@amudd 

If the modem has more than one interface going into it then i would assume you are using a Gateway as the modem.  Is this correct?  Gateways are modem and router combo devices.  If so try putting the Deco in AP mode.  This will unfortunately disable all routing functions and your Deco would get IP assignments from the modem/router combo device but the double NAT would clear up and would likely the ping times.  Double NAT can play havoc on connections like this. 

Also i would consider a SMB level switch. If you have this many units on one ISP line using a switch to segment and assign separate VLANs to each unit could help improve overall network performance, and ensure that no unit can see another units' traffic.

@amudd 

If you don't have access to the interface of the gateway then yes a switch would not help.  I was under the impression you were the network owner and had access to it.  Another thing to try is connect your PC to the Ethernet cable and see if the pings are still spiking.  Its possible with the number of units connected to the gateway that the bottleneck is the gateway.  If you have high pings directly to the computer then that would cofrim this.