Omada Hardware Controller fails to update any device firmware on remote sites
My OC200 used to only manage a local site, now it manages 3 sites with a total of 3 gateways and 25 APs. The controllers is located in one of the sites (behind one of the gateways) alongside 18 of the APs.
The second site resulted from migrating it from a software controller that runned on that newwork managing one gateway and 5 APs,
The third site is a new site with only one gateway and 2 APs.
ISP router is in bridge mode in all 3 sites.
Recently there have been a great number of new firmware releases. I've never had a problem updating firmwares before on any of the sites when they were stanalone, but now I can't get the firmware upload to work on any of the devices on any of the remote sites, while it works flawleslly on all devices in the local site. Phisically moving a device from remote to local site (plus forgetting and adopting) lets the device update to succeed.
Googling the problem I found one has to foward certain ports. While this needed step should obviously be done by the omada controller automatically (and only during the updating process), I went ahead and forwareded the ports. Which ports one need to forward depends on what tp-link page you land on, so I've forwarded the ports described on any and all related tp-link pages, forums, reddits and those provided by tp-link support in response to the ticked I opened. Still it doesn't work. I've forwarded ports
8443
443
29810-29820 (currently only untill 29816 is needed, but since they've been adding more ports, I went ahead and left a few extra ports)
All TCP + UPD.
The devices use the controllers dns name, however the controller is ona a fixed public IP. I can see all devices in all sites in OC200. I can otherwise manage all the devices so why can't I just update them?
Also, NONE of the FW update methods work: Single device update, rolling update, manually updating the new firmware file. Manually updating gets stuck at 99%, then fails. His is both if using the web interface locally or through https://omada.tplinkcloud.com/, or android app.
References:
https://community.tp-link.com/en/business/forum/topic/559150
https://community.tp-link.com/en/business/forum/topic/656120
https://www.tp-link.com/en/support/faq/3281/
I'm at a loss. So is TP-Link support. After some back and forth emails they've requested access to my controller, but I'm not about to let that happen just yet for security reasons.
Does anyone has any further suggestions I might try?
Edit: DMZ the controller also didn't work, so it doesn't seem to be a port forwarding issue.