Omada Hardware Controller fails to update any device firmware on remote sites

Omada Hardware Controller fails to update any device firmware on remote sites

Omada Hardware Controller fails to update any device firmware on remote sites
Omada Hardware Controller fails to update any device firmware on remote sites
2024-06-19 22:15:24 - last edited 2024-06-19 23:09:44
Model: OC200  
Hardware Version: V1
Firmware Version: 1.29.21 Build 20240401 Rel.52515

My OC200 used to only manage a local site, now it manages 3 sites with a total of 3 gateways and 25 APs. The controllers is located in one of the sites (behind one of the gateways) alongside 18 of the APs.

 

The second site resulted from migrating it from a software controller that runned on that newwork managing one gateway and 5 APs,

 

The third site is a new site with only one gateway and 2 APs.

 

ISP router is in bridge mode in all 3 sites.

 

Recently there have been a great number of new firmware releases. I've never had a problem updating firmwares before on any of the sites when they were stanalone, but now I can't get the firmware upload to work on any of the devices on any of the remote sites, while it works flawleslly on all devices in the local site. Phisically moving a device from remote to local site (plus forgetting and adopting) lets the device update to succeed.

 

Googling the problem I found one has to foward certain ports. While this needed step should obviously be done by the omada controller automatically (and only during the updating process), I went ahead and forwareded the ports. Which ports one need to forward depends on what tp-link page you land on, so I've forwarded the ports described on any and all related tp-link pages, forums, reddits and those provided by tp-link support in response to the ticked I opened. Still it doesn't work. I've forwarded ports

8443

443

29810-29820 (currently only untill 29816 is needed, but since they've been adding more ports, I went ahead and left a few extra ports) 

All TCP + UPD.

 

The devices use the controllers dns name, however the controller is ona a fixed public IP. I can see all devices in all sites in OC200. I can otherwise manage all the devices so why can't I just update them?

 

Also, NONE of the FW update methods work: Single device update, rolling update, manually updating the new firmware file. Manually updating gets stuck at 99%, then fails. His is both if using the web interface locally or through https://omada.tplinkcloud.com/, or android app.

 

References:

https://community.tp-link.com/en/business/forum/topic/559150

https://community.tp-link.com/en/business/forum/topic/656120

https://www.tp-link.com/en/support/faq/3281/

 

I'm at a loss. So is TP-Link support. After some back and forth emails they've requested access to my controller, but I'm not about to let that happen just yet for security reasons.

 

Does anyone has any further suggestions I might try?

 

Edit: DMZ the controller also didn't work, so it doesn't seem to be a port forwarding issue.

  0      
  0      
#1
Options
16 Reply
Re:Omada Hardware Controller fails to update any device firmware on remote sites
2024-06-20 04:09:50

  @Tintronic 

 

you must open port 443 to the controller, if you have changed the managed port to something else, that port must be opened.
if you do not get the device updated, there is a 99% chance that the remote device does not have access to the controller management port.

 

so double check that this is correct

 

 

  0  
  0  
#2
Options
Re:Omada Hardware Controller fails to update any device firmware on remote sites
2024-06-20 09:12:05

  @Tintronic 

All of the information you search up is useful. Alternatively, you may paste your setup below so that we can check to see if anything is wrong. Please show us which port you use for management.

Best Regards! >> Omada EAP Firmware Trial Available Here << >> Get the Latest Omada SDN Controller Releases Here << *Try filtering posts on each forum by Label of [Early Access]*
  0  
  0  
#3
Options
Re:Omada Hardware Controller fails to update any device firmware on remote sites
2024-06-24 22:28:58

  @Tintronic 

Hi Tintronic,

This works in my case, I have OC300 and I have 2 sites, with a single controller in one site.

I have opened on the controller site 3 kind of ports for the Controller IP (Static One):

443 TCP

29811-29814 TCP

29810 UDP

 

I have also set a VPN through OpenVPN Server-Client on the Router IP (Static One):

500 UDP

4500 UDP

 

 

On the remote site, instead, I have this on the Router IP (Static One):

500 UDP

4500 UDP

 

 

In this way, I can reach the IP of the controller when I'm in the remote site.

Also, be sure to have different IPs for everything in order to avoid overlaps.

 

Thank you,

Fra

  0  
  0  
#4
Options
Re:Omada Hardware Controller fails to update any device firmware on remote sites
2024-08-21 22:51:36

  @nicolati 

Thanks for the suggestion.

 

While I haven't had time to pursue this workaround and I do want to VPN some of the networks, what you suggest is a workaround and thus doesn't address the origin of the problem.

 

Also, the OpenVPN throughput of most omada routers (at least the SOHO ones) is quite low, not even 50Mbps), which is way to low for my VPN purposes.

  0  
  0  
#5
Options
Re:Omada Hardware Controller fails to update any device firmware on remote sites
2024-08-21 23:34:03

UPDATE:

This is going from stange to strangest:

 

I hastely added a 4th site about 1000km away, behind a non-omada router on a different ISP. I connected a brand new EAP110-Outdoor, set it up and adopted it to my central Omada OC200 controller. It showed a new firmware and I went ahead with the update process not expecting it to work.

 

BUT IT WORKED RIGHT AWAY!

 

Maybe they fixed the issue? So I tried updating a device on one of the omada routed sites, this failed as it has always failed.

 

So this would mean there is a problem on the omada routers on the remote sites that prevents the firmware update process from completing! Nothing to do with the router on the controller site!

 

The logical step then is to DMZ one of the devices on the remote site and try to update its firmware.

 

I did this, but it DIDN'T work.

 

This is just baffling.

 

The only thing left is some ISP related port filtering.

  0  
  0  
#6
Options
Re:Omada Hardware Controller fails to update any device firmware on remote sites
2024-08-25 21:58:22

  @nicolati 

This morning I used Omada easy way to VPN remote sites to the controller hosted site es per https://www.tp-link.com/us/support/faq/3049/

 

 

And confirmed the connection:

 

 

While I'm now able to access this remote site LAN, firmware update of remote APs still fails.

  0  
  0  
#7
Options
Re:Omada Hardware Controller fails to update any device firmware on remote sites
2024-08-25 22:12:55 - last edited 2024-08-25 22:19:03

 Hi @Hank21 

Hank21 wrote

  @Tintronic 

All of the information you search up is useful. Alternatively, you may paste your setup below so that we can check to see if anything is wrong. Please show us which port you use for management.

 

Controller config:

 

Device Management Hostname is the name of my public IP

NAT on controller hosted site

 

WAN2 is the fiber optic ISP with fixed public IP.

WAN/LAN3 is irrelevant as it is a Starlink backup WAN with CGNAT so no public IP.

 

I also added port 19810 since I recently got a message that stated this port, along the 291xx ports, need to be available for FW update. I don't remeber how I got this message.

 

Even if I DMZ the controller, still the firmware update process fails.

 

I recently posted that I created a 4th remote site with no omada router, just the (quite restrictive) ISP router, and was able to adopt and update the frimware using this same controller, so there must be something on both remote site omada routers messing with the device FW update process.

 

I even DMZ one of the remote site AP with pending FW update, but still the update failed.

  0  
  0  
#8
Options
Re:Omada Hardware Controller fails to update any device firmware on remote sites
2024-08-28 06:47:57

  @Tintronic 

Would you kindly see if you are able to upgrade the firmware locally? Do you have access to it? It appears from what you said that you are not even able to upgrade the firmware locally.

Best Regards! >> Omada EAP Firmware Trial Available Here << >> Get the Latest Omada SDN Controller Releases Here << *Try filtering posts on each forum by Label of [Early Access]*
  0  
  0  
#9
Options
Re:Omada Hardware Controller fails to update any device firmware on remote sites
2024-08-30 23:21:49

Hank21 wrote

  @Tintronic 

Would you kindly see if you are able to upgrade the firmware locally? Do you have access to it? It appears from what you said that you are not even able to upgrade the firmware locally.

 

@Hank21 

Once an Omada AP is adoptend into a controller, it cannot be upgraded any other way but through the controllers interface (web browser or android app), unless it is "forgotten" (unbinded) from the controller. While this may seem trivial for APs, it is not the case for the router for which there is also a firmware update available. As soon as I forget the router, that network will lose internet connection (and thus VPN) making firmware upgrade impossible.

 

There are a couple more reasons why I haven't done that, both aiming at solving the problem in question instead of using a workaround that will keep the problem from happening.

 

For one, firmware failing to upgrade is not a problem with only this remote site, I was unable to update firmare on the other remote site that also has an (different model) omada router. So the problem is not one site or router model, it's both remote sites. So it is not a problem with a specific router model.

 

Secondly, to discard a problem with the devices themselves, I forgot and removed 3 different model devices from both remote sites, connected and adopted them to the local (controller hosted) site and managed to upgrade their firmware flawlessly. So it is not a problem with a particular AP model nor firmware.

 

Lastly, I recently added a 3rd remote site, this time without an omada router, adopted a single brand new unboxed AP, and was able to update it without a single problem.

 

In conclusion:

- It is not a problem with controller hosted port forwarding

- It is not a problem with the remote devices

- It is unique to omada-routed remote sites.

 

I'm guessing not many clients use a single HW router to manage remote sites.

  0  
  0  
#10
Options
Re:Omada Hardware Controller fails to update any device firmware on remote sites
2024-08-31 12:23:32

  @Tintronic 

I do, but I prefer doing it only when I'm on site.

 

However, technically speaking, I think it's like the upgrade is happening on a remote site, because I upgrade it by connecting to Omada Controller connected in VPN, which is in the main site, different to the one where the ER605 V2 router is located (and where I'm temporary phisically).

So, if my assumption is true, I should be in the same your case, but upgrade process gets through.

I download the firmware manually and upload it through the controller interface of the router device.

 

Thank you,

Fra

  0  
  0  
#11
Options