Very high CPU utilization after upgrade from 5.9 to 5.12 (or beta 5.13)

This thread has been locked for further replies. You can start a new thread to share your ideas or ask questions.

Very high CPU utilization after upgrade from 5.9 to 5.12 (or beta 5.13)

This thread has been locked for further replies. You can start a new thread to share your ideas or ask questions.
Very high CPU utilization after upgrade from 5.9 to 5.12 (or beta 5.13)
Very high CPU utilization after upgrade from 5.9 to 5.12 (or beta 5.13)
2023-10-28 15:23:25 - last edited 2023-10-28 15:24:30
Tags: #Controller #Migration #Controller Upgrade
Model: EAP610   ER605 (TL-R605)  
Hardware Version: V2
Firmware Version: 2.1.2

Dear all

 

I'm running the Omada software as docker container on my VPS server with the following specs:

 

  • 1 vcpu
  • 2 GB RAM
  • 30G disk

 

Up until 5.9.31 this was working pretty smooth and the CPU utilization was close to 0%. After I upgraded from 5.9.31 to 5.12.7, CPU utilization spiked to 100% and also disk util increased and rendered the whole system unusable (it didn't recover even after days).

 

Even after upgrading to 2 vcpus the same situation was present. With 4 vcpus the situation improved (and the costs increased).

 

The CPU graphs pretty much show when exactly the upgrade was done. As you can see the CPU and disk utilization stayed that high for multiple days - so it's not just an upgrading issue.

 

Has anyone else experienced this after the upgrade?

 

I'm using these docker containers: https://hub.docker.com/r/mbentley/omada-controller

I also built a docker container for myself which included 5.13.10 - the situation stayed the same.

 

So I now went back to 5.9.31 and restored from a backup - smooth sailing again in this configuration.

 

Any ideas on what is causing this?

 

  1      
  1      
#1
Options
4 Reply
Re:Very high CPU utilization after upgrade from 5.9 to 5.12 (or beta 5.13)
2023-10-28 18:17:02

  @Raphi 

 

Never attempted a 1 core Omada controller install before.  I have it running as a VM in VMware, but it's 2 Core, 8GB, and 100GB of space, and it's cruising.  

 

I would be too scared to run it on 1 core, but I also know it has worked before for you.  

 

Do you have the guest portal enabled?  That can suck up some horsepower.  

I can not teach anyone anything - I can only make them think - Socrates
  0  
  0  
#2
Options
Re:Very high CPU utilization after upgrade from 5.9 to 5.12 (or beta 5.13)
2023-10-29 06:26:24

  @KimcheeGUN 

 

Good morning

 

We're talking about a 1 router, 1 AP home use setup right now, so no SLA whatever involved.

 

From a technical standpoint there has to be a reason why the new setup uses that much more resources. We're not talking about a bit more, the difference,:

 

Before 5.12: 1vcpu and disk usage barely 1%

On 5.12: 3vcpus 100% and disk fully utilized.

 

So the difference is huge!

  0  
  0  
#3
Options
Re:Very high CPU utilization after upgrade from 5.9 to 5.12 (or beta 5.13)
2023-10-29 13:05:50

  @Raphi 

 

I have 5.13.x running on Linux.  Also at 2 vcpu's.  CPU and Storage running at very minimal.  Something is def quirky going on there.  

 

You're up to 3 with the same issue.  Yikes!

 

@Clive_A can chime in?

 

 

I can not teach anyone anything - I can only make them think - Socrates
  0  
  0  
#4
Options
Re:Very high CPU utilization after upgrade from 5.9 to 5.12 (or beta 5.13)
2023-10-29 20:56:04

In the meantime I could dig a bit deeper on that issue and found out that the main problem isn't CPU here - it's actually memory.

 

It seems that the memory got fully used and then the kswapd0 tries to put some stuff into swap (that's where the IOPs are coming from) which kind of lead to the problems I've seen. This also explains why just increasing the vcpus didn't really help as the bottleneck stayed the same.

 

I re-created my docker container, restored from a backup and this time didn't enable cloud access - since then everything is working properly (memory footprint has been reduced from 0.9G to 0.8G which is just enough to work). However I don't consider that as a final solution - seems I just need more resources at the end.

 

So from my point of view this doesn't really seem to be an issue with the SW itself - it's just increased resource requirements. Based on the official requirements I'd call myself lucky that it's working at all with these reduced resources ;)

 

https://www.tp-link.com/us/support/faq/2967/

  0  
  0  
#5
Options