@APRC-P3-Tel :  I have a problem on on our controllers which is now running 51 APs. We are collecting the logs from this controller on a centralized syslog server. Sometimes the controller will just oscilate between LAN connection up and down for few minutes in a day (many times during off-peak hours):

Here the Controller MAC 50-D4-F7-1C-3F-1E having 51 APs connected to it is facing this problem and the log report from the switch (also collected at centralized server) is listed below:

 Mar  2 07:20:04     :port 13 link down
 Mar  2 07:20:06     :Port 13 link up, 100Mbps FULL duplex
 Mar  2 07:20:10     :port 13 link down
 Mar  2 07:20:12     :Port 13 link up, 100Mbps FULL duplex
 Mar  2 07:20:16     :port 13 link down
 Mar  2 07:20:18     :Port 13 link up, 100Mbps FULL duplex
 Mar  2 07:20:22     :port 13 link down
 Mar  2 07:20:24     :Port 13 link up, 100Mbps FULL duplex
 Mar  2 07:20:28     :port 13 link down
 Mar  2 07:20:30     :Port 13 link up, 100Mbps FULL duplex
 Mar  2 07:20:33     :port 13 link down
 Mar  2 07:20:35     :Port 13 link up, 100Mbps FULL duplex
 Mar  2 07:20:39     :port 13 link down
 Mar  2 07:20:41     :Port 13 link up, 100Mbps FULL duplex
 Mar  2 07:20:45     :port 13 link down
 Mar  2 07:20:47     :Port 13 link up, 100Mbps FULL duplex
 Mar  2 07:20:51     :port 13 link down
 Mar  2 07:20:53     :Port 13 link up, 100Mbps FULL duplex
 Mar  2 07:20:57     :port 13 link down
 Mar  2 07:20:59     :Port 13 link up, 100Mbps FULL duplex
 Mar  2 07:21:03     :port 13 link down
 Mar  2 07:21:05     :Port 13 link up, 100Mbps FULL duplex
 Mar  2 07:21:09     :port 13 link down
 Mar  2 07:21:11     :Port 13 link up, 100Mbps FULL duplex
 Mar  2 07:21:15     :port 13 link down
 Mar  2 07:21:17     :Port 13 link up, 100Mbps FULL duplex
 Mar  2 07:21:21     :port 13 link down
 Mar  2 07:21:23     :Port 13 link up, 100Mbps FULL duplex
 Mar  2 07:21:27     :port 13 link down
 Mar  2 07:21:29     :Port 13 link up, 10Mbps HALF duplex
 Mar  2 07:21:36     :port 13 link down
 Mar  2 07:21:38     :Port 13 link up, 100Mbps FULL duplex
 Mar  2 07:22:45     :port 13 link down
 Mar  2 07:22:47     :Port 13 link up, 100Mbps FULL duplex
 Mar  2 07:22:51     :port 13 link down
 Mar  2 07:22:53     :Port 13 link up, 100Mbps FULL duplex
 Mar  2 07:22:57     :port 13 link down
 Mar  2 07:22:59     :Port 13 link up, 100Mbps FULL duplex
 Mar  2 07:23:03     :port 13 link down
 Mar  2 07:23:05     :Port 13 link up, 100Mbps FULL duplex
 Mar  2 07:23:09     :port 13 link down
 Mar  2 07:23:11     :Port 13 link up, 100Mbps FULL duplex
 Mar  2 07:23:14     :port 13 link down
 Mar  2 07:23:17     :Port 13 link up, 10Mbps HALF duplex
 Mar  2 07:23:23     :port 13 link down
 Mar  2 07:23:25     :Port 13 link up, 100Mbps FULL duplex

I have verified from the switch fiorwarding table that this Controller MAC is connected indeed to Port 13 on the switch. The Controller log (collected at centrralized server) shows following abnormalities for the day (likely at this time only as all 51 APs, emiited ):

1 2020-03-02T01:55:18.464Z - Omada Controller - - Mayflower-R-S(98-DA-C4-58-9F-9E) connected to LAN
1 2020-03-02T01:55:18.516Z - Omada Controller - - Lotus-Hibiscus Walkway(98-DA-C4-97-09-62) connected to LAN
1 2020-03-02T01:55:18.546Z - Omada Controller - - Jacaranda-K(98-DA-C4-96-F9-02) connected to LAN
1 2020-03-02T01:55:19.438Z - Omada Controller - - Jacaranda-G(98-DA-C4-96-F9-74) connected to LAN
1 2020-03-02T01:55:19.508Z - Omada Controller - - Mayflower-S(98-DA-C4-58-9C-FE) connected to LAN
1 2020-03-02T01:55:20.080Z - Omada Controller - - Hibiscus-L-M(98-DA-C4-58-8C-56) connected to LAN
1 2020-03-02T01:55:20.956Z - Omada Controller - - Daffodils-B-C(68-FF-7B-CA-FD-48) connected to LAN
1 2020-03-02T01:55:21.048Z - Omada Controller - - Lotus-Rear(98-DA-C4-58-83-9E) connected to LAN
1 2020-03-02T01:55:21.218Z - Omada Controller - - Maple-T(98-DA-C4-96-F9-E8) connected to LAN
1 2020-03-02T01:55:22.091Z - Omada Controller - - Bougenvilla-B(98-DA-C4-58-84-20) connected to LAN
1 2020-03-02T01:55:22.292Z - Omada Controller - - Bougenvilla-A-B(98-DA-C4-58-8D-38) connected to LAN
1 2020-03-02T01:55:22.497Z - Omada Controller - - Clubhouse Side (Near Lotus)(98-DA-C4-58-8D-3A) connected to LAN
1 2020-03-02T01:55:23.456Z - Omada Controller - - Mayflower-N-P(98-DA-C4-97-09-8A) connected to LAN
1 2020-03-02T01:55:23.644Z - Omada Controller - - Hibiscus-M(98-DA-C4-58-88-8E) connected to LAN
1 2020-03-02T01:55:23.721Z - Omada Controller - - Bougenvilla-E(98-DA-C4-96-FC-50) connected to LAN
1 2020-03-02T01:55:24.524Z - Omada Controller - - Jacaranda-H(98-DA-C4-97-02-2E) connected to LAN
1 2020-03-02T01:55:24.708Z - Omada Controller - - Daffodils-G-H(68-FF-7B-CA-F3-8E) connected to LAN
1 2020-03-02T01:55:25.076Z - Omada Controller - - Mayflower-Q-R(98-DA-C4-96-FA-B8) connected to LAN
1 2020-03-02T01:55:25.403Z - Omada Controller - - Jasmine-W(98-DA-C4-97-02-58) connected to LAN
1 2020-03-02T01:55:25.629Z - Omada Controller - - Bougenvilla-D(98-DA-C4-58-83-C2) connected to LAN
1 2020-03-02T01:55:25.640Z - Omada Controller - - Daffodils-H Outer(68-FF-7B-CA-F3-AA) connected to LAN
1 2020-03-02T01:55:25.860Z - Omada Controller - - Mayflower-N(98-DA-C4-58-8C-E0) connected to LAN
1 2020-03-02T01:55:26.122Z - Omada Controller - - Bougenvilla-E-F(98-DA-C4-96-FC-A0) connected to LAN
1 2020-03-02T01:55:26.208Z - Omada Controller - - Jacaranda-J(98-DA-C4-97-09-52) connected to LAN
1 2020-03-02T01:55:26.446Z - Omada Controller - - Jacaranda-G-H(98-DA-C4-97-09-D2) connected to LAN
1 2020-03-02T01:55:26.960Z - Omada Controller - - Bougenvilla-F(98-DA-C4-58-84-18) connected to LAN
1 2020-03-02T01:55:26.978Z - Omada Controller - - Clubhouse Side (Near Hibiscus-M)(98-DA-C4-58-8D-2C) connected to LAN
1 2020-03-02T01:55:26.995Z - Omada Controller - - Visitor Parking - Bougenvilla-E-F(CC-32-E5-B5-53-10) connected to LAN
1 2020-03-02T01:55:27.464Z - Omada Controller - - Hibiscus-L(98-DA-C4-58-88-9C) connected to LAN
1 2020-03-02T01:55:27.476Z - Omada Controller - - Jacaranda-J-K(B0-BE-76-53-CC-1A) connected to LAN
1 2020-03-02T01:55:27.486Z - Omada Controller - - Clubhouse Reception(D8-0D-17-B7-BE-F8) connected to LAN
1 2020-03-02T01:55:27.896Z - Omada Controller - - Bougenvilla-A(98-DA-C4-96-FB-F4) connected to LAN
1 2020-03-02T01:55:27.957Z - Omada Controller - - Daffodils-D(68-FF-7B-CA-FD-E0) connected to LAN
1 2020-03-02T01:55:28.008Z - Omada Controller - - Mayflower-P-Q(98-DA-C4-58-88-DC) connected to LAN
1 2020-03-02T01:55:28.324Z - Omada Controller - - Maple-T-U(98-DA-C4-58-8C-20) connected to LAN
1 2020-03-02T01:55:28.438Z - Omada Controller - - Daffodils-F-G(68-FF-7B-CA-F9-00) connected to LAN
1 2020-03-02T01:55:28.568Z - Omada Controller - - Daffodils-A(68-FF-7B-CA-FA-D0) connected to LAN
1 2020-03-02T01:55:28.751Z - Omada Controller - - Bougenvilla-D (Towards Park)(98-DA-C4-58-84-08) connected to LAN
1 2020-03-02T01:55:28.755Z - Omada Controller - - Bougenvilla-C(98-DA-C4-58-9F-F2) connected to LAN
1 2020-03-02T01:55:28.972Z - Omada Controller - - Clubhouse Entrance Front(98-DA-C4-97-09-8C) connected to LAN
1 2020-03-02T01:55:29.169Z - Omada Controller - - Maple-U(98-DA-C4-96-F9-72) connected to LAN
1 2020-03-02T01:55:29.249Z - Omada Controller - - Lotus-Front(98-DA-C4-58-8D-12) connected to LAN
1 2020-03-02T01:55:29.434Z - Omada Controller - - Bougenvilla-C (Towards Park)(98-DA-C4-97-02-30) connected to LAN
1 2020-03-02T01:55:29.612Z - Omada Controller - - Visitor Parking - Jacaranda-G-H(CC-32-E5-B5-53-9C) connected to LAN
1 2020-03-02T01:55:29.626Z - Omada Controller - - Lotus Approach(98-DA-C4-97-08-B0) connected to LAN
1 2020-03-02T01:55:29.870Z - Omada Controller - - Visitor Parking - Jacaranda-J-K(CC-32-E5-B5-50-AE) connected to LAN
1 2020-03-02T01:55:30.022Z - Omada Controller - - Jasmine-V-W(98-DA-C4-96-FE-46) connected to LAN
1 2020-03-02T01:55:30.027Z - Omada Controller - - Clubhouse Basement Corridor(68-FF-7B-CA-FE-66) connected to LAN
1 2020-03-02T01:55:30.158Z - Omada Controller - - Daffodils-E(68-FF-7B-CA-F1-DC) connected to LAN
1 2020-03-02T01:55:30.355Z - Omada Controller - - Jasmine-V(98-DA-C4-96-F9-A6) connected to LAN
1 2020-03-02T01:55:30.428Z - Omada Controller - - Maple-T (Special)(98-DA-C4-58-A2-40) connected to LAN

The APs above just emitted a connection status. No IP and mask assignment log. The uptime of the APs also shows that these APs did not reboot or restart together at this time (rulls out power failure possibilitry also).

So is overload of APs (51 against max recommendation of 50) triggering this ?

During daytime the average user load on this 51 AP system in around 20, while the peak shoots to about 40 which is not even 1 user per AP on average. The number of users connecting to any AP never exceeds 5, which is defintely nota  problem for EAP225s (which are used here).

@APRC-P3-Tel 

It is recommended that the number of APs should not more than 50, some customers use the OC200 to manage more than 50 APs and they can also work normally. But if the traffic of the AP is very large, we still recommend you to manage less than 50 APs. 

For the logs you see on the Controller, they mean the AP is managed by the OC200. Normally we will see these logs when we adopt the AP first or the AP is rebooted. 

For the log of port 13, I guess there is something worng with the connection between the OC200 and switch. We recommend you to change the cable and another port to have a try. By the way, how about the status of the OC200? It is solid green?

No other logs of controller, its LAN port light is always solid green. The problem happens for about 3 minutes on some days, at no particular fixed time. Somedays it does not happen. If I see controller uptime, its clear that around the time the logs came, the controller has rebooted. But why it has rebooted, it was not able to log.

Also our APs are now set to reboot on 1st of every month, 4:00 AM against every day earlier. We noticed that few of the APs out of 51 autonomously reboot sometimes within the 1 month period. Its very random. And this reboot is not related to power but software related. And yes their are no logs for that also.

That's why my initial suspicion on overload ...

Our load pattern is like this

Number of users registered with this OC200 system ~1400

Number of APs being handled = 51 // ~50 is recommended value.
Normal user load ~20
Peak Hour user load ~40
maximum speed per user allowed ~ 2 mbps DL/UL

For  a 51 AP system, this type of daily pattern (almost similar every day), is the system really overloaded ??? and if yes, on what resource ?

forrest wrote

@APRC-P3-Tel 

 

It is recommended that the number of APs should not more than 50, some customers use the OC200 to manage more than 50 APs and they can also work normally. But if the traffic of the AP is very large, we still recommend you to manage less than 50 APs. 

 

For the logs you see on the Controller, they mean the AP is managed by the OC200. Normally we will see these logs when we adopt the AP first or the AP is rebooted. 

For the log of port 13, I guess there is something worng with the connection between the OC200 and switch. We recommend you to change the cable and another port to have a try. By the way, how about the status of the OC200? It is solid green?

@forrest 

@R1D2 :  The contractor who installed the system is a Windows *only guy. If I do install a linux system, and run the software for him and something does go wrong, then I have to again intervene and repair system for him. Others who administer system, use cloud or mobile app interface and are also not exactly linux comfortable. That is what is keeping me away from introducting a linux system for running controller.  However, I do have an atom D2550 (Dual Core, 1.8 Ghz) 1U micro server (CentOS 8.1) with 4G RAM, lying  spare and we can try it out if no other option. Is this configuration suitable ?

The windows PC we have does other function like network management, health monitoring, CCTV network management,  etc and is rebooted for unrelated reasons like OS updates, AV engine updates, some driver updates and what not. Also its motherboard does not recover from power shutdowns. This is why we moved from PC to OC200 in the first place.

To be frank, I am actually looking at a OC200ng/v2.0 or something which we can support 4X (~200) users atleast. Its a really a no worries kind of device. Simplicity of using and maintaining an appliance is what makes me reject both Windows and Linux as first choice for running controller application

R1D2 wrote

@APRC-P3-Tel, why don't you set up a Linux server with much larger processing capabilities and a Gigabit interface if you need a central server to serve 50+ EAPs? In my opinion the software server has been designed especially for this use scenario while OC200 has been designed primarily for stand-alone installations.

@R1D2 : Yes, I understand the OC200 is a tiny Linux based SoC board. But its advantage is low power, and Just Enough OS, wherein you can just power cycle the device if something goes wrong (hoping that no dynamic database data gets corrupted beyond self-appair on reboot). With a general purpose Linux OS on x64 hardware, I am not sure that i would have same level of confidence with os and other bundled services/terminals apps, etc getting updated independently of Omada application. Sooner or later something may break.

BTW, the Omada Controller User Guide recommends that:

Omada Controller can manage up to 1500 EAPs if the Controller Host has enough hardware resources.

To guarantee operational stability for managing 1500 EAPs, we recommend that you use the hardware which meets or exceeds the following specifications:

CPU: Intel Core i3-8100, i5-6500, or i7-4700 with 2 or more cores and 4 or more threads.Memory: 6 GB RAM or more

This is a much costly affair also, than a OC200 V1 (or V2/ng in future) besides running more hotter, taking more rack footprint, etc. I do have a 1U rack microserver spare with following Config

Motherboard+CPU Combo - Giada MI-NAS25 with Intel Atom D2550 2C4T 1.86 Ghz & 2xGbE networking

RAM - 4 GB SO-DIMM DDR3 RAM

Secondary Storage - 2xSSD (64 GB) for boot drive configured in Software-RAID-1 mode

OS - Linux Distro CentOS 8.1 Server Edition

PSU - Seasonic 250W SMPS [seems to run very cool without sipping too much power (sub 20W i think for this system)]

Would it be able to handle about 75-100 EAPs ? Am only worried about the CPU ...

I need to expand current 51 AP system by another 20-25 APs  in 2020 sometime. Am not able to split among multiple OC200 beacuse multiple OC200 cannot share configuration,  and we need smooth mobility between all these 75 APs

If no new powerful OC200 type appliance, I will have to try the above hardware first.

R1D2 wrote

@APRC-P3-Tel,

 

with OC200 you already introduced a Linux system, albeit a Linux running on a tiny System-on-Chip (SoC) with limited resources, but I understand your concerns. If used as a server, Windows is a real pain compared to Linux, but if you are more comfortable to administrate a Windows server, you can run Omada controller on Windows, too.

 

In my opinion, a Linux server is much more stable compared to Windows and once a Linux server has been set up it runs for years w/o problems. Even software updates would not disrupt the operation.

 

You also could use a managed server system or a cloud instance in a data-center to install Omada controller there.

 

Another alternative is two split up the number of EAPs on two or more OC200.

 

 

I am now  getting much severe form of this problem. Controller downtime freqiuency of 2-3 times per day and for 2+ hours and at random times. Like Yesterday, the downtime is

Hi @APRC-P3-Tel,

"Am not able to split among multiple OC200 beacuse multiple OC200 cannot share configuration,  and we need smooth mobility between all these 75 APs"

I don't agree with your logic here.  While you are correct that two OC-200's cannot "share" their configuration.  You can export the settings from one, and import them into the other.  Or just setup both by hand, it's very easy and you only have to do it once.

There is no reason why clients cannot move smothly between these segments.  Surely not all 50+ EAPs are in the same room anyway?  Unless this is an auditorium or stadium.

Clients can roam from AP to AP without the OC-200 at all.  Granted the controller is needed for "Fast Roaming" but this new innovation is really oversold.  STA's have been roaming from AP to AP for more than a decade before Fast Roaming was invented.

But certainly if you want a more powerful Omada solution deploy it on a workstation, server, VM, or AWS instance.  That is why the software solution exisits.

As far as your wonky Ethernet connection, I also doubt this is in any way related to crossing the magic AP# threshold.  It is much more likely to be a bad, cracked, kinked cable, poor termination, or cracked soldering joint on one of the RJ45 ports (in the OC200 or your switch).

-Jonathan

@APRC-P3-Tel 

Re: Power - Why don't you use one of the PoE ports to power the OC200 or perhaps use the PoE Injector that came with one of the APs? This would solve your issue related power.

Re: PC - You could look at a Raspberry Pi 4 instead of a full blown PC. The cost would be similar to aother OC200 but you would have a higher CPU/RAM/Storage options with a very low power consumption. There are insturction on this thread how to install the controller on Raspberry Pi: https://community.tp-link.com/en/business/forum/topic/162210