Omada controler offline for minutes
This thread has been locked for further replies. You can start a new thread to share your ideas or ask questions.
Omada controler offline for minutes
Hi,
I have two EAP225 access points and I'm running the Omada Controller as a docker image on my Synology NAS. Every couple of days, the controller and EAP's go offline. I have no WiFi signal for a couple of minutes and then everything comes back online again. I checked that I still have internet access on the wired network during this time.
Below's a snippet of the log during the most recent outage. (Note that it is in reversed order: the most recent log line is at the top.)
Any help in resolving this issue would be highly appreciated!
Best regards,
Bart
2021-01-06 08:03:41,stdout,2021-01-06 09:03:41 [scheduled-pool-2] [INFO]-[SourceFile:80] - Start pushing new connected devices. 2021-01-06 08:03:35,stdout,2021-01-06 09:03:35 [Thread-15] [INFO]-[SourceFile:366] - The result of connection is true. 2021-01-06 08:03:11,stdout,2021-01-06 09:03:11 [discovery-handler-pool-0] [INFO]-[SourceFile:129] - old connection Adopting of 1C-3B-F3-A1-4C-A0 is not timeout 2021-01-06 08:03:11,stdout,2021-01-06 09:03:11 [discovery-handler-pool-1] [INFO]-[SourceFile:129] - old connection Adopting of 1C-3B-F3-A1-4C-A0 is not timeout 2021-01-06 08:03:11,stdout,2021-01-06 09:03:11 [discovery-handler-pool-1] [INFO]-[SourceFile:129] - old connection Adopting of 1C-3B-F3-A1-4C-A0 is not timeout 2021-01-06 08:01:19,stdout,2021-01-06 09:01:19 [net-event-handler-pool-11] [WARN]-[SourceFile:133] - adopt timeout for 74-DA-88-D8-93-D4. 2021-01-06 08:01:19,stdout,2021-01-06 09:01:19 [device-timeout-service-0] [WARN]-[SourceFile:94] - Adopting 74-DA-88-D8-93-D4 time out 2021-01-06 08:00:01,stdout,2021-01-06 09:00:00 [net-event-handler-pool-12] [WARN]-[SourceFile:133] - adopt timeout for 74-DA-88-D8-93-D4. 2021-01-06 08:00:00,stdout,2021-01-06 08:59:59 [device-timeout-service-0] [WARN]-[SourceFile:94] - Adopting 74-DA-88-D8-93-D4 time out 2021-01-06 07:58:09,stdout,2021-01-06 08:58:09 [net-event-handler-pool-4] [WARN]-[SourceFile:133] - adopt timeout for 74-DA-88-D8-93-D4. 2021-01-06 07:58:09,stdout,2021-01-06 08:58:09 [device-timeout-service-0] [WARN]-[SourceFile:94] - Adopting 74-DA-88-D8-93-D4 time out 2021-01-06 07:58:06,stdout,2021-01-06 08:58:06 [Thread-15] [INFO]-[SourceFile:320] - Exception occurs when connecting service server:java.net.UnknownHostException: n-devs-smb.tplinkcloud.com: Temporary failure in name resolution 2021-01-06 07:56:49,stdout,2021-01-06 08:56:49 [net-event-handler-pool-5] [WARN]-[SourceFile:165] - Failed to auto adopt 74-DA-88-D8-93-D4 5 times as timeout 2021-01-06 07:56:49,stdout,2021-01-06 08:56:49 [net-event-handler-pool-5] [WARN]-[SourceFile:133] - adopt timeout for 74-DA-88-D8-93-D4. 2021-01-06 07:56:49,stdout,2021-01-06 08:56:49 [device-timeout-service-0] [WARN]-[SourceFile:94] - Adopting 74-DA-88-D8-93-D4 time out 2021-01-06 07:56:41,stdout, at java.lang.Thread.run(Thread.java:748) [?:1.8.0_272] 2021-01-06 07:56:41,stdout, at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_272] 2021-01-06 07:56:41,stdout, at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_272] 2021-01-06 07:56:41,stdout, at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_272] 2021-01-06 07:56:41,stdout, at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_272] 2021-01-06 07:56:41,stdout, at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_272] 2021-01-06 07:56:41,stdout, at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_272] 2021-01-06 07:56:41,stdout, at com.tplink.omada.common.b.a.a.run(SourceFile:40) [omada-common-4.1.5.jar:?] 2021-01-06 07:56:41,stdout, at com.tplink.omada.cloud.common.device.task.DeviceStatusPushTask.b(SourceFile:57) [omada-cloud-4.1.5.jar:?] 2021-01-06 07:56:41,stdout, at com.tplink.omada.cloud.common.device.task.DeviceStatusPushTask.f(SourceFile:85) [omada-cloud-4.1.5.jar:?] 2021-01-06 07:56:41,stdout, at com.tplink.omada.cloud.platform.a.b.h.a(SourceFile:144) [omada-cloud-4.1.5.jar:?] 2021-01-06 07:56:41,stdout, at com.tplink.omada.cloud.platform.a.b.h.a(SourceFile:209) [omada-cloud-4.1.5.jar:?] 2021-01-06 07:56:41,stdout, at com.tplink.omada.cloud.platform.b.a(SourceFile:89) ~[omada-cloud-4.1.5.jar:?] 2021-01-06 07:56:41,stdout, at com.tplink.eap.cloudsdk.httpclient.ApiHttpClient.sendPostRequest(SourceFile:204) ~[cloudsdk-1.0.5.jar:?] 2021-01-06 07:56:41,stdout, at com.tplink.eap.cloudsdk.httpclient.ApiHttpClient.sendPostRequest(SourceFile:259) ~[cloudsdk-1.0.5.jar:?] 2021-01-06 07:56:41,stdout, at com.tplink.eap.cloudsdk.httpclient.ApiHttpClient.sendPostRequest(SourceFile:289) ~[cloudsdk-1.0.5.jar:?] 2021-01-06 07:56:41,stdout, at com.tplink.eap.cloudsdk.httpclient.ApiHttpClient.a(SourceFile:325) ~[cloudsdk-1.0.5.jar:?] 2021-01-06 07:56:41,stdout, at com.tplink.eap.cloudsdk.httpclient.a.sendPostRequest(SourceFile:148) ~[cloudsdk-1.0.5.jar:?] 2021-01-06 07:56:41,stdout, at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[httpclient-4.5.7.jar:4.5.7] 2021-01-06 07:56:41,stdout, at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.7.jar:4.5.7] 2021-01-06 07:56:41,stdout, at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.7.jar:4.5.7] 2021-01-06 07:56:41,stdout, at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.7.jar:4.5.7] 2021-01-06 07:56:41,stdout, at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.7.jar:4.5.7] 2021-01-06 07:56:41,stdout, at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[httpclient-4.5.7.jar:4.5.7] 2021-01-06 07:56:41,stdout, at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.7.jar:4.5.7] 2021-01-06 07:56:41,stdout, at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.7.jar:4.5.7] 2021-01-06 07:56:41,stdout, at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374) ~[httpclient-4.5.7.jar:4.5.7] 2021-01-06 07:56:41,stdout, at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112) ~[httpclient-4.5.7.jar:4.5.7] 2021-01-06 07:56:41,stdout, at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45) ~[httpclient-4.5.7.jar:4.5.7] 2021-01-06 07:56:41,stdout, at java.net.InetAddress.getAllByName(InetAddress.java:1127) ~[?:1.8.0_272] 2021-01-06 07:56:41,stdout, at java.net.InetAddress.getAllByName(InetAddress.java:1193) ~[?:1.8.0_272] 2021-01-06 07:56:41,stdout, at java.net.InetAddress.getAllByName0(InetAddress.java:1277) ~[?:1.8.0_272] 2021-01-06 07:56:41,stdout, at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324) ~[?:1.8.0_272] 2021-01-06 07:56:41,stdout, at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929) ~[?:1.8.0_272] 2021-01-06 07:56:41,stdout, at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[?:1.8.0_272] 2021-01-06 07:56:41,stdout,java.net.UnknownHostException: n-smb-api.tplinkcloud.com: Temporary failure in name resolution 2021-01-06 07:56:41,stdout,2021-01-06 08:56:41 [scheduled-pool-9] [WARN]-[SourceFile:214] - Failed to push message [omadaEAPStatus]. 2021-01-06 07:56:20,stdout,2021-01-06 08:56:20 [scheduled-pool-9] [INFO]-[SourceFile:84] - Start pushing new disconnected devices. 2021-01-06 07:55:29,stdout,2021-01-06 08:55:29 [net-event-handler-pool-6] [WARN]-[SourceFile:133] - adopt timeout for 74-DA-88-D8-93-D4. 2021-01-06 07:55:29,stdout,2021-01-06 08:55:29 [device-timeout-service-0] [WARN]-[SourceFile:94] - Adopting 74-DA-88-D8-93-D4 time out 2021-01-06 07:55:19,stdout,2021-01-06 08:55:19 [net-event-handler-pool-9] [INFO]-[SourceFile:185] - handleConnectedTimeout for 1C-3B-F3-A1-4C-A0 2021-01-06 07:55:19,stdout,2021-01-06 08:55:19 [device-timeout-service-0] [INFO]-[SourceFile:121] - Connected device time out for 1C-3B-F3-A1-4C-A0 2021-01-06 07:55:02,stdout,2021-01-06 08:55:02 [Thread-15] [INFO]-[SourceFile:320] - Exception occurs when connecting service server:java.net.UnknownHostException: n-devs-smb.tplinkcloud.com: Temporary failure in name resolution 2021-01-06 07:53:59,stdout,2021-01-06 08:53:59 [net-event-handler-pool-2] [WARN]-[SourceFile:133] - adopt timeout for 74-DA-88-D8-93-D4. 2021-01-06 07:53:59,stdout,2021-01-06 08:53:59 [device-timeout-service-0] [WARN]-[SourceFile:94] - Adopting 74-DA-88-D8-93-D4 time out 2021-01-06 07:53:19,stdout,2021-01-06 08:53:19 [Thread-15] [INFO]-[SourceFile:320] - Exception occurs when connecting service server:java.net.UnknownHostException: n-devs-smb.tplinkcloud.com: Temporary failure in name resolution 2021-01-06 07:52:29,stdout,2021-01-06 08:52:29 [net-event-handler-pool-7] [WARN]-[SourceFile:133] - adopt timeout for 74-DA-88-D8-93-D4. 2021-01-06 07:52:29,stdout,2021-01-06 08:52:29 [device-timeout-service-0] [WARN]-[SourceFile:94] - Adopting 74-DA-88-D8-93-D4 time out 2021-01-06 07:52:18,stdout,2021-01-06 08:52:18 [Thread-15] [INFO]-[SourceFile:320] - Exception occurs when connecting service server:java.net.UnknownHostException: n-devs-smb.tplinkcloud.com: Temporary failure in name resolution 2021-01-06 07:51:58,stdout,"2021-01-06 08:51:58 [Thread-15] [INFO]-[SourceFile:127] - Connect service server automatically, ConnectionType is PERSISTENT_CONNECTION. " 2021-01-06 07:51:58,stdout,2021-01-06 08:51:58 [Thread-15] [INFO]-[SourceFile:114] - Close connection to SEF server. 2021-01-06 07:51:58,stdout,"2021-01-06 08:51:58 [Thread-15] [INFO]-[SourceFile:194] - The result of connection is false, service host is null, port is 0. " 2021-01-06 07:51:58,stdout,2021-01-06 08:51:58 [Thread-15] [INFO]-[SourceFile:155] - Exception occurs when connecting SEF server:java.net.UnknownHostException: n-deventry-smb.tplinkcloud.com: Temporary failure in name resolution 2021-01-06 07:51:28,stdout,2021-01-06 08:51:28 [Thread-15] [INFO]-[SourceFile:155] - Exception occurs when connecting SEF server:java.net.UnknownHostException: n-deventry-smb.tplinkcloud.com: Temporary failure in name resolution 2021-01-06 07:51:10,stdout,2021-01-06 08:51:10 [net-event-handler-pool-0] [WARN]-[SourceFile:133] - adopt timeout for 74-DA-88-D8-93-D4. 2021-01-06 07:51:09,stdout,2021-01-06 08:51:09 [device-timeout-service-0] [WARN]-[SourceFile:94] - Adopting 74-DA-88-D8-93-D4 time out 2021-01-06 07:51:03,stdout,2021-01-06 08:51:03 [Thread-15] [INFO]-[SourceFile:155] - Exception occurs when connecting SEF server:java.net.NoRouteToHostException: No route to host (Host unreachable) 2021-01-06 07:51:02,stdout,2021-01-06 08:51:02 [Thread-19] [INFO]-[SourceFile:839] - Thread 'recvThread' is stopped 2021-01-06 07:51:02,stdout, ... 3 more 2021-01-06 07:51:02,stdout, at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:815) ~[?:1.8.0_272] 2021-01-06 07:51:02,stdout, at sun.security.ssl.SSLSocketImpl.access$200(SSLSocketImpl.java:72) ~[?:1.8.0_272] 2021-01-06 07:51:02,stdout, at sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1095) ~[?:1.8.0_272] 2021-01-06 07:51:02,stdout, at sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:68) ~[?:1.8.0_272] 2021-01-06 07:51:02,stdout, at sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:457) ~[?:1.8.0_272] 2021-01-06 07:51:02,stdout, at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_272] 2021-01-06 07:51:02,stdout, at java.net.SocketInputStream.read(SocketInputStream.java:171) ~[?:1.8.0_272] 2021-01-06 07:51:02,stdout, at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[?:1.8.0_272] 2021-01-06 07:51:02,stdout, at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_272] 2021-01-06 07:51:02,stdout,Caused by: java.net.SocketException: Socket closed 2021-01-06 07:51:02,stdout, at java.lang.Thread.run(Thread.java:748) [?:1.8.0_272] 2021-01-06 07:51:02,stdout, at com.tplink.eap.cloudsdk.client.s.run(SourceFile:830) [cloudsdk-1.0.5.jar:?] 2021-01-06 07:51:02,stdout, at com.tplink.eap.cloudsdk.client.p.a(SourceFile:120) ~[cloudsdk-1.0.5.jar:?] 2021-01-06 07:51:02,stdout, at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:831) ~[?:1.8.0_272] 2021-01-06 07:51:02,stdout, at sun.security.ssl.SSLSocketImpl.access$300(SSLSocketImpl.java:72) ~[?:1.8.0_272] 2021-01-06 07:51:02,stdout, at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1303) ~[?:1.8.0_272] 2021-01-06 07:51:02,stdout, at sun.security.ssl.TransportContext.fatal(TransportContext.java:262) ~[?:1.8.0_272] 2021-01-06 07:51:02,stdout, at sun.security.ssl.TransportContext.fatal(TransportContext.java:267) ~[?:1.8.0_272] 2021-01-06 07:51:02,stdout, at sun.security.ssl.TransportContext.fatal(TransportContext.java:324) ~[?:1.8.0_272] 2021-01-06 07:51:02,stdout, at sun.security.ssl.Alert.createSSLException(Alert.java:127) ~[?:1.8.0_272] 2021-01-06 07:51:02,stdout,javax.net.ssl.SSLException: Socket closed 2021-01-06 07:51:02,stdout,2021-01-06 08:51:02 [Thread-19] [INFO]-[SourceFile:837] - javax.net.ssl.SSLException: Socket closed 2021-01-06 07:51:02,stdout,2021-01-06 08:51:02 [Thread-15] [INFO]-[SourceFile:110] - Connect SEF server automatically. 2021-01-06 07:51:02,stdout,2021-01-06 08:51:02 [Thread-15] [INFO]-[SourceFile:146] - Close connection to service server. 2021-01-06 07:51:02,stdout,2021-01-06 08:51:02 [Thread-15] [INFO]-[SourceFile:366] - The result of connection is false. 2021-01-06 07:51:02,stdout,"2021-01-06 08:51:02 [Thread-15] [INFO]-[SourceFile:127] - Connect service server automatically, ConnectionType is PERSISTENT_CONNECTION. " 2021-01-06 07:51:02,stdout,2021-01-06 08:51:02 [Thread-20] [INFO]-[SourceFile:796] - Thread 'expiredRequestCleanThread' is stopped 2021-01-06 07:51:02,stdout,2021-01-06 08:51:02 [Thread-20] [INFO]-[SourceFile:792] - expiredRequestCleanThread is interrupted. 2021-01-06 07:51:02,stdout,2021-01-06 08:51:02 [Thread-15] [INFO]-[SourceFile:146] - Close connection to service server. 2021-01-06 07:51:02,stdout,2021-01-06 08:51:02 [Thread-21] [INFO]-[SourceFile:921] - Thread 'heartBeatThread' is stopped 2021-01-06 07:50:44,stdout,2021-01-06 08:50:44 [device-monitor-timeout-task-0] [INFO]-[SourceFile:263] - Device 1C-3B-F3-A1-4C-A0 status change from Connected to Heartbeat Missed 2021-01-06 07:49:54,stdout,2021-01-06 08:49:54 [discovery-handler-pool-0] [INFO]-[SourceFile:569] - Old connected device 74-DA-88-D8-93-D4 reconnect to controller 2021-01-06 07:49:53,stdout,"2021-01-06 08:49:53 [nioEventLoopGroup-5-1] [INFO]-[SourceFile:225] - 74-DA-88-D8-93-D4 reset to discovery status from CONNECTED, close old connection " 2021-01-06 07:49:04,stdout,2021-01-06 08:49:04 [device-monitor-timeout-task-0] [INFO]-[SourceFile:263] - Device 74-DA-88-D8-93-D4 status change from Connected to Heartbeat Missed 2021-01-06 02:02:40,stdout,2021-01-06 03:02:40 [discovery-handler-pool-1] [INFO]-[SourceFile:569] - Old connected device 74-DA-88-D8-93-D4 reconnect to controller 2021-01-06 02:02:07,stdout,2021-01-06 03:02:07 [discovery-handler-pool-0] [INFO]-[SourceFile:569] - Old connected device 1C-3B-F3-A1-4C-A0 reconnect to controller 2021-01-06 02:01:23,stdout,2021-01-06 03:01:23 [device-monitor-timeout-task-0] [INFO]-[SourceFile:263] - Device 1C-3B-F3-A1-4C-A0 status change from Connected to Heartbeat Missed 2021-01-06 02:01:04,stdout,2021-01-06 03:01:04 [nioEventLoopGroup-7-2] [WARN]-[SourceFile:61] - handle message failed 2021-01-06 02:01:04,stdout,2021-01-06 03:01:04 [nioEventLoopGroup-7-1] [WARN]-[SourceFile:61] - handle message failed 2021-01-06 02:01:04,stdout,2021-01-06 03:01:04 [nioEventLoopGroup-7-2] [WARN]-[SourceFile:98] - Failed to updateManageRebuildDeviceContext for 74-DA-88-D8-93-D4 2021-01-06 02:01:04,stdout,2021-01-06 03:01:04 [nioEventLoopGroup-7-1] [WARN]-[SourceFile:98] - Failed to updateManageRebuildDeviceContext for 1C-3B-F3-A1-4C-A0 2021-01-06 02:01:04,stdout,2021-01-06 03:01:04 [nioEventLoopGroup-7-2] [ERROR]-[SourceFile:189] - Failed to find device context for 74-DA-88-D8-93-D4 after receive REBUILD_REQUEST during manage 2021-01-06 02:01:04,stdout,2021-01-06 03:01:04 [nioEventLoopGroup-7-1] [ERROR]-[SourceFile:189] - Failed to find device context for 1C-3B-F3-A1-4C-A0 after receive REBUILD_REQUEST during manage 2021-01-06 02:01:04,stdout,tail: '/opt/tplink/EAPController/logs/server.log' has appeared; following new file 2021-01-06 02:01:04,stdout,tail: '/opt/tplink/EAPController/logs/server.log' has become inaccessible: No such file or directory
Re:Omada controler offline for minutes
@Fae Thanks for your reply! The needed information is below. I hope this is enough and you can help me. Let me know if you need more information!
Best regards,
Bart
The APs are both EAP225(EU) v3.0, software version: 5.0.0
The Omada controller is version 4.1.5
I've added the config backup as attachment. (I had to split the zip archive, I'll post the second part in a follow up post. Silly to have a 10MB limit...)
Topology:
File:
Re:Omada controler offline for minutes
Dear @bkummel,
As you previously mentioned that the SSID will disappear for a couple minutes and come back again. I doubt that the EAP may be rebooted when the problem occurs. To figure out the problem, could you please help to check more information for analysis?
1. Do you use the original PoE adapter to power up them? Are the two EAP225s both hardwired to the network via Ethernet cables?
Checking your configuration, I noticed that the LED is disabled, perhaps you may enable it to check the LED status when the problem occurred. In addition, if the two EAPs are both hardwired via cables, I suggest you disable the Mesh function in case one AP may link wirelessly to the other when it lost the LAN connection.
2. As for the problem that the Omada Controller offline, may I know how you access the Omada Controller when the problem occurs? Do you access it via cloud access (https://omada.tplinkcloud.com/)? If yes, please try accessing via local access by typing the IP address of the controller in the local network. Besides, ensure the IP address of the controller is static.
3. Could you please provide the Running log for checking? If you can recall the date when the problem occurred, please also let me know.
>> Omada EAP Firmware Trial Available Here <<
*Try filtering posts on each forum by Label of [Early Access]*
Re:Omada controler offline for minutes
Hi @Fae,
Tanks for your reply.
- Yes, I do use the supplied PoE adapters. And both EAPs are hard wired to my network with ethernet cables. I have changed the LED to be on and Mesh to be off.
- The IP address of the controller is static. I get notifications from the Omada cloud that my controller is offline. I didn't try to get in locally, I'll try that next time.
- I've attached the requested log. The last time the problem occured was yesterday, January 18th, between 10:00 and 10:45. (The EAP's weren't offline for the whole 45 minutes, I guess it was 10 minutes or somehting like that, but I don't know exactly when that was.)
Best regards,
Bart
File: