Hello  @Clive_A,

good questions!

> How many PoE devices are connected to this switch? Number and model of them?

4 PoE devices.

Port 12: Instar Camera

Port 15: EAP610-Outdoor

Port 17: Olimex ESP23-PoE Board
Port 18: EAP660 HD

> So you said this happened twice in a quarter, was this stable before? Did you add anything to the switch prior to the first time it happened?

> Was there any firmware update before the first time?

These two are the only occurrences so far. Unfortunately, I cannot really tell anymore what has changed when from the first time. If there were firmware updates available, I would have done it. I really don't remember when it first happened as I brushed it off and didn't think much of it. But it was recent enough to claim within a quarter.

One significant change I did was add an RTSP loop to port 1. So now port 1is disabled automatically. Unfortunately, I don't remember whether the first occurrence is before or after that change.

Temperatures started to drop this week but I doubt this was the case of the first time. Could this behavior occur if one PoE connection is damaged by water? The EAP610-Outdoor is (surprisingly) outside. Maybe water ran into the connection?

> My guess is out of the PoE budget. Can you take a screenshot of the remaining PoE power on the switch?

Is there a way to pull a detailed switch log of the last X weeks from somewhere? I couldn't figure it out.

  @Clive_A 

Thank you for that! Seems like the switch logs were cleared after the restart so I couldn't find anything interesting.

The controller logs only indicate the event by showing:

status change from Connected to Heartbeat Missed

Apart from that, no unexpected message.

Would you like me to send you the logs (I don't really want to post them here publicly).

I would otherwise wait until the issue shows up again and grab the logs before restarting the switch.

From the logs, I would expect the first occurrence of the issue to have happened either on

2023-07-23 at 15:00 UTC or 2023-10-16 13:37 UTC.

Hello,

this night, this happened again after an uptime of 40day(s) 23h 54m 34s. Hopefully that matches with my previous past time frame :)

1. Issue was detected by the controller at 2024-01-09 01:57:46 (CET) by indicating the disconnected APs. Interestingly, the missing AP message does not show up on the controller global alerts. Only on the site alerts. In my opinion, the device alerts should show up globally if you can also see the devices in the global known device list...
2. The exported logs show a file time stamp of 0:03 although the containing syslog.log file seems to indicate an issue at a later time.
3. cpuUtilizationThread.txt contains some high cpu usage but no time stamp. Also not sure how relevant this really is.
4. syslog.log contains this:
   #2024-01-09 02:22:28,[Link]/5/Gi1/0/16 changed state to up.
   #2024-01-09 02:22:26,[Link]/5/Gi1/0/16 changed state to down.
   #2024-01-09 02:22:18,[Link]/5/Gi1/0/16 changed state to up.
   #2024-01-09 01:54:39,[LLDP]/6/Delete a neighbor from port Gi1/0/15.
   #2024-01-09 01:54:28,[LLDP]/6/Delete a neighbor from port Gi1/0/18.
   #2024-01-09 01:53:02,[PoE]/3/I2C read/write fail occurs on PSE 1.
   #2024-01-09 01:53:02,[PoE]/3/I2C read/write fail occurs on PSE 0.
   #2024-01-09 01:52:57,[PoE]/3/I2C read/write fail occurs on PSE 4.
   #2024-01-09 01:52:55,[PoE]/3/I2C read/write fail occurs on PSE 3.
   #2024-01-09 01:52:53,[PoE]/3/I2C read/write fail occurs on PSE 2.
   #2024-01-09 01:52:52,[Link]/5/Gi1/0/18 changed state to down.
   #2024-01-09 01:52:52,[Link]/5/Gi1/0/15 changed state to down.
   #2024-01-09 01:52:52,[Link]/5/Gi1/0/17 changed state to down.
   #2024-01-09 01:52:52,[Link]/5/Gi1/0/12 changed state to down.
   #2024-01-09 01:52:52,[PoE]/3/I2C read/write fail occurs on PSE 5.
   #2024-01-09 01:04:09,[Link]/5/Gi1/0/16 changed state to down.
   #2024-01-09 01:03:05,[Link]/5/Gi1/0/16 changed state to up.
   
   For this point, ignore "[Link]/5/Gi1/0/16". I only included it to have the continuous time series.
5. Overall, I do seem to have a strange "[Link]/5/Gi1/0/16" as it goes up and down multiple times very often. Can you tell me which port it is?

6 Sorry, I missed your message "Also, please monitor the LED changes if you can catch this "down" time.". I didn't do that... Will remember for next time as the switch is in a separate building.

How can I get the logs to you, if required? I can't upload them (I get a "Failed to upload" message here.

Thanks and best wishes,

sb0373

Hello  @Clive_A ,

just now it happened again. This time during daytime, which is a bit inconvenient to reboot. Also firmware is 5.20.0 Build 20230818 Rel.72032.

Again, I forgot to check the LEDs. I really have to remember doing so.

This time, a soft restart did not help (I assume I did it via the controller all the other times).

After the soft restart, POE was still not available and all LEDs apart from the power LED (green) were off (literally no other LED was alight on the whole switch).

Hard restart (power cable out/in) made it come back.

Any suggestions what else I can send to support the analysis?

I have the logs before soft restart and before hard restart.
 

Best wishes,

sb0373

Thank you, I also started thinking along the lines but hoping it could be fixed via software.