Switch suddenly disables stops delivering power to POE

This thread has been locked for further replies. You can start a new thread to share your ideas or ask questions.

Switch suddenly disables stops delivering power to POE

This thread has been locked for further replies. You can start a new thread to share your ideas or ask questions.
Switch suddenly disables stops delivering power to POE
Switch suddenly disables stops delivering power to POE
2023-11-29 20:23:28 - last edited 2024-01-25 06:08:39
Tags: #PoE
Model: TL-SG2428P  
Hardware Version: V5
Firmware Version: 5.0.5 Build 20230602 Rel.73473

Last night, the switch suddenly stopped delivering any power to POE devices. This is the second time this happened this quarter.

Only a hardware switch restart seems to help (or at least was the mehod that worked for me).

 

Here a CPU graph. At 1:01 AM, the controller logged the unavailability of the POE access points. The switch was restarted at just before 8 AM.

 

 

Any suggestions to avoid this in future?

  0      
  0      
#1
Options
1 Accepted Solution
Re:Switch suddenly disables stops delivering power to POE-Solution
2024-01-25 03:25:47 - last edited 2024-01-25 06:08:39

Hi @sb0373 

Thanks for posting in our business forum.

sb0373 wrote

Hello  @Clive_A ,

 

just now it happened again. This time during daytime, which is a bit inconvenient to reboot. Also firmware is 5.20.0 Build 20230818 Rel.72032.

Again, I forgot to check the LEDs. I really have to remember doing so.

 

This time, a soft restart did not help (I assume I did it via the controller all the other times).

After the soft restart, POE was still not available and all LEDs apart from the power LED (green) were off (literally no other LED was alight on the whole switch).

Hard restart (power cable out/in) made it come back.

 

Any suggestions what else I can send to support the analysis?

 

I have the logs before soft restart and before hard restart.
 

Best wishes,

sb0373

Judging from the log again, you might start an RMA with the local support team because it looks like this is a failure in the hardware. Replace one and check if this issue happens again or not.

Best Regards! If you are new to the forum, please read: Howto - A Guide to Use Forum Effectively. Read Before You Post. Look for a model? Search your model NOW Official and Beta firmware. NEW features! Subscribe for the latest update!Download Beta Here☚ ☛ ★ Configuration Guide ★ ☚ ☛ ★ Knowledge Base ★ ☚ ☛ ★ Troubleshooting ★ ☚ ● Be kind and nice. ● Stay on the topic. ● Post details. ● Search first. ● Please don't take it for granted. ● No email confidentiality should be violated. ● S/N, MAC, and your true public IP should be mosaiced.
Recommended Solution
  0  
  0  
#9
Options
9 Reply
Re:Switch suddenly disables stops delivering power to POE
2023-11-30 02:30:16

Hi @sb0373 

Thanks for posting in our business forum.

How many PoE devices are connected to this switch? Number and model of them?

So you said this happened twice in a quarter, was this stable before? Did you add anything to the switch prior to the first time it happened?

 

Was there any firmware update before the first time?

 

My guess is out of the PoE budget. Can you take a screenshot of the remaining PoE power on the switch?

Best Regards! If you are new to the forum, please read: Howto - A Guide to Use Forum Effectively. Read Before You Post. Look for a model? Search your model NOW Official and Beta firmware. NEW features! Subscribe for the latest update!Download Beta Here☚ ☛ ★ Configuration Guide ★ ☚ ☛ ★ Knowledge Base ★ ☚ ☛ ★ Troubleshooting ★ ☚ ● Be kind and nice. ● Stay on the topic. ● Post details. ● Search first. ● Please don't take it for granted. ● No email confidentiality should be violated. ● S/N, MAC, and your true public IP should be mosaiced.
  0  
  0  
#2
Options
Re:Switch suddenly disables stops delivering power to POE
2023-11-30 06:56:16 - last edited 2023-11-30 06:58:41

Hello  @Clive_A,

 

good questions!

 

> How many PoE devices are connected to this switch? Number and model of them?

4 PoE devices.

Port 12: Instar Camera

Port 15: EAP610-Outdoor

Port 17: Olimex ESP23-PoE Board
Port 18: EAP660 HD

 

> So you said this happened twice in a quarter, was this stable before? Did you add anything to the switch prior to the first time it happened?

> Was there any firmware update before the first time?

 

These two are the only occurrences so far. Unfortunately, I cannot really tell anymore what has changed when from the first time. If there were firmware updates available, I would have done it. I really don't remember when it first happened as I brushed it off and didn't think much of it. But it was recent enough to claim within a quarter.

 

One significant change I did was add an RTSP loop to port 1. So now port 1is disabled automatically. Unfortunately, I don't remember whether the first occurrence is before or after that change.


Temperatures started to drop this week but I doubt this was the case of the first time. Could this behavior occur if one PoE connection is damaged by water? The EAP610-Outdoor is (surprisingly) outside. Maybe water ran into the connection?

 

> My guess is out of the PoE budget. Can you take a screenshot of the remaining PoE power on the switch?

 

 

Is there a way to pull a detailed switch log of the last X weeks from somewhere? I couldn't figure it out.

  0  
  0  
#3
Options
Re:Switch suddenly disables stops delivering power to POE
2023-11-30 07:33:31

Hi @sb0373 

Thanks for posting in our business forum.

Two files might be helpful.

1. Device info. Click the device and go to config, find the Device Info.

2. Running log. Global View.

Both are in plaint text.

Best Regards! If you are new to the forum, please read: Howto - A Guide to Use Forum Effectively. Read Before You Post. Look for a model? Search your model NOW Official and Beta firmware. NEW features! Subscribe for the latest update!Download Beta Here☚ ☛ ★ Configuration Guide ★ ☚ ☛ ★ Knowledge Base ★ ☚ ☛ ★ Troubleshooting ★ ☚ ● Be kind and nice. ● Stay on the topic. ● Post details. ● Search first. ● Please don't take it for granted. ● No email confidentiality should be violated. ● S/N, MAC, and your true public IP should be mosaiced.
  0  
  0  
#4
Options
Re:Switch suddenly disables stops delivering power to POE
2023-11-30 09:27:01

  @Clive_A 

Thank you for that! Seems like the switch logs were cleared after the restart so I couldn't find anything interesting.

The controller logs only indicate the event by showing:

 

status change from Connected to Heartbeat Missed

 

Apart from that, no unexpected message.

 

Would you like me to send you the logs (I don't really want to post them here publicly).

 

I would otherwise wait until the issue shows up again and grab the logs before restarting the switch.

 

From the logs, I would expect the first occurrence of the issue to have happened either on

2023-07-23 at 15:00 UTC or 2023-10-16 13:37 UTC.

 

  1  
  1  
#5
Options
Re:Switch suddenly disables stops delivering power to POE
2023-12-01 01:12:43 - last edited 2023-12-01 09:51:15

Hi @sb0373 

Thanks for posting in our business forum.

sb0373 wrote

  @Clive_A 

Thank you for that! Seems like the switch logs were cleared after the restart so I couldn't find anything interesting.

The controller logs only indicate the event by showing:

 

status change from Connected to Heartbeat Missed

 

Apart from that, no unexpected message.

 

Would you like me to send you the logs (I don't really want to post them here publicly).

 

I would otherwise wait until the issue shows up again and grab the logs before restarting the switch.

 

From the logs, I would expect the first occurrence of the issue to have happened either on

2023-07-23 at 15:00 UTC or 2023-10-16 13:37 UTC.

 

Wait for your further reply and monitor the results. Will also update this information with the dev & test team and see what they suggest.

 

Also, please monitor the LED changes if you can catch this "down" time.

Best Regards! If you are new to the forum, please read: Howto - A Guide to Use Forum Effectively. Read Before You Post. Look for a model? Search your model NOW Official and Beta firmware. NEW features! Subscribe for the latest update!Download Beta Here☚ ☛ ★ Configuration Guide ★ ☚ ☛ ★ Knowledge Base ★ ☚ ☛ ★ Troubleshooting ★ ☚ ● Be kind and nice. ● Stay on the topic. ● Post details. ● Search first. ● Please don't take it for granted. ● No email confidentiality should be violated. ● S/N, MAC, and your true public IP should be mosaiced.
  0  
  0  
#6
Options
Re:Switch suddenly disables stops delivering power to POE
2024-01-09 06:43:27 - last edited 2024-01-09 06:44:46

Hello,

this night, this happened again after an uptime of 40day(s) 23h 54m 34s. Hopefully that matches with my previous past time frame :)

 

  1. Issue was detected by the controller at 2024-01-09 01:57:46 (CET) by indicating the disconnected APs. Interestingly, the missing AP message does not show up on the controller global alerts. Only on the site alerts. In my opinion, the device alerts should show up globally if you can also see the devices in the global known device list...
  2. The exported logs show a file time stamp of 0:03 although the containing syslog.log file seems to indicate an issue at a later time.
  3. cpuUtilizationThread.txt contains some high cpu usage but no time stamp. Also not sure how relevant this really is.
  4. syslog.log contains this:
    #2024-01-09 02:22:28,[Link]/5/Gi1/0/16 changed state to up.
    #2024-01-09 02:22:26,[Link]/5/Gi1/0/16 changed state to down.
    #2024-01-09 02:22:18,[Link]/5/Gi1/0/16 changed state to up.
    #2024-01-09 01:54:39,[LLDP]/6/Delete a neighbor from port Gi1/0/15.
    #2024-01-09 01:54:28,[LLDP]/6/Delete a neighbor from port Gi1/0/18.
    #2024-01-09 01:53:02,[PoE]/3/I2C read/write fail occurs on PSE 1.
    #2024-01-09 01:53:02,[PoE]/3/I2C read/write fail occurs on PSE 0.
    #2024-01-09 01:52:57,[PoE]/3/I2C read/write fail occurs on PSE 4.
    #2024-01-09 01:52:55,[PoE]/3/I2C read/write fail occurs on PSE 3.
    #2024-01-09 01:52:53,[PoE]/3/I2C read/write fail occurs on PSE 2.
    #2024-01-09 01:52:52,[Link]/5/Gi1/0/18 changed state to down.
    #2024-01-09 01:52:52,[Link]/5/Gi1/0/15 changed state to down.
    #2024-01-09 01:52:52,[Link]/5/Gi1/0/17 changed state to down.
    #2024-01-09 01:52:52,[Link]/5/Gi1/0/12 changed state to down.
    #2024-01-09 01:52:52,[PoE]/3/I2C read/write fail occurs on PSE 5.
    #2024-01-09 01:04:09,[Link]/5/Gi1/0/16 changed state to down.
    #2024-01-09 01:03:05,[Link]/5/Gi1/0/16 changed state to up.

    For this point, ignore "[Link]/5/Gi1/0/16". I only included it to have the continuous time series.
  5. Overall, I do seem to have a strange "[Link]/5/Gi1/0/16" as it goes up and down multiple times very often. Can you tell me which port it is?

 

6 Sorry, I missed your message "Also, please monitor the LED changes if you can catch this "down" time.". I didn't do that... Will remember for next time as the switch is in a separate building.

 

 

 

How can I get the logs to you, if required? I can't upload them (I get a "Failed to upload" message here.

 

Thanks and best wishes,

 

sb0373

  0  
  0  
#7
Options
Re:Switch suddenly disables stops delivering power to POE
2024-01-24 09:59:40 - last edited 2024-01-24 10:41:31

Hello  @Clive_A ,

 

just now it happened again. This time during daytime, which is a bit inconvenient to reboot. Also firmware is 5.20.0 Build 20230818 Rel.72032.

Again, I forgot to check the LEDs. I really have to remember doing so.

 

This time, a soft restart did not help (I assume I did it via the controller all the other times).

After the soft restart, POE was still not available and all LEDs apart from the power LED (green) were off (literally no other LED was alight on the whole switch).

Hard restart (power cable out/in) made it come back.

 

Any suggestions what else I can send to support the analysis?

 

I have the logs before soft restart and before hard restart.
 

Best wishes,

sb0373

  0  
  0  
#8
Options
Re:Switch suddenly disables stops delivering power to POE-Solution
2024-01-25 03:25:47 - last edited 2024-01-25 06:08:39

Hi @sb0373 

Thanks for posting in our business forum.

sb0373 wrote

Hello  @Clive_A ,

 

just now it happened again. This time during daytime, which is a bit inconvenient to reboot. Also firmware is 5.20.0 Build 20230818 Rel.72032.

Again, I forgot to check the LEDs. I really have to remember doing so.

 

This time, a soft restart did not help (I assume I did it via the controller all the other times).

After the soft restart, POE was still not available and all LEDs apart from the power LED (green) were off (literally no other LED was alight on the whole switch).

Hard restart (power cable out/in) made it come back.

 

Any suggestions what else I can send to support the analysis?

 

I have the logs before soft restart and before hard restart.
 

Best wishes,

sb0373

Judging from the log again, you might start an RMA with the local support team because it looks like this is a failure in the hardware. Replace one and check if this issue happens again or not.

Best Regards! If you are new to the forum, please read: Howto - A Guide to Use Forum Effectively. Read Before You Post. Look for a model? Search your model NOW Official and Beta firmware. NEW features! Subscribe for the latest update!Download Beta Here☚ ☛ ★ Configuration Guide ★ ☚ ☛ ★ Knowledge Base ★ ☚ ☛ ★ Troubleshooting ★ ☚ ● Be kind and nice. ● Stay on the topic. ● Post details. ● Search first. ● Please don't take it for granted. ● No email confidentiality should be violated. ● S/N, MAC, and your true public IP should be mosaiced.
Recommended Solution
  0  
  0  
#9
Options
Re:Switch suddenly disables stops delivering power to POE
2024-01-25 06:08:10

Thank you, I also started thinking along the lines but hoping it could be fixed via software.

  0  
  0  
#10
Options