Troubleshoot 'heartbeat missed' on EAP650

Troubleshoot 'heartbeat missed' on EAP650

Troubleshoot 'heartbeat missed' on EAP650
Troubleshoot 'heartbeat missed' on EAP650
2024-10-06 20:20:27 - last edited 2024-10-16 02:27:41
Model: EAP650  
Hardware Version: V1
Firmware Version: 1.1.0 Build 20240830 Rel. 50826

Hi,

 

I'm running Omada in my house and there's 1 location where the AP gives 'HEARTBEAT MISSED' after a couple of minutes of operation. I would love to get some ideas to troubleshoot.

 

Here's the setup:

- ER605 Router

- 3 - wired - EAP650 access points on floor 1,2 and 3 - wired into ER605.

- Not using PoE - powering APs with adapter.

- Omada controller

 

Problem:

Floor 1 and 3 are totally fine. The access point on floor 2 gives 'HEARTBEAT MISSED' after some random duration, then resets, and continues working until next random failure. Typically failure are 10 - 20 minutes apart. Uptime resets. Other APs are up for days, floor 2 always < 1 hr.

 

What I've tried:
- Pulled a new CAT5e wire up to floor 2, because I suspected the wire. Problem persists.

- Swapped APs on floor 1 with 2. Problem persists for that same floor, floor 2.

- Every time post-'crash' I find 'IP Address Changed' and 'AP floor 2 got IP address <whatever>.' in the events of Omada controller. 

- I *think* the crashes are correlated with changing traffic patterns or heavy traffic. I *think* I triggered it once or twice while doing a speedtest and kicking off chromecast. The *think* inidicates there seems to be correlation, but I'm unsure about causation.

- It seems this location is also the slowst when doing speedtests. Floor 1 & 3 are way over 500 MBit, whereas 2 seems to be 100-200. I suspect this floor has the most interference from neighbour networks and electronic equipment.

 

I attempted to eliminate cable (hence the new cable) and AP (hence the swap). But the problem still persists. Running out of ideas here. Does anyone have any suggestions?

- Does anyone still suspect the cable or cable category? Should I go CAT6e or so?

- Would getting a managed switch (TL-SG105MPE e.g.) on floor 2 and then link AP to the switch give me more information to troubleshoot on?

- Could it be that heavy traffic messes with heartbeat? 

 

Again - love to get some fresh ideas.

  0      
  0      
#1
Options
1 Accepted Solution
Re:Troubleshoot 'heartbeat missed' on EAP650-Solution
2024-10-07 22:48:02 - last edited 2024-10-16 02:27:41

  @spike82 

 

I dont see any behaviour like this on my busienss network (10x EAP650 v1.2) with all omada gear.

 

Things i would be checking in your scenario:

 

- Is there an IP conflict anywhere, any 2 devices sharing the same IP whether statically assigned or by DHCP will cause all manner of goofy behaviour - which the omada controller is particularly sensitive to

 

- Do you have any VLANs? specifically, more than 1 untagged vlans on one trunk / uplink somewhere?  again the omada controller doesnt seem to like having multiple untagged vlans on the network on trunk links very much, even if nowhere near the EAPs

 

- Is the IP address of the EAP correctly set, including network mask, gateway and the DNS ?

 

- Try a forget device and re-adopt if not already tried, a factory reset using the pinhole on the EAP has the same effect

 

- On any of your cabling, do you have a run which has one end pinout A and the other pinout B ?

 

- Are all link from source to AP negotiating at 1gbit ?

 

- Did you swap the power supplies when you swapped the access points to rule out those?

 

- are you certain that the power socket your are using at the troublesome location is actually stable and working properly?

 

Given that the AP swap carried the same issue to the swapped AP, i would definitely be inclined to look more closely at cabling, including any patch leads either side of the main run, punchdowns, keystones, crimped RJ45s literally from source port to endpoint.  Make sure you dont have any whacky combinations of A>B pinouts going on,, make sure every complete cable is the same both ends.  Can you swap the source port for the troublesome location to rule out a whacky switch/router port ?

 

I have found my EAPs to be very resilient to somewhat dubious cabling and power.  My business is located in an old large building, many spaces, corridors, seperated buildings, doors etc.  All my APs are powered from main IT rack over POE.  I have one that is about 90m away from the rack, going over a combination of cat5e,6, through 2 patch panels, and finally powered through a Tenda poe-powered POE-in POE-out switch so i can use the one link i have going into the space for other things, and it never misses a beat, ever.

Recommended Solution
  1  
  1  
#4
Options
5 Reply
Re:Troubleshoot 'heartbeat missed' on EAP650
2024-10-07 07:05:20

  @spike82 

 

The first thing that comes to my mind is the cable. how long is the cable? 

Do you have any switches between the access point and the omada controller? and if you connect to a PC on the cable, is the speed just as bad?

 

  0  
  0  
#2
Options
Re:Troubleshoot 'heartbeat missed' on EAP650
2024-10-07 15:47:22 - last edited 2024-10-07 15:48:37

Hi MR.S,

 

yeah I thought cable as well, that's why the new cable. Cable is somewhere 5 - 10 meters or so. Like I said, pulled a new CAT5e yesterday and added connectors myself. I'm not a pro - so it could be my cabling skills, but it works fine until it doesn't.

 

No switch between Router and AP, the Omada controller is behind a switch though - but that's totally out-of-line with the AP.

 

"I *think* the crashes are correlated with changing traffic patterns or heavy traffic. I *think* I triggered it once or twice while doing a speedtest and kicking off chromecast. The *think* inidicates there seems to be correlation, but I'm unsure about causation." (my original text) - I was able to reproduce this again this morning:

 

- AP was up for 5 hours in the night

- Early morning - Ookla speedtest on my phone

- shoots up to 470 mbit and then - boom - AP is gone, and I see reset in Omada again.

- After it comes back up - speed is approx. 170 mbit during consecutive speedtests - but - doesn't crash anymore.

  0  
  0  
#3
Options
Re:Troubleshoot 'heartbeat missed' on EAP650-Solution
2024-10-07 22:48:02 - last edited 2024-10-16 02:27:41

  @spike82 

 

I dont see any behaviour like this on my busienss network (10x EAP650 v1.2) with all omada gear.

 

Things i would be checking in your scenario:

 

- Is there an IP conflict anywhere, any 2 devices sharing the same IP whether statically assigned or by DHCP will cause all manner of goofy behaviour - which the omada controller is particularly sensitive to

 

- Do you have any VLANs? specifically, more than 1 untagged vlans on one trunk / uplink somewhere?  again the omada controller doesnt seem to like having multiple untagged vlans on the network on trunk links very much, even if nowhere near the EAPs

 

- Is the IP address of the EAP correctly set, including network mask, gateway and the DNS ?

 

- Try a forget device and re-adopt if not already tried, a factory reset using the pinhole on the EAP has the same effect

 

- On any of your cabling, do you have a run which has one end pinout A and the other pinout B ?

 

- Are all link from source to AP negotiating at 1gbit ?

 

- Did you swap the power supplies when you swapped the access points to rule out those?

 

- are you certain that the power socket your are using at the troublesome location is actually stable and working properly?

 

Given that the AP swap carried the same issue to the swapped AP, i would definitely be inclined to look more closely at cabling, including any patch leads either side of the main run, punchdowns, keystones, crimped RJ45s literally from source port to endpoint.  Make sure you dont have any whacky combinations of A>B pinouts going on,, make sure every complete cable is the same both ends.  Can you swap the source port for the troublesome location to rule out a whacky switch/router port ?

 

I have found my EAPs to be very resilient to somewhat dubious cabling and power.  My business is located in an old large building, many spaces, corridors, seperated buildings, doors etc.  All my APs are powered from main IT rack over POE.  I have one that is about 90m away from the rack, going over a combination of cat5e,6, through 2 patch panels, and finally powered through a Tenda poe-powered POE-in POE-out switch so i can use the one link i have going into the space for other things, and it never misses a beat, ever.

Recommended Solution
  1  
  1  
#4
Options
Re:Troubleshoot 'heartbeat missed' on EAP650
2024-10-08 15:48:32 - last edited 2024-10-08 15:49:45

Hi GRL,

 

thank you for the suggestions.

 

"Did you swap the power supplies when you swapped the access points to rule out those?" - NO, I hadn't. Did that do that, but did it now - so far so good.

 

- Just 1 VLAN

- Checked IP settings of AP. Set by Omada, and consistent.

- I'm replacing more cables around the house to 6e now. But have not yet pulled a new CAT 6e from the router to the AP. (have to order new roll + connectors and stuff)

- All links to APs are at 1gbit - checked

- Pretty sure power socket is OK, as it als has TV and Sonos on it and decent quality.

  0  
  0  
#5
Options
Re:Troubleshoot 'heartbeat missed' on EAP650
2024-10-10 06:12:05

Update: I think it was the power source indeed. Swapped the adapter and the signal is now stable for 2 days. 

  0  
  0  
#6
Options