@Rodzilla68 

A way to test to see if it is the router, bypass the router and connect a PC direct to the modem.

  @ArcherC8 

Thanks. Great idea! I tested it without the router and it works fine. So we know there is an issue with the router. How can I use this router and iCloud together?

  @Rodzilla68 

try signing out of icloud and then back in.

I would look at any security features such as firewalls and vpn. 

  @ArcherC8 

Sign out now that I'm back on the network with the router?

I'm not using VPN.

Firewall is whatever comes with Windows 11 and I haven't changed any config on that (not do I know how really).

  @Rodzilla68 

By firewall and security, I am refering to the router's security in advance settings. 
 

Try signing out of icloud when attached to modem and then power cycle the router and pc, attach the pc to the router and see if you can connect. Sometimes software wants to remain attached to a prior device. See it a lot with apple devices and other devices when router is changed and using same ssid.