@forlop
What is the current network diagram? How do you connect these devices together?
As you mentioned, you have another Deco M9 plus connect to this network and work properly, does it work in wireless router mode? And generally spearking, there will be ONE master Deco existed on the same network.
Then regarding to the error message, try to skip this setup and finish the configuration. After that, please open Deco app, click more-advanced-IPV4 to verify whether the Deco has obtained valid IP address, if so, please show us a screenshot of this page and change the DNS to 8.8.8.8.
And please unplug the master Deco M5 and modem, connect your computer to the LAN port of the modem and click here to verify its IP address and gateway.
Note: Please show us a screenshot of the details-status page.
If it works well, connect the modem back to the master Deco and reboot the modem to verify whether the master Deco M5 works properly.
Good day.