@AdrianHi 

to makes some things clear, I use a pi-hole as dhcp server and in router mode I can see the guest option but not the isolate from main network option, I can see this in ap mode but then I don't have a internet connection, I get an ip adress tho, but in the same range. 
so when I put the m5 in router mode I disable the dhcp server of my pi-hole but then the devices connected to the guest network can still see the rest of my main network and even connect to my server for example. 
 

tl:dr 

my m5 in router mode no isolation from main network for the guest WiFi. 

in ap mode no internet for the guest network. 

@TP-Link 

Hi there,
Thank you for the information that is provided until now. I would like to confirm that everything is working as it should and will supply detailed information about my setup and the steps I took to verify this here:

I have 3 deco M5 units in my home. They are in AP mode. My Moden/Router is the router. I have NAS1 wired to the moden/router with a fixed IP address. I have NAS2 wired to my deco M5 main unit (in the 2nd RJ45 port) with a fixed IP address. I have a wireless network enabled on the deco m5 units. I also have a wireless guest network enabled on the deco m5 system with the "isolation from main network" option enabled.

When I have my laptop on the wireless network from my moden/router I can connect to both NAS systems through the external IP address, the DDNS address & the internal network address.

When I have my laptop on the wireless network from deco m5 I can connect to both NAS systems through the external IP address, the DDNS address & the internal network address.

When I have my laptop on the isolated wireless guest network from deco m5 I can still connect to both NAS systems through the external ip address and the DDNS address. But I CANNOT connect to either NAS systems (or my modem/router) through the internal network address. I can still access the internet as I am writing this message when connected to the guest network.

In my opinion I can now conclude that, at least for me, everything is working as it should.

If there are any steps that I have missed out on, please let me know.

Thank you very much, keep up the good work and stay safe,

Nick

Can confirm this work, kinda!

TLDR; In AP mode, with the “Guest Isolation” option checked, devices connected to the guest network can only connect to the gateway and external IP addresses but not to any other IP address on the same subnet.

Details: In theory guest network isolation should be achieved via a separate subnet, but as Deco is not the router it cannot do this. So TP-Link has implemented this isolation but not allowing communications to any IP on the same subnet (except the gateway) from any device on the guest network, this also includes not being able to communicate to other devices on the guest network. i.e. each device on the guest network is isolated from the rest of the subnet.

You may still see ARP broadcasts and a discovery service may show that other devices are on the network but you will not be able to ping ore connect to them in any other way. This may not suit some who would like to run separate services like a media server on the guest network but works fine for devices you want to keep separate from your main network.

Overall this is a decent workaround by TP-link given the Deco is not the router. Maybe the next version can allow communications between IP address of devices connected on the guest network.

Note, in route mode, guest devices can communicate with each other.

(I am running firmware 1.5.7 Build 20210819 Rel. 43499)

I have accidentally stumbled on this thread and checked to see that the guest isolation now works for AP mode and my guests now cannot access my NAS storage or smart home devices.

Great, thanks TP-Link!

Now I'm just waiting for the option to turn off the "Smart DHCP" feature of my Decos, which is supposedly coming out of beta soon.