@Jaam 

I think the only way you can do what you want is to make sure the user network(s) used for wireless are the same for the A and B APs at any given site.  This means you need that user traffic to terminate on the same L3 router, which given your topology means in the datacentre.  To achieve this I think you need to manage each site router, ie 1a, 1b, 2a, 2b, etc. as it's own site and configure those routers with L2TP tunnels back to a new router (probably a 7206 or better) in the DC, using a defined subnet for each site...ie 1a and 1b both share say 192.168.1.x/24, 2a and 2b share 192.168.2.x/24 etc.  You would then use a Route Policy to force all local traffic on these subnets back over the L2TP tunnel to the DC router. If you don't want your users sharing the IP space as the APs, then you need to find a way to route VLAN tagged traffic back over the tunnel, or create additional tunnels, one per user VLAN with associate Route Policies.  Your APs would need to be configured for remote management... I'm not sure if the auto discovery will work or not.  You can re-use one of the two sites for each location to manage the APs at that location... ie you would adopt APs from both 1a and 1b into the 1a site for management... that will allow you to manage them as a pool, despite them having different uplink IPs from your ISP.  The downside of all of this is that no user traffic is offloaded, it all has to hairpin back through the DC router and then out to the internet... though you could create additional SSIDs that don't 'roam' and can directly egress the site through the local modem, rather than tunneling back to the DC.

The DC router/managed switch can have ACLs created to prevent sites talking to each other if desired, or if not, they'll be able to route to each other over the tunnels via the DC by default.

I obviously haven't mocked this up, but I can confirm that AP's on one modem can be managed by a controller+APs hanging off another modem.  For seamless roaming to work, you need to keep the same internet facing IP for the user sessions...hence the hairpin at the DC...so regardless of what AP the traffic comes in on, the same user IP (ie 192,168.1.100) and public IP are used and that's the reason for the tunnels from site router to DC.

I hope you find this helpful...or at least interesting to ponder :)

  @Jaam 

Individual SSIDs is messy. 

Scenario 1 -- I would keep the same SSID at each site which you can do by configuring the APs at each site with the remote IP of the Omada Controller in the DC (and opening the 4 ports or so needed for it to work on the internet at the DC).  The remote APs will register with the Omada controller and you can assign them to sites for management and if you group both the A and B APs into the same logical site you can push the same SSID profiles to both simultaneously.  In the simplest solution, you use the ISP router to provide IP's both to the APs and the clients who share the same subnet and all of them hit the internet directly via their local modem.  Zero hairpinning.  You would use the 'Guest' SSID feature to prevent the clients from talking to each other or accessing the APs.

Scenario 2 -- which is what you drew initially where you have a router at each site, though I think you will also need to put a managed switch like an SG2008(P) in conjunction with each of those routers.  Then you make each router/switch pair its own site and you can configure them all with the remote credentials of the controller in the DC...no VPN tunnels needed.  Down side is you have to manage the two halves of the same site individually, and sessions will still drop when users move from one half of the site to the other (even if they are on the same SSID, their uplink IP address would change breaking any SSL/HTTPS/RTP type sessions.  With this scenario you can have multiple isolated SSIDs, one for internet access, one for local communication, one that links to other sites via a VPN tunnel back to DC (hairpins the data though)....etc.