DNS over TLS/HTTPS Issues
Hello,
I'm facing a weird bug using my ER8411 and DNS Proxy. DNS Proxy is configured to forward my requests to a DNS via HTTPS (same behavior also on DNS over TLS).
I have a simple setup, ER8411 as internet gateway (PPPoE connection), a switch and 3x EAP650 connected to the switch.
Long story short: I have multiple devices (iPhones/iPad) already using secure DNS via native iOS integration (DNS over TLS or DNS over HTTP). So the devices are already performing DOT/DOH DNS requests (the only plain DNS request they perform is the bootstrap server DNS resolution, looking for the IP to send the DOT/DOH queries).
I noticed that all those devices (using different DOH/DOT server), are affected by random and sporadic slow connectivity, like a very slow DNS resolution.
The issue suddenly disappears once I let the devices to use plain DNS towards ER8411 (which is using DNS over TLS/HTTP) or viceversa: DOT/DOH from devices, while ER8411 is using plain DNS (UDP 53).
While I notice slow responsiveness on my devices using my configuration (DOH/DOT on both device AND ER8411), if I switch to cellular data, responsiveness is back and fast.
My ISP does not perform filtering and connectivity is working as expected on other devices using ER8411 as DNS resolver.
I was able to replicate the issue with and without DNS Cache enabled (with no TTL specified) and also disabling iCloud Private Relay on the devices.
Working:
- ER8411 with DNS over TLS/HTTPS configured + device with ER8411 as DNS server
- ER8411 with plain DNS UDP 53 upstream + device with DNS over TLS/HTTPS configured
Not working:
- ER8411 with DNS over TLS/HTTPS configured + device with DNS over TLS/HTTPS configured (DNS Cache ON and OFF, iCloud Private Relay ON and OFF)
I also tested multiple secure DOT/DOH DNS server providers.
Thank you.
- Copy Link
- Subscribe
- Bookmark
- Report Inappropriate Content
Hi @Bianco8
I am gonna reply to this publicly so that others can view this as well.
First, we don't get the same result in our lab test.
We suppose this is a problem in your network environment instead of a problem with the mechanism of the router DoH or DoT.
Second, supposedly, the DoH from the client is gonna sent to the DoH or DoT server directly bypassing the router's upstream DNS servers. As it is already encrypted and transferred in an encrypted manner.
There is only gonna be a problem when your DoH or DoT is resolved incorrectly when it queries the DoH/DoT domain from your iOS devices. After the resolution, it will become a private conversation to the servers bypassing the DoH/DoT upstream servers.
This could only be the one that goes wrong. You probably should confirm if the DoH/DoT you use on the iOS devices is correctly resolved based on your DoH/DoT setup on the router.
Basically, it confirms what I thought in the first reply. It should not be a problem with the router but your DNS resolution again.
You can reply to the ticket and send your new captures based on the new requests.
Primitive conclusion is that there is no problem with the DoH/DoT servers. Let's see how it moves.
- Copy Link
- Report Inappropriate Content
Hi @Bianco8
Thanks for posting in our business forum.
I gotta say that you have described the problem so well and clearly. It saved a lot of your and my time before we were on the same page which I used to find it hard to reply to someone else's.
My first impression of this might figure out the part from client > DoH/DoT upstream server as they have been configured DNS servers on them. IMO, if they have encrypted this part, it should not be forwarded to the router to resolve from the router's upstream. It should be forwarded directly.
Based on what you describe, it seems to be that it travels twice: Client _encrypted_ to router _ encrypted_ to Upstream DoH/DoT which causes slowness.
I hope you can verify this first. If you can Wireshark any, that would be better. I think it should be forwarded to the Upstream DNS set on your clients directly instead of passing through the router's upstream again.
I will also inform the test team to try this. Can you be specific about your slowness? Like it never loads or does it load after you refresh it? Can you reproduce it every time?
- Copy Link
- Report Inappropriate Content
Hello Clive,
Clive_A wrote
Hi @Bianco8
Thanks for posting in our business forum.
I gotta say that you have described the problem so well and clearly. It saved a lot of your and my time before we were on the same page which I used to find it hard to reply to someone else's.
My first impression of this might figure out the part from client > DoH/DoT upstream server as they have been configured DNS servers on them. IMO, if they have encrypted this part, it should not be forwarded to the router to resolve from the router's upstream. It should be forwarded directly.
Based on what you describe, it seems to be that it travels twice: Client _encrypted_ to router _ encrypted_ to Upstream DoH/DoT which causes slowness.
If I bypass DOH/DOT on my mobile devices while using the ER8411 network, everything is working fine.
As far as I understood, there is no double DNS query:
- DOH/DOT mobile device performs a standard UDP query to ER8411 to resolve the DOH/DOT DNS hostname (in my case, dns [.] controld [.] com)
- This query is encrypted and forwarded from ER8411 with DOH to my upstream provider (ControlD itself again), which provides its IP address (76.76.2.22).
- ER8411 might or might not use DNS Cache at this step (according to the answer TTL value).
- My iPhone gets DNS answer (76.76.2.22) and establish the DNS-over-TLS/HTTPS with the provider.
- All future DNS queries performed by my iPhone should be encrypted and transparent like HTTPS traffic to my ER8411 (I'm no longer performing DPI or IPS/IDS on it).
I can verify this from a packet capture performed at ER8411 level. This is the behaviour I expect for DOH configured in DNS Proxy ER8411 and DOH/DOT configured on my iPhone.
Clive_A wrote
I hope you can verify this first. If you can Wireshark any, that would be better. I think it should be forwarded to the Upstream DNS set on your clients directly instead of passing through the router's upstream again.
I will also inform the test team to try this. Can you be specific about your slowness? Like it never loads or does it load after you refresh it? Can you reproduce it every time?
Please note that I'm facing this issue with multiple DOH/DOT providers (including Cloudflare).
About the slowness, when I face the issue, it get stuck loading: for web browsing via Safari, I need to close/reopen the tab (reloading is not enough). On the apps facing the issue, they never complete loading (i.e. Slack app messages not updated). Checking the logs at resolver levels, I see queries flowing nominally.
As soon as I revert to plain DNS (on my iPhone), everything works fine.
The same websites might work as expected at other times, this is what is causing me difficulties in troubleshooting.
Thank you!
- Copy Link
- Report Inappropriate Content
@Clive_A I was able to perform a packet capture during the issue. Can you please point me how to share it securely?
TY!
- Copy Link
- Report Inappropriate Content
Hi @Bianco8
Thank you so much for taking the time to post the issue on TP-Link community!
To better assist you, I've created a support ticket via your registered email address, and escalated it to our support engineer to look into the issue. The ticket ID is TKID240612315, please check your email box and ensure the support email is well received. Thanks!
Once the issue is addressed or resolved, welcome to update this topic thread with your solution to help others who may encounter the same issue as you did.
Many thanks for your great cooperation and patience!
- Copy Link
- Report Inappropriate Content
Hi @Bianco8
I am gonna reply to this publicly so that others can view this as well.
First, we don't get the same result in our lab test.
We suppose this is a problem in your network environment instead of a problem with the mechanism of the router DoH or DoT.
Second, supposedly, the DoH from the client is gonna sent to the DoH or DoT server directly bypassing the router's upstream DNS servers. As it is already encrypted and transferred in an encrypted manner.
There is only gonna be a problem when your DoH or DoT is resolved incorrectly when it queries the DoH/DoT domain from your iOS devices. After the resolution, it will become a private conversation to the servers bypassing the DoH/DoT upstream servers.
This could only be the one that goes wrong. You probably should confirm if the DoH/DoT you use on the iOS devices is correctly resolved based on your DoH/DoT setup on the router.
Basically, it confirms what I thought in the first reply. It should not be a problem with the router but your DNS resolution again.
You can reply to the ticket and send your new captures based on the new requests.
Primitive conclusion is that there is no problem with the DoH/DoT servers. Let's see how it moves.
- Copy Link
- Report Inappropriate Content
- Copy Link
- Report Inappropriate Content
Hi @Bianco8
Thanks for posting in our business forum.
Bianco8 wrote
Hello, I can confirm this issue is still on-going with the last firmware. I tested for a couple of months a DNS resolver in my LAN, advertised by DHCP and faced no issues with devices using secure DNS profiles (iPhone/iPad). Yesterday, I revert the config, enabled DNS Proxy configured to a DOH upstream and enabled DNS Cache with no parameters. Again, I'm facing connectivity issues on iOS devices. Something is not working properly with DNS Cache or DNS Proxy, possibly the bootstrapping of the DOH DNS server by iOS devices is blocked at some point by the Omada gateway. This behaviour was already shared with the support via capture, my CPEs performed DNS query to resolve the bootstrap of the DOH server address. Unfortunately, it is not possible to perform a capture on ER8411 WAN interface.
Mirror the port. And provide the Wireshark.
If you cannot provide proof or anything for us to reproduce or to see the issue, we cannot even troubleshoot this. Nor locate the cause.
Even if we have a remote session, the only thing you have a problem with is your iOS. I have tested DNS with my iPhone without any VPN or private relay and it worked as expected with the available DNS servers in my region.
I don't think you should use any of the secure profiles or any stuff on your iOS if you tend to use DNS proxy on the router. First, there is no point in complexing the DNS query.
Second, you should examine if there is an issue with your profile/software/config on your iOS.
Well, you are the only person reporting this issue ever since this feature arrived.
I have tried at least several DNSs. SmartDNS with profiles installed on my iOS, Adguard, and Pihole at home, and upstream on the Omada, I don't experience a problem. SmartDNS is restricted in my area so I removed the profile. Private Relay is not good enough for me as well.
Simply using the Pihole and Adguard in my LAN, with SmartDNS as my upstream, I experienced zero issues.
- Copy Link
- Report Inappropriate Content
Information
Helpful: 1
Views: 749
Replies: 7
Voters 1