If your gateways DNS is slower than Google DNS, it is either doing a recursive lookup (small time difference only) or probably overloaded at the ISP (huge time difference).

Usually, ISPs set up caching DNS servers, which are - by network topology - "nearer" then any DNS in the Internet. Of course, you can set up such a caching server for yourself inside your own network, e.g. on your router (if it supports setting up a full DNS) or a dedicated server.

For example, if looking up the domain tp-link.us, querying Google's DNS took 45ms:

[CODE]
$ dig tp-link.us @8.8.8.8
; <<>> DiG 9.9.5-9+deb8u15-Debian <<>> tp-link.us @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28362
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1


;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;tp-link.us. IN A


;; ANSWER SECTION:
tp-link.us. 3599 IN A 52.23.134.86


;; Query time: 45 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Wed Apr 18 23:35:59 CEST 2018
;; MSG SIZE rcvd: 55
[/CODE]

Doing a recursive lookup might be somewhat slower for the first time, but once the answer is cached locally, lookups on my caching DNS server are much faster:

[CODE]
$ dig tp-link.us @192.168.1.10
; <<>> DiG 9.9.5-9+deb8u15-Debian <<>> tp-link.us @192.168.1.10
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16426
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 13, ADDITIONAL: 1


;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;tp-link.us. IN A


;; ANSWER SECTION:
tp-link.us. 3590 IN A 52.23.134.86


;; AUTHORITY SECTION:
. 496737 IN NS h.root-servers.net.
. 496737 IN NS g.root-servers.net.
. 496737 IN NS k.root-servers.net.
. 496737 IN NS i.root-servers.net.
. 496737 IN NS e.root-servers.net.
. 496737 IN NS l.root-servers.net.
. 496737 IN NS a.root-servers.net.
. 496737 IN NS d.root-servers.net.
. 496737 IN NS m.root-servers.net.
. 496737 IN NS f.root-servers.net.
. 496737 IN NS j.root-servers.net.
. 496737 IN NS c.root-servers.net.
. 496737 IN NS b.root-servers.net.


;; Query time: 0 msec
;; SERVER: 192.168.1.10#53(192.168.1.10)
;; WHEN: Wed Apr 18 23:36:23 CEST 2018
;; MSG SIZE rcvd: 266

[/CODE]

Therefore, all devices in my subnet use two caching proxy servers in my local network, which in turn use the forwarders of my ISP. In addition, queries are forwarded to public DNS such as Google's DNS if the ISP's DNS should fail for whatever reason.

BTW, you need to set up DNS on CPEs only for two reasons:

- If you want to use domain names rather than IPs to define NTP servers to use for time synchronization

- or if your CPEs provide DHCP services and need to communicate the DNS server's IP to be used to the DHCP clients.

In the former case, it's more efficient to also set up a local NTP service and use IPs on CPEs (or fast-resolving local DNS names). But if you use the ntp.org pool of time-sync servers, the ntp.org community asks you to only use domain names, because they use DNS for load-balancing queries.

In the latter case, a CPE providing DHCP service is in most cases configured to act as a router, not as a dumb AP, albeit it's possible to even use DHCP on APs in addition to DHCP on the router for high availability or load-balancing (but then ensure to separate the DHCP's IP pools carefully).

Yes.