The biggest issue with DNS is not the protocol, or even the reference implementation. It's the people who think they are clever and try to make things better by making them worse.
The most egregious of course is ISPs rewriting TTLs (or resolvers that just ignore them). But there are other implementation issues too, like caching things that shouldn't be or doing it wrong. I've seen resolvers that cache a CNAME and the A record it resolves to with the TTL of the CNAME (which is wrong).
I'm also very concerned about the "WHY DNS MATTERS FOR SYSTEM DESIGN" section. While everything there is correct enough, it doesn't dive into the implication of each and how things go wrong.
For example, using DNS for round robin balancing is an awful idea in practice. Because Comcast will cache one IP of three, and all of a sudden 60% of your traffic is going to one IP. Similar issue with regional IPs. There are so many ways for the wrong IP to get into a cache.
ISP DNS servers really ought to be banned, they are always so bad. I've seen traffic days later on a record with 1 hour TTL. In general I see like 50% traffic move after the initial 1-2x TTL interval, another 40-45% over next several hours up to one day, and then the last 5-1% can take forever.
For round-robin, I've actually had it work reasonably well for API usage. Of course it's not ideal, but when I wanted to roll out new things slowly over several days and could not use a load balancer or reverse proxy, it kind of worked. I think most API users are just running with a reasonable resolver and not residential ISP ones.
When I moved reddit from one datacenter to another, about 70% of the traffic shifted within the TTL. Another 20% moved within a week. Took till the end of a month after the change to get to about 98%
But after two months, about 1% was still going to the old server (I had set it up as a proxy for the cutover). Most of that traffic looked like crawlers that were written in things like Python or Ruby and had probably hard coded the IP or done something where it just didn't know what a TTL was.
So at that point I just shut down the old server.
You're probably right about API clients using better resolvers though. I was talking about consumer facing things where a lot of people would be on ISP DNS.
I have to admit - I still grind my teeth every time I see "dns propagation" used without a direct follow-up that it's a myth, you're looking at cascading cache expiry.
Propagation might be a useful way to visualise it, but doesn't match reality unless every cache is a warm cache.
DNS changes propagate. They just do-so in a pull, not push, way.
It’s accurate to say that a user is waiting for the change to propagate if they are sitting there clicking re-try as they wait for the cascading cache expirations to do their thing.
Yes! The idea of DNS records "propagating" gave me entirely the wrong mental model of DNS very early in my career. Granted, the confusion didn't last long because I read the cricket book soon after, but it was still pretty jarring.
> Without DNS, you'd need to memorize IP addresses for every website.
This used to be true until virtual hosting came along, allowing for several domains to point to the same IP address, but only for non-HTTPS traffic. Then a bit later we got SNI (Server Name Indication) that did the same thing for HTTPS.
I remember having web servers with 10-12 public IP adresses when I started working. The number of IPv4 addresses needed has been greatly reduced since.
> You can change which resolver you use in your network settings. I switched to 1.1.1.1 on my machines - it's noticeably faster than my ISP's default resolver.
Noticeably faster as in just loading a website? Or in some script where small differences add up? I thought typical DNS lookup was sub 100ms, but I've never tried switching my resolver so I'm curious
It's the most baroque protocol that is still somehow surviving from the initial Internet. There are so many weird limitations, like not being able to use CNAME for apex zones. Or the entire DNSSEC fiasco.
Only oddity was the reference to the "router cache". I agree if your browser tried to lookup example.com the local cache would be used, but then it would be the system's configured DNS server - and that would most likely be an ISP, rather than your local router.
(Assuming a typical home connection, your router is _probably_ not a DNS server with local cache, it probably is a DHCP server which will hand out the upstream/ISPs' nameservers.)
I think this is probably quite dependent on what’s normal for ISPs in the region. In the UK for example, every ISP router I’ve had runs a DNS server and it’s that which is given out via DHCP. It then forwards onto the ISPs DNS platform.
American here, most of ISPs here do it as well. With modern router hardware, there is plenty of hardware available to run tiny DNS server that caches and forwards all requests to ISP upstream. Memory overhead is probably about 50MB and CPU overhead is trivial, probably .1% or less.
My parents are with Bell (the biggest ISP in Canada) and use the Bell Gigahub (Router/AP/Switch in one). It does have a DNS cache and the its set as the DNS resolver in their DHCP configuration.
I would argue the contrary - most home routers are running a DNS server of some kind. They forward to upstream, but will resolve local names like your printer and whatnot.
dnsmasq is the defacto tool on these embedded devices for dhcp+dns. probably a billion deployments. it's up there with sqlite for most used tech.
IIRC a resolver is what people would think of as a DNS server only it's not an authority for any domains. Like you said, they're used to get load off of authoritative servers and are very common. I think dnsmasq is mentioned explicitly in the O'Reilly locust book but it's been a while.
This is not a bad overview of DNS from a theoretical perspective. It’s also pretty well written and has good examples and figures.
What I think is missing is a bit more of the “in practice” side. If the author was surprised about TTL values, I doubt they have much experience with some of the other pitfalls, so I’m not surprised (not a knock on the author). But there is a reason why the phrase “It’s always DNS” exists.
As an example, it could be helpful to mention that ISP DNS resolvers (or any caching resolver in the path) could decide to ignore the TTL. In this case, your 360 sec TTL might not get updated for an hour or a day or longer. This can be infuriating to troubleshoot.
A section on troubleshooting might also be beneficial. But this mainly consists of checking results from different resolvers in your path - does it work with a local resolver? Your ISPs DNS? The authoritative server?
> it could be helpful to mention that ISP DNS resolvers (or any caching resolver in the path) could decide to ignore the TTL
not exactly the same but it reminds me of MTU. You can set it to whatever you want but it only takes one router that you don't control in the path to undo what you're trying to accomplish.
Honestly, I guess it's a fine article for someone who isn't very technical, but it provides very little real detail, and this wasn't an example of DNS breaking anything - it worked as designed.
The biggest pain of DNS for most people is if someone has set the TTL to an absurdly large number, or if a resolver isn't respecting TTL. And once you get into advanced configurations, SOAs and delegation certainly create their own headaches!
This might be the easiest-to-understand breakdown of DNS that I've seen to date. I've owned a domain since the late 90s, but never really understood everything the acronyms or concepts involved in making it work. Well done!
The biggest issue with DNS is not the protocol, or even the reference implementation. It's the people who think they are clever and try to make things better by making them worse.
The most egregious of course is ISPs rewriting TTLs (or resolvers that just ignore them). But there are other implementation issues too, like caching things that shouldn't be or doing it wrong. I've seen resolvers that cache a CNAME and the A record it resolves to with the TTL of the CNAME (which is wrong).
I'm also very concerned about the "WHY DNS MATTERS FOR SYSTEM DESIGN" section. While everything there is correct enough, it doesn't dive into the implication of each and how things go wrong.
For example, using DNS for round robin balancing is an awful idea in practice. Because Comcast will cache one IP of three, and all of a sudden 60% of your traffic is going to one IP. Similar issue with regional IPs. There are so many ways for the wrong IP to get into a cache.
There is a reason we say "it's always DNS".
ISP DNS servers really ought to be banned, they are always so bad. I've seen traffic days later on a record with 1 hour TTL. In general I see like 50% traffic move after the initial 1-2x TTL interval, another 40-45% over next several hours up to one day, and then the last 5-1% can take forever.
For round-robin, I've actually had it work reasonably well for API usage. Of course it's not ideal, but when I wanted to roll out new things slowly over several days and could not use a load balancer or reverse proxy, it kind of worked. I think most API users are just running with a reasonable resolver and not residential ISP ones.
When I moved reddit from one datacenter to another, about 70% of the traffic shifted within the TTL. Another 20% moved within a week. Took till the end of a month after the change to get to about 98%
But after two months, about 1% was still going to the old server (I had set it up as a proxy for the cutover). Most of that traffic looked like crawlers that were written in things like Python or Ruby and had probably hard coded the IP or done something where it just didn't know what a TTL was.
So at that point I just shut down the old server.
You're probably right about API clients using better resolvers though. I was talking about consumer facing things where a lot of people would be on ISP DNS.
> It's the people who think they are clever and try to make things better by making them worse.
you mean DNSSEC, right? RIGHT?
I have to admit - I still grind my teeth every time I see "dns propagation" used without a direct follow-up that it's a myth, you're looking at cascading cache expiry.
Propagation might be a useful way to visualise it, but doesn't match reality unless every cache is a warm cache.
DNS changes propagate. They just do-so in a pull, not push, way.
It’s accurate to say that a user is waiting for the change to propagate if they are sitting there clicking re-try as they wait for the cascading cache expirations to do their thing.
https://jvns.ca/blog/2021/12/06/dns-doesn-t-propagate/
And checkout their Mess with DNS playgound!
Yes! The idea of DNS records "propagating" gave me entirely the wrong mental model of DNS very early in my career. Granted, the confusion didn't last long because I read the cricket book soon after, but it was still pretty jarring.
I grind my teeth every time I hear "I need an urgent DNS change" :/
> Without DNS, you'd need to memorize IP addresses for every website.
This used to be true until virtual hosting came along, allowing for several domains to point to the same IP address, but only for non-HTTPS traffic. Then a bit later we got SNI (Server Name Indication) that did the same thing for HTTPS.
I remember having web servers with 10-12 public IP adresses when I started working. The number of IPv4 addresses needed has been greatly reduced since.
You'd still need to memorize the IP address without DNS, otherwise how would you know to which server to connect?
The fact that a server can serve multiple vhosts and do TLS cert selection via SNI is not related to the lookup of what server to connect to.
> You can change which resolver you use in your network settings. I switched to 1.1.1.1 on my machines - it's noticeably faster than my ISP's default resolver.
Noticeably faster as in just loading a website? Or in some script where small differences add up? I thought typical DNS lookup was sub 100ms, but I've never tried switching my resolver so I'm curious
Curious how you’d measure this
I really like Julia Evan's explainer on this as well: https://wizardzines.com/zines/dns/
> DNS broke my site for three hours. But now I actually understand it
I have been broken for three decades and I still don't understand DNS. It is a simple protocol but people use it in complicated manners.
why you need DNS for at server? just use hosts file. why your server would need to resolve domains on the internet? client yeah, server no.
> why you need DNS for at server? just use hosts file.
IP's can change without warning.
interesting.
But it's not an issue at all, and it provides a convenience that can be depended on by a lot of your dependencies.
Code may use domains instead of ip addresses (which provides resiliency), package managers like apt depend on domains. And so on.
clusters and other load-balanced workloads. who wants to maintain hosts files across a fleet of containers or multiregion load-balanced situations?
Simple? Oh no. Simple it is not.
It's the most baroque protocol that is still somehow surviving from the initial Internet. There are so many weird limitations, like not being able to use CNAME for apex zones. Or the entire DNSSEC fiasco.
Only oddity was the reference to the "router cache". I agree if your browser tried to lookup example.com the local cache would be used, but then it would be the system's configured DNS server - and that would most likely be an ISP, rather than your local router.
(Assuming a typical home connection, your router is _probably_ not a DNS server with local cache, it probably is a DHCP server which will hand out the upstream/ISPs' nameservers.)
I think this is probably quite dependent on what’s normal for ISPs in the region. In the UK for example, every ISP router I’ve had runs a DNS server and it’s that which is given out via DHCP. It then forwards onto the ISPs DNS platform.
In Scotland I was with Telewest, then Virgin, and my memory is always that the DHCP pushed out the external IP of the ISP's DNS servers.
Nowadays I'm in Finland and definitely the router runs no DNS service, the DHCP service advertises the ISP resolvers.
Probably depends on the region/ISP I guess, but I had no expectation that it would be the more common option.
American here, most of ISPs here do it as well. With modern router hardware, there is plenty of hardware available to run tiny DNS server that caches and forwards all requests to ISP upstream. Memory overhead is probably about 50MB and CPU overhead is trivial, probably .1% or less.
My parents are with Bell (the biggest ISP in Canada) and use the Bell Gigahub (Router/AP/Switch in one). It does have a DNS cache and the its set as the DNS resolver in their DHCP configuration.
The system's configured DNS resolver is usually your router.
I would argue the contrary - most home routers are running a DNS server of some kind. They forward to upstream, but will resolve local names like your printer and whatnot.
dnsmasq is the defacto tool on these embedded devices for dhcp+dns. probably a billion deployments. it's up there with sqlite for most used tech.
IIRC a resolver is what people would think of as a DNS server only it's not an authority for any domains. Like you said, they're used to get load off of authoritative servers and are very common. I think dnsmasq is mentioned explicitly in the O'Reilly locust book but it's been a while.
I love DNS!
https://www.instagram.com/p/DUTSLcjkfJn/
well written
would really be happy to have had these explanations before I had to figure it out for myself.
then you have these guys who reached the next level
https://www.dns.toys/
This is not a bad overview of DNS from a theoretical perspective. It’s also pretty well written and has good examples and figures.
What I think is missing is a bit more of the “in practice” side. If the author was surprised about TTL values, I doubt they have much experience with some of the other pitfalls, so I’m not surprised (not a knock on the author). But there is a reason why the phrase “It’s always DNS” exists.
As an example, it could be helpful to mention that ISP DNS resolvers (or any caching resolver in the path) could decide to ignore the TTL. In this case, your 360 sec TTL might not get updated for an hour or a day or longer. This can be infuriating to troubleshoot.
A section on troubleshooting might also be beneficial. But this mainly consists of checking results from different resolvers in your path - does it work with a local resolver? Your ISPs DNS? The authoritative server?
> it could be helpful to mention that ISP DNS resolvers (or any caching resolver in the path) could decide to ignore the TTL
not exactly the same but it reminds me of MTU. You can set it to whatever you want but it only takes one router that you don't control in the path to undo what you're trying to accomplish.
Honestly, I guess it's a fine article for someone who isn't very technical, but it provides very little real detail, and this wasn't an example of DNS breaking anything - it worked as designed.
The biggest pain of DNS for most people is if someone has set the TTL to an absurdly large number, or if a resolver isn't respecting TTL. And once you get into advanced configurations, SOAs and delegation certainly create their own headaches!
This might be the easiest-to-understand breakdown of DNS that I've seen to date. I've owned a domain since the late 90s, but never really understood everything the acronyms or concepts involved in making it work. Well done!
Thanks! That was exactly what I was going for - making DNS approachable for everyone, not just sysadmins. Glad it helped!
Namasté. First blog post I’ve read from Nepal, very well done.
One nitpick I had was that the subdomain examples are more likely to be hosts. They could be either, but was not mentioned. Perhaps I missed.
Well written! Is so easy to understand and read that I can easily share it even with non-tech people c:
Appreciate it! That's the goal - tech concepts shouldn't need a CS degree to understand.
DNS is an example of a globally distributed DB
Well written article!
Glad you liked it. Thank you