Kitz ADSL Broadband Information
adsl spacer  
Support this site
Home Broadband ISPs Tech Routers Wiki Forum
 
     
   Compare ISP   Rate your ISP
   Glossary   Glossary
 
Please login or register.

Login with username, password and session length
Advanced search  

News:

Pages: [1] 2

Author Topic: Another daft idea - musing again - DNS precaching  (Read 3863 times)

Weaver

  • Senior Kitizen
  • ******
  • Posts: 11459
  • Retd s/w dev; A&A; 4x7km ADSL2 lines; Firebrick
Another daft idea - musing again - DNS precaching
« on: October 15, 2018, 01:08:42 AM »

Say you are a big ISP. You run a DNS relay proxy cache for your customers. Knowing what the pattern of requests in the cache is, you could decide on a ‘greatest hits’ ‘top ten’ or whatever, a list of most requested domain names. Using this, you could then actively re-query these domain names from an upstream source or from the authoritative server even, just before the cache entry expires. That way, when a user re-queries the domain name she will get an answer instantly as you will already have done the work.

A second option, a further improvement would be to query the domain name from the authoritative servers at half the time to live, thus getting an extension on the time to live, that would mean a better, longer-lived, result for users who re-query the domain name later within the original time-to-live validity period. This of course would be unkind to the authoritative servers by putting unnecessary extra load on them, but since it is only large ISPs doing this, not millions of individual users, the net effect would be a reduction in load that more than offsets the extra imposed by this small trick.

Do you think it would work? Would it be any good?

If someone had knowledge that certain domain names are important then that user could pre-query such names in advance, just to make sure that the cache is always populated.

I realise that I don’t know what happens when a domain name is unnecessarily re-queried within the validity time. Do you get a new response with the same end time, that is a shortened validity time? (Because the server is not re-querying the authoritative server and so there is less and less of the original validity period left.) I have absolutely no idea at all of what I am talking about. Of course it seems like a really bad idea to start giving out shorter and shorter validity time values. If you as a server just ignore the whole thing, and just give out certain fixed validity times then you are at least not going with the madness of eventually shortening validity times down to nothing, but then the idea of a certain actual defined validity time window with a real start and end point goes out of the window, but perhaps a very approximate concept which is vague is very much more than good enough.
« Last Edit: October 16, 2018, 10:12:55 PM by Weaver »
Logged

chenks

  • Kitizen
  • ****
  • Posts: 1106
Re: Another daft idea - musing again - DNS precaching
« Reply #1 on: October 15, 2018, 08:06:23 AM »

Do you think it would work? Would it be any good?

possibly, and no
Logged

niemand

  • Kitizen
  • ****
  • Posts: 1836
Re: Another daft idea - musing again - DNS precaching
« Reply #2 on: October 15, 2018, 12:10:43 PM »

Probably a topic to do some more research on. Musing over things you, in your own words, 'have absolutely no idea at all of what I am talking about it' is a fairly rapid route to a head-spinning experience  :)

Just as a thought, though, how much of a time saving are you expecting from this? The only time people querying a caching DNS server are actually going to hit the delay you want to avert is in between TTL expiry and the first query that goes to the authoritative, and you've said you want a 'greatest hits' list so the percentage of requests this would actually benefit is miniscule.
Logged

Weaver

  • Senior Kitizen
  • ******
  • Posts: 11459
  • Retd s/w dev; A&A; 4x7km ADSL2 lines; Firebrick
Re: Another daft idea - musing again - DNS precaching
« Reply #3 on: October 15, 2018, 11:56:27 PM »

That is a very good point about the greatest hits thing. I was just thinking, or rather not thinking properly, about limiting the amount of load, the thing not going too crazy, but this turns out to be rather a backwards approach.
Logged

Chrysalis

  • Content Team
  • Addicted Kitizen
  • *
  • Posts: 7382
  • VM Gig1 - AAISP L2TP
Re: Another daft idea - musing again - DNS precaching
« Reply #4 on: October 16, 2018, 12:26:02 AM »

on my phone so shortened reply

lookup dns prefetch existing tech
also lookup unbound serve-expired < killer feature

also major dns providers like google will have so many hits from clients that major dns names will stay cached a lot
Logged

Weaver

  • Senior Kitizen
  • ******
  • Posts: 11459
  • Retd s/w dev; A&A; 4x7km ADSL2 lines; Firebrick
Re: Another daft idea - musing again - DNS precaching
« Reply #5 on: October 16, 2018, 03:53:56 AM »

So how / when do validity times get extended - this is what I need to look into. If clients keep querying popular names and the answer is in the cache so nothing happens, then there is never any recheck that the name has not changed, not until the cache entry expires, then presumably the first unlucky client gets a delay - is that right? So the idea would be to extend some validity periods, by doing an unnecessary query straight to the authority, never mind cache entries, and then write a new cache entry with an extended validity period that stretches further into the future, and then no client is randomly unlucky, getting the avoidable delay. Am I completely wrong on this?

The thing is possibly not to extend lifetimes of entries that no one has any more interest in. How to decide that ?

It doesn’t amount to more load on the authoritative servers, if you are getting the design vaguely right and not prolonging stuff noone has any interest in. You are doing a query a bit earlier that’s all, before the point when validity period would have run out anyway.

I presume that I couldn’t just try and change this by doing ordinary queries anyway? Because they would just get answered from the cache and nothing would happen?
Logged

d2d4j

  • Kitizen
  • ****
  • Posts: 1103
Re: Another daft idea - musing again - DNS precaching
« Reply #6 on: October 16, 2018, 08:37:34 AM »

Hi weaver

I’m sorry, I hope I have understood your post correctly but apologies if I have not

In simple terms, the person in charge of the domain dns sets the TTL, and I suspect they would not want that to be altered in any way away from their set time. I certainly would not.

So if your idea is to say bring a new query into cache to extend the time, then that’s wrong, even if your query say brought the record into cache 1 second early to extend from say TTL 3600 to TTL 3601, which you would be doing.

Also, you are still querying so I am not sure what you are trying to achieve fully

Lastly, how is your idea going to change the users router/computer/etc dns cache

As I said, not fully understanding your post sorry but think you periodically keep raising this as a thread

Many thanks

John
Logged

jelv

  • Helpful
  • Kitizen
  • *
  • Posts: 2054
Re: Another daft idea - musing again - DNS precaching
« Reply #7 on: October 16, 2018, 10:29:05 AM »

There's nothing intricately wrong with re-fetching early as you will also get the current TTL - so exactly the same as if the entry had expired from the cache and had to be fetched anyway.

I'm sure that the operators of the major DNS services and writers of the DNS server software will have looked at every way possible of optimising their systems to get the best performance. People using https://www.grc.com/dns/benchmark.htm and similar will have driven them to make sure they come out in the best light. When it comes to ISP and hosting providers DNS servers I'd bet that hardware and network configuration is far more significant than minor tweaks to optimise the method.

Speculating with our very limited knowledge and without the masses of data to analyse performance of actual servers strikes me as a waste of time!
Logged
Broadband and Line rental: Zen Unlimited Fibre 2, Mobile: Vodaphone
Router: Fritz!Box 7530

j0hn

  • Kitizen
  • ****
  • Posts: 4093
Re: Another daft idea - musing again - DNS precaching
« Reply #8 on: October 16, 2018, 01:51:08 PM »

Quote from: Weaver
If clients keep querying popular names and the answer is in the cache so nothing happens, then there is never any recheck that the name has not changed, not until the cache entry expires, then presumably the first unlucky client gets a delay - is that right?

That's not my understanding of how DNS works.

You click a link, www.somesite.com
Your client (your iPad) will query the DNS server you have set. It will get the IP from the DNS server and it will cache it for the current TTL set.
There's no more querying of DNS servers for that particular hostname/IP by the iPad until the TTL expires.

It's your comment
Quote
then presumably the first unlucky client gets a delay - is that right?
that has me lost.

What unlucky client are you referring to?
Every client needs to query the DNS server. Each individual client has their own DNS cache.

Looks like (from the extremely limited documentation I can find) that the Firebrick does a form of DNS cache/relaying where it stores recently queried addresses for a short period.
Firebrick docs don't say how long for, if they use TTL, who knows.

Not all routers have their own DNS cache.

A DNS query only takes me about 20ms.
I can't see how you could optimize this.

You reference a "greatest hits". Can you explain how that would help? You would still need to query the DNS server to get this greatest hits list and it's associated IP's.

The current system works perfect. You query a host, the DNS server responds. Your client (and router in some cases) will cache it for a set time period so it doesn't need queried again.
Logged
Talktalk FTTP 550/75 - Speedtest - BQM

jelv

  • Helpful
  • Kitizen
  • *
  • Posts: 2054
Re: Another daft idea - musing again - DNS precaching
« Reply #9 on: October 16, 2018, 02:08:58 PM »

You are looking at it from the point of view of a user doing DNS lookups on EU equipment. Weaver's opening post made it clear he wasn't speculating about this:

Say you are a big ISP. You run a DNS relay prix cache for your customers. Knowing what the pattern of requests in the cache is, you could decide on a ‘greatest hits’ ‘top ten’ or whatever, a list of most requested domain names.

...
Logged
Broadband and Line rental: Zen Unlimited Fibre 2, Mobile: Vodaphone
Router: Fritz!Box 7530

Chrysalis

  • Content Team
  • Addicted Kitizen
  • *
  • Posts: 7382
  • VM Gig1 - AAISP L2TP
Re: Another daft idea - musing again - DNS precaching
« Reply #10 on: October 16, 2018, 02:41:46 PM »

weaver both prefetch and serve expired still fetch updated records they just dont make the client wait for it they do it in background to update cache for future lookups

they superior in that respect to the older techniques of bumping up ttl against domain admin wishes

jelv a dns relay that caches is a dns cache they the same thing, so tricks a home user can do apply tp isps as well, e.g. google dns is really just a dns cache hosted by google.
« Last Edit: October 16, 2018, 02:45:13 PM by Chrysalis »
Logged

j0hn

  • Kitizen
  • ****
  • Posts: 4093
Re: Another daft idea - musing again - DNS precaching
« Reply #11 on: October 16, 2018, 02:56:36 PM »

You are looking at it from the point of view of a user doing DNS lookups on EU equipment. Weaver's opening post made it clear he wasn't speculating about this:

Point still applies.
I can't think of a single scenario where a "greatest hits" would make any difference.

The customer still needs to query every single hostname once. It gets cached and the customer doesn't need to query the DNS server again till TTL.

DNS is pretty efficient and if it could be easily optimised I'm sure it would have been already.
Logged
Talktalk FTTP 550/75 - Speedtest - BQM

Weaver

  • Senior Kitizen
  • ******
  • Posts: 11459
  • Retd s/w dev; A&A; 4x7km ADSL2 lines; Firebrick
Re: Another daft idea - musing again - DNS precaching
« Reply #12 on: October 16, 2018, 10:23:31 PM »

> still fetch updated records they just dont make the client wait for it they do it in background to update cache for future lookups

I am no completely clear about the context, but this is good news, sound like someone is already doing the kind of thing I was looking for. But what limits the number of calls on through to the authoritative server? Apologies if I have failed to understand.

And to d2d4j, my sincere apologies if I have been guilty of repeating myself, which is highly likely, repeated posting. I can only ask your indulgence. Searching the database of posts is often difficult as deciding exactly what queries to use is not easy. The amount of drugs is the culprit, the pain medication means that my memory is completely shot and I can’t even remember stuff that happened only a week or two back. So mea maxima culpa,
Logged

d2d4j

  • Kitizen
  • ****
  • Posts: 1103
Re: Another daft idea - musing again - DNS precaching
« Reply #13 on: October 16, 2018, 10:54:40 PM »

Hi weaver

Many thanks and I fully understand sorry.

There’s more to this though then mention, eg geolocation, specific dns depending upon query il address, load balancers etc... so your just scratching the surface but as been mentioned, these have already been optimized and is still ongoing.  Eg powerdns

Many thanks and have a lovely night

John
Logged

Weaver

  • Senior Kitizen
  • ******
  • Posts: 11459
  • Retd s/w dev; A&A; 4x7km ADSL2 lines; Firebrick
Re: Another daft idea - musing again - DNS precaching
« Reply #14 on: October 16, 2018, 11:11:11 PM »

I would indeed love to find out more about this, about the innards of it. Then my questions might be a little less annoying, not so completely ill-informed. If I could find the source code for one popular DNS subsystem then I could read it, drugs permitting.

[Reading other people’s code is something that I loathe and I am very bad at it. But I am good at spotting performance cockups and bad practice, having done code reviews on occasion. (Am obsessive about micro performance for speed as I am an asm programmer through and through, and often look at the compiled asm output from a high level language compiler to see what kind of sins it is committing or to clarify exactly what some code is actually really doing.)]
Logged
Pages: [1] 2