Kitz ADSL Broadband Information
adsl spacer  
Support this site
Home Broadband ISPs Tech Routers Wiki Forum
 
     
   Compare ISP   Rate your ISP
 
Please login or register.

Login with username, password and session length
Advanced search  

News:

Pages: 1 [2]

Author Topic: How big is the DNS?  (Read 525 times)

Weaver

  • Kitizen
  • ****
  • Posts: 3847
  • Retd sw dev; A&A; 3 × 7km ADSL2; IPv6; Firebrick
Re: How big is the DNS?
« Reply #15 on: January 28, 2017, 07:50:28 AM »

Yes, I thought that access restrictions and partial support would make crawling the DNS a very iffy affair. Overloading one server is hardly necessary as you have the entire DNS to access so you could and should interleaved the accesses and spread the load around across servers queried.

I suspect the entire process of gathering the data would be a nightmare unless people signed up to contribute towards an acceleration project.

One other area that might be worth looking at some time might be the use of multicast in the DNS, but I am doubtful initially.
Logged

adrianw

  • Member
  • **
  • Posts: 67
Re: How big is the DNS?
« Reply #16 on: January 28, 2017, 08:14:33 AM »

Yes, interleaving would make you less noticeable.

Do you feel a RFC coming on (list of DNS related ones at http://www.bind9.net/rfc )?

Multicast DNS, limited to local networks, already exists - https://en.wikipedia.org/wiki/Multicast_DNS and https://tools.ietf.org/html/rfc6762
Logged

Weaver

  • Kitizen
  • ****
  • Posts: 3847
  • Retd sw dev; A&A; 3 × 7km ADSL2; IPv6; Firebrick
Re: How big is the DNS?
« Reply #17 on: January 28, 2017, 08:18:42 AM »

I was unclear - I am aware of mDNS and LLMNR. I meant servers pushing changes to downstream servers that subscribe, not clients using multicast to query servers or other hosts directly for the purposes of zero-config.
Logged

adrianw

  • Member
  • **
  • Posts: 67
Re: How big is the DNS?
« Reply #18 on: January 28, 2017, 08:59:18 AM »

Ah. With you now, I think. No idea of how to achieve it.
Logged

d2d4j

  • Reg Member
  • ***
  • Posts: 390
Re: How big is the DNS?
« Reply #19 on: January 28, 2017, 09:07:31 AM »

Hi weaver

Interworx already do this for dns to a degree, where a publisher is set and a listener is setup

Interworx can also use axfr as it is not bind, so axfr is needed for bind dns servers (which can be set either record, domain(s) or server wide for dns zones)

I can post a picture of dns sync between 2 or more services

Many thanks

John
Logged

d2d4j

  • Reg Member
  • ***
  • Posts: 390
Re: How big is the DNS?
« Reply #20 on: January 28, 2017, 09:10:51 AM »

Sorry, I should have also said, this can be 1 way or 2 way direction, as some of the dns servers acting as listeners, can also be more then dns slaves, they could be primary dns as well

Many thanks

John
Logged

d2d4j

  • Reg Member
  • ***
  • Posts: 390
Re: How big is the DNS?
« Reply #21 on: January 28, 2017, 09:33:28 AM »

Hi

Sorry, iPhone updated

The sync is not immediate though, it is timed sync, so if your idea was immediate sync, then interworx does not do that, but I could envisage big issues on high volume dns servers given the number of changes made

Many thanks

John
Logged

Weaver

  • Kitizen
  • ****
  • Posts: 3847
  • Retd sw dev; A&A; 3 × 7km ADSL2; IPv6; Firebrick
Re: How big is the DNS?
« Reply #22 on: January 28, 2017, 09:59:40 AM »

I really, really need to read up more on this subject. Memory is also faulty.

I wonder what log2(size of a typical ISP's cache) is? That is a question someone could answer. That would be an interesting way of getting round the problem of how to do crawling, instead use a very large organisation's cache as a source for broadcast.

But this idea if it were to be any use has to be delivered over zero-cost, unused bandwidth. It is not supposed to slow things down by competing for downstream bandwidth with important activity on your main pipes.

A lot of this would really really need to be done based on a classification of upstream servers as trusted or untrusted and their identities ought to be checked. Lying servers or redirects or men-in-the-middle or whatever would not be good.

Yahoo for example has published quite a bit on web browser performance and stresses what a killer the startup delay is due to DNS queries having to complete before a web browser can get anything else at all done in the very beginning. This suggests that this latency is never going to go away as bandwidth gets better and better, so its relative importance as a fraction of the initial webpage load time is in fact going to increase. Other items in a webpage, such as bitmaps or JS or ads on third party sites are going to be paralleliseable if there are several of them and if the browser can get to parsing enough stuff to discover references to these domain names soon enough. It would be a performance boost if pages could give prefetch domain name references very early, but that is often impossible, with dynamic stuff, rotating ads, references hidden inside material yet to be fetched.

This could be avoided by having servers take copies of third party objects and keeping them on the same server or at least on a server inside the same domain. Would stuff up ad blockers and defeat privacy champions, bad for us and good for the bad guys. Also thus kind of duplication would prevent multi-site caching of say shared JS and CSS, very bad for everyone. So I had better not mention it.

[Moderator edited to fix the damaged "Subject" line.]
« Last Edit: January 28, 2017, 03:39:00 PM by burakkucat »
Logged

Weaver

  • Kitizen
  • ****
  • Posts: 3847
  • Retd sw dev; A&A; 3 × 7km ADSL2; IPv6; Firebrick
Re: How big is the DNS?
« Reply #23 on: January 28, 2017, 10:11:29 AM »

A couple of other ideas:

1. Switching topics: cache-to-cache low-priority traffic delivered overnight over the conventional internet to populate caches from very fat upstream caches might be a useful idea, but only if the DNS entries have a long lifetime, and they are the ones that least need this help anyway. It is no use if the lifetime expires again halfway through the day.

2. (Unrelated.) I also wonder about whether or not it would be useful for a cache to refresh its entries before the lifetime expires, i.e. prefetch, but only selectively. If a cache entry has been repeatedly queried (scoring a hit) many times recently, then that means it is still useful to clients, or useful to many clients, anyway it means it is therefore worthy of being speculatively refreshed before it expires which will also get you a lifetime extension. Something could perhaps be worked out from the complete time distribution of hits on a cache entry.

3. I wonder if it would be worthwhile for intermediate upstream DNS caches to give out info about how often an entry is queried as part of a response or as part of a sync transfer. Would give downstream servers extra information about how useful a possible prefetch might conceivably be, but the downstream caches can know the pattern of their own clients’ queries, and it is that of course that is the most valuable statistical information on which to base decisions, as it is the more relevant, because it is client behaviour-specific and driven by the distribution of clients present.
Logged

d2d4j

  • Reg Member
  • ***
  • Posts: 390
Re: How big is the DNS?
« Reply #24 on: January 28, 2017, 10:13:08 AM »

Hi weaver

I think I'm understanding a little more, I am rather slow sometimes sorry

Please remember pc dns cache, browser cache, isp dns cache, which all play a factor in your idea

If you go to pingdom.com, test a web page and it will show you dns lookup times. There is another web page tester which we use, but I just cannot think of it right now, but again, shows where the slowness comes from, dns lookup, file size, CDN etc

Also, you may want to look at powerDNS and cloudflare

I thought last night, if I understood correctly, you have a new compression algorithm, perhaps this may be better aimed at browser compression for internet or even for live sat nav

I know I have most likely fully misunderstood your idea sorry, but it does take time for my single brain cell to understand

Many thanks

John
Logged
Pages: 1 [2]
 

anything