Kitz ADSL Broadband Information
adsl spacer  
Support this site
Home Broadband ISPs Tech Routers Wiki Forum
 
     
   Compare ISP   Rate your ISP
 
Please login or register.

Login with username, password and session length
Advanced search  

News:

Pages: [1] 2

Author Topic: How big is the DNS?  (Read 710 times)

Weaver

  • Kitizen
  • ****
  • Posts: 4004
  • Retd sw dev; A&A; 3 × 7km ADSL2; IPv6; Firebrick
How big is the DNS?
« on: January 27, 2017, 11:14:39 AM »

I'm just wondering how much space it would take to store the entire DNS? Any ideas?

It is highly compressible and for my purposes the entire thing could be sorted and delta encoded as one of a number of multiple heavy compression methods.
Logged

d2d4j

  • Reg Member
  • ***
  • Posts: 399
Re: How big is the DNS?
« Reply #1 on: January 27, 2017, 11:29:10 AM »

Hi weaver

It is not a storage issue, rather the speed and accuracy of the served records that count, which the quicker the better, circa 0.2ms or better

Compressing dns records just adds to resolution of record, which is/has to be public, unless it's clustered and your talking of dns sync between NS

Many thanks

John
Logged

Weaver

  • Kitizen
  • ****
  • Posts: 4004
  • Retd sw dev; A&A; 3 × 7km ADSL2; IPv6; Firebrick
Re: How big is the DNS?
« Reply #2 on: January 27, 2017, 09:02:16 PM »

Apologies, it's a design research idea, not a question about how the dns works. How much storage is needed to fully pre-populate a complete dns cache that covering the entire space down to every leaf? And I would think later about caching such a single database file into an optimised complete download.
Logged

Weaver

  • Kitizen
  • ****
  • Posts: 4004
  • Retd sw dev; A&A; 3 × 7km ADSL2; IPv6; Firebrick
Re: How big is the DNS?
« Reply #3 on: January 27, 2017, 09:20:22 PM »

Imagine transmitting the entire DNS by broadcast one-way satellite links, on several channels. On one group of channels you transmit simply everything, in a rotating fashion, in order to populate a DNS cache. On a second channel group, you transmit all the frequently-used stuff only, otherwise it's the same as the first. On a third group of channels, you transmit all the updates as they happen. One final channel group, group four, has only one channel in it and duplicates of all the group three content combined into a single channel.

Each channel is very highly compressed, and is pre-sorted so that delta compression could be one of the early compression layers. After every so-many records, the stream of data has to be broken up in such a way that a listener can synch up with the compression scheme and get into understand it having only missed a certain number or seconds worth of entries before they hear a synchronisation symbol followed by enough info to get them started with the compression scheme

When you first become a client, you listen to as many channels in group two as you have radios, in order to get up and running with something useful as quickly as possible. If you have even more radios, you listen to group one channels as well, otherwise you switch to absorbing group one after group two. Group two content is not duplicated in group one, it is simply omitted silently, so a new client always has to listen to all of both channel groups eventually.

The groups one to three are partitioned so that a channel in a group only carries a known part of the DNS, which increases the speed of rotation, and allows faster startup time if you have multiple radios. A fast, cheap hash function tells you which channel in a group a particular domain name will be mentioned on. Info with a list of parameters to the hash function and a map of the set of channels are transmitted on every channel frequently. Each channel simply rotates, it starts transmitting again from the beginning when it gets to the end of the data.

This partially completed design has some outstanding problems, the main one currently being the synchronisation between group one and group three / four content. Including some data in group three that is slightly old - but how old? - bridges the gap between the intialisation-time ‘full’ database and the live updates’ stream. Actually though, getting it wrong is not critical, because this is only a cache and being incomplete simply means that an attached normal DNS server has to consult the real live DNS more often. Also, without an estimate of database sizes and channel throughput I don't know if this plan is useful or even feasible. Neither do I know what the rate of DNS updates is, so I don't know whether channel four might be able to keep up with the rate of change of the entire world.

If the numbers are too large however, I could always simply reduce the scope of problem, and deal with only part of the database, such as the well-known TLDs plus one or more ccTLDs.
Logged

d2d4j

  • Reg Member
  • ***
  • Posts: 399
Re: How big is the DNS?
« Reply #4 on: January 27, 2017, 09:55:28 PM »

Hi weaver

Many thanks

I'm sorry, I am not sure what your idea is or it's use in the real world.

Are you proposing to replace the dns cache held on computers, or a new dns server

Many thanks

John
Logged

Weaver

  • Kitizen
  • ****
  • Posts: 4004
  • Retd sw dev; A&A; 3 × 7km ADSL2; IPv6; Firebrick
Re: How big is the DNS?
« Reply #5 on: January 27, 2017, 10:31:39 PM »

I was wondering if a performance booster for ISP or smaller networks was feasible or even possible. It is a thought experiment. Having a hugely prepopulated cache locally would slash latency for DNS queries and cut startup times for web browsing down to near zero. But it would need a source of free broadcast bandwidth, something like satellite, to fill the cache before users need it.

I have used to be working on new designs professionally for many years. I keep coming up with new / stupid / impractical ideas. But some of them occasionally turn out to be very good. If no one checks out the stupid ideas to see if they are actually worth anything then things will never advance. It doesn't mean that I am optimistic, but that doesn't matter.
Logged

Weaver

  • Kitizen
  • ****
  • Posts: 4004
  • Retd sw dev; A&A; 3 × 7km ADSL2; IPv6; Firebrick
Re: How big is the DNS?
« Reply #6 on: January 27, 2017, 10:33:52 PM »

If anyone ever sees some rough stats on the size of the DNS then pls let me know. I don't know how one might research such a thing.

Does anyone know if there are difficulties with crawling the DNS? I don't know enough about the subject. I realise that I have not considered the difficulties in obtaining the data to begin with. Perhaps that is a showstopper.
Logged

d2d4j

  • Reg Member
  • ***
  • Posts: 399
Re: How big is the DNS?
« Reply #7 on: January 27, 2017, 10:40:31 PM »

Hi weaver

Sorry, I'm not having a go at you, or criticising you in any way sorry

I think your proposing to replacing pc dns cache, but this works fine from what I see/experience, my personal thoughts only

The TTL is the determining factor for record cache, but it is unclear over when TTL starts, dns server record TTL or cache TTL. I think cache TTL


You maybe better looking at dns servers and how they can be improved, but large leaps forward have been made, but the overriding factors are speed and qps (queries per second) which they can handle

Just my thoughts so apologies if I seem to come across as been negative

Many thanks

John
Logged

Weaver

  • Kitizen
  • ****
  • Posts: 4004
  • Retd sw dev; A&A; 3 × 7km ADSL2; IPv6; Firebrick
Re: How big is the DNS?
« Reply #8 on: January 27, 2017, 10:50:00 PM »

Not proposing to replace PC DNS cache at all. Indeed could never do so. This is a design for an accelerator to add on the side of an upstream proxy cache to be located one or two hops further up, solely in order to reduce query round-trip time on many normal-cache misses to near zero.

And it is merely a thought experiment, as I have stressed. I am not trying to sell the merits of this idea, just to get a feel for whether or not it is even possible. That is what a thought experiment is. You perhaps are not used to living with theoretical physicists. :-)
« Last Edit: January 27, 2017, 10:53:09 PM by Weaver »
Logged

d2d4j

  • Reg Member
  • ***
  • Posts: 399
Re: How big is the DNS?
« Reply #9 on: January 27, 2017, 11:14:50 PM »

Hi weaver

Many thanks and sorry, your correct, I do not live in the theoretical world, only the real world, with real problems and real people. Pleas do not overreact to that, I mean no disrespect to you

I have always been labeled as seeing only black and white, no grey areas, which is true, but this has never stopped my thinking of real world applications, so I do try to think out side the box, but until the penny drops and I understand what your trying to do/against what is already in use, I cannot see what benefit it would produce.

Cache dns gives near zero lookups for dns, unless it's TTL has expired, at which it should take less then 0.2ms real world lookup time, so please forgive me, but I fail to see the application other then say saving a few 0.01ms time

Once again I apologise if I am wrong, it is just me and black and white look on life

Many thanks

John
Logged

Weaver

  • Kitizen
  • ****
  • Posts: 4004
  • Retd sw dev; A&A; 3 × 7km ADSL2; IPv6; Firebrick
Re: How big is the DNS?
« Reply #10 on: January 27, 2017, 11:32:19 PM »

No problem d2d4j, and nor was there ever any, your comments were taken as helpful and valid opinion, we were just thinking along different lines. An you are also probably not full of (a) crap, and (b) NHS drugs, which kitizens are asked to make allowance for.  ;D ;D
Logged

Weaver

  • Kitizen
  • ****
  • Posts: 4004
  • Retd sw dev; A&A; 3 × 7km ADSL2; IPv6; Firebrick
Re: How big is the DNS?
« Reply #11 on: January 27, 2017, 11:33:12 PM »

 if anyone knows anything about crawling the dns, then please comment.
Logged

burakkucat

  • Global Moderator
  • Senior Kitizen
  • *
  • Posts: 18790
  • Over the Rainbow
    • The ELRepo Project
Re: How big is the DNS?
« Reply #12 on: January 27, 2017, 11:38:21 PM »

I'm just wondering how much space it would take to store the entire DNS?

I read that question and thought "Huh?"  ???  It gave me a very funny feeling which only subsided when I consulted the Wikipedia entry for Domain Name System.

Having stated that, I don't think I can contribute anything to your thought experiment . . . though I will certainly read whatever you are prepared to post.  :)
Logged
:cat:  100% Linux and, previously, Unix. Co-founder of the ELRepo Project.

Please consider making a donation to support the running of this site.

Weaver

  • Kitizen
  • ****
  • Posts: 4004
  • Retd sw dev; A&A; 3 × 7km ADSL2; IPv6; Firebrick
Re: How big is the DNS?
« Reply #13 on: January 27, 2017, 11:58:39 PM »

If you pick a particular DNS server, can you suck the entire contents out of it, without particular privileged or non-standardised access?
Logged

adrianw

  • Member
  • **
  • Posts: 67
Re: How big is the DNS?
« Reply #14 on: January 28, 2017, 07:37:28 AM »

If you pick a particular DNS server, can you suck the entire contents out of it, without particular privileged or non-standardised access?

Some, not all, DNS servers support "zone transfer" (AXFR and an incremental version IXFR) to pull all the entries for a domain (sanitised example extracts below for my home system). This is normally used to allow slave DNS servers to catch up with the master when a change is made. Normally only permitted from certain IP addresses, such as the slaves. See https://en.wikipedia.org/wiki/DNS_zone_transfer

I can envisage an organisation's security people having fits if all the publicly addressable machines names and IP addresses were available as a neat list to all and sundry.

I can also envisage them thinking themselves under attack if you were to start doing a lot of lookups against their name servers, with reactions ranging from rate limiting you to hunting you down.

Also:

While some ISP's have good DNS servers for their customers, some have very poor ones with their wiser customers using Google, OpenDNS or other name servers. If an organization is not prepared to run a few good customer name servers, are they likely to pay out to enhance what they have?

There is not a simple 1:1 mapping between a name and address. Consider CNAMEs (a sort of alias), pools (where one name is supported by several addresses), the games played by content delivery networks, ...

An interesting project though. I seem to recall scorn being paid on the ideas of indexing and archiving the entire web, so don't be too put off.

Example zone transfer
Made on my home slave name server - a Beaglebone Black running FreeBSD and BIND.
[aw1@beaglebone ~]$ dig @localhost mydomain axfr

; <<>> DiG 9.10.3-P4 <<>> @localhost mydomain axfr
; (1 server found)
;; global options: +cmd
mydomain.     60      IN      SOA     swelter.mydomain. myemail. 2016123000 60 60 3600000 86400
mydomain.     60      IN      NS      titus.mydomain.
...
access4.mydomain. 60  IN      A       192.168.1.12
access4.mydomain. 60  IN      MX      1 access4.mydomain.
... (ad nauseaum - A and often MX entries for many hosts and all DHCP pool addresses))
mydomain.     60      IN      SOA     swelter.mydomain. myemail. 2016123000 60 60 3600000 86400
;; Query time: 12 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sat Jan 28 06:04:27 GMT 2017
;; XFR size: 325 records (messages 1, bytes 6834)

Logged
Pages: [1] 2