Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Weaver]

I'm just wondering how much space it would take to store the entire DNS? Any ideas?

It is highly compressible and for my purposes the entire thing could be sorted and delta encoded as one of a number of multiple heavy compression methods.

[d2d4j]

Hi weaver

It is not a storage issue, rather the speed and accuracy of the served records that count, which the quicker the better, circa 0.2ms or better

Compressing dns records just adds to resolution of record, which is/has to be public, unless it's clustered and your talking of dns sync between NS

Many thanks

John

[Weaver]

Apologies, it's a design research idea, not a question about how the dns works. How much storage is needed to fully pre-populate a complete dns cache that covering the entire space down to every leaf? And I would think later about caching such a single database file into an optimised complete download.

[Weaver]

[d2d4j]

Hi weaver

Many thanks

I'm sorry, I am not sure what your idea is or it's use in the real world.

Are you proposing to replace the dns cache held on computers, or a new dns server

Many thanks

John

[Weaver]

I was wondering if a performance booster for ISP or smaller networks was feasible or even possible. It is a thought experiment. Having a hugely prepopulated cache locally would slash latency for DNS queries and cut startup times for web browsing down to near zero. But it would need a source of free broadcast bandwidth, something like satellite, to fill the cache before users need it.

I have used to be working on new designs professionally for many years. I keep coming up with new / stupid / impractical ideas. But some of them occasionally turn out to be very good. If no one checks out the stupid ideas to see if they are actually worth anything then things will never advance. It doesn't mean that I am optimistic, but that doesn't matter.

[Weaver]

If anyone ever sees some rough stats on the size of the DNS then pls let me know. I don't know how one might research such a thing.

Does anyone know if there are difficulties with crawling the DNS? I don't know enough about the subject. I realise that I have not considered the difficulties in obtaining the data to begin with. Perhaps that is a showstopper.

[d2d4j]

Hi weaver

Sorry, I'm not having a go at you, or criticising you in any way sorry

I think your proposing to replacing pc dns cache, but this works fine from what I see/experience, my personal thoughts only

The TTL is the determining factor for record cache, but it is unclear over when TTL starts, dns server record TTL or cache TTL. I think cache TTL


You maybe better looking at dns servers and how they can be improved, but large leaps forward have been made, but the overriding factors are speed and qps (queries per second) which they can handle

Just my thoughts so apologies if I seem to come across as been negative

Many thanks

John

[Weaver]

Not proposing to replace PC DNS cache at all. Indeed could never do so. This is a design for an accelerator to add on the side of an upstream proxy cache to be located one or two hops further up, solely in order to reduce query round-trip time on many normal-cache misses to near zero.

And it is merely a thought experiment, as I have stressed. I am not trying to sell the merits of this idea, just to get a feel for whether or not it is even possible. That is what a thought experiment is. You perhaps are not used to living with theoretical physicists. :-)

[d2d4j]

Hi weaver

Many thanks and sorry, your correct, I do not live in the theoretical world, only the real world, with real problems and real people. Pleas do not overreact to that, I mean no disrespect to you

I have always been labeled as seeing only black and white, no grey areas, which is true, but this has never stopped my thinking of real world applications, so I do try to think out side the box, but until the penny drops and I understand what your trying to do/against what is already in use, I cannot see what benefit it would produce.

Cache dns gives near zero lookups for dns, unless it's TTL has expired, at which it should take less then 0.2ms real world lookup time, so please forgive me, but I fail to see the application other then say saving a few 0.01ms time

Once again I apologise if I am wrong, it is just me and black and white look on life

Many thanks

John

[Weaver]

No problem d2d4j, and nor was there ever any, your comments were taken as helpful and valid opinion, we were just thinking along different lines. An you are also probably not full of (a) crap, and (b) NHS drugs, which kitizens are asked to make allowance for. 

[Weaver]

 if anyone knows anything about crawling the dns, then please comment.

[burakkucat]

I'm just wondering how much space it would take to store the entire DNS?

I read that question and thought "Huh?"    It gave me a very funny feeling which only subsided when I consulted the Wikipedia entry for Domain Name System.

Having stated that, I don't think I can contribute anything to your thought experiment . . . though I will certainly read whatever you are prepared to post. 

[Weaver]

If you pick a particular DNS server, can you suck the entire contents out of it, without particular privileged or non-standardised access?

[adrianw]

If you pick a particular DNS server, can you suck the entire contents out of it, without particular privileged or non-standardised access?

Some, not all, DNS servers support "zone transfer" (AXFR and an incremental version IXFR) to pull all the entries for a domain (sanitised example extracts below for my home system). This is normally used to allow slave DNS servers to catch up with the master when a change is made. Normally only permitted from certain IP addresses, such as the slaves. See https://en.wikipedia.org/wiki/DNS_zone_transfer

I can envisage an organisation's security people having fits if all the publicly addressable machines names and IP addresses were available as a neat list to all and sundry.

I can also envisage them thinking themselves under attack if you were to start doing a lot of lookups against their name servers, with reactions ranging from rate limiting you to hunting you down.

Also:

While some ISP's have good DNS servers for their customers, some have very poor ones with their wiser customers using Google, OpenDNS or other name servers. If an organization is not prepared to run a few good customer name servers, are they likely to pay out to enhance what they have?

There is not a simple 1:1 mapping between a name and address. Consider CNAMEs (a sort of alias), pools (where one name is supported by several addresses), the games played by content delivery networks, ...

An interesting project though. I seem to recall scorn being paid on the ideas of indexing and archiving the entire web, so don't be too put off.

Example zone transfer
Made on my home slave name server - a Beaglebone Black running FreeBSD and BIND.

[aw1@beaglebone ~]$ dig @localhost mydomain axfr

; <<>> DiG 9.10.3-P4 <<>> @localhost mydomain axfr
; (1 server found)
;; global options: +cmd
mydomain.     60      IN      SOA     swelter.mydomain. myemail. 2016123000 60 60 3600000 86400
mydomain.     60      IN      NS      titus.mydomain.
...
access4.mydomain. 60  IN      A       192.168.1.12
access4.mydomain. 60  IN      MX      1 access4.mydomain.
... (ad nauseaum - A and often MX entries for many hosts and all DHCP pool addresses))
mydomain.     60      IN      SOA     swelter.mydomain. myemail. 2016123000 60 60 3600000 86400
;; Query time: 12 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sat Jan 28 06:04:27 GMT 2017
;; XFR size: 325 records (messages 1, bytes 6834)