While setting up PowerDNS with geobackend, there are a few configuration files that need to be supplied. In this post, I will talk about how I am trying to generate the
What does the map-zonefile do?
Like the name suggests, this file maps IP addresses to servers. For instance,
Here, the IP address
22.214.171.124/22 is mapped to the pseudo IP address of the server.
127.0.0.1 is some server X which is geographically closest to this IP address.
How to generate the file
Until the last week, I had used the Whois database to get IP addresses of networks and check which networks fall into which Voronoi cells. Now, I am just converting these IPs from the Whois database into the CIDR format and printing a
:127.0.0.X next to it where X is the serial number assigned to a server.
Getting the Required Data
I parsed the XML-style formatted file containing information about the mirror servers which are a part of the CERN CDN. I had done this earlier, but all of that had to be changed to get more information for generating the PDNS config files, for instance the Localsite field along with the domain name. The code which does this specifically can be found on this repository.
This is how a typical entry in the Whois database looks:
There is a
126.96.36.199 and the ‘toIP
is '188.8.131.52. From this I had to get the network IP in CIDR format so as to write into the map-zonefile. After looking around, I found out about the netaddr library which does the conversion from IP range to CIDR format IP addresses. Like:
>>> import netaddr >>> start_ip = "184.108.40.206" >>> end_ip = "220.127.116.11" >>> ip_range = list( netaddr.iter_iprange( start_ip, end_ip ) ) >>> netaddr.cidr_merge( ip_range ) [IPNetwork('18.104.22.168/24')]
cidr_merge and found that this library is quite slow. So, it was not practical to use it for a million conversions. An alternative to netaddr suggested on StackOverflow was the the cidrize library. It is much faster as compared to netaddr. Here is how it works:
>>> from cidrize import cidrize >>> fromIP = "22.214.171.124" >>> toIP = "126.96.36.199" >>> ip_range = fromIP >>> ip_range = fromIP + "-" + toIP >>> cidrize( ip_range ) [IPNetwork('188.8.131.52/24')]
Using this I have tried to generate the map-zonefile. For 160000 or so entries even this library takes quite a lot of time.
Here are the first few lines from the generated file:
184.108.40.206/24 :127.0.0.59 220.127.116.11/22 :127.0.0.25 18.104.22.168/22 :127.0.0.59 22.214.171.124/21 :127.0.0.5 126.96.36.199/20 :127.0.0.25
Bringing Together Data Pre-processing and The Computation
Until now, I was using many separate scripts to process the server names followed by getting their location from the GeoIP database and writing the outputs to files. These intermediate files were causing trouble. So, in place of writing the data into flat files, I am just building a dictionary of the data and passing it to the main method of the script that does all the computation. Further details and code are in this repository!