DNSDB Export Requirements
Introduction
DNSDB Export is an “on-premises” version of DNSDB API, perfect for situations where:
- Query latency must be minimized
- Total control over operation is required
- Technically-assured query privacy is mandated, or
- A partitioned network architecture may be in use.
DNSDB Export is intended to be installed on servers owned or controlled by the customer within a secure site. These servers must ensure that DNSDB data can only be accessed by authorized users.
DNSDB Export data is leased, not sold. If the contract for DNSDB Export terminates, all DNSDB data and related proprietary software must be deleted and may not be retained. For that reason, DNSDB data must be segregated from other data sets and cannot be merged into other databases or comingled with non-
DomainTools data.
DNSDB Export Standard Search is the foundation service. All DNSDB export customers will normally take DNSDB Standard Search data for regular DNS. Flexible Search data and Standard Search DNSSEC data are available as add-ons.
DNSDB Export Standard Search data gets stored in MTBL files, which are compressed, indexed binary files that store key-value entries. There are separate file sets for regular DNS and DNSSEC data. Flexible Search data is also split into two sets (one set for RRnames, and another for Rdata). If you take either, you will normally take both Flexible Search RRname and Rdata datasets.
MTBL files come in minutely, hourly, daily, monthly, and yearly file varieties. Shorter duration files normally get “rolled up” or aggregated into longer duration files. This saves space and reduces the number of files that need to be accessed during queries.
DNSDB Export customers can also choose the level of “near real-time delivery” they require, purchasing either monthly, daily, hourly, or minutely updates to their DNSDB Export data, with pricing representing the value and network traffic associated with those options.
Farsight provides the software to query these files using the same RESTful APIs as seen in the Farsight DNSDB service.
Data in MTBL files can be exported into a JSON format. This JSON data will typically be 3.8x the size of the MTBL file from which it is exported. The Flexible search data files are in a proprietary format, we do not support extracting the raw data.
DNSDB Export NVMe Storage Requirements
DNSDB Export works best when run on NVMe storage. Rotating disk will NOT deliver adequate performance.
Data storage capacity requirements continually increase. Our general recommendation is to give yourself as much headroom as possible. That is, use a chassis with plenty of slots for NVMe disks and use the highest density NVME disks certified for use in that chassis.
Estimated Usable Disk Required Per Product (rounded up to the next TB) Future growth is hard to predict. Please treat these values as estimates.
DNSDB Standard Search Regular DNS MTBL Files
2012-2023: 32 TB
2024: 11 TB
2025: 13 TB
2026 15 TB
Total through 2026: 71 TB
DNS Flexible Search (RRname and Rdata) Data Files
2012-2023: 19 TB
2024: 7 TB
2025: 8 TB
2026: 9 TB
Total through 2026: 43 TB
DNSDB Standard Search DNSSEC MTBL Files
2012-2023: 106 TB
2024: 48 TB
2025: 55 TB
2026: 64 TB
Total through 2026: 273 TB
Notes
- Yearly rollups for DNSDB Standard Search MTBL files are only available through 2020. After December 2020, monthly rollups are the most-aggregated files available for DNSDB Standard Search.
- Flexible Search is a finding aid; it requires the customer to have DNSDB Standard Search (either DNS only or DNS+DNSSEC) for full data.
- Standard Search DNSSEC files are large because they consist of unique encrypted content that compresses poorly. Most customers will not need nor want to take DNSSEC, but it is available at no additional charge with Standard Search if customers want it. Note the substantial disk space required!
- The disk space usage estimates shown above reflect data from our sensor network, and do NOT include CZDS zone file data. (Due to ICANN Terms of Service, we can’t provide Zone File data in bulk, but we can help customers convert and ingest Zone File data they may already have or which they may get during the term of their contract with DomainTools.)
- The above estimates are for USABLE storage, not RAW storage. Most customers will use RAID 6, in which case usable storage can be estimated at roughly 75% of raw storage (for example, 100TB of raw disk translates to 75TB of usable RAID 6-protected disk).
- A customer can limit what data they download and synchronize if necessary (current year vs prior years). Some analysts might care only what is available in the last few weeks or months and not want to include historical data (or they’re not willing to pay for the storage required to hold historical data).
- The above disk estimates do not consider potential intra- or inter-site replication/redundancy.
Recommended Hardware Configuration (as of October 2023):
- Chassis: Supermicro 2125HS-TNR
- CPU: 2 CPUs, total of 64 cores, 128 threads (AMD EYPC 9334)
- Because we are using NVMe disks, we need lots of PCIe lanes. EPYC CPUs tend to have more lanes than Xeon CPUs.
- RAM: 1.5TB DDR5 ECC REG
- Storage: 24x 7.8TB Solidigm D7-P5520 NVMe – 2 disks for OS (RAID 1), 21 disks for data (RAID 6), 1 hot spare / additional space as needed. This server also supports higher density NVMe drives, check the SuperMicro website for a list of certified models.
- Network: 2x10GigE
Network Requirements
A high speed network connection with sustained excess capacity of 100 Mbps is recommended. The database files are transferred over multiple HTTPS connections and throughput is normally limited to a maximum of 100 Mbps to mitigate unexpected resource exhaustion. Depending on which file sets are chosen it is possible that full 100 Mbps sustained network usage will be experienced for several days during the initial synchronization as well as future updates. (Initial DNSDB Export data can also be provided over couriered encrypted drives to avoid initial startup delays while awaiting network synchronization).
DNSDB Operating System Requirements
We support and have binary packages for:
- Debian 11 (Bullseye) and 12 (Bookworm – currently in testing)
- Ubuntu 20.04 LTS (Focal Fossa) and Ubuntu 22.04 LTS (Jammy Jellyfish – currently in testing)
- Rocky 9 (Blue Onyx)
- Packages for older Linux versions may be available on a case by case basis
Consultation
The Customer Solutions Engineering team is always available for consultation to assist with design and implementation.
