An Improved Farsight DNSDB Experience
The Farsight DNSDB Engineering Team is relentlessly “on the hunt” for ways to improve our DNSDB customers’ experience. One area prioritized for attention has been the speed of some DNSDB time-fenced queries. We’re happy to report that the engineering team has developed and deployed an optimization to help speed up those time-fenced queries.
What’s Time Fencing?
As discussed in past blog articles (such as Farsight’s DNSDB Time Fencing: A Post-Attack “Time Machine) DNSDB can limit its results to just results seen during a particular period of time.
For example, perhaps you’re only interested in results from the last seven days or last thirty days, or you’re only interested in results from a period of time when an incident was taking place. DNSDB has the ability to constrain its results to match those preferences using “time fencing.”
Traditional Query Performance WITHOUT Time Fencing
Historically, most DNSDB queries have run quite quickly. We can use the Un*x time command to measure how long sample commands take to run. Let’s consider two examples.
Example #1A (Without Time Fencing): Let’s see how long it takes to get up to a million results for *.uoregon.edu “A” records:
$ time dnsdbq -r \*.uoregon.edu/A -l0 -j > uoregon.edu.jsonl
real 0m1.542s
user 0m0.335s
sys 0m0.050s
$ wc -l uoregon.edu.jsonl
37560 uoregon.edu.jsonl
Example #2A (Without Time Fencing): Let’s also see how long it takes to get up to a million results for instagram.com “A” records. On March 10th, 2023, even including the time it took to download the million results to a remote broadband-connected Mac laptop, that took only a little over 15 seconds:
$ time dnsdbq -r instagram.com/A -l0 -j > instagram.com.jsonl
Database limit: Result limit reached
real 0m15.382s
user 0m7.906s
sys 0m0.594s
$ wc -l instagram.com.jsonl
1000000 instagram.com.jsonl
Both of those sample queries were fine from a timing point-of-view – they ran quickly.
Traditional Query Performance WITH Prior Time Fencing
But now what if we were to timefence those same queries? Many times, there’d be nothing comment-worthy:
Example #1B (WITH Traditional Time Fencing): Repeating our previous query for *.uoregon.edu “A” records WITH a 30 day timefence, our performance is actually even BETTER than it was un-timefenced:
$ time dnsdbq -r \*.uoregon.edu/A -l0 -j -A30d > uoregon2.edu.jsonl
real 0m1.181s
user 0m0.190s
sys 0m0.039s
$ wc -l uoregon2.edu.jsonl
16903 uoregon2.edu.jsonl
Example #2B (WITH Traditional Time Fencing): Our second example is more problematic when rerun with timefencing added. This time, when we queried DNSDB for instagram.com for “A” records and timefenced our results to just the last 30 days, our query took substantially longer to run – over four minutes instead of just roughly 15 seconds – even though we received far fewer results:
$ time dnsdbq -r instagram.com/A -l0 -j -A30d > instagram2.com.jsonl
real 4m1.517s
user 0m0.099s
sys 0m0.151s
$ wc -l instagram2.com.jsonl
63 instagram2.com.jsonl
Timefencing on the “New World” Servers
That difference in timefenced query performance bothered the engineering team (and we’re sure some users, too!), so we’re happy to report that the engineering team has successfully identified and resolved the associated issue. Rerunning the previous queries using the “New World” timefencing that went live on our production DNSDB servers as of March 24, 2023, we see:
Example #1C (Without Timefencing, on the “New World” server):
$ time dnsdbq -r \*.uoregon.edu/A -l0 -j > uoregon3.edu.jsonl
real 0m1.516s
user 0m0.307s
sys 0m0.053s
$ wc -l uoregon3.edu.jsonl
37561 uoregon3.edu.jsonl
Example #2C (Without Timefencing, on the “New World” server):
$ time dnsdbq -r instagram.com/A -l0 -j > instagram3.com.jsonl
Database limit: Result limit reached
real 0m12.541s
user 0m7.644s
sys 0m0.728s
$ wc -l instagram3.com.jsonl
1000000 instagram3.com.jsonl
Example #1D (WITH TIMEFENCING, on the “New World” Server):
$ time dnsdbq -r \*.uoregon.edu/A -l0 -j -A30d > uoregon4.edu.jsonl
real 0m2.128s
user 0m0.287s
sys 0m0.055s
$ wc -l uoregon4.edu.jsonl
16904 uoregon4.edu.jsonl
Example #2D (WITH TIMEFENCING, on the “New World” Server):
$ time dnsdbq -r instagram.com/A -l0 -j -A30d > instagram4.com.jsonl
real 0m0.693s
user 0m0.028s
sys 0m0.032s
$ wc -l instagram4.com.jsonl
63 instagram4.com.jsonl
Bottom line, on the “New World” server running the improved time fencing code, all the queries completed in just seconds, with or without timefencing! That makes us happy, and we think it will make you and other DNSDB users happy, too.
FAQ
Q0. What’s the “secret” behind this big performance improvement?
A. The actual meat of the change happened in our open source software stack (dnstable and dnstable-convert), so the changes aren’t really “secret.” The short form summary is:
- DNSDB stores data in multiple chunks, roughly split by time. We recently extended the storage format to include aggregate time ranges for data within each chunk and, when processing time fenced queries, we use these time ranges to search only those chunks which may contain data matching the query’s time filters. The results of this focused search are then “filled in” with data from the full DNSDB data set to provide accurate time ranges and counts, and returned to the user after a final filtering.
- The above method works for strict mode time fencing, where it particularly shines for domains with “combinatorial” A records (‘pick <n> IPs from a larger network range of <m> IPs for each “A” record returned’).
Q1. Do I need to change anything on my end? For example, do I need to use a new API key or change the name of the server I’m using?
A. No changes are needed on your side if you use DNSDB API (e.g., dnsdbq, DNSDB Scout, one of our DNSDB integrations, etc.) The “New World” server code is the new default, and your old API key will continue to work fine.
Q2. When will the “New World” code be available for DNSDB Export customers?
A. The code is public on GitHub and can be found https://github.com/farsightsec/dnstable and https://github.com/farsightsec/mtbl – the latest versions of both are required for the time filtering speedups. Packages will be available soon for Export users and we will provide an update once available.
Q3. Will the “New World’ server code ALWAYS return the same result counts as the code that was running on the previous server?
A. In general, yes, however note that new data is continually flowing into DNSDB from sensors, so newer results may be slightly larger than older results in some cases, as can be seen in this summary table for the example results shown above:
| Query | Time Fenced? | Code Version | Run Time | Result Count |
| *.uoregon.edu/A | N | Previous Code | 0m1.542s | 37,560 |
| *.uoregon.edu/A | N | “New World” Code | 0m1.542s | 37,561 |
| *.uoregon.edu/A | Y | Previous Code | 0m1.181s | 16,903 |
| *.uoregon.edu/A | Y | “New World” Code | 0m2.128s | 16,904 |
| instagram.com/A | N | Previous Code | 0m15.382s | 1,000,000 |
| instagram.com/A | N | “New World” Code | 0m12.541s | 1,000,000 |
| instagram.com/A | Y | Previous Code | 4m1.517s | 63 |
| instagram.com/A | Y | “New World” Code | 0m0.693s | 63 |
Q4. Will the “New World” server code ALWAYS run strictly faster than the previous code?
A. The “New World” code will normally be faster and have less variation in terms of extreme run times than the previous code, although caching and other day-to-day workload differences (including contention along the network path) may occasionally make the “New World” code run an insignificant amount slower. For example, compare the timefenced query times for *.uoregon.edu/A at 1.181 seconds (for the traditional code) vs 2.128 seconds (for the “New World” code) in the above table.
Q5. Is the old server code still running somewhere for comparison purposes? I’d like to do my own testing!
A. We’re not currently planning on doing so. The servers that handle DNSDB API queries are quite beefy, so dedicating one to running deprecated code wouldn’t be a prudent use of resources.
Q6. Were changes made to DNSDB Standard Search AND to DNSDB Flexible Search, or just DNSDB Standard Search?
A. We are only changing standard search. Flexible search already optimizes most time fenced searches.
Q7. I’ve run into an issue, or I’ve got other questions you haven’t answered!
A. Please drop us a note. We’d be glad to address your concern or question!
