1. Introduction
In a recent Farsight blog article, three new features were announced, including “Incremental Result Transfers.” The article discussed the new features in the context of DNSDB Scout. In this article, I will focus specifically on how “Incremental Result Transfers” work if you’re using dnsdbq and it’s -O (“offset”) option. The motivation for this new feature is simple: Farsight knows that some users may want more than a million results. While you’re still limited to no more than a million results for any single query, you can now “dive deeper” by making additional queries “offset” into the pool of total results.
2. So How Deep Can I Dive?
The extent to which your account is able to request additional results can be seen with dnsdbq -I
$ dnsdbq -I quota: reset: 2019-09-10 00:00:00 limit: 1000 remaining: 1000 results_max: 1000000 offset_max: 3000000
This quota report means that you can get up to FOUR million total results using the -O option in dnsdbq:
Query Offset Limit (results/query) ------ Total: 4,000,000 results
If your account is NOT permitted to use the offset feature, you’ll see "N/A" for offset_max. For example, someone using a Farsight DNSDB Community Edition API key would see something like:
$ dnsdbq -I quota: reset: n/a expires: 2019-10-09 13:47:47 limit: 500 remaining: 500 results_max: 256 offset_max: n/a burst rate: 25 per hour
3. A Worked Example
Request a million results for *.mit.edu domains in JSON Lines format using dnsdbq:
$ dnsdbq -r \*.mit.edu -l 1000000 -j > mit1.jsonl
Now let’s request a second tranche, offset by a million:
$ dnsdbq -r \*.mit.edu -l 1000000 -O 1000000 -j > mit2.jsonl
And a third tranche, offset by two million:
$ dnsdbq -r \*.mit.edu -l 1000000 -O 2000000 -j > mit3.jsonl
And add a fourth tranche, offset by three million:
$ dnsdbq -r \*.mit.edu -l 1000000 -O 3000000 -j > mit4.jsonl
Those four queries left us with:
$ wc -l mit[1234].jsonl 1000000 mit1.jsonl 1000000 mit2.jsonl 1000000 mit3.jsonl 146830 mit4.jsonl 3146830 total
Now let’s combine and deduplicate those results…
$ cat mit[1234].jsonl | sort -u > mit-combo.jsonl $ wc -l mit-combo.jsonl 3146826 mit-combo.jsonl <-- there were apparently 3146830-3146826=4 dupes
Note that we had four “dupes” even when we ran the four queries one right after the other.
4. Understanding the Limitations To The -O option to dnsdbq
Asking for all four tranches of results would require expenditure of four queries from your quota, not just one.
Farsight also cannot guarantee that there will be zero overlap between the sets of results returned since the underlying data may change while your queries are running as new data is received and incorporated. That is, we don’t “save” or “freeze” a copy of all results for each query you make, so we can’t strictly guarantee continuity of query results across tranches. If you think you may want more than one tranche full of results, we recommend you run all tranches in close succession for minimal overlap (like the four dupes in our example above).
If you’re going to sort and deduplicate your results (as shown in the example in the previous section), you’ll want to be sure to use JSON Lines format for output (e.g., dnsdbq's -j option) to ensure each observation is contained on a single line. Attempting to use traditional presentation format for output would “complicate” sorting since traditional presentation format spans multiple lines.
You should also note that the same RRname/RRtype Bailiwick/Rdata value may be seen multiple times in the output, perhaps with a count (or last-seen time) that has updated in the time between an initial run and a subsequent run.
5. Conclusion
If you’ve ever been frustrated when you have bumped into the max-million-results-per-query limit, the ability to now get up to four million results on most DNSDB API keys will be most-welcome news. We encourage you to explore this new feature and all the other new DNSDB API features!
The Farsight Security Sales Team can be reached at [email protected].
Joe St Sauver Ph.D. is a Distinguished Scientist with Farsight Security®, Inc.
