1. Introduction
Most users interact with DNSDB either through the web interface or via the sample command line interface (CLI) tools Farsight provides).
However, sometimes you may need to make complex or conditional queries that don’t fit well with either of those options. In that case, DNSDB API access customers can call the DNSDB API directly from their own application.
The DNSDB API documentation describes how to make “bulk, automated DNSDB queries via the HTTP API” but it doesn’t provide any actual code samples.
Rather than trying to build a complex application for this post, let’s just build a basic example that uses libcurl to make calls to the DNSDB API.
2. libcurl
libcurl is the API version of the curl command line web client we all know and love.
Installation instructions for libcurl are available here and documentation is available at here.
Some alternatives to libcurl are available here should you want to check them out.
Please note that libcurl is under active development. The version I installed and used for the following example was:
$ curl-config --version libcurl 7.50.3 [released on Sept 14, 2016]
For details about libcurl’s release history and changes, go here and here. I strongly encourage you to use the most recent version of libcurl, and be sure that your installation of openssl is fully up-to-date, too.
3. Sample simple skeleton C code to make DNSDB queries
For this example, we just want to build a small C program that will take a list of fully qualified domain names from stdin and pump them through DNSDB, returning the results to stdout. We could easily do this with the existing CLI clients available for use with DNSDB, but we wanted a simple example that nicely illustrates the basics of making DNSDB API calls with libcurl.
The full range of API command options described at in the documentation can be easily added to this example, if desired.
Note: Speaking of extending this example, this skeleton code is just that, an example, and does not purport to be “production-grade” (e.g., it embeds the user’s API key, it assumes the input is trusted and doesn’t explicitly sanitize it, it does not do extensive error checking, it only takes the first dozen results for each domain, etc.).
Anyone using this example as the basis for actual production code should enhance/rewrite the sample as may be appropriate for their environment and their unique requirements.
libcurl-sample.c
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <curl/curl.h>
int main (int argc, char **argv)
{
CURL *curl;
CURLcode res;
char mydomain[1024], tempstring[1024], fullcommand[1024];
/* initialize curl. must be called once and only once. */
if (curl_global_init(CURL_GLOBAL_ALL) != 0)
{
fprintf(stderr, "curl_global_init() failed\n");
return (EXIT_FAILURE);
}
/* get base domain from stdin */
while (scanf("%s", mydomain) == 1)
{
/* build the command we want to pass to curl */
/* all our commands use the same basic RESTFUL API endpoint... */
strcpy(fullcommand, "https://api.dnsdb.info/lookup/rrset/name/");
/* now tack on the domain */
strcpy(tempstring, mydomain);
strcat(fullcommand, tempstring);
/* just give me a token dozen results */
strcpy(tempstring, "?limit=12");
strcat(fullcommand, tempstring);
/* get curl ready for action */
curl = curl_easy_init();
/* pass the API key */
struct curl_slist *chunk = NULL;
chunk = curl_slist_append(chunk, "X-API-Key: [elided]");
res = curl_easy_setopt(curl, CURLOPT_HTTPHEADER, chunk);
/* do the actual curl command */
curl_easy_setopt(curl, CURLOPT_URL, fullcommand);
res = curl_easy_perform(curl);
if (res != CURLE_OK)
{
return (EXIT_FAILURE);
}
curl_easy_cleanup(curl);
}
curl_global_cleanup();
return (EXIT_SUCCESS);
}
Normally we’d use a makefile to build the code, but it in this case it’s such a small and simple sample we can build via:
$ gcc -Wall -O3 -o libcurl-sample libcurl-sample.c -I/usr/local/include -L/usr/local/lib/ -lcurl
Next, assume that we have a small test file that perhaps looks something like:
$ cat test-domains.txt www.google.com www.apple.com www.microsoft.com www.amazon.com [etc]
We can run the test by saying:
$ ./libcurl-sample < test-domains.txt > output.txt
4. Performance
This is a very simple single-threaded and untuned example. Nonetheless, how does it perform?
If we take 1,000 domains from https://raw.githubusercontent.com/opendns/public-domain-lists/master/opendns-top-domains.txt and time how long it takes to run those domains through DNSDB with this simple application, we see:
$ time ./sample < top-1000.txt > /dev/null real 1m17.653s (e.g., 77.653 seconds) user 0m22.417s sys 0m0.828s
That implies 1,000/77.653=12.87 queries/second even for just this very simple code. How long would it take to run a million queries with the simple code? Hypothetically, by extrapolation, it should take roughly 1,000 times the time it took to run 1,000 queries: 1,000*77.653/(60*60)=21.57 hours
Obviously, parallelization (e.g., using ten concurrent threads, or simply running multiple parallel jobs), could reduce that time by an order of magnitude. [Farsight generally asks that users limit their queries to no more than ten concurrent threads unless special arrangements have been made in advance.]
5. Conclusion
You’ve now learned how to write C code with libcurl that will let you run basic DNSDB queries using your DNSDB API credentials.
If you aren’t currently a Farsight DNSDB API customer, and want to lean more, please go here for information on how to purchase DNSDB API service.
Joe St Sauver, Ph.D. is a Scientist with Farsight Security, Inc.
