Farsight DNSDB API Version 2 Documentation
Introduction
DNSDB is a database that stores and indexes both the passive DNS data available via Farsight Security’s Security Information Exchange as well as the authoritative DNS data that various zone operators make available. DNSDB makes it easy to search for individual DNS RRsets and provides additional metadata for search results such as first seen and last seen timestamps as well as the DNS bailiwick associated with an RRset. DNSDB also has the ability to perform inverse or rdata searches.
This document describes how to make bulk, automated DNSDB queries via the RESTful API.
There are two versions of the API; the first, implicitly API version 1, is described at DNSDB APIv1. API version 2 is described on this page.
Audience
This document is intended for programmers who want to write applications that can interact with the RESTful DNSDB API using JSON and HTTP.
Summary of differences between APIv1 and APIv2
- Results that require processing DNSDB data are now encapsulated in Farsight’s Streaming API Framing protocol, described at DNSDB SAF Protocol and below.
- The “text” and “html” output formats have been removed. Only the “jsonl” output format is allowed now (though this should be specified in an HTTP ACCEPT header as ‘application/x-ndjson’).
- All existing paths in API URLs now start with /dnsdb/v2.
- The rate_limit request, which in APIv1 is at URL path /lookup/rate_limit, is moved to URL path /dnsdb/v2/rate_limit. It requires different content-types than previously — see HTTP Content-Types below.
- The rate structure returned by a rate_limit request is now on a single line as minimized JSON. In APIv1, it was returned with embedded newlines and spaces.
- rdata values are always returned as an array.
- Error code changes:
- If there are no results for an otherwise successful lookup request, instead of returning 404 “Not Found” error with a message body as in APIv1, an 200 OK status is returned without any SAF “obj” lines.
- New 401 “Unauthorized” error case.
- New 415 “Unsupported Media Type” error case.
- Some APIv1 error messages had a trailing period — these have been removed in APIv2. The actual error messages are not part of this specification.
- There is a new ping request at URL path /dnsdb/v2/ping.
Streaming API Framing
As just mentioned, DNSDB now returns streamed data encapsulated in DNSDB Streaming API Framing (SAF) Protocol. The SAF protocol is only applicable for HTTP responses that had an HTTP status of 200.
In APIv1, if the TCP connection broke, or there was some other error after the HTTP status code was returned, the DNSDB client could not tell that only a subset of the data was transferred. In API v2, if the returned result stream from a lookup does not end with a “cond” attribute with value “succeeded” or “limited” then the response must be treated as truncated.
In APIv1, an answer to query https://api.dnsdb.info/lookup/rrset/name/fsi.io/A?limit=1
had a form like:
{"count":271,"time_first":1578076118,"time_last":1580765117,"rrname":"fsi.io.",
"rrtype":"A","bailiwick":"fsi.io.","rdata":["104.244.14.108"]}
In APIv2 with SAF, an answer to the equivalent query
https://api.dnsdb.info/dnsdb/v2/lookup/rrset/name/fsi.io/A?limit=1 might be transmitted
as:
{"cond": "begin"}
{"obj":{"count":271,"time_first":1578076118,"time_last":1580765117,"rrname":"fsi.io.",
"rrtype":"A","bailiwick":"fsi.io.","rdata":["104.244.14.108"]}}
{"cond": "succeeded"}
If you have an existing DNSDB APIv1 processing pipeline, you may need to remove the first and last JSONL objects, remove any keep alive messages, and then unwrap the rest of JSONL objects to extract the “obj” values to feed the returned data into your pipeline.
Getting Started
To request a demonstration of DNSDB or to inquire about a trial API key please contact the DomainTools sales team.
Authentication
The DNSDB API is provided over an encrypted HTTPS transport over the Internet at the following URL:
Authentication is performed by providing an API key (a long string of hexadecimal digits or dashes) in a special HTTP request header, X-API-Key. For example, if your API key is d41d8cd98f00b204e9800998ecf8427e, then the following HTTP header should be added to the HTTP request:
X-API-Key: d41d8cd98f00b204e9800998ecf8427e
If the X-API-Key header is not present, or the provided API key is not valid, a 403 Forbidden response will be returned to the HTTP client.
Requests supported
The following kinds of requests are supported in DNSDB:
- lookup — this request is the primary query to search for individual DNS RRsets. This is the classic kind of request supported in DNSDB.
- summarize — this request returns a summary of RRsets that would be returned by a Lookup query. This gives you an estimate of result size. At-a-glance, it provides information on when a given domain name, IP address or other DNS asset was first-seen and last-seen by the global sensor network, as well as the total observation count.
- ping — this request is for end to end connectivity tests to the DNSDB API endpoint.
- rate_limit — this request returns quota information.
Of those requests, the results from lookup and summarize are streamed back encapsulated in the SAF protocol. Results from the other requests are a bare JSON object.
Name encoding
All IDN DNS names used with the DNSDB API must be encoded in Punycode. In particular, using non-basic ASCII characters may result in a 500 Internal Server error.
lookup requests
All DNSDB lookup requests are rooted in URL paths under the /dnsdb/v2/lookup hierarchy. There are two separate lookup methods, “rrset” and “rdata”, which are accessed via the URL paths /dnsdb/v2/lookup/rrset and /dnsdb/v2/lookup/rdata respectively.
rrset lookups
The “rrset” lookup queries DNSDB’s RRset index, which supports “forward” lookups based on the owner name of an RRset.
rrset lookups follow the URL path scheme:
/dnsdb/v2/lookup/rrset/TYPE/VALUE/RRTYPE/BAILIWICK
with placeholders TYPE, VALUE, RRTYPE, and BAILIWICK.
Both TYPE and VALUE are required; RRTYPE and BAILIWICK are optional. If RRTYPE is not specified, then the query will function as if “ANY” was specified.
The TYPE specifies how VALUE is interpreted. TYPE may be one of the following:
| Type | Description |
|---|---|
| name | The VALUE is a DNS owner name in presentation format or wildcards as described below. We sometimes call this just an rrset search. |
| raw | The VALUE is an even number of hexadecimal digits specifying a raw octet string. |
Wildcards are one of two forms: a left-hand (*.example.com) or right-hand
(www.example.*) wildcard domain name. An owner name with a leading asterisk and
label separator, (i.e., *.) will perform a wildcard search for any RRsets whose owner
names end with the given domain name. An owner name with a trailing label separator
and asterisk (i.e., .*) will perform a wildcard search for any RRsets whose owner names
start with the given label(s). Note that left-hand wildcard queries are somewhat more
expensive and slower than right-hand wildcard queries.
The OWNER_NAME and BAILIWICK are DNS names specified in DNS presentation format. TYPE is specified as a DNS RRtype mnemonic.
The RRtype ANY is modified somewhat from its usual meaning. A DNSDB lookup for RRtype ANY will match any RRtype except the DNSSEC-related RRtypes: DS, RRSIG, NSEC, DNSKEY, NSEC3, NSEC3PARAM, DLV, CDS, CDNSKEY, and TA. A new pseudo-mnemonic ANY-DNSSEC has been introduced that will return only those records matching the aforementioned RRtypes. RRtype ANY may be specified for RRTYPE in order to perform bailiwick filtering without also filtering on a particular RRtype.
The BAILIWICK may not be specified with raw queries.
As of July, 2022 DNSDB was changed to reduce the amount of junk wildcard domains in its database. We are gradually rolling out a change to replace multiple wildcarded DNS rrnames with a single rrname that starts with a _WILDCARD_. label. No other rrname labels contain uppercase letters, so records with this (all upper case) _WILDCARD_. were never in DNSDB before. Note that there are existing, real, domain names that contain a _wildcard_. label (all lower case).
The results of an rrset lookup return zero or more RRsets, along with the following metadata for each result:
| Item | Description |
|---|---|
| count | The number of times the RRset was observed via passive DNS replication. |
| bailiwick | The “bailiwick” of an RRset in DNSDB observed via passive DNS replication is the closest enclosing zone delegated to a nameserver which served the RRset. The “bailiwick” of an RRset in DNSDB observed in a zone file is simply the name of the zone containing the RRset. |
| first seen | A UTC timestamp with seconds granularity indicating the first time an RRset was seen in the given bailiwick via passive DNS replication. |
| last seen | A UTC timestamp with seconds granularity indicating the last time an RRset was seen in the given bailiwick via passive DNS replication. |
| first seen in zone file | A UTC timestamp with seconds granularity indicating the first time an RRset was seen in the given bailiwick via zone file import. |
| last seen in zone file | A UTC timestamp with seconds granularity indicating the last time an RRset was seen in the given bailiwick via zone file import. |
An rrset search result may be missing either the pair of (first seen, last seen) timestamps from passive DNS replication or from zone file import, but at least one pair of timestamps will always be present. This may change if a fundamentally new data source is introduced in the future, but there will always be at least one timestamp pair associated with an RRset.
Note: The DNSDB API returns RRSIG records with the expiration and inception times in the Unix epoch time format, seconds since 1 January 1970 00:00:00 UTC. RFC 4034 also allows presentation of these times in YYYYMMDDHHmmSS format, which may DNSSEC tools and presentation libraries use. It is always possible to distinguish between these two formats because the YYYYMMDDHHmmSS format will always be exactly 14 digits, while the decimal representation of a 32-bit unsigned integer can never be longer than 10 digits.
rdata lookups
The “rdata” lookup queries DNSDB’s Rdata index, which supports “inverse” lookups based on Rdata record values. In contrast to the rrset lookup method, rdata lookups return only individual resource records and not full resource record sets, and lack bailiwick metadata. An rrset lookup on the owner name reported via an rdata lookup must be performed to retrieve the full RRset and bailiwick.
rdata lookups follow the URL path scheme:
/dnsdb/v2/lookup/rdata/TYPE/VALUE/RRTYPE
with placeholders TYPE, VALUE, and RRTYPE.
Both TYPE and VALUE are required. Placeholder TYPE specifies how VALUE is interpreted. TYPE may be one of the following:
| Type | Description |
|---|---|
| name | The VALUE is a DNS domain name in presentation format, or a left-hand (“.example.com”) or right-hand (“www.example.”) wildcard domain name. Note that left-hand wildcard queries are somewhat more expensive than right-hand wildcard queries. |
| ip | The VALUE is one of an IPv4 or IPv6 single address, with a prefix length, or with an address range. If a prefix is provided, the delimiter between the network address and prefix length is a single comma (“,”) character rather than the usual slash (“/”) character to avoid clashing with the HTTP URI path name separator. |
| raw | The VALUE is an even number of hexadecimal digits specifying a raw octet string. |
If RRTYPE is not specified, then the query will function as if “ANY” was specified.
For “name” and “raw” rdata lookups, RRTYPE optionally filters the results by RRtype in the same manner as the rrset lookup.
For “ip” rdata lookups, the supported RRTYPE values are A, AAAA, and ANY, which can be used interchangeably: you can send an IPv4 or IPv6 address to either value and the data returned will be based on the IP address sent, not on the RRTYPE value. Any other RRTYPE value will return an “HTTP 400 Bad Request” response.
Some examples of “ip” rdata URLs (noting that colon needs to be expressed as %3A):
https://api.dnsdb.info/dnsdb/v2/lookup/rdata/ip/10.0.0.1/ANY
https://api.dnsdb.info/dnsdb/v2/lookup/rdata/ip/10.0.0.1-10.1.255.255
https://api.dnsdb.info/dnsdb/v2/lookup/rdata/ip/10.0.0.1,24/ANY
https://api.dnsdb.info/dnsdb/v2/lookup/rdata/ip/2620%3A11c%3Af008%3A%3A,126
https://api.dnsdb.info/dnsdb/v2/lookup/rdata/ip/2620%3A11c%3Af008%3A%3A1-2620%3A11c%3Af008%3A%3Aff
Lookup result formats
The DNSDB API only supports one Lookup result format, the “jsonl” format (though this should be specified in an HTTP ACCEPT header as ‘application/x-ndjson’).
The rrset JSON result format is a document containing one JSON-encoded result object per line. Each result object is an associative array with the following keys.
rrset results
| Key | Description |
|---|---|
| rrname | The owner name of the RRset in DNS presentation format. |
| rrtype | The resource record type of the RRset, either using the standard DNS type mnemonic, or an RFC 3597 generic type, i.e. the string TYPE immediately followed by the decimal RRtype number. |
| rdata | An array of one or more Rdata values. The Rdata values are converted to the standard presentation format based on the rrtype value. If the encoder lacks a type-specific presentation format for the RRset’s rrtype, then the RFC 3597 generic Rdata encoding will be used. |
| bailiwick | The “bailiwick” metadata value described in the section above. |
| count | The number of times the RRset was observed via passive DNS replication. |
| time_first, time_last | UNIX epoch timestamps with second granularity indicating the first and last times the RRset was observed via passive DNS replication. Will not be present if the RRset was only observed via zone file import. |
| zone_time_first, zone_time_last | UNIX epoch timestamps with second granularity indicating the first and last times the RRset was observed via zone file import. Will not be present if the RRset was only observed via passive DNS replication. |
rdata results
| Key | Description |
|---|---|
| rrname | The owner name of the resource record in DNS presentation format. |
| rrtype | The resource record type of the resource record, either using the standard DNS type mnemonic, or an RFC 3597 generic type, i.e. the string TYPE immediately followed by the decimal RRtype number. |
| rdata | The record data value. The Rdata value is converted to the standard presentation format based on the rrtype value. If the encoder lacks a type-specific presentation format for the resource record’s type, then the RFC 3597 generic Rdata encoding will be used. |
| count | The number of times the resource record was observed via passive DNS replication. |
| time_first, time_last | UNIX epoch timestamps with second granularity indicating the first and last times the resource record was observed via passive DNS replication. |
| zone_time_first, zone_time_last | UNIX epoch timestamps with second granularity indicating the first and last times the resource record was observed via zone file import. |
summarize requests
All DNSDB summarize requests are rooted in URL paths under the /dnsdb/v2/summarize hierarchy. Everything underneath /dnsdb/v2/summarize is the same as for /dnsdb/v2/lookup queries. For example, URL paths /dnsdb/v2/summarize/rrset and /dnsdb/v2/summarize/rdata respectively:
/dnsdb/v2/summarize/rrset/name/OWNER_NAME/RRTYPE/BAILIWICK
/dnsdb/v2/summarize/rdata/TYPE/VALUE/RRTYPE
Summarize result formats
The DNSDB API only supports one Summarize result format, the “jsonl” format (though this should be specified in an HTTP ACCEPT header as ‘application/x-ndjson’).
rrset and rdata results
| Key | Description |
|---|---|
| count | The number of times the RRset was observed via passive DNS replication. |
| num_results | The number of results (RRsets) that would be returned from a Lookup. |
| time_first, time_last | UNIX epoch timestamps with second granularity indicating the first and last times the RRset was observed via passive DNS replication. Only present if the RRsets were observed via passive DNS replication. |
| zone_time_first, zone_time_last | UNIX epoch timestamps with second granularity indicating the first and last times the RRset was observed via zone file import. Only present if the RRsets were observed via zone file import. |
An example JSON object is:
{"count":528,"num_results":4,"time_first":1557864746,"time_last":1560524861}
As encapsulated by the SAF, this would be expressed on the wire as:
{"cond":"begin"}
{"obj":{"count":528,"num_results":4,"time_first":1557864746,"time_last":1560524861}}
{"cond":"succeeded"}
If there are no RRsets found by the underlying query, the DNSDB API will return the JSON object:
{"count":0, "num_results":0}
As encapsulated by the SAF, this would be expressed on the wire as:
{"cond":"begin"}
{"obj":{"count":0,"num_results":0}}
{"cond":"succeeded"}
Optional query parameters
Lookup and summarize requests accept optional query parameters mostly in common, though each has an additional optional query parameter described below.
Time-fencing query parameters
The DNSDB records when a specific DNS record was first and last observed. You may filter results by time using the “time_first_before,” “time_first_after,” “time_last_before,” and “time_last_after” query parameters. These parameters expect a integer (Unix/Epoch time) with seconds granularity or a relative time in seconds (preceded by -).
Note that the DNSDB API server may have API key-dependent time-fencing restrictions that cannot be expanded with these parameters but can reduce the results (narrow the time ranges, or completely override them).
The following time-fencing optional parameters may be present in a query URL:
| Parameter | Description |
|---|---|
| time_first_before | provide results before the defined timestamp for when the DNS record was first observed. For example, the URL parameter “time_first_before=1420070400” will only provide matching DNS records that were first observed before (or older than) January 1, 2015. |
| time_first_after | provide results after the defined timestamp for when the DNS record was first observed. For example, the URL parameter “time_first_after=-31536000” will only provide results that were first observed within the last year. |
| time_last_before | provide results before the defined timestamp for when the DNS record was last observed. For example, the URL parameter “time_last_before=1356998400” will only provide results for DNS records that were last observed before 2013. |
| time_last_after | provide results after the defined timestamp for when the DNS record was last observed. For example, the URL parameter “time_last_after=-2678400” will only provide results that were last observed after 31 days ago. |
Combinations of the time parameters may be used to strictly provide or exclude results for specific time-ranges. For example, to only have results when the first observed date and the last observed date are both only in 2015, you can use “time_first_after=1420070399” combined with “time_last_before=1451606400”. As another time combination example, to get DNS records that were first observed before 2012 and last observed within the last month (recently-observed records which have not changed in a very long time), use “time_first_before=1325376000” and relative “time_last_after=-2678400”.
Other query parameters
The following other optional parameters may be present in a query URL:
| Parameter | Description |
|---|---|
| limit | Limit for the number of results returned via these lookup methods. There is a built-in limit to the number of results that are returned via these lookup methods. The default limit is set at 10,000. This limit can be raised or lowered by setting the “limit” query parameter. There is also a maximum number of results allowed; requesting a limit greater than the maximum will only return the maximum. See results_max below for information on that maximum. If “?limit=0” is used then DNSDB will return the maximum number of results allowed. Obviously, if there are less results for the query than the requested limit, only the actual amount can be returned. Example: appending “?limit=20000” to the URL path will set the response limit to 20,000 results. |
| swclient | Name of the API client software generating the DNSDB query. Limited to twenty alphanumeric characters. This may be logged by the DNSDB API server. Farsight support can help you debug a new API client using this and the following parameter. There is no default. Example: “?swclient=dnsdbq”. |
| version | Version number of the API client software generating the DNSDB query. Limited to twenty alphanumeric characters plus dash, underscore, and period. This may be logged by the DNSDB API server. There is no default. Example: “?version=1.1a”. |
| id | Client software specific identity of the user of the API client. Comprised of an alphanumeric string, a colon, and an alphanumeric string, limited to thirty characters. This may be logged by the DNSDB API server. There is no default. Example: “?id=dnsq:91e6245ad31387”. |
| aggr | Aggregated results group identical rrsets across all time periods and is the classic behavior from querying the DNSDB. This means you could get the total number of times an rrset has been observed, but not when it was observed. Unaggregated results ungroup identical rrsets, allowing you to see how the domain name was resolved in the DNS across the full-time range covered in DNSDB (subject to time fencing). This can give a more accurate impression of record request volume across time because it will reveal the distinct timestamps of records whose values are repeated. You can answer questions like, “Was a domain parked for a long time, mostly unused, until it was repurposed for serving malware or relaying spam, but then was abandoned again?” It allows you to see if a record was observed heavily in the last week vs. having been observed constantly for years. This is a boolean value. Use True, the default, for the aggregated results or False for unaggregated results. The value is case insensitive and can be abbreviated. Example: “?aggr=f”. |
| humantime | A boolean value that is True if time values (in time_first, time_last, zone_time_first, zone_time_last) should be returned in human readable (RFC3339 compliant) format or False if Unix-style time values in seconds since the epoch should be returned. False is the classic behavior from querying the DNSDB and is the default value for this option. The value is case insensitive and can be abbreviated. Example: “?humantime=t”. |
Additional query parameter for lookup
The following optional parameter may be present in a lookup form of query URL:
| Parameter | Description |
|---|---|
| offset | How many rows to offset (e.g. skip) in the results. This implements an incremental result transfer feature, allowing you to view more of the available results for a single query. The rows are offset prior to the limit parameter being applied, therefore offset allows seeing additional results past a limit that matches the maximum number of results. Note that DNSDB recalculates the results for each query and the order of results might not be preserved. Therefore, this capability is not a valid way to walk all results over multiple queries — some results might be missing and some might be duplicated. The actual offset that can be used is limited or for certain API keys, offset is not allowed — see the offset_max rate_limit key below. The offset value must be a positive integer. The default is 0, which means do not offset the rows. Example to return the second million results (assuming results_max is 100000): “?limit=100000&offset=1000000”. |
Additional query parameter for summarize
The following optional parameter may be present in a summarize form of query URL:
| Parameter | Description |
|---|---|
| max_count | max_count controls stopping when we reach that summary count. The resulting total count can exceed max_count as it will include the entire count from the last rrset examined. The default is to not constrain the count. Example: appending “?max_count=100” to the URL path will stop the summary when its total count reaches 100. |
ping requests
DNSDB ping requests have the URL path /dnsdb/v2/ping. This request is for end to end connectivity tests to the DNSDB API endpoint, letting you know that there are no firewall blockages. This request does not require an API key. It returns just a JSON object {“ping”: “ok”}.
Service limits and Quotas
The number of concurrent connections to a DNSDB API server may be limited. This limit is separate from the quota limit described below. If this limit is exceeded, the HTTP 503 “Service Unavailable” response code will be generated.
API keys have a primary quota, which limits the number of requests that can be made to the data-fetching endpoints (which are lookup, /dnsdb/v2/lookup, summarize, and /dnsdb/v2/summarize). There are three types of quotas: time-based, block-based, and unlimited. The DNSDB API server tracks the usage of the quotas. If the quota’s limit is exceeded (if applicable), the HTTP 429 “Too Many Requests” response code will be generated with message ‘Error: Rate limit exceeded’. If a block quota is expired, then a 401 “Unauthorized” response code will be generated with message ‘Error: Quota is expired’.
There may also be a secondary, burst rate, quota associated with an API key. The burst rate limits how many requests may be made in a short time window. For example, 5 requests in 360 seconds. The parameters for burst rate are burst size and burst window.
You may query the /dnsdb/v2/rate_limit endpoint to obtain a JSON map containing a top-level map “rate” that contains quota information as described below. Querying the /dnsdb/v2/rate_limit endpoint does not count against the quota limit.
Time-based quotas
Time-based quotas are usually applied on a daily basis and resets daily at 00:00 (midnight) in the UTC time zone. Time-based quotas can also be applied for arbitrary time-quantum, but this is less common.
Block-based quotas
For block quotas, there are a number of queries that are bought, with all, or a subset, of that number “split” off for a particular API key; e.g. a split is an allocation from the amount of queries purchased. Block quotas have a specific number of queries split with an expiration time (the expiration time is usually much longer than a day). For each API key, the DNSDB API server tracks the number of queries split, number used, and the expiration time. The number of queries bought is not visible to the DNSDB API server.
/dnsdb/v2/rate_limit response
The /dnsdb/v2/rate_limit endpoint returns a JSON map named rate containing some or all of the following:
| Key | Description |
|---|---|
| limit | The maximum number of API lookups that may be performed. This is the initial quota. |
| remaining | For time-based quotas: the remaining number of API lookups that may be performed until the reset time. For block-based quotas: the remaining number of API lookups in the block quota. |
| reset | For time-based quotas: UNIX epoch timestamp with second granularity indicating the next point in time when the quota limit will be reset. Usually this is at 00:00 (midnight) UTC. For block-based quotas: the value will be “n/a”. |
| expires | Only present for block-based quota: UNIX epoch timestamp with second granularity indicating when the quota will expire. |
| results_max | Returns the maximum number of results that can be returned by these lookup methods. This overrides a “limit” query parameter if provided. For example, if “?limit=20000” is appended to the URL path but results_max=1000 then only up to 1000 results will be returned. |
| offset_max | The maximum value that the offset query parameter can be. If it is higher then an HTTP 416 “Requested Range Not Satisfiable” response code will be returned with message “Error: offset value greater than maximum allowed.” If the value is “n/a” then the offset parameter is not allowed for this API key, and similar 416 error will be generated. |
| burst_size | The maximum number of API lookups that may be performed within this burst_window number of seconds. |
| burst_window | The number of seconds over which a burst of queries is measured. |
If burst_size and burst_window are not returned then there is no burst rate quota applicable.
For a time-based quota, the following is an example of a /dnsdb/v2/rate_limit response performed on June 10, 2015 that indicates that the API key’s quota limit will be reset at midnight UTC on June 11, 2015, and that 999 lookups are remaining out of a total quota of 1000 lookups:
{"rate": {"reset": 1433980800, "limit": 1000, "remaining": 999}}
For a block-based quota, the following is an example of a /dnsdb/v2/rate_limit response which indicates that the API key’s quota limit will expire at Mon Apr 15 19:28:34 2019, 592 lookups were used out of a total quota of 600 lookups, the burst size is 10 queries in 5 minutes, the maximum number of results that can be requested from a single query are 256, and offset will fail if higher than 3000000.
{"rate": {"reset": "n/a", "burst_size": 10, "expires": 1555370914, "burst_window": 300,
"offset_max": 3000000, "results_max": 256, "limit": 600, "remaining": 8}}
An unlimited quota API key has the corresponding /dnsdb/v2/rate_limit response:
{"rate": {"reset": "n/a", "limit": "unlimited", "remaining": "n/a"}}
X-RateLimit- Headers
Responses from the data-fetching endpoints contain information in the HTTP response headers that are a subset of that obtained from the /dnsdb/v2/rate_limit endpoint. These values are embedded as the X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, and for block-based quotas, X-RateLimit-Expires headers.
For a time-based quota, responses to lookups will contain response headers like:
X-RateLimit-Limit: 1000
X-RateLimit-Remaining: 999
X-RateLimit-Reset: 1433980800
For a block-based quota, responses to lookups will contain response headers like:
X-RateLimit-Limit: 600
X-RateLimit-Remaining: 8
X-RateLimit-Reset: n/a
X-RateLimit-Expires: 1555370914
For an unlimited API key, responses to lookups will contain response headers like:
X-RateLimit-Limit: unlimited
X-RateLimit-Remaining: n/a
X-RateLimit-Reset: n/a
HTTP Content-Types
The preferred HTTP Content-Type is application/x-ndjson. Multiple synonyms for that are supported, even though the actual data returned is identical for all of those types. The API will return a Content-Type with that type.
The following are the allowed content-types in the HTTP Accept: header:
- application/x-ndjson
- application/ldjson
- application/x-ldjson
- application/ndjson
- application/jsonl
- application/x-jsonl
Addtionally,
- */* is a wildcard. If the Accept header specifies the */* wildcard then the API will return application/x-ndjson.
If the request specifies multiple types in its Accept: header, then the first one that matches this supported list will be returned. The ;q= (q-factor weighting) feature is not supported.
If none of the content-types in the Accept: header matches this supported list, then the API will return a 415 Unsupported Media Type error with content-type text/plain.
For example:
- if a request has “Accept: application/jsonl” it will get back “Content-Type: application/jsonl“.
- If a request has “Accept: text/plain, application/x-ndjson” it will get back “Content-Type: application/x-ndjson“.
- If a request has “Accept: text/plain” it will get back an HTTP 415 error.
For a 500 Internal Server Error, it returns text/html (no matter what was requested).
In case of other error responses generated by the DNSDB API, it returns text/plain (no matter what was requested).
For errors generated by nginx, such as a 404 Not Found for an unknown request URL, nginx returns text/html.
HTTP Response codes
A table of HTTP response codes sent by the DNSDB API:
| Response code | Description |
|---|---|
| 200 OK | The request was successfully processed. For requests that stream back data, the server interpreted the request correctly and began to reply, but look at the final “cond” for success or failure of returning the streamed data. |
| 400 Bad Request | If the URL is formatted incorrectly, such as unrecognized lower-level components. |
| 401 Unauthorized | Your API key may not query this API version. (New in APIv2) |
| 401 Unauthorized | The API key is not authorized (usually indicates the block quota is expired). |
| 403 Forbidden | If the X-API-Key header is not present, the provided API key is not valid, or the Client IP address not authorized for this API key. |
| 404 Not Found | If the URL path had unrecognized or missing top-level components. Note: in APIv1 this error indicated that there were no results found. In APIv2, this is not an error. |
| 415 Unsupported Media Type | The Accept: header does not specify a supported content type for this query. |
| 416 Requested Range Not Satisfiable | If the offset value is greater than the maximum allowed or if an offset value was provided when not permitted. |
| 429 Too Many Requests | If you have exceeded your quota and no new requests will be accepted at this time. For time-based quotas: The API key’s daily quota limit is exceeded. The quota will automatically replenish, usually at the start of the next day. For block-based quotas: The block quota is exhausted. You may need to purchase a larger quota. For burst rate secondary quotas: There were too many queries within the burst window. The window will automatically reopen at its end. |
| 500 Internal Server Error | If there is an error processing the request. |
| 503 Service Unavailable | If the limit of number of concurrent connections is exceeded. |
| 504 Gateway Timeout | If the DNSDB Server timed-out before returning any results. Your query might be too complex, such as doing a wildcard search with restrictive time-fencing. You might simplify the query and retry it. |
Example clients
A full-featured example API client written in C is available from https://github.com/dnsdb/dnsdbq.
Example results
Examples are provided in the JSON result format. For each example, a description of the lookup is provided, then the URL used to retrieve the results and sample curl commands with the current results (at the time of the writing of this document). For brevity, the results are limited to a total of two records in the examples below. Note that we provide source code to convert the JSON format into the old APIv1 ad hoc text format inside the dnsdbq example client referenced above.
The examples assume the shell variable $DNSDB_API_KEY is set to your API key:
DNSDB_API_KEY=d41d8cd98f00b204e9800998ecf8427e...
1. Lookup all RRsets whose owner name is www.farsightsecurity.com, limited to 2 results
URL
/dnsdb/v2/lookup/rrset/name/www.farsightsecurity.com?limit=2
Sample query
curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \
"https://api.dnsdb.info/dnsdb/v2/lookup/rrset/name/www.farsightsecurity.com?limit=2"
The values in the SAF “obj” values are:
{"cond": "begin"}
{"obj":{"count":5059,"time_first":1380139330,"time_last":1427881899,
"rrname":"www.farsightsecurity.com.","rrtype":"A","bailiwick":"farsightsecurity.com.",
"rdata":["66.160.140.81"]}}
{"obj":{"count":17381,"time_first":1427893644,"time_last":1468329272,
"rrname":"www.farsightsecurity.com.","rrtype":"A","bailiwick":"farsightsecurity.com.",
"rdata":["104.244.13.104"]}}
{"cond": "limited", "msg": "Result limit reached"}
Note: The “Result limit reached” message does not imply that re-trying the query with a higher limit value will return more results, only that the limit was reached.
1a. Summarize all RRsets whose owner name is www.farsightsecurity.com, with limit and max_count
In this call, the summarize processing will limit itself to looking at two underlying results rows and it will stop at the row when the count value reaches max_count. Effectively, since the first row in Example 1 above is over that max_count, the summarize will only account for the first row in Example 1. Notice how the count and time_first and time_last are identical to that first row.
URL
/dnsdb/v2/summarize/rrset/name/www.farsightsecurity.com?limit=2&max_count=5000
Sample query
curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \
"https://api.dnsdb.info/dnsdb/v2/summarize/rrset/name/www.farsightsecurity.com\
?limit=2&max_count=5000"
{"cond":"succeeded","obj":{"count":1127,"num_results":2,"time_first":1557859313,
"time_last":1560537333}}
1b. Summarize all RRsets whose owner name is www.farsightsecurity.com, with only limit
In this call, the summarize processing will limit itself to looking at two underlying results rows. Notice how the count is the sum of the counts from the two rows in Example 1. And notice how the time_first is from the first row and the time_last is from the second row.
URL
/dnsdb/v2/summarize/rrset/name/www.farsightsecurity.com?limit=2
Sample query
curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \
"https://api.dnsdb.info/dnsdb/v2/summarize/rrset/name/www.farsightsecurity.com?limit=2"
{"cond":"succeeded","obj":{"count":1127,"num_results":2,"time_first":1557859313,
"time_last":1560537333}}
1c. Summarize all RRsets whose owner name is www.farsightsecurity.com, with only max_count
URL
/dnsdb/v2/summarize/rrset/name/www.farsightsecurity.com?max_count=50000
Sample query
curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \
"https://api.dnsdb.info/dnsdb/v2/summarize/rrset/name/www.farsightsecurity.com\
?max_count=50000"
{"cond":"succeeded","obj":{"count":1078,"num_results":2,"time_first":1573594176,
"time_last":1576187607}}
2. Lookup all RRsets whose owner name ends in farsightsecurity.com, of type NS, in the farsightsecurity.com zone
URL
/dnsdb/v2/lookup/rrset/name/*.farsightsecurity.com/ns/farsightsecurity.com
Sample query
curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \
"https://api.dnsdb.info/dnsdb/v2/lookup/rrset/name/*.farsightsecurity.com/ns/farsightsecurity.com"
{"cond": "begin"}
{"obj":{"count":51,"time_first":1372688083,"time_last":1374023864,
"rrname":"farsightsecurity.com.","rrtype":"NS","bailiwick":"farsightsecurity.com.",
"rdata":["ns.lah1.vix.com.","ns1.isc-sns.net.","ns2.isc-sns.com.",
"ns3.isc-sns.info."]}}
{"obj":{"count":495241,"time_first":1374096380,"time_last":1468324876,
"rrname":"farsightsecurity.com.","rrtype":"NS","bailiwick":"farsightsecurity.com.",
"rdata":["ns5.dnsmadeeasy.com.","ns6.dnsmadeeasy.com.","ns7.dnsmadeeasy.com."]}}
{"cond": "succeeded"}
3. Lookup all resource records whose Rdata values are the IPv4 address 104.244.13.104
URL
/dnsdb/v2/lookup/rdata/ip/104.244.13.104
Sample query
curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY"
"https://api.dnsdb.info/dnsdb/v2/lookup/rdata/ip/104.244.13.104"
{"cond": "begin"}
{"obj":{"count":24,"time_first":1433550785,"time_last":1468312116,
"rrname":"www.farsighsecurity.com.","rrtype":"A","rdata":"104.244.13.104"}}
{"obj":{"count":9429,"time_first":1427897872,"time_last":1468333042,
"rrname":"farsightsecurity.com.","rrtype":"A","rdata":"104.244.13.104"}}
{"cond": "succeeded"}
4. Lookup all resource records whose Rdata values are addresses in the 104.244.13.104/29 network prefix
This also shows some keep alive messages.
URL
/dnsdb/v2/lookup/rdata/ip/104.244.13.104,29
Sample query
curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \
"https://api.dnsdb.info/dnsdb/v2/lookup/rdata/ip/104.244.13.104,29"
{"cond": "begin"}
{}
{"obj":{"count":24,"time_first":1433550785,"time_last":1468312116,
"rrname":"www.farsighsecurity.com.","rrtype":"A","rdata":"104.244.13.104"}}
{"obj":{"count":9429,"time_first":1427897872,"time_last":1468333042,
"rrname":"farsightsecurity.com.","rrtype":"A","rdata":"104.244.13.104"}}
{}
{"cond": "succeeded"}
4a. Summarize all resource records whose Rdata values are addresses in the 104.244.13.104/29 network prefix
URL
/dnsdb/v2/summarize/rdata/ip/104.244.13.104,29
Sample query
curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \
"https://api.dnsdb.info/dnsdb/v2/summarize/rdata/ip/104.244.13.104,29"
{"cond":"succeeded","obj":{"count":528,"num_results":4,"time_first":1557864746,
"time_last":1560524861}}
5. Lookup all resource records whose Rdata values are the IPv6 address 2620:11c:f004::104
URL
/dnsdb/v2/lookup/rdata/ip/2620:11c:f004::104
Sample query
curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \
"https://api.dnsdb.info/dnsdb/v2/lookup/rdata/ip/2620:11c:f004::104"
{"cond": "begin"}
{"obj":{"count":14,"time_first":1433845806,"time_last":1467828872,
"rrname":"www.farsighsecurity.com.","rrtype":"AAAA","rdata":"2620:11c:f004::104"}}
{"obj":{"count":5307,"time_first":1427897876,"time_last":1468333042,
"rrname":"farsightsecurity.com.","rrtype":"AAAA","rdata":"2620:11c:f004::104"}}
{"cond": "succeeded"}
6. Lookup all resource records whose Rdata values are addresses in the 2620:11c:f000::/126 network prefix
URL
/dnsdb/v2/lookup/rdata/ip/2620:11c:f000::,126
Sample query
curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \
"https://api.dnsdb.info/dnsdb/v2/lookup/rdata/ip/2620:11c:f000::,126"
{"cond": "begin"}
{"obj":{"count":2,"time_first":1574082633,"time_last":1574082633,
"rrname":"gw.fmt1.fsi.io.","rrtype":"AAAA","rdata":["2620:11c:f000::1"]}}
{"obj":{"count":261,"time_first":1573589461,"time_last":1576188661,
"rrname":"r1.fmt1.fsi.io.","rrtype":"AAAA","rdata":["2620:11c:f000::2"]}}
{"obj":{"count":241,"time_first":1573611061,"time_last":1576188661,
"rrname":"r2.fmt1.fsi.io.","rrtype":"AAAA","rdata":["2620:11c:f000::3"]}}
{"cond": "succeeded"}
7. Lookup all domain names delegated to the nameserver ns5.dnsmadeeasy.com
URL
/dnsdb/v2/lookup/rdata/name/ns5.dnsmadeeasy.com
Sample query
curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \
"https://api.dnsdb.info/dnsdb/v2/lookup/rdata/name/ns5.dnsmadeeasy.com"
{"cond": "begin"}
{"obj":{"count":1078,"zone_time_first":1374250920,"zone_time_last":1468253883,
"rrname":"farsightsecurity.com.","rrtype":"NS","rdata":"ns5.dnsmadeeasy.com."}}
{"obj":{"count":706617,"time_first":1374096380,"time_last":1468334926,
"rrname":"farsightsecurity.com.","rrtype":"NS","rdata":"ns5.dnsmadeeasy.com."}}
{"cond": "succeeded"}
8. Lookup all domain names whose mail exchanges are the server hq.fsi.io
URL
/dnsdb/v2/lookup/rdata/name/hq.fsi.io
Sample query
curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \
"https://api.dnsdb.info/dnsdb/v2/lookup/rdata/name/hq.fsi.io"
{"cond": "begin"}
{"obj":{"count":45644,"time_first":1372706073,"time_last":1468330740,
"rrname":"fsi.io.","rrtype":"MX","rdata":"10 hq.fsi.io."}}
{"obj":{"count":19304,"time_first":1374098929,"time_last":1468333042,
"rrname":"farsightsecurity.com.","rrtype":"MX","rdata":"10 hq.fsi.io."}}
{"cond": "succeeded"}
9. Lookup wildcard search for RRsets whose owner name is farsightsecurity.com, rrtype is NS, bailiwick is farsightsecurity.com, last observed after July 11, 2016 with a limit of 100 results
URL
/dnsdb/v2/lookup/rrset/name/*.farsightsecurity.com/ns/farsightsecurity.com?limit=100\
&time_last_after=1468281600
Sample query
curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \
"https://api.dnsdb.info/dnsdb/v2/lookup/rrset/name/*.farsightsecurity.com\
/ns/farsightsecurity.com?limit=100&time_last_after=1468281600"
{"cond": "begin"}
{"obj":{"count":989291,"time_first":1374096380,"time_last":1499964330,
"rrname":"farsightsecurity.com.","rrtype":"NS","bailiwick":"farsightsecurity.com.",
"rdata":["ns5.dnsmadeeasy.com.","ns6.dnsmadeeasy.com.","ns7.dnsmadeeasy.com."]}}
{"cond": "limited", "msg": "Result limit reached"}
10. Lookup any DNSSEC records under farsightsecurity.com
URL
dnsdb/v2/lookup/rrset/name/*.farsightsecurity.com/ANY-DNSSEC?limit=2
Sample query
curl -k -H "Host: api.dnsdb.info" -H "Accept: application/x-ndjson" \
-H "X-API-Key: $DNSDB_API_KEY" \
"https://api.dnsdb.info/dnsdb/v2/lookup/rrset/name/*.farsightsecurity.com/ANY-DNSSEC?limit=2"
{"cond": "begin"}
{"obj":{"count":1696,"zone_time_first":1374250920,"zone_time_last":1521734545,
"rrname":"farsightsecurity.com.","rrtype":"DS","bailiwick":"com.",
"rdata":["60454 5 2 3672C35CFA8FF14C9C223B84277BD645C0AF54BAD5790375FE797161E4801479"]}}
{"obj":{"count":3,"zone_time_first":1374250920,"zone_time_last":1374423636,
"rrname":"farsightsecurity.com.","rrtype":"RRSIG","bailiwick":"com.",
"rdata":["DS 8 2 86400 1374774350 1374165350 8795 com.
cuOdo+2G0yJpBN5ba2zxiljSzgtTzminrVc3CrsNxQPqc5YVQX4eBWMB
+kpgSEXPT+DF2D9HwIsPpBDNdJekBpXIRW41Yl7IdZYHySqabn7hgt9M
mk5KNy9gqCOK/JLRs07LPAm3wvfyYer8e0/7VCTEjF9/DMbMGsLLH3xr kBA="]}}
{"cond": "limited", "msg": "Result limit reached"}
Sample query post-formatted with jq
curl -k -H "Host: api.dnsdb.info" -H "Accept: application/x-ndjson" \ -H "X-API-Key: $DNSDB_API_KEY" \
"https://api.dnsdb.info/dnsdb/v2/lookup/rrset/name/*.farsightsecurity.com/ANY-DNSSEC?limit=2" \
| jq .
{
"cond": "begin"
}
{
"count": 1696,
"zone_time_first": 1374250920,
"zone_time_last": 1521734545,
"rrname": "farsightsecurity.com.",
"rrtype": "DS",
"bailiwick": "com.",
"rdata": [
"60454 5 2 3672C35CFA8FF14C9C223B84277BD645C0AF54BAD5790375FE797161E4801479"
]
}
{
"count": 3,
"zone_time_first": 1374250920,
"zone_time_last": 1374423636,
"rrname": "farsightsecurity.com.",
"rrtype": "RRSIG",
"bailiwick": "com.",
"rdata": [
"DS 8 2 86400 1374774350 1374165350 8795 com.
cuOdo+2G0yJpBN5ba2zxiljSzgtTzminrVc3CrsNxQPqc5YVQX4eBWMB
+kpgSEXPT+DF2D9HwIsPpBDNdJekBpXIRW41Yl7IdZYHySqabn7hgt9M
mk5KNy9gqCOK/JLRs07LPAm3wvfyYer8e0/7VCTEjF9/DMbMGsLLH3xr kBA="
]
}
{
"cond": "limited",
"msg": "Result limit reached"
}
11. Lookup with no results
URL
/dnsdb/v2/lookup/rrset/name/this.name.does.not.exist
Sample query
curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \
"https://api.dnsdb.info/dnsdb/v2/lookup/rdata/name/this.name.does.not.exist"
HTTP 200 OK status with
{"cond": "begin"}
{"cond": "succeeded"}
In APIv1, the equivalent query
curl -H "Accept: application/json" "X-API-Key: $DNSDB_API_KEY" \
"https://api.dnsdb.info/lookup/rdata/name/this.name.does.not.exist"
results in an HTTP 404 Not Found status with message Error: no results found for query.
12. rrset/raw query for any records under fsi.io
URL
dnsdb/v2/lookup/rrset/raw/0366736902696f00
Sample query
curl -k -H "Host: api.dnsdb.info" -H "Accept: application/x-ndjson" \
-H "X-API-Key: \
$DNSDB_API_KEY" "https://api.dnsdb.info/dnsdb/v2/lookup/rrset/raw/0366736902696f00?limit=2"
{"cond": "begin"}
{"obj":{"count":10392,"time_first":1381265499,"time_last":1428418529,
"rrname":"fsi.io.","rrtype":"A","bailiwick":"fsi.io.","rdata":["66.160.140.76"]}}
{"obj":{"count":69435,"time_first":1428433465,"time_last":1538014110,
"rrname":"fsi.io.","rrtype":"A","bailiwick":"fsi.io.","rdata":["104.244.13.104"]}}
{"cond": "limited", "msg": "Result limit reached"}
13. rdata/raw query for any records under fsi.io
URL
dnsdb/v2/lookup/rdata/raw/0366736902696f00
Sample query
curl -k -H "Host: api.dnsdb.info" -H "Accept: application/x-ndjson" \
-H "X-API-Key: $DNSDB_API_KEY" \
"https://api.dnsdb.info/dnsdb/v2/lookup/rdata/raw/0366736902696f00?limit=2"
{"cond": "begin"}
{"obj":{"count":6,"time_first":1413228451,"time_last":1413228451,
"rrname":"local-data.fsi.io.","rrtype":"SOA",
"rdata":"fsi.io. hostmaster.fsi.io. 2014050101 7200 3600 604800 3600"}}
{"obj":{"count":25,"time_first":1412912798,"time_last":1412942807,"rrname":"dnstap.info.",
"rrtype":"SOA","rdata":"fsi.io. hostmaster.fsi.io. 2014052824 7200 3600 25920000 3600"}}
{"cond": "limited", "msg": "Result limit reached"}
14. rate_limit request
This call shows the rate_limit response for a time-based quota.
Sample query
curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \
"https://api.dnsdb.info/dnsdb/v2/rate_limit"
{"rate": {"reset": 1539129600, "limit": 1000, "remaining": 990}}
15. ping request
This call shows a successful ping:
Sample query
curl "https://api.dnsdb.info/dnsdb/v2/ping"
{"ping": "ok"}
Note that since an Accept: header was not specified on the curl command-line, curl defaults to sending “Accept: */*“.
16. ping request with unsupported content-type
This call shows a ping with an unsupported content-type:
Sample query
curl -H "Accept: text/plain" "https://api.dnsdb.info/dnsdb/v2/ping"
Error: The Accept: header does not specify a supported content type for this query
17. request with unrecognized lower-level components
This call shows a request with unrecognized lower-level components (the “/rrset” component is missing):
Sample query
curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \
"https://api.dnsdb.info/dnsdb/v2/lookup/name/fsi.io"
HTTP “400 Bad Request” status with body:
Error: unable to parse request
18. request with missing top-level components
This call shows a request with missing top-level components:
Sample query
curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \
"https://api.dnsdb.info/dnsdb/"
HTTP “404 Not Found” status with body:
404 Not Found
404 Not Found
nginx/1.10.3
