User Guides

DNSDB Flexible Search API Programmers Guide

Introduction

This page documents the Flex Search API extensions to DNSDB APIv2. DNSDB APIv2 has two components: an enhanced Standard Search capability and Flexible Search. Flexible search adds ways to search DNSDB by regular expressions and globs (aka wildcarding). The Farsight Flexible Search Reference Guide should be read first, before reading this API programmer’s guide.

In conjunction with the DNSDB APIv2 page, this page describes how to make DNSDB Flexible search queries via the RESTful API.

Audience

This document is intended for programmers who want to write applications that can interact with the Flex API flexible search extensions to the RESTful DNSDB APIv2 using JSON and HTTP.

High level features

Flexible Search and its Flex API:

Flex provides a flavor of “regular expressions” known as “extended regular expressions,” as used by the Unix egrep command. It supports most egrep-style regular expressions (except for capturing groups and backreferences), including simple string searches and partial label search. The particulars are known as “Farsight Compatible Regular Expressions” (FCRE) and are documented at FCRE Reference Guide.
Flex provides advanced wildcard search features, known in the Unix world as globs. They are documented at Glob Reference Guide.
Flex returns JSONL (newline delimited JSON) data, similar to what Standard Search returns, but the data are not the IETF Passive DNS – Common Output Format (COF) format.

URL encoding

When you’re building URIs, some characters can’t be used as normally written. These characters need to be written in their percent encoded form instead. Those special characters used in Flex regular expressions and globs patterns are:

special character	use in URLs
!	%21
“	%22
/	%2F
?	%3F
%	%25
[	%5B
\	%5C
]	%5D

See the Example URLs section below for some examples.

curl issue

In this document we show many curl examples. When using the Flex API with curl and you have use square brackets or curly brackets in the URL, you are supposed to encode them as mentioned above. curl usually works anyway without them encoded, but by default curl will interpret them itself. For example, if you ran a curl command with URL

.../dnsdb/v2/regex/rrnames/example[1-2]

it would actually fetch two separate URLs:

.../dnsdb/v2/regex/rrnames/example1

.../dnsdb/v2/regex/rrnames/example2

to stop this, ideally use URL encoding; second best, use the --globoff (or -g) parameter to curl. All our curl examples show this parameter in use, even when not strictly necessary.

Summary of differences between APIv2 and the Flex API

APIv1 and APIv2 only allow whole-label left-hand-side and right-hand-side wildcards, like *.fsi.io or www.fsi.*. Flexible search allows searches using regular expressions and globs (aka wildcarding) on rrnames and rdata. – DNSDB calls searches on the owner name (i.e. left-hand-side of a DNS query-response) an “rrset” query. Flexible search calls that an “rrnames” query, which is more consistent with DNS standards and reflects that the results from the search contain rrnames, not full RRsets. – The URL structure below /dnsdb/v2/ is completely different.
The returned data stream, while still encapsulated in Farsight’s Streaming API Framing protocol, are different JSON objects. – The Flex API does not provide count, time_first, and time_last in returned results.
In the Flex API, the aggr, humantime, and max_count query parameters are not supported; the other APIv2 parameters are supported. – The limit parameter behaves slightly differently. In the Flex API, limit applies to the number of unique RRnames returned and not the number of JSON response objects returned, as it does in APIv2. That is, in the Flex API, limit=1 might return 1 unique RRname, but multiple JSON response objects, one per RRType for that name.
There are new error messages within the SAF “msg” object. – The X-RateLimit- headers are not returned in Flex API responses. Use the Standard Search API rate_limit request to get your quota information.

Summary of commonalities between APIv2 and the Flex API

Both are encapsulated using the Farsight’s Streaming API Framing protocol.
Both use the same API keys.
Both share the same quotas.
Both use the same HTTP Content-Types.
Both return rrtype.
Both return the same HTTP Response codes.

Streaming API Framing

The Flex API returns streamed data encapsulated in Farsight’s Streaming API Framing protocol. The SAF protocol is only
applicable for HTTP responses that had an HTTP status of 200.

Here is an example Flex API query and response. Line breaks were added for readability, but the actual results were on one line:

https://api.dnsdb.info/dnsdb/v2/regex/rrnames/example\..*/ANY?limit=1

{"cond": "begin"}
{"obj":{"rrname":"example.s.ac.","rrtype":"A"}}
{"cond": "succeeded"}

Authentication

The Flex API, as part of the DNSDB API 2.0, is provided over an encrypted HTTPS transport over the Internet at https://api.dnsdb.info/.

Authentication is performed by providing an API key (a long string of hexadecimal digits or dashes) in a special HTTP request header, X-API-Key. For example, if your API key is d41d….8427e, then the following HTTP header should be added to the HTTP request:

X-API-Key: d41d....8427e

If the X-API-Key header is not present, or the provided API key is not valid, a 403 Forbidden response will be returned to the HTTP client.

Getting an API Key

To request a demonstration or a trial API key please contact the DomainTools sales team.

Query URL structure

The query URL structure is:

api.dnsdb.info/dnsdb/v2/SEARCH_METHOD/WHAT_TO_SEARCH/VALUE/RRTYPE?query-parameters

RRTYPE is optional, so the following is also valid:

api.dnsdb.info/dnsdb/v2/SEARCH_METHOD/WHAT_TO_SEARCH/VALUE?query-parameters

/dnsdb/v2 – The first two path components.

SEARCH_METHOD = Third-level path component:
- regex – regular expression search
- glob – full wildcarding
WHAT_TO_SEARCH = Fourth-level path component:
- rrnames – search in rrnames, supports “forward” searches based on the owner name
  of an RRset.
- rdata – search in rdata, supports “inverse” searches based on RData record values
VALUE = Fifth-level path component:
- For rrnames queries, “LHS” DNS owner name regular expression or glob
- For rdata queries, “RHS” normalized resource record value regular expression or
  glob
RRTYPE = Sixth-level path component – Optional:
- The official DNS RRType string value or TYPE# where # is the numeric value of the RRType.
  - If an undefined RRType (but not TYPE# form) is specified
    (i.e. not in the DNS RFCs), then an HTTP 400 (Bad Request)
    error will be returned with message “Error: RRTYPE has an
    unsupported value”.
- The RRtype ANY is modified somewhat from its usual meaning. A search for RRtype ANY will match any supported RRtype.
  - ANY is the default if RRTYPE is not specified.
- DNSSEC-related RRtypes are not permitted: CDS, CDNSKEY, DNSKEY, DS, DLV, NSEC, NSEC3, NSEC3PARAM, RRSIG, TA, nor our own ANY-DNSSEC.
  - If one of those DNSSEC RRTypes is specified, then an HTTP 400 (Bad Request)
    error will be returned with message “Error: RRTYPE has an unsupported
    DNSSEC value”.
- Only SOA, CNAME, HINFO, MX, NS, NAPTR, PTR, RP, SRV, SPF, or TXT (plus their
  TYPE# equivalents) are indexed for rdata queries. If another RRType is
  provided, then an HTTP 400 (Bad Request) error will be returned with message
  “Error: rdata searches are not allowed for the specified RRType”.
If any of the path component values are not recognized, then an HTTP 400 (Bad
Request) error will be returned with message “<X> has an unsupported value”.
The maximum length of the VALUE path component and the exclude
parameter value (which is described below) are 4096 characters each
before URL encoding.

Optional query parameters

Time-fencing query parameters

The DNSDB records when a specific DNS record was first and last observed. You may filter results by time using the “time_first_before,” “time_first_after,” “time_last_before,” and “time_last_after” query parameters. These parameters expect a integer (Unix/Epoch time) with seconds granularity or a relative time in negative seconds, as relative times must be in the past from now.

Note that the DNSDB API server may have API key-dependent time-fencing restrictions that cannot be expanded with these parameters but can reduce the results (narrow the time ranges, or completely override them).

The following time-fencing optional parameters may be present in a query URL:

Parameter	Description
time_first_before	provide results before the defined timestamp for when the DNS record was first observed. For example, the URL parameter “time_first_before=1420070400” will only provide matching DNS records that were first observed before (or older than) January 1, 2015.
time_first_after	provide results after the defined timestamp for when the DNS record was first observed. For example, the URL parameter “time_first_after=-31536000” will only provide results that were first observed within the last year. (365 days * 24 hours/day * 60 minutes/hr * 60 minutes/sec = 31536000)
time_last_before	provide results before the defined timestamp for when the DNS record was last observed. For example, the URL parameter “time_last_before=1356998400” will only provide results for DNS records that were last observed before 2013.
time_last_after	provide results after the defined timestamp for when the DNS record was last observed. For example, the URL parameter “time_last_after=-2678400” will only provide results that were last observed after 31 days ago.

Combinations of the time parameters may be used to strictly provide or exclude results for
specific time-ranges. For example, to only have results when the first observed date and
the last observed date are both only in 2015, you can use “time_first_after=1420070399”
combined with “time_last_before=1451606400”. As another time combination example, to get
DNS records that were first observed before 2012 and last observed within the last month
(recently-observed records which have not changed in a very long time), use
“time_first_before=1325376000” and relative “time_last_after=-2678400”.

Other query parameters

The following other optional parameters may be present in a query URL:

Parameter	Description
exclude	The “exclude” parameter is used to exclude (i.e. filter-out) results that match it. It is described below this table.
limit	Limit for the number of unique rrnames or rdata value results returned via these search methods. The default limit is set at 10,000. This limit can be raised or lowered by setting the “limit” query parameter. There is also a maximum number of results allowed; requesting a limit greater than the maximum will only return the maximum. See results_max in a rate_limit response for that maximum. If “?limit=0” is used then DNSDB will return the maximum number of results allowed. Obviously, if there are less results for the query than the requested limit, only the actual amount can be returned. Example: appending “?limit=20000” to the URL path will set the response limit to 20,000 results.
offset	How many rows to offset (e.g. skip) in the results. This implements an incremental result transfer feature, allowing you to view more of the available results for a single query. The rows are offset prior to the limit parameter being applied, therefore offset allows seeing additional results past a limit that matches the maximum number of results. Note that DNSDB recalculates the results for each query and the order of results might not be preserved. Therefore, this capability is not a valid way to walk all results over multiple queries — some results might be missing and some might be duplicated. The actual offset that can be used is limited or for certain API keys, offset is not allowed — see the Farsight DNSDB API Version 2 section on offset_max. The offset value must be a positive integer. The default is 0, which means do not offset the rows. Example to return the second million results (assuming results_max is 100000): “?limit=100000&offset=1000000”.
id	Client software specific identity of the user of the API client. Comprised of an alphanumeric string, a colon, and an alphanumeric string, limited to thirty characters. This may be logged by the DNSDB API server. There is no default. Example: “?id=dnsq:91e6245ad31387”.
swclient	Name of the API client software generating the DNSDB query. Limited to twenty alphanumeric characters. This may be logged by the DNSDB API server. Farsight support can help you debug a new API client using this and the following parameter. There is no default. Example: “?swclient=dnsdbflex”.
version	Version number of the API client software generating the query. Limited to twenty alphanumeric characters plus dash, underscore, and period. This may be logged by the DNSDB API server. There is no default. Example: “?version=1.1a”.

The exclude parameter

As mentioned above, the exclude parameter is used to exclude (i.e. filter-out) results that match it. Conceptually, this is like the shell pipeline: egrep $VALUE <DNSDB | egrep -v $exclude. Its value is a regular expression or glob, depending upon the search_method.

Here is an example with two rrnames searches, first without exclude and then excluding rrnames that end in .com., .site., .bid., .net, or .io.

$ curl -g -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \
'https://api.dnsdb.info/dnsdb/v2/regex/rrnames/^fsi.io/A?limit=3'
{"cond":"begin"}
{"obj":{"rrname": "fsi.io.", "rrtype": "A"}}
{"obj":{"rrname": "fsi.ioc.abc.cimpress.io.", "rrtype": "A"}}
{"obj":{"rrname": "fsi.io.ua.", "rrtype": "A"}}
{"cond":"succeeded"}

$ curl -g -H “Accept: application/x-ndjson” -H “X-API-Key: $DNSDB_API_KEY” \
‘https://api.dnsdb.info/dnsdb/v2/regex/rrnames/^fsi.io/A?exclude=.(com|site|bid|net|io).$&l\
imit=3′
{“cond”:”begin”}
{“obj”:{“rrname”: “fsi.io.ua.”, “rrtype”: “A”}}
{“cond”:”succeeded”}

Example URLs

Here are some example URLs showing a variety of URL components and parameters. We show each URL in its internal form and the form sent on the wire with URL encoding:

https://api.dnsdb.info/dnsdb/v2/regex/rrnames/example\..*/ANY?limit=1

https://api.dnsdb.info/dnsdb/v2/regex/rrnames/example%5C..%2A/ANY?limit=1


https://api.dnsdb.info/dnsdb/v2/glob/rrnames/example.*

https://api.dnsdb.info/dnsdb/v2/glob/rrnames/example.%2A


https://api.dnsdb.info/dnsdb/v2/glob/rdata/example.*/TXT?limit=1\
    &time_last_after=-2678400&exclude=*.com.&swclient=dnsdbflex

https://api.dnsdb.info/dnsdb/v2/glob/rdata/example.%2A/TXT?limit=1\
    &time_last_after=-2678400&exclude=%2A.com.&swclient=dnsdbflex

Result formats

As an extension to DNSDB APIv2, the Flex API only supports one underlying result format, the “jsonl” format. “jsonl” has multiple equivalent names for it. It should be specified in an HTTP ACCEPT header as application/x-ndjson. See the HTTP Content-Types section below.

Pedantic note: all of our JSON / json outputs are really JSONL (newline delimited JSON). We use the terms JSON or json throughout this document unless that would cause confusion.

If there are no results to the query, then the HTTP status will be 200 with the following SAF objects returned:

{"cond":"begin"}
{"cond":"succeeded","msg":"No results found"}

This is different than the DNSDB APIv1.

If there are results to the query, each result object is an associative array with the following keys for the terse mode (which is the only currently supported mode). See the Technical Reference Guide representation of rrnames and rdata section for details.

rrnames results

Key	Description
rrname	The owner name of the RRset in DNS presentation format.
rrtype	The resource record type of the RRset, either using the standard DNS type mnemonic, or an RFC 3597 generic type, i.e. the string TYPE immediately followed by the decimal RRtype number.

rdata results

Key	Description
rdata	The record data value.
rrtype	The resource record type of the resource record, either using the standard DNS type mnemonic, or an RFC 3597 generic type, i.e. the string TYPE immediately followed by the decimal RRtype number.
raw_rdata	The record data value as pairs of hex digits specifying a raw octet string. This value is used for pivoting from flexible search into standard search to get more details on rdata.

HTTP Content-Types

The preferred HTTP Content-Type is application/x-ndjson. Multiple synonyms for that
are supported, even though the actual data returned is identical for all of those types.
The API will return a Content-Type with that type.

The following are the allowed content-types in the HTTP Accept: header:

application/x-ndjson
application/ldjson
application/x-ldjson
application/ndjson
application/jsonl
application/x-jsonl

Addtionally,

*/* is a wildcard. If the Accept header specifies the */* wildcard then the API
will return application/x-ndjson.

If the request specifies multiple types in its Accept: header, then the first one that matches this supported list will be returned. The ;q= (q-factor weighting) feature is not supported.

If none of the content-types in the Accept: header matches this supported list, then the API will return a 415 Unsupported Media Type error with content-type text/plain.

For example:

If a request has “Accept: application/jsonl” it will get back “Content-Type: application/jsonl“.
If a request has “Accept: text/plain, application/x-ndjson” it will get back
“Content-Type: application/x-ndjson“.
If a request has “Accept: text/plain” it will get back an HTTP 415 error.

For a 500 Internal Server Error, it returns text/html (no matter what was requested).

In case of other error responses generated by the DNSDB API, it returns text/plain (no matter what was requested).

For errors generated by nginx, such as a 404 Not Found for an unknown request URL, nginx returns text/html.

HTTP Response codes

The following is a table of HTTP response codes sent by the DNSDB Flex API. A single response code may be used with more than one message.

Response code	Description
200 OK	The request was successfully received, but errors may still be detected as it’s further processed. Look at the final “cond” for success or failure of the request. See below for more information.
400 Bad Request	The URL is formatted incorrectly, such as unrecognized lower-level components. See below for more information.
401 Unauthorized	Your API key may not query this API version. Or the API key is not authorized (usually indicates the block quota is expired).
403 Forbidden	The X-API-Key header is not present, the provided API key is not valid, or the Client IP address not authorized for this API key.
404 Not Found	The URL path had unrecognized or missing top-level components. This may also be returned for an unknown SEARCH_METHOD.
415 Unsupported Media Type	The Accept: header does not specify a supported content type for this query.
416 Requested Range Not Satisfiable	The offset value is greater than the maximum allowed or an offset value was provided when not permitted.
429 Too Many Requests	You have exceeded your quota and no new requests will be accepted at this time. For time-based quotas: The API key’s daily quota limit is exceeded. The quota will automatically replenish, usually at the start of the next day. For block-based quotas: The block quota is exhausted. You may need to purchase a larger quota. For burst rate secondary quotas: There were too many queries within the burst window. The window will automatically reopen at its end. Note: Burst rate quotas are not applicable currently for the Flex API.
500 Internal Server Error	There is an internal error processing the request.
503 Service Unavailable	The limit of number of concurrent connections is exceeded.

Errors after a HTTP 200 (OK) status

Errors that occur after the query has started procesing, and an HTTP 200 OK was returned, will be reported by a SAF "cond":"failed" result in its "msg" value. For example:

$ curl -g -k -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \
  'https://api-dev2.dnsdb.info/dnsdb/v2/regex/rrnames/$^?limit=2'
...
< HTTP/1.1 200 OK
...
{"cond":"begin"}
{"cond":"failed","msg":"Regex syntax error: Invalid characters after '$' at 1"}

See the Flexible Search Technical Reference Guide section for descriptions and examples of these errors.

400 Bad Request errors

The response body after the HTTP 400 Bad Request will contain the following messages with a Content-Type: text/plain:

Message
Error: exception occurred– If the URL is very mangled or an internal bug, please email
Error: unable to parse request– Query parameters have illegal values
Error: SEARCH_METHOD has an unsupported value
Error: WHAT_TO_SEARCH has an unsupported value
Error: RRTYPE has an unsupported value
Error: RRTYPE has an unsupported DNSSEC value
Error: REQUEST_URI is missing path segment
Error: REQUEST_URI has too many path segments
Error: REQUEST_URI not available from environment– An internal bug, please email
Error: REQUEST_URI does not start with /dnsdb/v2/– An internal bug, please email

An example showing an illegal limit value (it should be a non-negative integer):

$ curl -g -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \
    'https://api.dnsdb.info/dnsdb/v2/glob/rrnames/*.google.*/ANY?limit=ILLEGAL'  -v
< HTTP/1.1 400 Bad Request
...
< Content-Type: text/plain
Error: unable to parse request

An example showing an unsupported WHAT_TO_SEARCH value:

$ dnsdbflex --regex 'test-search' -t badtype
dnsdbflex: warning: libcurl 400 [https://api.dnsdb.info/dnsdb/v2/regex/rrnames/
    test-search/badtype?swclient=dnsdbflex&version=1.0.0]
dnsdbflex: warning: libcurl: [Error: RRTYPE has an unsupported value]
Query status: ERROR (Error: RRTYPE has an unsupported value)

dnsdbflex can be invoked with parameters that will trigger some of those 400 Bad Request error messages. For example:

$ dnsdbflex --regex &#039;test-search&#039; -t badtype
dnsdbflex: warning: libcurl 400 [https://api.dnsdb.info/dnsdb/v2/regex/rrnames/
 test-search/badtype?swclient=dnsdbflex&amp;version=1.0.0]
dnsdbflex: warning: libcurl: [Error: RRTYPE has an unsupported value]
Query status: ERROR (Error: RRTYPE has an unsupported value)

Service limits and Quotas

The number of concurrent connections to a DNSDB API server may be limited. This limit is separate from the quota limit described below. If this limit is exceeded, the HTTP 503 “Service Unavailable” response code will be generated.

API keys have a primary quota, which limits the number of requests that can be made to the data-fetching endpoints. There are three types of quotas: time-based, block-based, and unlimited. The DNSDB API server tracks the usage of the quotas. If the quota’s limit is exceeded (if applicable), the HTTP 429 “Too Many Requests” response code will be generated with message ‘Error: Rate limit exceeded’. If a block quota is expired, then a 401 “Unauthorized” response code will be generated with message ‘Error: Quota is expired’.

There may also be a secondary, burst rate, quota associated with an API key. The burst rate limits how many requests may be made in a short time window. For example, 5 requests in 360 seconds. The parameters for burst rate are burst size and burst window. Note: Burst rate quotas are not applicable currently for the Flex API.

You may query the /dnsdb/v2/rate_limit endpoint to obtain a JSON map containing a top-level map “rate” that contains quota information. See the APIv2 Rate_limit response section.

Example client

dnsdbflex, a full-featured example API client written in C, is available from https://github.com/farsightsec/dnsdbflex
Its manual page.

Example results

For each example, the URL used to retrieve the results and sample curl commands with the current results (at the time of the writing of this document). For brevity, the results are limited to a total of two records in the examples below.

SAF format compared to dnsdbflex output format

For many examples, the equivalent dnsdbflex invocation and outputs are provided. Note that the json output from dnsdbflex is the “unwrapped” form of the SAF format.

For example, the following SAF:

{"cond":"begin"}
{"obj":{"rrname": "fsi.iosdjf.cn.", "rrtype": "CNAME"}}
{"obj":{"rrname": "fsi.iosdnb.cn.", "rrtype": "CNAME"}}
{"cond":"succeeded"}

is output from dnsdbflex when using its default json (-j) format as:

{"rrname": "fsi.iosdjf.cn.", "rrtype": "CNAME"}
{"rrname": "fsi.iosdnb.cn.", "rrtype": "CNAME"}

and is output from dnsdbflex in -F batch mode output format as:

rrset/name/fsi.iosdjf.cn./CNAME
rrset/name/fsi.iosdnb.cn./CNAME

Setting DNSDB_API_KEY

The examples assume the shell variable DNSDB_API_KEY is set to your API key:

DNSDB_API_KEY=0123456890abcd...

1. Search for all rrnames whose owner name matches regular expression ^fsi\.io, limited to 2 rrnames. Note: This is not right anchored

URL

/dnsdb/v2/regex/rrnames/^fsi\.io/ANY?limit=2

Sample query

$ curl -g -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \
 'https://api.dnsdb.info/dnsdb/v2/regex/rrnames/^fsi\.io/ANY?limit=2'

{"cond":"begin"}
{"obj":{"rrname": "fsi.iosdjf.cn.", "rrtype": "CNAME"}}
{"obj":{"rrname": "fsi.iosdnb.cn.", "rrtype": "CNAME"}}
{"cond":"succeeded"}

2. Search for all rrnames whose owner name matches regular expression ^fsi\.io$, limited to 2 rrnames. Note: This is right anchored

This shows a problem where we forgot to add a regex expression that matches a dot before the right anchor.

URL

/dnsdb/v2/regex/rrnames/^fsi\.io$/ANY?limit=2

Sample query

$ curl -g -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \
 'https://api.dnsdb.info/dnsdb/v2/regex/rrnames/^fsi\.io$/ANY?limit=2'

{"cond":"begin"}
{"cond":"succeeded","msg":"No results found"}

To fix this query, try one of the following

/dnsdb/v2/regex/rrnames/^fsi\.io.$/ANY?limit=2
/dnsdb/v2/regex/rrnames/^fsi\.io\.$/ANY?limit=2

3. Search for all rdata which matches regular expression ^fsi\.io, limited to 2 results

URL

/dnsdb/v2/regex/rdata/^fsi\.io/ANY?limit=2

Sample query

$ curl -g -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \
 'https://api.dnsdb.info/dnsdb/v2/regex/rdata/^fsi\.io/ANY?limit=2'

{"cond":"begin"}
{}
{"obj":{"rdata": "fsi.io. hostmaster.fsi.io.", "rrtype": "SOA",
 "raw_rdata": "0366736902696F000A686F73746D61737465720366736902696F00"}}
{"obj":{"rdata": "fsi.ioza15pw68tk.k0l33r.info.", "rrtype": "CNAME",
 "raw_rdata": "036673690C696F7A61313570773638746B066B306C33337204696E666F00"}}
{"cond":"succeeded"}

That output includes one keep alive message.

dnsdbflex equivalent

$ dnsdbflex --regex '^fsi\.io' -l 2 -s rdata

{"rdata":"fsi.io. hostmaster.fsi.io.","rrtype":"SOA"
 "raw_rdata":"0366736902696F000A686F73746D61737465720366736902696F00"}
{"rdata": "fsi.ioza15pw68tk.k0l33r.info.", "rrtype": "CNAME",
 "raw_rdata": "036673690C696F7A61313570773638746B066B306C33337204696E666F00"}

4. Use dnsdbflex to search all rrnames matching a complex range glob

This shows how some of the special characters in the glob are quoted by dnsdbflex.

$ dnsdbflex --glob '[!a-vx-z]sillyputty.info.'

which issues URL

/dnsdb/v2/glob/rrnames/a-vx-zsillyputty.info.?swclient=dnsdbflex&version=1.0.0&limit=1

which gets an API response:

{"cond":"begin"}
{"rrname": "wsillyputty.info.", "rrtype": "A"}
{"rrname": "wsillyputty.info.", "rrtype": "NS"}
{"rrname": "wsillyputty.info.", "rrtype": "SOA"}
{"rrname": "wsillyputty.info.", "rrtype": "MX"}
{"rrname": "wsillyputty.info.", "rrtype": "TXT"}
{"rrname": "wsillyputty.info.", "rrtype": "AAAA"}
{"cond":"succeeded"}

Butdnsdbflex only outputs:

{"rrname": "wsillyputty.info.", "rrtype": "A"}
{"rrname": "wsillyputty.info.", "rrtype": "NS"}
{"rrname": "wsillyputty.info.", "rrtype": "SOA"}
{"rrname": "wsillyputty.info.", "rrtype": "MX"}
{"rrname": "wsillyputty.info.", "rrtype": "TXT"}
{"rrname": "wsillyputty.info.", "rrtype": "AAAA"}

Note here that even though the “limit=1”, we got back 6 results — the reason is that there is only one unique rrname in those results.

5. Request with missing lower-level components

This call shows a request with a missing lower-level component (the “/rrnames” component is missing):

Sample query

$ curl -g -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \
 'https://api.dnsdb.info/dnsdb/v2/regex/^fsi\.io/ANY?limit=2' -v

HTTP “400 Bad Request” status with body:

Error: WHAT_TO_SEARCH has an unsupported value

6. Request with missing top-level components

This call shows a request with missing top-level components:

Sample query

$ curl -g -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \
 'https://api.dnsdb.info/dnsdb/'

HTTP “404 Not Found” status with body:


404 Not Found

404 Not Found
nginx/1.10.3

7. Search for possible phishing sites for a popular bank brand, with IPv6 addresses, limited to 2 rrnames

URL

dnsdb/v2/regex/rrnames/(www|web)\.wellsfargo\..*/AAAA?limit=2

Sample query

$ curl -g -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \
 'https://api.dnsdb.info/dnsdb/v2/regex/rrnames/(www|web)\.wellsfargo\..*/AAAA?limit=2'

{"cond":"begin"}
{"obj":{"rrname": "www.wellsfargo.cm.", "rrtype": "AAAA"}}
{"obj":{"rrname": "www.wellsfargo.lima-city.de.", "rrtype": "AAAA"}}
{"cond":"succeeded"}

8. Search for rdata that encodes 20 or more Punycode labels and pretty-print the JSON

The regular expression (xn-.*\.){20,99} matches twenty to 99 occurrences of xn- followed by anything and a dot. That’s a domain name with at least twenty punycode encoded labels.

URL

/dnsdb/v2/regex/rdata/xn-..2099

Sample query

$ dnsdbflex --regex '(xn-.*\.){20,99}' -s rdata | jq .
{
 "rdata": "xn--mgbta3g.xn--mgbnd.xn--mgbjk8g.xn--hhbac90btf.xn--ugbe1e7uue.xn--ckb3e.xn--
 mgbatf8ic.xn--ngbr0a94c.xn--pgbr9cf.xn--ugbe.xn--khb7q.xn--mgb7d.xn--
 ugbf4df.xn--ikb2d.xn--hhbac90btf.xn--mgbz3c.xn--ugbf4df.xn--ikb2d.xn--
 ugbe.xn--mgba3a4f16a.ir.",
 "rrtype": "CNAME",
 "raw_rdata": "0B786E2D2D6D67627461336709786E2D2D6D67626E640B786E2D2D6D67626A6B386
 70E786E2D2D686862616339306274660E786E2D2D7567626531653775756509786E2D2D636B623365
 0D786E2D2D6D67626174663869630D786E2D2D6E67627230613934630B786E2D2D706762723963660
 8786E2D2D7567626509786E2D2D6B6862377109786E2D2D6D676237640B786E2D2D75676266346466
 09786E2D2D696B6232640E786E2D2D686862616339306274660A786E2D2D6D67627A33630B786E2D2
 D7567626634646609786E2D2D696B62326408786E2D2D756762650F786E2D2D6D6762613361346631
 366102697200"
}

A website like https://www.punycoder.com/ can be used to convert that punycode string to its international character representation.

References

What’s A Regular Expression?
What is Globbing?
For more information about DNSDB Time Fencing please see Joe St Sauver’s Farsight’s DNSDB Time Fencing: A Post-Attack “Time Machine”
Latest draft of IETF Passive DNS – Common Output Format (COF) format

DNSDB Flexible Search API Programmers Guide

Share this entry

Introduction

Audience

High level features

URL encoding

curl issue

Summary of differences between APIv2 and the Flex API

Summary of commonalities between APIv2 and the Flex API

Streaming API Framing

Authentication

Getting an API Key

Query URL structure

Optional query parameters

Time-fencing query parameters

Other query parameters

The exclude parameter

Example URLs

Result formats

rrnames results

rdata results

HTTP Content-Types

HTTP Response codes

Errors after a HTTP 200 (OK) status

400 Bad Request errors

Service limits and Quotas

Example client

Example results

SAF format compared to dnsdbflex output format

Setting DNSDB_API_KEY

1. Search for all rrnames whose owner name matches regular expression ^fsi\.io, limited to 2 rrnames. Note: This is not right anchored

2. Search for all rrnames whose owner name matches regular expression ^fsi\.io$, limited to 2 rrnames. Note: This is right anchored

3. Search for all rdata which matches regular expression ^fsi\.io, limited to 2 results

4. Use dnsdbflex to search all rrnames matching a complex range glob

5. Request with missing lower-level components

6. Request with missing top-level components

7. Search for possible phishing sites for a popular bank brand, with IPv6 addresses, limited to 2 rrnames

8. Search for rdata that encodes 20 or more Punycode labels and pretty-print the JSON

References

Sign up for our newsletter

Related Content

The AXA User Guide

The AXA Technical Overview

The AXA API Reference