DNSDB API Technical Datasheet
DNSDB
The DNSDB® is built from Farsight Security’s (now a part of DomainTools) Passive DNS data. In addition to the most recent information, DNSDB contains historical data going back to 2010. Farsight stores and indexes two types of data:
- Observed DNS data, as recorded by Farsight’s network of globally distributed sensors, and
- DNS records gleaned from TLD zone files.
DNSDB makes it easy to find related domain names and IP addresses: assuming you have an initial domain name or IP address as a starting point. DNSDB can answer questions, such as:
- Where did this domain name point to in the past?
- What domain names are hosted on a given IP address?
- What domain names use a given name server?
- What fully qualified domain names exist below a delegation point?
What is Passive DNS?
“Passive DNS” or “Passive DNS replication” is a technique invented by Florian Weimer in 2004 to opportunistically reconstruct a partial view of the data available in the global Domain Name System into a central database where it can be indexed and queried. During the initial collection stage, packets between DNS resolvers and authoritative DNS servers are collected together at a central processing point. There are many steps required to process passive DNS data into a cohesive DNS database that can be queried.
Passive DNS uses observed cache miss traffic collected from above recursive resolvers to build a database detailing relationships between domain names, IP addresses, and nameservers. That historical database can then be queried to get a report of:
- Domains that have been seen associated with a particular IP or IP range
- IPs that have been seen associated with a particular domain name
- Domain names that are known to be using a particular authoritative nameserver, etc.
- The date and time range associated with associations and changes
One example of the value of using passive DNS can be seen when you
compare what passive DNS finds vs what you may get when you just request
a PTR record for that same IP address. For instance, let’s request
the PTR record for 128.223.142.89:
$ dig -x 128.223.142.89 +short
www.uoregon.edu.
This PTR makes it appear as if 128.223.142.89 is home to
www.uoregon.edu, and at one time perhaps it was. However, at the
time this example was prepared, www.uoregon.edu was actually at:
$ dig www.uoregon.edu +short
drupal-cluster5.uoregon.edu.
128.223.142.244
If we check dnsdb and ask to just see passive DNS records for
128.223.142.89 from the last 6 months (24 weeks), we can see more
current results that look like:
$ dnsdb_query.py -i 128.223.142.89 --after=24w | sort
cfc.uoregon.edu. IN A 128.223.142.89
culjp.org. IN A 128.223.142.89
oregon-ix.com. IN A 128.223.142.89
oregon-ix.net. IN A 128.223.142.89
oregonix.net. IN A 128.223.142.89
oregonix.org. IN A 128.223.142.89
virt-www.uoregon.edu. IN A 128.223.142.89
www.culjp.org. IN A 128.223.142.89
The equivalent command using the dnsdbq CLI command is:
dnsdbq -i 128.223.142.89 -A 24w | fgrep -v ';' | grep . | sort
Limitations of Regular DNS
“Regular” DNS is a one-way view. It only shows what’s currently configured for a particular domain or IP address. It doesn’t show many key relationships, nor does it provide a historical view. DNSDB “synthesizes” many latent or implicit DNS relationships, and thereby enables a broader range of queries. For example, DNSDB makes it possible to find virtually all the domains using the same name server. Or, given an IP address, DNSDB can tell a user all the hostnames that have historically been seen using that IP address. DNSDB can do these things and much more.
How Can DNSDB Be Used?
There are many ways that security teams can use DNSDB. Some ways include:
- Identify shared infrastructure: Users may be confronting unwanted traffic from a given source. Should they block it? If they do so, and the IP is used for shared hosting, there may be undesirable collateral damage. DNSDB will let the analyst tell what’s been seen on that IP over time.
- Validate DNS configuration: IT teams may need to identify changes when they happen for a particular host or domain. DNSDB can be used to track these movements: even if the adversary is using a fast flux domain.
- Identify related infrastructure: DNSDB is all about making connections. DNSDB lets a security analyst find related domains using the same nameserver infrastructure used by a “known bad” domain, thereby avoiding incomplete take downs.
- Conduct research and analysis: DNSDB also helps you do a post-hoc analysis of some operational issues. Why did a given service break? Was there a DNS server administrator typo?
Why Does DNSDB Work?
The internet relies heavily on DNS and criminals are not exempt. DNSDB exploits the fact that cyber criminals share and reuse resources. This can include sharing nameservers or sharing IP addresses. Other times cyber criminals will obtain a consecutive range of IP addresses, so finding one will often let you locate additional bad addresses in the same network neighborhood or netblock. In this regard, criminal activity can leave footprints in DNS. Using DNSDB, investigators can follow those trails even after the criminals have covered their tracks in regular DNS.
DNSDB Access Methods
Farsight’s DNSDB can be accessed via:
- DNSDB API
- Third party software integration (which use the DNSDB API)
- DNSDB Export (DNSDB with a copy of the DNSDB data located “on premises”)
This document focuses on DNSDB API. This datasheet does not cover DNSDB Export. For more information on these products, please read the DNSDB Export User Guide.
DNSDB Capabilities and limits
Access to DNSDB can be licensed in a number of ways and access can be granted via a number of interfaces and tools. These licenses and tools have different capabilities and limits that a user needs to be aware of.
This table summarizes these capabilities and limits summary:
Trial Products
| Product | Quota | Maximum Results | Duration | Data Available | Rate Limit | Query Privacy |
|---|---|---|---|---|---|---|
| Maltego Free Queries | 12 per hour | 12 | N/A | 2010 to now | 12 per hour | No |
To request a demonstration of DNSDB please contact the DomainTools sales team.
Subscription Products
| Product | Quota | Maximum Results | Duration | Data Available | Rate Limit | Query Privacy |
|---|---|---|---|---|---|---|
| Queries per Day (QPD) | 1K – Unlimited | 10K – 1M | 1 Year | 2010 to now | None | Yes |
DNSDB API
Farsight’s DNSDB API service can be accessed many different ways. Farsight supports key portability allowing customers to use their purchased key with many different tools. Access methods include:
| Delivery Method | Target Audiences |
|---|---|
| API for developers | Targets security application developers who can integrate DNSDB API into an existing application, or write their own front-end interface to DNSDB’s API |
| Company-provided demo command line clients like dnsdbq | Security, incident response, SOC, and research teams can easily leverage the power of DNSDB API without having to be a programmer |
| Third party software integrations | These provide a convenient way for analysts to easily/automatically enhance data managed by their favorite existing tools |
The DNSDB API for Developers
There are two versions of the DNSDB API developers can use:
- DNSDB API V2 was released in August 2020 and is recommended for all new development against the DNSDB API.
- DNSDB API V1 remains available for existing tools and ongoing development of them.
System Requirements
DNSDB API can be accessed from any Internet connected host. DNSDB API requires the ability to create a TLS-secured “https” RESTful connection. These connections can be established from software/systems of the user’s choice.
Dependencies
As an API, there are few absolute dependencies other than network access. DNSDB API is delivered via a RESTful HTTPS API and requires HTTPS access. Network connectivity issues, outbound firewalls, or air-gapped networks could prevent or limit access to the API server. Note: If an un-encrypted transport is required, DNSDB-Export is the only option.
Most of the dependencies for DNSDB are based on the client software integrating DNSDB. This can be anything from Splunk or Maltego integration tools, to a custom-built integration. Refer to specific dependencies and requirements for that software.
When working with the command line reference client, you must have a unix shell. Each of the API command line clients has its own requirements for being installed.
Capacity Planning
Similar to dependencies, capacity planning for DNSDB is largely based on the client software integrating DNSDB. Capacity planning is determined by your use cases.
Additional Considerations
- There are no DNSDB API user-side storage requirements, except to the extent that you elect to save results.
- Conducting right-hand side wildcard RRname searches may be resource intensive.
- When working with large result sets, up to the million results limit per query, adequate RAM must be available.
- Opening more than ten concurrent connections per key per IP address will cause service interruption and/or performance issues. Therefore, some client configuration, IP, and network planning may be required if more than 10 concurrent connections are needed. If your subscriptions allows, you may want to request a unique user name and key for each user.
Refer to the capacity requirements for your associated software or platform and make sure there is enough additional CPU, RAM, and storage resources to run DNSDB queries and analyze the results.
Limitations
The following are limitations with DNSDB API:
- By default, DNSDB API users are limited to 10 concurrent connections.
- The maximum number of results you can request is 1,000,000 (one million).
- Partial label queries are not supported; a search has to be done within a domain (e.g. *.example.com) or for a complete domain component (e.g. example.*). Wildcards inside a domain are not supported (can’t search for exam*ple.com). If you need these capabilities, you should investigate the DNSDB Flexible Search API
- The order in which DNSDB API returns results does not necessarily mean you are being shown the most recent results first by default. You should keep this in mind if you receive only a subset of results, as when results are limited to no more than N results.
- Farsight’s data processing (as better explained in our “PassiveDNS Architechture” white paper) prevents non-authoritative answers from being collected and attempts to that the authoritative answers come from the valid authoritative name servers for a particular domain.
- Non-authoritative records are removed from the DNSDB API when identified. Additionally, Farsight may filter other known low-value data, which while we do not disclose the exact details, may include records like randomized subdomain attacks, inverse address records tied to dynamic domain names, DNS block list queries and responses, and some busy domains that use wildcard CNAMES, such as blogspot.com.
DNSDB Company-provided Demo Command Line Clients
Company-provided demo command line clients are reference implementations of the DNSDB HTTP API. Output is compliant with the Passive DNS Common Output Format. Demo command-line-interface clients are provided:
| Implementation | Description | Required Prerequisites | Additional Information |
|---|---|---|---|
| dnsdbq | dnsdbq is one of the most feature-rich command line interfaces available for use with DNSDB API. | Linux, BSD, macOS, or WSL; jansson; libcurl | dnsdbq |
Third Party Software Integration (Using the DNSDB API)
By augmenting an organization’s internal log data with real-time Internet DNS information, security teams will be better able to analyze threats and adversary infrastructure and capabilities. This will enable them to identify, detect, correlate and take action on the intelligence.
Farsight works leading security platform partners to integrate Farsight solutions into their platforms. Some of the major platform integrations we support includea Anomali, Domaintools, Maltego, IBM’s Resilient system and Splunk. For a complete list of our third party integrations, please visit our Integrations page.
Application Examples
Using libcurl to make calls to the DNSDB API
- Making Programmatic DNSDB Queries With libcurl (API V1 only)
