User Guides

An Introductory Guide to Flexible Search With DNSDB Scout

DNSDB® 2.0 introduces a host of powerful new features and improvements – the biggest of which is Flexible Search.

Flexible Search allows for more complex searching parameters in DNSDB queries and enables more granular and accurate search results. Before DNSDB 2.0 queries were unable to contain metacharacters, groupings, repetitions, and other pattern matching values – Flexible Search enables all of these techniques by way of the Regular Expression (Regex) and Global Wildcard Expansion (Globbing) syntaxes.

In this guide we’ll explore how Flexible Search can be used in DNSDB Scout® to find threats and other interesting things recorded in DNSDB.

Accessing DNSDB Scout

DNSDB Scout is a GUI for the DNSDB API. Available in all popular web browsers via a website or browser extension, it supports all of the major features of the DNSDB API including the new Flexible Search features introduced in DNSDB 2.0.

Website Version

DNSDB Scout can be accessed by visiting in any popular web browser, including on smaller devices like phones and tablets. Accessing DNSDB Scout this way may be preferred or required depending on your device owner’s restrictions, device platform restrictions, or web browser compatibility. Some platforms, such as iOS, do not allow for web browser extensions, and other web browsers may not have a DNSDB Scout extension release available yet.

Google Chrome Extensions & Mozilla Firefox Add-ons

To use DNSDB Scout in your browser you need to have the browser extension installed. If you haven’t installed DNSDB Scout, please visit the store page for your respective web browser and install it per the instructions.

Google Chrome
- click Add to Chrome
Mozilla Firefox
- click Add to Firefox.

After the extension has been added to your browser, click the orange Scout icon in the top-right corner to bring up the Scout “Popup” menu. This is the primary way to open the Scout extension to check on your API key status and to navigate to the Dashboard and Options pages.

Regular Expressions (Regex) and the FCRE Syntax

To enable DNSDB 2.0’s new Regular Expression (Regex) searching capabilities Farsight has adopted a subset syntax of POSIX-Extended Standard Regular Expressions. This new syntax is called the Farsight Compatible Regular Expression (FCRE) syntax.

The FCRE syntax has the following restrictions and simplifications over POSIX-Extended Standard Regular Expressions syntax:

Only printable characters are allowed in a search query.
Hexadecimal and octal escape sequences are not allowed at all.
Only special characters may be escaped with \ (backslash).
- Note: ] and } are not considered special characters, but [ and { are.
- Note: PCRE extensions and shorthands such as \w and \d are not allowed.
POSIX collating elements (e.g., [=ch=], [.a.]) in character classes are not supported. The sequences [= and [. are not allowed in character classes.
Capturing groups and backreferences are not supported.

In other words, only printable metacharacters may be escaped and collating elements aren’t available. Otherwise, your favorite regexes should work when making a Regex query in Flexible Search.

While using this Regex syntax Flexible Search queries will return precise results related to your investigation.

Examples of what Flexible Search Regex queries look like and how to use them are later in this guide.

For a more detailed explanation about the FCRE Syntax itself please visit the DNSDB API Documentation Site.

For an in-depth exploration into how Regular Expressions apply to domains and DNS we recommend the What’s a Regular Expression?.

Global Wildcard Expansion (Globbing) Syntax

To enable DNSDB 2.0’s use of Global Wildcard Expansions Farsight has adopted the standard glob(7) semantics as its ‘Globbing’ syntax. This syntax is sometimes referred to as “Extended Globbing”, and also as “globs” as shorthand.

Globbing is front and back anchored by default; Regular Expressions are not anchored by default. As an example, you might try a glob of farsight* to find all matches that begin with “farsight”, where the equivalent Regular Expression would be ^farsight.*$. For an in-depth exploration into how Globbing applies to domains and DNS we recommend the What is Globbing?.

While using this Globbing syntax Flexible Search queries support real wildcards on both sides of a domain label or hostname. If a Glob query is constructed correctly then it can result in broad Keyword-like searches.

Examples of what Flexible Search Glob queries look like and how to use them are later in this guide.

For a more detailed explanation about the Globbing syntax itself, please refer to the Glob Reference Guide.

Flexible Search Tab

Flexible Search is front-and-center in DNSDB Scout. The Flexible Search tab contains all of the new DNSDB 2.0 searching features. When you open DNSDB Scout you can start making Flexible Searches right away.

Syntax

Starting from the top of the Flexible Search options, the Syntax option allows you to select between three syntaxes – Keyword, Regex, and Globbing:

Keyword is the pre-selected default – it allows for searching of simple terms like “paypal”, “rolex”, or “covid”. Under the hood, Keyword searches perform a double-ended wildcard Globbing query, so think of it as a shorthand for Globbing.
Regex is the second option – it allows for the aforementioned FCRE syntax to be used to craft search queries. With Regex, you can search for things that are “fuzzy” and would not have worked with the original DNSDB.
Globbing is the third option – it allows for the aforementioned Globbing syntax to be used to craft search queries. With Globbing, real wildcards can be used with literal values instead of the limited wildcards found in the original DNSDB.

For the easiest, but broadest, search results try Keyword.

When you start building more complex sequences or want finer control on the results, you’ll find that Regex will work better for you. Globbing is a middle ground with finer control than Keyword but is less complex than Regex.

RRSet Fields

Next is the Left-Hand (RRName) and Right-Hand (RData) Field option. Which one you select will tell the search which field of an RRSet to search in. Both support all three of the syntaxes, above. If you’re unfamiliar with the Left/Right terminology, the “Left” part of an RRSet is the DNS query while the “Right” part is the answer. For example, let’s look at an A Record RRSet:

www.farsightsecurity.com IN A 66.160.140.81

www.farsightsecurity.com is the “Left” part of this record, a queryable domain name.

66.160.140.81 is the “Right” part of this record, an answer to the query.

These fields are searched separately in Flexible Search. Left-Hand (RRName) search results will not have an RData column/field, and Right-Hand (RData) search results will not have an RRName column/field. To see both fields at the same time and exactly when they were observed a subsequent Standard Search or a pivot into a Standard Search is required.

Find

The Find field is next; this is the bulk of how you craft your query. The Find field is the primary thing you’re searching for in DNSDB – for example, a keyword like paypal, a Regex like p(a|o)ypal, or a Glob like payp*.

Exclude

Nearby the Find field is the Exclude field. The Exclude field is a way to restrict observations you know you don’t want to see in your search results. A great use case for this is for excluding your own brands, your own corporate website, or a common result that you know will pollute your results. For example, searches involving covid might get polluted by covideo so excluding that may be a good idea.

The Exclude field is disabled for Keyword searches. The expression in the exclude field uses the Regex or Glob syntax, depending which Syntax is selected.

Record Type

Next is the Record Type dropdown. By default, it uses ANY which means any record type is allows to appear in your search results. You can restrict which record types appear by changing it in this dropdown. For example, changing it to NS will restrict your search to only return name server results.

Note: Unlike in Standard Search, Flexible Search does not have any DNS-SEC related record types to select from. DNS-SEC related observations are not available in Flexible Search. Otherwise, this Record Type dropdown works the same way in Flexible Search as Standard Search.

Limit

Over on the right side of the tab we have the Limit dropdown. Changing the Limit field will restrict how many results your search returns. By default, DNSDB Scout uses a Limit of 5000.

Note: The number of results returned for Flexible Search RRNames queries may seemingly exceed the Limit, but the results for these queries use the Limit for the number of unique RRNames seen in the results and does not account for multiple RRTypes that may have been observed with the same RRNames. Otherwise, this Limit dropdown works the same way in Flexible Search as in Standard Search.

Offset

To the right of the Limit dropdown is the Offset field. This field lets you specify how many results to “skip” over if you want to go beyond DNSDB Scout’s maximum Limit. The Offset field works the same way in Flexible Search as Standard Search. This effectively raises the maximum Limit.

note: DNSDB recalculates the results for each query and the order of results might not be preserved. Therefore, this capability is not a valid way to walk all results over multiple queries – some results might be missing and some might be duplicated.

Time Fencing

Finally, the Time Fencing area. Time Fencing is used to improve the signal to noise ratio of results by filtering results to specific time ranges that are more relevant to an investigation or event.

Time Fencing is controlled by a set of three query parameters, all of which are optional:

Seen After: Allows for results where they were observed after a given date. This is the most frequently used Time Fencing option in DNSDB Scout.
Seen Before: Allows for results where they were observed before a given date.
Strict Mode: Only allow for observations that fall strictly on the specified side of a Time Fence. If enabled and both Time Fences are used then the observations will be strictly between the Time Fences. This is also known as a ‘complete’ Time Fence

For a more in-depth exploration into DNSDB Time Fencing we recommend DNSDB Time Fencing: A Post-Attack “Time Machine”.

Other Notes

If you’re already familiar with DNSDB Scout or a previous version of the DNSDB API then you may be looking for a De-Aggregate or Summarize option. These options are not available in Flexible Search.

Example Search #1: Enumerated Subdomains

An investigation involving a series of enumerated subdomains can be difficult to keep track of with DNSDB Standard Search. For example, say we find the CNAME drupal-hosting-web-cluster5-prod.uoregon.edu. while looking at observations for www.uoregon.edu. That cluster5 gives us a clue – there are probably more CNAMES like it.

With Standard Search we would have to make multiple RRSet searches, one for each of these possibilities until we find one that fails:

drupal-hosting-web-cluster1-prod.uoregon.edu.
drupal-hosting-web-cluster2-prod.uoregon.edu.
drupal-hosting-web-cluster3-prod.uoregon.edu.
…
drupal-hosting-web-clusterN-prod.uoregon.edu.
etc.

With a single Flexible Search query in DNSDB Scout we can figure out how many CNAMES there are that match this pattern instead of guessing at numbers that follow our clue.

Open DNSDB Scout; the Flexible Search tab should already be selected
Select the Regex syntax
Enter ^drupal-hosting-web-cluster[0-9]+-prod\.uoregon\.edu\.$ into the Find field. This Regex means we’re looking for some numerical digits after cluster in otherwise the same pattern
Click the Search button

Depending on your investigation, you may also want to restrict the Record Type, but in this example we’ll leave it alone.

In the results we find there are six RRNames matching this pattern – now we don’t have to guess and check at numbers.

If there was a gap in the numbers then we wouldn’t have known how large of a gap without this pattern searching. What if there was a cluster8 thru cluster11 even though there’s no cluster7? Flexible Search has eliminated the queries we need to “burn” to verify if we found all of the relevant RRNames. This technique is not only useful for CNAMEs, but also RRNames associated with A, AAAA, MX, and NS records.

From here, one available option to continue this investigation is to export our results for another tool to use. We can export the results as a different file format using the Export dropdown above the results table.

Another option is to pivot on one of these RRNames into DNSDB Standard Search. Hover over one of the RRNames, like hosting-web-cluster6-prod.uoregon.edu. and it will present you with a pivot tooltip. In the tooltip will be a number of pivot options. Clicking the first pivoting option, which says Standard for RRSet: drupal-hosting-web-cluster6-prod.uoregon.edu., will perform a DNSDB Standard Search for the RRName as part of an RRSet query which will then allow you to see the full DNS resource record as observed by DNSDB.

Once again, we can export our results as CSV or JSON for another tool to use or we can continue to pivot by clicking the pivot links presented in the table.

Example Search #2: Lookalikes and Typosquatting

An investigation involving payment and banking systems may involve domain label lookalikes and typosquatting. Let’s use some simple regexes to find some domains that look similar to Paypal and Wells Fargo’s.

First, Paypal. A common phishing technique is to use dashes in a label. Let’s see what’s out there with a wildcard and some dashes while keeping in mind we want to exclude results for the real Paypal.

Open the Flexible Search tab
Select the Regex syntax
Enter www\..*-paypal- into the Find field. This Regex means we’re looking for something with www and also paypal between dashes
Enter ^.*paypal\.com\.$ into the Exclude field – we know this is the real Paypal so we can exclude it from our results
Click the Search button

In the results are thousands of RRNames that fit our search.

When faced with a lot of results you can narrow them down by re-running the search with a Record Type selection or Time Fencing to keep the results relevant to your investigation.

Next, Wells Fargo. A common phishing technique is to duplicate hard-to-distinguish letters, like l. Let’s see what’s out there by using a repetition regex pattern.

Open the Flexible Search tab
Select the Regex syntax
Enter wel{3,6}sfargo into the Find field. This Regex means we’re looking for something with three to six l characters that looks likes similar to wellsfargo
Click the Search button

Although there aren’t as many results in this example as in the previous one, there’s still a lot to sort through. You can narrow the results down by re-running the search with the Record Type option and by using Time Fencing, whichever helps you narrow your results so they’re relevant to the investigation.

Example Search #3: Brand Protection

Companies and brands need to defend their intellectual property, even in DNS. Searching for brand names among FQDNs and hostnames to find infringers can be challenging, but Flexible Search and DNSDB Scout makes this much easier. DNSDB Scout provides a novel way to make use of simple string searches by offering a customized Globbing search option as a syntax: Keyword. By using the Keyword syntax you have an easy way to see what’s going on with a brand name that’s part of a domain label. Let’s use rolex as a Keyword and see what we find.

Open the Flexible Search tab
Select the Keyword syntax
Enter rolex into the Find field
Click the Search button

In the results you’ll see domains that contain rolex somewhere in them; you’ll likely find something that doesn’t quite represent the brand’s true owner very quickly.

You can narrow your Keyword Search results by using a Record Type filter for A, CNAME, or NS records and also a Seen After Time Fence for the last month. Doing this will ensure your results contain only recent observations that result from “normal” web traffic patterns.

Example Search #4: Domain Generation Algorithms (DGAs)

Modern botnets and malware sometimes use Domain Generation Algorithms (DGAs) to register and communicate over seemingly random domain names as part of their infrastructure in order to make them robust and hard to take down. Security experts and researchers have published regular expressions that match the these DGAs and these regular expressions are then used to find or block botnets and malware. With a DGA regular expression in hand, you can search DNSDB for observations related to botnets and malware to make a map of when they were spotted and how widespread they were.

Open the Flexible Search tab
Select the Regex syntax
Enter the DGA regular expression into the Find field. In this example we’ll use ^f[a-z]{5,6}-[a-z]{4,5}.(ru|com).$
Click the Search button

The results will contain FQDNs that match this particular DGA.

Caution: Not all search results you find using DGA regular expressions will actually be part of a botnet or malware; results will contain everything that matches the pattern, both malicious and benign. Take precautions if you decide to probe the results outside of DNSDB Flexible Search and DNSDB Scout.

Example Search #5: Record Misconfiguration

Flexible Search can be used as a way to find DNS misconfigurations. For example, a common SPF Record mistake is to use an ipv4 or ipv6 modifier instead of the correct ip4 or ip6 modifier. A common SOA Record mistake is to use an IP address instead of an MNAME (Primary Name).

Let’s look for some SPF Record mistakes observed in the last month.

Open the Flexible Search tab
Select the Regex syntax
Select the Right-Hand (RData) field option
Enter (ipv4|ipv6) into the Find field
Select the SPF Record Type
Enter a Seen After date for a month ago; in this example we’ll use July 12, 2020 12:00 AM UTC
Enable Strict Mode so observations are First Seen after the Time Fence
Click the Search button

The results will show recent observations that have misconfigured SPF records using an invalid IP modifier of ipv4 or ipv6.

Next, let’s look for SOA Record mistakes observed in the last month.

Open the Flexible Search tab
Select the Regex syntax
Select the Right-Hand (RData) field option
Enter 8\.8\.8\.8 into the Find field
Select the SOA Record Type
Enter a Seen After date for a month ago; in this example we’ll use July 12, 2020 12:00 AM UTC
Enable Strict Mode so observations are First Seen after the Time Fence
Click the Search button

The results will show recent observations that have misconfigured SOA records trying to use Google DNS as an SOA MNAME.

COVID-19 has impacted everything, and a way to observe it is through DNS observations. With Flexible Search we can see when COVID-19 related domain registrations and websites started reacting to the news.

Open the Flexible Search tab
Select the Globbing syntax
Select the Right-Hand (RData) field option
Enter *.covid??* into the Find field. We’re looking for domain labels containing covid with at least two characters following it.
Select the NS Record Type as they’re usually the first-movers with fresh domain registrations
Enter a Seen After date for a month ago; in this example we’ll use February 20th, 2020 12:00 AM UTC
Enable Strict Mode so observations are First Seen after the Time Fence
Click the Search button

In the results we see RData that fits our loose COVID Globbing Search.

A more generic Globbing Search for *coronavirus* for ANY Record Type yields different results, though this may not be relevant to COVID19 specifically but rather other strains of coronavirus.

To get an idea of the activity periods of these results try pivoting over to Standard Search to see the Time First Seen, Time Last Seen, and Count fields.

Summary

If you’ve followed this guide to the end then you should now be familiar with how Regular Expressions (Regex) and Global Wildcard Expansions (Globbing) work in DNSDB Flexible Search and within DNSDB Scout. With Flexible Search you now have much more flexibility in crafting searches related to your investigations. We hope this guide demonstrates how Flexible Search is a great addition to DNSDB and how it can be used.

If you have any feedback or would like to see us break down more complicated DNSDB searches in the future please contact us and let us know,

References

For a more detailed explanation about the Farsight Compatible Regular Expression (FCRE) Syntax please visit the DNSDB API Documentation Site.
For more information about how Regular Expressions apply to domains and DNS please see Joe St Sauver’s What’s a Regular Expression?
For a reference guide to Global Wildcard Expansion (Globbing) please see the Glob Reference Guide.
For more information about how Globbing applies to domains and DNS please see Joe St Sauver’s What is Globbing?.
For more information about DNSDB Time Fencing please see Joe St Sauver’s Farsight’s DNSDB Time Fencing: A Post-Attack “Time Machine”.