The AXA Technical Overview
Farsight Security’s (now a part of DomainTools) Advanced Exchange Access (AXA) is a suite of tools and library code that brings the Security Information Exchange (SIE) directly to the end-user’s network.
SIE is a scalable and adaptable real-time data streaming and information sharing platform. SIE collects and provides access to more than 200,000 observations per-second of raw data from its global sensor network. Farsight also applies unique and proprietary methods for improving usability of the data, directly sharing the refined intelligence with SIE customers and DNSDB®, one of the world’s largest passive DNS (pDNS) databases.
The diverse set of data available from SIE includes the following and is relevant and useful for practitioners in various technology roles:
- Raw and processed passive DNS data
- Darknet/darkspace telescope data
- SPAM sources and URLs
- Phishing URLs and associated targeted brands
- Connection attempts from malware-infected systems (as seen by a sinkhole)
- Network traffic blocked by Intrusion Detection Systems (IDS) and firewall devices
Each unique set of data in SIE is known as a channel and the data acquired from a specific channel can be customized to meet the needs of each customer, enabling you to subscribe to and access only the channels needed to solve your problem. A channel in SIE may be the result of raw data analysis, or a subset of data from other channels.
The data available from SIE channel subscription packages includes:
- Raw Passive DNS: Real-time observations of DNS cache-miss traffic sent from DNS recursive resolvers on the Internet to authoritative name servers. The DNS information includes authoritative DNS data that various zone operators make available and responses from authoritative name servers to recursive resolver queries
- Value-Added Passive DNS: Real-time de-duplicated, filtered, and verified passive DNS (pDNS) data observed on the Internet
- Newly Observed Domains (NOD) and Newly Observed Hostnames (NOH): Real-time actionable insights for domains and hostnames, fully qualified domain names (FQDNs), when they are first successfully resolved on the Internet
- Base Channels: A collection of threat intelligence channels that provide access to honeypot data (darknet and spam) and botnet (e.g., Conficker) sinkhole data. The data also includes threat intelligence for phishing campaigns and log data for network traffic blocked by Intrusion Detection Systems (IDS) and firewall devices
- Premium Channels: A range of premium security-related feeds including malware metadata, IOCs and other telemetry. Subscribers consume the intelligence as real-time event flows rather than traditional batch transfers – which are inherently delayed
The SIE Channel Guide provides an overview of the channels available from SIE.
Advanced Exchange Access Toolkit
The Advanced Exchange Access (AXA) toolkit contains tools and a C library to bring Farsight’s real-time data and services directly from the Farsight Security Information Exchange (SIE) to the subscriber’s network edge.
AXA enables subscribers to connect to Farsight’s subscription-based SRA (SIE Remote Access) servers. These servers provide access to data and services built from Farsight’s SIE.
Contents of the Advanced Exchange Access Toolkit
Farsight freely provides several end-user tools to stream AXA data in the Advanced Exchange Access Toolkit. This distribution contains the following:
- sratool: A command-line tool used to connect to an SRA server, set watches, enable SIE channels, and stream data.
- sratunnel: A command-line tool that streams SIE data to the local network.
- libaxa: A C library providing an API for the AXA protocol including:
- connection instantiation/teardown
- message encapsulation/decapsulation
- watch parsing/loading
- control packet rate limits, sampling rates, window sizes, and many other AXA-specific functions
Many of the features and options we’ll be discussing are the same amongst all of the tools and examples will be given where appropriate.
Farsight SIE Remote Access
SIE Remote Access (SRA) is Farsight’s software solution to make SIE content available to remote users. SRA enables SIE channel traffic to be delivered through a TCP stream across the Internet. In order to reduce bandwidth, SRA provides subscribers with the ability to invoke a server-side filtering capability across a set of channels, selecting only that subset of records that match specific domain name / IP address search criteria.
The Advanced Exchange Access suite of tools and library code is the software that implements Farsight’s SRA service.
The AXA suite consists of two Unix command line tools and one C library developers can use to build custom SRA applications.
sratool
sratool is the AXA Swiss Army Knife. It is a versatile tool used to test, debug, or stream AXA connections. It connects to an SRA server, sends protocol messages and displays the responses. It can also tunnel SIE data like sratunnel does.
sratunnel
sratunnel transfers selected SIE data from the remote server to the local network. The connection to the server is created and restored, with binary exponential delays between retries.
sratunnel is the workhorse of the AXA family. It is used to transfer SIE data from the remote server to the local network. It is what Farsight uses for production deployment of SIE data to the customer. sratunnel can be thought of as a fast, efficient, and smart conduit for SIE data. Data goes in one end and sratunnel can emit the data into different output formats, including:
- NMSGs to a UDP port
- NMSGs to a TCP port
- NMSGs to a file
- pcap to a file
- pcap to a network interface
libaxa
libaxa is the C programming library that exposes the AXA API to the application programmer.
The AXA Protocol
The AXA protocol is documented in the section titled AXA Protocol in the README file found on the GitHub repository for the Farsight Advanced Exchange Access Toolkit.
AXA Limits
Some of the channels offered by the SIE network burst to an extremely high bitrate. AXA has two ways to deal with such network-hungry situations: optional filtering and loss-tolerance built into the protocol.
Filtering can take one of the following forms:
- Via the rate limit option to reduce the flow of ingress data to a certain number of packets per second.
- Via one or more IP-based or DNS-based “watches” to limit the flow of data to specific assets the subscriber wishes to observe.
Finally, AXA is a deliberately lossy protocol. If a subscriber requests more data than the network can carry, data overruns will occur. When this happens, loss markers are transmitted reliably within the AXA stream to inform the subscriber via the AXA accounting subsystem (see below). At this point, the subscriber’s possible mitigation strategies include:
- ask for less data via rate limiting
- increase network capacity and/or other host resources
- treat the SRA stream as a chunky and non-representative sample of the total SIE data
- pursue Direct Access to SIE
AXA Watches
In the AXA world, the way an end-user registers interest in an asset is through what’s known as a watch. These are fundamental building blocks of AXA sessions. In order to get anything done, an end-user must specify one or more watches to inform the AXA server what she is interested in seeing.
There are four different types of AXA watches, each is described below:
- IP Watches: used to express interest in SIE messages containing a specified IP address or CIDR block. AXA supports both IPv4 and IPv6 address types.
- DNS Watches: used to express interest in SIE messages containing a specified hostname, domain, or wildcard hostname.
- Channel Watches: used to express interest in all SIE messages from a given channel. This is useful to enable “the firehose” for a given channel and ask SRA to send everything from the specified channel rather than matching IP address information or DNS names. Only valid for SRA connections.
- Error watches: used to express interest in SIE messages that cannot be decoded by the server. Only valid for SRA connections and intended only for debugging.
Watches are referenced by tags. A tag is simply an arbitrary 16-bit integer label used to keep track of individual watches.
When connected to an SRA server, tags must be unique. Each watch will use a different tag.
Depending on the tool, tagging may be done explicitly by the end-user or implicitly, by the tool.
SRA and AXA RESTful Interface
The Farsight Security Advanced Exchange Access (AXA) RESTful Interface adds a streaming HTTP interface on top of the AXA toolkit to enable developers of web-based applications to interface with Farsight Security’s SIE Remote Access (SRA) servers.
The SRA module facilitates the real time streaming of data from the Security Information Exchange (SIE) over HTTP using a RESTful API.
Access is controlled via an API key that is passed as the X-API-Key HTTP header.
The Farsight Security AXA RESTful Interface does not have any specific operating system requirements as it is delivered over a RESTful API. Farsight Security provides a convenience command line interface (CLI) tool that doubles as a Python extension module which is compatible with various modern operating systems. AXA REST requires HTTPS permitted outbound to axa-sie.domaintools.com. Subscribers must have purchased a service entitlement from Farsight Security and have been provisioned an API key.Accounting Messages
By default, axamd will return AXA accounting messages containing current counter statistics relevant to the current session. For more details on these packet counts, see the Accounting section below
Rate Limiting
Rate limiting is used to limit the rate of incoming AXA messages transmitted from the server to the client. While it works for SRA, it is primarily useful for high bitrate SIE channels such as the DNS Errors channels.
Sampling
Sampling is a way to statistically sample a percentage of messages from the server. For example, if a value of 50 is chosen, AXA will return a uniformly distributed random sample of 50% of the messages caught by watches.
TCP Buffer Sizing
AXA allows the end-user to get or set the TCP send buffer size used by the server. These buffers serve to accumulate outgoing data that the network stack has not yet been able to put on the wire and data that has been received from the wire but not yet read by applications. In simpler terms, if you know what you’re doing, these options allow the end-user to adjust the size of the in-kernel TCP buffers to optimize network performance. As an aside, these options do not directly adjust the TCP window size (see TCP Windows and Window Scaling for more information).
Note that internally, TCP actually allocates twice the size of the buffer requested (using the extra space for internal purposes. This is why AXA returns a buffer size that is twice the size requested. The minimum size is 1024 and the maximum is 262142. Only sratool support this option for TCP-based connections.
Accounting
AXA has a series of server-side packet counters used to track traffic totals. This is the mechanism by which AXA tracks, logs, and communicates server-side packet information. This information is intended for users of SIE Remote Access (SRA).
What is Accounting?
Accounting is AXA’s way of keeping track of traffic totals. Server-side, AXA maintains a series of per-client packet counters (a full list is below). The AXA protocol message AXA_P_OP_ACCT sent from client to server is used to query this data. The feature is available from sratool via the acct command. It is also available from sratunnel via the -A command line option. Accounting messages are also logged server-side.
Accounting Glossary
The AXA accounting counters are described below.
- total-congested: Packets that were unable to be forwarded to the client due to connection issues, although it could also be client load (i.e., the connection isn’t removing data from the server fast enough, or the client isn’t reading and processing the data fast enough). Note that: high server load might increase total-missed, but it would generally decrease total-congested by throttling the input to the congested pipe / process.
- total-filtered: This is an SRA input counter. It measures packets/nmsgs that have been read on the input side (usually directly from the SIE network) and will be submitted to the filtering logic (i.e.: an AXA watch).
- total-missed: This is an SRA loss counter. It measures packets that the SRA server failed to receive on the input side either because the daemon was too busy or because they were lost during transit. This counter tracks NMSG and pcap-based loss.
- total-ratelimited: This measures packets that were dropped instead of being forwarded to the SRA client to comply with the rate limits specified for the client.
- total-sent: This measures packets that have been sent to an SRA client.
