Abstract
This article explains Farsight’s Network Message (NMSG) loss tracking feature.
To get the most from this article, it is recommended that you be comfortable with the material from the following Farsight Security Blog articles:
- Farsight’s Network Message, Volume 1: Introduction to NMSG
- Farsight’s Network Message, Volume 2: Introduction to nmsgtool
- Farsight’s Network Message, Volume 3: Headers and Encoding
- Google Protocol Buffer Deserialization The Hard Way
Introduction
NMSG is an extensible container wire and file format for storing and transmitting blobs of data with support for dynamic message types, compression, fragmentation, sequencing, and rate limiting. We will cover NMSG’s data sequencing feature.
NMSG’s data sequencing feature, referred to as “seqsrc” was added in version
0.8.0 to provide a mechanism for tracking container loss across UDP-socket
based NMSG sessions. This is especially useful when a user wants to track
packet loss in high-throughput NMSG sessions.
The seqsrc feature is available in nmsgtool and exposed via the C API,
examples of both are given below.
How it Works
Seqsrc loss tracking is performed by nmsg on the receiver side on a per-container basis (as opposed to per-payload). Each NMSG container carries with it a sequence number and sequence_id. The sequence number is an unsigned 32 bit number that serves as — you probably guessed — the container’s sequence number. The sequence_id is an unsigned 64 bit nonce used as an opaque pseudo-random cookie to prevent inadvertent sequence space reuse. From the nmsg API documentation, the sequence_id is:
a randomized ID number identifying the sequence number space that the ‘sequence’ parameter exists in. This ID number is used by NMSG consumers to disambiguate multiple disparate sequences of NMSG containers when consuming a multiplexed stream from multiple sources. This ID number should be generated by a cryptographically secure PRNG such as the one provided by nmsg_random_init() to minimize the possibility of collisions.
Indeed, during the development of seqsrc and prior to the addition of a sequence_id, Farsight Security did actually see birthday collisions due to the very high UDP port number reuse on high bit-rate SIE channels.
To the NMSG consumer, seqsrc is implemented as a series of separate and
independent sequence number spaces. Each of these sequence number spaces is
relevant only to the NMSG connection that it references (there should be one
per nmsg_output_t object on the sender-side), and only for the time that it
exists. In a lossless system, this value should increase monotonically without
gaps.
NMSG will partition a new seqsrc space using the IP source address, IP protocol, source port, and sequence_id to create a tuple that gets its own sequence space. As such, a seqsrc key is defined as:
struct nmsg_seqsrc_key {
uint64_t sequence_id;
sa_family_t af;
uint16_t port;
union {
uint8_t ip4[4];
uint8_t ip6[16];
};
};
Internally, Farsight Security uses seqsrc to monitor container loss in real-time on its Security Information Exchange (SIE).
Loss Tracking with nmsgtool
The reference implementation of nmsg, nmsgtool, supports seqsrc when invoked
with (at least) four levels of debug (-dddd).
To see seqsrc in action, we’ll use a corpus of channel 202 NMSGs containing just over 2,000,000
payloads (we use the nmsgpcnt program to count containers and payloads) and send these,
from one unbuffered nmsgtool instance, over the loopback interface, to another
listening nmsgtool instance.
The nmsg file we’ll use:
$ ls -l 202-2000000.nmsg -rw-r--r-- 1 mschiffm staff 753800034 Mar 7 2015 202-2000000.nmsg $ nmsgpcnt-fsi 202-2000000.nmsg containers: 720 payloads: 2000481
First, we instantiate an nmsg listener on loopback/UDP port 8888 with the proper debug level and instruct it to write binary nmsgs to file:
$ nmsgtool -l 127.0.0.1/8888 -dddd -w foo.nmsg ...<debug messages omitted>...
Next, we fire off an nmsgtool instance with the --unbuffered option which
will send each payload in its own container, as fast as it can (over the
loopback interface on my several year old MacBook Pro this should result in
some container loss):
$ nmsgtool -r 202-100000payloads.nmsg -s 127.0.0.1/8888 --unbuffered
We then quit the initial nmsgtool session (via ctrl-c) and inspect the debug
output:
... ^Cnmsgtool: signalled break nmsg_io: iothr=0x7faf8c6002c0 count_nmsg_payload_in=1989299 _input_seqsrc_destroy: input=0x7faf8c502e50 count_recv=1989299 count_drop=11182 (0.0056) nmsg_io: io=0x7faf8c502fe0 count_nmsg_payload_out=1989299
nmsgtool reveals that of the 2,000,481 containers sent, it received 1,989,299
and 11,182 were lost.
Loss Tracking with the nmsg C API
The nmsg C library exposes a very simple API to enable, disable, and access
seqsrc data. The following code snippet is taken from the working sample nmsg
program nmsgjsonseqsrc:
First, we explicitly enable seqsrc verification on a previously instantiated
nmsg input object (note that as of version 0.12.1, nmsg enables it by
default):
...
/* Instruct io engine to enable container loss tracking. */
res = nmsg_input_set_verify_seqsrc(input, true);
if (res != nmsg_res_success)
{
fprintf(stderr, "nmsg_input_set_verify_seqsrc(): %s\n",
nmsg_res_lookup(res));
return (EXIT_FAILURE);
}
printf("verify seqsrc enabled\n");
...
After running the nmsg io engine and (presumably) processing many NMSGs, we cull and emit the received and dropped container count:
...
/* Retrieve the total number of NMSG containers that have been
* received since the nmsg_input_t object was created. */
res = nmsg_input_get_count_container_received(input, &count);
if (res != nmsg_res_success)
{
fprintf(stderr,
"nmsg_input_get_count_container_received(): %s\n",
nmsg_res_lookup(res));
return (EXIT_FAILURE);
}
printf("\nrecevied %llu containers\n", count);
/* Retrieve the total number of NMSG containers that have been
* dropped since the nmsg_input_t object was created. */
res = nmsg_input_get_count_container_dropped(input, &count);
if (res != nmsg_res_success)
{
fprintf(stderr,
"nmsg_input_get_count_container_dropped(): %s\n",
nmsg_res_lookup(res));
return (EXIT_FAILURE);
}
...
Using the same NMSG file and unbuffered sending invocation as in the example above, we see the following result:
$ ./nmsg-seqsrc -l 127.0.0.1/8888 nmsg JSON emitter / seqsrc example program nmsg initialized nmsg io engine initialized nmsg socket input initialized verify seqsrc enabled socket input added to io engine callback initialized callback added to io engine entering io loop, <ctrl-c> to quit... ...<newline delimited json output omitted>... ^C received 1041381 containers dropped 959079 containers
nmsgjsonseqsrc reveals that of the 2,000,481 containers sent, it received
1,041,381 and 959,100 were lost (we lost more here because the nmsgjsonseqsrc
program deserializes each incoming NMSG as JSON, a more computationally
expensive operation than simply writing binary NMSGs to a file).
Mike Schiffman is a personal packet shopper for Farsight Security, Inc.
