Farsight Streaming API Framing Protocol Documentation
Introduction
This document describes the Farsight Streaming API Framing (SAF) protocol, a data transfer protocol. It is initially used by DNSDB to transfer pre-standard IETF COF format JSON objects. SAF is basically a wrapper protocol around JSON objects.
Audience
This document is intended for programmers who want to write applications that can interact with the RESTful DNSDB API using JSON and HTTP.
Format
The data is transmitted as newline delimited JSON lines, aka JSONL or NDJSON.
All string values are UTF-8. All attribute names are US-ASCII.
Whitespace within the JSONL structure should be ignored.
Each JSONL object contains the optional attributes:
“cond” is a condition. It is a scalar enum of string values:
- “begin”
- “ongoing”
- “succeeded”
- “limited”
- “failed”
“msg” is a human readable message. It is a scalar string which ought to be reported to the end-user or in the system log.
“obj” contains data appropriate for the protocol using SAF encapsulation. It is a wrapped JSON object. For the DNSDB API, it contains a pre-standard IETF COF object.
Other attributes, if present, would be the subject of some future protocol revision.
The condition “begin” MUST be in the first, initiating JSONL object. This object MUST contain a “cond” and MAY contain a “msg”.
The conditions “succeeded”, “limited”, and “failed” are terminating conditions and MUST be in the last JSONL object.
JSONL objects after the first and before the last MAY contain an “obj” attribute, MAY contain a “msg” attribute, and MAY either specify “cond” to be “ongoing” OR omit the “cond”. The default value for “cond” is “ongoing”.
If a terminating condition is missing (i.e. the last object in the data stream does not contain one) then the client should interpret this as truncation and generate an appropriate error message to report to the end-user or in the system log. The previously received data objects may be discarded or used subject to the knowledge they may be incomplete. This is also the case if the data stream terminates with a TCP error. A client is free to retry a truncated operation.
If any JSONL object cannot be parsed as valid JSON then both that JSONL object and all remaining JSONL objects must be discarded and some warning must be issued to the user or system log.
If the terminating condition is “succeeded” or “limited” then the data objects are valid.
If the terminating condition is “failed” then the data objects may be discarded or used subject to the knowledge they may be incomplete.
If a JSONL object after the first and before the last, either having no “cond” attribute or having a “cond” value of “ongoing”, contains a “msg” attribute then the “msg” describes an informational or warning condition which has no decisive bearing on the validity of the objects in the response stream. Other values of “msg” SHOULD be reported to the end-user or in the system log.
If a JSONL object after the first and before the last, either having
no “cond” attribute or having a “cond” value of “ongoing”, contains
neither a “obj” nor a “msg”, then this is a “keep alive” message. A
keep alive message is designed to signal to the client that the
connection is still open. A legal representation of a keep alive
message is the empty JSON object {}. Keep alives are useful for
queries that take a long time to process before reporting anything to
the user. Keep alives are also known as “heartbeats”.
State Machine
The state machine grammar is:
initiating JSONL object :=
{ "cond": "begin" [, "msg": "$message"]}
ongoing JSONL object :=
{ [ "cond": "ongoing",] ["obj": { $object }] [, "msg": "$message"]}
terminating JSONL object :=
{ "cond": "succeeded" [, "msg": "$message"]}
|
{ "cond": "limited" [, "msg": "$message"]}
|
{ "cond": "failed" [, "msg": "$message"]}
START => initiating JSONL object => ongoing JSONL object OR terminating JSONL object
ongoing JSONL object => ongoing JSONL object OR terminating JSONL object
terminating JSONL object => END
$message => scalar string
$object => JSON object wrapped by the SAF protocol
Examples
Simple successful example
{"cond": "begin"}
{"obj":{"count":10392,"time_first":138126549...}}
{"cond": "succeeded"}
The second JSONL object implicitly has the default condition of “ongoing”.
Equivalent simple successful example
{"cond":"begin"}
{"cond":"ongoing", "obj":{"count":10392,"time_first":138126549...}}
{"cond":"succeeded"}
Long-running example with “alive” messages, as implemented
This example shows a query that took a while to return the first
response, and then had occasional delays in responding. Because
“cond”:”ongoing” is the default, it is normally omitted, which means
that a “keep alive” message is transmitted as an empty JSON object,
{}. The example also shows varying amounts of whitespace, as the
whitespace should be ignored.
{"cond":"begin"}
{}
{}
{"obj":{"count":10392,"time_first":138126549...}}
...
{}
{"obj": {"count": 1234,"time_first": 238126549...}}
...
{}
...
{"obj" : {"count" :456, "time_first":338126549...} }
{"cond":"succeeded"}
Long-running example with “alive” messages, as possible.
This is the same as the previous example, but explicitly shows the default “cond”:”ongoing” attribute.
{"cond":"begin"}
{"cond":"ongoing"}
{"cond":"ongoing"}
{"cond":"ongoing", "obj":{"count":10392,"time_first":138126549...}}
...
{"cond":"ongoing"}
{"cond":"ongoing", "obj": {"count": 1234,"time_first": 238126549...}}
...
{}
...
{ "cond" : "ongoing", "obj" : {"count" :456, "time_first":338126549...} }
{"cond":"succeeded"}
Limited example
{"cond": "begin"}
{"obj":{"count":10392,"time_first":138126549...}}
{"obj":{"count":33,"time_first":19126549...}}
{"cond":"limited", "msg":"Result limit reached"}
In this case, the results are valid and SHOULD be used.
Note: The “Result limit reached” message does not imply that re-trying the query with a higher limit value will return more results, only that the limit was reached.
Failure example
{"cond": "begin"}
{"obj":{"count":33,"time_first":19126549...}}
...
{"cond": "failed", "msg": "Processing timeout; results may be incomplete"}
In this case, the results MAY be used, but it is recommended to retry the request.
Successful but empty examples
{"cond": "begin"}
{"cond": "succeeded"}
{"cond": "begin"}
{}
{}
{"cond": "succeeded"}
These are successful examples that are empty of data.
