Abstract
If you read any security industry press you will likely see at least one reference to the term “threat intelligence”. But what exactly is it and what does it mean? Unfortunately as it has become a saturated marketing term, threat intelligence means different things to different people. Much like “Advanced Persistent Threat” (APT) has been for a few years, you have to know who’s speaking to understand the context in which they’re using the term.
Threat Intelligence
Gartner defines threat intelligence as:
…threat intelligence is evidence-based knowledge, including context, mechanisms, indicators, implications and actionable advice, about an existing or emerging menace or hazard to assets that can be used to inform decisions regarding the subject’s response to that menace or hazard.
Based on this definition, true threat intelligence would need come in a rich format such as STIX or another format capable of expressing a great level of detail, such as a detailed report. To meet the expectation of the Gartner definition both indicators and mechanisms as well as the confidence associated with that information would need to be included in the output.
The DIKW Pyramid
Though it may be a trope, the DIKW Pyramid gives us a simple framework for discussing the relationship of various forms of security feed. Each layer is discussed below.
Data
At the first layer, D stands for data which is the least valuable and most plentiful stage. This stage can be thought of as a collection of atomic facts in isolation. These facts may be interesting, they may be repetitious and they may even be obvious. We could be talking about NetFlow, pcap, log lines, SNMP traps, or anything else that exists in “raw” form as it would be collected in the field. This raw data is a critical source material but it would be a real stretch to call it threat intelligence. While it might be rich in indicators it is sorely lacking in context, implications or advice. In this form it wouldn’t even be appropriate to call this a threat feed, the raw data as collected would include both harmless and malicious activity.
Information
The next layer, I represents information. Information is often filtered to include only a defined context possibly de-duplicated or otherwise reduced to the set from data that has value for a specific application. This stage would also see the filtering of common false-positives and the removal of non-critical privacy impacting information.
The I layer is the stage where many researchers start to produce feeds as an output and most community data sharing projects function at this stage. However, according to Gartner’s definition, we cannot yet call this information threat intelligence. With careful filtering we can start to add depth and context to the information that it does provide and in some cases begin to formulate advice for consumers that share the assumptions made by the publisher.
The data that Farsight collects as part of our PassiveDNS platform exists at the I layer. We have designed the collection process such that we gain the benefit of existing caching as a first-stage de-duplication mechanism. We can also leverage the abstraction provided by this cache to anonymize the data collected (this is just one example, it illustrates how some careful planning can result in easy value-add).
Knowledge
The K layer sitting on top of information, is knowledge. At this layer we can take multiple information-level feeds and their associated context and combine them such that the output includes additional details or is filtered in such away that it becomes actionable. Extremely effective products functioning at the knowledge layer also take as input context from the consumer. For example, this can be their networks or domain names, security posture, risk tolerance and so on. Here we start to see the early signs of threats intelligence, however many outputs of knowledge level systems can be ingested by automation.
Wisdom
Finally we have the wisdom layer W. If you’re going to stick strictly to the Gartner definition true threat intelligence products must exist here. This is where you would take elements from the lower layers and combine them with a detailed analysis and even more external data and produce either a report or a rich data feed providing almost step by step actionable countermeasures and detailed risk analysis to the consumer.
Timing, Action, and Value
There are many products and services in the security marketplace today that can be called threat intelligence, there are also many that cannot. A key factor that we have not yet discussed is timeliness. Wisdom layer threat intelligence, at the very least, takes hours if not days to collect, vet, and compile. With sufficiently advanced automation, knowledge can be generated often in a matter of minutes. The big advantage of the information layer is that it can often be acted on automatically within seconds. Last we have data, while it’s much harder to effectively act on in a real-time manner, it can be on hand within milliseconds.
The value of a feed has a complex relationship with its timeliness. Feeds that are completely digested into a human readable report with context and confidence are often actionable, and therefore valuable operationally and often more expensive. It is worth noting that in order to reach the level of detail require to meet Gartner’s definition takes time. While a report is more immediately actionable the delay required to generate and ingest that report would reduce its effectiveness in preventing attacks.
Further down the stack, feeds at the information or data layers often lend themselves to automation. With careful architecture systems can consume lower confidence, higher volume feeds that enable the security team to defend first and understand later. With the reduction of manual oversight these lower layer feeds often have a higher false positive rate, however if the use of the feed is carefully aligned with the assertions made by the feed the risks to operations can be minimized. It is often preferable to block a site for a few minutes based on an automated feed while a manual determination is made.
As you’d expect wisdom layer products are usually more expensive then the other lower layers. Knowledge less so and information can often be found in the open-source domain. Data is somewhat interesting while it is the least valuable in terms of actionability, access to it often touches on privacy or regulatory issues. Sources at the data layer are also encumbered by volume and transmission challenges. So while it may be the least processed, data, at times is only available as under NDA or agreement. As a result of this you often find the data partners prefer to exchange information layer products.
Conclusion
Defining precisely what a “threat intelligence” service looks like can be a challenge. No two vendors build to the exact same parameters. The DIKW Pyramid gives us some simple building blocks with which to assemble an array of products. Knowing how your feeds come together can help you leverage them to the best effect. Be sure to understand how the source material is collected and processed, this will enable you to select the most appropriate actions to take or inference to draw.
Ben April is the Director of Engineering for Farsight Security, Inc.
