How Some People Got Started in Security and Anti-Abuse Work: Anybody Remember Usenet?
Introduction
You may sometimes wonder what prompts people to get started in the crazy world of system and network security work, or how they got started fighting cyber crime and network abuse. Why would anyone fight hacker/crackers, or phishing and online fraud, or spamming?
For a surprising number of older anti-abuse people, the answer may be “Usenet.” Usenet began way back in 1980, and is a service that easily predates the World Wide Web.
Usenet was, and remains, a sort of distributed international bulletin board
system. Usenet messages look a lot like email messages, except instead of being
sent person-to-person, users “post” or “submit” messages to one or more Usenet
newsgroups. If you wanted to read articles about a particular computer science
topic, talk a little politics, discuss chess, or chat about what was going on
in France or Spain or Germany (in French or Spanish or German no less), there
was likely a newsgroup meant just for that purpose. Usenet news also carried a
lot of text-encoded binary content, particularly in the alt.* (alternative)
newsgroup hierarchy.
The administrator of a Usenet server at each site arranged to “peer” with news servers at other sites, offering new articles from the server’s spool via a flooding protocol known as NNTP (Network News Transfer Protocol).
Because new articles were constantly coming in, and spool storage space is finite (or at least it was back then), old articles would be periodically expired (removed) to make room for new ones. Duplicates were prevented through the use of unique message-IDs, loops were prevented with path headers updated on a hop-by-hop basis, and transfers were optimized with a variety of techniques including parallel message streams and cyclical news file systems. All-in-all, Usenet was a pretty cool distributed information sharing environment, and one that still exists today. In fact, as of the start of this year, Usenet traffic volume had increased to the point where it was running over 17 TB per day*.
If you find yourself intrigued by the thought that there’s a part of the Internet you didn’t know existed, and you’d like to check it out, a list of 3rd party Usenet providers can be seen here.
Bottom line, if you’re a user, Usenet was (and is) a terrific place to discuss various topics. If you were a sysadmin responsible for running a Usenet news server, it was (and is) a terrific proving ground for high performance networking, disk-intensive I/O prototyping, and so on.
It also, unfortunately, tended to quickly motivate a personal and professional interest in anti-abuse efforts.
Usenet Abuse
The continued usability of Usenet has always really depended on the courtesy of its users. If a Usenet newsgroup was devoted to high energy physics, common sense dictates that articles about scuba diving or recent movies would be off-topic. Those discussions really should be held in a more appropriate newsgroup, instead.
Most users respected that, and were careful to post to appropriate newsgroups. Some, however, simply posted whatever they wanted, wherever they wanted, either because they were confused or careless, or had malicious intent, or simply wanted to blast their commercial message to everyone since the cost to them of doing so was effectively zero. These behaviors, if tolerated, could obviously cause problems for users and Usenet administrators alike: discussions would get derailed, users would complain to administrators, spool space might become exhausted prematurely, etc., etc., etc.
Kill files, implemented client-side in News reading client software, allowed users to selectively address minor irritations by automatically and silently suppressing the display of posts made by certain obnoxious people.
Larger-scale and more serious Usenet abuse issues were often handled with cancel messages. Cancel messages were originally envisioned as a way for an individual user to request deletion of one of their own posts (e.g., if you developed second thoughts about a message you might have posted accidentally or in the heat of the moment, you could send a cancel message and it would have your earlier posting deleted at many sites). However, because cancel messages weren’t cryptographically authenticated, cancel messages could be “forged” to delete ANY arbitrary message, regardless of who originated it, at least if the site receiving the cancel messages chose to honor/process those cancel messages.
This lead to some of the first “bot wars:” automatic spam bots (run by the bad guys) would post waves of garbage to Usenet newsgroups, while automatic cancel bots (run by the good guys) would rapidly clean up that mess by issuing cancels for those messages as quickly as they were posted.
Of course, the bad guys then tried to discourage sites from accepting cancels at all by issuing forged cancels for ALL articles in Usenet… and the good guys responded by “aliasing out” (systematically shunning) all traffic from the problematic servers that were trying to “cancel everything,” a classic “arms race.”
Spam wasn’t the only sort of problematic content in Usenet. Fraudulent scams of various sorts, for instance, were seen from time to time in Usenet just as they’re seen today in email, social media, and elsewhere.
And because text-encoded binaries were a material part of a typical feed, the potential for Usenet to act as a vector for malware distribution was also very real. Binary content also meant that copyright infringing content (such as pirated software, pirated music, and pirated movies) was another potentially problematic area, as was illegal online child sexual abuse material, although what was present in a given server’s spool would vary dramatically depending on what newsgroups your News admin elected to carry.
In spite of all these real or potential issues, Usenet resulted in a very rich and creative intellectual environment that facilitated a lot of productive work. It was a true community, and at least among some Usenet administrators, the impetus for an intense focus on system and network performance, and measurement work.
By way of example, one colleague at Unidata in Colorado evaluated using NNTP as a potential data distribution protocol for pushing binary weather data. She found that from a latency and article completeness perspective, NNTP rocked.
The peer-to-peer nature of the Usenet feed environment also resulted in many Usenet system administrators forming close collaborative relationships with each other. Relationships of that sort often formed the foundation for later abuse mitigation efforts, and were as important as technical advances.
How Does This Relate to Cyber Security Today, and Farsight?
Farsight is a cyber security data company. Multiple Farsight staff members have at least historical connections to Usenet, AND deep connections to the anti-abuse and security communities. Now, having read along to this point, you know a little bit about why those connections tend to exist.
You now also understand a little about why we care about data distribution technologies: whether folks were pushing Usenet articles in the old days or pushing cyber security data today, we want and need to quickly, efficiently and scalably move large quantities of data over the wire. This means we care about advanced networking, I/O optimization, data structures, data transfer protocols, securing data in flight and at rest, traffic measurement and analysis, etc.
We also count on our colleagues and friends to help us continue to win the fight against cyber crime and other online abuse. Just like personally-arranged Usenet newsfeeds, collaborative data-driven security only works if people agree to share. It’s your data, your contribution, your telemetry that you share with Farsight, that makes all the difference. We deeply appreciate your generosity and we literally couldn’t continue the fight against the bad guys without your help and your data. A big THANK YOU to all our sensor operators out there!
We’d also like to take this opportunity to remind researchers at accredited academic institutions that we welcome the chance to help support your work with full or partially underwritten grant access to Farsight Security’s data.
Looking Forward
We’re also curious: while we don’t currently work with Usenet data as a cyber security data source, is there data actually in Usenet traffic that you think would be helpful to cyber security and anti-abuse efforts? Is this an area where you’re currently lacking visibility, and need to fix that? After all, there’s probably something going on in over 17TB of data a day — wouldn’t it be nice to know what?
Anyhow, we always like to hear from our friends and customers. Please feel free to send along feedback.
* 17TB/day ==> 17,000,000 MB * 8 bits per byte / (24 hrs/day * 60 min/hr * 60 seconds per minute==> ~1.574 Gigabits/second, assuming traffic is uniform (which it isn’t) and there’s no duplication of incoming traffic (which there will be). See here for more information.
Joe St Sauver, Ph.D. is a Scientist with Farsight Security, Inc.
