Coming up this week on Breaking Badness. Today we discuss: further analysis conducted by a number of individuals of the SolarWinds supply chain incident and Two Truths and a Lie.
Here are a few highlights from each article we discussed:
Recent Incident Leaves Organizations to Twist in the SolarWinds
- This yarn ball is far from fully unraveled. Here’s how we first came to know about this:
- The news broke about this incursion as we know it on December 13. I phrase it that way because we now know what we didn’t know at first, which is that the FireEye breach, which we learned about earlier, was in fact related to SUNBURST as well.
- The first takeaways about this breach were that it was a Big Deal. At the very beginning of the SolarWinds part of the story (on the 13th) we knew that several government agencies including the Commerce and Treasury departments had been breached; but it took only a few hours before it was broadly known that many more agencies were involved, including the Pentagon, all branches of the US military, the State Department, the NSA, and more additional ones than I have time to list here. In fact, there seem to be only a couple of agencies that were *not* breached, as far as we know. And that’s just the US government. Many, many more entities were involved:
- 425 (at least) of the Fortune 500
- ALL of the top 10 telecom companies
- ALL of the top 5 accounting firms
- The Los Alamos national lab, as well as other labs
- Hundreds of universities and colleges
- Scores of local and state governmental agencies
- In fact, if there was any sort of silver lining here, and trust me, it’s not much, it’s that there was so much data available to the adversary that they could basically be drowning in it. But it stands to reason that any entity capable of pulling off this breach is also capable of figuring out where the really sensitive stuff is.
- I’d say there have been two major ways that this incident has evolved. The first is the scope of it. Remember the beginning of Star Wars Episode IV, A New Hope, where we see the rebel ship being pursued by the Star Destroyer? What makes that moment so impressive is how the destroyer keeps growing…and growing…and growing…you have these mini moments of thinking “oh, it’s that big…no, it’s THAT big…holy smokes, it’s THAT big…does this thing NEVER END?” That was the first way this has evolved. The second is that, as is expected with this sort of disclosure, we got a more focused picture of how the intrusion began—that is, the details of the modified package delivered by SolarWinds Orion.
- FirEye was the first entity to sound the alarm on this campaign about 10 days ago. We can assume that this was discovered based on telemetry with their own breach and those of their customers followed by code analysis from the software package that was beaconing. We don’t know the duration of FirEye’s analysis prior to the announcement—meaning, how long were they sitting on the analysis? Presumably, they want to ensure that whatever they put out is airtight. What is more of interest to me is the NSA aspect of things. NSA is the primary agency charged with protecting US government cyber assets. The fact that this was not released by NSA in advance is a strong indicator that they did not know about this, which implies that government agencies were fully affected.
- The primary command and control domain was first enabled in February, 2020. However the properties for the domain were modified in December 2019. The domain itself is much older, most likely originally registered by a party that is not affiliated with the attackers. If we look at the timeline, we can infer that planning for this campaign probably occurred no later than Q1 of 2019. So planning occurred in two parallel verticals: Modifying the code base, and selecting and preparing infrastructure. Of course this would have been run as an operation, with multiple moving parts.
- Reversing Labs tied the first instance of tampered software to a release that was dated as early as October 2019.
- The version did not contain the full back door, however it did instantiate some classes that would be used in the weaponized release, whose release date was March 2020. It is not known whether this release was a “trial run”, testing the waters, so to speak.
- The attackers very astutely wrote code that mimicked the style and syntax of Solarwinds developers that worked on the codebase in an effort to remain undiscovered. Additionally, much of the code itself runs in memory, and makes attempts to remove traces of itself in relevant logs.
- All of this indicates a slow and methodical approach and ramp up, with a strong focus on OpSec. e.g. the type of OpSec we would expect to see by a group of centrally managed professionals.
- There was some additional analysis performed by Volexity, I’d like to highlight a few conclusions including that they identified a group responsible for some specific breaches that used this supply chain attack as “Dark Halo.” This group is called UNC2452 by FireEye (UNC stands for Uncategorized), and they have been active for years now, including three major incidents that Volexity worked. So to explain this a bit, they did some response work for a think tank in late 2019 and early 2020 that was infected with, as they put it, “multiple tools, backdoors, and malware implants.” Volexity was able to initially kick them out of the network, but as we’re seeing here, these are actors who truly embody the idea of an advanced persistent threat, and they were able to get back into the think tank’s networks by exploiting a vulnerability in Microsoft’s Exchange control panel. One of their main goals seemed to be getting at the emails of senior personnel at the think tank, and to that end they did things like using novel techniques to bypass Duo’s two-factor authentication to get at an individual’s OWA (Outlook Web Access) account.
- I’m gonna pause the narrative here and make a 2021 prediction, even though that’s not the theme of this podcast. Books are going to be written about this activity (I don’t even know what to call it, since the “it” is evolving and growing so much), and not just the Andy Greenberg “Sandworm” kinds of books. This is because the operations of Dark Halo are literally textbook examples of how an advanced actor works.
- They lived off the land for years at the breached think tank. They used advanced exploits and tools only when they had to, most of the time just quietly operating on the victim networks like any other authorized user.
- The abuse of Duo’s 2FA is interesting and it carries a lesson for blue teamers, because Dark Halo did not actually exploit a vulnerability in Duo. Instead, they were able to bypass Duo entirely by including a cookie in their session that had been derived via Duo’s integration secret key. If they hadn’t been able to get that key, they wouldn’t have been able to bypass the 2FA in this way. One of the takeaways Volexity emphasizes about this is the importance of changing ALL passwords and keys after an intrusion, and not just changing them to something guessable by the attacker, since you have to assume that even when it appears that they’ve been eradicated from your network, they might not be. So if they know your password pattern and your next password is just a sequence of that pattern, guess what—they’re back in.
- This seems, and I want to emphasize *seems,* since we’re learning more pretty much by the hour, that Microsoft was not breached. Microsoft isn’t denying that there was an incursion, but they’re saying that it only involved some servers that were isolated from customers of services such as Office 365. If in fact it comes to light that customer services were involved, then the implications are staggering—think of the millions of businesses, agencies, and individuals who have sensitive data residing in Office 365. But again, I want to stress that right now we don’t have any evidence that O365 has been compromised. Reuters walked back the short, initial story they published suggesting that the breach could affect Microsoft customers.
- In terms of C2 nodes, a primary domain was used, and this domain was initially registered in 2018. However, the details around domain ownership seem to have been modified around late 2019. The “late 2019” modifications to the primary domain paces the development date of the initial code found by Reversing Labs. Looking at this holistically, it backs up the initial timeline assessment – We see code modifications, then a few months later we see modifications to existing domains that will be used for Command and Control.
- Researchers found about 10 other secondary C2 domains—they used generally innocuous naming conventions to appear as normal traffic from the infected SolarWinds server back out.
- One interesting note about the primary C2 domain: I was impressed by the speed and breadth of Microsoft’s seizure efforts in taking control of the domain. Sinkholing the domain will help defenders by allowing Microsoft to warn companies as they detect infected instances of SolarWinds that are trying to “phone home.” This effectively created a kill-switch, neutering the ability of infected SolarWinds packages from calling the mothership.
- These modifications show us that the attacker domains were selected carefully, with a goal of using domains that had already been established in an effort to avoid raising suspicion. Many threat intelligence platforms will factor “newness” into a security rating. Older, uninteresting domains are less likely to raise alarms.
- Domain hosting was primarily via cloud service providers, which allows an attacker to easily set up and tear down or move infrastructure.
- Another aspect of the SolarWinds Supply Chain attack is the ability to decode lists of victims after Prevasio published the algorithm publicly. This is one of the more interesting tidbits forensically speaking. Most listeners will be familiar with the concept that DNS can be abused for various kinds of malicious activities, far beyond just resolving domains used by adversaries. In this case, part of the C2 for SUNBURST involves DNS traffic where the subdomain part of the query string is an encoded blob representing the name of the victim entity that the query is coming from. It looks like gibberish on the wire, but it turns out to be a pretty basic substitution cipher—you could decode it with a paper decoder ring—which tells the threat actor who this query is coming from, evidently to help them prioritize their targeting. Looking for these random-seeming subdomain strings is one of the ways an organization can use DNS logging to help figure out if they’ve been compromised with this malware.
- Regarding the SUNBURST infection chain, generally speaking: The malicious update is downloaded via solarwinds, a check is performed to make sure that the host’s IP address is not a private (RFC 1918) address, or one in a narrow public range. Once checks pass, an outbound call is made to the primary C2 server, avsvmcloud[.]com. Then a CNAME response is returned using a DGA derived by using the hostname of the infected host. From there, the attacker can decide whether to further prosecute the target, whereby it is possible that command and control will occur via one of the secondary C2 servers, or the attacker may simply choose not to further prosecute the attack.
- As Tim pointed out, this attack cast a very wide net. Multiple industry and government verticals were affected. The attackers obviously knew the types of targets sets that were of greatest interest to them. So this particular attack type, and the nature of the infection chain, allowed them to keep the fish that they wanted, and throw back the ones they didn’t.
- There was initially a rush by some media outlets to attribute the attack to APT29, CozyBear, or Russia’s Foreign Intelligence Service (SVR). Many government entities are also blaming “APT29” for the attack, possibly as a catch-all for “Russian Government Sponsored Entity.” It is noteworthy that Fireye, Volexity, and Microsoft are attributing the attack to unknown entities, refraining from making such a strong conclusion. However, the general consensus seems to be that it is Russian. Certainly the FBI is working very diligently to determine true attribution. This includes technical investigation of TTPs, as well as investigation of SolarWinds personnel, especially those in a position of modifying the code base. This is all speculation, but one should never rule out insider compromise.
- Having been a vulnerability manager and penetration tester for a large company for several years, I would have said “Patch Patch Patch!”. However, it is almost perverse to say but organizations that failed to update Solarwinds would have avoided introducing the backdoor into their environments! That said, a “Defense in Depth” approach is almost always the best strategy. This means proper controls at the firewall, including SSL stripping. Internal segmentation between test, QA, production, CDE, and admin environments. Endpoint protection, to include FIM and DLP protection. A robust threat hunting team. And for goodness sake disable scripting languages on hosts if they’re not needed. But most importantly, select an appropriate security framework like the CIS 20 or ISO 27001/27002, and work to bring the organization to a place of maturity with those controls. Security frameworks such as these take a comprehensive approach, and help practitioners set up a defense in depth.
- The essentials of blocking and tackling apply, always. I’m going to depart from the letter of your question a little bit here before coming back to it, in order to address some of the IR practices that Volexity emphasized. Guess what their first recommendation was—look for traffic to the malicious domains and IPs used by SUNBURST. This thing is so extensive that the first order of business is to figure out if you’ve been affected, especially if you’re a SolarWinds shop. The forensics on this intrusion underscore how valuable DNS is to figuring out what’s going on with a known or suspected breach.
- As for guarding against future incursions or attacks like this, I 100% agree with everything Matt just said. I mean, literally EVERY major area of infosec best practices is involved here, from least privilege and zero trust to good password and key management to carefully developed network traffic monitoring. I was on a panel about threat hunting with SANS last week, and one of the provocative questions on the panel was “could the SolarWinds breach have been prevented by threat hunting. There was some variation in opinions on that; some of the panelists said no, but I said yes. My reasoning is NOT to throw any victim under the bus here, because the spirit of my answer was more theoretical than practical, but it morphs into practical. What I mean is that we’re going to develop new ways to hunt based on what we’re learning about SUNBURST, and those methods may allow us to detect similar incursions earlier in the future. In this case, for example, if you had some regex in place that alerted on weird DNS queries, which is a very attainable goal, you might have flagged some of the early stage C2 where the actor was identifying victims. If I sound like a broken record, call it a broken A record, since DNS offers so many avenues both to the attacker and the defender.
Two Truths and a Lie
Introducing our newest segment on Breaking Badness. We are going to play a game you are all likely familiar with called two truths and a lie, with a fun twist. Each week, one us with come prepared with three article titles, two of which are real, and one is, you guessed it, A LIE.
You’ll have to tune in to find out!
Current Scoreboard
This Week’s Hoodie/Goodie Scale
Recent Incident Leaves Organizations to Twist in the SolarWinds
[Tim]: 10/10 Hoodies
[Matt]: 10/10 Hoodies
That’s about all we have for this week, you can find us on Twitter @domaintools, all of the articles mentioned in our podcast will always be included on our podcast recap. Catch us Wednesdays at 9 AM Pacific time when we publish our next podcast and blog.
*A special thanks to John Roderick for our incredible podcast music!
