Podcast audio-only versions of weekly webcasts from Antisyphon Training
Patterson, are you ready to get started in teaching us rapid endpoint investigations?
Patterson Cake:I am. I'm so ready.
Zach Hill:You're so ready. You're always ready, man. You're you're like a OG at this. Everybody loves you.
Patterson Cake:Let's hope.
Zach Hill:No. It's gonna go great. I will go ahead and head to the backstage, let you get started. I'll come back when there's, you know, five, ten minutes left or whenever you're wrapping up. We'll go over some q and a.
Zach Hill:But I hope you all have a great session here, and we'll see you all in a little bit. Good luck. Have fun.
Patterson Cake:Thank you, Zach. Welcome. Thank you so much for being here today. I I I often start these sessions with a with a story, and so I thought I'd mix it up just a little bit today, especially following that pre show banter. I feel like we need to, like, raise the IQ a little bit, throw in a little maybe culture and sophistication, and so so I wanna read a little little poem to you, and maybe this will be a new thing for Anticasts, we'll see.
Patterson Cake:This is the first one that came to mind, of course, because April is the cruelest month. Having said that, it's over 400 lines long, so it's a little too lengthy for today. So pulled in a slightly different classic, and I'll read this to you ever so quickly, just to level set. Every OS wastes your time from the desktop to the lap. Everything since Apple DOS, just a bunch of crap.
Patterson Cake:Now there's Linux or Linux. I don't know how to say it or how you install it or use it or play it or where you download it or what programs run, but Linux or Linux don't look like much fun. However you say it, it's getting great press, though how it survives is anyone's guess. If you ask me, it's a great big mess for elitist nerdy schmucks. It's free, they say, if you can get it to run, the geeks say, hey, that's half the fun.
Patterson Cake:Yeah, but I got a girlfriend and things to get done. The Linux OS sucks. I'm sorry, but it does. So potentially, may be familiar with three dead trolls in a baggie. If you are, you're probably old like me.
Patterson Cake:The the level set, of course, is that operating systems are challenging, and today we're gonna talk about Nix. Thank you for your patience. We got about an hour, and that's not nearly enough time to talk about the Knicks universe, so I'm gonna do my best to hit the high points. I do have a four hour workshop coming up next Friday to dig a little deeper and actually put hands on keyboard for some of these things, but today we're just gonna jam pack an hour's worth of fun and games, and at the end, of course, there will be some external resources, so you don't have to take all the notes, capture all the things. Ultimately, again, I'll leave you with some takeaways that you can follow-up with later.
Patterson Cake:I like to start with the context and presuppositions. If you don't know me, I am the Director of Incident Response for Black Hills Information Security, and that's important because it is the context from which I approach almost all of my tactical operations, and what I mean by that is I do incident response for a living, and so my day job is helping businesses respond to and recover from active incidents. That is, again, the stage, if you will, for everything that follows. My first career was carpentry, honestly. My father was a carpenter and I grew up swinging a hammer, and my dad used to say, Strive for perfection and come within a quarter inch.
Patterson Cake:Strive for perfection and come within a quarter inch. And his point was, put the appropriate amount of effort into this for the desired outcome, and so close enough for horseshoes and hand grenades is my paraphrase for this does not need to be perfect. Do not let the perfect be the enemy of the good. We need to accomplish a task in the context of active IR, and that's what we're here to talk about today. Not all the finer, nuanced details of NICS as a whole, but how do we have simple, effective, repeatable process for investigating NICS endpoints.
Patterson Cake:I have sort of this same preamble for a lot of my rapid endpoint investigation conversations, and this is me trying to simplify the universe, honestly, and the vast majority of cases that we work now and into the future center around endpoint and identity, and I'll talk a little bit more about what I mean by endpoint and identity as we progress. Endpoints in general, of course, where human beings interact with technology, like it or not, that is a critical attack service today, tomorrow, and again into the foreseeable future. When we talk about endpoints, we talk about a handful of critical components, and legitimately, we could be talking about a phone, a tablet, a PC, a switch, a router, etcetera. Ultimately, these are the critical components of an endpoint for today's context. Operating system, operating system, probably most important for today's context.
Patterson Cake:I put a little star by network attached, because that's a critical sort of force multiplier. If you have an endpoint sitting in the corner not attached to the network, chances are I don't care about it, you don't care about it. The minute it's attached to my overarching environment, it increases in importance. Every time I do this slide, think about a healthcare investigation we did for a while, and intermittently we'd get these high severity AV EDR alerts, and then they would just vanish and disappear, and then we would go to investigate this host, and we couldn't find it in the environment. It was just no longer there.
Patterson Cake:This would happen once a week, twice a week. Finally, we partnered with biomed, and we literally did physical investigations of the environment, and we found this computer in the corner. And we asked the biomed tech, Well, what is that? So, Well, that's our CR, it's attached to our computer, our x-ray systems, and it's got a virus on it, so we only plug it into the network when we have a patient. Sorry, I can't help myself but think about that particular moment when it's sitting in the corner detached, I don't care, and the minute it's attached to the network, and of course, the malware begins to propagate, we care a lot more.
Patterson Cake:Today, again, we're focusing on some specific operating system components. This is the core. Honestly, this is the core of our investigation. If you go back and you read the entirety of today's poem, it makes a really, really germane point and that is this is really not about the hardware. Hardware is an important underlying part, but it's really not about the hardware.
Patterson Cake:It's really about the operating system, the user interface, apps, data and services. And we need to be aware of the differences, similarities, features and functions of operating systems, and you know, necessarily we spend a lot of time talking about Windows. And that's reasonable for market share, for vulnerabilities, for attack surface, etcetera. That is often the focus. I was thinking last night, coming up with fictitious statistics in my mind, and I'm thinking somewhere around one in 51 in a 100 endpoint investigations that we do typically is not Windows.
Patterson Cake:The vast majority are Windows. Having said that, obviously there is significant footprint for Linux, especially less so for Mac, but this needs to be in our wheelhouse, in my humble opinion, and so we need to spend a little bit of time thinking about the propagation, the prevalence, the use case. Use case is critical. It is critical in our understanding of how these systems are used in a typical business environment, and of course, that will drive our investigation, so Windows, still top of the list for general deployment, for number of vulnerabilities, for frequency of impact, but having said that, we will see that Linux is monstrous from an entirely different use case, and then Mac falls somewhere towards the bottom of the list, but still extremely relevant, and I think growing in relevance from my perspective and experience. So we're gonna start today talking about Linux.
Patterson Cake:And the Linux use cases, as mentioned, are distinctly different in many ways, at least statistically proportionately from Windows, and we see that Linux is, well, the infrastructure. Right? It is the Internet. It is the supercomputers of the planet of this day and age, and just did a super simple Shodan query, show me OSs that are not Windows, and obviously you can see we have a lot, a lot, a lot of Linux in terms of the infrastructure that is running the planet. It's important, and maybe this is obvious, maybe not, it's important that we keep in mind that typically we don't have a user sitting down using a Linux system for their daily workflow, typically.
Patterson Cake:That does not mean that does not happen, it does not mean that is not a positive trajectory, but generally speaking, the footprint, the use case is what we see on this screen. And so again, this changes our perspective. One of the things we're most interested in on a Windows endpoint, in many situations, investigations, is web artifacts. What did the user do? Where did they browse to?
Patterson Cake:What did they download? That is not the same use case we see on Linux, typically. And those kinds of observations can and should inform our endpoint investigations specifically. I have mixed feelings honestly about which is easier. In some ways Linux investigations are much simpler than Windows investigations.
Patterson Cake:In other ways, we're often less familiar with Linux, and because Linux is so customizable, you may find unique situations and circumstance, so it's a little bit of a give and take, but long story short, obviously there are tons of distros, lots of different flavors, and I'm lumping them all into one large category, and I'm here to tell you that in the vast majority of instances, legitimately, I can only think of one case in recent memory where we had to do something special to investigate a custom Linux distribution. In the vast majority of cases, the things we're going to talk about today, they apply, even though there's lots of variability, lots of complexity. If you've been to my Endpoint Investigations classes before, or discussions before, once again, I'm trying to simplify. There's way too much here, way too much going on on any typical operating system, millions and millions and millions of lines of code, average of a couple 100,000 man pages, just as an example, potentially thousands of configuration files, boils down to, I don't even know how to quantify. I made up those numbers.
Patterson Cake:One to 5,000,000 forensic artifacts, depends on how you quantify that. Long story short, in a previous discussion, we talked about searching for a needle in a stack of needles, and this is way, way too many needles to wade through in the midst of active compromise. Remember, that is our context. So what are we gonna do about it? We're gonna need to find a way to simplify.
Patterson Cake:I like to throw around the word rapid triage, and by triage I mean we have limited resource, that's time, money, expertise, etcetera. We're in a bit of a hurry because we're responding to active compromise. We need to figure out how to use the resources we have for maximum effect in the midst of active IR. Our plan is to take everything that we just talked about and distill it into a finite set of the prioritized, the most important artifacts so that we can begin our analysis here. Now, it may not end here, right?
Patterson Cake:My goal, and what we will do shortly, is to take the vast majority of these artifacts, boil them down into the most critical components, the most critical categories, and we're going to come up with a list of 10. This is where we're going to start. This is where we're going to invest our resources for maximum effect. Again, you may need to come back later and look at something else, But this is going to drive our investigation, drive our response, drive our prioritized response. I'm here to tell you that the threat actors have standard operating procedure, and I am super, super grateful for that, because that makes them in many ways predictable.
Patterson Cake:Predictable. So this is my short summary of Threat Actor Standard Operating Procedure when it comes to Linux, and I'm talking based on recurrence in the wild over the last six months, year, two years, etcetera, and there are consistent patterns. Now, they do vary somewhat slightly, but only slightly, and so we can start with some perspective on what are we looking for, investigating active compromise on a Linux or NixBox in general, but Linux specifically. Initial access is one of three choices, SSH, SSH, or SSH. Legitimately, almost always.
Patterson Cake:You you know, know because of who you are and what you do, and because you're here today, you create a cloud based instance, a Linux based instance, and you spin it up in AWS or GCP or whatever, DigitalOcean, Linode, how do you get to it from your current location? Well, SSH. And so this is the attack vector of choice, absolutely, completely, and of course, there's a network communications component, whether that's command and control, etcetera, we'll talk a little bit more about that in just a second. Fortunately, it's pretty plain Jane lately. It's straight up TCPIP, HTTP, HTTPS, because why not?
Patterson Cake:It works, and it's simple. The detection of agent is not as much of a thing as it is on Windows. There is potentially, I would say there is a growing proportion of Linux systems that have AVEDR running, but it is not nearly as much of a component as it is on Windows, so maybe not applicable. What I see, and this is evasion, is the doppelganger thing, and what I mean by that is just naming something cleverly so it's hidden in plain sight. One of the favorites, and I think I may have it in a screenshot, is a recent case where they renamed DHCP to DHPC, just as an example of doppelgangers to hide a malicious service, malicious executable, so you might not notice.
Patterson Cake:The next one, actions and objectives, threat actor pops a Linux box, why are they there, what are they trying to accomplish, and recently, mostly, it is to abuse your resources, abuse your resources, and that comes in a couple different flavors. Often it's crypto mining, often it's crypto mining in conjunction with looking for other vulnerable Linux systems. Outbound SSH attacks are huge, the box gets popped, begin to do some crypto mining, use up all of your critical system resources, maybe make a little money, and while they're at it, your systems to attack other systems and propagate this life cycle. And how do they accomplish persistence? Unsurprisingly, if you're not familiar with Cron, we'll talk a little bit about that.
Patterson Cake:It's just scheduled tasks effectively. Chronological table on Linux for recurrent execution, persistence for configuration changes, and then often again SSH for ongoing action. Threat Actor SOP, and I'm talking like 98% of what we're seeing currently in the wild with a very, very small delta. This is important, this is philosophical for just a second. I think we focus way too much, often, on the cause, on the cause.
Patterson Cake:We like to give threat actors clever names, Fuzzy Panda, whatever. We like to give malware clever names, whether it's Sock Gullish or Raspberry Robin or Lemon Duck or I don't care. I don't care. I think we need to focus, and I am consistently focusing on the effect, not the cause. The cause is not irrelevant, but what I care most about is the impact in my environment.
Patterson Cake:For today's conversation, the effect is the endpoint attack surface. If we know this is what the threat actor is likely to do, then I wanna know how to see this, how to find it, how to identify it discreetly. Much more concerned about the effect, the impact, than I am on the overarching cause. So let's get practical. We got tons of information, tons of code, a complex operating system, we know how the threat actors behave.
Patterson Cake:We're gonna take those things and map them high level categorically to one of four primary components on our endpoint, memory, identity, network, and disk, and distill that into how are the threat actor behaviors, malware behaviors, going to manifest on my operating system, on my Linux operating system? And the answer is, I am most likely to find them in running processes, network sockets, world writable directories, scheduled tasks, AKA cron, running services, and finally SSH, SSH config. Important context, you can stare at this later, this is the overarching, again, rapid triage workflow. What questions are you trying to answer? This is so important.
Patterson Cake:It will drive everything that follows. Once we define that, what artifacts will help us to discreetly answer those questions, and then last but not least, we need a way to collect, parse, and then reduce and refine so that we can derive actionable intelligence, What I mean by actionable intelligence is usually looking for discrete indicators of compromise, something that we can use from our investigation to further harden, contain, eradicate a threat, whether that's a file name or a file hash or an IP address, etcetera, actionable intelligence. I'm gonna show you in just a second the artifacts that I think are most important. We have to have a way to acquire those. We have to have a way to process those so that we can analyze them.
Patterson Cake:I'll feed you that momentarily. Ultimately, again, identification of indicators of compromise so that we can harden our environment and contain, eradicate the threat. The 10 artifacts that I mentioned a little bit ago, relatively self explanatory based on what we've seen and discussed so far. These are the artifacts that we are looking for, the top 10, if you will. My favorite artifact, honestly, is running process.
Patterson Cake:Running process with command line. If I get to start somewhere, especially with an active engagement, that's priority number one for me. Network sockets, of course. Chances are the threat actors are not physically in your building, in your facility, sitting with hands on keyboard, so there's network communications involved. Shell history is a fantastic way just to learn a bit contextually about what a threat actor has done on a particular endpoint.
Patterson Cake:Config files, crontabs, scheduled tasks, services, executable files. One of my favorite things to do to interrogate on an operating system on an endpoint is show me all newly created executable files in world writable directories. Hugely, hugely beneficial from an active investigation. Since we know SSH is likely the inroad, we need to take a look at auth. We need to take a look at logging access, etcetera.
Patterson Cake:As mentioned, ties into executable files, writable executable directories, and I would really like to have hash values associated with those, if possible. One fascinating thing that is a really sort of tangential artifact is we see the threat actors, and usually if they're popping a box via SSH, it's weak creds, right? They guessed a lousy password. The fascinating artifact immediately thereafter, because the threat actor wants to take advantage of your resources, the next thing they do is they harden your box for you, which is again a fascinating artifact. They will usually change the password, they will usually instantiate key based access to SSH, so it's again a fascinating thing to evaluate.
Patterson Cake:If you see SSH off access that's unauthorized, often immediately it's followed by the threat actor performing your due diligence for you. So how are we gonna grab these? Well, lots of different options, and again, I'm a huge fan of the aforementioned Theodore Roosevelt quote of, Do what you can with what you have where you are. You don't have to do it my way, this is just a way. So we're gonna talk about a couple different solutions, specifically Velociraptor offline.
Patterson Cake:If you've heard me talk before, you know I like it. For the Linux components, super, super useful. We're gonna talk about UAC in the context of Mac, so we'll touch base on that in just a minute. I like the Velociraptor offline piece because it fits into my overarching general workflow and some distinct advantages. So we'll talk about collection process.
Patterson Cake:Parsing, I don't have to do a lot of parsing, a lot of processing. However, we're gonna grab still a whole ton of data, and so I'm gonna introduce you to a script that I created to take the Velociraptor CAT scale output and simplify that, And then we're gonna use the the three greatest incident response investigative tools, which of course are Excel, Notepad, and GREP for maximum effect. As mentioned, don't panic, don't take mad notes necessarily. I'll give you a link to this. This is all documented in my GitHub repos, step by step by step.
Patterson Cake:We don't have a lot of time today, so I'm going through it quickly. You can circle back, review this, test it, customize it for your environment, and make it your own. So what I'm gonna do, I am going to leverage Velociraptor offline collector. I am not going to leverage Velociraptor like the developers want or wish that I did. I am going to use this one discrete component, the offline collector, because it's easy, it's scalable, it's functional, it is compatible, I don't need any infrastructure, it works.
Patterson Cake:And so I'm gonna build a collector, I'm gonna use a collection of scripts called Catscale, which we can easily import into Velociraptor. You can run Catscale independently, but it works very nicely with the collector itself. I'm gonna select a handful of additional discrete indicators. The reason I like this is the output is an MUSL file, a C standard library that works on almost every Linux endpoint. Almost every.
Patterson Cake:Not every, but almost every. We're going to build this collector. We can do it super quickly. We can pre stage it in advance. We can test it in advance.
Patterson Cake:It can just be sitting there ready to go. When we execute the collector, it will grab the artifacts that we specify, create a zip file locally on the file system, and if we want, we can also do automated upload to secure cloud storage, which is my favorite, upload to S3, for example, do our analysis in EC2, really slick, super quick, I don't need a Velociraptor server, I don't need VPNs, I don't need any infrastructure really, other than to create this executable in advance, test it, be ready to go. We can do it one to one, there is an expect script on my GitHub repo, so that we can do this one to many. If you have the ability to authenticate to a bunch of Linux systems, you can do a script driven execution of the collector on ten, twenty, a 100 systems at a time. Pretty slick, scalable, the price is right.
Patterson Cake:These are the things I want. As I'm building my collector, these are the things I want, and again, forgive me, we're cramming an awful lot into an hour. You can reference the GitHub repo, I will share that with you in just a minute, and it'll step you through how to do this. These are fairly self explanatory. These are the things that I want.
Patterson Cake:I want Netstat, thank you very much. I want running process, AKA PS list. I want Bash history. It also incorporates other common shell history like ZSH, etcetera. I want crontab, which is scheduled tasks, more or less.
Patterson Cake:I want last user login because it's a very discreet show me who authed last, why not? I wanna find SSH login logs if I can, not always standardized and accessible, but if so, for the win. Then CAT Scale is this clump, it is this large section of additional forensic components. There is some duplication, and I did that on purpose, there's some duplication, like CAT Scale collects, you know, running process and Netstat, although not using the Netstat tool, and there's method to my madness, and you'll see that in terms of the scripted output in just a minute, but I'm gonna go in once and I'm gonna grab a little more than I think I need, because then it's easier than going back and getting a second set of triage data, to speak. Categorically, these are the things I'm going for.
Patterson Cake:From an analysis workflow perspective, these are the pieces and parts that I care most about. You remember my slide several slides ago, we took 50,000,000 lines of code, standard operating procedure, broke them down into memory, identity, network, and disk, and then basically had five categories. Well, here they are. This is what I'm going to prioritize for my analysis workflow. I'm gonna build my collector, I'm gonna execute my collector, I'm gonna either grab that data off disk or preferentially have it upload to secure cloud storage, I'm going to create a script because I got too much stuff still, I'm gonna create a script and I'm gonna extract these five categories for my initial analysis, and this is the, again, the rapid triage process.
Patterson Cake:What this looks like is this. This is on my GitHub repo, RTW, VR, Linux, CAT scale. This is a simple script, and all it does, we put the script in the same directory by default as my triage data zip file. It's going to extract the zip file, which is a little bit cumbersome because there are multiple nested components of zip and tar and gzip, so it's a little clunky if you have to do it manually. The script takes care of that, And then it also goes through the zip file process and pre stages these files specifically for my immediate analysis.
Patterson Cake:And again, we're talking about a quick look at overarching scheduled tasks, a quick look at established connections from a netstat perspective, those are ongoing active connections at the time of this collection. The PS list unique hash list, which I'll show you in just a second. These are all the running processes, I'm going to grab the running processes, and then I'm going to prioritize my analysis by calculating the unique entries, a hash value, and then we'll take a look at a quick way to potentially identify the unauthorized components, and now again, the off success for SSH. I'm gonna try and show you this really quickly, and you know how fun and scary demos are, right? So bear with me.
Patterson Cake:So here I have staged the, this is the output from running my collector, and again, upload it, copy it, whatever you need to do. The output is a zip file named for the host name with a date and timestamp, which I've removed for brevity, and then I'm gonna put this very basic script in the same folder as my zip file. It will process Wow, I made up two words at the same time. You know what I mean? It will parse and process multiple zip files if you have them all in that folder.
Patterson Cake:Then I'm gonna execute my script and it should take, oh, seconds. And if all of those according to plan, Work with me. It's going to extract the zip file, and it's also going to search through this bevy of triage data for me, and pre populate these five files. There's a six file here for a little more deep analysis, but it pre stages these. If you take a look at this, there's a whole bunch of stuff here.
Patterson Cake:Lots of good information, still a bit too much. So the idea of course is that the script extracts, prioritizes, and then I can immediately jump into what I wanna look at next. Well, truthfully, I usually wanna look at running process, and this is part of what the script does. We have this verbose PS list, over 2,000 lines, super useful, super important information, too much for me to stare at. So the script is gonna say, okay, stop, simplify it, I can go back to the discrete JSON should I want to, or I can just extract the unique entries, unique entries with hash, and knowing what I know about ThreatActor standard operating procedure and typical Linux configuration, root cannot SSH by default.
Patterson Cake:And in most instances, the threat actors are popping some other account. So legit, all this does is stack these running components, running processes, entry with hash by user ID. And this is an actual example of a compromised box, and right out of the gate, knowing just what described, we can prioritize review of the non admin accounts for interactive users, and sure enough, out of these handful of entries, these eight discrete entries, three of those are malicious. This is again, just prioritized analysis. We can do some fancy footwork.
Patterson Cake:If you want, I have a script for taking those and submitting those via API to VirusTotal, you so can cycle through this and go, what are those? If you wanna Google those later, those are related specifically to crypto mining and SSH outbound attacks. That's just one example, I don't have time to go through all of the examples, but this is your stage data. This is the output of the discrete workflow that we've just described, fully documented on the GitHub repo, so take a look, and this is the idea. Pretty confident, pretty confident that if you analyze those five discrete files, you will come out the other side with a clear understanding of, Have you been compromised?
Patterson Cake:Yes or no. Not the end of your Linux Endpoint investigation, but a fantastic starting point Let's talk let's talk a little bit about Mac. MACs are weird. Nobody likes to do MAC forensics. It's one of the rare things I've encountered many DFIR consultants, DFIR professionals, people that I respect, who say, I don't do MAC forensics.
Patterson Cake:I'm like, wait, is that an option? I don't do them. Just forget it. Yeah. And I understand.
Patterson Cake:They are a unique beast. Now, it is, you know, theoretically, well, theoretically, it is actually, of course, sort of an underlying Linux variation, if you will. But then there's this additional complexity layered on top from a Mac perspective. A lot of that complexity is really good from a security perspective, if you will, but it makes our lives really challenging from an investigations perspective. There's also this unique component from my experience, about MAC investigations, the sort of threat actor or abuse use cases.
Patterson Cake:They're changing a little bit over time, but the vast majority of them are a particular type and a particular flavor, and we'll talk about that in just a sec. One thing that's really fascinating about this is that I basically purport that there are kind of two operating systems that we need to be aware of, Windows and NICS, and ironically, Windows is moving a little bit towards Linux all the time, and Mac seems to be moving a little bit away from Linux all the time, so maybe I'll have to revisit this in a year and say, yeah, okay, sorry, there are three operating systems. I don't know, we'll see. Limited use case. Limited use case.
Patterson Cake:I have not looked at a Mac server, a Mac OS server, well, ever. I've encountered them, but I don't think I've ever done an investigation, but quite a few investigations on Mac laptops in corporate environments, and there are a few features functions, Threat Actor SOP, if you will. Full disclosure, the vast majority of cases that I have worked on a Mac were insider threat. They were legit investigating the Mac user themselves, the Mac user behavior. Fascinatingly, in many instances, those are IT people, those are tech people, and in many instances, they have tried to cover their tracks.
Patterson Cake:In many instances, they've wiped the entirety of the drive, which is, again, really challenging, really fascinating. Having said that, we have seen very recent examples, oh Axios, if you will, of compromise, of abuse of things like AppleScript, etcetera, to compromise the endpoint. This is newish in my experience, and so it does, in some ways, drive our investigations, but these are the things I think we should care about and that we should specifically look at, and for the rest of this conversation, I wanna focus on things that are a little different than the previous conversation, a little different than NICS, Linux specifically. Initial compromise, we do see initial compromise through web components, through web lures, malvertising, etcetera, and then of course we do often see the insider threat piece. Actions on objective, you know, info theft has been the latest from a malware perspective, and often that ties directly into sort of insider threat behaviors, so visibility into web, to disk, to file access behaviors.
Patterson Cake:In terms of TTPs, this is kind of a short list, to be honest, but in recent memory, we do see the threat actors sort of living off the land, or living off the orchard, lube ins, that was brand new to me recently, I think that's kind of a stretch, living off the orchard binaries, but you get it. So this again, informs what comes next, and what comes next, I wanna talk, how is Mac a little different than Linux? You'll recognize these, These are the artifact categories. I'll spare you the diatribe, because the aforementioned workflow and analysis approach, works perfectly for Mac and Windows, honestly. I care about running processes greatly.
Patterson Cake:I wanna know about network sockets. I'm interested in shell history. It may or may not be relevant. Generally, ZSH on current iterations of Mac, but Bash is an option, so I kinda wanna pull both, just in case. Same thing with general Linux configuration files, scheduled tasks, services.
Patterson Cake:And then there's this weird final category, and this is different, very different from typical Windows Linux, and it's this private portion of sensitive data that is sequestered on modern Mac iterations, and most of what we're talking about today has been tested and documented on Tahoe. I have a GitHub link for you at the end, but this is weird. Most situations, Windows and Linux, you just need root or admin, not so much on Mac. If you want to get all of these artifacts, specifically the private artifacts, you're gonna need admin level and full disk access. If you don't have full disk access, you can still go through this workflow process, but you're not gonna get the private parts.
Patterson Cake:Moving right along. The web history, which is critical, the unified log, which we'll talk about, some other log components, we want the private parts, there's just no way around it. So this is an important consideration as you attempt to execute and extract information from the Mac, full disk access required, and depending on how you're running the collector, can be a little different. Remote login is slightly different than terminal, is slightly different than UI, so pay attention to that. Some notes on the GitHub repo.
Patterson Cake:I'm gonna stop using Velociraptor at this point when it comes to Mac. You can use UAC. UAC is an available triage package on Velociraptor. The problem is that the executable is troublesome. We grab the VR executable, Velociraptor executable, and try and build or collect on Mac, a Mac is not happy about it.
Patterson Cake:The security components say, sorry, you can't run that. And then sort of poking holes in the security on the Mac to get that executable to run is just a non starter for me. You might be able to do that in your environment, but as a consultant, it's just, I can't do that. I can't have you turn off this and this and this and this to get this executable to run, so I pivot to UAC, and UAC is largely YAML files and associated scripts. I say largely because you can sort of bring in your own external customized executable content, so you could be leveraging executables, but I typically do not.
Patterson Cake:Very compatible. This is what I would fall back on if I don't wanna use Catscale and VR on Linux. Works beautifully. Non executable, same sort of features and functions, local zip file creation, potential upload to secure storage in the cloud if you want to, pre stage it, pre test it, pre deploy it should you want to, execute as root, make sure you have full disk permissions if you want the private parts, and we're ready to go. Relatively simple, you're gonna acquire UAC, you're gonna transfer the entirety of the UAC zip to your target environment, you're gonna extract it, you're gonna execute it specifying an investigation profile.
Patterson Cake:One more time, there's the short link, I'll share the link at the end, check out the GitHub repo, because you can customize all these pieces and parts, and be ready to go. I've pre staged a script for you, should you want to use it, that will unzip, execute, configure for upload, and much like our previous forays, we're ready to go. As mentioned, don't have a ton of time today, so I wanna talk about some critical components. Running processes is honestly in network sockets, they're pretty much ubiquitous, whether it's Windows, NICs, general, Mac specifically. One of my favorite parts of the UAC output is the live response, and you'll see on the screen, the upper right hand corner, those are the top level categories of output, and so live response is a fantastic place to go to immediately view things like running process, network sockets, super useful.
Patterson Cake:You can drill down into the root folder and see a bunch of file system components, config components, etcetera. You'll notice if you drill down into root, there's this private section, and this is, at least in part, where we're gonna find some of the critical sort of hidden components like the unified log, and then we'll find the web components under the user subdirectory specific to the user one more time, we have to have that full disk access or we don't get there, we don't see this. So I wanna talk about three sort of critical components of this output. And first, let's talk about web artifacts. And generally speaking, this is Safari.
Patterson Cake:Obviously, you could have other browsers running on your Mac, but we'll focus on Safari because it is default, de facto standard. These are the file paths. This is where you want to go to investigate the things. And really, there are kind of two, three primary things I wanna look at. History itemshistory visit, which is part of the history.
Patterson Cake:Db file, it's a SQLite database, so we need to have a tool or a way to sort of query this. I'll show you an option in just a second. So this is gonna tell me basically browser history, right? Then the next component is the downloads. Plist, and this is a property list, which is a uniquely Mac sort of a thing, property list, it's not a discreetly text readable file, but we can cheat on the next slide and sort of enumerate this.
Patterson Cake:The downloads are not tracked in the history. Db, so that's why we need these kind of two things. Where do they go? A little bit of information about the sites they visited, and then I wanna pivot to the downloads. Plist to see did they download and execute any software?
Patterson Cake:And then the last one's sort of a cheat. It's the standard downloads directory, so why not just take a quick peek at the file system components and see if they downloaded and executed something? There is a tool called iBackup Viewer, and if you're gonna do your analysis on Windows, then you can download iBackup Viewer for free. This is not an endorsement of this product. Be careful if you search iBackup Viewer, because there's lots of bogus links, you might download the wrong thing, so be careful.
Patterson Cake:But this is actually kind of a cool utility that will allow you to browse SQL and immediately view plist entries, which is a rare combination that's not designed for this process, but I stumbled across it and it allows me to sort of interrogate these files directly, so you might wanna check that out by Backup Viewer. This is DB Browser. This should be fairly familiar territory if you've done browser analysis, it's nothing fancy special. The kicker is, again, that this gives us some information about the visit history, the history items specifically, and then we can Here I'm just cheating in the lower right hand corner and using another really cool useful forensic tool called strings, and I am reading the downloads. Plist instead of using iBackupViewer or something.
Patterson Cake:I'm just gonna nuke it with strings, and I can see it's not pretty. It's kinda actually gross, but I can look at that and go, okay, they've been to Avast, they download Avast security online, and it's just it's down and dirty, to be honest. The iBackup PLIST viewer is better, but this is just, why not? It's already in my WSL, do a strings query, and kinda get some immediate correlation between the two. We gotta talk about Apple Unified Log, and I really don't wanna talk about the Apple Unified Log, if I'm honest, because I don't find a lot of value in it.
Patterson Cake:It's supposed to be all the things, right? It's supposed to be all It's the promise of unified log, I'm still waiting, all the things that you care about, except for it doesn't really seem to be, from my perspective, for driving analysis, especially for active compromise, but you need to know, you need to know that this thing exists, and the idea is that it concatenates all the important things, including SSH authentication, which are really hard to parse and really hard to derive value from. These are some of the things, the components of the unified log, you need to have full disk access to grab these things, and then when you grab them, you need to parse them somehow, some way. If you're using a Windows based system, then I think the best solution at the moment is the unified log iterator created and published by Mandiant. Thank you very much.
Patterson Cake:This will take the unified log output from UAC and parse it and create a monumental CSV file of all of the log items. Again, I don't find this super useful, to be frank, but you need to know it's there, you need to know how to extract it, and then once you have it in this format, you can always do the grep thing. You can do your other analysis, and then go back and search through this data for keywords, key indicators, etcetera. Document it on the Git Hub repo, link there, of course, to Manu, it's right up. It is a useful tool, and I think, again, you need to know, you need to have that in your toolbox.
Patterson Cake:We're almost done. Hang with me, thank you. Firehose, but that's the way we like it. And somebody's thinking, if you hadn't wasted so much time on poetry, it would be done by now. Other other artifacts.
Patterson Cake:This is a UAC output component. You've heard of this before. This is a body file. This is an attempt to take all of the things and put them in a timeline for your analysis, and it is crazy verbose. Crazy verbose.
Patterson Cake:On a typical Mac 4A, this is like six, seven, 800 megabytes of text file, of CSV output, which is huge, way too much. So I like it in concept, I do not like it in terms of rapid endpoint investigation. So what I have provided here is my approach, which is take the body file, convert it to plaza output, use p source to slice it. And again, you can reference this through the GitHub repo, through the slide deck, all that really does is say, okay, I found evil, I found evil, and it happened around 01:30 a. M.
Patterson Cake:On Tuesday, so instead of looking at this gargantuan output, show me what happened thirty minutes before, and thirty minutes after this critical moment. This I find really useful. Even that output is still extremely verbose, but it's a way to say, Okay, I already found something, so focus in on this particular point, this initial access, or something like that. There are a couple requirements here. I do this through WSL on my Windows analyst systems, need to have the plaza tools installed, this is pretty effective, again, just to extract a critical component of that body file overarching timeline.
Patterson Cake:I like Grip a lot. I don't know how I would survive without WSL, Windows Subsystem for Linux. Apologies if I didn't say that before. WSL is the bomb, and I use it largely for searching, formatting, structuring, unstructured textual data. Now this is me saying, Okay, I'm on a mission.
Patterson Cake:I'm investigating some thing, and I've got this pretty large output from UAC, and well, I like my colors. I like this format. If you just grep, recurse, search for the things, you get this kind of monochromatic output, and and I don't like it. I like this, it's a little complicated, you don't have to use this full syntax, but basically what I'm saying is, hey, Grep, can you search for all of the things in my triage data? I've created this keywords list, and this is an example of crypto related keywords.
Patterson Cake:You could use anything, of course. And then I want you to search through all of this and find me discrete entries and quantity of entries for each of my keywords, and so I get this pretty user friendly, readable output that says, Where did you find these things? And this is a great way for me to prioritize my next analytical steps, because if I'm looking for all XMRIG, then now I know where to look. And we can see that my timeline slice, the body file slice, is the largest number of specific XMRIG entries based on one, five, the quantity of hits, so I now know I should probably go check that out. I probably should go pivot to some of these additional components, or maybe I wanna look at hashed executables.
Patterson Cake:If I have XM ring related MD five hashes, again, this is a red flag, a warning, so you can tweak this, tune this based on your investigation, based on your keyword list, etcetera. It's not super fancy, obviously, but I love me some grip, and works. That's the tip of the iceberg, right? I know it is, I know it is. But the goal, the goal is to define this, to define the endpoint impact for the type and nature of my investigation to focus on the effect on my endpoints, to simplify this into something useful and meaningful, this needs to be iterative.
Patterson Cake:Things change, things change rapidly, you know it. So don't get stuck here, be sort of continually aware of what are general threat actor behaviors, general malware behaviors, instead of focusing on the latest and greatest, the named threat or the named threat actor, talk to me about how does it impact my endpoints. I like to break that into memory identity, network, and disk. Those are not discrete technical components, right? That's not the point.
Patterson Cake:The point is just to guide my theoretical analysis, my analysis process. Then I'm gonna take those categorically and interpret the specific artifacts that are likely to manifest as a result of threat actor malware behaviors, and in this example, processes, sockets, writable dirs, tasks, services, remote management, and monitoring. Spoiler alert, those pretty much work today across all endpoint operating systems, legit. And all we're doing is slightly modifying this process, because Cron is different than scheduled tasks. Apple Unified Log is different than the SSH off log on Linux.
Patterson Cake:Get But the if we can consolidate those things, then we have discrete components to guide our investigation, the rapidity of our investigation, and like I said, you're not done, but you're off to a roaring start, an effective start, simple, repeatable, and repos, repos for the win. More information, the overarching rapid triage workflow, which walks you through the Windows components, the Velociraptor, the S3 AWS configuration uploads, the NIC specific components, which are Catscale and UAC broken out separately in the GitHub repo, and then a couple different just discreet one line interrogated and active endpoint, whether it's Linux or Windows via PowerShell. And as always, I like hanging out with you. I love to communicate with you. If you take a look at this stuff, you kick the tires, you have a suggestion, you have an issue or an error, great way to communicate and interact with me.
Patterson Cake:Huge, huge thank you to Black Hills Information Security that they let me share this stuff with you. This is largely our actual functional investigative workflow, and we get to share it with you, which is really unusual and crazy awesome from my perspective. So, thank you, thank you, thank you for your time today. Thank you for enduring the whirlwind, including poetry. Reach out if you have questions and if nothing else, if that was just not enough for you, we can do it again on Friday for a four hour workshop.
Zach Hill:Yeah. Definitely. I put the link to your workshop in the chat. I'll do it again just in case Nice. You guys wanna check that out, please go ahead and sign up for the workshop.
Zach Hill:I believe it is pay what you can. Starts at $25, I believe.
Patterson Cake:I think it is. I think that sounds right.
Zach Hill:Yep. $25. So yeah. If you enjoyed Patterson's teaching, which I know I do, always, you're phenomenal. Thank you so much for sharing your knowledge with us, sir.
Zach Hill:We appreciate you. Do you have, time to answer a few questions?
Patterson Cake:Heck yeah. I can talk about this all day.
Zach Hill:We did I have a question about the workshop though. If somebody cannot sit for the workshop, is it possible to follow with the labs on another day?
Patterson Cake:100%. Yeah. Absolutely. So you register, if you can come for some of it, sweet. If can come for all of it, even better.
Patterson Cake:If you can't, then the Zoom recording is made available to you. The lab data I will happily provide you to take with you, so to speak, and then the labs themselves are all through Cloud VM, So you can you can run those labs, you can do them locally should you want to, or you can do the labs through Meta CTF, or it's not Meta CTF anymore, is it? SkillBit. Okay. But yes.
Patterson Cake:The answer to your question is yes.
Zach Hill:Awesome. Thank you, sir. I have a little bit of a longer question here from Rocky. Do you what chat should I put it in so that you can see it?
Patterson Cake:Probably the Zoom chat.
Zach Hill:The Zoom chat. Alright.
Patterson Cake:Put it
Zach Hill:I'll put it where it's so to everyone. So it's starts with regarding the collection of Safari web history. And I'll read it for everybody so you so they can and I'll let you read it so you can answer it. It's saying regarding the collection of Safari web history, does the current artifact parser account for the state of right ahead logging, WAL, files to ensure zero loss extraction of the SQLite database or is it limited to the primary DB file? I should start there.
Zach Hill:It's longer in-depth.
Patterson Cake:I refuse to answer that question. No, just kidding. Fantastic question. And so, and sort of. The the UAC script itself, and I should have pointed this out, I forgot, but UAC just went through a major revision, like last week, and so I have yet to fully test that, but some cool looking improvements, so you should check that out.
Patterson Cake:In answer to your question, the UAC extracts all of the underlying, the temporal history, the temporal .db files as well. You have to go search those out, so to speak, and launch it and enumerate those via the SQLite browser or something like that, but those components are there if you want to go look for them, extracted from the file system. So great, great question.
Zach Hill:Thank you, sir. Are the workshops and resources for people who work from Windows machines or can work they'll work somebody on a Mac. So all the labs are in a browser, so you shouldn't have any issues accessing everything you need for that workshop.
Patterson Cake:A 100%. That's the primary reason we do that, so we don't have to wrestle with your ARM processor or your whatever whatever.
Zach Hill:For sure. Max is asking, the subject of Mac forensic investigations, what do you think is the biggest trap that a Windows familiar forensic analyst jumping into Mac devices would fall into? Oh, boy.
Patterson Cake:That's a fantastic question. I think honestly the things that I talked about today were kind of my effort to say, if you're familiar with Windows and maybe even secondarily Linux, these are parts you might not be aware of. They're just a little different, and so being aware of those things, being aware of the primary differentiators, like the unified log as a specific example, and the sequestered private components of the disk, those are the things that I think are the weirdest, and if you weren't aware, you might just miss them. You might do your analysis on an endpoint and come away thinking you had a full picture, but you didn't get those important pieces. Beyond that, I think it's very Linux like, for the most part, but there are some gotchas.
Patterson Cake:This SSH auth log, well, it doesn't exist. It's part of the unified log, that kind of thing, but great question.
Zach Hill:Thank you, sir. This one's coming from Orthicon and Siri. I put it in the Zoom chat as well because there's some string there. But what about search logs for other browsers? Same general location, and they say IE, and they put like root users, username
Patterson Cake:Yes. Etcetera. Yes. Great question. And and absolutely.
Patterson Cake:Generally, follow that same file path construct. Obviously, every browser is just a little bit different, but that is exactly the kind of thing, in my humble opinion, that you should be kicking the tires in your environment, in your specific use case, to make sure that you have standardized paths, standardized output, but in general, the answer is yes.
Zach Hill:Awesome. What kind of motorcycle do you ride?
Patterson Cake:That is the best question of the day. I have a Triumph Street Twin. That's my old man bike at the moment.
Zach Hill:Thank you. Pre Tom's asking, in modern Linux compromises, attackers often use fileless or in memory techniques. How do you reliably detect persistence mechanisms that don't leave obvious disk artifacts, especially outside traditional cron or system d vectors?
Patterson Cake:You remember way back at the beginning of this presentation where I call myself an IR contrarian? Here it goes. I I think fileless malware is not a primary concern. I I think that I I almost always pivot to to permanent, to non volatile artifacts. And and forgive me, let me qualify that.
Patterson Cake:A couple things are commonalities. I rarely get memory. I rarely do. Somebody's already rebooted it, right, every single time. And so I sort of gave up on memory analysis a long time ago.
Patterson Cake:I just don't expect to get it. I'll take it, by all means. Even running process, I don't really expect necessarily to get it, but it is possible that we have transient artifacts in memory only. But if that's the case, then the threat actor was not done. The malware did not complete.
Patterson Cake:The actions objective did not continue. So, don't focus a lot on that, and maybe, again, that's just me and Active IR as a consultant, but I sort of minimize the priority of that, because I almost always, and I mean 98.4% of the time, see other non volatile indicators of that exact behavior. Sorry. Thank you. That was a long answer.
Patterson Cake:Sorry.
Zach Hill:All good. I'm sure everybody appreciates it who or especially the one who asked it, for sure. Would Velo on Mac collect the same files as UAC or is it missing artifacts?
Patterson Cake:Sorry. Say that one more time.
Zach Hill:Would Velo on Mac collect the same files as UAC or is it missing artifacts?
Patterson Cake:So so, yeah, it can be configured effectively to collect the same artifacts. It just is hard to do. And and I don't like I don't like the abstraction between Velociraptor and UAC because I don't find much value, and I find a huge challenge, specifically of the execution of the binary, so to speak, that didn't come from the Apple ecosystem, and it may not be signed depending on how you create the offline collector, and on and on and on. That's just it's painful. And I like that UAC is super easy to customize.
Patterson Cake:You know, you wanna create your own IR triage artifact prioritization, you you edit one YAML file, and you're done. And so, it's just so simple to take that and make it your own outside of Velociraptor, but if you if you need to or it makes sense, you can totally do that same level of of collection through Velociraptor.
Zach Hill:Thank you, sir. What is the trusted link for iBackupViewer or the hash of the legit file?
Patterson Cake:That is a great question, and I stand by, because I was looking at that earlier, and I really wanted to find it, because I think that the right place to go, if I'm not mistaken, is iMac Tools, I M A C Tools, I believe is the actual proprietor of that solution. Again, I'll be honest, every time I use that tool, I use that tool on a transient VM for my analysis. I do not install that on my corporate PC. I use it with caution, but imactools.com, I'm pretty sure, is the real deal. Well, I shouldn't say that.
Patterson Cake:That is the version that I have used and tested.
Zach Hill:Looking that up.
Patterson Cake:Think I can find it. I hate sharing links because, you know.
Zach Hill:Thank you, sir. Can probably answer this question from Wren and their experience, the most persistence mechanisms happen in launch agents, launch daemons. Is that true in your experience?
Patterson Cake:I think that that is extremely relevant. And I think that, I think as an overarching category, that should be at the top of your list. Oddly, oddly lately, just based on my experience and in very recent investigative forays, just Chrome seems to be Well, honestly, scheduled tasks. And I think primarily scheduled tasks because you don't require admin root level privileges to create scheduled tasks in the context of the compromised user, and so that seems to be like the easy button lately, whether it's Windows, Linux, etcetera, is the ability to do Cron and or scheduled tasks on Windows because no local admin required. Not as powerful, so to speak, you know, not system wide, they're user specific context, but I think, you know, I've been seeing the threat actors leverage that because it's a given that you're gonna be a standard user if you compromise an account, so to speak.
Zach Hill:When does your book of poetry drop?
Patterson Cake:Yeah. I completely plagiarized, so it's you know, I feel like I get in trouble. The a work in progress, let's just say.
Zach Hill:The anonymous attendee says that they appreciate your war of ages shirt. Nice. Thank you as well.
Patterson Cake:Nice. Thank you.
Zach Hill:You're welcome. In your experience, what subtle misconfigurations in Linux environments most often lead to privilege escalation but are missed during audits?
Patterson Cake:Wow. That's a fantastic question. As you started that question, before you were done, I I was ready to answer, and it was incomplete, I'll answer what came to mind, and I think this is of critical importance. It's not what you're asking, but what I see a lot are SSH misconfigurations that lead to initial compromise, and that is just, man, there's zero room for error on that specifically. So, again, that's not what you're asking, but that's immediately where my mind went, is weak password, or in many instances where we think we disabled password off, that we required key based access, and then we made a mistake on the SSH config, or something along those lines for initial access.
Patterson Cake:And maybe I'm just I have a distinct myopic view of the world, but I swear for most of the cases I work after initial access, the universe thinks that Linux is secure already, and so it's just a devil's playground. Once that threat actor gets initial access, then there are very few constraints in the typical enterprise Linux distribution. It's pivot to root. Often again, don't see the threat actors even necessarily needing that pivot. If they pop a box as standard user, SOP right now is behave like standard user, and do all the things.
Patterson Cake:Do all the things from the SOP that I talked about. You can often execute applications. There's some directories that will be world writable and world executable. Huge hole directly to your question, I think that would be sort of top of my list, and then of course, standard user can execute outbound network connections, standard user can potentially execute crypto mining, etcetera. So maybe, again, in summary, another long answer to a short question, I would pay particular attention to sort of least privilege when it comes to writable and executable directories.
Zach Hill:Thank you, sir. Harry's asking, how challenging is it to make an image from a Mac with a build in non removable storage?
Patterson Cake:Oh, god. It's terrible. It is extremely challenging, and I'll I'll tell you the truth. The even in most enterprises that, distribute Macs to a certain extent, the centralized management is just terrible. It's terrible.
Patterson Cake:So getting a physical device and imaging a Mac, this is something, one more time, in your environment that you should test and work through and document in advance, because there's so many complexities. End user wipe device, or My favorite one is in the MDM, we have great management of our Mac, we have it managed through Intune, and the minute we decided to terminate this employee, or we believe that the device was compromised, we marked it for deletion in Intune, and then you're hosed. The drive is poised for destruction, and there's not a dang thing you can do about it, so it's a huge pain. Did I mention it's a huge pain? That's part of the reason that I pivot to a collector, like give me data, as opposed to full disk image, that doesn't obviate that question or that process, but it is really tough.
Patterson Cake:Can I just say, sorry, I should've said this in the thing, if someone comes to you and says, I need you to investigate an endpoint, one of the things you obviously should ask is, what operating system is it running? And if they say Windows, Linux, or Mac, you should immediately try to manage their expectations. That's critical. If somebody says, I want you to investigate a MAC, I immediately go, I'm happy to do that. I need you to know that I'm going to need full administrative access.
Patterson Cake:I'm going to need full DISC access. I'm gonna need administrative access physically, potentially to the box, and if you can provide those things, then we can absolutely do an investigation. If you can't, then this is gonna be really challenging. So just kind of managing that at the beginning is pretty useful.
Zach Hill:Thank you, sir. I think we're all caught up here, so if you guys have any other questions, you can definitely reach out to Patterson, or if you want to learn more with Patterson, definitely consider joining his workshop next Friday, May 1. It's four hours long. It'll be a rapid endpoint investigation for Linux and Mac. So you'll be diving a little bit deeper than you, did today.
Zach Hill:So definitely check that out if you guys, want to learn more with Patterson. Do you have any parting words for us, my friend?
Patterson Cake:Well, thank you, Zach, and thank you everybody who showed up, and especially if you're still here. Man, We I just, I love hanging out. I love talking about this stuff, and it is my great joy to help you in any way I can to, you know, sort of thwart evil. So, again, thank you for being here. I hope this is useful.
Patterson Cake:Love to stay in touch. Questions, comments, criticism, feedback of any kind. So, again, thank you. Thank you. Thank you.
Zach Hill:Absolutely. Thank you everybody for joining us. You can tune in next week for another webcast where we'll have Natalia Salmon joining us for Break Free From Cybersecurity Burnout. So hope to see you all there. I should be there and I hope you all have a great week and a great weekend.
Zach Hill:Take care everybody and bye bye.