The IR Mindset (Part 2: Practical Approach)
In the first part, I went through the first half of an investigation, which is about the mindset facing an incident. This is a crucial part, but it still has to be backed up with actual investigative skills that can only be obtained through actual hands-on work and by playing around with artifacts of some kind. Let’s dive in!
As a defender, you have to defend something. Could be a system, a domain environment, or anything that was once in a development cycle. Part of this cycle is to log or audit important information. In other words, for most things, there should be some sort of a recording of what happened for security or troubleshooting purposes.
In a perfect world, every action that had happened is recorded and kept for a long period of time. Oh, and auditing features are enabled by default. Since we do not live in one, depending on the security auditing posture of companies, these things are not guaranteed.
In an IR case, you work in hindsight to unravel actions that have already taken place in the past. Therefore, auditing is incredibly crucial. An unaudited environment is a pain for every investigator, and sometimes without a proper one, it will be impossible to find the root cause of the incident.
Disclaimer — In these next paragraphs I am going to demonstrate a general investigation guideline that assumes the existence of security tools, to some extent. Investigating at scale could have a different strategy, but the guidelines should stay the same.
As part of this artice, I will demonstrate this guideline in a scriptblock font with a simplified scenario - a download of a malicious program that attempts credential theft and beaconing to a C2 server. In an attempt to keep this article brief, try testing yourself on your own by thinking about the next steps and artifacts you will need to collect in order to complete each step! A general guideline for an investigation process should look something like this:
Get A Grip On The Incident
Before any decision or escalation is made, a general gathering of information should take place. It is not recommended to make biased assumptions that may not be correct, it happens, but it’s just an unpleasant situation (shoutout to the third agreement).
An incident usually gives away the general concept of what happened and what caused it, some incidents are more telling than others. Before going forward to the next step, you should have a general knowledge of what happened so you can decide how to move forward.
You, as a responder, are there for a reason. Your sense of judgment matters, and is even expected. Taking responsibility in this field should always be prioritized and appreciated.
In our custom scenario, we received an alert on our SIEM system that a blacklisted malicious DNS query to much.evil was observed from a desktop.Determine Estimated Impact and Prioritize
Now, after you know the general direction of the incident, you can start making a plan to unravel the incident. It is QUESTIONS time!
First, from a business approach- does the company need to close any services? Shut anything down? Any effect on clients? Was there any exfiltration of PII (Personally identifiable information) or any other important data? etc. etc.
And from a technical approach, if it has not yet been escalated to higher-ups, should it be? Does this pose an immediate threat to the network? Does it require an immediate response? How can this case even be solved? What do you need to know to get the puzzle pieces for the incident, and how will you get them?
What I mean by the last question is that you should think about which specific systems and which specific artifacts can enable you to obtain this information. To answer these questions, I recommend writing down the direction you are planning to go in bullet points.
Once you have established which pieces you need to look for, the next step is to prioritize and determine what will bring you the most benefit in the least amount of time.
Is a callback to a blacklisted address definitely malicious? How can we make sure of that?
What would have happened if this happened on an important server?Investigate
Once you have visualized how you are going to kick this incident’s butt, you can let the games begin.
It is important to understand that while investigating and finding other leads, just as tree branches work, you must embark on a different quest to find a different puzzle piece, going through the same investigation process I’m about to describe.
Just as the sketch above suggests, the investigation process can be divided into four parts:
Logs & Artifacts
For declaring the incident as successfully investigated, you have to have sufficient information, right? Logs and security artifacts are the information source. Process executions and network logs are also classified as logs/artifacts. Really, it could be anything that gives you data.
Depending on your environment, you can utilize systems such as SIEM, EDR, or any other log repository available to present you with enough information.
If the data provided to you by any kind of system is not enough and you have sufficient privileges, you can get more data by obtaining artifacts on the root endpoint’s OS — event logs (Event Log Explorer), binaries, MFT (for file system), Prefeteches, etc.
Most questions have answers if you gather the right artifacts and look hard enough (although sometimes it is just impossible :|)
Let's figure out why this connection happened. Assuming we don't collect endpoint logs to a centralized place and we don't have an EDR system in place, let's look into the endpoint's artifacts.Network Communication
To communicate with other people you have to talk, right? The same goes for computers. However, keep in mind that network communication is more ephemeral than other artifacts and harder to catch.
A few resources you can utilize to understand who the system has talked to are FW logs, EDR communication logs, custom logs such as Sysmon event ID 3/22, filtering platform, DNS cache(PowerShell: Get-DnsClientCache), browsing history (Nirsoft) and SRUM.
Keep in mind that a network connection is initialized by a process of some kind, so a good approach is to try and bind the network communication to an effective process.
We know a suspicious DNS query was in place, so consequently, looking at DNS logs can assist us. We managed to bind the act to a process started on the system (PID 316).
Try to think about we should look for next.Processes and commands execution
A good ole saying I like is — Malware can hide, but it has to run.
Somehow somewhere, an application must run in the computer’s memory for it to give instructions to the CPU. You may not see it, but it is there.
It could occur as a binary execution commonly known as process, which can be categorized into several categories.
Every process has a parent process that spawned it (well, except for the very first one that is initiated on the machine, or injected into another process’ memory). Utilizing this information and forming a tree of execution, even in hindsight, will help you build the big picture (Process Explorer or just matching Parent Process IDs (PPID) to Process IDs (PID) will do the trick).
In looking for process creation logs of any kind, we found out about the process that initiated the suspicious DNS query. Since explorer.exe is the parent process (which is legitimate in this case), it could indicate that the user itself started it.
In looking for anything else that the supsicious process could have done, we found out a "legit.exe" process was spawned.
This leads to the question - how does age_of_empires_II.exe even get to disk?? 
The "Downloads" folder was pretty telling, don't you think? It led us to look for browsing history where we could easily see illegitimate web traffic, which can lead to malware running on your computer.
Note how the timing alings.Quick tip - when viewing CSV files with Excel, press ctrl+A to mark all data, and then ctrl+T to make it into a table for easy viewing and filtering. Pressing ALT+H -> O -> I, will also autofit the cells' width.
IOCs
IOCs, or Indicators of Compromise, are essentially unique identifiers of malware.
File hashes, file names, domain names and IPs are all IOCs. Diving even deeper can lead to registry keys, strings, functions, blobs of code, etc.
They can be easily searched for online and within the security appliances’ search. We use IOCs for the same reasons police use IDs or SSNs. Once you stick identifiers to malware, you can robustly block it or gather more information online (these IOCs can be altered more easily, however if found, it is definitely a quick win). For instance, entering the hash of a suspected executable into VirusTotal (VT), or any sandbox engine such as Hybrid-Analysis, can save a significant amount of self-analyzing time. IPs/URLs could go to VT as well, IP/URLvoid, ipinfo.io, RiskIQ, etc.
My goto for any IOC is the simple Google dork intext:”<Enter IOC>”.
Time is priceless!
For instance, after submitting legit.exe's hash in VT, we now know this is in fact mimikatz.exe, which is mostly known for extracting passwords, so any logged in user on this desktop is not secured now.
We can also google intext:"AppData\Roaming\legit.exe", to look for similar occurrences in the wild.Persistence mechanisms
Imagine yourself as a threat actor, and imagine you’ve worked your butt off to be able to run code on a particular machine. Would you risk it all over a simple reboot?!
Attackers will almost always attempt to set a persistence mechanism to make their payload run even if their code has stopped. It could be triggered at every logon, a specific time, following a certain event, or whenever their mind desires.
These persistence mechanisms are commonly called autoruns. Microsoft’s SysInternals suite has a tool just for that for a Windows machine, but the concept exists on every OS.
To deem the machine as clean, it must not have persistence mechanisms, so an investigator always keeps an eye out for that. When in doubt, consider rebuilding the machine from scratch to be on the safe side.
O_o
As part of the remediation process, we should look for this Run key IOC and remove it from every infected machine. Lateral Movement
Attackers are as greedy as it gets. Once they gain a foothold in one of the computers, they will most likely try to spread out to find more valuable assets over the network. Many attacks start from a plain phishing attack aimed at compromising a user, privileged or not. Once it is obtained, the domain admin is the limit.
My point is, once an entity is compromised, you should always look for other assets over the network that might have been compromised as well, such as other endpoints, servers, users, and infrastructure equipment.
How, you ask? Well, you already know! Suspicious network connections and processes with an informative command line that contains IPs or usernames for example are just the tip of the iceberg.
O_o x2
We now added the endpoint with the IP 10.100.102.7 to our scope, and we must see if the attacker managed to spread over there too. Could they use the credentials they extracted via Mimikatz? What should you check first on that endpoint?Assemble A Timeline
Congratulations! You made it to this point. It means that you have the puzzle pieces, and now all you have to do is put them together.
A timeline is essential for you to really understand what happened and also for you to be able to convey it to other people in person or in writing. Noting timestamps along the way, as suggested in the first part of the article, will most certainly help. Sketching the attack flow can also come in handy.
As a rule of thumb, you can attempt to think about the complete attack timeline chronologically. To simulate the thought process, I like to act as if I am the attacker and see if I would do anything else to make the attack more venomous, and that is where I investigate next.
When you feel like you have covered the attack’s entire evolution and have come up with a timeline that makes sense, you have basically finished the investigation part! Now all that is left are the consequential resolutions.
So after checking ALL loose ends and making sure that there aren't any missing pieces in our investigation puzzle, we now know how the attack started, what happened in the middle, and how it ended. We can now look at the incident from a bird's eye view chronologically:
User downloaded a malicious program and ran it -> persistence mechanisms & lateral movement to other hosts -> passwords were extracted -> and eventually beaconing to C2 servers occurred (which was the very first piece of information we got).The Verdict
Decision-making time!
After you have concluded the investigation process, there is the inevitable question— is it malicious? (Some incidents seem 100% positively evil just until you talk to a guy that was testing Mimikatz for fun, you never know!)
A1:Not malicious- Awesome! You can keep on with your life!
A2:Malicious- Once you are done investigating, meaning you have a thorough attack timeline, a few main points to keep in mind:
- How do I remediate the attacker’s actions and get things back to an operational level?
- What is the business impact that occurred as a result of the attack, and is it recoverable?
- Will the attackers be able to restore their foothold in the future?
- What are the lessons learned from it, and how do we prevent it from happening again tomorrow?
Our example scenario was definitely malicious. However the attack vector was fairly simple- it most likely wasn't a targeted attack. The user's judgment of downloaded software was the root cause of the attack.
In this case, remediating persistence mechanisms and resetting passwords/secrets should be enough.
To be on the safe side, if it is possible, I would have the machine rebuilt and start from scratch. Closure
I hope now Incident Response is now more clear to anyone who reads it, and that your IR mindset & toolset were enriched!
How can you get better with IR (and DFIR in general), you ask?
Mostly practice. Keeping up-to-date with the ever-changing IR artifacts landscape through Twitter, Reddit, or any other platform. Reading DFIR reports, playing with DFIR CTFs (there aren’t many relatively), or just old-school dumping artifacts and seeing what you get. Set up a lab, with all your custom tweaks, and just examine away everything you find interesting, be curious about it!
For the most basic level I suggest trying every tool in the amazing Eric Zimmerman’s tools list, especially seeing what artifacts KAPE provides and parse them (here and here too).
Don’t forget that Linux and MAC are out there too (who said Android & IOS or God forbid IoT/embedded :O), but that’s too much for one blog post :)
Feel free to keep in touch or leave your thoughts, Omri 🙃 @0xMRI_
