The IR Mindset (Part 1)
A Way of Thinking Stepping Into the Incident Response World And Practical Investigation Guidelines To Tackle Security Incidents
Part 1 — The mindset of an investigator when facing an incident, along with five tips & tricks.
Part 2 — A detailed practical guideline for an investigation process with an example scenario.
So you might want to or have already found your way into a SOC (Security Operation Center), looking at a SIEM (Security Information and Event Management) or an EDR (Endpoint Detection & Response) system, waiting for an alert to pop up so you can roll up your sleeves and get cracking. This article may sort some things out in your head when approaching a new case - seeking an accurate resolution.
The goal of this article is to share methods and perspectives of mine that hopefully will come in handy whether you are an entry-level or an experienced responder, so let’s get started!
As an Incident Responder, the number one goal is... well, responding to security incidents when they come. You are on the frontline. You have to analyze the situation well enough, fathom the impact of every incident that arises, and as a result — respond accordingly and in a timely manner.
For that to be done properly, you must comprehend the inner workings of the incident that you are looking at. A good approach is the good old 5 Ws (Well 4 Ws and an H). For an accurate resolution, you must understand the what, who, when, how, and the why of the incident, don't underestimate them!
- What — You need to know what actually happened, there is no way around it. If you see a process running with a command line that you are not sure what it does- look it up, perhaps even simulate it yourself if it is safe (please do not run malware on your personal computer though, stay safe). You should know what you are dealing with. No one can know everything, therefore it is crucial to keep yourself up-to-date and familiar with things as much as possible.
It is not a shame to not know something. However, it is a shame not to look it up afterwards. - who — Every action that happens in the computing world is initiated by something, and when it comes to operating systems, every action runs under the context of some user. It could be a local user, domain user or a system user. Someone or something should always be responsible.
- When — Everything should have a timestamp if it was set up correctly. It is important to keep track of timing so you can link different sources and build an accurate timeline.
- How — With permissions set up correctly, and given the fact that you are looking into a security incident, something had to happen. You need to know how. Was there something that was maliciously exploited, or did normal actions that in a specific situation triggered the alert? Questions that need answers!
- Why — Was there a specific reason the alert happened? If you recognize a threat, what are their alleged motives? Money (Cryptomining, Adware..), Information (user information stealth, organizational proprieties..), gaining access for further infection, etc. Knowing why will point you in the right direction to continue looking into.
Making unbiased assumptions won’t do you any good most of the time (especially when writing down the resolution reasoning at the ticketing system, which is the system that is used to centralize incidents and conveniently keep track of them). As a responder, you will have to find the bread crumbs that will lead you to the answers to all of the questions above.
In order to do this, you will eventually have to determine if the incident is — False Positive or True Positive. If it is False Positive(Detected? Positive. Justified? False), this means the threat is not real and was falsely detected. If it is a True Positive (Detected? Positive. Justified? True), this means it is a real threat that was rightfully detected. This determination should always be backed up with evidence.
During the discovery of the answers to all these kinds of questions, the one word you should constantly recite in your head is — TIMELINE!
There are many clues that can be gathered from many kinds of sources, whether it’s from Active Directory logs, host logs, FW logs, IPS, web, etc. The one thing they all have in common is that they all happened at a specific point in time, and logs almost always have a timestamp to look for. Only after you assemble a timeline and complete the story, you can figure out what happened, come to an accurate resolution, and finally determine whether the incident is a False Positive (FP) or a True Positive (TP).
Before we go down to the actual process of investigating, I’ll let you know of an approach I find works well for me, and sometimes prevents me from going down the rabbit hole of a useless direction.
Simply put, just ask yourself questions.
When an incident comes up, this is in fact the final stage of a chain of events that started earlier in time. Questions will help you unravel the mystery.
“Okay, I see this process running, what made it run? Who was logged in to the computer at that time? Are there any other processes that were spawned by this process?”.
Depending on the auditing maturity and how deep you are willing to go (Office fans I’m sorry ;), almost all questions have answers one way or the other, either from logs or from actual forensic artifacts. Once you know what questions to ask you can think about the answers, and most importantly, which artifacts or log sources can provide the information to answer them.
For instance, let’s say I see the file evil.exe running. Okay, what made it run? Something had to activate it! Let’s check process execution logs for parent process ID (Windows event ID 4688, Symon event ID 1, EDR custom logs, etc.), we see that cmd.exe is the parent process, then let’s keep that going until we find a definitive answer to our question in the form of the primal process.
Who was the logged-in user? Okay then, what is the last successful login event for that host (Windows event ID 4624)? Any suspicious network connections (FW logs)? What even is this source host I am looking at? Is it a server? An endpoint? Does it maybe make sense that this server with its designated functionality would be doing what the incident is indicating?
For instance, seeing lots of outbound network communication coming out from an IP that belongs to the company, towards many different external addresses on multiple ports? Sounds suspicious right? A quick check on that IP address might uncover that this IP is in fact a NAT gateway which makes it a bit less suspicious.
In the second part of the article, I’ll be going over the actual investigation process which should provide a general guideline for every incident you will be facing, with more technical information.
But for now, here are some general tips to keep in mind. These can save a significant amount of time:
#1 — History Repeats Itself!
I can guarantee that you will be facing an incident, which will first look like it is going to take hours to figure out. However, the nature of things is to repeat themselves. There is a big chance that this exact incident, or a pretty similar one, was either already resolved and explained in the past by one of the team members, or was described in a report online. Try using keywords (IOCs) when searching (using Google Dorks). This can save A LOT of time and only takes two seconds to verify.
#2 — Documenting As You Go Will Make Your Life Easier
As an incident responder, you will probably work as a part of a team. All incidents should be followed with a corresponding ticket in a ticketing system, for the sake of keeping track of every incident. It is best practice to make them as detailed as possible. If it is for the sake of reviewing it when a future similar incident occurs, or either for getting the approval of the direct supervisor to make sure this incident is indeed resolved, the investigation process should be documented. Therefore, you might want to note down any leads you find along the way to make sure you cover everything. Including pieces of evidence that are also crucial to get documented along the way (did someone say the magic TIMELINE word?).
My personal favorite is using snapshots (WinKey+Shift+S), saving them on a dedicated OneNote page, and later coming back and reviewing what I found along the way. WinKey + PrtScrn, which automatically saves a print screen on disk (under the user profile’s Pictures\Screenshots folder) is also a good way to quickly capture evidence.
#3-Don't Forget What Brought You Here!
As an incident responder, you are literally responding to an incident that is the result of a set of rules that got triggered. This set of rules was written with care and lots of testing (hopefully). Each rule should cover an alleged threat that you, as an incident responder, should determine as rightfully triggered (True Positive), or if the actual action that got it triggered is actually benign and does not pose a threat (False positive). That is the most important question of all, mother of all questions if you’d like, and should always be in mind during the investigation process. An incident is a direct result of bad activity. Understand the bad and only then you can start resolving it.
#4-Don’t Go Down Unnecessary Rabbit Holes
If you stepped inside the incident response world and even got all the way down here in this article, you must have a curious mind. Which is obviously awesome! That being said, I can assure you that you will often find yourself encountering an unknown command or chain of events, that you will try to figure out what the hell happened there. Besides an aching forehead from all the banging against the keyboard you will most likely do, this can waste a lot of your time, and nobody likes wasting time. It is indeed good to aspire to know everything but more often than not, achieving resolution and actually responding is more urgent.
When actively investigating, trying to find missing pieces so you can complete the incident puzzle, you have to have a clear objective in mind. A vision for the puzzle piece you are currently seeking to find. When you feel a bit lost, try to remember what was exactly that puzzle piece you were looking for.
A good question to ask yourself when you feel like you are going down a rabbit hole is “If I were to figure the thing out, will it actually enable me to connect the puzzle piece? Will it make me achieve the objective I am currently after?”. If the answer is no and you were just shooting in the dark, then drop it and regroup. Worse comes to worst, you can always come back and try again.
#5- It’s All Connected
Think of the network environment as a living body. You got multiple systems that are working separately, and together they seamlessly form one organization that works properly (well, most of the time).
The same goes for your company’s network. It has multiple security systems that work separately with a focus on different aspects, but together they form a secured functioning business. Or at least aspire to be as secure as possible. One should not limit oneself to the pleasure of just one log source.
Seeing a suspicious process activity on a host? You got the IP address of the host? Great, you can now check FW logs to see where it tried to connect. You got the current logged-in user for the host? great, you can now check for Active Directory or Azure AD logs to see if there is something suspicious there, and so on and so on. Utilize everything!
I hope you enjoyed the reading and might even have added a couple of methods to your arsenal. I truly believe that you are as good of a responder as your mindset when investigating, try these out and let me know what you think :)
Mindset changes are always the toughest but they are also the most rewarding ones too!
Part 2 can be found here
