Preparing for an Incident

By Colby Clark ·

Out of the hundreds of incident response investigations I have been involved with, I find that most are a result of not taking proper steps to prepare. Fewer than 40% of companies have meaningful logs that cover the period of time they were compromised. The logs roll over quickly, often lasting less than a few days.

Moreover, even fewer companies have rolling pcap files or the ability to capture host-based activity. In general, they lack the visibility required at the endpoint or network to be able to adequately determine what is going on and identify malicious activity in a timely manner themselves. In fact, most companies discover they have had a breach from a third party, often law enforcement.

Typically when we respond to incidents, we have to configure customer environments to facilitate a good incident response engagement. This usually includes a combination of customer resources and those we bring to augment their capabilities.

Of primary concern is the ability to monitor the following:

  • Host activity – Use one of the several continuous monitoring agents  out there that capture all host activity (process execution, network activity and file writes) and feed all activity to a centralized location for viewing, searching and retrospection. Make sure you evaluate each of these solutions carefully to ensure they map to your specific needs. They appear the same from a surface-level examination, but each of the options is actually quite different.
  • Network activity – It is true that recording network activity can be as easy as throwing a Linux box on a spanned port and using TCP Dump, but this solution nets a lot less value with each passing day. With an estimated 85% of malware traffic being encrypted these days, not to mention most websites of interest, leveraging an SSL decryption tool has become a necessity. Do yourself a favor and capture decrypted traffic if possible.
  • Log activity – It should go without saying that logs from all ingress/egress devices - such as firewalls, proxies, VPNs or other remote access devices - in addition to security devices, authentication systems, critical systems and applications should all be directed to a SIEM to provide the ability to parse, correlate and preserve.

In addition to having these items logged for later review, it is essential to be able to do something about it. Once an attack is detected, affected hosts are identified and behavior is understood, the next needs are as follows:

  • Containment – It is critical to be able to stop malicious activity before it spreads further or sensitive information is removed from your environment. Containment is an intermediate clean-up step that allows you to hit the pause button on an intrusion or malware outbreak, freeze malicious activity and do a deep-dive investigation or buy some time to perform remediation.
  • Deep dive – Being able to do deep dive investigations on targeted nodes is key to identifying the attack vectors used, what data is affected and just how bad your organization has been compromised.
  • Surgical remediation – Because malware outbreaks can be widespread and performing full remediation of affected machines can be slow, the ability to execute surgical remediation and sanitize infected hosts is essential.

Having said that, it is important to ensure that agents for your containment, deep-dive and remediation solutions are always deployed on endpoints and ready to be used at a moment’s notice. There are few things more frustrating than trying to get an agent on an endpoint and going through all the change control and authorization processes necessary during the heat of an incident. These processes can take days or hours when minutes and seconds matter.

One thing I would add, but it is probably asking too much, is to have a good understanding of your environment →BEFORE← you have a problem. It will assist you in being able to detect abnormal or malicious behavior. Behavioral analysis should include both the host and the network environments to ensure you have a good baseline of activity.