Incident Response Primer

By Tim Dillman ·

Not long ago, the cost of doing business seemed to be something most companies had a handle on. Sales, Marketing, Finance, Production, Human Resources, Information Technology (IT) and Legal departments were the building blocks of most successful ventures. As the “Information Age” matures, the cost of doing business is increasing in ways we might not have imagined 5 or 10 years ago. Today it is common to see most companies’ IT departments containing a Security subset. Depending on industry or maturity level, some organizations have even gone a step further and added an Incident Management sub-subset. This latest evolution is the predictable result of a trend as old as civilization itself, ‘the bad guys go where the money is!’ Why walk into a bank to rob it when you can steal it from the comfort of your home computer? The same concept holds true for any other item of value that has been digitized.

In March 2012, Robert Mueller, director of the Federal Bureau of Investigations gave a keynote address at the RSA security conference in San Francisco, CA. He said - "I am convinced that there are only two types of companies: those that have been hacked and those that will be. And even they are converging into one category: companies that have been hacked and will be hacked again,"1 This perspective makes the investigation procedure and management of the resulting evidence, critical to the success of managing an incident.

In an article posted last year by the SANS Security Laboratory, contributor Rob Lee predicted, “The virtual incident response team concept will begin to fade in favor of full time Incident Responders, Forensic Analysts, and Reverse Engineering Malware Specialists. Many companies responded to incidents by using the volunteer fire department model where individuals were drafted into serving on an IR team temporarily during an incident. However, with the scale of incidents growing and the time to respond shifting from weeks to months and even years, organizations find that they cannot pull core IT staff off their normal duties to respond to incidents continually. Security Operations Centers will begin to form dedicated full-time incident response teams to respond to ongoing security threats.” 2

This paper touches on some of the major building blocks of an Incident Management program. The goal is to inspire the readers to start thinking of and planning for, a major incident within their own enterprise. This information is structured to apply to a broad audience and the reader is encouraged to conduct additional research.


This topic has been described, as ‘you don’t know what you don’t know’. In other words, you could be breached or under attack this very moment and not realize it. The technology for detecting and alerting on malicious activity is effective, affordable, and necessary if you want to know what is entering and exiting your kingdom and whether or not the keys to it just walked out.

Fortunately, almost every modern piece of computing and networking hardware you deploy will generate some type of log. Those logs can typically be pointed toward a centralized repository where they can be stored, correlated with other logs and prioritized for alerting purposes. Alerting can be streamlined as well, notifying the right people when suspicious or known bad events are discovered. Albeit a reactive solution, log alerts can keep a bad situation from getting worse.

Proactive detection is a little more complex but well worth the effort when done correctly. Technologies such as Damballa’s Failsafe and FireEye’s Malware Analysis System (MAS) capture files from the network, make a near-real-time determination about the risk they may pose, and can be set to block suspicious file packets - depending on your risk tolerance level. Other technologies such as Cisco’s ASA Content Security and Control Security Services Module (CSC-SSM) and Websense’s Security Gateway also provide network based proactive detection of malware but only at the Internet gateway.

For completeness, vulnerability scanning and penetration testing also should be mentioned. Both will proactively identify the risks of a compromise by looking for specific vulnerabilities on the target hosts. The resultant report, gives an administrator specific exposures that need to be remediated, and the general steps required to rectify the identified issue(s). This is proactive detection of unexploited breach points, similar to detection of actively exploited breach attempts.


Detection is an important first step but without a documented, detailed, and practiced response plan, individuals with information security responsibilities will forever be fighting an uphill battle. Certainly, every incident comes with a unique set of variables but the response should contain several constant foundational elements.

Depending on the size of an organization, the incident response team should always be made up of certain individuals or groups. Information Security, Networking, Legal, Public Relations, and Executive Management should all be considered when developing the core incident response team.

Having an incident response plan is also critical to an effective response. At a minimum, the plan should include:

  • Criteria to qualify an incident (compared to business as usual events)
  • A list of incident response team members (may vary by incident type)
  • Logistical details (war room, conference bridge, email distribution lists)
  • A flow chart depicting:
    • Incident declaration
    • Escalation paths
    • Decision points
    • Incident resolution
    • Post mortem analysis and feedback

Complexity of the incident response plan and team may fluctuate depending on the size of the organization, sensitivity of trade secrets, and other factors. Regular tabletop exercises should also be part of the response to help maintain a state of readiness.


An important subcomponent of the response may include the collection of evidence. Companies whose operations are subject to government or industry regulations (HIPAA or PCI-DSS for example) may be required to collect evidence, perform an investigation, and publish the results. PCI-DSS for example, requires that specific breach response procedures from the payment card brands (Amex, Discover, JCB, MasterCard, and Visa) be adhered to, and any subsequent investigation can only be conducted by an approved organization. When developing an incident response plan, this is an area for careful consideration and legal counsel should be consulted.

Depending on the nature of the incident, collection duties can range from collecting the logs of affected devices to extracting a complete forensic image of those same devices. It is important to note is that all subsequent investigative activities performed on the evidence collected, will be inextricably tied to the integrity of the collection. Sound evidence collection practices should always include:

  • Maintaining evidence that complies with the “best evidence”3 Federal Rules of Evidence requirements
  • Establishing and maintaining a documented chain of custody
  • Secure, environmentally controlled storage
  • An evidence retention policy commensurate with regulatory requirements

In the worst case, improperly collected evidence (and subsequent analysis) may not be admissible in court. In April 2008, the National Institute of Justice (NIJ) issued a special report entitled Electronic Crime Scene Investigation: A Guide for First Responders, Second Edition.4 It was originally created for law enforcement however, a good forensic investigator will recognize (or learn from) the many valid procedures it contains. Even though it is a bit dated, the processes have stood the test of time and more importantly, litigation. In the introduction of the guide, it states, “First responders to electronic crime scenes should adjust their practices as circumstances—including level of experience, conditions, and available equipment—warrant.”4 While it is impossible to anticipate every possible set of circumstances that could present themselves, it should be easy to plan for what you know. Very basic questions to ask:

  1. Where does my most critical data (and most likely targets) reside?
  2. Does my Incident Response team or individual have the tools and skills to collect evidence from those systems?
  3. Is my organization prepared to store the evidence properly for analysis and potential prosecution?

Three simple questions that could take weeks to answer completely, involve multiple departments, and may cost thousands of dollars to address. Although answering these questions may be tough, scrambling for answers during an attack will be orders of magnitude more difficult. Do not wait! The NIJ offers a detailed flowchart5 on their website for collecting digital evidence.  


As a specialized and complex element of a complete incident management program, analysis is at the heart of every investigation. The duties performed during this phase can be as simple as reviewing log entries for specific event types to reverse engineering malicious code to understand its operation. Because analysis is a specialized skill set, it can be a challenge to staff these individuals. Depending on the frequency of major incidents, many companies choose to go the route of retaining this capability with an outside agency. Regardless, it is still a good idea to have designated staff trained on the basic tactics for reviewing evidence. Most forensic software companies offer introductory courses on how to perform analysis with their tools. Reputable forensic software companies include:

  • Guidance Software’s EnCase
  • AccessData’s Forensic Tool Kit
  • e-fense’s Helix

As with each of the topics mentioned in this paper, only a high-level overview is presented, and the reader is encouraged to perform additional research.

Incident Management Framework

It is interesting to note that while evidence management is a subset of incident management, it is itself, comprised of many different components. The oversimplified view of evidence management is making sure the item is securely stored and accounted for. However, consideration on a commercial scale over many years includes many key elements.

At a foundational level, evidence is a record of an event or incident. From this vantage point, it becomes easier to recognize the process framework. Classification, retention, preservation, storage, disposal, and integrity must all be included in the creation of a comprehensive evidence management plan. Despite being a subcomponent of the larger investigative process, treatment of each record of evidence should follow strict guidelines. This protects the evidence from the hunch factor; which is based on the false perceptions an investigator may have. The location where evidence physically resides is not always conducive to separately managing individual records. Values, which would be attributed to the most sensitive records by default, shall apply to all records sharing the location.

Here are six elements to a good incident management framework:

  1. Conjecture is an opinion formed through speculation and based on incomplete information. In incident management, this means avoiding conjecture by having a sound methodology for producing factual reports based upon the evidence identified. Additionally we can see the consideration of potential objections, which may be raised by the defense when taking your case to court, especially if conjecture leads to hearsay. Classic objections offered in cases involving digital evidence include hearsay and best evidence.
  • Hearsay: “Many courts have categorically determined that computer records are admissible under Federal Rule of Evidence 803(6), the hearsay exception for “records of regularly conducted activity”—or more commonly, the “business records” exception—without first asking whether the records are hearsay… Increasingly, however, courts have recognized that many computer records result from a process and are not statements of persons—they are thus not hearsay at all.”6
  • Best evidence: The best evidence rule states that to prove the content of a writing, recording, or photograph, the original writing, recording, or photograph is ordinarily required... Properly copied electronic data is just as admissible as the original data. Rule 1003 states that a duplicate is admissible to the same extent as an original unless there is a genuine question about the original’s authenticity”.6
  1. Retention length should be appropriate for a record and determined by a previously assigned ranking and any relevant legal precedence.
  2. Preservation of a record considers environmental elements such as humidity, temperature, and cleanroom level.
  3. Storage has two components to consider. Initially it is necessary to store evidence locally for a period of time. A locked facility with logged or auditable electronic access generally satisfies the chain of custody requirements. Long-term options include using a professional storage company. Issuing a formal request for proposal to service providers should include questions about the security, longevity, and facility access. In addition, determining an appropriate container for storage should coincide with decisions made regarding retention and preservation.
  4. If each previous element has been given appropriate consideration, integrity of the record will remain intact. This can easily be verified at any time in the future by comparing the hash value generated during the initial acquisition.
  5. In the event an evidence record is to be disposed, a limited number of options should be explored. If the media housing the record is expendable, burning, shredding, and other forms of irreversible destruction are acceptable. If media is to be reused, overwriting electronic records using Department of Defense standard 5220.22-M methods is acceptable.

Despite the different requirements and steps involved, these six elements contribute to a single concept. Admissibility is the underlying benchmark used to measure the effectiveness of each process in its contribution to a successful framework.


From detection and response through collection and into analysis, every element of incident response has much to consider. In the references section that follows, the reader will find links to additional sources of information. Even more important though, is the sense of urgency this paper should inspire. If your organization has not yet proactively taken the time to develop, test, and tune a comprehensive incident response plan, do not put it off any longer. Hackers and cyber criminals are already organized and you should be too.