Having a solid incident response capability isn’t an accident.  It’s the result of focused preparation, training and culture.  Incidents come at unexpected times, frequently with little warning, and can have a severe impact on an organization.  It’s during these times that inadequate planning, documentation and missing tools become painfully apparent.  That high level incident response plan that made the auditor happy suddenly doesn’t seem to be very useful.  Perhaps we think that the plan will give us the overall guidance and we’ll be able to figure it out as we go from there.  Think this will work?  Think again.  Let’s look at one possibility outside of information security.

Thankfully, scenes like this are not played out in our fire departments.  The men and women who become firefighters are trained, professional and prepared.  Their turnouts are always ready that they can grab them, climb into the engine and race to save someone’s life.  They don’t have to rummage around the fire engine checking for equipment, because when they got back from the last call they made sure they were ready for the next call.  And they did so in a way that the next call can be a fire, an accident or the proverbial cat stuck in the tree.

Fire fighters are the consummate “incident responders”.  They are able to drop what they are doing or wake up from a dead sleep, grab their gear and start saving lives.  Their preparation is such that they are ready to respond to a variety of issues. 

As computer incident responders, there are some things that we can learn from these professionals.  First, they are selected carefully and trained in the proper tools and techniques.  Next, they assemble the equipment and store it on a mobile “jump kit” known as a fire engine.  Last, after they use their fire engine, they restock it and prepare it for the next emergency.  So let’s look at the nearest equivalents in computer incident response.

Selection of Incident Handlers

Incident handlers need to be selected with serious consideration.  When an incident occurs the incident response team is called in to deal with the situation.  The business is willing to make this call because the incident handlers are the experts in this stuff.  They trust them.  The team is made up of people with solid technical backgrounds, they know what needs to be done, how it needs to be done and can guide everyone else through the process.

An incident is stressful on everyone around it.  People may fear for their jobs or careers.  They aren’t sure what to do or how to even begin.  It may be pure panic.  Incident response team members need to be able to handle their own stress and the stress of those around them.  They need to be calm and help calm down everyone else around them.  They must make good decisions under pressure.  Communication is a key skill that will be relied on heavily.  They will need a good understanding of the business and what’s important to it.  Plus they must know the procedures to handle evidence gathered in response properly so that the business is protected or has legal recourse.  Last, they must to have the technical knowledge and analytic capabilities to get the business out of the incident and back into regular operations.

This kind of expertise becomes a reality because people have made a deliberate choice to pursue and gain these skills.  They are not individuals who can be picked at random.  Choose carefully and deliberately.  Then support them in fulfilling the mission they’ve been assigned.

Equipment Preparation

Incident response requires some equipment on hand and ready to go.  It’s not a kit filled with terribly unusual or expensive items.  It involves deciding what needs to be in the kit and putting it together into something frequently referred to as the “Jump Kit”.  It’s an organized bag with all the basics needed to respond to a variety of incidents.  As you assemble it an iron clad rule must be followed.

With that rule out of the way, what should be in a jump kit?  NIST has created Special Publication 800-61 – Computer Security Incident Handling Guide to help government agencies prepare for their incident response capabilities.  It contains a number of recommendations for a jump kit.  SANS also has assembled a similar list of information, which is distributed to students of Security 504, Hacker Techniques, Exploits and Incident Handling.  Both are excellent sources of information.

The list below is a combination of ingredients from both sources.  I have some of these items but not all yet.  I’m still working on building my kit.

• Contact information for team members and others within and outside the organization (primary and backup contacts). If you’re depending on the online address list, but that server is compromised you may be in trouble.
• Pagers or cell phones. Don’t forget spare batteries and chargers
• Laptop with lots of RAM, a large hard drive and pre-loaded with forensics applications
• Virtual machine software such as Xen, VMware, or Virtual PC so you can run multiple operating systems
• Live CDs of various types. Some options are Helix, CAIN or DEFT for forensics. Backtrack is another possiblity, though it is focused on penetration testing
• DVDs or CDs with trusted, statically linked versions of programs to be used to gather evidence from systems
• Packet sniffers and protocol analyzers to capture and analyze network traffic
• Encryption software. If the network is not trustworthy, you need encryption to keep your communication confidential
• Blank media, such as floppy disks, CD-Rs, and DVD-Rs
• USB thumb drives. 8 and 16 GB drives aren’t too expensive
• A network hub or a tap. A tap would be better, but they cost more
• Write blocking device(s) to use when imaging hard drives or other media
• A large external hard drive. Maybe more than one.
• Tools such as screwdrivers, flashlight, tweezers, telescoping magnet or hands. I tend to bring a Gerber utility knife with me where ever I go. Watch out when flying some where though. Be prepared to check your kit if you carry a pocket knife or screwdrivers
• USB to serial port adapter
• Network cables. Straight and crossover. I carry a crossover adapter
• Hard drive jumpers
• Cisco rollover cable and a serial cable
• Female to female RJ45 connector
• Hard-bound notebooks with numbered pages
• Digital camera and audio recorder
• Chain of custody and other incident forms
• Evidence storage bags and tags, and evidence tape. Amazon.com has these items
• Desiccants for protecting against moisture in the bags
• Port lists, including commonly used ports and Trojan horse ports
• Cryptographic hashes of critical files to speed the analysis, verification, and eradication of incidents
• Media, including OS boot disks and CD-ROMs, OS media, and application media
• Security patches from OS and application vendors

Edit — 7/21/2010
One quick note to add to this list. Consider what items are practical to have a couple of. In my case the power switch on my write blocker failed when I was imaging a number of systems. The vendor, Digital Intelligence, responded in short order to replace my write blocker and was awesome. But in the mean time I was stuck. The lesson of redundancy was re-taught to me at that point. If duplicate equipment is not possible consider what your backup plan is if something dies on you.

— Continuing the original post…

If you are doing incident response internally…

• Documentation for operating systems, applications, protocols, and intrusion detection and anti-virus systems
• Network diagrams and lists of critical assets, such as Web, email, and database servers
• Baselines of expected network, system and application activity
• Backup images of OS, applications, and data stored on secondary media

You may decide that this list isn’t complete. It’s not. It’s just a starting place. There may be that one time when you really needed something and didn’t have it. So now you always make sure to have it with you. After each incident, review how things went and ask the team if there was some piece of equipment that wasn’t there. Adjust, improve, and then restock.

Jump Kit Restocking

After the incident is over several items from the kit may have been used and need to be replaced.  Compare your inventory of what the bag should have to what is left in it.  Note any missing or nearly depleted items.  The team may have noted some items that they wish that they had had or that needed to be purchased during the incident.  If appropriate, add them to the list of jump bag items and purchase it for next time.  Make sure that everything is stored properly, so that it can be used without needing to untangle it from something else first.  Once the bag is ready, put it in its normal place and have it ready for next time.  Your team is now ready to respond to the next incident with confidence that when they grab the bag that it is ready to go.

How Are We Doing Now?
The team has been selected, vetted and trained. Everyone knows where the jump kit is and that it has the gear necessary. When the phones start ringing and an incident is declared, the team starts gathering their gear and congregating as planned. Instead of scrambling for equipment and wondering what to do next, the team is ready, confident and it shows. The incident has a much better chance of being resolved quicker, more completely and according to accepted practices.