Incident Response -...

© 2019 CrySyS Lab, BME

Incident Response

Levente Buttyán

CrySyS Lab, BME HIT

[email protected]

w w w . c r y s y s . h u

|

Terminology

Attack

A deliberate action or activity carried out with malicious intent

aiming at defeating the security policy of a system

System compromise

Result of a successful attack

Security incident

A detected system compromise

Incident Response 2

|

Security incidents

Examples for security incidents

– Admin password is leaked

– Private customer records are stolen and published

– Data storage is encrypted by ransomware

– Service is disabled by a flood of requests

Security incidents can have a grave impact on an organization’s

business operations and reputation

Effective and efficient response to security incidents is

extremely important

Incident Response 3

|

Incident response goals

Minimize disruption to business operations

– Containment of compromise

– System recovery

Prevent future incidents of the same kind

– Incident analysis

– Recommendations for eliminating identified vulnerabilities

Support law enforcement

– Forensically sound retrieval and handling of evidence

– Accurate analysis reports

Incident Response 4

|

Incident response process

Incident Response

Pre-incident preparation

Strategy formulation

System recovery

Data collection and analysis

Reporting and recommendations

EducationEliminating vulnerabilities

Detection of the incident

pre-incident

post-incident

incident

5

|

Pre-incident preparation

Risk assessment

Preparation of hosts/devices (e.g., hardening, golden images,

backups, logging, host based IDS)

Preparation of networks (e.g., segmentation, logging, network

IDS, free monitoring ports)

Establishment of incident response policies and procedures

Establishment and training of a Computer Security Incident

Response Team (CSIRT)

Incident Response 6

|

Need for deep investigation

usually, primary goal is to recover from an incident (business continuity), but ...

you should also make efforts to better understand– the scope of the attack

» what parts of your system are affected? (space)

» for how long has the attack been going on? (time)

» what is the amount and nature of information that has been stolen? (impact)

– how the attacker actually compromised your system» what was the initial entry point?

» how did the attacker move laterally in your system?

» what are the vulnerabilities that made the attack possible?

– the goals of the attacker» why have you been attacked? (~ who is the attacker?)

» based on the collected evidence, did the attacker reach their goals?

results of deep investigation enable more complete response and better protection in the future

Incident Response 7

|

Some good advise

Be prepared!

– Security incidents did, do, and will happen

– Prepare your IR plan, have policies and procedures in place!

– Make backups, switch on logging!

Be capable of taking the first steps!

– Timely response is usually crucial

– You must have a CSIRT!

Don’t spare deep investigation!

– Better understanding of what happened enables better future protection

– Outsource deep analysis work to specialized companies (contracted well before the incident)!

Learn from the experience!

– What doesn’t kill you makes you stronger

– Improve your protection and educate your employees!

Incident Response 8

PRE-INCIDENT PREPARATION

|

Introduction

incident response is reactive in nature, with pre-incident

preparation being the only phase that comprises of proactive

measures

good preparation can

– save time, effort, and money later when an incident happens

– greatly increase the chances of successful incident response

common pre-incident preparation steps:

– identify corporate risks

– prepare hosts and networks for incident response and recovery

– establish policies and procedures that support incident response

– create and sustain a CSIRT that can assemble to handle incidents

– create and maintain a response toolkit (used by the CSIRT)

Incident Response 10

|

Preparing individual hosts

common steps that help any investigator respond effectively:

– record cryptographic checksums of critical files

– increase or enable secure audit logging

– build up your host’s defenses

– back up critical data and store media securely

– educate users about host-based security

as hosts change over time with new users, software, and

network configuration, the above steps need to be repeated

regularly

– a good approach is to incorporate them into organizational policies and

procedures


|

Recording cryptographic checksums

fingerprint a “known-good” system state by computing the cryptographic hashes of all important files (e.g., executables, DLLs, and drivers used by the OS) and storing those hashes in a secure location– checksums of “known-good” system state should be taken before an

intruder has the opportunity to compromise the system

then, regularly fingerprint the current system state and compare itagainst the stored “known-good” state– any changes to the system state will be identified and investigated

tool support– simple digest computing programs (e.g., md5sum)

» available on most Linux systems

» on Windows, use CygWin or download a Windows equivalent (e.g., md5sums)

– automate checksum computation for multiple files with scripts

– free and commercial products (e.g., TripWire, FCIV utility for Windows)» http://www.tripwire.com/it-security-software/scm/file-integrity-monitoring/

» https://support.microsoft.com/en-us/kb/841290/en-us


|

Enabling secure audit logging

when logging is enabled, answering the question of “What

happened?” after an incident becomes much easier

almost every operating system and many applications provide

significant logging capabilities

some hints to consider:

– default logging capabilities are often less than ideal some tweaking is

necessary

– log as much useful information as possible, but not too much…

– log messages to a file that only the administrator can access, or to a

secure, remote log host


|

Building up host defenses

actions taken to secure hosts

– reduce the exposure to security incidents

– make it easier to resolve incidents when they happen

cornerstones of host security:

– make sure that all operating system and application software are the most recent

» all patches, hot fixes, and updates are installed

– disable unnecessary services

» unnecessary services introduce unnecessary risk

– when faced with configuration choices, choose wisely

» many security exposures are introduced through sloppy system administration

for a complete discussion of secure host configuration choices

– refer to a book devoted to the subject

– or check the web for security configuration of a particular platform

» e.g., http://www.microsoft.com/security/pc-security/default.aspx

» e.g., http://www.debian.org/security/

» e.g., https://help.ubuntu.com/community/Security

» ...


|

Backing up critical data

regular, complete system backups may be

– a useful reference during incident response

» help to discover what was deleted and what was added

» help checking the times files and directories were last accessed, modified, or created

– indispensable for recovery from an incident

Unix/Linux backup tools

– common utilities include dump/restore, rsync, tar, dd, ...

» caution: some backup utilities reset the time of last access

» choose a backup solution that preserves all three time/date stamps (e.g., dump)

– there are many other backup tools available on the Internet...

Windows backup tools

– Windows has built-in backup solutions (Backup, File History, ...)

– there are many other backup tools available on the Internet...

backups are only as good as the original from which they were created


|

Educating users about host security

users play a critical role in an organization’s overall security

educating them improves security and incident response

users should know what types of actions they should and should not take on their systems– computer security perspective:

» users should follow the organization’s password policy

» users should not be allowed to install software on their computer– in particular, a web server or FTP server accessible from the Internet

» users should not be allowed to change configuration of their computer– e.g., to disable virus scanning or automatic updates

» users should be trained to identify phishing attempts

» users should not open attachements or URLs originating from unknown sources

» ...

– incident response perspective:» users should be trained to identify potential security incidents

» users should not take investigative actions

» rather, they should immediately notify a designated contact


|

Common network security actions

installing firewalls and intrusion detection systems

– rather than configuring firewalls and IDS devices to simply protect thenetwork, one should also configure them to log activities

using access control lists on routers

– allows certain types of traffic while prohibiting potentially dangerous traffic

encrypting network traffic

– encrypting network traffic enhances the security of any network

– one can use IPsec or other VPN solutions at the network layer, and TLS and SSH for applications

– caution: encrypting network traffic can also hinder the detection and the investigation of an incident

requiring authentication

– authenticating users can increase network security and provide an effective audit trail for incident response teams


|

Creating a network topology map

without information about the network topology, it is difficult to figure out which other systems may be affected by an incident

example:

– assume you identify a rough network sniffer on a compromised host

– the attacker might have obtained passwords by sniffing

– you need to know which other systems may have been compromised by the stolen passwords

network topology maps include details on – how the hosts are connected

– which networks use switches versus routers

– locations of external connectivity

even if all details are not available in a large network, one should have a detailed view of mission-critical networks such as the DMZ or Internet-facing e-commerce applications

topology maps can be created manually (e.g., in Visio) or automatically (as the output of some network scanning program)

information about the physical location of network devices and cables can alsobe very useful during the handling of an incident


|

Supporting network monitoring

network monitoring is often one of the first steps that one may

take when responding to an incident

make sure that the network architecture supports monitoring

– it should be possible to attach a network-monitoring platform to a

network device that has access to all network traffic on a given network

segment

– e.g., leave open ports on hubs and a spanning port on switches

network monitoring should be supported by appropriate

policies and procedures


|

Incident response policies and procedures

an incident response policy defines

– how potential incidents should be reported?

– who has the authority to approve response actions?

– when monitoring or investigation will be started?

– which type of incident warrants which type of monitoring or

investigation?

– when and how employees will be notified about investigative actions?

incident response procedures

– the policy states what you intend to do

– procedures are the implementation of the policy

– procedures entail much of technical details about how monitoring and

investigations will be carried out


|

Need for policies

policies save money and headaches– with proper policies, corporate investigators can advance the pace of an

investigation, and perhaps take investigative steps that usually require court orders when performed by law enforcement personnel

» performing a trap-and-trace of traffic (only headres) on corporate networks

» performing full-content monitoring on corporate networks

» searching and reviewing an employee’s machine

» coordinating with upstream sites (e.g., network service providers) involved in anincident

policies should be defined before an incident occurs, and employees should be informed about them– policies need to be advertised throughout the corporation and incorporated

into new employee orientation

– all current employees need to positively acknowledge their existence with a written signature

– a refresher overview course should be given when major changes are made to policies


|

Computer Security Incident Response Team

a team of designated individuals (employees of the

organization) with the appropriate technical, legal, and other

expertise necessary to resolve a computer security incident

may involve technical experts, security professionals, corporate

security officers, business managers, legal counsel, HR

personnel, helpdesk workers, ...

as incident response is not required at all times, the CSIRT can

be a dynamic team assembled when an organization requires

its capabilities


|

Creating a CSIRT

it is too late to establish the CSIRT after a computer security

incident has occured

– one cannot expect untrained and unprepared personnel to succeed

steps of CSIRT establishment

– define the team’s mission

– obtain top-level management support

» helps to involve team members from different departments

» helps to define and enforce policies

» important to get resources for the operation of the team

– select the team members

– train the team

» the team needs hands-on hacking and incident response training

» team building exercises may also be a good idea


|

Creating a response toolkit

includes the hardware, software, and documentation used during response

forensic hardware platform– should have high performance

» lot of disk space or external disk drives (for storing evidence duplicates)

» high-end processor and lot of memory (for analysis tasks)

– should be robust and configurable

– should have attachments for various external devices

– portable hardware duplicating tools may be useful

– cables, converters, adapters, plugs, labels and markers, digital camera, ...

response software– multiple operating systems on the forensics computer

– a forensic software package (e.g., EnCase, X-Ways, ...)

– other special tools (e.g., Volatility)

– all drivers for all the hardware of the forensics platform (computer and external devices)

– a selection of bootable USB pen drives and disks with different OSs

– disk-write blocking utilities

– a backup of the complete setup


|


network monitoring platform

– should be able to handle the amount of traffic the observed network has

» performant CPU, lot of memory and disk space

– network card should support promiscous mode

– network card should not respond to Address Resolution Protocol (ARP) packets and maintain network silence

– special network tap devices may be useful

documentation

– documentation is necessary for furtherdisciplinary, civil, or criminal action, as well as for a thorough response

– the CSIRT should document

» how the evidence is obtained

» all actions taken

» where and how the evidence is stored

– standardized report templates and forms are useful

– evidence tags and evidence labels


|

Summary on pre-incident preparation

proper prior preparation prevents poor performance during incident response

– preparation for system administrators involves configuring hosts and networks in a manner that reduces the risk of incidents and eases the task of resolving incidents

– preparation for the organization as a whole involves the establishment of a CSIRT and creation of a response toolkit from appropriate hardware and software tools, as well as templates that help documenting investigations

in the real world, pre-incident preparation is extremely difficult, both technically and ideologically

– few controls are in place to monitor user activities (due to basic human rights)

– logging is not enabled for performance reasons

– hosts do not have security software

– updates are not installed regularly

– no regular backup

– networks have too many entry points and configuration nightmares

– ...


INCIDENT INVESTIGATION

|

Introduction

the goal of this phase is to determine the who, what, when,

where, how, and why surrounding an incident

– however, determining the identity of the source of an incident may

sometimes be very difficult

– thus, many organizations choose to focus solely on what was damaged,

how it was damaged, and how to fix it

incident investigation can be divided into two phases:

– data collection

» gather all the relevant information needed to resolve the incident in a

manner that meets the response strategy

– forensic analysis

» examine all the data collected to determine the who, what, when, where,

and how


|

Incident investigation – data collection

generic requirements on data collection:

– the collection process itself may change relevant data and destroy evidence electronic data must be collected in a forensically sound manner

– collected data must be handled in a manner that protects its integrity (proper evidence handling)

types of data collected:

– host-based information

» live data collection

– volatile information obtained before the system is shut down

» forensic duplication

– provides a “mirror image” of the hard drive(s) of the investigated system, which can be used as a working copy for analysis purposes

– network-based information

» IDS, router, firewall logs

» full network traffic capture (may be initiated as part of the incident response)

– other information

» testimony and other information obtained from people (e.g., via interviews)


|

Incident investigation – forensics analysis

preparation of data

– create a working copy of all evidence data collected

– create a file list

– perform file signature analysis

– identify known good files (e.g., known OS components)

– recover deleted data and unallocated space

analysis of data

– extract e-mails and attachments

– review browser history

– review files in the file system

– identify and decrypt encrypted files

– search for relevant strings (in mails, attachments, browser history, files, deleted and unallocated parts of the hard disk, ...)

– review installed applications

– perform analysis of unkown programs

– review data collected during live response (e.g., recover OS structures from memory)

– review network-based evidence


|

Live data collection

objectives of live data collection:

– to confirm that there is an incident

– to retrieve the system’s volatile data that will no longer be available after the system is powered off

it is important that during the initial live response, one performsas few operations as possible

– in order to minimize alteration of the investigated system

but gather enough information for decision making in later steps

live data collection steps in general:

– creating a response toolkit (preparation)

– obtaining volatile data

– storing acquired information

– in-depth live analysis (if needed)


|


during live data collection, it is critical to use trusted commands, and make particular attention to not destroying or altering the evidence

collect and write a carefully selected set of tools to a USB stick

– collect tools that are compatible with the target system

– command line tools are preferred, because they do less ”behind the scene” interactions with the system

– check for dependencies and ensure that all DLLs and any other files theresponse tools depend on are written to the stick

– create a checksum for all the files on the stick and store them separately

– preferably, use a USB stick that has a physical write protect switch

» e.g., Kenguru FlashTrust or Kenguru Defender

label the response stick

– put case number, time and date, name of the investigator

– indicate whether or not the response stick contains output files or evidence from the victim system


|


at a minimum, you may want to collect the following volatile

data prior to forensic duplication:

– system date and time

– a list of the users who are currently logged on

– a list of currently running processes

– a list of past and current network connections

– the applications using those network connections

– a list of the systems that have or recently had connections to the system

– time/date stamps for the entire file system


|

Forensic duplication

forensic duplication is the process of creating copies of hard disks and other storage media for the purpose of analysis

– also called acquisition or (forensic) imaging

important general requirements:

– the duplication process must not alter the original evidence

– the duplicate should be an accurate representation of the original evidence

types of duplicates:

– forensic duplicate

– qualified forensic duplicate

– restored image

– mirror image

– logical copy


|

Forensically imaging a drive

any time when the original medium is accessed, take precautions to avoid writing to it

be aware that the following actions may modify the suspect drive:

– booting up the forensic acqusition system in Windows with the suspect drive attached to it

– plugging in the suspect drive with a USB/FireWire hard drive enclosure in Windows

– mounting the suspect drive in read/write mode in Linux

– booting up the suspect system in Windows with the drive still in it

– choosing the wrong drive to write to when performing duplication

if you create a mirror image, then wipe the image drive before using it

– wiping means overwriting every sector on the drive with some data

– this will remove old data from previous cases

– you can use dd or any forensics software package for wiping

» e.g., dd if=/dev/random of=/dev/<image drive>


|

Analysis of forensic duplicates

analysis of forensic duplicates of storage media can be

performed at two levels:

– disk level (dealing with partitions, clusters, and meta-data)

» usually requires special tools (e.g., DFF) which can open raw images and

qualified forensic duplicates (e.g., EWF or AFF files)

» allows for recovering deleted files

» allows for recovering unallocated and slack spaces

– file system level

» requires restoring the file system from the forensic duplicate

» in Linux, it is easy to mount a raw image

» qualified forensic duplicates need special mount programs (e.g., affuse) or

they need to be converted to a raw image

» once the file system is mounted, analysis can use standard Linux commands

and special tools for file system level analysis


|

Generating file lists

browsing through the entire file system manually is not feasible

creating informative file listings can be of great value, where each item on the list contains:– full path of the file

– MAC timestamps of the file

– logical size of the file

– an MD5 hash of the file

the file list should be in a format that can be imported into other tools for further analysis and correlation– spread sheet or database applications

– other analysis tools

MD5 hashes can be used – to filter out known good system files (and thus reduce the number of

remaining files that may need to be reviewed)

– to identify known bad files (e.g., based on some indicator of compromise information obtained by threat intelligence)


|

Reviewing relevant files

one can identify potentially interesting files in the file system

– by file type

» determine file type by extension, as well as by true file headers (signature)

» typically interesting file types: .doc(x), .xls(x), ..., .txt, .wpd, ..., .gif, .jpg, .png, ..., .pdf, .tmp, .log, .exe, .dll, ...

» you can limit the possible file types to those relevant to the case you are investigating

– by determining the timeframe in which the incident occurred, and scrutinizing those files created, modified, or accessed during this timeframe

» most forensics software packages offer possibility to filter files by their access times

» use standard Linux find command

– by performing a keyword search on the entire file system...

find /usr/tom –ctime -24


|

Performing keyword searches

string searches can be conducted on the logical file system or at the physical level to examine the contents of an entire drive

– a potential difficulty is that files may be compressed or encrypted

– some applications store their data in binary format

– so these files must be first uncompressed / decrypted / converted

useful keywords are case specific

examples:

– marijuana, dope, ...

– girls, teens, sex, ...

– ...

tools:

– use special programs such as Findwild

– use standard Linux grep command


|

Recovering deleted files

one of the most common tasks requested in any investigation is to

find and recover the files that have been deleted from the system

fortunately, most forensic analysis tools allow us to find, recover, and

examine many deleted items on a system

– when file systems are designed, different requirements are considered

– for end-user systems, the top two priorities are usually speed and

throughput

– so when a deletion takes place, data is not really wiped (that would take

time), but only meta-data on disk is modified to mark the clusters used by

the deleted file as unused (this is much faster)

– analysis tools rely on this and look for remains of files in unallocated disk

space


|

Summary on incident investigation

the goal of incident investigation is to determine the who, what, when, where, how, and why surrounding an incident

it has two phases: data collection and forensic analasys

types of data collected:– host-based information

» live data collection

» forensic duplication

– network-based information

– other information

data collection requires appropriate tools and careful procedures– live data collector toolkit

– forensic duplication

analysis of forensic duplicates can be carried out at two levels:disk level and file system level

forensic analysis can be based on standard (e.g., grep comand) and special tools (e.g., DFF) for searching disk or file system content,recovering deleted files, review and analyze logs, ...


INCIDENT RESPONSE IN ICS/SCADA

|

General challenges in ICS/SCADA

Impact of an incident may be cross-domain

Availability and safety concerns

Legacy equipment

Proprietary systems and protocols

Lack of ICS specific security incident response tools


|

Containment challenges

Containment of the compromise may require fast system

reconfiguration

– Network segmentation, firewall reconfiguration

– Disconnection of infected hosts/devices, disabling services

System reconfigurations in ICS/SCADA

– may not be allowed at all

– should preserve safety

– should have no adverse effect on the controlled processes


|

Recovery challenges

System recovery should be possible from backups, golden

images, and known good configurations (if available)

However, system recovery

– may require stopping and re-starting physical processes

– may take a considerable amount of time

– should be carefully scheduled

– should take into account interdependence between

different sub-systems

» requirements on the order in which sub-systems can be re-started

» requirements on consistent global state of the entire system


|

Data collection challenges

In ICS/SCADA systems, logging focuses mostly on process monitoring and troubleshooting, not on incident response

– Legacy equipment may not create any useful logs

– Field devices (e.g., sensors) typically don’t create any logs

– Network traffic is typically not logged

– PLCs may have proprietary or non-standard log format

– IT equipment may be modified in order to serve the system in the most efficient way --» logging may be switched off

– Logging may have adverse effects on system performance (availability, real-time guarantees)

– Logging would need storage capacity in PLCs and field devices

– Network based log data gathering increases network load

– Gathering log information manually may be slow and difficult


|

Ingredients of a solution

Up-to-date asset inventory, network topology, dependency

graph --» makes reconfiguration easier

Backups, golden images, proper configuration management --»

makes system recovery easier

Passive network monitoring --» makes intrusion detection and

data collection possible even in legacy systems

Physically separated management network for log gathering

and orchestration of security controls --» to avoid interference

with the system and increased network load


Incident Response -...

Documents

Transcript of Incident Response -...