44CON London 2015 - 15-Minute Linux Incident Response Live Analysis
-
Upload
44con -
Category
Technology
-
view
643 -
download
1
Transcript of 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis
![Page 1: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/1.jpg)
15-Minute Linux Live Analysis
Dr. Phil (Polstra)Bloomsburg University of Pennsylvania
![Page 2: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/2.jpg)
What is this talk about?
● Determining with some certainty if you have been hacked● In a matter of minutes● With minimal disturbance to the subject system● Automating the process with shell scripting
![Page 3: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/3.jpg)
Why should you care?
● Someone calls you about a suspected breach● You need to need to figure out if they were hacked
● Quickly so as to avoid further harm to your client● Without destroying evidence● Without taking down a critical machine
![Page 4: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/4.jpg)
Who is the handsome man before you?
● Digital forensics professor at Bloomsburg University● Programming since age 8 (in Assembly since age 10)● Known to hack electronics from age 12● Linux user from its start● Author USB
Forensics
Insert coolCover art
![Page 5: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/5.jpg)
Roadmap
● Opening a case● Talking to the users● Mounting known-good binaries● Minimizing disturbance to the system● Collecting Data● Automation with scripting● Next steps if there is a breach
![Page 6: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/6.jpg)
Opening a Case
● Decide on case name (date?)● Create a case folder on your laptop● Start making entries in your notebook
● Bound notebook with numbered pages● Easy to carry● Hard to insert/remove pages● No batteries required
![Page 7: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/7.jpg)
First Talk to the Users
● They know the situation better than you● Might be able to tell a false alarm before digging in● Why did you call me?
● What they suspect● No internal experts or policy to use outsider?
● Why do you think there was an incident?● Everything they know about the subject system
![Page 8: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/8.jpg)
USB Response Drive
● Contains known-good binaries● Minimum /bin, /sbin, /lib for same architecture● Might also grab /usr/sbin, /usr/bin, /usr/lib● Must be on an ext2, ext3, or ext4 partition
● Could contain a bootable Linux on another partition● This partition will probably be FAT ● Should be first partition● See http://linuxforensicsbook.com/hdvideos.html Chapter 1
![Page 9: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/9.jpg)
Mounting Known-Good Binaries
● Insert response drive● Exec your bash binary● Set path to your binaries (and only your binaries)● Set LD_LIBRARY_PATH● Run all shell scripts as bash <script>
● Don't use she-bang (#!) in scripts!
![Page 10: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/10.jpg)
Demo: Mounting Binaries
![Page 11: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/11.jpg)
Minimize Disturbance to System
● You will always change the system a little● Goal is to
● Minimize memory footprint● Never write to subject media
● Two basic options● Save everything to your USB response drive● Send it over the network
![Page 12: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/12.jpg)
Sending data over the network
● Better than USB drive due to caching● Use netcat
● Create a listener for “log” information on forensics workstation● Send “log” information from client● Also create a listener for interesting files on forensics
workstation– Spawn a new listener when files are sent
![Page 13: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/13.jpg)
Setting Up Log Listener
● netcat -k -l 9999 >> case-log.txt● (-k) keep alive (-l) listen (>>) append● From subject
● {command} | netcat {forensic ws IP} 9999
● Let's use shell scripting to automate this● Shell not Python because we want to minimize memory
footprint
![Page 14: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/14.jpg)
Automating the Log Listener
usage () {
echo "usage: $0 <case number>"
echo "Simple script to create case folder and start listeners"
exit 1
}
if [ $# -lt 1 ] ; then
usage
else
echo "Starting case $1"
fi
#if the directory doesn't exist create it
if [ ! -d $1 ] ; then
mkdir $1
fi
# create the log listener
`nc -k -l 4444 >> $1/log.txt` &
echo "Started log listener for case $1 on $(date)" | nc localhost 4444
# start the file listener
`./start-file-listener.sh $1` &
![Page 15: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/15.jpg)
Automating the Log Client-Part 1
usage () {
echo "usage: $0 <IP> [log port] [fn port] [ft port]"
exit 1
}
# did you specify a file?
if [ $# -lt 1 ] ; then
usage
fi
export RHOST=$1
if [ $# -gt 1 ] ; then
export RPORT=$2
else
export RPORT=4444
fi
if [ $# -gt 2 ] ; then
export RFPORT=$3
else
export RFPORT=5555
fi
if [ $# -gt 3 ] ; then
export RFTPORT=$4
else
export RFTPORT=5556
fi
![Page 16: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/16.jpg)
Automating the Log Client – Part 2
# defaults primarily for testing
[ -z "$RHOST" ] && { export RHOST=localhost; }
[ -z "$RPORT" ] && { export RPORT=4444; }
usage () {
echo "usage: $0 <command or script>"
echo "Simple script to send a log entry to listener"
exit 1
}
# did you specify a command?
if [ $# -lt 1 ] ; then
usage
else
echo -e "++++Sending log for $@ at $(date) ++++\n $($@) \n----end----\n" | nc $RHOST $RPORT
fi
![Page 17: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/17.jpg)
Automating Sending Files
● Listener on forensics workstation listens for file name● When a new file name is received
● Create a new listener to receive file● Redirect file to one with correct name● Also log in the main case log (optional)
● On the client side● File name is sent● After brief pause send file to data listener port
![Page 18: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/18.jpg)
Automating the File Listener
usage () {
echo "usage: $0 <case name>"
echo "Simple script to start a file listener"
exit 1
}
# did you specify a file?
if [ $# -lt 1 ] ; then
usage
fi
while true
do
filename=$(nc -l 5555)
nc -l 5556 > $1/$(basename $filename)
done
![Page 19: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/19.jpg)
Automating the File Client
# defaults primarily for testing
[ -z "$RHOST" ] && { export RHOST=localhost; }
[ -z "$RPORT" ] && { export RPORT=4444; }
[ -z "$RFPORT" ] && { export RFPORT=5555; }
[ -z "$RFTPORT" ] && { export RFTPORT=5556; }
usage () {
echo "usage: $0 <filename>"
echo "Simple script to send a file to listener"
exit 1
}
# did you specify a file?
if [ $# -lt 1 ] ; then
usage
fi
#log it
echo "Attempting to send file $1 at $(date)" | nc $RHOST $RPORT
#send name
echo $(basename $1) | nc $RHOST $RFPORT
#give it time
sleep 5
nc $RHOST $RFTPORT < $1
![Page 20: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/20.jpg)
Cleaning Up
# close the case and clean up the listeners
echo "Shutting down listeners at $(date) at user request" | nc localhost 4444
killall start-case.sh
killall start-file-listener.sh
killall nc
![Page 21: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/21.jpg)
Collecting Data
● Date (date)● Clock skew on subject● Time zone on subject
● Kernel version (uname -a)● Needed for memory analysis● Might be useful for researching vulnerabilities
![Page 22: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/22.jpg)
Collecting Data (continued)
● Network interfaces (ifconfig -a)● Any new interfaces?● Strange addresses assigned?
● Network connections (netstat -anp)● Connects to suspicious Internet addresses?● Strange localhost connections?● Suspicious ports?● Programs on wrong ports (i.e malware on port 80)
![Page 23: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/23.jpg)
Collecting Data (continued)
● Open files (lsof -V)● What programs are using certain files/ports● Might fail if malware installed
● Running processes (ps -ef and/or ps -aux)● Things run as root that shouldn't be● No login accounts logged in and running things● Might fail if malware installed
![Page 24: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/24.jpg)
Collecting Data (continued)
● Routing info (netstat -rn and route)● Routed through new interface● New gateways● Conflicting results = malware● Failure to run = malware
![Page 25: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/25.jpg)
Collecting Data (continued)
● Mounted filesystems (mount and df)● Things mounted that shouldn't be (especially tempfs)● Mount options that have changed● Filesystem filling up● Disagreement = malware
● Kernel modules (lsmod)● New device drivers● Modules that have changed
![Page 26: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/26.jpg)
Collecting Data (continued)
● Who is logged in now (w)● System accounts that shouldn't be allowed to login
● Who has been logging in (last)● System accounts that shouldn't be allowed to login● Accounts that don't normally use this machine
● Failed logins (lastb)● Repeated failures then success = password cracked
![Page 27: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/27.jpg)
Collecting Data (continued)
● User login info (send /etc/passwd and /etc/shadow)● Newly created login● System accounts with shells and home directories● Accounts with ID 0● Accounts with passwords that shouldn't be there
![Page 28: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/28.jpg)
Putting It Together with a Script
usage () {
echo "usage: $0 [listening host]"
echo "Simple script to send a log entry to listener"
exit 1
}
# did you specify a listener IP?
if [ $# -gt 1 ] || [ "$1" == "--help" ] ; then
usage
fi
# did you specify a listener IP?
if [ "$1" != "" ] ; then
source setup-client.sh $1
fi
# now collect some info!
send-log.sh date
send-log.sh uname -a
send-log.sh ifconfig -a
send-log.sh netstat -anp
send-log.sh lsof -V
send-log.sh ps -ef
send-log.sh netstat -rn
send-log.sh route
send-log.sh lsmod
send-log.sh df
send-log.sh mount
send-log.sh w
send-log.sh last
send-log.sh lastb
send-log.sh cat /etc/passwd
send-log.sh cat /etc/shadow
![Page 29: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/29.jpg)
Running the Initial Scan
![Page 30: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/30.jpg)
Have I been hacked?
![Page 31: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/31.jpg)
Who is Johnn?
/etc/passwd
![Page 32: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/32.jpg)
Why do these accounts have passwords?
/etc/shadow
![Page 33: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/33.jpg)
Who's been logging in?Results from last
![Page 34: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/34.jpg)
Who failed to login?
Results from lastb
![Page 35: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/35.jpg)
Looks Like They Were HackedNow What?
![Page 36: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/36.jpg)
Live Analysis
● Use techniques described to● Grab file metadata for key directories (/sbin, /bin, etc.)● Grab users' command history● Get system log files● Get hashes of suspicious files
● Dump RAM● Must compile LiME (off subject machine!)● Risky – can cause a crash
![Page 37: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/37.jpg)
Dead Analysis
● Unless the machine absolutely cannot be taken offline it is strongly recommended that you shut it down and get a filesystem image
● If you cannot shutdown the machine● You can still get a filesystem image with dcfldd● You probably cannot use this evidence in court
![Page 38: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/38.jpg)
More on Dead Analysis
● Filesystem analysis is much more mature and powerful than memory analysis
● The Linux support in Volatility is somewhat lacking● Relatively new addition to a new tool● Seems to fall down a lot with late 3.x and 4.x kernels
● None of the investigators I've talked to could come up with a case where evidence existed only in memory
![Page 39: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/39.jpg)
Finding Out More
● Heard there was a new book out (1 kg+ of knowledge)● Resources on http://linuxforensicsbook.com● Feel free to talk to me later
![Page 40: 44CON London 2015 - 15-Minute Linux Incident Response Live Analysis](https://reader031.fdocuments.in/reader031/viewer/2022030308/58ec8a251a28ab73208b4591/html5/thumbnails/40.jpg)
Questions?