• Which systems are affected?• Has any of the information stored on

your systems been modified and if so,what?

• What was the intruder aiming to do?It is quite often possible to answer ques-tions like these after examining theFirewall and IDS protocols – providedthe admin implemented a security sys-tem and adequate logging. Butcomprehensive monitoring of systemsand networks often conflicts with protec-tion of data privacy policies.

Forensic analysis of the affected sys-tems after the event is the only way toobtain clear evidence. The exact proce-dures are complicated and depend on anumber of factors. If you are looking fora comprehensive introduction to the sub-ject matter, there are a few documents –some of them available online – whichmay help [5]. The goal of any forensicinvestigation is to ascertain the eventsand their timeline: the series of activitiesthat led to the compromise and anyactions performed by the intruder on thevictim machines. If the results of thisanalysis are to be used as evidence at alater stage, there is also a requirementfor unambiguous documentation of theindividual investigative steps.

Performing the individual investigativesteps in a fixed order will ensure that theresults are reproducible: “Secure and iso-late the scene. Record the scene,Conduct a systematic search for evi-dence.” [6]

Firewalls and Intrusion DetectionSystems alert administrative usersand provide protection against

most attacks. But intruders continue tofind loopholes and attack machines. Theadministrator then acts like a coronerperforming forensic tests on the living(i.e. running) or deceased (i.e. poweredoff) system to salvage vital evidence.

Tools like TCT, the classic forensictoolkit (The Coroner’s Toolkit [4]) or thenewer Sleuthkit [1], which is the subjectof this article, help solve these problems.After a compromise, there are a numberof difficult questions to answer, such as:• How did the compromise occur?• What prevented the firewall from fend-

ing off the attack?• Why did the IDS not recognize the

attack (until it was too late)?

Sleuthkit performs forensic analysis on both Microsoft and UNIX filesystems,

applying its sleuth-like skills from the command line to identify evidence,

recover deleted files and reconstruct scenarios.

All of which play a vital role in the art of digital computer forensic work.

In this growing field it is better to prepare and practice for the problems

that lie in the future.

BY RALF SPENNEBERG

Sleuthkit, the Digital Forensic Toolkit

Detective Work

60 October 2003 www.linux-magazine.com

SleuthkitSYSADMIN

Ralf Spenneberg is afreelance Unix/Linuxtrainer and author.Last year saw therelease of his firstbook:“IntrusionDetection Systems forLinux Servers”. Ralfhas also developed various trainingmaterials.

TH

E A

UT

HO

R

Secure and IsolateThe first step in forensic analysis is asubject of much controversy. Mostexperts recommend disconnecting thesystem from the active network immedi-ately. This prevents the intruder fromrecognizing the analysis and covering histracks.

Some experts disagree with this; theintruder might have installed a hiddenscript that automatically wipes thesystem if it is disconnected from thenetwork. It is difficult to make a generalrecommendation, but I tend to recom-mend and use the former approach.

Recording the ScenarioWhen analyzing a compromised com-puter the forensic expert should firstlook to saving volatile information,

Once the enhanced loopback driver and themodified losetup command have beeninstalled you have to create the appropriatedevices using the supplied createdev com-mand.Then you can simply mount theimage of a whole harddisk using the follow-ing commands:

# losetup -r /dev/loopa host_Uhda.dd# fdisk -l /dev/loopaNow you can access the individual parti-tions using /dev/loopa1, /dev/loopa2, etc.Since the -r option sets up the loopbackdevice as read-only, even journaling file-systems may not write.

Enhanced Loopback Driver

Core Knowledge Foundation,w

ww

.coreknowledge.org

before moving on to non-volatile data.The following are volatile:• Main memory• Kernel messages• Time/date• Active processes• Open files• Network configuration• Network connectionsThis information is lost, when you downthe system, but it can be extremely valu-able. Normal Linux commands aretypically all you need to secure this dataand the grave-robber command from tctis useful. The command is not part ofSleuthkit. It is important to use programsfrom trustworthy sources when securingthe evidence; these could be staticallycompiled commands on a CD or a write-protected floppy.

Non-volatile data is next on the list.This means the filesystems and the swapmemory. You should additionally look torecording any documents in the vicinityand the general appearance of the sys-tem. A camera can be a useful tool fordocumentation purposes.

A simple backup is notsufficientA simple backup of the system is not suf-ficient. The filesystem copy must be

identical to the original; you might missfiles deleted by the intruder otherwise.The backup should also include slack-space. The dd tool for UNIX can performthis task; a Windows version is alsoavailable from [7].

The admin should use dd to backupthe whole hard disk and not just individ-ual partitions. When doing so, it isimportant not to store any data on theoriginal disk. Instead you should use anexternal hard disk, running the Netcattool, or its cryptographic counterpartCryptcat, or an SSH tunnel, to transferthe data to the forensic system on whichyou will test and examine.

Migrating the DataIf you use Netcat to transfer the contentof the disk, the Netcat listener must berunning on the target machine:

nc -l -p 3000 > host_hda.ddAnd then feed the hard disk data fromthe system you need to back up to Net-cat:

dd if=/dev/hda | nc Server 3000

Forensic investigators are well advised tocalculate a checksum for the disk imageimmediately after backing up. The

checksum later provides evidence thatthe backup has not been manipulated.

md5sum host_hda.dd

The next task is to divide the hard diskinto individual partitions, as the Sleuthkitcan only handle individual partitions.The fdisk -lu host_hda.dd command liststhe partitions (see Listing 1).

You can again use dd to extract theindividual partitions. To do so, you willneed to calculate the size of the partitionusing the start and end cylinders. In thecase of hda1 this amounts to 15 119 999 -63 + 1 = 15 119 937. dd can then usethis information to store the first parti-tion in a file of its own:

dd if=host_hda.dd of=host_hda1U.dd bs=512 skip=63 count=U15119937.

If these steps sound too complicated,you can back up individual partitionsfrom the outset, but this does imply the danger of missing informationneeded for forensic analysis if it is storedin non-partitioned areas. The enhancedloopback driver [8], which is capable of accessing the complete hard disk as a loopback device, is more practical

61www.linux-magazine.com October 2003

SYSADMINSleuthkit

The first forensic toolkit for UNIX systemswas written by Wietse Venema and DanFarmer in 1999 [5].The Coroner’s Toolkit (TCT)is a collection of commands that admins canuse to perform post-incidents forensics onUNIX computers.Wietse Venema still main-tains this toolkit and regularly offers updateson his homepage [4].TCT can analyze the fol-lowing operating systems: Sun Solaris,SunOS, Linux, FreeBSD, OpenBSD, andBSD/OS.Brian Carrier from Atstake (typically spelled

@stake [3]) started adding additional toolsto TCT at an early stage, releasing these add-ons as the TCT Utils.The TCT Utils offer arange of extended filesystem functions tothose found in TCT.Analyzing Non-UNIX FilesystemsEarly on in 2002 Brian Carrier implementedFAT and NTFS support to provide analyticalfacilities for addition filesystem types. Healso reworked the TCT sources and this wasreason enough to change the name to TASK,The Atstake Sleuth Kit. Brian additionally

released the Autopsy Forensic Browser.Despite its name Autopsy it is in fact a Webserver; admins performing such forensicwork may find Autopsy useful as a simpleGUI front-end for TASK commands.In April 2003 Brian Carrier again changed thename of the project to The Sleuthkit to markthe release of the current 1.61 version.Thisemphasizes the Open Source nature of theproject and its independence of the Atstakecorporation.The project files are availablefrom Sourceforge [1, 2].

TCT and Sleuthkit

Disk /dev/hda: 0 heads, 0 sectors, 0 cylindersUnits = sectors of 1 * 512 bytes

Device Boot Start End Blocks Id System/dev/hda1 63 15119999 7559968+ 83 Linux/dev/hda2 15120000 80408159 32644080 5 Extended/dev/hda5 15120063 16178399 529168+ 82 Linux swap/dev/hda6 16178463 24373439 4097488+ 83 Linux/dev/hda7 24373503 32568479 4097488+ 83 Linux/dev/hda8 32568543 80408159 23919808+ 83 Linux

Listing 1: Partitions for the first example

Figure 1: When the IDS reports an intruder (in this case Snort isusing Win-Popup to alert on an IMAP exploit), it is the admin’stask to initiate incident response procedures. Sleuthkit can helpanalyze the filesystems

vides a number of commands for thisstep – we will be looking at some of themajor tools in the following sections.The second part of this series describes ahands-on investigation using Autopsy.

Installation is extremely simple: down-load the package from the homepage,extract it, and call make. This places thetools in the bin/ subdirectory. TheAutopsy Forensic Browser is quite easy

to compile, however, you might prefer touse the RPM packages.

Sleuthkit tools are divided into fourcategories. The first category displaysinformation for complete filesystems andcontains only the fsstat command. Thesecond group starts with d, and allowsyou to access data stored in files: dcalc,dcat, dls, and dstat. The Sleuthkitprovides the following tools for meta-

than manual separation of the individualimages on the disk.

Systematic SearchingAfter storing the filesystem of the com-promised computer, creating checksumsand storing a copy in a safe place, theinvestigator can go on to analyze a copyof the filesystem, leaving the originaluntouched as evidence. Sleuthkit pro-

62 October 2003 www.linux-magazine.com

SleuthkitSYSADMIN

Filesystem Command line OptionBSDi FFS bsdiFAT Filesystem FAT12 fat12FAT Filesystem FAT16 fat16FAT Filesystem FAT32 fat32FreeBSD FFS freebsdLinux Filesystem EXT2 linux-ext2Linux Filesystem EXT3 linux-ext3NTFS ntfsOpenBSD FFS openbsdSolaris FFS solaris

Table 1: File Systems inSleuthkit

class|host|device|start_timeils|kermit.spenneberg.de|honeypot.hda5.dd|1052056153st_ino|st_alloc|st_uid|st_gid|st_mtime|st_atime|st_ctime|st_dtime|st_mode|st_nlink|st_size|st_block0|st_block11|a|0|0|973385737|973385737|973385737|0|0|0|0|0|01890|f|17275|20|973695367|973695367|973695367|973695367|40755|0|0|6172|01891|f|17275|20|846630170|973695173|973695367|973695367|100644|0|1082|6173|01892|f|17275|20|973695367|973695367|973695367|973695367|40755|0|0|6174|01893|f|17275|20|846630171|973695173|973695367|973695367|100644|0|1458|6175|01894|f|17275|20|846630171|973695173|973695367|973695367|100644|0|1362|6176|02243|f|17275|20|846630171|973695173|973695367|973695367|100644|0|1465|6177|0

Listing 2: Deleted Files

Restoring deleted files is by no means a triv-ial task on UNIX-type filesystems. I am notaware of an unerase command for restoringdeleted data without any danger of loss, nomatter what UNIX operating system youuse.Thus, some background knowledge ofhow the filesystem stores and deletes data isessential to run tools like TCT or Sleuthkit,and to cope with issues that may arise whenrestoring deleted files.

The Second Extended Filesystem, EXT2, is auseful example of a heritage Inode basedUNIX filesystem. Each file is represented by aspecial structure, its Inode.The Inode stores the meta-information belonging tothe file. Additionally, datablocks are required to storethe payload. Directories aresimply special files thatagain comprise an Inode.Their data blocks store thedirectory list which containsfilenames and links to theappropriate Inodes.

The Inode in EXT2

Inodes store the meta-infor-mation for a file except itsname.This includes the filesize, type, permissions, theowner and group, the refer-

ence counter and three UNIX timestamps(ctime, atime and mtime).The Inode alsocontains twelve direct references to datablocks, that is the addresses of the firsttwelve data blocks in the file.

Inodes use additional indirect references, thefirst of which points to a data block con-taining direct references to additional data blocks.The last two references pointto a double or triple indirect block (see Figure 2).

When the filesystem deletes a file, it simplymarks the directory entry and Inode as

deleted and frees up the space occupied bythe Inode and the data blocks.The referencesto the data blocks and the link between the filename and the Inode tend to remainunchanged.Thus, it is possible to use toolssuch as ils to display deleted Inodes and icat to restore the file content. However,this only works if the filesystem has notalready reassigned the Inode or data blocksto another file.The fls command shows the names of deleted files.

Modern Implementations Delete More

Many modern Linux distributions delete filesin a way that effectively pre-vents them from beingrestored. Unfortunately, thissecurity feature tends to im-pede forensic analysis withtools such as ils and icat. Inthis case, the admin per-forming the investigation istypically forced to resort tothe dls Sleuthkit commandwhich reads the slackspaceon the hard disk.The investi-gator can then use grep orthe sorter tool to investigatethe results. Autopsy is an ele-gant front-end that helpsout with this more complexinvestigative step.

Waking the Dead

Figure 2: The Inode (left) stores meta-information for a file. It references the data blockswith the file contents. Larger files may also require the use of indirect references

Datablock

Datablock

Datablock

Datablock

Datablock

Datablock

Datablock

Datablock

Datablock

Datablock

Datablock

dataMeta

Direct Double indirectIndirect

Triple indirect

Inode

information stored in Inodes:icat, ifind, ils, and istat. Thecommands in the fourth groupstart with f, and are designed for file level tasks such as creat-ing file lists and searching forfiles.

The mactime command dis-plays events chronologically ona system. The file and sortercommands sort the files in ahard disk image by type.

Supported FilesystemsSleuthkit analyzes the filesystem typesshown in Table 1. Unfortunately, it doesnot support typical Linux filesystems likeReiser FS, JFS, and XFS. The “Wakingthe Dead” insert describes how UNIXfilesystems work, indicating the informa-tion on which tools are oriented, andpointing out their limitations.

EXT2 and EXT3 can produce someunpleasant side effects with moremodern distributions. Depending onwhat patches you have applied, these

filesystems delete the reference to thedata blocks from the Inode, and thisprevents simple tools like icat fromrestoring the content of a deleted Inode.

Practical Hints: ReanimationNow it is time to analyze the hard diskimages you backed up. Our examplesuse the Forensic Challenge filesystems.This competition was organized by theHoneynet project in January 2001. A tar-ball containing the data is available fordownloading from various addresses [9].

Since the partitions havealready been separated, wecould not use these files to illus-trate the first step of backing upthe harddisk.

After downloading, the bud-ding forensic scientist will need to extract the data and verifythe checksums. The latter arelocated in the readme file. Thefollowing command extracts theMD5 sums and verifies themagainst the individual images:

# tail +12 readme | head U

-6 | md5sum -choneypot.hda1.dd: Okhoneypot.hda5.dd: Okhoneypot.hda6.dd: Okhoneypot.hda7.dd: Okhoneypot.hda8.dd: Okhoneypot.hda9.dd: Ok

The function of the individual partitionsis supplied as background knowledge(see Table 2). We will be mounting

SYSADMINSleuthkit

Figure 3: In order to analyze the modification order across multiplefilesystems, the investigator needs to collate file and Inode informa-tion in the body file

p-p-p-pick upa dedicated linux server

EasyVserver solutionsDebian or RedHat O/S

True “root” access4, 6 or 8GB raid space

1IP addressHighly secure

Established in 1997 with over 1/2 million customers. Accredited ICANN registrar & nominet member. Prices exclude VAT.

EASY TO BUY • EASY TO SET UP • EASY TO SEE

Make the most of Linux technology with the big name in Web registration and hosting

packages. Our flexible, scalable, secure EasyVserver solutions start at just £39 per

month. Back up by unrivalled support and know-how. If you want the best of Linux

come along for the ride.

log on today at: www.easyspace.com

obviously run on the host kermit.spenneberg.de for a file called honey-pot.hda5.dd. Things start to becomemore interesting in line 3; this is thesecond header for the table and is usedfor any remaining lines. The firstcolumn, st_ino contains the Inode num-ber, column two, (st_alloc), indicateswhether the Inode is free (f) or allo-cated (a).

There are many approaches to discov-ering interesting files. The first approachassumes any files with a non-zero size(column 11, st_size). These files areinvestigated using the file command, inorder to categorize the file content. To dothis, the tool accesses a large databasecontaining nearly every known file for-mat (magic*). The Sleuthkit version offile is no different from the versionincluded with many distributions,although the database may be larger.

Interesting FilesListing 3 shows the possible syntax. Line 1 cuts off the first four lines of theInode information output by the ils com-mand, and passes the rest to an awkcommand (line 2). This ascertains anyInodes with a non-zero size, and passesthe Inode numbers to the icat command.The latter extracts the files from theimage and stores them in a subdirectorydata/hda5.icat/ using the Inode numberas the filename.

The file command is then used toascertain the file type, and egrep filtersout the interesting files (line 7). Aninvestigation on the first tarball (line 15)shows that we have discovered SSH Ver-sion 1.27.

Logbook Stardate 2000-11-07The second approach to analyzingdeleted files involves the forensic investi-gator first creating a chronologicaljournal of the file operations on the sys-tem. The journal will cover any files,rather than just deleted files. Addition-ally, it makes sense to cover all thefilesystems with this analysis.

The admin first collects data from thefilesystems (see Figure 3). In addition tothe familiar ils command, this involvesusing fls. This command collects infor-mation on any files that still exist on thefilesystem. The -m option creates output,

the partitions as read-only using the loopback device to provide an initialoverview – but this can be dangerousand may modify the data, see the “Read-only Write Sometimes” insert.

Sleuthkit commands provide a more in depth look. The deleted files are thefirst point of interest – if the attacker has attempted to cover his tracks, this is often the best way of uncoveringthem.

Inodes of Deleted FilesThe ils command displays Inode information for a filesystem; the -r flagtells the command to concentrate ondeleted Inodes (default). Listing 2 showsthe results for ils -f linux-ext2 -rhoneypot.hda5.dd.

Output is in table format, with line 1containing the header and line 2 the cor-responding data. The command was

64 October 2003 www.linux-magazine.com

SleuthkitSYSADMIN

ed Nov 08 2000 14:51:53 17969 .a. -/-rwxr-xr-x 1010 100 109832 /usr/man/.Ci/scan/x/x1760 .a. -/-rwxr-xr-x 1010 100 109829 /usr/man/.Ci/scan/bind/ibind.sh15092 .a. -/-rwxr-xr-x 1010 100 109836 /usr/man/.Ci/scan/x/pscan4096 .a. d/drwxr-xr-x 1010 100 109841 /usr/man/.Ci/scan/port/strobe1259 .a. -/-rwxr-xr-x 1010 100 109834 /usr/man/.Ci/scan/x/xfil4096 .a. d/drwxr-xr-x 1010 100 109831 /usr/man/.Ci/scan/x

[...]Wed Nov 08 2000 14:52:09 9 m.c l/lrwxrwxrwx 0 0 46636 /root/.bash_history -> /dev/null

9 m.c l/lrwxrwxrwx 0 0 23 /.bash_history -> /dev/null[...]

Listing 4: Mactime summary

When performing forensic analysis it isoften useful to mount the hard disk parti-tion to look for files.This particularly appliesto Reiser FS, as the Sleuthkit does not sup-port the Reiser filesystem.To ensure dataintegrity, investigators should generatechecksums for all filesystems and mountthem as read-only.Unfortunately read-only cannot always betaken at face value: non-journaling filesys-tems really don’t change anything if they aremounted read-only. However, journalingfilesystems like EXT3 or Reiser FS tend toupdate the journal even for read-only mountops. Of course, this changes the MD5 check-sum.There is a simple test for this:You willneed a test image and its checksum for thetest:

# dd if=/dev/zeroof=/tmp/testimage.orig U

bs=1024k count=50

# losetup /dev/loop0/tmp/testimage.orig# mkreiserfs /dev/loop0# md5sum /tmp/testimage.orig2ceed9e819bf4348a33a21f7697149c8U/tmp/testimage.orig

After mounting read-only and immediatelyunmounting, the MD5 sum has changed:

# mount -o ro -t reiserfs/dev/loop0 /mnt# umount /mnt# md5sum /tmp/testimage.origc6ffc8a13a6821cd327d1db9ee351fafU/tmp/testimage.orig

There are only three ways to avoid this atpresent:• Modify the EXT3 or Reiser FS driver

• Save the filesystem on a CD-ROM

• Use the enhanced loopback driver [8]

Read-only, Write Sometimes

Partition Filesystem/dev/hda8 //dev/hda1 /boot/dev/hda6 /home/dev/hda5 /usr/dev/hda7 /var/dev/hda9 swap

Table 2: ChallengePartitions

which in turn is processedby the mactime command.The original mount point is also output. The -m flag is also required for ils,although the mount point is not.

The file used to collectthe mass of data is calledbody for historical reasons.Calling mactime for this file generates a logbook of filesystem modifications.You can restrict the outputto a period you are inter-ested in. In our exampletaken from the ForensicChallenge, we can work on the assumption that the attack took place onNovember 7 2000.

Consistent TimekeepingTo provide consistent timekeeping, youshould use the -z flag to specify the time-zone used by the system that created thedata. In this case, it is CST (GMT-0600).The output shows unusual activities at14:51 on November 8 (see Listing 4): The/usr/man/.Ci/ directory should not reallybe there.

Files with content changes are also ofinterest. These files are easily recognized

by the m in the third column. An a indi-cates access to the file, and c shows thatthe meta-information (permissions,owner, …) has been modified. You cansee where compilers have been calledand libraries linked, which allows you toreconstruct the steps the attacker took.

There is one drawback to thisapproach. If the attacker has modified afile more than once, the Inode will onlysave the date of the last modification.The mactime tool will thus display onlythe last modification.

GUI to the RescueSleuthkit commands can bequite difficult to use, andthis makes it hard to keepan eye on the results.Searching for specificstrings on all filesystemsand analyzing the contentsof deleted files can betedious, and may requiresome extensive scripting.

Given these circum-stances a GUI can be quiteuseful when performingforensic analysis. It pro-vides an abstraction layerfor the commands and pre-sents only the results,meaning that forensicinvestigators does not need

to consult manpages to achieve results.The Autopsy Forensic Browser (see

Figure 4) provides a GUI of this type. Italso allows the investigator to documentand comment on data and results. Undersome specific circumstance it may stillmake sense or be necessary to resort tothe command line, but investigators willbe able to perform major steps fromwithin the GUI.

Part 2 of this series will be looking atthe Autopsy Forensic Browser using theForensic Challenge examples. ■

65www.linux-magazine.com October 2003

SYSADMINSleuthkit

# ils -f linux-ext2 -r honeypot.hda5.dd | tail +4 |> awk -F '|' '$11 > 0 {print $1}' |> while read in; do> icat honeypot.hda5.dd $in > data/hda5.icat/$in> done

# file data/hda5.icat/* | egrep "tar|gzip|RPM|ELF"data/hda5.icat/109791: GNU tar archivedata/hda5.icat/109861: GNU tar archivedata/hda5.icat/109865: RPM v3 bin i386 nfs-utils-0.1.9.1-1data/hda5.icat/109866: RPM v3 bin i386 wu-ftpd-2.6.0-14.6xdata/hda5.icat/109943: ELF 32-bit LSB relocatable, Intel 80386, version1 (SYSV), not strippeddata/hda5.icat/109944: ELF 32-bit LSB relocatable, Intel 80386, version1 (SYSV), not stripped

# tar tf data/hda5.icat/109791ssh-1.2.27/ssh-1.2.27/COPYINGssh-1.2.27/READMEssh-1.2.27/README.SECURID[...]

Listing 3: Discovering Interesting Data[1] Sleuthkit: http://www.sleuthkit.org

[2] Autopsy Forensic Browser:http://autopsy.sf.net

[3] Atstake:http://www.atstake.com

[4] The Coroner’s Toolkit:http://www.porcupine.org/forensics/tct.html

[5] Dan Farmer and Wietse Venema,“Computer Forensic Analysis”:http://www.porcupine.org/forensics/

[6] Richard E. Saferstein,“Criminalistics: An Introduction to Forensic Science”, Prentice Hall

[7] dd for Windows – Unx-Utils:http://unxutils.sf.net/

[8] Enhanced Loopback:ftp://ftp.hq.nasa.gov/pub/ig/ccd/enhanced_loopback/

[9] Forensic Challenge files:http://project.honeynet.org/challenge/images.html

INFO

Figure 4: The Autopsy Forensic Browser provides a Web front-end for Sleuthkitand facilitates filesystem analysis