d1 02 Carrier Slides

1

An Investigator’s Guide to File System Internals

Brian [email protected]

@stake

? 2002 @stake, Inc. FIRST File System Tutorial2

Overview

• Motivation

• Disk Layout (MBR and Partitions)

• Basic File System Model

• Classical File Systems

– FAT (16-bit and 32-bit)

– UNIX (FFS and EXT2FS)

• Journaling File Systems

– NTFS

– EXT3FS

• Other File Systems

2

@stake


Motivation

• Currently, most evidence is found in the file system

• A thorough knowledge of the file systems allow one to:

– Make better use of forensics tools

– Recover deleted content when tools do not exist

– Allow one to describe what your automated tools are doing

• This talk is going to cover basic & advanced topics of file systems for:

– Manual file recovery

– Finding evidence of “wiping” tools

– Knowing how to most efficiently use tools

@stake


What is a File System?

• File Systems manage data storage.

• Organized into files, unlike memory.

• Provide an addressing scheme that is easy for humans to understand, unlike memory.

• Examples: FAT, EXT2FS, FFS, NTFS, Reiser FS,

3

@stake


Tools

• The @stake Sleuth Kit (TASK) will be used throughout this to show examples on real images.

• The Coroner’s Toolkit (TCT) and TCTUTILs could also be used to collect some of the UNIX file system data.

• TASK is a collection of forensics command line tools for file system analysis

• The tools parse file system structures and do not rely on kernel support

@stake


Disk Layout (MBR and Partitions)

4

@stake


The Disk

• A disk can be imagined as a long stream of millions bytes

• A disk is typically broken up into one or more groups (or partitions)

• Logical volume management systems do not directly follow this model because they can have several physical disks appear as one volume, which is then broken up into partitions.

• We will cover x86 (DOS) partitions and disk labels

@stake


x86: Disk Layout

• The bytes are grouped into sectors, usually 512-bytes each

• Each sector is addressed, using Logical Block Addressing (LBA) or Cylinder / Head / Sector (CHS)

• The first sector contains the Master Boot Record (MBR)and the Partition Table

• The remaining sectors are organized into Partitions, which contain file systems

5

@stake


x86: Master Boot Record (MBR)

• Located in first 512-bytes of disk

• The first 446 bytes are x86 assembly instructions that read the partition table to identify a bootable partition

• This is the sector that the systems reads when booting

• The Partition Table starts at offset 446 and contains four 16-byte entries (48 bytes total)

• The final 2-bytes contain a signature (0x55AA)

@stake


x86: Partition Table

• Contains 4 entries, each can describe a partition

• Entries are each 16-bytes long

• Entries contain:

– Starting address of partition in CHS and LBA format

– Ending address of partition in CHS format

– Size of partition in sectors

– Flags (is it bootable?)

– Type of Partition

6

@stake


x86: Partition Types

• There are more than one partition type because only four entries exist in the MBR Partition Table.

• Primary Partition: Contains 1 file system and can only be described by one the four MBR partition table entries.

• Extended Partition: Contains 0 or 1 Secondary Partitions and 0 or 1 additional Extended Partitions (a linked list). Used when more than 4 partitions are needed.

• Secondary Partition: Contains 1 file system and occupies either all or part of an Extended Partition.

@stake


x86: Partition Example

ExtendedSec(FAT)

Secondary(NTFS)

Primary(NTFS)

Primary(EXT2FS)

Primary(EXT2FS)

ExtendedMBR

Disk

7

@stake


x86: Partition Tools

• Displaying:

– fdisk

– Partition Dump (pd) <http://heat.ca.sandia.gov/projects/software.html>

– GNU Parted <http://www.gnu.org/software/parted/>

• Finding partitions when the table is gone

– gpart <http://www.stud.uni-hannover.de/user/76201/gpart/>

@stake


disk labels: Overview

• BSD systems use disk labels to describe a disk

• The disk label contains the disk layout (partitions), disk geometry, label, model etc. (but no boot code)

• It is located somewhere in the “beginning” of the disk

• There is not a standard format, but the contents include:

– Offset (in sectors or cylinders)

– Size

– Type

– Flags

8

@stake


disk labels: x86-based BSD flavors

• x86-based BSD systems (i.e. OpenBSD, FreeBSD) use both x86 partitions and disk labels

• An x86 partition is created as the primary partition for the BSD system (so that it can be booted)

• The disk label is located in the beginning of the BSD primary partition and is used by the kernel to identify where the partitions exist

• The disk label can describe partitions that do not exist in the x86 partition tables

• The labels can be viewed with the ‘disklabel’ command.

@stake


Windows XP OpenBSD

a b d ef

•

disk labels: x86-based BSD flavors example

• Only two x86 partitions are defined

• The BSD disk label is located at the beginning of the OpenBSD partition (a)

• The disk label describes 5 partitions (partition f has an offset earlier than it does)

9

@stake


disk labels: Sun

• Sector 0 of the disk (slice 2) contains the Virtual Table Of Contents (VTOC) and partition descriptors:

– ID (boot, root, unassigned, swap, usr, backup, stand, var, home etc.)

– Permission Flag (not -mountable, Read Only)

– Starting Sector (or cylinder in some versions)

– Size of partition

• Maximum of 8 or 16 partitions

• Sectors 1-15 contain the bootblock code

• The contents can be listed with:

prtvtoc /dev/rdsk/c0t0d0s2

• The format command can also display the partition details

@stake


Basic File System Model

10

@stake


Very Basic File System Model

• In general, data within a file system can fall into 1 of 4 categories

• File System Layer

– Descriptive data about the file system (i.e. block sizes, offsets, dirty status)

• Content Layer

– File content (i.e. blocks, clusters). Most data falls into this category.

• Meta Data Layer

– Descriptive data about files (i.e. file size, permissions, time of last access)

• Name Layer

– Human addressing format (i.e. file and directory names)

@stake


TASK tool names

• Each tool in TASK is named based on the file system layers:

• File System Layer:

– fsstat

• Content Layer (data):

– dls, dcalc, dstat

• Meta Data Layer (inode):

– ils, istat, icat, ifind

• Name Layer (file name)

– fls, ffind

11

@stake


FAT

@stake


FAT12, FAT16, and FAT32 File Systems

• Primitive File System – has been used since early DOS versions

• Still used in Windows 9x and on floppies

• Size limits of file system:

– FAT12: 32 megabytes (used in floppy disks)

– FAT16: 4 gigabytes

– FAT32: 8 terabytes

12

@stake


FAT: File System Layout

• The file system is organized into 3 areas:

• Boot Sector

– Contains file system details such as sizes and offsets

• Data Area

– Contains file and directory contents. Consecutive sectors in it are grouped into clusters.

• Primary and Backup File Allocation Tables (FAT)

– Contains tables that manage the clusters in the Data Area

@stake


FAT: File System Layout Diagram

Boot Sector

File Allocation Tables

Data Area

Cluster

13

@stake


FAT: Data Area

• All file and directory content is stored in clusters within the Data Area

• Each cluster is 1 or more (power of 2) consecutive 512-byte sectors. The size of which is saved in the Super Block.

• Each cluster has an address and the first usable one is number 2

• The File Allocation Table (FAT) manages the clusters

@stake


FAT: File Allocation Table

• The File Allocation Table (FAT) contains as many entries as there are clusters in the Data Area

• The table is indexed by the cluster number and the contents describe that cluster

• Table entries can be:

– An unallocated marker, used when we need to allocate new clusters

– The address of the next cluster in a file or directory

– End of File (EOF) marker if it is the last cluster in the file or directory

– Bad sector marker, so it is not reallocated

• The size of each table entry depends on the version of FAT (12-bits, 16-bits or 32-bits)

14

@stake


FAT: File Allocation Table Example

0 1 2 3 4 8 9

0 0 3 0 8 9 E

• Files are located in Clusters 2, 3, 4, 5 and 7, 8, 9• Cluster 6 is unallocated• E is our End-Of-File marker

4 5 E

5 6 7

@stake


FAT: Directory Entries

• File and directory meta data are stored in Directory Entry structures

• One directory entry is allocated for each file and directory

• Directory entries have a static size (32-bytes) and form a list within the clusters allocated by the parent directory

• The clusters for the Root Directory (C:\) are statically located in FAT12 and FAT16 at the first cluster (#2) and are dynamically located in FAT32

• The Root Directory does not have a directory entry (i.e. it has no meta data associated with it)

15

@stake


FAT: Directory Entry Structure

• name in ASCII (8 bytes)

• extension in ASCII (3 bytes)

• attribute flags (1 byte) [Directory, Read Only, Hidden, System, Volume ID, Archive, Long Name Entry]

• lowercase flags (1 byte)

• Create time & date (5 bytes) (optional, and is 0 on some Windows9X directories)

• Access date (2 bytes) (no time, but is required by spec)

• Written time & date (4 bytes) (optional)

• Address of the first allocated cluster (4 bytes total)

• Size of file (4 bytes)

@stake


FAT: Date and Time structures

• FAT stores the actual time, not the number of seconds from a known date

• Lower 5-bits (0x001f) of time are seconds (0-29) divided by 2

• Next 6-bits (0x07e0) of time are minutes (0-59)

• Upper 5-bits (0xf800) of time are hours (0-23)

• Lower 5-bits (0x001f) of date are day (1-31)

• Next 4-bits (0x01e0) of date are month (1-12)

• Upper 7-bits (0xfe00) of date are year from 1980 (0-127)

16

@stake


FAT: Long File Name Directory Entries

• The Directory Entry contains the file name, but is constrained to the 8.3 format.

• Long File Name (LFN) Directory Entries are the same size as normal directory entries, but have a special value in the attribute flag (all bits are set).

• Older versions of DOS happen to ignore directory entries with this flag.

• Files with long names also have a short name directory entry after the LFN directory entries

• Each LFN can hold 13 UNICODE letters, multiple ones exist for long names.

@stake


FAT: LFN Structure

• sequence (1 byte) [EOF or linear value]

• 5 UNICODE letters of name (10 bytes)

• attributes flag (1 byte)

• reserved (1 byte)

• checksum (1 byte) (equal for all LFN of a file)


• reserved (2 bytes)


17

@stake


FAT: LFN Example

abcdefghijklmnopqrstuvwxyz

1234.txt

abcdef~1.txt

LFN

LFNLFN

FILE

File is: abcdefghijklmnopqrstuvwxyz1234.txt

Name Attribute

NOTES: Stored in “reverse” order and the SFN entry is theonly one with the starting sector and other meta data

@stake


FAT: Example

5 6 7 8 9 13 14

6 E E 0 0 14 E9 10 E

10 11 12

file1 - 5

cluster 5

dir1 - 7foo.c - 8

bar.h - 13

Root Dir

FAT

FileContent

FileContent

FileContent

FileContent

cluster 8

cluster 13

cluster 7

cluster 6

18

@stake


FAT: File Lookup Example

• Examine contents of c:\dir1\file1.dat

1. Identify location of the Root Directory sectors (from the super block)

2. Read the directory entries in the root directory sectors, looking for one with the name “dir1” and a directory attribute flag

3. When the entry is found, identify the starting cluster (16 for example)

4. Read cluster 16 and process the data as directory entries, looking for the entry with “file1” as the name and “dat” as the extension.

@stake


FAT: File Lookup Example

5. If an entry is not found, look in entry 16 in the FAT to identify the next cluster.

6. Repeat the lookups until the entry is found. Identify the starting cluster from the directory entry (23 for example).

7. Read the contents of cluster 23 and then lookup entry 23 in the FAT for the next cluster, 24 for example.

8. Read the contents of 24 and repeat the FAT lookups until <EOF> is reached

19

@stake


FAT: File System Layer

• Two structures describe the file system

• The Boot Sector structure describes file system attributes. The structure is different in FAT16 and FAT32.

– Cluster size

– Location and size of the FAT

– Start and end of the Data Area

– Magic value (0xAA55)

• The File Allocation Table (FAT) manages the clusters within the Data Area

@stake


FAT: TASK & sectors

• TASK uses sectors for addressing instead of clusters

• The first cluster is hundreds of sectors into the file system, after the Super Block and FATs

• Therefore, sectors are used as addresses so that we can access the contents of the FAT and Super Block

20

@stake


FAT: fsstat

• The fsstat tool lists the important Boot Sector values and a summary of the FAT.

# fsstat –f fat fat.dd

• Boot Sector Values:

– Volume ID

– Sector and Cluster size

– Sector of first Cluster

– Sector Range

– Locations of primary and backup FAT

• FAT Data

– Sequence of consecutive sectors that are allocated and either the next nonconsecutive sector or EOF.

@stake


FAT: Content Layer

• The clusters in the Data Area contain the file and directory content data.

• We can view the contents of the clusters using the ‘dcat’ command in TASK, or using ‘dd’.

• NOTE: TASK will require the sector address and not the cluster.

# dcat –f fat fat.dd 390

• We can get the statistics of a sector using the ‘dstat’ command:

# dstat –f fat fat.dd 390

21

@stake


FAT: Meta Data Layer

• The Directory Entries describe the files and directories in the file system.

• We can list of details of which using the ‘ils’ command from TASK:

# ils –e –f fat fat.dd

• We can get more details of a specific one using ‘istat’:

# istat –f fat fat.dd 5

• We can view the contents of the file using that directory entry, use ‘icat’:

# icat –f fat fat.dd 5

@stake


FAT: TASK & Directory Entries

• Directory entries in FAT are not given addresses

• TASK treats every 32-bytes as a possible directory entry

• The Root Directory has no directory entry, so it is given an address of 2 and the first 32-bytes in the first cluster are given the address 3. The second 32-bytes are given the address 4 etc.

• Not every address will be a directory entry

22

@stake


FAT: Name Layer

• All names are stored in the Directory Entry structure.

• We can get a listing of the file and directory names using the ‘fls’ command:

# fls –rp –f fat fat.dd

@stake


FAT: File Allocation

• Results from testing a Windows 2000 system

• The new directory entry is placed at the end of the existing list. Entries from deleted files are not reallocated, this results in a LONG list of deleted files.

• The clusters are allocated by scanning the FAT for clusters that are free. I’m not 100% sure of the algorithm yet:

– If the size is known, it will look for enough consecutive clusters (i.e. it will skip over single free ones if it needs several of them)

– If the size is unknown or it is from a redirection, it allocates single free clusters (but sometimes it skips over single ones and allocates several consecutive ones - maybe due to buffering)

• In the last cluster, only the sectors that contain data are overwritten (i.e. slack space)

23

@stake


FAT: File Deletion

• Directory entries are unallocated by setting the first letter (which is also the first byte of the structure) to ‘0xEF’

• No other directory entry values are changed:

– The name will still exist

– The times do not reflect the deletion time

• The clusters are unallocated by setting the corresponding FAT entries to 0

@stake


FAT: Allocation & Deletion Tools

• We can see how files are allocated by using the following tools:

– ‘ils –e’ to list the status of all ‘inodes’

– ‘fls –rp’ to list all known directory entries

– ‘fsstat’ to get the contents of the FAT

– ‘dls -le’ to list the status of sectors

• Save the output of these tools, create or delete files, and then compare the new tool output

24

@stake


Forensics Notes

• Because meta data is not in a static location, it can exist for a long time before it is overwritten

• Deleted directory entries still contain names and starting cluster -but not a deleted time

• Directory entry allocation algorithm is such that deleted entries are not overwritten

• Only the Access time is required to be updated and it is only accurate to the day. Created and written times are not consistently written

• Deleted data can be found in the slack space of files

• Wiping tools will sometimes leave long file names such as “aaaaaaaaaaaaaaaaaaaaaaaaaaaaa”

@stake


UNIX File Systems (FFS and EXT2FS)

25

@stake


UNIX File Systems

• Solaris, HP, and other BSD flavors use the Berkley Fast File System (FFS)

• Linux uses the Extended 2 File System (EXT2FS) and the Extended 3 File System (EXT3FS)

• EXT2FS is based on FFS

• We will present FFS first and then show the differences with EXT2FS

@stake


UNIX: File System Layout

• The File System is divided into groups, called Cylinder Groups in FFS (Block Groups in EXT2FS)

• All groups are the same size

• Groups contain both meta data structures and file content

• Each group manages its own resources

26

@stake


UNIX: File System Layout Diagram

Group

@stake


UNIX FFS: Cylinder Groups

• All Cylinder Groups in a given file system are the same size, which is identified in the Super Block

• The first group starts at offset 0 (but the first 16 are reserved)

• Each group contains a copy of the Super Block, but it increments its offset (it has an offset of 16 for the first group due to the disk labels)

• Each group contains a Group Descriptor structure that describes the group. The inode and block bitmaps are stored in the group descriptor.

27

@stake


UNIX FFS: Blocks & Fragments

• Sequential disk sectors (within the same group) are organized into Blocks

• Blocks may also be broken into Fragments (which are usually larger than a sector)

• Each Fragment is addressable

• Example:

– Sector Size: 512-bytes

– Block Size: 4096-bytes (8 sectors)

– Fragment Size: 1024-bytes (2 sectors)

• Each group has a bit-map that specifies the allocation status of each fragment

@stake


UNIX FFS: Blocks & Fragments Diagram

0 2 41 3 5 6 7

512-bytesector 1024-byte

fragment

4096-byteblock

28

@stake


UNIX FFS: inodes

• File and directory meta data are stored in inode structures

• A static number of inode structures are allocated at file system creation time

• Each group contains the same number of inode structures, which are stored in an inode table.

• Multiple files may use the same inode (hard-links)

• Inodes have unique addresses, starting with the number 2

• The Root directory (/) is always described by inode 2

• Each group contains a bit-map that describes the allocation status of inodes in that group

@stake


UNIX FFS: Summary of the inode Structure

• mode (2-bytes) [ permissions & type]

• number of links (2-bytes)

• size (8-bytes)

• Last access date & time (8-bytes)

• Last modified date & time (8-bytes) – file data modifications

• Last change date & time (8-bytes) - file status (inode) change

• User ID (4-bytes) and Group ID (4-bytes)

• Partial list of allocated blocks (direct pointers) and list of blocks with additional pointers (indirect pointers)

29

@stake


UNIX FFS: Block Pointers

• FFS was designed on research that showed that small files were more common the large files

• An inode contains 12 direct, 1 single-indirect, 1 double-indirect, and 1 triple-indirect pointer

• Direct Pointer: Points directly to a block that contains the file or directory content

• Single-indirect Pointers: Points to a block that contains a list of direct pointers, which point to the data

• Double-indirect Pointers: Points to a block that contains a list of single-indirect pointers

• Triple-indirect Pointers: Points to a block that contains a list of double-indirect pointers

@stake


UNIX FFS: Block Pointer Diagram

19 20 21 22 282726252423 29 30 31

direct

19 - filecontent

...32 33 34 35

99

32 - filecontent

100 101

200 - filecontent

...

...

single

double

direct

200 201direct

single

30

@stake


UNIX FFS: Time & Date Format

• UNIX saves the time & date as the number of seconds since 1/1/1970 UTC.

@stake


UNIX FFS: Directory Entries

• File and directory names are stored in Directory Entrystructures that are located in blocks allocated by the parent directory (similar to FAT, but different contents)

• Each structure contains a name and points to an inode structure

• Directory Entry structures have a dynamic size and form a linked list

• Multiple directory entries can point to the same inode (symbolic links)

• The ‘.’ and ‘..’ directory entries are the first two in all directories

31

@stake


UNIX FFS: Directory Entry Structure

• inode (4-bytes)

• record length (2-bytes)

• type (1-byte) {This does not exist in all versions}

• name length (1-byte) {This is 2-bytes if type does not exist}

• name (less than 255, NULL terminated)

• The record length is rounded up to a multiple of 4

@stake


UNIX FFS: Directory Entry Example

16 file1 file28357 16 8358 16 8359 file3

NOTE: The ‘.’ and ‘..’ entries are not shown

file35835916

file25835816

file15835716

namename leninoderec len

32

@stake


UNIX FFS: File Lookup Example

• Lookup /dir1/file1.dat

1. Locate inode 2 in the Inode Table of Group 1.

2. Read the blocks allocated to inode 2 and process them as directory entries.

3. Analyze each directory entry for the one with the name “dir1”. Save the inode value in the directory entry, 1810 for example.

4. Calculate which group inode 1810 is part of by dividing it by the number of inodes per group, group 2 for example.

5. Locate inode 1810 in the Inode Table of Group 2.

@stake


UNIX FFS: File Lookup Example

6. Read the blocks allocated to inode 1810 and process as directory entries.

7. Look for an entry with the name “file1.dat” and save the associated inode value, 2020 for example.

8. Calculate which group inode 2020 is part of by dividing it by the number of inodes per group, group 3 for example.

9. Read the inode 2020 structure from the Group 3 Inode Table and the blocks allocated by it.

33

@stake


UNIX FFS: File System Layer

• Two types of structures describe the file system

• The Super Block structure describes file system attributes.

– Size of Blocks & Fragments

– Size of groups

– Number of blocks and inodes per group

– Offset of group descriptor within each group

• The Group Descriptor structure within each Cylinder Group describes the group attributes

– Location of inode bitmap

– Location of block bitmap

@stake


UNIX FFS: fsstat

• The fsstat tool lists important Super Block values and the groups layout

# fsstat –f solaris solaris.dd

• Super Block Values:

– Block and fragment sizes

– Range of block addresses

– Range of inode addresses

• Group Info:

– Block and inode range of each group

– Location of inode table, group descriptor, and Super Block in each group

34

@stake


UNIX FFS: Content Layer

• The blocks and fragments contain the file and directory content data.

• We can view the contents of the clusters using the ‘dcat’ command in TASK (& TCT/TCTUTILs), or using ‘dd’.

# dcat –f solaris solaris.dd 390

• We can get the statistics of a sector using the ‘dstat’ command:

# dstat –f solaris solaris.dd 390

@stake


UNIX FFS: Meta Data Layer

• The inode structures describe the files and directories in the file system.

• We can list of details of which using the ‘ils’ command:

# ils –e –f solaris solaris.dd

• We can get more details of a specific one using ‘istat’:

# istat –f solaris solaris.dd 200

• We can view the contents of the file using that inode with ‘icat’:

# icat –f solaris solaris.dd 200

35

@stake


UNIX FFS: Name Layer

• All names are stored in Directory Entry structures.

• We can get a listing of the file and directory names using the ‘fls’ command:

# fls –rp –f solaris solaris.dd

@stake


UNIX FFS: Block Allocation

• Blocks are allocated to the same Cylinder Group as the inode. If the Cylinder Group is full or near full, a new one ischosen based on disk rotation and percentage of free blocks.

• Solaris will not fill up a group with a large file. It will fill up half of the available blocks, then move on to the next group. OpenBSD will fill up the group before moving on.

• The remainder of the fragment is wiped (i.e. no slack space)

36

@stake


UNIX FFS: Inode & Directory Entry Allocation

• File inodes are allocated in the same Cylinder Group as the parent directory. If it is full, a quadratic hash search is performed.

• Directory inodes are allocated in the next Cylinder Group with a greater than average number of free inodes

• The first available inode (lowest address) in the group is used

• Directory Entries are inserted into the list at the first available location

@stake


UNIX FFS: File Deletion

• Blocks and Fragments: Blocks are unallocated in the bitmap (including indirect blocks).

• Inode: The inode is set to unallocated in the group’s inode bitmap. The size and block pointer fields are reset to 0. The Modified & Changed times are set to the current time.

• Directory Entry: The record length of the previous directory entry is increased by the length of the deleted entry. Solaris will also reset the inode value to 0.

37

@stake


UNIX FFS: Deleted File Example

32 file1 file28357 16 8358 16 8359 file3

file2 is deleted

file35835916

file25835816

file15835732


file35835916

file25835816

file15835716


@stake


UNIX FFS: Allocation & Deletion Tools

• We can see how files are allocated by using the following tools:

– ‘ils –e’ to list the status of all inodes

– ‘istat’ to get a list of all blocks and fragments that were allocated to a given inode

– ‘fls –rp’ to list all known directory entries

– ‘dls -le’ to list the status of blocks and fragments

38

@stake


EXT2FS

@stake


UNIX EXT2FS: Block Groups

• EXT2FS calls the groups Block Groups - They are not organized by disk geometry

• Each group manages itself and contains a duplicate copy of the Super Block (which is always at the same offset)

• Each group has a table of Group Descriptors (FFS only had the local one)

• Entire blocks are allocated to the inode and block bitmaps

• The Sparse Super Block option prevents the Super Block from existing in every group and the space is used for data

39

@stake


UNIX EXT2FS: inode

• The EXT2FS inode contains much of the same data as FFS

• They have a different layout though

• The EXT2FS inode has a deletion time

@stake


UNIX EXT2FS: Block Allocation

• Blocks are allocated in the same Block Group as the inode (if there is room). It tries to allocate a block where the next 7 are also free.

• Attention is not paid to one file filling up a Block Group.

40

@stake


UNIX EXT2FS: Inode & Dir Entry Allocation

• File inodes are allocated to the same block group as parent directory, unless there is no room. Then they are allocated to another based on a Quadratic Hash or linear search.

• Directory inodes are allocated to the block group with the most free blocks and greater than average free inodes. The search begins at group 0, so it could be the same one as the parent directory.

• In both cases, the first (lowest address) inode is used.

• Directory entries are placed in the first available location in the list.

@stake


EXT2FS: File Deletion

• Linux changed how they deleted files from the 2.2 kernel to 2.4

• Blocks & Fragments: Unallocated in the group bitmap.

• Inode: The inode is set to unallocated, but the block pointers and size are not reset (the size is set to 0 when a directory is deleted though).

• The Modified, Changed, and Deleted times of the inode are set to the current time.

• Directory Entries: The length of the previous entry is increased. In the 2.2 kernel the inode value was not cleared. As of 2.4 it is set to 0, as FFS does.

41

@stake


EXT2FS: Other Tools

• debugfs runs on Linux and will let you view and change any structure

@stake


UNIX: Forensics Notes

• Times still exist in unallocated inodes, but usually not the directory entry pointer (except Open & FreeBSD FFS).

• With EXT2FS we still have access to the block pointers

• We can identify meta data wiping tools because there will be zeroed inodes in between allocated inodes in a CG

• Blank directory entries in the middle of a directory are also suspect

• To find the blocks for an unallocated inode, start looking in the CG, not the beginning of the disk (use ‘fsstat’ and ‘dls’)

• By manually parsing the directory entry structure, you can sometimes tell some order of file deletions

42

@stake


Journaling File Systems

@stake


Journaling File Systems: Motivation

• When a system crashes or is not shutdown properly, the file system must be validated

• The ‘fsck’ tool scans inodes, directory entries, and bitmaps to ensure that all structures agree

• This can take a LONG time on large file systems

43

@stake


Journaling File Systems: Overview

• A log is kept that contains meta data (and sometimes content) changes, atomic transactions are used.

• Before the data is written, the intended updates are recorded

• If the system crashes, the log is consulted and it can easily identify which transactions did not complete. It then either retries the update or turns back to the previous state.

• No public forensic tools currently take advantage of these logs.

• Examples: NTFS, EXT3FS, IBM JFS

@stake


NTFS

44

@stake


NTFS

• Based on HPFS (designed by Microsoft and IBM)

• Used in Microsoft Windows NT, 2000, & XP

• Design is not widely published by Microsoft, but a sourceforge project exists to write a Linux driver and document it

• NTFS is object oriented and very flexible

• There are too many data structures to address in detail in this talk (refer to TASK or Linux source)

@stake


NTFS: File System Layout

• 3 Major Areas (similar to FAT)

• Boot Sector

– Contains file system details such as sizes and offsets

• Master File Table (MFT)

– Manages the files and other objects

• Data Area

– Content area for files and objects

45

@stake


NTFS: Layout Example

Boot Sector

Master File Table

Data Area

@stake


NTFS: Master File Table

• The Master File Table (MFT) contains an entry for each user and system file

• MFT entries are called File structures and have a static size of 1024 bytes

• MFT entries store file and directory meta data

• Data is stored in attribute structures (objects)

• Some files will require multiple MFT entries (due to static size)

• Multiple files may use the same MFT entry (hard-links)

• MFT entries are addressed (File Reference) and have a sequence number

46

@stake


NTFS: MFT Entry Examples

• The MFT itself and mirror copy (0 & 1)

• LogFile (2)

• Root Directory (5)

• Volume Bitmap (3)

• Bad Clusters (8)

• Quota data (9)

• All files and directories (25+)

@stake


NTFS: Attributes

• An attribute describes a specific property of the file or directory (i.e. the name or dates)

• Attributes have a header and a value.

• The header is always stored in the MFT entry

• The value is stored in either the MFT entry (resident) or in the Data Area (non-resident)

47

@stake


NTFS: Attribute Header

• The attribute header is always located in the MFT

• It has the following fields:

– type (4-bytes) & id (2-bytes)

– length (4-bytes)

– resident flag

– name length and name (all have a name except the primary data one)

– If resident: A pointer to attribute stream

– If non-resident: A run-list of the clusters that contain the attribute data

– Virtual Cluster Number (VCN): The address of the cluster in a run with respect to the beginning of the attribute (0 - n), not the beginning of the file system (Logial Cluster Number - LCN)

@stake


NTFS: Attribute Types

• Standard Information: Read Only, Archive, MAC Times, number of links (type 16)

• File Name(s) (type 48)

• Data (type 128)

• Index Root (type 144) and Index Allocation (type 160) (for directory contents)

• Security Descriptor (type 80)

• Anything: For example, the icon on a Macintosh file is stored in an attribute so it gets moved with the file (more type 128)

• Each attribute type has its own data structure

48

@stake


NTFS: Standard Information Attribute

• Attribute Type 16

• Dates and Times:

– Creation

– Last MFT change

– Last write

– Last access

• Permissions in DOS format

• version

• Owner Id

• Security Id

@stake


NTFS: File Name Attribute


• MFT Reference Number

• MFT Sequence Number

• Dates and Times:

– Creation

– Last MFT change

– Last write

– Last access

• Character set

• Name length and name

49

@stake


NTFS: Data Attribute


• Data attributes just contain the raw data

• The name of the data is given in the attribute header

• The only attribute that can not have a name is the primary data attribute, which is typically given the name $Data by default.

• Directories can have data attributes in addition to their B-Tree attributes

@stake


TASK & NTFS Attributes

• MFT entries are addressed the same way in NTFS as inodes are with UNIX file systems

• One can give the Type and Id (if there are more than one of a given type) for a specific instance in the form of:

MFT-TYPE-ID

• For example:

# icat ntfs.dd 300

# icat ntfs.dd 300-128 (both are the same)

# icat ntfs.dd 300-128-3 (if an ADS exists)

50

@stake


NTFS: Data Area

• Data that can not fit in the MFT entry is stored in the Data Area

• The Data Area is organized into Clusters (groups of consecutive sectors)

• Cluster numbering starts with the first partition sector (unlike FAT)

• The clusters are managed using the Volume Bitmap MFT Entry (#6)

@stake


NTFS: Directories

• Directories store the file descriptors in B-trees, which are sorted by any descriptor (file name by default)

• The B-tree nodes contain the:

– MFT reference number

– The structure that is being sorted by (file name structure for example)

• The MFT entry also stores the file name in an attribute

• Names are stored in UNICODE

• The Root Directory (\) is pointed to by a static MFT entry (number 5)

51

@stake


NTFS: File Lookup Example

• Lookup \dir1\file1.dat

1. Read the Boot Sector to locate the beginning of the MFT

2. Read the MFT Entry for the Root Directory (#5)

3. Find & read the Index Root and Index Allocation attributes

4. Process the B-Tree and analyze the file name structures for the one with the name “dir1”. Save the file reference value, 180 for example.

5. Read MFT entry 180

6. Find & read the Index Root and Index Allocation attributes

@stake


NTFS: File Lookup Example

7. Process the B-Tree and find the file name structure with “file1.dat”. Save the reference value, 181 for example.

8. Read MFT entry 181.

9. Find & read the “$Data” attribute

52

@stake


NTFS: File System Layer

• Several structures describe the file system

• The Super Block contains the location of the MFT and the cluster sizes

• The MFT entry in the MFT describes the location and size of itself

• The $Log MFT entry contains the atomic transaction logs

• All of these can be viewed with the ‘fsstat’ command

# fsstat –f ntfs ntfs.dd

@stake


NTFS: Content Layer

• Data is stored in the MFT itself if it is small enough

• Larger data is stored in clusters in the Data Area

• Clusters are numbered from the beginning of the file system

• The dcat tool will allow us to display the contents of any cluster

# dcat –f ntfs ntfs.dd 8016

• The icat tool will allow us to display the contents of attributes that are stored in the MFT:

# icat -f ntfs ntfs.dd 76-160

53

@stake


NTFS: Meta Data Layer

• All file and directory meta data is generally located in the MFT

• If the value of the meta data is too large, it will be located in the Data Area

• Each MFT is addressed using a File Reference Number

• We can view the details of an MFT entry using istat:

# istat –f ntfs ntfs.dd 5

• We can list them all using ils:

# ils –f ntfs –e ntfs.dd

• The contents of the file can be viewed using icat (with no type or id value):

# icat -f ntfs ntfs.dd 9901

@stake


NTFS: Naming Layer

• File names are stored in a node of the B-Tree in the directory’s MFT entry

• The file’s MFT entry also has all names that it uses (multiple ones if it has hard-links)

• The list of known file names can be shown with fls:

# fls –f ntfs –rp ntfs.dd

54

@stake


NTFS: File Allocation

• An area following the MFT is reserved for its expansion

• MFT entries are allocated linearly

• Names are allocated into directories based on their name and location in the B-tree

• Only the sectors in the last cluster that are used are overwritten (slack space)

@stake


NTFS: File Deletion

• Content Blocks are unallocated by clearing the bits in the bitmap

• MFT entry is freed by setting the in-use flag. The attributes are unchanged. (link count never goes to 0).

• No times are set to the current time

• The name still exists in the MFT entry

• Parent directory Index Tree is resorted (the data may still exist in the tree depending on the other files in the directory)

55

@stake


Forensics Notes

• Because of MFT allocation, it is likely that unallocated entries will be reused faster than with UNIX inodes

• No times are set when an MFT is unallocated, so time lines of file activity are not as revealing

• Deleted file names will not exist as frequently as with FAT or UNIX because they will get overwritten when the tree is sorted

• We can collect slack space data

• Clean MFT entries in the middle of the MFT is likely a result ofwiping tools

• If times in Standard Info are different than File Name, wiping tools may have been used

@stake


NTFS: Other Tools

• NTFS File Sector Information Utility (nfi) (Microsoft)

• NTFS Info (www.sysinternals.com)

56

@stake


EXT3FS

@stake


EXT3FS: Overview

• EXT3FS is identical to EXT2FS except it interfaces with the Journal layer

• The journal is kept in a special file or a specific journaling device

• Currently, no tools publicly exist to utilize the journal for forensics

57

@stake


Summary of Structures

Directory Entries

B-tree Index trees

Directory Entries

Name Layer

InodesMFT EntriesDirectory Entries

Meta Data

Fragments & Blocks in groups

Clusters in the Data Area

Clusters in the Data Area

Content

Super Block, Group Desc

Boot Sector, MFT

Boo Sector, FAT

File System

UNIXNTFSFAT

@stake


Log-based File Systems

58

@stake


Basics of Log-structured File Systems

• “Classical” File Systems waste time by seeking to allocated blocks and overwriting the contents (for data or inodes)

• Log-structured File Systems work like a log file and append new data to the “end” of the data

• All data and meta data are written to the current pointer.

• An inode index file points to the current location of the inode

• The inode points to the current location of its data blocks

• Deleted Content is not overwritten for a while...

• No Forensic Tools currently take advantage of Log FS...

• Example: BSD Log File System, Linux Log-Structured FS

@stake


PALM OS

• Everything is a database (even applications)

• Each database stores the dates of Creation, last modification, and last backup

• When a database is deleted, it is flagged as not in use, but not deleted (until a hotsync is done)

• See Joe Grand’s talk on the pdd Palm Acquisition tool (Wednesday)

59

@stake


Summary

• Knowledge of file system structures will help interpreting the results of tools

• Knowledge will also help write tools when ones do not exist

• Evidence of wiping tools can be found by examining the structures and their location

• Using allocation algorithms, one can sometimes recover deleted data more quickly

@stake


Software References

• TASK can be found at:

http://www.atstake.com/research/tools/task

• TCT & TCTUTILs (for UNIX only) can be found at:

www.fish.com

www.porcupine.org

www.cerias.purdue.edu/homes/carrier

60

@stake


References

• FAT

– Microsoft FAT32 File System Specification

– Microsoft Long Filename Specification

• FFS

– The Design and Implementation of the 4.4 BSD Operating System (McKusick, Bostic , Karels , Quarterman)

• EXT2FS

– The extended-2 filesystem overview (Oxman)

– Design and Implementation of the Second Extended Filesystem (Card, Ts’o,Tweedie)

@stake


References

• EXT3FS

– Transcript of Stephen Tweedie talk (olstrans .sourceforge.net/release/OLS2000-ext3/OLS2000-ext3.html)

• NTFS

– Inside Windows 2000 - 3rd Edition (Solomon, Russinovich)

– Linux NTFS Project (linux-ntfs.sourceforge.net)

d1 02 Carrier Slides

Documents

Transcript of d1 02 Carrier Slides