Extensible File Allocation Table (ExFAT) AHTCC V1.1
Transcript of Extensible File Allocation Table (ExFAT) AHTCC V1.1
Extensible File Allocation Table (exFAT)
Zoran Iliev 2010
History
• 2006, on Windows Hardware Engineering Conference (WinHEC), Microsoft presented “Personal Storage: Opportunities and challenges for pocket-sized storage devices in the Windows world”.
• Same Conference Vishal Ghotge - Program Manager, Core File Services, Microsoft Corporation presents exFAT
The design goals of exFat
• Be as similar to FAT32 as possible
• Overcome FAT32’s limitations
• Design for the future
• Make it extensible
• Scale to large volume sizes
• Manage large files
• Designed to optimize for flash media
• exFAT – A simple, extensible, rich filesystem for next generation devices and flash media.
Some Key Features Of exFAT
• Free space bitmap Identifies free and available clusters on the volume allowing Faster look-up for free space allocation, Faster deletes
• Support for large files File size field is 8 bytes, Theoretical maximum file size of 264 bytes
• Large cluster size The on-disk structure limit for cluster size is 2255,
Implementation limit is 32 MB Source: download.microsoft.com/download/5/b/9/5b97017b.../sto072_wh06.ppt
Some Key Features Of exFAT • Extensibility for the future
- Directory structure is flexible and extensible - Support for more than 216 (65,536) files in a single directory - Timestamps can be in UTC rather than only local time (starting with Vista SP2). - Easy to add new features and can add some features without breaking compatibility
• Name hashes - 2 byte checksum of up-cased file name, allows for faster checks of filenames
• OEM specific parameters and types - Feature to store OEM specific volume information
• Support for access control lists (not supported yet )
• Support for Transaction-Safe FAT File System (TFAT) (optionally WinCE activated function)
exFAT vs FAT 32
• exFAT is not backward compatible with FAT32
• There are no “short names” in exFAT
Disk Space Overhead
Default cluster sizes for exFAT
VOLUME Structure
BOOT Regions – Main and Backup
Boot Sector – On Disk Hex View
Boot Sector – On Disk Hex View
JumpBoot (0, 3)
Contains the jump instruction for CPUs common in personal computers, which, when executed, "jumps" the CPU to execute the boot-strapping instructions in the BootCode field
Boot Sector – On Disk Hex View
File System Name (3, 8)
The FileSystemName field contains the name of the file system on the volume, "EXFAT“ in ASCII characters, which includes three trailing white spaces.
Boot Sector – On Disk Hex View
Must Be Zero (11, 53)
The MustBeZero field directly corresponds with the range of bytes the packed BIOS parameter block consumes on FAT12/16/32 volumes. The valid value for this field is 0, which helps to prevent FAT12/16/32 implementations from mistakenly mounting an exFAT volume.
Boot Sector – On Disk Hex View
Partition Offset (64, 8) 02 00 = 8192
The PartitionOffset field describes the media-relative sector offset of the partition which hosts the given exFAT volume. This field aids boot-strapping from the volume using extended INT 13h** on personal computers. **Extended Int13h functions address sectors using Logical Block Addressing
Boot Sector – On Disk Hex View
Volume Length (72, 8) F4 40 00= 16007168 * 512= 8195670016 ≈ 7.6GB
The VolumeLength field describes the size of the given exFAT volume in sectors.
Boot Sector – On Disk Hex View
FAT Offset (80, 4) 08 00 = 2048
The FatOffset field describes the volume-relative sector offset of the First FAT.
Boot Sector – On Disk Hex View
FAT Length (84, 4) 07 C0 = 1984
The FatLength field describes the length, in sectors, of each FAT table (the volume may contain up to two FATs, not implemented yet).
Boot Sector – On Disk Hex View
Cluster Heap Offset (88, 4) 10 00 = 4096
The ClusterHeapOffset field describes the volume-relative sector offset of the Cluster Heap. This field enables implementations to align the Cluster Heap to the characteristics of the underlying storage media and it is the Physical Sector Location of the Bitmap.
Boot Sector – On Disk Hex View
CLUSTER COUNT (92, 4) 03 D0 C0 = 250048
The ClusterCount field describes the number of clusters the Cluster Heap contains. The value of the ClusterCount field determines the minimum size of a FAT.
Boot Sector – On Disk Hex View
First Cluster of Root Directory (96, 4) 04 = 4
The FirstClusterOfRootDirectory field contains the cluster index of the first cluster of the root directory. The FirstClusterOfRootDirectory field contains the cluster index of the first cluster of the root directory.
Boot Sector – On Disk Hex View
Volume Serial Number (100, 4)
The Volume Serial Number field contains a unique serial number.
Boot Sector – On Disk Hex View
File System Revision (100, 4) 1.0
The high-order byte is the major revision number and the low-order byte is the minor revision number. For example, if the high-order byte contains the value 01h and if the low-order byte contains the value 05h, then the FileSystemRevision field describes the revision number 1.05.
Boot Sector – On Disk Hex View
Volume FLAGS (106, 2)
The VolumeFlags field contains flags which indicate the status of various file system structures on the exFAT volume.
Boot Sector – On Disk Hex View
Bytes Per Sector Shift (108, 1) 29 = 512 Bytes per Sector
The BytesPerSectorShift field describes the bytes per sector expressed as binary logarithm log2(N), where N is the number of bytes per sector.
Boot Sector – On Disk Hex View
Sectors Per Cluster Shift (109, 1) 26 = 64 Sectors per Cluster
The Sectors PerClusterShift field describes the bytes per sector expressed as binary logarithm log2(N), where N is the number of bytes per sector.
Boot Sector – On Disk Hex View
Number of Fats (110, 1)
The NumberOfFats field describes the number of FATs and Allocation Bitmaps the volume contains. Can be 1 or 2.
Boot Sector – On Disk Hex View
Drive Select (111, 1)
The DriveSelect field contains the extended INT 13h drive number, which aids boot-strapping from this volume using extended INT 13h on personal computers.
Boot Sector – On Disk Hex View
Percent In Use (112, 1) 0% - no files written
The PercentInUse field describes the percentage of clusters in the Cluster Heap which are allocated. FFh-indicates the percentage of allocated clusters in the Cluster Heap is not available; 00 – indicates backup of the Main Boot region.
Boot Sector – On Disk Hex View
Percent In Use (112, 1) 9% in use
Boot Sector – On Disk Hex View
Reserved (113, 7)
Boot Sector – On Disk Hex View
Boot Code (120, 390)
The BootCode field contains boot-strapping instructions.
Boot Sector – On Disk Hex View
Boot Signature (510, 2)
Boot Sector – Cont’d
• First 9 sectors (0 to 8) are marked with the x55 AA signature
• Sector 11 contains Boot Checksum, 4 bytes value is repeated from the beginning to the end of the sector. This value, generated during format and is copied in the backup VBR, is a checksum of the other areas of the boot region, not including the volume flags or the percent-in-use fields.
• Sectors 12-23 containing a complete backup of the first 12 sectors of the volume
Sector 10
Sector 11 Boot Checksum
Sector 12
FAT Region
• A FAT describes cluster chains in the Cluster Heap. A cluster chain is a series of clusters which provides space for recording the contents of files, directories, and other file system structures.
• A FAT represents a cluster chain as a singly-linked list of clusters indices.
• With the exception of the first two entries, every entry in a FAT represents exactly one cluster and each entry is 4 bytes in size.
FAT Region
Data Region
• The Data region contains the Cluster Heap, which provides managed space for file system structures, directories, and files.
• The Cluster Heap's structure is very simple; each consecutive series of sectors describes one cluster, as the SectorsPerClusterShift field defines.
• Importantly, the first cluster of the Cluster Heap has index two, which directly corresponds to the index of Fat Entry 2.
Allocation Bitmap
• In an exFAT volume, an Allocation Bitmap maintains the record of the allocation state of all clusters.
• This is a significant difference from exFAT's predecessors (FAT12, FAT16, and FAT32), in which a FAT maintained a record of the allocation state of all clusters in the Cluster Heap.
Allocation Bitmap
• An Allocation Bitmap records the allocation state of the clusters in the Cluster Heap.
• Each bit in an Allocation Bitmap indicates whether its corresponding cluster is available for allocation or not.
• An Allocation Bitmap represents clusters from lowest to highest index.
• For historical reasons, the first cluster has index 2. • Note: the first bit in the bitmap is the lowest-order
bit of the first byte.
Directory Structure
• The exFAT file system uses a directory tree approach to manage the file system structures and files which exist in the Cluster Heap.
• Directories have a one-to-many relationship between parent and child in the directory tree.
• The directory to which the “FirstClusterOfRootDirectory” (96, 4) field refers is the root of the directory tree.
• All other directories descend from the root directory in a singly-linked fashion. Each directory consists of a series of directory entries.
Example: Calculating the first sector of the Root entry
Directory Structure
• One or more directory entries combine into a directory entry set which describes something of interest, such as a file system structure, sub-directory, or file.
Volume Label Directory Entry
• The Volume Label is a Unicode string which enables end users to distinguish their storage volumes.
• In the exFAT file system, the Volume Label exists as a critical primary directory entry in the root directory .
Volume Label Directory Entry
Entry Type (0, 1)
Volume Label Directory Entry
• The CharacterCount field contains the length of the Unicode string the VolumeLabel field contains.
Character Count (1, 1)
Volume Label Directory Entry
• The VolumeLabel field contains a Unicode string, which is the user-friendly name of the volume.
• Maximum 11 characters Volume Label Name.
Volume Label (2, 22)
Volume Label Directory Entry
Reserved (24, 8)
Volume Label Directory Entry
• If the volume was not given a label, the first value for the first entry will be: 0x03 instead of 0x83.
• Incidentally, renaming of the Volume name changes the inUse flag to zero to the previous Volume Label Directory Entry and creates new Volume Label Directory Entry. Nothing else is changed within the previous Volume Label Directory Entry.
Allocation Bitmap Directory Entry
• In the exFAT file system, a FAT does not describe allocation state of clusters; rather, an Allocation Bitmap does.
• Allocation Bitmap exist in the Cluster Heap and have corresponding critical primary directory entries in the root directory.
• The “NumberOfFats” field determines the number of valid Allocation Bitmap directory entries in the root directory.
Allocation Bitmap Directory Entry
• If the “NumberOfFats” field contains the value 1, then the only valid number of Allocation Bitmap directory entries is 1.
• If the NumberOfFats field contains the value 2, then the only valid number of Allocation Bitmap directory entries is 2.
Allocation Bitmap Directory Entry
Entry Type (0, 1)
Allocation Bitmap Directory Entry
Bitmap Flags (1, 1)
0, which means the given directory entry describes the First Allocation Bitmap 1, which means the given directory entry describes the Second Allocation Bitmap and is possible only when “NumberOfFats” contains the value 2
Allocation Bitmap Directory Entry
Reserved (2, 18)
Allocation Bitmap Directory Entry
First Cluster (20, 4)
This field contains the index of the first cluster of the cluster chain, as the FAT describes, which hosts the Allocation Bitmap.
Allocation Bitmap Directory Entry
Data Length (24, 4) 7A 18 = 31256 = 30.5 Kb
Up-Case Table Directory Entry
• The Up-case Table defines the conversion from lower-case to upper-case characters.
• This is important due to the File Name directory entry using Unicode characters and the exFAT file system being case insensitive and case preserving.
• The Up-case Table exists in the Cluster Heap and has a corresponding critical primary directory entry in the root directory.
Up-Case Table Directory Entry
• Entry Type 82
Entry Type (0, 1)
Up-Case Table Directory Entry
Reserved (1, 3)
Up-Case Table Directory Entry
• The “TableChecksum” field contains the checksum of the Up-case Table (which the “FirstCluster” and “DataLength” fields describe).
• Implementations shall verify the contents of this field are valid prior to using the Up-case Table.
Table Checksum (4,4)
Up-Case Table Directory Entry
Reserved (8, 12)
Up-Case Table Directory Entry
• This field contains the index of the first cluster of the cluster chain, as the FAT describes, which hosts the Up-case Table.
First Cluster (20, 4)
Up-Case Table Directory Entry
Data Length (20, 4) 16 CC = 5836 = 5.699 Kb
Volume Guid Directory Entry
• The Volume GUID directory entry contains a GUID which enables implementations to uniquely and programmatically distinguish volumes.
• The Volume GUID exists as a benign primary directory entry in the root directory.
File Directory Entry
• File directory entries describe files and directories. They are critical primary directory entries and any directory may contain zero or more File directory entries.
• For a File directory entry to be valid, at most one Stream Extension Directory entry and at least one File Name Directory entry immediately follow the File directory entry.
File Directory Entry
• Entry Type 85
Entry Type (0, 1)
File Directory Entry
• The “SecondaryCount” field describes the number of secondary directory entries which immediately follow the given primary directory entry.
• These secondary directory entries, along with the given primary directory entry, comprise the directory entry set.
Secondary Count (1, 1)
File Directory Entry
• The “SetChecksum” field contains the checksum of all directory entries in the given directory entry set.
• The checksum excludes this field
Set Checksum (2, 2)
File Directory Entry
File Attributes (4, 2) 20h = 32 (25)-> Archive
File Directory Entry
Reserved 1 (6, 2)
File Directory Entry
Create Timestamp 1 (8, 4)
• In combination, the “CreateTimestamp” and “CreateTime10msIncrement” fields describe the date and time the given file/directory was created.
• These two fields conform to the definitions of the Timestamp and 10msIncrement fields.
Create 10ms Increment1 (20, 1)
File Directory Entry
Timestamp Field Structure :
• 0 – 4 Seconds divided by 2 (0 = 0 sec., 29 = 58 sec.)
• 5 – 10 Minutes 0 – 59
• 11 – 15 Hours 0 – 23
• 16 – 20 Days 1 – 31
• 21 – 24 Months (01 = January 02 = February etc. )
• 25 – 31 Years since 1980
File Directory Entry
10ms Increment Fields Structure :
• The valid range of values for these fields is:
– At least 0, which represents 0 milliseconds
– At most 199, which represents 1990 milliseconds
File Directory Entry Create Timestamp 1 (8, 4)
25/02/2010 16:39:54 Create 10ms Increment1 (20, 1)
50h = 80 (* 10) = 800ms
File Directory Entry Last Modified Timestamp 1 (12, 4)
25/02/2010 16:20:52 Last Modified 10ms Increment1 (21, 1)
00 ??
File Directory Entry
Time Zone Created (22, 1)
Time Zone Accessed (24, 1)
Time Zone Modified (23, 1)
Time Zone Calculation
• There is one byte for each Time Stamp (Created, Modified and Accessed).
• The bit assignment is: - 1 bit reserved (the most significant bit) - the other 7 bits are representing a signed integer that is recording the number of 15 minutes intervals.
Time Zone Calculation
A8h = 10101000 -> 101100 = 40 X 15 = 600 min
=> Time Zone +10 (Canberra, Sydney)
File Directory Entry
Reserved 2 (25, 7)
Stream Extension Directory Entry
• The Stream Extension directory entry is a critical secondary directory entry in File directory entry sets.
• This directory entry is valid only if it immediately follows the File directory entry.
Stream Extension Directory Entry
• Entry Type C0
Entry Type (0, 1)
Stream Extension Directory Entry
Secondary Count (1, 1)
Stream Extension Directory Entry
Reserved 1 (2, 1)
Stream Extension Directory Entry
• The NameLength field contains the length of the Unicode string the subsequent File Name directory entries collectively contain.
Name Length 1 (3, 1)
Stream Extension Directory Entry
The “NameHash” field contains a 2-byte hash of the up-cased file name. This enables implementations to perform a quick comparison when searching for a file by name. Importantly, it provides a sure verification of a mismatch. Implementations shall verify all “NameHash” matches with a comparison of the up-cased file name.
Name Hash 1 (4, 2)
Stream Extension Directory Entry
Reserved 2 (6, 2)
Stream Extension Directory Entry
• The “ValidDataLength” field describes how far into the data stream user data has been written.
• Implementations shall update this field as they write data further out into the data stream.
Valid Data Length (8, 8)
Stream Extension Directory Entry
Reserved3 (16, 4)
Stream Extension Directory Entry
First Cluster (20, 4)
Stream Extension Directory Entry
Data Length (28, 4)
File Name Directory Entry
• File Name directory entries are critical secondary directory entries in File directory entry sets.
• The valid number of File Name directory entries in a File directory entry set is “NameLength”/15, rounded up to the nearest integer.
• File Name directory entries are valid only if they immediately follow the Stream Extension directory entry as a consecutive series.
• File Name directory entries combine to form the file name for the File directory entry set.
File Name Directory Entry
• Entry Type C1
Entry Type (0, 1)
File Name Directory Entry
General Secondary Flags Field (1, 1)
File Name Directory Entry
File Name(2, 30)
File Name Directory Entry
• The file names "." and ".." have the special meaning of "this directory" and "containing directory", respectively. Implementations shall not record either file name in the FileName field.
• However, implementations may generate these two file names in directory listings to refer to the directory being listed and the containing directory.
File Creation
By Examining the FAT entries we can find the allocated clusters.
We know that the First cluster is 78h, so we go to the beginning of the FAT, at sector 128, which is defined in the boot sector:
• So, if we do the math, we have:
• First Fragment: 4999-119 = 4880
• Second Fragment: 7135-5607 = 1528
• FILE SIZE: = 6408 CLUSTERS
Deleted Files
Directory Entry after file deletion: • As we already mentioned, The “InUSE” bit describes
whether the given directory entry in use or not. The valid values for this field are:
• 0, which means the given directory entry is not in use; this means the given structure actually is an unused directory entry
• 1, which means the given directory entry is in use; this means the given structure is a regular directory entry
• Accordingly, when a file or directory is deleted, the flag changes to “0”, indicating that this directory entry is free or unused directory entry.
Deleted Files
FAT after file deletion
• No change until re-used for the next fragmented file.
BITMAP after file deletion
• The bits that were showing “in use (1)” are immediately changed to “not in use (0)”
Recovery of Contiguous files in exFAT
Questions?
• Special Thank you to my colleague and friend Steve Ilett.
• Thank you all for attending this session!
Bibliography
• Ravisankar V. Pudipeddi and Vishal Ghotge and Ravinder Thind 2009: Contiguous Allocation in an Extensible File System, US Patent Application Publication, Pub. No.: US 2009/0164539 A1.
• Ravisankar V. Pudipeddi and Vishal Ghotge and Ravinder Thind 2009: Quick Filename Lookup Using Name Hash, US Patent Application Publication, Pub. No.: US 2009/0164440 A1.
• Jeff Hamm, Extended FAT File System, 2009 • Article ID: 955704 - Last Review: September 29, 2009 - Revision:
3.1: Description of the exFAT file system driver update package, http://support.microsoft.com/kb/955704, accessed February 2009.
• Raymond Chen 2006: Windows Confidential, A Brief and Incomplete History of FAT32, TechNet Magazine.
• Vishal Ghotge, (Program Manager, Core File Services Microsoft Corporation), exFAT (PPT), Win HEC 2006