File System Implementation CISC3595, Fall 09 1. Outline File system introduction File system...

File System Implementation

CISC3595, Fall 09

1

Outline File system introduction File system implementation Disk space allocation and management Efficiency and Performance Recovery NFS

2

Objectives for a File Management System Meet the data management needs of the

user Provide I/O support for a variety of storage

device types Provide a standardized set of I/O interface

routines to user processes Provide I/O support for multiple users (if

needed) Guarantee that the data in the file are valid Minimize lost or destroyed data Optimize performance

Requirements for a general purpose system1. user should be able to create, delete, read, write

and modify files2. user may have controlled access to other users’

files3. user may control what type of accesses are

allowed to his/her files4. user should be able to restructure his/her files5. user should be able to move data between files6. user should be able to back up and recover files

in case of damage7. user should be able to access files using

symbolic names

File-System Structure File: logical storage unit,

collection of related information

File system resides on secondary storage (disks), or tertiary storage

File system organized into layers

5

File-System Structure Logical file system: manage

metadata information, all file system structures except actual data (i.e., file content) directory structure, file-control block,

protection and security File-Organization module:

File allocation: map logic block addr.(2nd block of a file) to physical block addr.

Free space management Basic file system

Issue commands to device driver Manage memory buffers and caches

Buffer allocated before data transfer Cache for storing meta-data

6

File-System Structure

I/O control: device drivers and interrupt handlers Responsible for starting I/O

operations on a device Processes completion of an

I/O request Translate high-level requests

to low-level, hardware-specific instructions (sent to disk controller)

7

Virtual File Systems

Virtual File Systems (VFS): same system call

interface (API) used for different types of concrete file systems

Support numerous file system types ext2, ufs, fat, vfat, hpfs,

minix, isofs, sysv, hfs, affs, NTFS

NFS, CoDA, AFS ncpfs Procfs, umsdos, userfs

8

Mount Various file systems are mounted at different directories

(mounting points) in the files system name space $ mount

$ /dev/sda3 on / type ext3 (rw,relatime,errors=remount-ro)

tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)

proc on /proc type proc (rw,noexec,nosuid,nodev)

sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)

varrun on /var/run type tmpfs (rw,nosuid,mode=0755)

varlock on /var/lock type tmpfs (rw,noexec,nosuid,nodev,mode=1777) udev on /dev type tmpfs (rw,mode=0755)

tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)

devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620) fusectl on /sys/fs/fuse/connections type fusectl (rw) lrm on /lib/modules/2.6.28-11-generic/volatile type tmpfs (rw,mode=755) securityfs on /sys/kernel/security type securityfs (rw) binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,noexec,nosuid,nodev) gvfs-fuse-daemon on /home/zhang/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev,user=zhang)

9


10

File system structures (metadata) on disk Boot control block: contain info. needed for

boot an OS from the volume Volume control block

contains volume (partition) detials: number of blocks, size of blocks, free-block count and list, free FCB count and FCB pointers

Directory structure (per file system): organize files

Per-file FCB: information about a file

11

File system structures in memory Mount table: contains info. about each mounted

volume In-memory directory-structure cache: contains

recent accessed directory info. For a directory that is a mounting point, contains flag

indicating it’s a mount point, and a pointer to an entry in mount table

System-wide open-file table: a copy of PCB for each open file

Per-process open-file table: contains pointer to appropriate entry in system-wide open-file table

Buffer: hold file system blocks being read from disk or written to disk

12

Supporting file system interface: file creation

Program issues system call, open(), create(), fopen(), …

Logical files system allocates a new FCB reads directory into memory,

update it with new file and its FCB, write it back Call file-organization module to

map directory I/O to disk-block number

13

directory I/O

disk block #

Supporting file system interface: open a file Program issues system call, open(), passing a file

name Logic file system (handler of open()) searches

system-wide open-file table for the file, if not found, search directory structure for the file, cache directory info, copy file’s PCB into system-wide open-file table

In per-process open-file table, creates an entry for the file, to store pointer to system-wide open-file table entry, current location pointer, access mode info.

Return a pointer to the per-process open-file table, i.e., file descriptor in Unix, or file handler in Windows

14

Open/Read a file

15


16

Directory: Contains information about files

File Name File type File Organisation

For systems that support different organizations Attributes, ownership Location:

Volume: Indicates device on which file is stored Starting Address Size Used : Current size of the file in bytes, words, or

blocks Size Allocated : The maximum size of the file

Operations Performed on a Directory A directory system should support a number

of operations including: Search Create files Deleting files Listing directory Updating directory

Directory Implementation

Linear list of file names with pointer to the data blocks. simple to program time-consuming to search Sorted list? Tree structure?

Hash Table – linear list with a hash table hash table takes a value computed from file name

and returns a pointer to the file name in a linear list decreases directory search time collisions – situations where two file names hash

to the same location

19


20

Allocation Methods Allocate disk space to files

Contiguous allocation

Linked allocation

Indexed allocation

21

Contiguous Allocation Each file occupies a set of

contiguous blocks on the disk Pros:

Simple – only starting location (block #) and length (number of blocks) are required

Random access Cons:

Wasteful of space: dynamic storage-allocation problem: how to satisfy request from list of non-contiguous free holes

External fragmentation Files cannot grow

22

Linked Allocation Each file is a linked list of disk

blocks: blocks may be scattered anywhere on the disk.

Directory contains pointer to the first and last blocks of the file.

Pros: Solve problems with contiguous

allocation Cons:

Inefficient direct access Reliability issues

23

File-Allocation Table

24

File-allocation table (FAT) – disk-space allocation used by MS-DOS and OS/2.

FAT: for each volumeone entry for each disk block

Pros?

Cons?

Indexed Allocation Brings all pointers (block #s) together into

index block. Logical view.

Each file has its own index block

index table

25

Indexed Allocation

26

Indexed Allocation: Multi-level indexing

outer-index

index table file

27

Combined Scheme: UNIX inode (4K bytes per block)

28

Free-Space Management Bit vector (n blocks)

First free Block number: (number of bits per word) *(number of 0-value words) + offset of first 1 bit Easy to get contiguous free blocks

Bit map requires extra space E.g., block size = 212 bytes, disk size = 230 bytes (1 gigabyte) n = 230/212 = 218 bits (or 32K bytes)

…

0 1 2 n-1

bit[i] = 0 block[i] free

1 block[i] occupied

29

Free-Space Management (Cont.)

30

Linked list (free list) Cannot get contiguous space

easily No waste of space

Grouping First free block contains address of

n free blocks The n-th block therein contains

address of another n free blocks, …

Counting Free blocks might be contiguous Keep starting block # and length


31

Efficiency and Performance Efficiency dependent on:

disk allocation and directory algorithms types of data kept in file’s directory entry

Performance disk cache – separate section of main memory for

frequently used blocks free-behind and read-ahead – techniques to

optimize sequential access improve PC performance by dedicating section of

memory as virtual disk, or RAM disk

32

Page Cache A page cache caches pages rather than disk

blocks using virtual memory techniques

Memory-mapped I/O uses a page cache

Routine I/O through the file system uses the buffer (disk) cache

This leads to the following figure

33

I/O Without a Unified Buffer Cache

34


35

Recovery Consistency checking – compares data in

directory structure with data blocks on disk, and tries to fix inconsistencies

Use system programs to back up data from disk to another storage device (floppy disk, magnetic tape, other magnetic disk, optical)

Recover lost file or disk by restoring data from backup

36

Log Structured File Systems Log structured (or journaling) file

systems record each update to file system as a transaction

All transactions are written to a log A transaction is considered committed once

it is written to log However, file system may not yet be updated

Transactions in the log are asynchronously written to file system When file system is modified, the transaction

is removed from log If file system crashes, all remaining

transactions in log must still be performed37


38

The Sun Network File System (NFS) An implementation and a specification of a

software system for accessing remote files across LANs (or WANs)

The implementation is part of the Solaris and SunOS operating systems running on Sun workstations using an unreliable datagram protocol (UDP/IP protocol and Ethernet

39

NFS (Cont.) Interconnected workstations viewed as a set of

independent machines with independent file systems, which allows sharing among these file systems in a transparent manner

A remote directory is mounted over a local file system directory Mounted directory looks like an integral subtree of local file

system, replacing the subtree descending from the local directory

Specification of remote directory for mount operation is nontransparent: host name of remote directory has to be provided Files in the remote directory can then be accessed in a

transparent manner Subject to access-rights accreditation, potentially any

file system (or directory within a file system), can be mounted remotely on top of any local directory

40

NFS (Cont.) NFS is designed to operate in a heterogeneous

environment of different machines, operating systems, and network architectures; the NFS specifications independent of these media

This independence is achieved through the use of RPC primitives built on top of an External Data Representation (XDR) protocol used between two implementation-independent interfaces

NFS specification distinguishes between the services provided by a mount mechanism and the actual remote-file-access services

41

NFS Protocol Provides a set of remote procedure calls for remote

file operations. The procedures support the following operations: searching for a file within a directory reading a set of directory entries manipulating links and directories accessing file attributes reading and writing files

NFS servers are stateless; each request has to provide a full set of arguments

(NFS V4 is just coming available – very different, stateful)

Modified data must be committed to the server’s disk before results are returned to the client (lose advantages of caching)

The NFS protocol does not provide concurrency-control mechanisms

42

Three Major Layers of NFS Architecture

UNIX file-system interface (based on the open, read, write, and close calls, and file descriptors)

Virtual File System (VFS) layer – distinguishes local files from remote ones, and local files are further distinguished according to their file-system types The VFS activates file-system-specific operations to

handle local requests according to their file-system types Calls the NFS protocol procedures for remote requests

NFS service layer – bottom layer of the architecture Implements the NFS protocol

43

Schematic View of NFS Architecture

44

NFS Path-Name Translation Performed by breaking the path into

component names and performing a separate NFS lookup call for every pair of component name and directory vnode

To make lookup faster, a directory name lookup cache on the client’s side holds the vnodes for remote directory names

45

NFS Remote Operations Nearly one-to-one correspondence between regular

UNIX system calls and the NFS protocol RPCs (except opening and closing files)

NFS adheres to the remote-service paradigm, but employs buffering and caching techniques for the sake of performance

File-blocks cache – when a file is opened, the kernel checks with the remote server whether to fetch or revalidate the cached attributes Cached file blocks are used only if the corresponding

cached attributes are up to date File-attribute cache – the attribute cache is updated

whenever new attributes arrive from the server Clients do not free delayed-write blocks until the

server confirms that the data have been written to disk

46

File System Implementation CISC3595, Fall 09 1. Outline File system introduction File system...

Documents

Transcript of File System Implementation CISC3595, Fall 09 1. Outline File system introduction File system...