1GWU CS 259 Brad Taylor Spring 2004
Systems Programming
Meeting 3:Meeting 3:
Handling Input/OutputHandling Input/OutputFiles, Directories & Special FilesFiles, Directories & Special Files
2GWU CS 259 Brad Taylor Spring 2004
Objectives
•Complete OS Basis MaterialComplete OS Basis Material
•OS Input/Output Handling Means OS Input/Output Handling Means
•User & Program Interaction (Files, User & Program Interaction (Files, Directories)Directories)
•Process Communication Primitives Process Communication Primitives (Pipes, Named Pipes, …)(Pipes, Named Pipes, …)
3GWU CS 259 Brad Taylor Spring 2004
Input / Output, Files & Directories
• What are they? Why use them?What are they? Why use them? • How do they work?How do they work? • How are they integrated?How are they integrated? • How deal with resource & time How deal with resource & time
contention concurrently?contention concurrently?
4GWU CS 259 Brad Taylor Spring 2004
Brainstorm onto board …Brainstorm onto board …
How many How many differentdifferent means means can we list / discuss?can we list / discuss?
Inputting Data
HelloHal.
Hello Dave
!
Its not behaving … what do Its not behaving … what do we do???we do???
5GWU CS 259 Brad Taylor Spring 2004
Inputting Data
KeyboardKeyboard
Mouse / TrackballMouse / Trackball
Drawing Tablet Drawing Tablet
Handwritten charactersHandwritten characters
Magnetic-ink & Optical Magnetic-ink & Optical character recognitioncharacter recognition
ScannerScanner
Optical mark recognitionOptical mark recognition
Digital CameraDigital Camera
Bar codesBar codes
Voice inputVoice input
Touch screensTouch screens
Eye Motion (Look)Eye Motion (Look)
Midi DevicesMidi Devices
Sensor DevicesSensor Devices
Clock, GPS Clock, GPS
Network Network
SERIAL, vs. PARALLEL, SERIAL, vs. PARALLEL, GPIB GPIB
6GWU CS 259 Brad Taylor Spring 2004
Sensor Devices
HomeHome-- Smoke Detectors Smoke Detectors-- Room Temperature Room Temperature-- Motion (Presence) Motion (Presence) DetectorsDetectors- - Continuity Continuity DetectionDetection-- Security Code or Security Code or Personal Personal IdentificationIdentification
IndustrialIndustrial- Valve Position- Valve Position- Electrical Contacts- Electrical Contacts- Gas & Fluid Flow - Gas & Fluid Flow - Photocell- Photocell- Pressure Detection- Pressure Detection- Level Measurement- Level Measurement- Safety Circuits- Safety Circuits- Electrical Current- Electrical Current- Finger /Voice - Finger /Voice /Retinal ID/Retinal ID
7GWU CS 259 Brad Taylor Spring 2004
Digital Cameras
Digital cameras take photographs and store Digital cameras take photographs and store them internallythem internally
Photos are then sent to the computer for Photos are then sent to the computer for output, or stored on disk.output, or stored on disk.
Photographic Data Compression StandardsPhotographic Data Compression Standards
8GWU CS 259 Brad Taylor Spring 2004
Output of Information
Ideas?Ideas?
Another list …Another list …
9GWU CS 259 Brad Taylor Spring 2004
Output of Information
Useful user information takes a variety of forms:Useful user information takes a variety of forms:Computer screens Computer screens (or virtual/holographic)(or virtual/holographic)Printers Printers (Impact vs. non)(Impact vs. non)Voice output Voice output
- by Analysis (words) or Rules (sounds)- by Analysis (words) or Rules (sounds)Music and Sound Music and Sound (Napster or peers …)(Napster or peers …)Actuating DevicesActuating DevicesNetworkNetworkStorage Devices (FDD/HDD/tapes/USB …)Storage Devices (FDD/HDD/tapes/USB …)HapticsHapticsGPIBGPIB
10GWU CS 259 Brad Taylor Spring 2004
Actuating Devices
HomeHome-- Fire Alarms Fire Alarms-- Heating & Heating & CoolingCooling-- Lighting Lighting (Inside & Outside)(Inside & Outside)- Refrigerator- Refrigerator
IndustrialIndustrial- Valve Positioners- Valve Positioners- Electrical - Electrical SwitchesSwitches- Laser Projection- Laser Projection- Safety Cutoffs- Safety Cutoffs- Mixers- Mixers- Lights- Lights- Sirens- Sirens- Ovens, Heaters, - Ovens, Heaters, Chillers …Chillers …
11GWU CS 259 Brad Taylor Spring 2004
UNIX Device Files
Peripheral DevicesPeripheral Devices accessed through filenames accessed through filenames in file systemin file system
readreads and s and writewrites s to theseto these device files transfers device files transfers data directly between system and devicesdata directly between system and devices
Block device files Block device files (Transfer unit is block; ex: disk)(Transfer unit is block; ex: disk)Character device files Character device files (Transfer unit is character; (Transfer unit is character;
terminal, modem, printer)terminal, modem, printer)Action Sequence Action Sequence
- Access device file’s inode- Access device file’s inode- Determine transfer unit- Determine transfer unit- Call specific device driver- Call specific device driver
12GWU CS 259 Brad Taylor Spring 2004
Files: named bytes on disk
File abstraction:File abstraction:– user’s view: named sequence of bytes user’s view: named sequence of bytes
– FS’s view: collection of disk blocksFS’s view: collection of disk blocks– file system’s job: translate name & offset to disk blocks file system’s job: translate name & offset to disk blocks
File operations:File operations:– create a file, delete a filecreate a file, delete a file– read from file, write to fileread from file, write to file
Want: operations to have as few disk accesses as possible & have minimal Want: operations to have as few disk accesses as possible & have minimal space overheadspace overhead
offset:int disk addr:int
int main() { … foo.c
13GWU CS 259 Brad Taylor Spring 2004
Low-level I/O: open, read, write, close
Example 1: Example 1: append.cappend.c main(int argc, char **argv)main(int argc, char **argv){{int n, in, out;int n, in, out;
char buf[1024];char buf[1024];
/* Open the first file for reading *//* Open the first file for reading */in = in = openopen (argv[1], O_RDONLY); (argv[1], O_RDONLY);
/* Open the second file for writing *//* Open the second file for writing */out = out = openopen (argv[2], O_WRONLY | O_APPEND); (argv[2], O_WRONLY | O_APPEND);
/* Copy data from the first file to the second *//* Copy data from the first file to the second */while ((n = while ((n = readread (in, buf, sizeof(buf))) > 0) (in, buf, sizeof(buf))) > 0)writewrite out, buf, n); out, buf, n);
}}
14GWU CS 259 Brad Taylor Spring 2004
select & poll functions
Means to monitor & access various I/O devices Means to monitor & access various I/O devices when unsure which is readywhen unsure which is ready
Considerations:Considerations:Blocking vs. non-blockingBlocking vs. non-blockingConditional vs. time-slicesConditional vs. time-slices
15GWU CS 259 Brad Taylor Spring 2004
In some sense, the problems we will look at are no different than those in In some sense, the problems we will look at are no different than those in virtual memoryvirtual memory– like page tables, file system meta data are simply data structures like page tables, file system meta data are simply data structures
used to construct mappings.used to construct mappings.– Page table: map virtual page # to physical page #Page table: map virtual page # to physical page #
– file meta data: map byte offset to disk block addressfile meta data: map byte offset to disk block address
– directory: map name to disk block addressdirectory: map name to disk block address
What hard Grouping Blocks?
Page table28 33
Unix inode 418 8003121
directoryfoo.c 3330103
Like usual, we’re going to call the same thing by different names. We’ll be using lists and trees of arrays to track integers, but instead of calling them that or page tables, now meta data. Purpose the same: construct a mapping.
Like usual, we’re going to call the same thing by different names. We’ll be using lists and trees of arrays to track integers, but instead of calling them that or page tables, now meta data. Purpose the same: construct a mapping.
16GWU CS 259 Brad Taylor Spring 2004
In some ways problem similar: In some ways problem similar: – Desire location transparency: oblivious to size & protectionDesire location transparency: oblivious to size & protection
In some ways the problem is easier: In some ways the problem is easier: – CPU time to do FS mappings not a big deal (no TLB)CPU time to do FS mappings not a big deal (no TLB)– Page tables deal with sparse address spaces and random access, Page tables deal with sparse address spaces and random access,
files are dense (0 .. filesize-1) & ~sequentialfiles are dense (0 .. filesize-1) & ~sequential
In some ways problem is harder:In some ways problem is harder:– Each layer of translation = potential disk accessEach layer of translation = potential disk access– Space a huge premium! (But disk is huge?!?!) Reason? Never Space a huge premium! (But disk is huge?!?!) Reason? Never
enough cache space: amount of data you can enough cache space: amount of data you can getget in one fetch in one fetch– Range very extreme: Many files <10k, some more than GB.Range very extreme: Many files <10k, some more than GB.– Implications?Implications?
File System vs. Virtual Memory
Recall: can fetch a track at a time, or about 64KRecall: can fetch a track at a time, or about 64K
17GWU CS 259 Brad Taylor Spring 2004
Some Working IntuitionsFS performance dominated by # of disk accessesFS performance dominated by # of disk accesses
– Each access costs 10s of millisecondsEach access costs 10s of milliseconds– Touch the disk 50-100 extra times = 1 *second*Touch the disk 50-100 extra times = 1 *second*– Can easily do 100s of millions of ALU ops in same timeCan easily do 100s of millions of ALU ops in same time
Access cost dominated by movement, not transferAccess cost dominated by movement, not transfer– Can get 20x the data for only ~5% more overheadCan get 20x the data for only ~5% more overhead– 1 sector = 10ms + 8ms + 50us (512/10MB/s) = 18ms1 sector = 10ms + 8ms + 50us (512/10MB/s) = 18ms– 20 sectors = 10ms + 8ms + 1ms = 19ms20 sectors = 10ms + 8ms + 1ms = 19ms
Observations:Observations:– All blocks in file tend to be used together sequentiallyAll blocks in file tend to be used together sequentially– Most files in a directory tend to be used togetherMost files in a directory tend to be used together– Most names in a directory tend to be used togetherMost names in a directory tend to be used together– How to exploit?How to exploit?
18GWU CS 259 Brad Taylor Spring 2004
Common addressing patterns
Sequential:Sequential:– File data processed in sequential orderFile data processed in sequential order– By far, the most common modeBy far, the most common mode– Ex: editor writes out new file, compiler reads in file, etc.Ex: editor writes out new file, compiler reads in file, etc.
Random access: Random access: – Address any block in file directly without passing through Address any block in file directly without passing through
predecessorspredecessors– Ex: data set for demand paging, databasesEx: data set for demand paging, databases
Keyed access: Keyed access: – Search for block with particular valuesSearch for block with particular values– Ex: associative data base, indexEx: associative data base, index– Usually not provided by OS Usually not provided by OS
19GWU CS 259 Brad Taylor Spring 2004
Problem: how to track file’s data?
Disk management: Disk management: – Need to keep track of where file contents are on diskNeed to keep track of where file contents are on disk– Must be able to use this to map byte offset to disk blockMust be able to use this to map byte offset to disk block– The data structure used to track a file’s sectors is called a The data structure used to track a file’s sectors is called a file file
descriptordescriptor– File descriptors often stored on disk along with fileFile descriptors often stored on disk along with file
Things to keep in mind while designing file structure:Things to keep in mind while designing file structure:– Most files are small Most files are small – Much of the disk is allocated to large filesMuch of the disk is allocated to large files– Many of the I/O operations are made to large filesMany of the I/O operations are made to large files– Want good sequential and good random access (what do these Want good sequential and good random access (what do these
require?)require?)
Fixed cost must be low, must be able to nicely represent large files, and accessing them must not take too much time
Fixed cost must be low, must be able to nicely represent large files, and accessing them must not take too much time
20GWU CS 259 Brad Taylor Spring 2004
File Representation:Handles & Descriptors
File HandlesFile HandlesTwo methods of Two methods of doing I/O in UNIX:doing I/O in UNIX:File descriptors: File descriptors: this is part of the this is part of the UNIX UNIX specification, specification, POSIX, etc.POSIX, etc.File pointers: this File pointers: this is from the C is from the C standard standard
File DescriptorsFile DescriptorsUse the POSIX Use the POSIX names names STDIN_FILENO, STDIN_FILENO, STDOUT_FILENO STDOUT_FILENO and and STDERR_FILENOSTDERR_FILENO
21GWU CS 259 Brad Taylor Spring 2004
Parent & Child Share Same File Pointer
Example 2: Example 2: fileshare.cfileshare.c main ()main ()
{{int in;int in;pid_t pid;pid_t pid;char buf[1024];char buf[1024];
system ("echo abc123 > junk");system ("echo abc123 > junk");in = open("junk", O_RDONLY);in = open("junk", O_RDONLY);
pid = fork();pid = fork();if (pid == 0) {if (pid == 0) {read(in, buf, 3);read(in, buf, 3);write(1, buf, 3); write(1, buf, 3); }}else {else {read(in, buf, 3);read(in, buf, 3);write(1, buf, 3);write(1, buf, 3);}}write(1,"\n", 1);write(1,"\n", 1);
}}
22GWU CS 259 Brad Taylor Spring 2004
Used since CTSS (1960s)Used since CTSS (1960s)– Unix picked up and used really nicely.Unix picked up and used really nicely.
Directories stored on disk just like regular filesDirectories stored on disk just like regular files– inode contains special flag bit setinode contains special flag bit set– Users can read just like any other fileUsers can read just like any other file– Only special programs can write (why?)Only special programs can write (why?)– File pointed to by the index may be File pointed to by the index may be
another directoryanother directory– Makes FS into hierarchical treeMakes FS into hierarchical tree
Simple. Simple. ++ Speeding up file ops = Speeding up dir Speeding up file ops = Speeding up dir ops! ops!
Hierarchical Unix
/
afs bin cdrom dev sbin tmp
awk chmod chown
<name, inode#><afs, 1021><tmp, 1020><bin, 1022><cdrom, 4123><dev, 1001><sbin, 1011> ...
23GWU CS 259 Brad Taylor Spring 2004
inodes
Directory entries Directory entries contain only a contain only a name and index name and index into file into file information information table. The table table. The table and index are and index are both referred both referred to as an inode.to as an inode.
24GWU CS 259 Brad Taylor Spring 2004
ptr 1ptr 2 …ptr 128
Multi-level indexed files: ~4.3 BSD
File descriptor (File descriptor (inodeinode) = 14 block pointers + “stuff”) = 14 block pointers + “stuff”
Ptr 1ptr 2ptr 3ptr 4...
ptr 13ptr 14
stuffdata blocks
ptr 1ptr 2 …ptr 128
Indirect block
Double indirect block
Indirect blks
25GWU CS 259 Brad Taylor Spring 2004
Indexed files (DOS FAT)Each file has an array holding all of it’s block pointersEach file has an array holding all of it’s block pointers
– (purpose and issues = those of a page table)(purpose and issues = those of a page table)– Max file size fixed by array’s size (static or dynamic?)Max file size fixed by array’s size (static or dynamic?)– Create: allocate array to hold all file’s blocks, but allocate on Create: allocate array to hold all file’s blocks, but allocate on
demand using free listdemand using free list
– Pro: both sequential and random access easyPro: both sequential and random access easy– Con? (When did you last defragment your HDD?)Con? (When did you last defragment your HDD?)
file a file b
26GWU CS 259 Brad Taylor Spring 2004
Pro:Pro:– Simple, easy to build, fast access to small filesSimple, easy to build, fast access to small files– Maximum file length fixed, but large. (With 4k blks?)Maximum file length fixed, but large. (With 4k blks?)
Cons:Cons:– What’s the worst case # of accesses?What’s the worst case # of accesses?– What’s some bad space overheads?What’s some bad space overheads?
An empirical problem:An empirical problem:– Because you allocate blocks by taking them off unordered Because you allocate blocks by taking them off unordered
freelist, meta data and data get strewn across diskfreelist, meta data and data get strewn across disk
Unix discussion
27GWU CS 259 Brad Taylor Spring 2004
Latency tricksLatency tricks– Caching (data centric)Caching (data centric)– Code migration (code centric)Code migration (code centric)– Prefetching (dual: asynchronous writes)Prefetching (dual: asynchronous writes)– Clever data layoutClever data layout
Throughput tricks:Throughput tricks:– Increase bandwidth? Duplicate device N timesIncrease bandwidth? Duplicate device N times– Hide latency? Run another thread while waitingHide latency? Run another thread while waiting– ““Money can buy bandwidth. Latency requires bribing God.”Money can buy bandwidth. Latency requires bribing God.”
These tricks are eternal themesThese tricks are eternal themes– Use whenever need fast access to data living in slow placeUse whenever need fast access to data living in slow place– Common straws: I/O bus, memory bus, network, space (bi-Common straws: I/O bus, memory bus, network, space (bi-
directional), time (uni-directional)directional), time (uni-directional)
Sucking & Blowing Data through Slow Straws
Once there, can usually get a chunk rather than one thing. So clusterOnce there, can usually get a chunk rather than one thing. So cluster
28GWU CS 259 Brad Taylor Spring 2004
Importance of writes?Importance of writes?
Write through (simple, but slow):Write through (simple, but slow):– Whenever modify cached block, write block to disk.Whenever modify cached block, write block to disk.– Con: makes writes slow.Con: makes writes slow.– Pro: keeps FS in consistent state. PC OSes do this. Pro: keeps FS in consistent state. PC OSes do this.
Write back (faster, but complex (and dangerous)):Write back (faster, but complex (and dangerous)):– Whenever modify block, mark as dirty. Flushed laterWhenever modify block, mark as dirty. Flushed later– Pro: fast writes, absorbs writes, enables batchingPro: fast writes, absorbs writes, enables batching– Con: More you defer write back, the worse a crash is.Con: More you defer write back, the worse a crash is.
Cache Write back Policies
size
time
Memory size
File size Implications?
Which do DOS and MacOS do as user can pull floppy out at any time?Which do DOS and MacOS do as user can pull floppy out at any time?
29GWU CS 259 Brad Taylor Spring 2004
Fundamental tension:Fundamental tension:– On crash, all modified data in cache is lost.On crash, all modified data in cache is lost.– Longer you postpone write back, the worst the damage is, but Longer you postpone write back, the worst the damage is, but
the faster you are.the faster you are.
Four times to flush:Four times to flush:– When block evicted (this is ~ what VM does)When block evicted (this is ~ what VM does)– When a file is closed (distributed file systems do this)When a file is closed (distributed file systems do this)– On an explicit flush (“man sync”)On an explicit flush (“man sync”)– When a time interval elapses (30 secs in Unix)When a time interval elapses (30 secs in Unix)
Write back doesn’t respect orderingWrite back doesn’t respect ordering– Crash = file in weird jumbled stateCrash = file in weird jumbled state
Finally: OS may not have any choice!Finally: OS may not have any choice!– Disk could be doing write buffering.Disk could be doing write buffering.
Write back complications
30GWU CS 259 Brad Taylor Spring 2004
Losing some blocks worse than losing others. Usually:Losing some blocks worse than losing others. Usually:
File system effects:File system effects:– Flush modified meta data Flush modified meta data
back quickly back quickly – Note: was an implicit Note: was an implicit
side-effect of synchronous writes for meta data modsside-effect of synchronous writes for meta data mods
Application effects:Application effects:– Have a really important file? Issue a manual flush (sync) to Have a really important file? Issue a manual flush (sync) to
make sure its saved to disk.make sure its saved to disk.– Databases are forced to do this. Frequently, they will just by-Databases are forced to do this. Frequently, they will just by-
pass file system. pass file system.
Flushing nuance: All blocks are not equal
Data block
inode
directory
importance
superblock
31GWU CS 259 Brad Taylor Spring 2004
File Sharing between Processes (Inheritance)
Active
32GWU CS 259 Brad Taylor Spring 2004
Filters and Redirection
Filters, pipes (“|”) Filters, pipes (“|”) provide a means provide a means for UNIX utilities for UNIX utilities to transform to transform input (read from input (read from standard input) standard input) to an output (on to an output (on standard output)standard output)
Examples: Examples: moremore, , grepgrep
Redirection: take Redirection: take input (“<“) or input (“<“) or provide output provide output (“>“) elsewhere(“>“) elsewhere
33GWU CS 259 Brad Taylor Spring 2004
Pipelines: Inter-Process Communication (IPC)
pipepipe calls, similar calls, similar to use at the to use at the command command prompt (“|”) prompt (“|”) provide a provide a communication communication buffer means to buffer means to process dataprocess data
A FIFO (or named A FIFO (or named pipe) processes pipe) processes data on first in, data on first in, first out basis first out basis and persists and persists beyond process beyond process execution.execution.
34GWU CS 259 Brad Taylor Spring 2004
Child Writes to Parent
Example 3: Example 3: pipedate.cpipedate.cmain()main()
{{pid_t pid;pid_t pid;int pfd[2];int pfd[2];int i, status;int i, status;char line[64];char line[64];
pipe(pfd);pipe(pfd);
pid = fork();pid = fork();
if (pid == 0) {if (pid == 0) {dup2(pfd[1], 1);dup2(pfd[1], 1);close(pfd[1]);close(pfd[1]);close(pfd[0]);close(pfd[0]);
execl("/bin/date", "date", 0);execl("/bin/date", "date", 0);}}
close(pfd[1]);close(pfd[1]);
read (pfd[0], line, read (pfd[0], line, sizeof(line));sizeof(line));printf("date from child is: %s\printf("date from child is: %s\n", line);n", line);
close(pfd[0]);close(pfd[0]);waitpid(pid, &status, 0);waitpid(pid, &status, 0);exit(0);exit(0);}}
35GWU CS 259 Brad Taylor Spring 2004
Parent Writes to Child
Example 4: Example 4: pipemail.cpipemail.cmain()main()
{{pid_t pid;pid_t pid;int pfd[2];int pfd[2];int i, status;int i, status;char *username;char *username;
username = cuserid(NULL);username = cuserid(NULL);
pipe(pfd);pipe(pfd);
pid = fork();pid = fork();
if (pid == 0) {if (pid == 0) {
dup2(pfd[0], 0);dup2(pfd[0], 0);close(pfd[1]);close(pfd[1]);
execl("/bin/mail", "mail", execl("/bin/mail", "mail", username, 0);username, 0);}}
close(pfd[0]);close(pfd[0]);
write(pfd[1], "Greetings and write(pfd[1], "Greetings and salutations,\n\n", 28);salutations,\n\n", 28);write(pfd[1], "This is your write(pfd[1], "This is your program saying hello.\n", 35);program saying hello.\n", 35);write(pfd[1], "Have a nice write(pfd[1], "Have a nice day.\n\n", 18);day.\n\n", 18);write(pfd[1], "Bye.\n", 5);write(pfd[1], "Bye.\n", 5);
close(pfd[1]);close(pfd[1]);waitpid(pid, &status, 0);waitpid(pid, &status, 0);exit(0);exit(0);}}
36GWU CS 259 Brad Taylor Spring 2004
I/O, Files & Directories Summary
. • Completes discussion of basic issuesCompletes discussion of basic issues• I/O handling in terms of filesI/O handling in terms of files• Manipulation of files, directoriesManipulation of files, directories
37GWU CS 259 Brad Taylor Spring 2004
Project Discussions
Code & concept issues …Code & concept issues …
Top Related