L_4-1

8/6/2019 L_4-1

http://slidepdf.com/reader/full/l4-1 1/40

Fall 2010 ~ Eric Meyer

8/6/2019 L_4-1


2

Lesson A

Extracting Information from Files

8/6/2019 L_4-1


3

Objectives Explain the UNIX approach to file processing

Use basic file manipulation commands

Extract characters and fields from a file using the cutcommand

Rearrange fields inside a record using the paste command

Merge files using the sort command

Create a new file by combining cut, paste, and sort

8/6/2019 L_4-1


4

UNIX Approach toFile Processing Based on the approach that files should be treated as nothing

more than character sequences

Because you can directly access each character, you canperform a range of editing tasks – this offers flexibility in terms of

file manipulation

8/6/2019 L_4-1


5

Understanding UNIX File Types Regular files, also known as ordinary files

Create information that you maintain and manipulate, andinclude ASCII and binary files

Directories

System files for maintaining file system structure

Special files

Character special files relate to serial I/O devices

Communicates one character at a time

Block special files relate to devices such as disks

Communicates using blocks of data

8/6/2019 L_4-1


8/6/2019 L_4-1


7

File Structures

Files can be structured in many ways depending on the kind ofdata they store

UNIX stores data, such as letters and product records, as flatASCII files

Three kinds of regular files are

Unstructured ASCII character

Unstructured ASCII records

Unstructured ASCII trees

8/6/2019 L_4-1


8

8/6/2019 L_4-1


9

Using Input and Error Redirection You can use redirection operators to retrieve input from

something other than the standard input device and sendoutput to something other than the standard output device

Examples of redirection: Redirect the ls command output to a file, instead of to the

monitor (or screen)

Redirect a program that receives input from the keyboardto receive input from a file instead

Redirect error messages to files, instead of to the screen by

default

8/6/2019 L_4-1


10

Using Input and Error Redirection

Create a file by:

typing in all thecommands, or by

redirecting the cat

command output to

a file

8/6/2019 L_4-1


11

Manipulating Files When you manipulate files, you work with the files

themselves, as well as their contents

Create files using output redirection

cat command - concatenate text via output redirection

touch command - used to create empty files

8/6/2019 L_4-1


12

Manipulating Files Delete files when you no longer needed

rm command - permanently removes a file or an emptydirectory

The -r option of the rm command will remove a directory

and everything it contains (recursive delete) The –f option forces it

So to kill a folder regardless rm –rf**do not do this to / - it will kill your system**

Copy files as a means of back-up or as a means to assist with

new file creation cp command - copies the file(s) specified by the source

path to the location specified by the destination path

8/6/2019 L_4-1


13

Manipulating Files Moving a file in order to change the directory that contains it

mv command - removes file from one directory and places it

in another

Finding a file helps you locate it in the directory structure

find command - searches for the file that has the name you

specify

8/6/2019 L_4-1


14

Manipulating Files

8/6/2019 L_4-1


15

Manipulating Files Combining files using output redirection

cat command - concatenate text of two different files via

output redirection

paste command - joins text of different files in side by side

fashion

Extracting fields of a file using output redirection

cut command - removes specific columns or fields from a

file

8/6/2019 L_4-1


16

Manipulating Files

8/6/2019 L_4-1


17

Manipulating Files Re-arranging the contents of a file

sort command - sorts a file’s contents alphabetically or

numerically

The sort command offers many options:

You can sort the contents of a file and redirect the outputto another file

Utilizing a sort key which provides the option of sorting on

a field position within each line

8/6/2019 L_4-1


18

Manipulating Files

8/6/2019 L_4-1


19

Lesson B

Assembling Extracted Information

8/6/2019 L_4-1


20

Objectives Create a script file

Use the join command to link files using a common field

Use the awk command to create a professional-looking report

8/6/2019 L_4-1


21

Using Script Files UNIX users create shell script files to contain commands that

can be run sequentially as a set – this helps with the issues of

command automation and re-use of command actions

UNIX users use the vi editor to create script files, then make the

script executable using the chmod command with the x

argument

8/6/2019 L_4-1


22

Using Script Files

Type out the script andthen make it

executable using the

chmod command.

8/6/2019 L_4-1


8/6/2019 L_4-1


25

Using the Join Command to Createthe Vendor Report

Use the join command

to create reportsshowing the

relationship between

two files

8/6/2019 L_4-1


8/6/2019 L_4-1


27

A Brief Introduction to theAwk Program

Awk uses a print

formatting function

from the Cprogramming

language to achieve a

more professional-

looking report

8/6/2019 L_4-1


8/6/2019 L_4-1


30

Standard input, output and error • standard input (0: stdin)

– The default place where a process reads its input

(usually the terminal keyboard)

• standard output (1: stdout)

– The default place where a process writes its output

(typically the terminal display)

• standard error (2: stderr)

– the default place where a process can send its error

messages (typically the terminal display)

8/6/2019 L_4-1


31

Redirecting standard I/O• Standard input and output can be redirected providing a

great deal of flexibility in combining programs and unix

tools

• Can redirect standard input from a file using <

a.out < input12

– any use of stdin will instead use input12 in this example

• Can redirect standard output to a file using >

testprog1 > testout1

cal > todaycal

a.out < input12 > testout – the stdout of a.out is directed to file testout1 in this

example

• Can also redirect stderr and / or stdout at the same time

8/6/2019 L_4-1


32

Appending to a file

• The >> operator appends to a file rather than redirecting

the output to a file

cat textinfo >assign4

prog1.exe >>assign4

prog2.exe >>assign4

cat endinfo >>assign4

8/6/2019 L_4-1


33

Pipes

• Pipes allow the standard output of one program to beused as the standard input of another program

• The pipe operator ‘|’ takes the input from the commandon the left and feeds it as standard input to the commandat the right of the pipe

Examples

ls | sort -r

prog1.exe < input.dat | prog2.exe | prog3.exe>output.dat

ls -l | cut -c 38-80

• Pipes are more efficient as compared to usingintermediate files

8/6/2019 L_4-1


34

Another Example

du -sc * | sort -n | tail

• The du command is for disk usage (default is in blocks of

512 bytes). The s and c flags are for summarize and give a

grand total respectively• the sort -n command will sort by numeric value

• head and tail commands print out a few lines at the head

or tail of the file respectively

8/6/2019 L_4-1


8/6/2019 L_4-1


36

8/6/2019 L_4-1


37

8/6/2019 L_4-1


38

Chapter Summary UNIX supports regular files, directories, and character and

block special files

File’s structures depend on data being stored and three kindsof regular files are unstructured ASCII characters, records andtrees

When running, UNIX receives input from the standard inputdevice (keyboard) also known as stdin, and sends output tothe standard output device (monitor) also known as stdout.Another standard device, stderr, refers to the error file thatdefaults to the monitor

8/6/2019 L_4-1


39

Chapter Summary The touch command updates a file’s time and date stampsand creates empty files

The rmdir command removes empty directories

The cut command extracts specific columns or fields from afile

To combine two or more files, use the paste command

Use the sort command to sort a file’s contents alphabeticallyor numerically

8/6/2019 L_4-1


40

Chapter Summary To automate command processing, include commands in a

script file that you can later execute as a program

Use the join command to extract data from two files sharing

a common field and use this field to join the two files

Awk is a pattern-scanning and processing language useful

for creating a formatted report with a professional look

L_4-1

Documents

Transcript of L_4-1