Unix

Unix History

• Evolved at AT&T Bell Labs in late sixties.• Ken Thomson, Rudd Canaday, Doug McIilroy, Joe Ossanna and Dennis

Ritchie - writers of Unix.• Originally written as OS for PDP-7 and later for PDP-11.• Liberal liscensing: Various versions.• System V in 1983 - Unification of all variants.

Features of UNIX

• Multi-User & Multiprocessor System

• Time Sharing Operating System

• Written in “C” (HLL)

• Consistent File Format - the Byte Stream

• Hierarchical File System

Features of UNIX

• Simple User Interface

• Supports Languages - FORTRAN, BASIC, PASCAL, Ada, COBOL, LISP,

PROLOG, C, C++ etc

Architecture of UNIX System

Hardware

Kernel

CompilersDatabasePackages

Other Applicationand systemsoftware

Unix commands

Features of Unix

Services Provided :

• Process Management

• Creation, Termination, Suspension and Communication

Features of Unix

Services Provided :

• File Management

• Aspects related to files like Creation & Deletion, File security etc.

File System

• Hierarchical Structure• Consistent Treatment of Data: Lack of file format• The Treatment of Peripheral Devices as Files • Protection of File Data

File System

Root (/)

bin dev etc mnt tmp unix usr

<Files> <Files> <Files> <Files> bin Kumar Sharma

<files> prog safe

File System

/ bin commonly used UNIX Command like /usr/bin who, ls, cat, wc etc. are stored here/dev - contains device files of all hardware

devices/etc - contains those utilities mostly used by

system administrator e.g. passwd, chmod, chown

File System

/tmp - used by some UNIX utilities especially vi and by user to store temporary files

/usr - contains all the files created by user, including login directory

/unix - kernel

Release V - does not contain / bin- contains / home instead of /usr

File System

• File Types

• Regular File

• Directory File

• Device File

File System

• Consists of Sequence of Logical Blocks(Each block: Multiple of 512 k bytes)

• An Installation may have• several physical disk units• each containing one or more file system

• Data from another File System can be Accessed by Mounting that File System

File System Layout

Boot Block

SuperBlock

Inode List

DataBlocks

File System

• Boot Block (numbered 0)

• Beginning of File System (1st Block)

• Contains Bootstrap Code

• Every File System has (possibly empty) Boot Block

File System

• Super Block (numbered 1)

• Size

• # of Files that can be Stored

• Free Space Information

• Inode Information

File System

• Inode blocks (numbered 2 onwards)

• Information like access permissions, data/time, phy.

Addr. etc. about each inode

File System

• Data Block : File Data & Administrative Data:stores actual contents of file

• Inode Table Fields :• File Owner Identifier - Owner & Group id• File Type - Regular, Directory, Character / Block / Pipe

File System

• File Access Permissionsr w x r w x r w x

user group othersDate & Time - Last File/Inode Modified/Accessed• No. of Links• Disk Addresses• File Size

File Permissions

• Associated with every file, useful for security.• 3 categories of users

• Owner (u)• Group (g)• Others (o)

• Access permissions• Read (r)• Write (w)• Execute (e)

Basic Unix Commands

• Logging in & out

• Logon name & password required

• Successful logon places user in home directory

• Changing password with passwd command

• file /etc/passwd stores username, password & other

login information

Basic Unix commands

• On line help with man command

• includes brief description, options & examples.

$man <command>

Commands related to Files & directories

• Checking current directory - pwd command

/usr/Kumar

• Changing directories - cd commands

/usr/Kumar

$ cd Prog

/usr/Kumar/Prog

• Moving one level up

$ cd ..

• Switching to home directory

3. Checking login directory - logname command

$ cd /usr/Sharma

$ logname

• Listing directory contents - ls commande.g. $ ls

a.outchap1chap2testtest.c$

Options available in lsOption Description

-x Displays multi columnar output (prior to Release 4)-F Marks executables with *and directories with /-r Sorts files in reverse order (ASCII collating sequence

by default)-l The long listing showing seven attributes of a file-dForces listing of a directory-a Shows all files including ., .. And those beginning with

Options available in lsOption Description

-t Sorts files by modification time-R Recursive listing of all files in sub-directories-u Sorts files by access time (when used with the -t

option)-i Shows i-node number of a file-s Displays number of blocks used by a file

E.g. $ ls - ltotal 8-rw-rw-rw- 1 Kumar group 44 May 9 09:08 dept.h-rw-rw-rw- 1 Kumar group 212 May 9 09:08 dept.q-rw-rw-rw- 1 Kumar group 154 May 9 09:08 emp.h

Consider first entry -Field1 --> mode

- r w x r w x r w x

--> user permissions --> group permissions --> others permissions

• File type

- --> regular file

d --> directory file

c --> character - read

b --> block read

Field2 - indicates number of linksField3 - File owner idField4 - Group idField5 - File size in bytesField6 - Date/time last alteredField7 - Filename

• Displaying and Creating Files - cat command

$ cat dept.lst

01|accounts|6213

02|admin|5423

06|training|1006

• Scrolling of large files can be freezed by pressing <ctrl s> and resumed by

<ctrl q>$ cat chap1 chap2

- display contents of chap2 immediately after displaying chap1

$ cat > testType here the text contentsAfter typing the entire textPress <ctrl d> to save and exit<ctrl d>

$ cat chap1 chap2- display contents of chap2 immediately after displaying chap1

• Copies a file or group of filese.g. $ cp chap1 temp/chap2Option - i (interactive)

$cp - i chap1 chap2cp: overwrite chap2 ? y

Option -r (recursive) to copy entire directory$cp - r temp newtemp

• To delete files$ rm chap1 chap2 chap3$ rm * Are you sure? y

Option - i (interactive delete)$ rm - i chap1 chap2chap1 : ? y

• Option - r (recursive delete)$ rm -r *(Warning: Pl. do not use this option)

• To rename: mv command

• Used to rename file or group of files; as well as directories

$ mv chap1 man1

• The destination file, if existing, gets overwritten

e.g. $ mv temp doc

e.g. $ mv chap1 chap2 chap3 man1

will move chap1, chap2 & chap3 to man1

directory

• To alter file permissionsFormat :chmod <category> <operation> <permission> <filenames>Category Operations Attributeu-user +assigns permission r-readg-group -remove permission w-writeo-others =assigns absolute permission x-executea-all

e.g. $ chmod u+x note$ ls - l note -rwx r-- r -- 1 …… note

e.g. $ chmod ugo+x note$ ls - l note

-rwxr-xr-x …… note

e.g. $ chmod u-x, go+r note$ chmod u+x note note1 note2$ chmod o+wx note$ chmod ugo=r note

• Octal notation• describes both category and permission• similar to =operator (absolute assignment)• read permission - assigned value

write permission - assigned value is 2

Execute permission - assigned value is 1

e.g. $ chmod 666 notewill assign read & write permission to all

e.g. $ chmod 777 notewill assign all permissions to all

e.g. $ chmod 753 note?

To create a directory - mkdir command• Creates a directorye.g. $ mkdir doce.g. $ mkdir doc doc/example doc/datae.g. $ mkdir doc/example doc

- will give error - Order important

rmdir command• Used to remove directory• Only empty dir can be deleted• More than one dir can be deleted in single command• Command should be executed from at least one level above in the

hierarchy

e.g. $ rmdir doc$ rmdir doc/example doc$ rmdir doc doc/example- will give error

Some other Unix commands :wc

• Counts lines, words and character depending on option• takes one or more filename as arguments

e.g. $ wc infile

3 20 103 infile

Some other Unix commands :wc

$ wc infile test 3 20 103 infile 10 100 180 test 13 120 283 total$ wc - l infile

3 infile$ wc - wl infile

20 3 infile

Some other Unix commands : more

• Allows user to view a file one screen at a timeSyntax :

more <options <+linenumber> <+/pattern> <filename>

more continued

• : appears after displaying screen• <spacebar> or <enter> - next screen• q - quit• n - switch to next file on command• p - switch to previous file• f - displays filename and line no.• /pattern - to locate line with pattern• ! - to run command

more continued

Command Descriptionk:n Skips to kth next file that has been

specified in the command linek:p Skips to kth previous file that has been

specified in the command line:f Displays current filename and line

number= Displays current line number

more continued

Command Descriptionspacebar Displays next screenks Skips k lines forwardkf Skips k screens forward, Reverts to beginning of file if no search

made, or to point where last search wasmade

/pattern Searches for pattern forward

more continued

Command Descriptionn Repeats last search forwardv Starts up the vi editor. Repeats previous command! cmd Executes the Unix command cmdq Exits from more

Some other Unix commands:cmp

• To compare two files & echo first mismatch• option -l gives detailed list of byte no. and different byte in octal

e.g. $ cmp note1 note2 note1 note2 differ : char3, line1$ cmp - l note1 note2 3 143 145 6 170 167 7 171 170

Some other Unix commands:comm

• To compare two sorted files• displays 3 - columnar output• first column - lines unique to first file

second column - lines unique to second filethird column - line common to both files

• option -1, -2, -3 can be used to display specific column only

Some other Unix commands:diff

• Display file differences

• Display which lines in one file have to be changed to make both files identical

Some other Unix commands:od

• Lists the octal value (ASCII) of any file content• used to display non-printing character• options

• b to display 16 byte of data on each line• c to display character above octal value

e.g. $ od - bc odfile

Some other Unix commands:od

• Lists the octal value (ASCII) of any file content• used to display non-printing character• options

• b to display 16 byte of data on each line• c to display character above octal value

e.g. $ od - bc odfile

Some other Unix commands: banner

• Creates poster by blowing up its arguments• maximum ten characters in a line

e.g. $ banner UNIX$ banner ‘UNIX’ ‘IS HERE’

Some other Unix commands:cal

• Displays calendar of a particular month/ year• any calendar from year 1 to 9999

e.g. $ cal 1991$ cal 01 1991

Some other Unix commands:date

• Displays system date• number of options available

option m - monthh - month named - day of monthy - last two digit of the yearH - hourM - minuteS - second

Some other Unix commands:date

option ( cont.. )T - time in hh:mm:ssa - day of week (Sun to Sat)r - time in AM/PM

e.g. $ date Fri Dec 7 15:00:21 EST 1990$ date +%m 12$ date +”%h%m” Dec 12

Some other Unix commands:who

• Displays current users of the system• displays logname, terminal no., login date and time

$ who$ who am i

logname of the user

Some other Unix commands

• Printing with lp command

• Spooling

• Job No

$ lp <file>

• tr command

• translation of characters

• squeeze spaces ( -s )

$ tr –s “ ” < file1

$ tr “[a-z]” “[A-Z]” < file1

• find command• locate files

find <path list> <selection criteria> <action>• The following command will locate all the .profile files in the system

$ find / -name .profile -print

• nl command• This command elaborates schemes for numbering lines.

$ nl file1.txt• This command starts numbering lines from 40 increment value is 7 and

width of numbers is 2. ( max 6 allowed )$nl -w2 -v40 -i7 file1.txt

Terminal Related Commands

• tty - displays device name of the terminal• stty -a - displays current settings• stty can also be used to change the setting

To use <ctrl c> as interrupt key instead of <Del> key

$stty intr \^c

Introduction to Bourne Shell

• Shell - The agency that sits between user and UNIX System

- much more than command processor

• Bourne Shell- named after its founder Steve Bourne

- widely used

• C Shell - product from Univ. of California, Berkeley

- advanced user interface and enhanced features

• Korn Shell - by David Korn of Bell Lab.

Introduction to Bourne Shell

Executables in /bin directory

• sh indicates - Bourne Shell

• csh if present indicates - C Shell

• ksh if present indicated - Korn Shell

Working of shell

Continous sleep-waking-waiting cycle• Performs following activities:

• Issues a $ prompt & waits for user to enter a command• After user enters command, shell scans & processes the

command.• The command is passed on to the Kernel for execution

& the shell waits for its conclusion.• The $ prompt appears so that the user can enter next

command.

Bourne shell metacharacters

• * : To match any number of characters• ? : To match with a single character• [] : Character class; Matching with any single character specified within

[]• ! : To reverse matching criteria of character class.

Bourne shell metacharacters

• \ : To remove special meaning attached to metacharacters• ; : To give more than one command at the same prompt

Shell redirections

• Every Unix command has access to:• Standard input• Standard output• Standard error

• Shell can redirect I/p, o/p or error to any physical file using meta characters “<“, “>” & “2>”

Shell redirections

Examples:

$ ls > temp

$ wc < file1.txt > result

$ cat nonexistantfile 2> err

Building block primitives

• Pipe - allows stream of data to be passed between reader & writer process

• The data that the first processor writes into the pipe is input to second process$ who | wc-l$ ls | wc -l$ ls | wc -l > fcount

Building Block Primitives

tee - standard output can be saved in a file as well as displayed - can be placed anywhere in pipeline

e.g. $ who | tee user.lstkumar tty01 May 18 09 : 32sharma tty02 May 18 11 : 18tiwary tty03 May 18 13 : 21

e.g. $ who | tee user.lst | wc-l3

Command Substitution

• Command enclosed in backquotes (`), the shell executes the command first, and the enclosed command text is replaced by the output of the commande.g.$echo The date today is `date`

The date today is Fri 27 00:12:55 EST 1990$echo The date today is; date- issues command echo and date sequentially

Command Substitution

e.g. $echo “There are `ls | wc - l` files in current directory”$cal `date “+%m 19%y”`

echo Command

• Displays argumente.g. $echo GOOD Morning

GOOD Morning$echo \**$echo ‘*’*

echo Command

• escape sequence to manipulate cursor position\t : tab\f : formfeed\n : newline\c : cursor immediately after argumente.g. $echo ‘Enter Your Name : \c’

Enter Your Name : $_

echo Command

• accepts ASCII Octal valuese.g. $echo ‘\007’

<… beep hear…:

Shell Variables

• User variables can be defined• no type declaration or initialization• Format : variable=value• Value is of string type• No space preceding and following =operator

Shell Variables

• Evaluated by prefixing variable name with $• multi word assignment must be quoted

e.g.1 $x = 37$echo $ x37

Shell Variables

e.g.2 $msg = You have mailhave not found

e.g.3 $msg = ‘You have mail’$echo $msgYou have mail

Shell Variables

$msg = You\ have\ mail

$echo $msg

You have mail

$x = aa

$y = bb$echo $x$y

Shell Variables

$echo ‘Amount = $1000’$echo Amount = \$1000- both will display : Amount =$1000$echo “Amount = $1000”Amount=000- here shell evaluated variable $1 as double quoted

Shell Variables

$p = `pwd`$echo $p$echo ${fname}emp.sh$echo ${fname}xemp.shx

Shell script: an example

$ cat script1.sh

echo ‘calender for the current month is’

cal ‘date”+%m 20%y”’

$ cat script2.shecho ‘enter your name.’read unameecho “hi $uname”

$script2.shenter your name:xyzHi xyz

Simple Filters

• Central tools of Unix toolkit

• useful text manipulators

• both horizontal & vertical filters available

head Command

• Displays the beginning of one or more file se.g. $head -3 emp.lst

• will display first 3 lines• if number not specified (default) 10 lines

$head -2 emp.lst dept.lst

tail command

• displays the end of a file• default - display last 10 lines• $tail -3 emp.lst

• displays last 3 lines• $tail +50 emp.lst

• displays till end of file starting from line no. 50• OR• tail -n +2 emp.lst

tail command

$tail -15c emp.lst• display last 15 characters

$tail -1b emp.lst• block is used as unit of measurement

Simple filters

Cut :• Allows slicing file vertically• identifies both column and fields

e.g. $cut -c6-22, 24-32 emp.lst• cuts the file emp.lst vertically from column 6 to 22 and

from column 24 to 32$cut -c-5, 10-15, 25 - emp.lst implies 1-5 implies 25 - end of file

Simple filters

Cut :• -d option to specify delimiter can be used with -f option

which specifies field no.• default delimiter is tab• cut selects entire line if not properly delimited

$cut -d “|” -f 2,3 emp.lst$cut -d “|” -f 1,5- empl.lst | tee temp

Simple filters

Paste :paste more than one file laterally$paste file1 file2<…file1> <file2>

• delimiter can be inserted between two files

$paste -d “|” file1 file2

Sorting a file

• Reorders a line in a ASCII collating sequence• sorting can be done on a field• delimiter can be specified, -t option

$sort -t “|” +1 emp.lst

+1indicates sort after skipping first field i.e.on 2nd fieldOr$sort –t “|” –k 2 emp.lst

• sorting in reverse order -r option$sort -r +1 emp.lst

Sorting a file

• output filename can be specified as argument$sort -0 sortedf emp.lst

• -c (check) option to check if sort successful. No message indicates sort successful

• char position can be specified$sort -t “|” +4.6 - 4.9 emp.lst• will sort 7th & 8th column of field 5• Or$sort –t “|” –k4.6, 4.9 emp.lst

• sorting on multiple fields$sort -t “|” +2-3 +1 file1

$sort -m file1 file2 file3

• merges sorted files• Numeric Sort

$sort -n file1

Sorting a file

Uniq Command

• removes duplicate records (-d option)

$uniq dept.lst

$sort dept.lst | unique - uniqlst• -u option if only unique records to be listed• -d if only one copy of repeated records to be selected

Editors

• Three Types• Line Editor - ed, ex• Screen Editor - vi• Stream Editor - sed

vi Editor

• Three Modules :

CommandMode

exMode

Input command

i, I, a, A,r, R, o, O,s, S

<Enter>

<ctrl d>or exit

Input Mode Commands

Command Functioni Insert text to left of cursorI Inserts text at beginning of linea Appends text to right of cursorA Appends text at the end of lineo Opens line belowO Opens line above

Input Mode Commands

Command Functionrch Replaces single character under

cursor with character ch (no<Esc>)R Replaces text from cursor to rights Replaces single character under

cursor with any number ofcharacters

S Replaces entire line

vi Editor - Save & Quit

• From input mode ZZ• From input mode to command mode press <Esc> and from command

modeTo Save :wTo Quit :qTo save & quit :wq

vi Editor - Command Mode

Deletionx - single char4x - deletes 4 characters from cursor to rightdd - line delete4dd - deletes 4 lines

Cursor Movement Commands

Command Functionh or backspace Moves cursor leftj Moves cursor downk Moves cursor upI or spacebar Moves cursor right^ Moves cursor to beginning of first word

of line (no repeat factor)

Cursor Movement Commands

Command FunctionO or l Moves cursor to beginning of line (no

repeat factor with O)$ Moves cursor to end of lineb Moves cursor backwards to beginning

of worde Moves cursor forward to end of wordw Moves cursor forward to beginning of

Paging Functions

Command Function<Control-f> Full page forward<Control-b> Full page backward<Control-d> Half page forward<Control-u> Half page backward<Control-l> Redraw page screen (no repeat factor)

Search and Repeat Commands

Commands Functions/pat Searches forward for pat?pat Searches backward for pattern patn Repeats search in the same direction along

which the previous search was made (norepeat factor)

N Repeats search in a direction opposite tothat which the previous search was made(no repeat factor)

Search and Repeat Commands

Commands Functions

fch Moves cursor forward to first occurrence of character ch in the current line

tch Moves cursor forward onto but before first occurrence of character ch in the current line

; Repeats search in the same direction along which the previous search was made with f or t

, Repeats search in a direction opposite to that along which the previous search was made with f or t

vi Editor - Other Features

• Joining lineJ - to join current line with next line4J - to join 4 lines from current line

• Marking Textm <single char label> - to mark line<single char label> - to locate marked line

vi Editor - Other Features

• Repeating last Instruction - . (dot key)• Undo last Instruction - u• Reverse all changes made to current line - U

A Few Specimen Operator-Command Combinations

Command Function

d$ or D Deletes from cursor to end of line5dd Delete five linesd/endif Deletes from cursor up to the first occurrence

of the string endif in the forward directiond30G Deletes from cursor up to line number 30df. Deletes from cursor to first occurrence of a

dotc0 Changes from cursor to beginning of linec$ or C Changes from cursor to end of line

A Few Specimen Operator-Command Combinations

Command Function3cw or c3w Changes three wordscc Changes current linecG Changes from cursor to end of filey$ or Y Yanks from cursor to end of line3yw or y3w Yanks three words from cursor position5yy Yanks five linesy?case Yanks from cursor up to the first occurrence

of the string case in the reverse!Gsort Sorts all lines from current position to end of

ex mode commands

Command Function:r note Reads file note into present cursor location:w saves buffer and remains in editing mode:w note1 Creates file note1 and writes buffer contents

to it:w! note1 Overwrites contents of file note1 with

contents of buffer:w>> note1 Appends buffer contents to file note1

ex mode commands

Command Function:x Saves buffer and quits editing mode:wq Saves buffer and quits editing mode:q Quits editing mode without saving:q! As above but after abandoning changes:e note1 Stops editing current file, and edits file note1“e! note1 As above, but after abandoning changes

made to current file

ex mode commands

Command Function:e! Loads last saved edition of current file:e # Returns to editing the most recently edited

file:n Edits next file (on the vi command line):rew Rewinds file list to start editing from first file

specified in the command line100 Cursor at start of 100th line

Customising editing environment

• set• map• abbreviate

Set Options Used by ex/vi

Option Abbreviation Significance

autoindent ai Next line starts at the previous indented level

autowrite aw Writes current file automatically when-ever

switching files with :n and escaping to the

shell with :sh

ignorecase ic Ignores case while searching for patterns

magic Treats the characters of the regular

expression set as special while searching

for patterns

Set Options Used by ex/vi

Option Abbreviation Significance

number nu Displays line numbers on screen

showmatch sm Shows momentarily the mach to a ) & }

showmode Displays a message when vi is in input

tabstop ts Sets tab for display (default : 8 spaces)

wrapscan ws Continues pattern search by moving to

the other end of a file

Map & umap commands

• Map : used to assign undefined key or reassign defined ones.:map v “ayyto display current settings:mapto cancel a mapping:umap v

Abbreviate

• For mapping of characters in input mode ab se standard error

Options to vi

• To salvage a file after crash $ vi -r <file>

• to begin from specified line$ vi +<line> <file>

• to begin with specified pattern$ vi +/<pattern> <file>

grep Command - Advanced Filter

• Searches for a given pattern in the files specifiedgrep <options> <pattern> <filename(s)>

• Options-c displays count of occurrences -l displays list of filenames only-n displays line number along with lines-v displays all but the line matching pattern

grep Command - Advanced Filter

• Options ( cont..)--i ignores case-h omits filename-f fname reads expression/pattern from file

(egrep & fgrep option)-x displays lines match in entirety

(fgrep option)

grep Command

e.g. $grep sales emp.lst- lists records with sales as dept$grep ‘sales’ emp.lst$str = ‘sales’$grep -c “$str” emp?.lst• will count no. of directors in all emp?.lst• display will be for each file separate

Regular Expressions with grep

• Used to specify pattern• specified in single quotes

Expressionch* - 0 or more occurrence of char ch[pqr] - matches single char p, q or r[^pqr] - matches single char which is not p, q

or r[c1-c2] - match single char within ASCII range

specified

Regular Expressions with grep

• Expression.(dot) -matches with any single character^pattern - matches with pattern at beginning of linepattern$ - matches with pattern at end of linech \ {m\} - matches m occurrence of char chch \ {m,n\} - matches min. m & max. n occurrence

of char chch \ {m,\} - matches min. m occurrence

Regular expression with grepConsider following R.E.(i) ‘[cC]ho [wu]dh*ury’(ii) ‘[7-9] …$’ char [7-9] followed by 3 more

chars rep by … at end of line(iii) ‘director.*[7-9]…$’(iv) ‘^[\t]*$’(v) ‘^ ……… 03’

.\{10\}(vi) [0-9] [0-9] [0-9] [0-9]

egrep Command

egrep - Extending grep• extends grep pattern matching capabilities

• Extending R.E. features

ch+ matches 1 or more char ch

ch? Matches or 1 char ch

exp1/exp2 matches exp1 or exp2 expression

(x1/x2)x3 matches expression x1x3 or x2x3

x1/x2 matches expression x1 or x2

egrep Command

e.g. $ egrep ‘(sen/das)gupta’ emp.lst

$ egrep ‘[aA]gg?[ar]+wal’ emp.lst

$ egrep -f pat.lst emp.lst

fgrep Command

fgrep - multiple string searching• only simple (pattern) string or group of strings can be searched• faster than grep and egrep

e.g. $fgrep -f pat.lst emp.lst$cat pat.lst sales admin

Some limitations of grep family

• No options to identify & work with fields• Output formatting, computations etc. Not possible• Extremely difficult to specify patterns/ R.E always.

Advanced Filter - awk

• Report writer : very powerful text manipulation utility• Named after Aho

WeinbergerKernigham

• As powerful as any programming language• It can access, transform and format individual fields in a record• Generally awk programs are slow.

• Syntaxawk <options> ‘line specifier {action}’ <files>

Advanced filter awk

Simple awk filteringawk ‘/director/ {print}’ emp.lst

line specifier Action(selects line (as has an effect of for processing) printing all selected lines)

• $0 variable stores entire line

• Line specifier and action option are optional, either of them needs to be specified

• line specifier not specified indicates all lines to be selected• {action} omitted indicates print (default)• fields are identified by special variable

$1, $2, ….; default delimiter is contiguous string of spaces

• Explicit delimiter can be specified using -F optione.g. awk -F “|” ‘/sales/{print $3, $4}’ emp.lst

• Regular expression of egrep can be used to specify pattern• Line numbers can be selected using NR built-in variable

$awk -F “|” ‘NR ==3, NR ==6 {print NR, $0}’ emp.lst

• Output can be formatted using printf• similar to printf in ‘c’ but not a function

e.g. $awk -F “|” ‘/[aA]gg?[ar]+wal/{>printf “%3d%-20s%-12s%d\n”, NR, $2,$3,$6}’>emp.lst

Record no./Line number will be printed

• , (comma) to be used as delimiter in printf

• Logical and Relational operators• Logical operator &&, ||

$awk - F “|” ‘$3 ==“director” || $3 == “chairman”{>printf “%-20s”,$2}’ emp.lstRelational Operators : <, <=, ==, !=, >=, >, ~, !~e.g. $awk -F “|” ‘$6>7500 {printf “%20s”, $2}’

emp.lste.g. $awk -F “|” ‘$3 == “director” || $6>7500 {

>print $0}’ emp.lst

• == tries to find perfect match but string may have trailing spaces to overcome this ~ and !~ (match & negate of match) can be used with R.E.

$awk -F “|” ‘$2~/director/||$~/g.m/{>printf $0}’ emp.lst

$3 ~/^g.m/

Number Processing• Computation on numbers can be done

+, -, *, /, % operator available• no type declaration for variables• variables initialized to zero

$awk -F “|” ‘$3 == director” || $6>7500 {>kount = kount+1>printf “%3d%-20s\n”, kount,$2}’ emp.lst

awk Command

• -f option• awk program can be written in a separate file and used in awke.g. $awk -F “|” -f emp.awk emp.lst

single quoted contents are written in this file without quotes

awk Command

• BEGIN and END SectionFormat: (i) BEGIN {action}

(ii) END {action}

awk<options> ‘BEGIN {action} line specifier {action} END {action}’ <files>

awk Command

e.g. $cat emp.awkBEGIN {printf “\n\t Employee details \n\n”}$6>7500{# increment sr. no. and sum salarykount++; tot+=$6printf “%d%-20s%d\n”, kount, $2, $6}END {printf “\n The Avg. Sal. Is %6d\n”, tot/kount}$_&awk -F “|” -f emp.awk emp.lst

awk Command

• Positional parameters and shell variable

• requires entire awk command should be in the shell script• to differentiate positional parameter and field identifier,

the positional parameter should be single quoted in awk program

e.g. $3 > ‘$1’

Enhancements to awk available as new command - nawk.• Its possible to use getline statement to accept I/p from the user or from a file.• Its possible to use the built-in variables ARGC & ARGV to check arguments

passed.• Its possible to define own functions.• Its possible to use the system function to run unix commands or any other

executable within nawk.• The ?: operator as in “C” is available within nawk. getline x < “Emp.lst”

getline x < “/dev/tty”

Built-in Variable

NR - Cumulative no. of records readFS - The i/p field separatorOFS - The output field separatorNF - No. of fields in current recordFILENAME- The current i/p fileARGC - No. of arguments in command lineARGV - List of arguments

Built in functions

Functionsint(x) - Returns integer value of xsqrt(x) - Returns square root of xindex(s1,s2) - Returns the position of string s2 in

s1length() - Returns length of the argument

Built in functions

Functionssubstr(s1,s2,s3) - Returns portion of string of length

- s3,starting position s2 in string s1split(s,a) - Split string s into array a optionally returns

no. of fields

Arrays

• Can handle single dimension array • index can be anything even string• no declaration required, initialized to zero automaticallye.g. totl[1] +=$6; tot[2]+=da; tot[3]+=HRA; temp[$3]

Interface with Shell

• Redirection > , piping | possible• restriction - filenames, commands that follows these symbol must be in

double quotesProgramming constructs :(I)if … then …

if (conditions is true)<statements>

else<statements>

Programming constructs :(II) For statement

for (i=1; i <= 9; i++){

(iii) for loop for arraysfor (k in array)

<commands>(iv) While loop

while (condition is true){}

(v) break and continue

Shell Variables

• System Variables

• Set during - booting- login

• script executed during login time - profile• profile alters operating environment of a user• to display list of system variable $set

Shell Variables

• list of variable

PATH - list of dir. In order of precedence that have to be scanned for hunting command

Home - dir. Named after Login name

IFS - string of char which are used as word separator in the command line

MAIL - mail directory

PS1 - Two prompts - primary and secondary

PS2 - (used by sed, awk)

TERM - Terminal type

Shell Variables

DATA in the same file - The here document• < redirection symbol, redirects the standard input to come from a file• << here document symbol signifies data is here rather than in a Separate

Shell Variables

e.g. $grep director <<mark - delimiter to indicate end of document

>1001 | Sanjay | g.m. | 5 >5001 | Rahul | director | 7

Read Statement

• Takes input from standard i/p• can be used to read one or more variables

e.g.echo “\n Enter the Pattern : \c”read patnameecho “\n Enter file name : \c”read fname grep “$patname” $fname

Command Line Arguments

• Shell procedures arguments in command• Arguments are specified along with the name of shell procedure• Arguments are assigned to special variables (positional parameters) $1 - First

parameters

$2 - Second parameters

• Special parameters also available$0 - Name of executed command$* - Complete set of positional parameters$# - No. of arguments$$ - PID of current shell$! - PID of last background job$? - Exit status of last command$@ - Same as $* except when enclosed in double quotes

E.g.1 if [$# -nc 3]then

fi2 $kill - 9 $$3 grep “$1” $24 grep director emp.lst >/dev/null;echo$?

Exit Status indicates successful1 indicates failure2 indicates unreadable file

Conditional Execution

• Logical Operators && and ||• && operator delimits two commands, the second command is executed only if

first succeeds• || operator delimits two commands, the second command is executed only if

first fails

e.g. $grep `director` emp.lst && echo “patern found”

$grep `manager` emp.lst || echo “pattern not found”

Conditional Execution

Script Termination - exit statement

• premature termination of program• control returned to the calling program

e.g. grep “$1” $2 || exit 2

echo “pattern found”

Return value

default zero

if Statement

• Format :(i) if <condition is true>

then<execute commands>

else(execute commands>

fi(ii) if <condition is true>

then<execute commands>

if Statement

(iii) if <condition is true>then

<execute commands>elif <condition is true>then

<…>else

<execute commands>fi

if Statement

• Condition can be specified either using test or [cond]• Relational Operator used

0 eq Equal to0 nc Not equal to0 gt Greater than0 gc Greater than or equal to0 lt Less than0 lc Less than or equal to

if Statement

• File related tests with test

-a <file> Trace if <file> exists-f <file> if exists & regular file-r <file> if exists & readable-w <file> if exists & writable-x <file> if exists & executable-d <file> if exists & is dir file-s <file> if exists & size > 0

Test - String Comparison

• String tests used by test

0 n str True, if str not a null string0 z str True, if str is a null string0 S1 = S2 True, if S1 = S20 S1! = S2 True, if S1 S20 str True, if str is assigned and not null

• Logical Operators

0 a .AND.0 O .OR.0 ! Not

Test - String Comparison

e.g. If test $x -eq $y if [$x -eq $y]

e.g. If [! -f fname]then

echo “file does not exists”fi

The Case Statement

• Format :case <expression> in<pattern 1> ) <execute commands> ;;<pattern 2> ) <execute commands> ;;

<…><…>

The Case Statement

e.g. echo “\n Enter Option : \c”read choicecase $choice in1) ls -l ;;2) ps -f ;;3) date ;;4) who ;;5) exit ;;

The Case Statement

E.g.Case `date | cat -d “ “ -fi ` inMon > <commands> ;;Tue > <commands> ;;

The Case Statement

Case “$ans” in[yY] [eE] [sS] ;;

[nN] [oO] exit ;;*) “invalid option” ;;

While Statement

• Format :while <condition is true>do

<execute statements>done

While Statement

e.g. while [$x -gt 3]do

ps -fsleep 5

donee.g. while true

dops -fsleep 5

While Statement

E.g. Script to edit, compile & execute a programwhile truedo

cc $1case $? In

o) echo “Compilation Successful” echo “Executing a.out” a.out ; exit ;;

*) echo “Compilation Error” echo “Press <Enter> to edit” read pause vi $1 ;;esacdone

Break & Continue Statement

• Continue : Suspends execution of all statements

following it, and switches control to

top of loop for next iteration• Break : causes control to break out of the loop

e.g. Validation of designation codefname = desig.lstwhile echo “Designation code : \c”do

read desigcase “$desig” in[0-9] [0_9] if grep “^$desig” $fname

>/dev/nullthen

echo “code exists”continue

fi ;;*) echo “invalid code”

continue ;;esacdone

Until Statement

• Complement of while stm• loop body is executed repeatedly as long as the condition remains false• e.g. until false

dops -fsleep 5

for Statement

• Format :for variable in listdo

<execute commands>done

for Statement

e.g. for x in 1 2 3do

echo “The value of x is $x”done

e.g. for var in $PATH $HOME $MAILdo

echo “$var”done

e.g. for file in *.cdo

cc $filedone

While Statement

cc $1case $? In

>/dev/nullthen

continue ;;esacdone

Until Statement

dops -fsleep 5

for Statement

echo “$var”done

cc $filedone

Process

• Instance of program in execution

• Many processes can run at the same time

• Every process identified by Process Identifier

• PID is allocated by kernel

Parent & child processes

• On logging to system, a process is set up due to execution of shell

• Shell is the parent process for every other process setup due to execution of commands

• Every process has a parent process

• Parent process waits for death of child process before resuming execution.

Process Status

• ps command to display characteristics of a process• options

-f - full form -u - details of only users processes-a - all processes details-l - detailed listing-e - system processes

Background Process

• To run a process in background, use & operator• $sort -0 emp.lst emp.lst &

• nohup (no hangup) - permits execution of process even if user has logged off$nohup sort emp.lst &

sends output to nohup.out

Process Related Commands

• wait• waits for child process to complete $wait 138 - waits for background job with pid 138

• kill$kill 1005 (default signal 15)• kills job with pid 1005$kill -9 1005

sure killing of job$kill 0 - kills all background process

Process Related Commands

• nice - job execution with low priority• prefix the command with nice

$nice wc -l xxx$nice -15 wc -l xxx

reduce priority by 15 units

Communication & scheduling

• News command invoked by user to read any message sent by administrator• Messages stored in /usr/news• options: -n,-s,-a

• motd• message od the day- a text file in /etc/motd• readable by all users

• write command:• allows 2-way communication between users who are logged on.• Message occurs on screen anytime it is sent.• The mesg command with parameter n prevents disturbances.

• mail command:• single channel communication.• User need not be logged on : message gets saved in users mailbox.• Many internal commands available within mail.

• wall commandused by system administrator to send messages to all logged on users.

• Some commands for scheduling:

• sleep : To introduce delay in script

• at : To schedule job execution at later date/time

• batch: To run the job when system load permits.

• cron : To schedule jobs for periodic execution.

While Statement

cc $1case $? In

>/dev/nullthen

continue ;;esacdone

Until Statement

dops -fsleep 5

for Statement

echo “$var”done

cc $filedone

Program Development

• CC

• Make

• SCCS

CC Command

• The UNIX system compiler for C. • cc compiles the file if no errors are found an executable version of the

file is created. The default name of the executable file is a.out• Invokes a series of programs – the preprocessor, the compiler and the

linker.• Subprograms can be compiled together.

Make Utility

• Make utility and makefiles help automate building of an application. • Modifying any one or more of the source files requires recompilation

of that part of the program and relinking.• Automate this process by specifying the interdependencies between

files that make up the application along with the commands needed to recompile and relink each piece.

Make Utility

• During program development a number of subroutines may be developed separately and linked at the end. However keeping a track of the updated versions is difficult. Further, files that are changed may not be recompiled.

Make Utility

• These instructions are placed in make file. The command make is used to execute instructions defined in make file.

$ make myprog

Make Utility

• A make file consists of a series of entries of the following form:

[...targetfile..]: [.....dependencies....][One Tab Space] [.....commands........]

Make Utility

main.o: main.c cc -c main.c

• Makefile and all files referenced by MAKE should be in the same directory.• Sees the dependencies upon program.Then checks to see if program is current

SCCS-Source code control system

• SCCS is a group of programs that help in controlling the various versions of a program.

• Multiple copies are not stored. Only incremental changes to the program are stored.

SCCS-Source code control system

• The following commands are used with SCCSAdmin : used to create initial versions of the fileGet : used to retreive the SCCS filesComb : used to combine two or more versions and remove redundant versionsRemdel : used to remove a versionPrs : removes information about a SCCS file

Unix

Documents

Transcript of Unix

Unix, Perl and BioPerlbarc.wi.mit.edu/education/bioinfo2006/unix-perl/... · Unix, Perl and BioPerl I: Introduction to Unix for Bioinformatics George Bell, Ph.D. WIBR Bioinformatics

Unix Stream Unix Advanced V2.0

Unix 101: Introduction to UNIX (i.e. Unix for Windows Users)mkegel/Unix_course/Course Notes.pdf · 2006-03-23 · Unix 101: Introduction to UNIX (i.e. Unix for Windows Users) Mark

1 Case Study 1: UNIX and LINUX Chapter 10 10.1 History of unix 10.2 Overview of unix 10.3 Processes in unix 10.4 Memory management in unix 10.5 Input/output.

UNIX OPERATING SYSTEM - WordPress.comText Book: Unix Concept and application- Sumitabhadas References: Unix Shell Programming-Yashwant Kanetkar Unix Programming Environment- RobPike

UNIX ™ /Linux Overview Unix/LINUX Intro. UNIX/Linux History.

Introduction to Unix - accre.vanderbilt.eduIntroduction to Unix 3 What is Unix? Unix is an operating system (OS) like Windows UNIX was originally created in the late 1960's by Ken

UNIX 2 Enhancing your UNIX skills - Docs.is.ed.ac.uk

Unix Introduction - Department of Systems Engineering and ...seem3460/tutorial/unix/tuto1-unix-intro-2019.… · Unix Introduction Zhang Wenxuan Isaac. Unix-like system is everywhere

Introduction to UNIX. 2 Unix File System zStores Files yText, Programs, Images, etc. zMade up of Many Files yUser, Unix System, Unix Commands.

UNIX/Windows Integration Jon Dowland ISS UNIX team.

Learing outcomes Introduction to Unix system Unix commands.

Unix - Introductionkschmidt/CS265/Lectures/Unix/unix.pdf · Unix - Introduction Kurt Schmidt Preface Intro Logging In ... these notes discuss Linux, ... Unix ﬁlenames can contain

Chapter 7 UNIX and LINUX. 2 Outline Overview Processes in UNIX Memory management in UNIX I/O in UNIX UNIX file system.

Lecture 2 The UNIX Filesystem. On the last episode of UNIX Tools… Course Info History of UNIX Highlights of UNIX The UNIX Philosophy System organization.

Linux / Unix TutorialLinux / Unix Tutorialcc.ee.ntu.edu.tw/~jhjiang/instruction/courses/fall11...Linux / Unix TutorialLinux / Unix Tutorial Sppgeaker: Cheng-Shen Han (韓承駪) Instructor:

unix-ch05 maurice batch, system call in unix

UNIX · 2.6 TRADITIONAL UNIX SYSTEMS ... Kernel Memory Allocation .....24 9.3 TRADITIONAL UNIX SCHEDULING ...

UNIX/LINUX OPERATING SYSTEM. Introduction to Linux Introduction to Unix History of UNIX What is Linux Linux Distributions Linux Installation Unix File.

The UNIX System: Program Design in the UNIX Environmenta.papnet.eu/UNIX/bltj/06771909.pdf · 2018-11-09 · The UNIX System: Program Design in the UNIX Environment By R. PIKE* and