Unix Overview This is not an exhaustive training on Unix, covering all the arenas with full details....

Unix Overview

This is not an exhaustive training on Unix , This is not an exhaustive training on Unix , covering all the arenas with full details.covering all the arenas with full details.

Even the topics covered in this presentation Even the topics covered in this presentation are not all inclusive.are not all inclusive.

But, it will definitely help you to be a But, it will definitely help you to be a confident sailor in the ocean of Unix – no confident sailor in the ocean of Unix – no matter what the circumstance is.matter what the circumstance is.

Expectation from this training

This training will teach you aboutThis training will teach you about 9090 commands!!!commands!!!

And will make you bore with more thanAnd will make you bore with more than 350350

slides!!!!slides!!!!

But don’t be nervous – remember the utilities But don’t be nervous – remember the utilities of the commands – for this presentation will of the commands – for this presentation will be with you for the rest of your life to provide be with you for the rest of your life to provide you the syntax for using them.you the syntax for using them.

Don’t be nervous,please

Duration

15 Hours ( 3 Days)

Schedule

29th Sept-1st Oct,2003

Timings

11:00 am – 1 pm

2:00 pm – 5 pm

Purpose

To get acquainted with the basic Unix commands

To learn a bit of Shell Programming

Get introduced to Unix Operating System

Operating System

An operating system is a living, breathing software entity. The soul of the computing machine, it is the nervous system that turns electrons and silicon into a personality. It brings life to the computer

- from Mike Gancarz's "The UNIX Philosophy"

The OS sits between users and hardware providing translation services. It speaks the language of the hardware to perform basic tasks such as the definition of memory or the allocation of disk space using the hardware.

Unix Operating System

In UNIX, the operating system is broken into three pieces: the kernel, the shell, and the built-in utilities. The kernel is responsible for low level hardware communication, the shell provides human users with a user-friendly interface, and the built-in utilities provide basic tools for doing work.

Hardware

Kernel

Shell

•Heart of The Unix OS.

•Collection of C programs directly communicating with hardware

•Part of Unix system loaded into memory when Unix is booted

Manages:-

1. System resources

2. Allocates time between user and processes

3. Decides process priorities

Kernel

Exit

•Human interface point for Unix

•Program layer – provides an environment for the user to enter commands to get desired results.

•Korn Shell, Bourne Shell, C Shell are various shells used by Unix users

Shell

Exit

User login process

Unix booted.

Program Unix(kernel) is booted into main memory, and remains active till the computer is shut down

Program init runs as a background task and remains running till shutdown

User attempts to log in.

Kernel calls program ‘init’.

‘init’ scans file /etc/inittab , which lists the ports with terminals and their characteristics and returns an active open terminal to ‘init’.

‘init’ calls program ‘getty’, which issues a login prompt in the monitor

User enters login and password

‘getty’ calls program ‘login’ which scans file /etc/passwd to match username and password

After validation, control passes to session startup program /bin/sh , session startup program

Program /bin/sh reads file /etc/profile and .profile and sets up system wide and user specific environment.

User gets a shell promptExit

Understanding Commands

General Purpose Utilities

Handling Files

Filters

Controlling Environment

Process

System Administration

Communication

Understanding Unix Commands

By end of this section , you will be able to know

What a command is?

‘Which’ – program related to a command

‘Man’ – get online help of commands

‘History’ – previously executed commands

‘Alias’ – Call a Command by another name

To get work done by Unix, there is no way but to execute : commands.

Commands can be typed in from keyboard or taken from a file

Unix offers a variety of commands for each category of jobs.

Commands execute a program in the background which performs the desired job

Independent Commands

Do not require any input for execution

$pwd

/home/ems2000

Dependent Commands

Require input for execution

$type ls

ls is a tracked alias for /usr/bin/ls

Mixed Commands:- Can work independently/dependently

$ls

input queue sentwebcatalogs web

$ls queue

queue

General syntax to use commands

<command> [ <option flag> ] [<arguments>]

A statement asking for execution of a command usually consists of three distinct sections:-

<command> : Keyword for the command

<option flag> : Starts with ‘-’ sign. Decides the nature of output from the results of command execution

<arguments> : May be a string or the name of file(s) on which the command will act upon for desired output.

Example :

$grep -l “ems2000” *.sh

The above command from the OS prompt searches for the string “ems2000” in all the files with extension : .sh and shows the name of the files containing it.

More than one commands can be stated from OS prompt at a time

$pwd;tty

/home/ems2000/queue

/dev/pts/tb

First, pwd command is executed which displays the output : /home/ems2000/queue

Then , tty command shows the terminal no: /dev/pts/tb

Any number of commands can be specified at the OS prompt for execution separated by a ‘;’

A command can spawn more than one lines

$ echo ‘

Hello’

Hello

General Purpose UtilitiesBy end of this section , you will be able know

‘banner’ – set up poster

‘cal’ – get Calendar

‘date’ – get current date

‘calendar’ – get schedule

‘who’ – User information

‘tty’ – terminal information

‘uname’- machine info

‘uptime’ – System info

‘login’ – server login

‘telnet’ – server login

‘exit’- exit current shell

‘lock’ – lock a terminal

‘script’ – get the whole job

‘bc’ – Unix Calculator

‘expr’ – Calculations

‘factor’ – factor : number

‘primes’-Prime number

‘units’-Unit conversion

‘tput’-Control display

‘time’-Time taken by a command

Example

Use Significance

date +”%A” Full weekday name(eg., ‘Wednesday’)

date +”%b” Full month name(‘Jan’)

date +”%c” Current date & time representation

date +”%C” Century(year/100, truncated to nearest number)

date +”%d” Current Day of the month

date +”%e” Current Day of the month

date +”%H” Current hour of time(00-23)

date +”%I” Current hour of time(12 hour clock)

date +”%m” Current Month as decimal two digit number

date +”%M” Current minute of time(00-59)

date +”%n” Newline character

date +”%p” AM or PM

Use Significance

date +”%R”

Current time as %H:%M

date +”%S” Current second of time(00-59)

date +”%t” Tab character

date +”%u” Weekday as 1 digit number[1-7 for Monday-Sunday]

date +”%w”

Weekday as 1 digit number [ 0-6 for Sunday-Saturday]

date +”%V”

Week number of the year

date +”%x” Current date as dd/mm/yy

date +”%X”

Current time in hh:mm:ss

date +”%y” Two digit year

date +”%Y” 4 digit year

date +”%Z”

Time zone name

Examples of Date

$ cat calendar

I have a meeting on 07/10/2003 with Subhomoy

I have a meeting on 08/10/2003 with Biswajit

I have a meeting on 06/10/2003 with Sugata

I have a meeting on 09/10/2003 with my delivery manager

I plan for a dinner on Sep 10, 2003 with my team

I have a meeting on 11/10/2003

I have a meeting on 12/10/2003

I will not come to office on 09/11/2003

$ calendar

I have a meeting on 09/10/2003 with my delivery manager

I plan for a dinner on Sep 10, 2003 with my team

I will not come to office on 09/11/2003

$who

sqian pts/td Sep 9 18:27

ems2000 pts/te Sep 10 01:41

$who -Hu

NAME LINE TIME IDLE PID COMMENTS

sqian pts/td Sep 9 18:27 7:32 28935 135.148.207.175

ems2000 pts/te Sep 10 01:41 0:02 29362 129.42.68.104

$who am i

ems2000 pts/te Sep 10 01:41

$uptime -u

2:17am up 122 days, 11:03, 2 users, load average: 0.41, 0.41, 0.41

$uptime -h

sqian pts/td 6:27pm 7:39 rlogin htstbw00

ems2000 pts/te 1:41am -ksh

$uptime -l

2:18am up 122 days, 11:04, 2 users, load average: 0.40, 0.41, 0.41

Scale:-

By default, bc performs truncated division.One have to set scale to the number of digits of precision before performing any division.

$bc

scale=2

10/4

2.50

^d

$

If answer to division is greater than the value as dictated by the scale variable, then the value dictated by the scale is ignored and the real value is shown.

ibase and obase:-

By default, the input and output are interpreted as decimal values.But, if the demand required input and/or output in different number system(binary, hexadecimal), variables ibase and obase are set.

To convert binary input to decimal output:-

ibase=2

To convert decimal input(default) into binary output:-

obase=2

For hexadecimal systems, value ’16’ is used in ibase/obase variable.

Handling Variables:-

Variables can be used in bc mode and values can be assigned to them.

$bc

X=12 ; y=19

Z=x+y

Z

31

Conditional logics(if) ,loops(for,while), arrays, functions are supported by ‘bc’

Square Root of a Number

Syntax : sqrt ( x )

Example :

$bc

Sqrt ( 4 )

2

Length of a Number

Syntax : length ( x )

Example :

$bc

length ( 1234.5678 )

8

Trigonometric Functions

To use trigonometric functions of bc, one have to include math library. For that , issue the following command from Os prompt:-

$ bc –l

Various trigonometric functions available with ‘bc’ are:-

s(x) sine

c(x) cosine

e(x) exponential

l(x) log

String handling

To find length of a string:-

expr “<string>” : ‘.*’

Example : $ expr “Unix” : ‘.*’

4

To extract a substring from a string:-

$ expr “Subhendu” : ‘… (\..\)’

he # Shows 4th to 5th character

To locate first position of a character in a string:-

$expr “Subhendu” : ‘[^d]*h’

4 # Shows ‘h’ is at 4th position

factorFinds out factor of the integer provided

Syntax : factor <number>

$factor 15

15

3

5

$

$factor 18

18

2

3

3

$

primesShows all prime numbers between integers <lower value> and <upper value>. If upper value is not provided, it is considered to be 2,147,483,647.

Syntax : primes <lower value> <upper value>

$primes 0 10

2

3

5

7

$

units

Converts quantities expressed in various standard scales to their equivalents in other scales. Acts interactively as follows:-

System Prompt User Response

You have: inch

You want: cm

The system responds with two factors; one used if multiplying (preceded by *), the other if dividing (preceded by /):

* 2.540000e+00

/ 3.937008e-01

For a complete list of units, examine the file:

/usr/share/lib/unittab

tputControls screen display

Options Significance

tput clear Clears the screen

tput cup <r> <c>

Moves cursor to row <r> and column <c>

tput bold Bold display

tput blink Blink display

tput rev Reverse display

tput cols Shows number of columns in the screen

tput bel Echo bell character

tput lines Shows number of lines in the screen

tput smso Starts reverse display

tput rmso Ends reverse display

Examples

time

Times a command.

Command <command> is executed and time prints:-

Elapsed time during the command

Time spent in the system

Time spent executing the command

Syntax : time <command>

Example:-

$ time grep -i "Subhendu" *

ABCDEF:subhendu

bounce_off.ksh:# Developed by : SUBHENDU MAJUMDAR

drawbox.ksh:# Developed by : SUBHENDU MAJUMDAR

header:# Developer : SUBHENDU MAJUMDAR

heading:# Developer : SUBHENDU MAJUMDAR

real 6.6

user 0.8

sys 0.7

Handling FilesThis section will introduce you with

Architecture

Types of Files

Inode

File System

‘pwd’-Current Directory

‘cd’ – Change Directory

‘ls’ – List contents of a dir

Cat – Create,View,append Files

VI – Visual Editor

‘more’ – display files

‘pg’ – View files

‘cp’,’mv’ – Copy and rename files

‘rm’ – Remove Files

‘wc’ – Count word,line and characters

‘file’ – know file type

‘chmod’ – Change permissions

‘chown’ – Change owner

‘chgrp’ – Change group

‘touch’ – Change time stamp of a file

‘ln’ – Link a file to other

Handling Files…contd

‘mkdir’ – Create Directory

‘rmdir’ – Remove Directory

‘cmp’ – compare two files

‘comm’ – Compare two files

‘sdiff’ – find differences between two files

‘dircmp’ Compare Directories

‘lp’Print a file in a printer

‘lpstat’View printer status

‘cancel’Cancel print jobs

‘pr’Format file contents

Compressing and Uncompressing Files

File maintenance architecture

Formatted Disk

Partition

Partition

Partition

File system

Directory Directory Directory Directory

Directory Directory

File File File

Types of Files

Normal files

Can be text/binary files.

Can be a text file, compiled source code , executables

Directory files

Contains no external data, but details of files and sub-directories it contains.

Device files

Printers, tapes, floppy drives, CD ROMs, hard disks, terminals – all are considered as files

Inode

•Inode is a fixed format structure containing the attributes of the files stored in the file system.

•Every file has one inode, and a list of such inodes is kept in a disk area not directly accessible by user.

•Each inode is accessed by inode number, specifying the position of the inode in the list.

Some of the important information that inode contains are:-

Information Significance

Mode Permission mask and type of file

Link count No. of links associated with the file

User id ID of the owner

Group Id ID of the group

Size No. of bytes in the file

Access time Time of last access of the file

Mod time Time of last modification of the file

Inode time Time at which the inode structure was last modified

Typical File System

/ (root)

Unix bin dev usr tmp etc

usr1 usr2 usr3 bin

/Unix folder contains the kernel

/bin contains binary executable files

/dev contains device related files

/usr is the home directory for all users

/usr/usr1 is the home dir. for user : usr1

/tmp contains temporary files

/usr/bin contains additional binary unix commands

/etc contains binary executable files for system administration

pwd

Shows the directory where the user is currently in

Syntax : pwd

Example

$pwd

/home/ems2000

cdNavigates from the current directory to another directory

Syntax : cd <new directory specification>

Commands Significance

cd scriptscd ./scripts

Moves to directory : scripts under current working directory

cd ../program Change to directory program residing in the current directory's parent directory

cd /usr/fin/subhendu/ manfiles Change to the directory whose absolute pathname is : /usr/fin/subhendu/ manfiles

cdcd ~

Move to home directory of the user


cd .. Move to the parent directory of the current working directory

cd ~/subhendu Move to folder : subhendu under the home directory for the user

cdpath

This is an environmental variable which specifies the list of directories to be searched when an user issues a ‘cd’ command.

$CDPATH=.:..:$HOME

This means, when a ‘cd’ command is issued, search for the new directory first in current working directory(.)

If not found, move to the parent directory of the current directory and search there.

If, still, not found, search for the directory under user’s home directory

ls

Shows the contents of a directory/existence of specific files with their attributes

Syntax : ls [ flag] [string for filename]


ls Shows the name of all the files and directories under the current directory, excluding those starting with .

ls –a Lists all files including those starting with ‘.’

ls –x Multi columnar output

ls –R Shows all files and recursive listing of all files in sub-directories

ls –l Long listing showing seven attributes of a file

ls –F Marks executables with ‘*’ and directories with ‘/’


ls –t Sorts files by modification time – the file modified most recently comes at the top

ls –ut Sorts files by access time

ls –r Sorts file in reverse order

ls –ltr Shows long listing of files with their attributes, sorted in reverse order by access time(most recently edited file comes last in the list)

ls –i Shows inode number of a file

ls *.ksh Shows the name of all files with ‘.ksh’ at the end of their name

ls [aeiou]* Shows the files with name starting with vowels

ls d*.sh Lists all files starting with ‘d’ and ending with ‘.sh’ in their name

ls d?l* Lists all files with first letter as ‘d’ and third letter as ‘l’

ls [!aeiou]* Shows the files with names not starting with vowels

catCreates , shows, concatenates,copies files


cat >file1 Creates file file1 where a user enters text and presses <Ctrl-D> to end text editing

cat >>file1 Append lines to existing content of file : file1 and is ended when <Ctrl-D> is pressed

cat file1 Shows the contents of the file: file1

cat file1>file2

Copies the contents of file : file1 into new or existing file : file2

cat file1 file2 > file3

Concatenates the content of file1 and file2 and places it into new or existing file file3

cat file1 >>file2

Appends the contents of file1 after the last line of file2. If file2 does not exist, new file is created


cat –n file1 Displays the contents of file : file1 with line number

cat –b file1 Displays the contents of file : file1 with line number for all lines excepting the blank lines

cat –e file1 Prints ‘$’ at the end of each line

cat –r file1 Replaces multiple consecutive empty lines with one empty line

cat –t file1 Prints tab character as ‘^I’ and form feed character as ‘^L’

cat –s file1 file2 >file3 Suppresses error and does the job. If file file2 does not exist, the command will copy the contents of file1 into file3

Examples

Examples…continued

VIFirst Unix Full screen Editor

Haters say that it is the worst thing ever happened in the Unix world.

Lovers are totally biased towards it and would go any length supporting its cause.

First full screen editor, developed by William(Bill) Joy, a graduate student from University of California, Berkley.

Divides Unix users into two camps:

•Those who hate vi

•Those who love vi

Modes of Operation

First session with vi

Append mode

Command mode

Ex mode

Environmental variables

.exrc & EXINIT variable

view

Mode of Operation

Command mode : Default mode when a file is opened using vi. All the keys pressed by the user are interpreted as user commands

Append Mode : Permits insertion of new text, editing existing texts.

Ex mode : Permits commands at the command line(last line of the screen)

Command Mode

Append Mode

Ex Mode

R,R,I,I,c,C,o,O,s,S,a,A

Esc:Enter

First Session with vi

Step 1 : Create a new file by typing the following command from the OS Prompt : vi newfile

•vi clears the screen and display a window.

•The ‘_’ on the top line indicates that the cursor is waiting for commands

•Every other line starts with ‘~’, symbol for empty line.

Step 2 : Press ‘i’ to enter into Append mode. Add text to the file

Step 3 : Press <Esc> key to return to command mode

Step 3 : Press ‘:’. The cursor takes to the ‘ex’ mode at the command line. Enter ‘wq’ and press enter.

Append Mode

Cmd Significance

i Appends text from the left of the current cursor position

I Appends text at the start of the current line.

a Appends text from the right of the current cursor position

A Appends text at the end of the current line.

o Opens a line immediately below the current line in input mode

O Opens a line immediately before the current line in input mode

Inserting Texts

Cmd Significance

<n>r Replaces <n> characters from current cursor posn. with inserted text

R Replaces text from cursor to right

<n>s Replaces <n> characters from cursor with entered text

<n>S

Replaces <n> lines from the current cursor line with entered text

c0 Changes from cursor to beginning of line with the text entered

c$ Changes from cursor to end of line with the text entered

C Change from current cursor posn. to end of line with the text entered

<n>cw

Changes <n> words from the current cursor position with text entered

<n>cc

Replaces <n> lines from the current cursor line with entered text

cG Changes from current cursor position to end of the file with entered text.

Changing Texts

Command Mode

Cmd Significance

ZZ Saves the work done in the file and quits editing by vi editor

Saving work in a file and quit

Deleting texts/lines

Cmd Significance

<n>x Deletes <n> characters from current cursor position

<n>dd or <n>D

Deletes <n> lines counting from current cursor line to below

d$ Deletes from current cursor position to end of line

dG Deletes from current cursot position to end of file

d<n>G Deletes from current line to line no <n>

df<char> Deletes from current cursor position to first occurrence of character <char>

d/<pattern> Deletes from cursor upto the first occurrence of string <pattern> in forward direction

d?<pattern> Deletes from cursor upto the first occurrence of string <pattern> in backward direction

Moving/Copying Texts

Cmd Significance

<n>yy or <n>Y

Yank <n> lines starting from current line onwards into undo buffer

<n>yw Yank <n> words starting from current cursor position onwards into undo buffer

y$ Yank from current cursor position to end of the line in undo buffer

yG Yank from current cursor position to end of the file in undo buffer

“a<n>yy Yank <n> lines starting from current line onwards into buffer named a

p Paste the contents of undo buffer( as a result of deleting or yanking) after the cursor position

P Paste the contents of undo buffer( as a result of deleting or yanking) before the cursor position

“ap Paste the contents of buffer a after the cursor position

“bd Delete text into named buffer b

Navigation in same line

Cmd Significance

<n>h Moves cursor left to nth previous character w.r.t. the current cursor position

<n>l Moves cursor right to nth next character w.r.t. the current cursor position

<n>b Moves cursor left to start of nth previous word w.r.t the current cursor position. Punctuation marks are taken into account.

<n>w Moves cursor right to start of nth next word w.r.t the current cursor position. Punctuation marks are taken into account.

<n>e Moves cursor right to end of nth next word w.r.t the current cursor position. Punctuation marks are taken into account.

f<ch> Move the character to the next character <ch> on same line

F<ch> Move the character to the prv. character <ch> on same line

t<ch> Move the character to one column before the next character <ch> on same line

T<ch> Move the character to one column after the next character <ch> on same line

; Repeats search in the same direction along which the prv. Search was made using f/t

Cmd Significance

, Repeats search in the opposite direction along which the prv. Search was made using f/t

<n>| Moves the cursor to specified column <n>

0 or ( Moves to 1st character of the current line

$ or ) Moves to last character of current line

^ Moves to 1st non-space character of the line

<n>B Moves cursor left to start of nth previous word w.r.t the current cursor position.Punctuation marks are ignored

<n>W Moves cursor right to start of nth next word w.r.t the current cursor position. Punctuation marks are ignored

<n>E Moves cursor right to end of nth next word w.r.t the current cursor position. Punctuation marks are ignored

Navigation across lines

Cmd Significance

<n>j or <n>^n

Move the cursor down to the <n>th next line in the same column

<n>k or <n>^p

Move the cursor up to the <n>th prv line in the same column

H Moves the cursor to the top line of the screen

L Moves the cursor to the last line of the screen

M Moves the cursor to the middle line of the screen

<n>G Moves to line number <n>

+ Moves the cursor to next line’s first non-blank character

- Moves the cursor to previous line’s first non-blank character

Redraw screen

Cmd Significance

z- Makes the current line the last line of the screenand redraws the screen

z+ Makes the current line the first line of the screenand redraws the screen

z. Makes the current line the middle line of the screenand redraws the screen

Ctrl-l Redraws the screen

/pattern/z- Find the next occurrence of <pattern> and make that last line of the screen

Scrolling across pages

Cmd Significance

<n>^f Move forward by <n> screens

<n>^b Move backward by <n> screens

<n>^d Move forward by <n> number of half screens

<n>^u Move backward by <n> number of half screens

<n>^e Scroll window down by <n> lines

<n>^y Scroll window up by <n> lines

Pattern searching

Cmd Significance

/pattern Searches for specified <pattern> forward. IF end of file is reached, search wraps around.

?pattern Searches for specified <pattern> backward.

n Repeat the last search in the same direction as was specified in the last search

N Repeat the last search in the opposite direction as was specified in the last search

/pattern/+<n> Positions the cursor <n> number of lines after the line where the specified <pattern> is found

/pattern/-<n> Positions the cursor <n> number of lines before the line where the specified <pattern> is found

% Find the matching braces or parenthesis

Joining lines

Cmd Significance

<n>J Joins current line and <n> lines below it together to form a single line

Undo changes

Cmd Significance

u Undo last change

U Undo all the changes in the current line

Marking text

Cmd Significance

m<char> Marks position of the file with mark <char>

‘<char> Moves to portion of the file marked with <char>

“ Toggle to most recently marked location

Restoring previously deleted lineCmd Significance

“<n>p Paste the content of <n>th last delete ( n<=9)

“1pu.u.u… Till the last change is found

Filtering texts

Cmd Significance

!<n>G sort Sort from current line to line no. <n>

!<n>G tr ‘[a-z]’ ‘[A-Z]’ Translates all the characters from current line to line <n> to uppercase

!! tr ‘[a-z]’ ‘[A-Z]’ Translates all the characters of current line to uppercase

Cmd Significance

<n>i<ch> Inserts <ch> character <n> number of times in input mode at a stretch

Repeat factor

Miscellaneous in command modeCmd Significance

~ Change the character under cursor from lowercase to uppercase and vice versa

. Repeat the last change

<n>. Repeat the last action ‘n’ times

<< Shift current line to shift width character left

>> Shift current line to shift width character right

Options available with vi commandCmd Significance

vi –r <filename> Recover the file <filename> as much as possible after system crash and open it

vi –R <filename> Open the file <filename> in read-only mode

vi +<n> <filename>

Opens the file <filename> with cursor positioned in line number <n>

vi + <filename> Opens the file <filename> with cursor at the last line

vi –w<n> <filename>

Opens file <filename> in vi mode with window size of <n> number of lines

vi +/<pattern> <filename>

Opens file <filename> in vi editor and places the cursor at first occurrence of pattern <pattern>

vi –x <filename> Opens encrypted file <filename> in vi mode and asks for the password before opening that

Ex ModeSaving work in a file

Cmd Significance

:w Save the changes made to the file

:w <filename> Same as “Save As..” in windows. Saves the contents to the specified file <filename> . If it does not exist previously, a new file is created

:w! <filename> Save the changes to file <filename>, if the file already exists

:w >> <file1> Append the contents of the opened file after the last line of the file <file1>. File <file1> should exist previously

:<n1>,<n2>w <newfile>

Copies the contents of lines <n1> to <n2> into a new file <newfile>

:<n1>,<n2>w! <newfile>

Moves the contents of lines <n1> to <n2> into an existing file <newfile> , overwriting its previous contents

Cmd Significance

:.,.+<n>w <newfile>

Appends from current line to <n> number of lines below it into file <newfile>

:.,.+<n>w >> <nextfile>

Appends from current line to <n> number of lines below it after the last line of the file <nextfile>

:q Quits the file editing in vi, provided no unsaved change remains

:q! Quits vi neglecting all the unsaved changes made to the file

:wq or :x Save the unsaved changes in the opened file and quit vi editor

Temporary exit to shell

Cmd Significance

:sh Temporarily allows the user to come out of the vi file and use the shell. After the job of the user is done and command : exit is triggered from OS prompt, control returns to vi editor again

Navigating to desired line

Cmd Significance

:<n> Custor moves to line number <n>

Search and replace texts in ex mode

Syntax :- :<line address>s/<old pattern>/<new pattern>/g

Line address Significance

% All lines where matching pattern is found

. Current line

<n1>,<n2> Refers from line <n1> to <n2>

$ Last line

1,$ First to last line

.,.+<n> From current cursor line to <n> number of lines downwards

Examples of Search and Replace

Example Significance

:%s/ex/vi/c Substitutes 1st occurrence of string ‘ex’ with ‘vi’ by showing them and asking for confirmation. When each string will be shown with pause in cursor, press ‘y’ for substitution

:%s /<amaze \ >\/delight/g

Replaces ‘amaze’ , where available as a full word, with ‘delight’. Note, any word like ‘amazed’ will not be replaced

:g/subhendu/s/majumdar/mazumder/g

Replaces every occurrence of string ‘majumdar’ with ‘mazumder’ on all lines containing the pattern ‘subhendu’

:g/.\ {9\ }9/s/0/*/g Replaces ‘0’ with ‘*’ at all lines having ‘9’ after 9th position

:g/^$/d Delete all blank lines

:g!/complete/s/$/To be done/

Append the string ‘To be done’ at the end of all lines not containing the string ‘complete’

:g/vi/s/^/editor/ Append the string ‘editor’ at the first of all lines containing the string ‘vi’

:%s/$/ : see my note/g

Appends the string ‘: see my note’ at the end of all lines

:g/^….$/d Deletes all lines containing 4 letters

:g/^..o/d Deletes all lines with ‘o’ as 3rd character

:%s/…$//g Delete the last three character of every line

Reading below the current line in the vi editor

Command Significance

:r <nextfile> Reads the contents of the file <nextfile> below current line

:r! <command> Places the output of the command <command> below the current line

Editing another file


:e <nextfile> Stops editing the current file; leaves the current file and starts editing file <nextfile>; provided there are no unsaved changes in the current file

:e! <nextfile> Edits file <nextfile> abandoning all the changes done to the current file

:e! Loads last saved version of current file

:n Edits next file mentioned in the vi queue

:rew Edits first file in the command line

:e +<n> <nextfile> Edit starts at line <n> of file <nextfile>

Abbreviating texts


:ab <short_string> <long_string>

When the user writes the string <short_string> in input mode, the <long_string> is written

Mapping


:map g :w^M Pressing ‘g’, one wants to save the file(:w is for saving, and ^M is for pressing Enter key. While writing it in the command line, write ^V^M)

:map z i^M^[ When you position your cursor to any character in a line and press ‘z’ , the line will be broken from that point and two lines will be formed. The control will remain in command mode(^[ represents <Escape> key)

:map z :w^M:!%^M

Pressing ‘z’ in command mode saves the file and executes it in one shot

To unmap a key, write at the command line :unmap <key>

Miscellaneous


:! <command>

Executes the command <command> remaining in vi editor

:f Shows the name of the current file and line

^g Same as :f

Setting environmental variables for vi

Works in ex mode.

To set an environment variable to customize vi, the following syntax needs to be followed:-

:set <env.variable> [= <value>]

Environment variables

Significance

autoindent(ai)

Newly inserted lines of text are indented to the same distance from left margin as the preceding line. Opposite of this option is noautoindent(noai)

autowrite(aw)

Automatically saves the unsaved changes in a file before opening the next file with :n or using a shell command with :! <command>. The opposite to this option is noautowrite(noaw)

errorbells(eb)

Sounds the bell when error occurs. Opposite is ‘noeb’

exrc(ex) Allows an .exrc file in the current directory to override the .exrc file in user’s home directory. Opposite is ‘noex’

list Displays special characters in the screen: tabs are shown as ^I, end of line are marked with ‘$’. Opposite is ‘nolist’

mesg System messages allowed when vi is running. Opposite is ‘nomesg’

number( nu ) Displays line numbers. Opposite is ‘nonu’

Environment variables Significance

report=<val> When any operation affects more lines than this settings, message is displayed

scroll=<val> Number of screen lines to scroll

shiftwidth(sw)=<val> Number of spaces to be used for backtabs/<</>>

showmatch(sm) Shows match for ) or } . Opposite is ‘nosm’

showmode Indicates type of mode

tabstop=<val> No. of spaces the tab character moves over

ignorecase(ic) Ignores case when searches patterns. Opposite is ‘noic’

wrapmargin(wrm)=<val>

When set to a value >0 , carriage returns are inserted automatically when the cursor gets to within that number of spaces from the right edge of the screen

.exrc file and EXINIT system variable

One can store all the values for environment variables, all the key mappings and all the abbreviations in a file ‘.exrc’ under the home directory for the user.VI looks for this file on startup and executes the instructions as ex mode commands.

Besides, there is also a system variable , ‘EXINIT’ which can also be used to save the settings.

EXINIT=“set report=5 ignorecase ai”; export EXINIT

view

Description :Shows the file <filename> in vi mode. File remains read-only. No changes done to the file cannot be saved.

Syntax : view <filename>

more

Filter for examining continuous text, one screenful at a time. It pauses after each screenful,printing the filename at the bottom of the screen.

•To display one or more next lines, press <Return>

•To display another screenful, press <Space>

Syntax : more [ -n <number> ] [ -<number>] [ -d] [-i] [-e] [ -c] [-f] [-s] [ +/pattern] [filename]

Command line option

Significance

-n <number>

Sets the no of lines in the display window to that <number>.

-<n> Same as above

-d Prompts user with message :- ‘Press space to continue, q to quit, h for help’ at the end of each screenful of display

-i Performs case independent pattern matching

-c Draw each page by starting at the top of screen, and erase each line before drawing on it

-s Squeeze multiple blank lines from the file , showing only one blank line

+/pattern Start listing such that the current position isset to two lines above the line matching the regular expression pattern.

f Scrolls forward one screen

b Scrolls backward one screen

q Quit

/pattern Searches pattern <pattern> forward

?pattern Searches pattern <pattern> backward

v Opens vi mode for the file viewed

n or N Repeats last search

<n>j Scrolls forward by <n> number of lines

<n>G Moves to line number <n>

G Goes to last line of the file

. Repeats prv. command

:n Moves to next file specified in the command line

Internal commands in ‘more’ mode

:p Moves to previous file specified in the command line

i<space>

Scrolls forward by i number of screens

i<return>

Scrolls forward by i number of lines

^d/ 

Scrolls forward i lines, with a default of ½ of the screen size

^u /u

Scrolls backward i lines, with a default of ½ of the screen size

<n>k/ <n>^y

Scrolls backward by <n> number of lines

<n>z Displays <n> more lines and sets the window size to <n>

<n>f / <n>^f

Moves forward by <n> lines

<n>b / <n>^b

Moves backward by <n> lines

^g Writes the name of the file currently being examined, the no. relative to the total number of files to be viewed, the current line no,current byte no, total no of bytes to write and what % of the file precedes the current position

h Display a descriptive list of all the more commands

!<command>

Invoke a shell with a command <command>

:e <newfile> Leaves the current file and starts viewing a new file <newfile>

:n Examines the th next file specified in the command line

r / ^l Refreshes the screen

. Repeats the prv command

m<char> Marks the current position with letter <char>

‘<char> Returns to the position previously marked by the specified letter <ch>

pgDisplays the content of a file one page at a time

Syntax :pg < flag and string> <filename(s)

Flag Significance

-c Clear the screen at the end of each page of display and start the display at the top of the screen

-e Continues to the next file after the end of one file, if more than one files are specified at the command line

-f Truncate lines longer than the width of the screen display

-p <string>

Display the <string> at the pg command prompt.Default is ‘:’ . If string is %d the pageno is displayed atr the prompt

-s Highlights all messages and prompts issued by the pg command

+<n> Start the display from line no <n> of the file specified

Flag Significance

-<n> Sets the size of the display screen to <n> number of lines

+/pattern/ Search for pattern <pattern> in the file and start the display at that line

Keystrokes in pg mode

Option Significance

-<n> Go backward by <n> number of pages

+<n> Go forward by <n> number of pages

l Go forward by one line

<n>l Start the display in the file at line specified by <n>

+<n>l Go forward by <n> lines

-<n>l Go backward by <n> lines

d Go forward by ½ screen

-d Go backward by ½ screen

^l Redraws the screen

Option Significance

$ Move to last page of the file

<n>/<pattern>

Searches forward for the pattern <pattern> in the file from beginning of the next page. If a number <n> is specified, pg searches for the specified occurrence number <n> of the <pattern>

<n>p Starts showing the <n>th previous file.

n Start showing the next file

q Quits the pg command

cpCopies one file to another file/one directory to another directory/files into directories

Syntax :cp [-flag] [ old files/directories] [ new files/directories]

Flag Significance

-i Interactive copying : prompt appears for user accent before copying

-f Force copying

-p Preserve permissions. Preserves modification time,access time,file mode, user id, user group etc

-r Recursive copying


cp file1 file2 Copies the contents of file1 into new or existing file file2

cp file1 file2 file3 dir1

Copies files: file1, file2 and file3 into directory dir1

cp –R dir1 dir2 Copies directory dir1 into a new directory dir2(if dir2 does not exist before) / copies the directory dir1 as a sub-directory under dir2(if directory dir2 exists before)

cp –R dir1 dir2 dir3

If dir3 exists, two sub-directories under it are created : dir1 and dir2.IF dir2 directory does not exist, then a new directory dir2 is created with the contents same as dir1 and one additional sub-directory dir2

Examples

mv

Moves/renames:-

A file to new or existing file

One/more files to existing directory

One or more directories to a new or existing directory.

If the access permission of the destination directory or existing destination file forbids writing, mv command asks for overwriting the file

Syntax :mv [-flag] [ old files/directories] [ new files/directories]

Flag Significance

-f Performs move operation without prompting for permission

-i Interactive moving

rm

Removes the entries for one/more files from a directory.

Destroys the file whose last link is deleted

Removal of a file requires write permission for that file

Removal of a file from a directory required write and execute permission in the directory

Syntax :rm [-f|-i] [ -r|-R] [files|directories]

Flag Significance

-f Forceful deletion

-i Interactive deletion

-r or –R Recursively delete the entire contents of the cirectory before removing the directory itself


mv *.txt ../testdir

Moves all the files with .txt extension under the directory testdir residing under the parent directory of the current directory

rm *.temp Remove all files with extension : .tmp under the current directory

rm –r backupdir

Remove the directory backupdir with all its contents

Examples

wc

Counts the number of words/bytes/characters/lines in a file

Syntax :wc [-c|-l|-w] <filename(s)>

Flag Significance

-w Counts the total no. of words

-l Counts the total no. of lines

-c Counts the total no. of characters

$ wc –c file1

32 file1

$ wc –l file1

2 file1

$ wc –w file1

8 file1

Examples

file

Determines the file type of a file or list of files.

Performs a series of tests on each file in an attempt to classify it. If file appears to be an ASCII file, file examines the first 512bytes and tries to guess its language.

File /etc/magic is ued to identify files that have some sort of magic number, that is, any file containing a numeric or string constant that indicates its type. Commentary at the beginning of /etc/magic explains the format.

Syntax : file [-f ffile] [-h] file ...

chmod

A file can have three type of permissions:-

Read : Authorized user can read the contents of the file.

Write : Along with read permission, it allows the allowed user to modify its contents.

Execute : If the file is an executable, any allowed user can execute it

Syntax : chmod [file permission] <files|directories>

A file can be accessed by:-

User : Person creating the file. He grants all the authorizations to the file

Group : Group user for the file

Others: All other users not belonging to the group or are not the creator.

Value Significance

4 Read permission

2 Write permission

1 Execute permission

6(=4+2) Read and write permission

7(=4+2+1) Read,write and execute permission

5(=4+1) Read and execute permission

Numeric representation of permissions

Example of granting numeric authorization

Syntax: chmod <val1><val2><val3> <filename(s)|directory name(s)>

val1 is for users

val2 is for group

val3 is for others

Any permission on a directory percolate down to the files and sub-directories under it.


chmod 744 file1 Grant all permissions to : User, and read permission to group and others

chmod 776 Grant all permission to user and group, read and write permission to others

chmod 777 file1 Grant all permission to all

Value Significance

r Read permission

w Write permission

x Execute permission

Alphabetic representation of permissions

Value Significance

u User creating the file

g Other users in the same group of the creator

o Any other users

a All(creator, other users in the same group, and other users)

Example of granting alphabetic authorization


chmod u=rwx,go=r file1 Assign read,write and execute permission to user, but only read permission to group and others

chmod o-x file1 Revoke execute permission from others

chmod a+x file Assign execute permission to everybody

chmod –R ug+r,o-r,a+x /home/ems2000

Traverse the directory subtree under directory /home/ems2000 making all regular files readable by user and group only, revoke read permission from others and grant execute permission to all

Chown and chgrp

chown changes owner of the file and chgrp changes group of each file specified.

To change user or groyp, one must own the file and have the CHOWN privilege granted to him by the system administrator

Syntax : chown [-h] [-R] [owner] [file…]

chgrp [-h] [-R] [group] [file…]


-h Change owner/group of a symbolic link

-R Recursively change the owner and group of all the files and sub-directories under the directory named


chown ems2000 auto.profile

User ‘ems2000’ becomes the owner of the file auto.profile

chgrp ems2000 auto.login

User ‘ems2000’ becomes the group user to access the file auto.login

chown –R ems2000: users shell_scripts

The command searches the directory : shell_scripts and changes each file in that directory to owner : ems2000 and group : users

touchUpdates access time/modification time of file(s)

Syntax : touch [-a|m|c] [ -r <ref-file> | -t <time> ] <filename(s)>

Flag Significance

-a Change the access time of the file to the time specified/ if no time is specified, use the current time

-m Change the modification time of the file to the time specified/ if no time is specified, use the current time.

-r ref-file Use the corresponding time of file ref-file to change the modification/access time of the file

-t <time> Use the specified time <time> instead of current time. The time format is : <YYYY><MM><DD><hh><mm>.<ss>


touch –a wot.ksh Changes the access time of wot.ksh with the current date and time(Changes can be perceived if the command : ls –ut is fired in the directory containing the file : the file wot.ksh will come at the top of the list)

touch –m wot.ksh Changes the modification time of wot.ksh with the current date and time(Changes can be perceived if the command : ls –lt is fired in the directory containing the file : the file wot.ksh will come at the top of the list)

touch -m -t 201012122300 new.del

Changes the modification time of file : new.del to Dec 12,2010 time : 23:00Seeing the attributes of the file new.del will reveal the information:--rw-rw-r-- 1 ems2000 dba 10 Dec 12 2010 new.del

touch -m -r new.del new1

Change the modification time of file : new1 and make it same as the modification time of the file : new.del

ln

Description Link files and directories


ln [-f|-i] file1 newfile Links file : file1 to a new or existing file : file1

ln [-f|-i] file1 file2 testdir

Creates link for files new1 and new2 under the directory : testdir with two new files/existing files new1 and new2

ln [-f|-i] dir1 dir2 testdir

Creates link for directories dir1 and dir2 under the directory : testdir with two new /existing directories : dir1 and dir2

Flag Significance

-f Force esisting files or directories to be removed to allow the link

-i Interactive linking

mkdirCreates directories

Flag Significance

-p Intermediate directories are created as necessary.Otherwise, the full path prefix of dirname must already exist. mkdir requires write permission in the parent directory.

-m <mode>

Mode of permission for the directory and all the files under it

Syntax : mkdir [-p] [-m <mode>] <directory name(s)>


Mkdir –m755 testdir Directory : testdir is created with permission = 755

mkdir –p testdir/subhendu

If testdir is not created, it is first created. Then, a directory : subhendu is created under the directory testdir

rmdir

Removes the directory entry for each empty directory referred

Flag Significance

-f Forcefully remove a directory, even though it is not empty

-i Interactive removal of empty directory

-p Path removal.If, after, removing a directory with more than one pathname component, the parent directory of that directory is empty, rmdir removes the parent directory also. This continues till rmdir encounters non-empty parent directory

Syntax : rmdir [-f|-I|-p] <directory>

cmp

Description Compares two files

Flag Significance

-l Print the byte number(decimal) and differing byte(octal) for each difference

-s Prints nothing; return exit codes only

Syntax : cmp [-l|-s] <file1> <file2>

Return codes Significance

0 Files identical

1 Files not identical

2 Inaccessible/missing arguments

comm

Description Select or reject lines common to both files.

Syntax : comm [-[123]] <file1> <file2>

Case Significance

comm filenew fileold Produces a three columnar output:-1st column displays lines only in file: filenew2nd column displays lines only in file:fileold3rd column displays lines common to both files

comm –1 filenew fileold

Suppress display of 1st column


Suppress display of 2nd column


Suppress display of 3rd column


Shows only lines common to both files

sdiff

Description Side by side file difference program.

Syntax : sdiff [-l|-s] <file1> <file2>

$cat file1

Suvendu

Subhasish

Dibyendu

Diptoman

Akash

$cat file2

Subhendu

Subhasish

Arka

Dibyendu

diptoman

$sdiff file1 file2

Suvendu ! Subhendu

Subhasish Subhasish

> Arka

Dibyendu Dibyendu

Diptoman Diptoman

Akash <

Flag Significance

-l Only print on the left hand side where columns are identical

-s Do not print identical lines

dircmp

Description Examines two directories and generates various tabulated info about the contents of the directories.

Syntax : dircmp [ -d|-s|-wn] <directory1> <directory2>

Flag Significance

-d Also compares the contents of the files with same name in both directories and output a list telling what must be done to bring them into agreement

-s Suppresses messages about identical files

$ >ls -ltr dir1 dir2dir1:total 10-rw-rw-r-- 1 ems2000 dba 13 Jul 20 07:23 abc-rw-rw-r-- 1 ems2000 dba 26 Jul 20 07:23 def-rw-rw-r-- 1 ems2000 dba 15 Jul 20 07:30 subhendu-rw-rw-r-- 1 ems2000 dba 28 Aug 13 00:30 subha-rw-rw-r-- 1 ems2000 dba 5 Aug 13 00:39 samefile1

dir2:total 8-rw-rw-r-- 1 ems2000 dba 26 Jul 20 07:24 def-rw-rw-r-- 1 ems2000 dba 12 Aug 13 00:32 nsf-rw-rw-r-- 1 ems2000 dba 30 Aug 13 00:32 abc-rw-rw-r-- 1 ems2000 dba 5 Aug 13 00:39 samefile1

$ dircmp dir1 dir2Aug 13 00:42 2003 dir1 only and dir2 only Page 1

./subha ./nsf

./subhendu

Aug 13 00:42 2003 Comparison of dir1 dir2 Page 1

directory .different ./abcsame ./defsame ./samefile1

lp

Description Prints one/more files to a specified printer

Syntax : lp -d<printer> [-m] [-n<copies>] [-t<title>] <file(s)

Flag Significance

-d<printer> Specify the printer <printer> where the print request is to be directed

-m Notify the requesting user at successful completion of the print request by mail

-n<copies> Specifies number of copies to print

-t<title> Print the specified title <title> on the banner page

lpstat

Description Shows current status of the print request

Syntax : lpstat [ -p<printer>] [-t] [-u<username>] [-v <printername>]

Flag Significance

-p<printer> Shows all printing jobs queued at printer <printer>

-t Displays detailed status information about the print from all users

-u<username> Displays the status of print requests triggered by user <username>

-v<printername>

Displays a list for the specified printername

cancel

Description

Used to cancel a specific print request or cancel all queued requests to a specific printer queue. Any ordinary user can cancel only those jobs triggered by his own userid

Example:-

$ cancel 734 # cancels print job with id 734

$ cancel lipi # cancels all queued jobs in printer lipi

prDescription Prepares a file for printing by adding suitable headers, footers and formatted texts.


pr file1 Adds 5 lines of margin at the top and 5 lines at the bottom of file : file1. Header shows date and time of last modification of the file along with filename and pageno

pr –l 72 file1 Changes default page size from 66 to 72

pr +10 file Show file file1 from 10th page

pr –3 file1 Prints a file in three columns

pr –t file1 Suppresses page headers and footers for file : file1

pr –f file1 Form feed is used for a new page instead of a sequence of line feed characters

pr –w 80 file1

Set the width of each page to 80 instead of a default of 72


pr –P file1 Pauses after each page is displayed in the terminal

pr –o5 file1 Indent each line by 5 columns

pr –d file1 Generate output with double specing

pr –h “List of files” file1

Print “List of files” instead of the filename as header on each page.

Compressing filesCompress

Compress <filename(s)>

Reduces the size of the named files using adaptive Lempel-zev coding. If reduction is possible, each file is replaced by a new file with same name suffixed sith ‘.Z’. Original ownership, modes, access and modification times are preserved.To uncompress a file, issue the command : uncompress <filename>.Z . To view a compressed file, issue : zcat <filename>.Z

Gzip

gzip <filename(s)>

Reduces the size of the named files . If reduction is possible, each file is replaced by a new file with same name suffixed sith ‘.gz’. Original ownership, modes, access and modification times are preserved. To uncompress a file, issue the command : gunzip <filename>.gz . To view a compressed file, issue : gzcat <filename>.Z

Filters

FiltersBy end of this section , you will be able know

‘head’First few lines of a file

‘tail’Last few lines of a file

‘cut’ View columns/fields

‘paste’join lines of files

‘split’Break long lines

‘fmt’Formats texts

‘fold’ Folds long lines

‘Sort’ Sorts file contents

‘tr’Translates file contents

‘nl’Shows line numbers

‘Spell’Catch spelling mistakes

‘find’ Find files

‘grep’Search Pattern in a file

‘Sed’Display specific lines

‘awk’Reporting Tool

head

Gives first few lines of a file


head –5 newfile Shows first 5 lines of file : newfile

head –n 5 newfile Same as above

head -l -n 5 newfile Same as above

head -c –n 14 newfile Shows first 14 characters of newfile

Syntax :head [-<n>] [-c] [-l] [ -n ] [file(s)]

tailtailGives last few lines/characters of a fileGives last few lines/characters of a file

Flag Significance

-n <no>

Shows last <no> lines of the file if <no> is specified with no sign or –ve sign. If specified with a ‘+’ sign, it shows from <no>th line onwards till end of the file

-c <no>

Shows last <no> bytes of the file if <no> is specified with no sign or –ve sign. If specified with a ‘+’ sign, it shows from <no>th byte onwards till end of the file

-f Works in constant pilot mode . Used for variable sized files those are growing constantly.

Syntax : tail [-f] [-c number] [file] tail [-f] [-n number] [file]


tail –5 newfile Shows last 5 lines of newfile

tail –n 5 newfile Same as above

tail +5 newfile Shows from 5th line to end of the file

tail –n +5 newfile

Same as above

tail -c 5 newfile Shows last 5 bytes of a file

tail –c +5 newfile

Shows from 5th byte till end of the file

tail –fn 3 newfile

Constantly displays the last three lines of file : newfile and leave the tail in “follow” mode. The ‘$’ prompt does not return even after the work is over. One have to abort the process to exit to the shell.

cutcut

Extracts selected fields/characters on each line of a file

Syntax : cut [-c list] [file] cut [-f list] [-d <char>] [file]


cut –c 6-22,24-32 file1

Extracts texts from column 6 to 22 and 24 to 32 and displays in the terminal in two columns

cut –c –22 newfile Extracts text from column 1 to 22 from the file : newfile and displays it in the terminal

cut –c –6, 22,24- file1 Extracts texts from column 1 to 6, column 22 and from column 24 to end of the line and display it in the terminal

cut –d”|” –f 2,3 newfile

Extracts field contents of 2nd and 3rd fields and display in the terminal. Use “|” as a field separator while extracting fields

cut –d”|” –f 1-5 newfile

Using ‘|’ as field separator, it extracts field contents of 1st to 5th fields and display in the terminal.

paste

Merges same lines of several files or subsequent lines of same file

Syntax : paste file1 file2 ...

paste -d list file1 file2 ...

paste -s [-d list] file1 file2 ...


paste file1 file2 Display in the screen the join of line 1 of file1 with line 1 of file 2 with a space between them, line 2 of file1 with line 2 of file2 with a space delimiter and so on.

paste –d”|” file1 file2 Same as above with ‘|’ as delimiter between the fields of both files on each line.

Paste –s file1 Joins all the lines of file1 into a single line and show it in the terminal


paste -s file1 file2 Displays in the screen a single line which is :-Join of all the lines in file1 + Join of all the lines in file2

paste –s -d” t\n\” file1

Combines 1st and 2nd lines into a single line, 3rd and 4th into another line and so on.

split

Splits up a file into equiline smaller lines.

Large files are sometime difficult to edit with an editor. The split command breaks up a larger file into several equi line smaller files, each containing a default of 100 lines.

It creates a group of files xaa,xab…till xaz and then again from xba,xbb.. till xbz. Total 26x 26 = 676 files can be created in this way.

Syntax : split [-<no. of lines to be put in each file>] [<initials>]

Example:

File: newfile consists of 100 lines.

$ split –20 newfile

5 files nfa,nfb,nfc,nfd and nfe will be prepared each containing 20 lines from newfile

fmt

Simple text formatter that fills and join lines /split lines to produce output lines upto the number of characters specified in the –w option(default is 72). It also counts the spaces between words in a lineand also considers a space between joining of two lines.

The –s option split lines only. It does not join short lines to form longer ones.

Syntax : fmt [-s] [-w <width>] [file…]

$ >cat file1

Today, we have a meeting.

It will start at 6 pm.

Bye

$ >fmt -w10 file1Today, wehave ameeting.It willstart at 6pm. Bye

foldfold

Folds long linesSyntax : fold [-b|-s] [-w <width>] <file….>Syntax : fold [-b|-s] [-w <width>] <file….>

Flag Significance

-b Width in bytes for counting

-s Break the line on the last blank character found before the specified number of column position specified in the –w<width> option(default : 80)

$ cat file1Today, we have a meeting. It will start at 6 pm. Bye

$ fmt -s -w15 file1Today, we havea meeting. Itwill start at 6pm. Bye

sort

Sorts the contents of a file.

Starts with 1st character on each line and proceeds to the next character only when the prv. Character in the two lines are identical.

Can also work on fields. Default field separator is the space.This can also be changed.


sort file1 Shows the sorted contents of file1 as per character sorting

Sort –t “|” +2 file1 Sort the contents of file1 based on 3rd field

Sort –t “|” –r +1 file1 Reverse sort on 2nd field

Sort –c file1 Checks whether the file : file1 is already sorted or not. Value of system variable $? Is 0 if it is sorted

sort –t “|” +2 –c file1 Checks whether the file is already sorted on 3rd field or not


Sort –o sort.lst +3 file1

Sort the contents of file : file1 based on 4th field and store the output in a file : sort.lst

Sort –t “|” +3 –4 file1 Start sorting of file1 after 3rd field(+3) and stop sorting after 4th field(-4). Basically sort file : file1 on 4th field

Sort –u file1 Sorts the contents of the file : file1 and removes duplicate entries

Sort –t “|” +7.5 – 11.6 file1

Start sort after 5th(+7.5) column of 8th field (+7.5) and stop sort on 6th column(11.6) of 12th field(11.6)

Sort –n file1 Sort numerically file : file1. Used when the file contains numeric entries.

tr

By default, it translates each character in <expr1> found in the file to its mapped character in <expr2>

Syntax : tr [-c] [-d] <expr1> <expr2> < <file>


tr ‘ABC’ ‘abc’ < filenew Translates the uppercase letters A,B and C in file : filenew to lowercase

tr ‘[a-z]’ ‘[A-Z]’ <filenew

Translates all the lowercase letters in file : filenew to uppercase

tr –d ‘|’ < fileold Deletes all the ‘|’ characters from file : fileold

tr –s ‘A’ <fileold Compress multiple consecutive presence of character ‘A’ to one ‘A’

tr –cd ‘|/’ < fileold Delete all characters except ‘|’ and ‘/’.

nl

Shows line number at the left of each line in the file. Reserves six characters for the number

Syntax : nl [-w < number>] [ -s <delimiter_char>] <filename….>


-w <number> Specifies the width option for line number display to <number> characters

-s <delimiter_char> Used to specify the character used for delimiting between line no and line content

spellspell

Lists all spellings in the file (s) the program recognizes as mistakes.

Syntax : spell [-a|-b] <filename(s)>


-a Use American spelling system

-b Use British spelling system

find

Recursively examines all files in the <path_list>

Matches each file for one/more occurrence of <sel-criteria> depending on <mode>

Takes some action on the files selected

Syntax : find <path_list> <mode> <sel-criteria> <action>

Mode and Selection criteria

Significance

-name <filename> Selects file with name <filename> . Pattern matching is allowed

-user <username> Selects file owned by user specified by <username>

-type <f> Selects type of files specified by <f>

-group <grpname>

Selects file if owned by group <grpname>

-size +<x>[<c>] Selects file if the size exceeds <x> blocks(characters if <c> is also specified)

-atime +<x> Selects file if accessed before <x> days

-amin +<x> Selects file if accesses before <x> minutes

-atime -<x> Selects file if accessed within <x> days

-amin -<x> Selects file if accesses within <x> minutes

-mtime -<x> Selects files if modified within <x> days

-mmin -<x> Selects files if modified within <x> minutes

-newer <flname> Selects file if modified after file <flname>

Action Significance

-perm <permission_mode>

Finds the files with permission mode specified

-exec <command> {} \;

Executes command <command> after finding every file on the search

-ok <command> {} ; Executes command <command> after user confirmation

-print Shows selected files in display

grep

Searches a file for a pattern

Syntax : grep [-c|-n|-v|-l|-I|-e|-h|-x ] <pattern…> <filename(s)>

Flag Significance

-c Counts number of occurrences. Output shows only the file name and number of times the searched pattern found within it

-n Displays the line number containing the pattern, along with the lines

-v Displays all lines excepting those containing the pattern

-l Displays only file names containing the pattern

-i Ignores case while searching for the pattern

-e Extended search. More than one pattern is to be searched

-h Suppresses printing filename when printing multiple files where the pattern is found

-x Matches are recognized only when the entire line of the file searched matches the fixed string


grep while wot.ksh Searches for the string ‘while’ in the file : wot.ksh and if found, displays the lines containing the pattern

grep while *.ksh Searches for the string ‘while’ in all files with extension ‘.ksh’ and shows the file name and the line containing the pattern

grep “List of names” *.txt

Searches for the string ‘List of names’ in all .txt files.

grep –li while *.ksh Case-independent searching of the string ‘while’ takes place in all the .ksh files and the the name of the files where matched is displayed only

grep –e “Subhendu” –e “suvendu” name.lst

Searches for pattern ‘Subhendu’ and ‘Suvendu’ in the file : name.lst

grep 'M[ao]noroma' name.lst

Searches for the string ‘Manoroma’ or ‘Monoroma’ in the file : name.lst

grep “^A” name.lst Searches all lines starting with A

grep "Sinha$" name.lst Searches all lines ending with ‘Sinha’

egrepDescription Extended grep

Expression Significance

<ch>+ Matches one/more presence of character <ch>

<ch>? Matches zero / presence of character <ch>

<expr1>|<expr2> Matches <expr1> or <expr2>

(<exp1>|<exp2>|<exp3>

Matches expression <exp1><exp3> or <exp2><exp3>


egrep '(Mano|Mono)roma' name.lst

Matches string ‘Monoroma’ or ‘Manoroma’

egrep 'Aka+sh' name.lst Matches ‘Akash’ , ‘Akaash’,’Akaash’

egrep –f srchlist name.lst

File : srchlist contains all the strings to be searched in file : name.lst

sed

Stream text editor, used for:-

•Displaying specific lines from a file by line no/pattern matching

•Inserting or changing texts in a file

•Deleting lines from a file

Can be addressed in two ways:-

By line number

By specifying a pattern which occurs in a line

Syntax : sed <options> <address,action> <filename(s)>

Done by specifying line number , or a pair of them separated by comma to specify the lower and upper boundaries of selection.

Single word specifying action are:-

p Print

q Quit

$p Last line

Negative indicator ‘!’ can also be used to signify negation

Addressing by lines

Examples Significance

sed ‘3q’ name.lst Show first three lines of file : name.lst and quit

sed –n ‘3,5p’ name.lst Show 3rd to 5th line of file : name.lst

sed –n ’10,$p’ name.lst Show from line no. 10 to end of file

sed –n –e ‘1,2p’ –e ‘7,9p’ –e ‘$p’ name.lst

Show 1st to 2nd line, 7th to 9th line and last line of the file : name.lst


$ sed 'a\> subhendu> ' del>del1

Inserts the line ‘subhendu’ after every line in the file : del and save the display in file : del1

$ sed ‘i\> subhendu> ' del>del1

Inserts the line ‘subhendu’ before every line in the file : del and save the display in file : del1

$ sed ‘$a\> subhendu> ' del>del1

Inserts the line ‘subhendu’ at the end of file : del and saves under file : del1


sed –n ‘/getopts/p’ emp.ksh

Searches the string ‘getopts’ in file : emp.ksh and displays it

sed –n ‘/getopts/, /charstring/p’ emp.ksh

Searches for the string ‘getopts’ and ‘charstring’ in file : emp.ksh

sed –n ‘/[cC]hatterjee/p’ emp.lst

Searches for the string ‘Chatterjee’ or ‘chatterjee’ in file : emp.lst

sed -n '/Aka*sh/p' name.lst

Searches for any string with the pattern specified

sed –n ‘/^A.a.s/p’ emp.lst

Searches for any line containing ‘A’ as 1st character, ‘a’ as 3rd character and ‘s’ as 5th character

sed –n ‘/^$/!p’ emp.lst sed ‘/^$’/d’ emp.lst

Shows all but the blank lines from file emp.lst

sed –n ‘/^….$/p’ emp.lst

Shows all lines with 4 characters only

sed -n '/. {101, }/p' del Shows lines longer than 100 characters

sed -n '/. {9 }9/p' del Shows lines containing ‘9’ after 9 characters in a line

Context addressing


sed –n ‘/if/w iflist /while/w wlist’ emp.ksh

Searches file emp.ksh for pattern : if and writes matching lines into file : iflistSearches file emp.ksh for pattern : while and writes matching lines to file : wlist

sed –n –f instr.lst emp.ksh

Searches file : emp.ksh for the strings specified in the file instr.lst

Substitution

Examples Significance sed ‘s/exhausted/tired/’ list1 sed ‘s/exhausted/tired/p’ list1

Shows the content of the file : list1 with the string ‘exhausted’ replaced by ‘tired’ at all occurences

sed ‘1,5s/basic/prelim/’ list1 Shows from 1st to 5th line of file : list1 with the string ‘basic’ replaced by ‘prelim’ at all occurences

sed –n ‘/marketing/s/staff/member/p’ list1

Replaces ‘staff’ with ‘member’ in all the lines containing ‘marketing’

sed ‘s/student/ex &/’ name.lst Replaces ‘student’ with ‘ex student’

awk

•Reporting tool for Unix.

•Developed by Alfred V Aho, Peter J Weinberger, Brian W Kernighan

•Emerging as programming language.

•Produces facilities similar to SQL language.

Syntax : awk <options> ‘ <address> { <actions> }’ <input|file(s)>

‘AWK’ is a pattern matching and processing language . It can search file(s) searching for a pattern, and when found, performs specified action.

Works best with ascii files, preferably not to use with binary files.

Using awk from command line

$ awk ‘ { print $0 }’

Good morning

Good morning

Hello

Hello

^d

$

$ awk ‘ { print $1,$3 }’

Happy Birth day

Happy day

How are you?

How you?

^d

$

$ awk –F”|” { print $1,$2 }’ name.lst

0001 Subhendu Majumdar

0005 Raghab Das

0006 Sumit kumar Basu

0011 Tamal Sen Sharma

0010 Ratna Sengupta

0015 Raghab Dasgupta

0025 Rajib Banerjee

0012 Damini Sen

0007 Akash Nag

0009 Anindya Das

0019 Sougata Das

$ cat name.awk

{ print "-->" , $2 }

$ awk -F"|" -f name.awk name.lst

--> Subhendu Majumdar

--> Raghab Das

--> Sumit kumar Basu

--> Tamal Sen Sharma

--> Ratna Sengupta

--> Raghab Dasgupta

--> Rajib Banerjee

--> Damini Sen

--> Akash Nag

--> Anindya Das

--> Sougata Das

Taking awk instructions from a file

Variables Significance

ARGC Number of command line arguments

CONVFMT Conversion format for numbers

FIELDWIDTHS Whitespace separated string for the width of input fields. Provides a facility for fixed-length fields instead of using field separators.

FILENAME Name of current input file

NR Current record number

FS Input field separator

IGNORECASE 0(Case sensitive)1 ( Case insensitive)

NF Number of fields in the current record

FNR Current record number

Pre-defined variables in awk

Variables Significance

OFS Output field separator

ORS Output record separator(default is newline)

RS Input record separator( default is new line)

FILENAME Name of current input file

NR Current record number

FS Input field separator

IGNORECASE 0(Case sensitive)1 ( Case insensitive)

ENVIRON Unix environment variables

ERRNO Unix system error message number

Pre-defined variables in awk…contd

$ cat name1.lst

0001 | Sourav Ghar | Team Lead | Avaya | 25000.00|

Example of using pre-defined variables

$ cat name2.lst

0001 | Sourav Dutta | Project M | Avaya | 25000.00|

0002 | Ratna Sengupta | Programmer| GFS $ cat name.awk

{

FS = "|" ;

OFS = " ** " ;

ORS = "\n-------------------------------------------\n";

print ARGC , ENVIRON["TERM"], CONVFMT , FILENAME ;

print NR , $2 , $3 , "Total fields" , NF ;

}

Example of using pre-defined variables…contd

$ awk -f name.awk name1.lst name2.lst

3 ** vt100 ** %.6g ** name1.lst

-------------------------------------------

1 ** Sourav Ghar ** Team Lead ** Total fields ** 6

-------------------------------------------

3 ** vt100 ** %.6g ** name2.lst

-------------------------------------------

2 ** Sourav Dutta ** Project M ** Total fields ** 6

-------------------------------------------

3 ** vt100 ** %.6g ** name2.lst

-------------------------------------------

3 ** Ratna Sengupta ** Programmer ** Total fields ** 4

-------------------------------------------


\n Newline

\t Tab

^ Starts match at the beginning

$ Matches at the end of the string

. Matches any single character

[ABC] Matches any of A,B or C

[A-Ca-c] Matches any of A,B,C,a,b or c

[^ABC] Matches any character other than A,B or C

Desk|Chair Matches any of ‘Desk’ or ‘Chair’

[ABC][DEF] Matches any of A,B or C followed by D,E or F

Regular Expression Metacharacters in Awk:-


[ABC]* Matches zero/more occurences of A,B or C

[ABC]+ Matches one/more occurences of A,B or C

[ABC]? Matches to an empty string or more of A,B or C

( ) (Blue|Black)berry matches to Blueberry or Blackberry

Regular Expression Metacharacters in Awk…contd

Operators Significance

== Is equal to

< Less than

<= Less than or equal to

> Greater than

>= Greater than or equal to

!= Not equal to

~ Matched by regular expression

!~ Not matched by regular expression

Comparison Operators in awk


&& Logical AND

|| Logical OR

! Logical NOT

( ) Used to group compound statements

Compound pattern Operators

Range pattern Operators


awk -F"|" 'NR==3,NR==5 { print NR,$2 } ' name.lst

Shows 3rd,4th and 5th record

BEGIN {

<declare variables>

< write headings>

}

. . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . .

END {

< show subtotals>

}

BEGIN and END blocks

$ awk ‘/Programmer/ { print }’ name.lst # Details of all programmers

0005 | Raghab Das | Programmer| Avaya | 12000.00|

0006 | Sumit kumar Basu | Programmer| Avaya | 12000.00|

0011 | Tamal Sen Sharma | Programmer| Avaya | 12000.00|

0015 | Raghab Dasgupta | Programmer| Avaya | 12000.00|

0025 | Rajib Banerjee | Programmer| Avaya | 12000.00|

$ awk ‘/Prog*/ { print }’ name.lst # Details of all programmers

0005 | Raghab Das | Programmer| Avaya | 12000.00|

0006 | Sumit kumar Basu | Programmer| Avaya | 12000.00|

0011 | Tamal Sen Sharma | Programmer| Avaya | 12000.00|

0015 | Raghab Dasgupta | Programmer| Avaya | 12000.00|

0025 | Rajib Banerjee | Programmer| Avaya | 12000.00|

$ awk '/Operator/ ' name.lst # Details of all Operators

0012 | Damini Sen | Operator | Nestle| 10000.00|

$ awk -F "|" '/Operator/ { print $2,$5 }' name.lst #Field 2 and 5

Damini Sen 10000.00

$ awk -F"|" '/(Sen|Das)gupta/ { printf "%3d % -20s n" , NR , $2 }' name.lst 5 Ratna Sengupta

6 Raghab Dasgupta

(Shows the record number and names of all Sengupta-s and Dasgupta-s)

$ awk -F"|" 'NR==3,NR==5 { print NR,$2 } ' name.lst

3 Sumit kumar Basu

4 Tamal Sen Sharma

5 Ratna Sengupta

(Shows line number and second field of 3rd-5th records)

$ awk -F"|" '$3==" Programmer" { print $2,$4,$5 }' name.lst Raghab Das Avaya 12000.00 Sumit kumar Basu Avaya 12000.00 Tamal Sen Sharma Avaya 12000.00 Raghab Dasgupta Avaya 12000.00 Rajib Banerjee Avaya 12000.00

(Shows 2nd,4th and 5th field of all records where 3rd field contains the word ‘ Programmer’)

$ awk -F"|" '$3~ /Manager/ && $4~ /Nestle/ ' name.lst

0009 | Anindya Das | Manager | Nestle| 30000.00|

(Shows the records where 3rd field contains the string ‘Manager’ and 4th field contains the string ‘Nestle’.

$ awk -F"|" '$3!~ /Programmer|Accountant/' name.lst

0001 | Subhendu Majumdar | Team Lead | Avaya | 25000.00|

0012 | Damini Sen | Operator | Nestle | 10000.00|

0007 | Akash Nag | Salesman | Nestle| 7800.00 |

0009 | Anindya Das | Manager | Nestle| 30000.00|

0019 | Sougata Das | Manager | Avaya | 50000.00|

(Shows all records where 3rd fiels contains strings other than Programmer and Accountant)

$ awk -F"|" '$5>=30000 { printf "%20s t t %d n" , $2, $5 }' name.lst

Anindya Das 30000

Sougata Das 50000

(Shows the name and salary of persons with salary >=30,000)

$ awk -F"|" '$3~ /Manager/ {

> kount = kount + 1

> print kount,$2 } ' name.lst

1 Anindya Das

2 Sougata Das

$ awk -F"|" '$5>=30000 { printf "%20s t t %d n" , $2, $5 }' name.lst

Anindya Das 30000

Sougata Das 50000

(Shows the name and salary of persons with salary >=30,000)

$ cat salavg.awk

BEGIN {

printf "\n\t\t Salary Report of Managers \n\n"

}

$3~ /Manager/ { kount++

tot = tot + $5

print kount,$2,$5

}

END {

print "\n\nTotal managers found is :",kount

print "Average salary is :",tot/kount

}

$ awk -F"|" -f salavg.awk name.lst

Salary Report of Managers

1 Anindya Das 30000.00

2 Sougata Das 50000.00

Total managers found is : 2

Average salary is : 40000

(A)Concatenating Strings

x = “abc””def” # x = abcdef

y = “ghi”

z = x y # z = abcdefghi

String Operators

Built-in String Functions

Functions Significance

gsub( <reg>,<string>,<target>)

Substitutes string <string> in string <target> every time the regular expression <reg> is matched

index( <search>,<string> )

Returns the position of the string <search> in string <string>

length(<string>) Returns the length of the string <string>

match( <string> , <reg>) Returns the position in string <string> that matches the expression <reg>

split(<string>,<store>,<delim>)

Splits string <string> into array elements of <store> based on delimiter <delim>

sub(<reg>,<string>,<target>)

Substitutes string <string> in <target> the first time the regular expression <reg> is matched.

Built-in String Functions

Functions Significance

substr(<string>,<pos>,<len>)

Extracts the portion of the string <string> starting from position <pos> of length <len>

tolower(<string>) Translates the string <string> rto lower case

toupper(<string>) Translates the string <string> rto upper case

Use of Character Functions in awk

$ cat awkf.awk

$5 >= 30000 { print "Length of the string is : " , length ( $2 )

print "First three characters are : " , substr( $2,1,3 )

print "String in lowercase is : " , tolower( $2 )

print "String in uppercase is : " , toupper( $2 )

nm = substr( $2,2,4 )

print " 2nd to 5th characters are :", nm

}

Use of Character Functions in awk…Contd

$ awk -F"|" -f awkf.awk name.lst

Length of the string is : 19

First three characters are : An

String in lowercase is : anindya das

String in uppercase is : ANINDYA DAS

2nd to 5th characters are : Anin

Length of the string is : 19

First three characters are : So

String in lowercase is : sougata das

String in uppercase is : SOUGATA DAS

2nd to 5th characters are : Soug

Use of Character Functions in awk…Use of Split function

$ cat split.awk

BEGIN {

FS="|"

printf "\n Details of name of the Employees : "

print "\n\t Name \t\t\t\t First name \t\t Last name"

print "\n\t............................................................"

}

$5 > 15000 { split( $2 , a , " " )

print "\n\t", $2, "\t\t", a[1] , "\t\t" , a[2]

}

Use of Character Functions in awk…Use of Split function

$ awk -f split.awk name.lst

Details of name of the Employees :

Name First name Last name

………………………...........................................................

Subhendu Majumdar Subhendu Majumdar

Anindya Das Anindya Das

Sougata Das Sougata Das

Special String Constants


\\ Backslash

\a Alert or bell character

\b Backspace

\f Formfeed

\n Newline

\r Carriage return

\t Tab character

\v Vertical tab

\” Double quote

Built-in Numerical Functions


cos(x) Cosine of x in radians

int(x) Integer value of x

exp(x) Returns e raised to power of x

log(x) Natural log of x

rand( ) Returns random number between 0 and 1

sin(x) Returns the sine of x in radians

sqrt(x) Returns square root of x

systime() Current time in seconds since midnight, Jan 1,1970

Arithmetic Functions


x^y Raises x to the power of y

x**y Raises x to the power of y

x%y Calculates the reminder of division of x by y

x+y Adds y to x

x-y Subtracts y from x

x*y Multiplies y with x

x/y Divides ‘x’ by ‘y’

x++ Increments x by 1 and then uses it

++x Uses x and then increments it by 1

y-- Decrements y by 1 and then uses it

--y Uses y and then decrements it

Arithmetic Functions


x+=y x = x+y

x-=y x = x-y

x*=y x = x*y

x/=y x = x/y

x++=y x = x + 1 + y

Arrays

Arrays are normally used to handle more than one related piece of data.

One accesses the individual elements within an array by enclosing the subscript within double brackets.

In AWK, one does not have to declare a variable to be an array, and does not have to define the maximum no. of elements. When one uses an element for the first time, it is created.

In awk arrays, subscript is a string ; viz.,

tot_sales[“Cal”] = 10.15

One can use this in conditional flow:-

If “Cal” in tot_sales

Delete tot_sales[“Cal”]

Multidimensional Arrays

One can use two subscripts to form a multi-dimensional array.

Example : tot_sales[“India”,”Calcutta”] = 100

And can use this in conditional statements also:-

If (“India”,”Calcutta”) in tot_sales

Examples of Arrays

BEGIN {

FS = "|"

print "\n\t Name \t\t Basic \t\t da \t\t hra \n"

print "---------------------------------------------------------------"

}

$3~ /Manager/ {

da = $5 * 0.25; hra = $5 * 0.3;

print $2 ,"\t" , $5 , "\t" , da , "\t\t" , hra ;

tot[1] += $5; tot[2] += da ; tot[3] += hra ; tot[4] += $5 + da + hra ;

}

END {

print "\n-------------------------------------------------------------"

print "Total" , "\t\t\t" , tot[1] , "\t\t" , tot[2] , "\t\t" , tot[3] ;

print "---------------------------------------------------------------"

print "\n\n Grand Total : ", tot[4] ;

}

Examples of Arrays…contd

$ awk –f salcalc.awk name.lst

Name Basic da hra

---------------------------------------------------------------

Anindya Das 30000.00 7500 9000

Sougata Das 50000.00 12500 15000

-------------------------------------------------------------

Total 80000 20000 24000

---------------------------------------------------------------

Multidimensional Arrays-Examples…contd

$ cat pop.awk

BEGIN {

system(" tput clear " );

print "\n\nPopulation in different parts of the cities in India\n\n"

FS = "|"

}

{ print $2 , "\t" , "\t" , $3 , "\t" , $4 ;

tot_pop[$1,$2]+=$4;

}

END {

print "--------------------------------------------------------"

print "Population for Calcutta : " ,tot_pop["India","Calcutta"];

print "Population for Chennai" , tot_pop["India","Chennai"];

}

Multidimensional Arrays-Examples…contd

$ awk –f pop.awk pop.lst

Population in different parts of the cities in India

Chennai West 150000

Calcutta North 234500

Calcutta East 134500

Chennai East 100000

Calcutta South 234508

Calcutta West 100500

--------------------------------------------------------

Population for Calcutta : 704008

Population for Chennai 250000

Conditional Flow

{

if ………..

else if ……..

else if ……..

else ………

}

$ cat sal.awk

4 ~/Nestle/ {

if ( $5 > 10000 )

print $2 , $5 , "Taxable";

else print $2 , $5 , "Non taxable";

}$ awk -F"|" -f sal.awk name.lst

Damini Sen 10000.00 Non taxable

Akash Nag 7800.00 Non taxable

Anindya Das 30000.00 Taxable

Conditional Flow

To substitute ‘if’ test anywhere in the code:-

condition ? True : false

$ cat sal.awk

$4 ~/Nestle/ {

$5 > 10000 ? taxyn = "Taxable" : taxyn = "Non taxable" ;

print $2 , $5 , taxyn ;

}$ awk -F"|" -f sal.awk name.lst

Damini Sen 10000.00 Non taxable

Akash Nag 7800.00 Non taxable

Anindya Das 30000.00 Taxable

LoopsLoops

Loops can be of three types : • Do• While• For

Used to perform some repeatitive jobs.

Do Loop

do

<statement>

While ( <conditions> )

statement in any of the above constructs may be either a simple statement or a series of statements enclosed in { }

Example of Do Loop

$ cat doloop.awk

BEGIN {

linesep = "-"; linedraw = "-"; FS = "|";

}

{

res = $5 / 1000 ;

do

{ linedraw = linedraw"-" ; res--; }

while ( res >= 1 );

print $2 , linedraw ;

linedraw = linesep ;

}

Example of Do Loop…contd

$ awk –f doloop.awk name.lst

Subhendu Majumdar --------------------------

Raghab Das -------------

Sumit kumar Basu -------------

Tamal Sen Sharma -------------

Ratna Sengupta -------------

Raghab Dasgupta -------------

Rajib Banerjee -------------

Damini Sen -----------

Akash Nag --------

Anindya Das -------------------------------

Sougata Das ---------------------------------------------------

While Loop

While ( <condition> )

statement

Example of While Loop

$ cat while.loop

BEGIN {


}

{

res = $5 / 1000 ;

while ( res >= 1 )

{

linedraw = linedraw linesep ; res--;

}



}

Example of While Loop…contd

$ awk –f while.loop name.lst


Raghab Das -------------







Akash Nag --------

Anindya Das -------------------------------

Sougata Das ---------------------------------------------------

For Loop

For ( <var> = <value> , var <operator> <value_last> , <var> [++|--]

<statements>

For ( <subscript> in <array> )

<statement>

Example of For Loop

$ cat for.loop

BEGIN {


}

{

res = $5 / 1000 ;

for ( res = res ; res >= 1 ; res -- )

{

linedraw = linedraw linesep ; res--;

}



}

Example of For Loop…contd

$ awk –f for.loop name.lst


Raghab Das -------------







Akash Nag --------

Anindya Das -------------------------------

Sougata Das ---------------------------------------------------

Loop Breaking Statements

Statement Significance

Break Exits the loop

Continue Leaves the current record and continues with the next record in the loop from beginning of the loop processing statements.

Pretty formatting - printf

Format Meaning

%c Ascii character

%d Integer

%i Integer

%e Floating point number using scientific notation

%f Floating point number(eg., 10.43)

%s String of characters

%g Awk chooses %e or %f display format(whichever is shorter) suppressing non-significant zeroes.

Syntax : printf( format_specifier, var1, var2,…..varn)

Output to another file

Statement Significance

Printf( “Hello world\n”) > “datafile” Creates a file called datafile with the output

Printf( “Hello world\n”) >> “datafile”

Appends the output at the end of existing file “datafile” or creates a new file.

A file should be closed after it receives output from an awk program.

Syntax : close(“filename”)

FunctionsFunctions

Block of code, accesses usually in multiple places in the code.

When awk reaches the end of the function, it implicitly returns the control to the calling routine.

To make an explicit return to the main program, one can explicitly use the return statement.

Syntax :

function fname(parameter list) {

< function_code>

}

Block of code, accesses usually in multiple places in the code.

When awk reaches the end of the function, it implicitly returns the control to the calling routine.

To make an explicit return to the main program, one can explicitly use the return statement.

Syntax :

function fname(parameter list) {

< function_code>

}

Example of Functions

$ cat subcalc.awk

BEGIN {

FS = "|"

show_heading( "\n\t Name \t\t Basic \t\t da \t\t hra \n" ) ;

show_heading( "-------------------------------------------------");

}

$3~ /Manager/ {

da = calc_comp(0.25,$5) ; hra = calc_comp(0.3,$5) ;

print $2 ,"\t" , $5 , "\t" , da , "\t\t" , hra ;

}

function show_heading(heading) {

print heading ;

} function calc_comp(rate,val) { val = val * rate ; return val ; }

Example of Functions…contd

$ awk –f salcalc.awk name.lst

Name Basic da hra

---------------------------------------------------------------

Anindya Das 30000.00 7500 9000

Sougata Das 50000.00 12500 15000

Use of inputs from user in awk program

BEGIN {

FS="|"

printf "%20s%3d\n\n", "Cut-off basic pay : ",cobp

}

$5 > cobp {

kount++

print $2 , $3 , $5 , "\n"

}

END {

print "\n\n The End\n\n"

}

Use of inputs from user in awk program…Contd

$ awk -v cobp=15000 -f tkinp.awk name.lst

Cut-off basic pay : 15000

Subhendu Majumdar Team Lead 25000.00

Anindya Das Manager 30000.00

Sougata Das Manager 50000.00

The End

Controlling Environment

The Unix environment is controlled by a number of pre-defined environment variables.

They are usually defined in the file ‘.profile’ for the user or are defined by user as and when required.

Pre-defined variable

Significance

ERRNO Non-zero exit code of last command that failed. Value changes only when a command fails

LINENO Meaningful only within a shell script. Its value is the line no. of the line in the script currently being executed

OLDPWD Value is always the full pathname of the directory where the control resided before coming into current working directory

PWD Current working directory

SECONDS Integer number of seconds since one invoked the Korn shell

PPID Value of the process id of the parent process of $$

PATH Absolute directory path under which the programs for the commands reside.


Significance

EDITOR Vi or emacs

CDPATH List of colon separated directory name, which is followed by the system when the user issues a ‘cd’ command

ENV Pathname of the shell script containing commands to be executed when the korn shell is invoked

HISTFILE Filename of the korn shell history file

HISTSIZE Integer number containing maximum number of commands to be retained in history files

HOME Pathname of home directory for an user


Significance

IFS Zero/more characters to be treated by the shell as de-limiters when parsing a command line into words or using the read command

COLUMNS Display width used by Korn shell edit mode- vi or emacs

LINES Integer number representing the number of lines displayed by the terminal.

MAIL Pathname of a file to be monitored by the shell for a change in its date of last modification. If change is noted, the shell issues the message : You have mail at the next oppurtunity

MAILCHECK

No. of seconds after which the shell should check for a change in the MAIL file

PS1 Primary shell prompt string.Prompt where commands are issued by the user


Significance

PS2 Secondary prompt screen. Shown when a command pawns more than one line.

PS4 Debug prompt screen , shown when a shell program is executed using –v option.

SHELL Pathname of the shell the user is using

TERM Symbolic alpha numeric string that identifies the type of your terminal

TMOUT Seconds after which automatic logout occurs.

stty

Controls terminal output and sets terminal characteristics


stty –a Displays all current settings

stty –echoe Enables backspacing remove character from the display

stty –echo Keyboard entry is not echoed

stty echo Keyboard entry is echoed

stty intr ^Z

Sets <Ctrl-Z> as the interrupt key

stty eof \ ^a

Sets <Ctrl-a> to terminate output and declare end-of-file, for eg., while creating file using cat command

stty erase ‘^H’

Typing <Ctrl-H> helps to remove the last character typed

stty quit ‘^d’ Typing <Ctrl-d> enables aborting the current shell

stty susp ‘^Z’

Enables suspending a foreground process when <ctrl-Z> is pressed

stty stop ‘^S’

Enables halting the current session by pressing ‘^S’

stty start ‘^Q’

Enables starting the current halted session by pressing ‘^Q’

stty rows 20

Set 20 rows in display

stty rows 20 column 80

Sets 20 rows and 80 columns in display

stty iuclc Maps uppercase alphabets to lowercase

stty olcuc Maps lowercase alphabets to uppercase

stty size Gives the current screen size in terms of rows and columns

stty eol ‘^J’ Pressing <Ctrl-J> does the job of ending a line

stty sane Set the terminal characteristics to values that will work for most terminals

Process

This section will give you a brief idea on

Running Jobs

Process

Scheduling Jobs

What is a process?

Process is a program that has its own address space. Every command fired in Unix has a process associated with it.

Is born when a program starts execution, and remains alive as long as the program is active.After execution is complete, it dies.

Has a name, usually the name of the program being executed.

Kernel controls the process

Kernel is finally responsible for managing the process.

Determines time and priorities allocated to processed, enabling multiple processes to share CPU resources.

Each one is uniquely identified by PID, Process Identifier, allocated by the Kernel when it is born.

Sh Process

A process is immediately set up by the Kernel when a user logs into Unix system.This os usually a Unix command(sh/ksh/bash etc).

Any command executed is actually input to the shell process.

Can be known from the value of shell variable ‘$$’. This process for the shell remains alive till the user logs out.

Parent and Child process

Every process has a parent process(except the root process).

When a command is executed from the shell prompt, that becomes a child process to the parent shell process.

A parent process can ave more than one child process

Cat name.lst| grep ‘student’

Both the process : cat and grep have same parent (shell) process

ps – process status

Displays the attributes of a process

Command Significance ps Selects the process associated with the current

terminal.Shows output in four columns: PID, TTY(terminal info), TIME(total cpu time used by the process), CMD(command gererating the process)

ps –f Displays process ancestry.Produces 8 column output:- UID(userid),PID,PPID(Parent PID), C( amt of CPU time consumed by a process),STIME(time the process started),TIME,CMD


ps –u ems2000 Displays processed of user : ems2000

ps –a Displays all processes of all users; excepting the system processes

ps –x Shows command line in extended format

ps –e Displays system processes

How a process is created in Unix?

Forking : A copy of the the process that invokes a new process is created.Child gets a new PID.

Exec : The parent then overwrites its image with a copy of the program to be executed. Done with exec system call. No additional process is created at this stage.

Wait: Parent then executes the wait system call to keep waiting for the child process to complete. After the child process is completely executed, a terminal signal is sent to the parent.

Internal and External Commands

External program : Commands like : cat, ls, grep, sed etc are external programs.Shell creates a process for each of these commands.

Shell Scripts : Shell executes shell scripts by spawning to another shell, which then becomes the parent of the commands featured in the script.

Internal Commands : These are commands which are executed directly by the current shell. No additional processes are generated. Example : cd , echo etc

Running jobs in background

Unix is a multi-tasking system , allowing to perform more than one job at a time.

One job can be executed in foreground at a time, but many can be executed at background.

‘&’ operator at the end of the command line interprets the job to be done at background. Shell returns the PID of the background job for tracking.

Job control in Korn and Bash Shells

Korn or bash shell users can use the job control facility of the shell to manipulate jobs.One can put a job in the background, bring it back to foreground, suspend it, run it later, or even kill it.

bg

You are running a job in the foreground.

It is taking a lot of time. You want to do some other jobs.

Press ^Z. The job running in the foreground will be suspended.

Enter the command : bg at the OS prompt and push it to be executed in the background.

Unix will return you a PID for the process pushed to background.

jobs

Shows the jobs running in background.

Command : jobs –l prints PID of each job besides job number.

This command is used by the user to push a job from background to foreground.

fg

Used to bring a background job to foreground and execute it.

fg [%<job_no>] brings the job from background to foreground and executes it. For example , issuing of the command : fg %2 executes job number 2 (viewed by jobs command) from background to foreground and executes it.

Execution of fg command alone from the OS prompt resumes latest created background process to be executed inforeground.

stop

Pauses a job which is being executed in background.

One can use the command : bg to resume execution of the stopped job in background or can issue command : fg to bring it to foreground and execute it.

Syntax : stop [%<job_no>]

kill

System often requires to control execution of a process. This is done by sending signals to the process.

The process, after receiving the signal, may ignore it, terminate itself, or do something else.

The command : kill is used to terminate or suspend a process running in background, taking unusual long time.

Syntax : kill [options] <PID>


kill 2905 Kills the process with Pid : 2905

kill –9 2905 kill –s SIGKILL 2905

Surely kills a process with PID 2905

kill –15 2906 kill –s SIGTERM 2906

Terminates the process with PID 2906

kill –24 2900 kill –25 2900

Pause the process with PID 2900

kill –26 2900 kill –s SIGCONT 2900

Start the paused process with PID 2900

kill &! Kill the latest background job

nice

Processes in the Unix are usually executed with equal priority.

But, high priority jobs must be executed at the earliest. Unix offers : nice command to change the priority of execution of jobs.

Ordinary users can only reduce priority; super user can do both.

Nice value ranges from 0 to 39 ; commands run with a value of 20 in both.

Example : nice –n 15 prog1.sh &

The above command reduces the pririty value to 35

The nice and priority values of the processes can be displayed with ps –l command.

batch

Schedules job for later execution, when the system load permits it.

Prevents too many high load jobs to run at the same time.

Syntax for running a program : batch < prog1.sh

Any job scheduled in this way also goes to the at queue and can be removed using : at –r , provided it is fired before the job has been executed.

atat

Schedules a program for execution at a specified time.

Displays a list of scheduled jobs.

Removes jobs from the scheduled list.

Flag Significance

-l Displays the list of jobs scheduled by the user

-m Mail a report of successful execution of the job

-r <joblist>

Remove the jobs specified in the joblist from queue


at 2300 my_job.shExecutes program my_job.sh at 11:00 pm tonight

at 11:00 pm my_job.sh

at 11:00 P my_job.sh

at 2300 today my-job.sh

at now + 6 hours njb.sh Schedules execution of njb.sh after 6 hours from now

at 6:30 pm next week njb.sh

Schedules execution of program : njb.sh at 6:30 pm next week

at now + 1 year njb.sh Schedules execution of program : njb.sh after 1 year from now

at 3:08 pm + 1 day njb.sh Schedules execution of program : njb.sh at 3:08 pm tomorrow.

at 9 am Mon njb.sh Schedules execution of njb.sh at 9 am coming Monday.

at 9 am tomorrow Schedules execution of njb.sh at 9 am tomorrow.

at now + 5 minutes < njb.sh Schedules execution of njb.sh after 5

minutes from now. at –f njb.sh now + 5 minutes

at 08:15 Jan 12 < njb.sh

Schedule execution of njb.sh at 08:15 am on January 24.

cron

Runs jobs periodically.

Mostly dormant, but wakes every minute and checks for a file in /usr/spool/cron/crontabs directory for programs to be executed at that instant.

A user may be permitted to place a crontab file after his login name in this directory. This file contains a list of commands, along with a schedule for execution.

First field indicates number of minutes after the hour when the command is to be executed. The range 00-10 schedules every minute in the first 10 minutes of the hour.

Second fields indicates the hour in the 24 hour format for scheduling.

3rd field(1 to 31) controls the day of the month. ‘*’ means all.

4th field(1-12) indicates the month

5th field(0-6) indicates the day of the week(Sunday with 0)

6th field contains the command

Sample format for a crontab file

User can modify the file containing his cron commands as follows:-

Create a file : cron.txt in the format described earlier.

Issue the following command from the Os prompt: crontab cron.txt.

There will be now a file ‘ems2000’( for eg., the name of the user) in /usr/spool/cron/crontabs with the contents of cron.txt.

One can see the contents of crontab file issuing the command : crontab -l


shutdownShutdown Unix server

suGo as you like

wallSend message to everybody

umaskSet Default Permission

ulimitSet maxm. Space for user

duShow disk usage

dfFree bytes available

fingerShow User information

cryptPassword protect a file

shutdown

Executed from the root user to shut down the system.


shutdown –g2 Powers down system after 2 minutes

shutdown –y –g0 Immediate shutdown

shutdown –y –g0 –i6 Shutdown and reboot

shutdown 17:30 Shutdown at 5:30 pm

shutdown –r now Shutdown and reboot

shutdown –m Bring the system down to maintenance(single user mode)

su

Takes to super user mode.

Commands Significance su Takes to super user mode. One can have all

the accesses same as the root user. Root password is required after executing the command

su mat One logs in as the user : mat. Asks for the password.

wall

Addresses all the users simultaneously.

$ wall

………….

…………

^D

All the text written will be reflected in the terminals of all the users.

$ wall < mesg.txt

Contents of the file mesg.txt are sent as the wall message.

umask

Used by System Administrator to set the default permissions to assign to each file to be created by the user.

User can modify this default settings.

To give permission 751 as a default to newly created files, umask value will be (777 – 751) = 026.

This value is assigned as a command : umask 026

ulimit

Faulty programs or processes can eat up disk space.So, a restriction is to be imposed on the mazimum size of a file that an user is permitted to create.

Ordinary user can reduce the default value . Super user can increase / decrease it.

Examples Significance ulimit –a Shows soft limits

ulimit –Ha Shows the hard limits

ulimit –t <size>

Set the CPU time in seconds

ulimit –f <size>

Set the maxm. Size in blocks

ulimit –d <size>

Sets the maxm. size of data blocks in KB

ulimit –m <size>

Sets the maxm. Size of memory in KB.

du

Shows disk usage – amount of space taken by a group of files in units of 512 bytes or KB.

Descends all sub-directories from the directory in which the command is fired.

Flag Significance-a Print entries for each file encountered in the directory

hierarchies in additional to the normal output

-k Give the block count in terms of 1024 bytes

-s Give the grand total of disk usage for each of the directories

-r Print messages about directories that cannot be read, files cannot be accessed etc.

df

Displays the number of free 512 byte blocks and free inodes available for file systems by examining the count kept in the superblocks.

Flag Significance

-b Reports only the number of KB free

-e Report the number of files free

-f Reports only the actual count of the blocks in the free list

-i Reports the total number of inodes, number of free inodes, number of used inodes,% of inodes in use

-k Reports the allocation in KB

finger

By default, the command finger lists for each user in the system:- Login name Full given name Terminal write status( if write permission is denied) Idle time Login time User’s home directory and login shell Project on which the user is working Any plan put by user in the file .plan under home directory Office location and telephone number Last time the user received the mail, the last time user read the mail.

Flag Significance

-b Suppresses information on home directory and shell

-f Suppresses the header info

-h Suppresses the project information

-i Suppresses the info on idle time

-l Suppresses long output

-p Suppresses plan

-R Print user’s hostname

-s Forces short output format

File Significance

/etc/utmp Who file

/var/admin/utmp Last login file

/etc/passwd User’s name, office etc

/etc/.plan User’s plan

/etc/.project User’s project info

/var/mail Mail directory

Files accessed while showing output

crypt

Used to encode/decode files using personal password


crypt < uncrypted_file > crypted_file

Crypts file ‘uncryped_file’ to crypted_file. Asks for a password.

crypt passwd < file1 > file2

Crypts file : file1 to : file2 based on the password : passwd

crypt passwd < file2 > file1

Decrypts file : file2 to : file1 using the password : passwd

vi –x file2 Helps to view an encrypted file : file2. Asks for a password

Communication

Communicationmailx

Send and receive mails

writeSend message to all users

ftpSend and receive files

rloginRemote login

rcpCopy files to remote server

mailx

Used to send/receive mails

Syntax for sending mails:-

echo <body> | mailx –s <subject> <remote_address>

cat <message file> | mailx –s <subject> <remote_address>

Syntax for receiving mails:-

mailx -e

mailx [-HL] [-u user]

mailx -f [-HL] [filename]

Sending mail

$ mailx –s “Test heading” [email protected]

this is a test mail for communication

~.

The job performed above will send a mail to : [email protected] with subject of the mail as : Test heading and body of the mail as : this is a test mail for communication.

Case : 1

$ mailx [email protected]

Subject : Test mail number 2


~.

The job performed above will send a mail to : [email protected] with subject of the mail as : Test mail number 2 and body of the mail as : this is a test mail for communication.

If no subject is entered at the command line, a prompt asks for the subject.

Case : 2

$ echo “Hello mike”|mailx –s “Heading1” [email protected]

The job performed above will send a mail to : [email protected] with subject of the mail as : “Heading1” and body of the mail as : “Hello mike”.

Case : 3

The job performed above will send a mail to : [email protected] with subject of the mail as : “Heading1” and body of the mail with the texts written in the file : msgfile.

Case : 4

$ cat msgfile| mailx –s “Heading1” [email protected]

$ mailx [email protected]

Subject : Test mail number 2


~h

To: [email protected]

Subject: testing

Cc: [email protected]

Bcc: [email protected]

(continue)

testing mail--please ignore

~.

EOT

.

Case 5- You want to send a mail to some user and cc it to some other users.

Viewing received mails

mailx -e

mailx [-HL] [-u user]

mailx -f [-HL] [filename]

Flag Significance

-e Test for presence of mail. mailx prints nothing and exits with a successful return code if there is mail to read.

0 Mail present1 No mail2 Other error

-f Read messages from filename instead of the user's system mailbox. If filename is not specified, the secondary mbox is used.

-H/-L Prints header summary only

-u user Read user's mailbox.Can be used only if read access to user's mailbox is not read protected.

Options available in viewing received mail

When one executes mailx command for viewing received mails:-

$ mailx

mailx Revision: 1.179.214.2 date: 98/12/01 01:29:55 Type ? for help.

"/var/mail/ems2000": 2 messages 2 unread

>U 1 [email protected] Mon Aug 18 04:50 33/1199 RE: 3rd mail

U 2 ems2000 Mon Aug 18 03:55 16/561 Program to be re-schedule

?_

It shows a list of unread mails with a ‘?’ prompt below, waiting for commands from the user.

Commands available at ‘?’ command prompt

Flag Significance<n> Shows message number <n> , the number typed

at the first column for every message.

type Shows all the messages in one shot, one by one

quit(q) Quit mail utility

next Shows next message

reply <n> Reply to message number <n>

mail <user>

Opens mail utility to write mail to user <user>

writeType text into other user’s terminal for information

Case : 1 $ write ems2000

Hello, how are you

^d

The command flashes the message : Hello, how are you to all person’s terminal who have logged in as user : ems2000.

Case : 2

$ write ems2000 pts/tc

Hello, how are you

^d

The command flashes the message : Hello, how are you to that person’s terminal who is working in terminal pts/tc and logged in as user : ems2000.

Case : 3

$ write ems2000 < fileshow.txt

The command flashes the message written in the file : fileshow.txt to all person’s terminal who have logged in as user : ems2000.

User must be logged in and mesg should be set to y( mesg=y) for successful write communication.

ftp

Copies file over a network connection between local host to remote host. ftp runs on client server.

Syntax : ftp [-i] [-v] [-n] <remote_host>

Flag

Significance

-i Disable interactive prompting by multiple file commands.

-n Disable auto-login. If auto-login is enabled and user issues an ‘open’ command to establish connection to a remote server, he gets login and passwd prompt. If it is disabled by ‘n’, user does not get such prompts. He then writes: user <username> <passwd> in the ftp mode to specify the user and password.

-v -v Enables verbose output. IT shows the actions when files are passed from one server to another.

Commands fired from ftp mode


! <command> Executes command <command> in the local server

append <lf> <rf>

Copy file <lf> in the local server to the end of file : <rf> in the remote server. If no such file exists in remote server, a new file is created.

ascii Sets the file transfer type to ASCII

binary Sets the file transfer type to binary

bye Close the connection to remote host, if it is open, and exit

case Remote file names with name containing all letters in uppercase are translated to lowercase when the file is transported from remote server to local.

cd <remdir> Set the working directory in the remote server to <remdir>

cdup Set the current directory of the remote server to the parent of current working directory.

chmod <mode> <file>

Changes the mode of the file : <file> to mode <mode>

close Terminate the connection to the remote server

mdelete <remote_files>

Deletes the files in the remote server

mdir <rem_files> <lcl_file>

Writes a list of remote files to local file <lcl_file>

mget <rem_files> Copy remote files to local system

get <rem_file> [<lcl_file>]

Copt the remote file to local file.

mkdir <dirname> Creates a directory <dirname> in remote server.

mls <remfiles> <lclfile>

Writes an abbreviated listing of remote files to local file <lclfile>

modtime <rem_file> Shows the last modification time of remote file <rem_file>

mput <local files> Copy files from local server to remote server

delete <rem_file> Delete file <rem_file> from the remote server. The file can be an empty directory.

dir <rem_dir> [<lcl_file>]

Shows a listing of remote directory <rem_dir> to terminal/ into local file <lcl_file> .

disconnect Close the connection to remote host

beep Beeps when command is executed

lcd [local_dir] Specifies the current working directory of the local server to directory : <local_dir> . If unspecified, user’s home directory is considered for navigation.

newer <file_name> get file if remote file <file_name> is newer than local file <file_name>

prompt Toggle forcing interactive prompting on multiple commands

runique Toggle store unique for local files. IF turned on, then while receiving files from remote host, if a remote file already exists with a name equal to a local file, a ‘.1’ is appended to the name. For another repeatition, a ‘.2’ is appended till 99 such cases are found.

size <remote_file> Shows the size of the remote file

quit bye

reget <remfile> [<lclfile>]

Acts like get. If local file <lclfile> exists and is smaller than the remote file <remfile>, local file is assumed to be partially transferred from remote server.Then, the transfer is continued from the apparent point of failure. This command is very useful to transfer very large files over the netwoprk that tend to drop connections.

status Shows the current status of ftp- informations are shown on remote server name and all the settings(prompt,type,bell,case etc)

sunique Toggle store unique for remote files. If turned on, then while transferring files to remote host, if a remote file already exists with a name equal to a local file, a ‘.1’ is appended to the name of the file transferred from local to remote server.. For another repeatition, a ‘.2’ is appended till 99 such cases are found.

system Shows the type of operating system running in remote server

umask [new mask] Set the default umask on the remote server to new value. IF nothing is specified, the current umask value of the remote server is displayed.

help [<command>] If only help is typed from ftp prompt, it shows a list of all commands that can be used from ftp prompt. If a command is mentioned for help, it shows one line help explaining its significance.

glob toggle metacharacter expansion of local file names. If put on, one can use ‘*’ , ‘?’ etc characters to specify portion / all of the file name.

rlogin

Logs into remote host <rhost> as user <username> with/without any password.

If the users are same on both remote and local server, then option <username> is not required.

Prompting for password depends on the entries in file : /etc/hosts.equiv or .rhosts under remote user’s home directory of the remote server

Syntax : rlogin <rhost> [-l <username>]

rcp

Copy Single File

rcp [-p] <srcfile1> <rem_user> @<remserver>:<pathname>/<dest_file> rcp [-p] <srcfile1> <remuser>@<remserver>:<dest_dir>

Copy Multiple Files

rcp [-p] <srcfile1> [<file2>]... <remuser>@<remserver>:<dest_dir>

Copy One or More Directory Subtrees

rcp [-p] -r <srcdir1> [<dir2>]... <remuser>@<remserver>:<dest_dir>

Copy Files and Directory Subtrees

rcp [-p] -r file_or_dir1 [file_or_dir2]... <remuser>@<remserver>:<dest_dir>

Copy files from server1 to server2 by server3

rcp <user1>@<server1>:<pathname>/<file> <user2>@<server2>:<pathname>[/<file>]

Copies file(s)/directory(ies) from one server to another

Shell Programming

List of instructions written within a file.

Programming logics can be built in.

Executes a set of related instructions, saving time,resource and imparts better control.

The file has to be executable to make the thing work.

Invoking Sub-Shell manually

When a script file with execute permission starts running, it runs in a sub-shell under the current shell.

One can invoke a sub-shell manually to execute the scripts :-

$ sh prog1.sh

Or

$ sh < prog1.sh

where prog1.sh is the name of the program. It may not have execute permission at this point of time.

Execute a script in the current shell

Syntax : . <progname>

Whenever a shell script is executed, it runs in a sub-shell under the current shell. To make the script run in the current shell, one have to trigger the program as : . <progname>. The program need not have execute permission before executing in this fashion.

Specifying the type of shell

There are different kind of shells with different utilities and syntaxes. A shell script running fine in Korn shell, may not run ok in Bourne shell.

To specify the interpreter shell, one can execute the program from OS prompt as :-

$ ksh < prog1.ksh

This will execute program prog1.ksh using Korn shell as the interpretor.

It is always a better approach to specify the shell at the first line of the program:-

#!/bin/ksh

Reading from Keyboard

The most common practice of any programming is to take input from user at runtime and act on that.

‘Echo’ and ‘read’ statement in shell scripts help us to achieve that.

$ cat prog1.sh

Echo “Enter name:\c”

read p_name

Echo Hello $p_name

$ prog1.sh

Enter name: Suman

Hello Suman

Defining own variables

Defining global and local variables for programming is an indispensable practice.

One can declare it at the top of the program /anywhere at the time of usage.


g_val=20 Declares a variable : g_val and assigns a value 20 to it.

g_name=‘two words’

Declares a variable : g_name and assigns a value ‘two words’ to it.Place the value to be assigned within quotes if contains more than one words


g_val=“”g_val=‘’g_val=

Assigns null value to the variable g_val

let g_cnt=20 Declares a variable : g_cnt and assigns a value 20 to it.

g_cnt=20 readonly g_val

Makes the value in the variable : g_cnt readonly. Further change in the value is not allowed.

unset g_cnt Wipes off the value from variable : g_val.

Special parameters used by Shell

Shell uses some special parameters which imparts us different information on a program/command , various arguments passed to it, success/failure of the program etc.

Parameters

Significance

$0 Name of executed program/command

$1,$2…. First, second etc arguments passed to a program

$* Complete set of positional parameters as a single string

$# Number of arguments passed in command line

$? Exit status of last command

Parameters

Significance

$! PID of last background job

$$ PID of the current shell

$_ Argument passed to the last command

Set – Manipulating Positional Parameters

The command : set can be used to manipulate different positional parameters($1,$2…) from a string / output of a command

$ set Happy birthday to you

$

$echo $1 $2

$ Happy birthday

$ set `date`

$

$echo $1 $3 $2 $6

$ Fri 19 Aug 2003

Shift- Shifting Positional Parameters

At a time, one can have values for 9 positional parameters( $1 to $9). To shift their values one place to the left, this command is used.

$ set I know this is not a valid issue to discuss at the meeting before the board of directors

$echo $1 $2 $3 $4 $5 $6 $7 $8 $9

$ I know this is not a valid issue to

$ shift; echo $1 $2 $3 $4 $5 $6 $7 $8 $9

$ know this is not a valid issue to discuss

Performing Calculations

Expr, echo let and (( )) comes handy

cnt_val=èxpr $cnt_val + 1`

cnt_val=ècho $cnt_val + 1|bc`

let cnt_val=cnt_val + max_limit

((z=x+y+2))

cnt=èxpr $a \* 2`

a=èxpr $rt / 2`

per=èxpr $ $m1 + $m2 $ / 5`

echo

Displays strings , values in terminal or stores them in files.


echo Hello dear String : Hello dear shown in terminal

echo hello `hostname`

String : Hello sb1dtl02 shown in terminal

a=256; echo $a The value of variable a is displayed in terminal

string=happy; echo “$string”>del

Creates a file called : del or overwrites an existing file , the content of the file being : happy

Echo “$string”>>del Appends the contents o the variable $string at the end of the file : del

Escape sequences used with echo to format output in terminal

Sequence

Significance

\033[0m Normal display

\033[1m Bold display

\033[4m Underlined display

\033[5m Blinking display

\033[7m Display in reverse video

\07 Beep

\b Backspace

\t Tab

\r Carriage return

\c Positioning the cursor after the statement

tput

Formats displayOptions Significance

clear Clears the screen

cup r c Moves cursor to row r and column c

bold Bold display

blink Blinking display

rev Reverse display

smul Starts mode underline

rmul Ends mode underline

bel Bells the terminal

lines Echoes no. of lines in the screen

ed Clear to end of display

el Clear to end of line

Syntax : tput [<options>]

Decision making – If

Making decisions based on conditions is the most popular practice in every programming language. Shell provides decision building using “If” statements.

If <conditions>

then

<actions>

fi

If <conditions>

then

<actions>

else

<actions>

fi

If <conditions>

then

<actions>

elif <conditions>

then

<actions>

else

<actions>

fi

echo "Enter a value:\c"

read val

if test $val -gt 100

then

echo Value $val greater than 100

elif test $val -ge 50 -a $val -lt 100

then

echo Between 50 and 100

elif test $val -gt 0 -a $val -lt 50

then

echo Between 50 and 100

else

echo No comments

fi

Test

Used to compare numerical values,strings,find properties of different files etc.

Numericl test operators

Significance

-gt Greater than

-lt Lesser than

-eq(=) Equal to

-ne(!=) Not equal to

-ge Greater than or equal to

-le Lesser than or equal to

String test operators Significance

string1 = string2 True if string1 is same as string2

string1 != string2 True if string1 is not equal to string2

-n string True if string length >0

-z string True if string length = 0

while [ -z "$userid" ] do tput cup 5 5 echo "Enter userid:\c" read userid done

if test $userid = super then tput cup 11 5 echo No such userid allowed exit fi

while [ -z "$pswd" ] do tput cup 7 5 echo "Enter password:\c" stty -echo read pswd stty echodone

while [ -z "$npswd" ] do tput cup 9 5 echo "Confirm password:\c" stty -echo read npswd stty echodone

if test $pswd = $npswd then tput cup 11 5 echo o.k else echo wrong entry.... fi

And,Or

echo "Enter city and state:\c"read city state if test $city = cal -a $state = wb then echo You stay in Calcutta and in West Bengal elif test $city != cal -a $state = wb then echo You stay in West Bengal, not in Calcutta elif test $state != wb -a $ $city = bom -o $city = del $ then echo you stay in other metros fi


-a and

-o or

E x a m p l e

Case

Used to build logic depending on different values of same object.

Syntaxcase <variable> in

<value1>) <actions> ;;

<value2>) <actions> ;;

*) <actions>

esac

echo "Enter an animal:\c" read animal

case "$animal" in [Tt]iger|[Ll]ion) echo He is the king;; Crow|Sparrow|parrot) echo They are birds;; [0-9]*) echo Name cannot start with numbers;; ?) echo You entered a single alphabet only;; ??) echo You entered two alphabets only;; [hH]ip*m*) echo Is it hippo?;; *) echo Correct entry , but could not be identified esac

Loop processing

Three common loops used in shell scripts are :-

1. While

2. Until

3. For

While

Syntax while <condition is true>

do

<actions>

done

while [ -z "$userid" ]

do

tput cup 5 5

echo "Enter userid:\c"

read userid

done

Until

Syntaxuntil <control does not return true>

do

<actions>

done

echo "Enter number:\c"

read no

until [ $no -gt 10 ] # till $no is not greater than 10

do

echo $no

((no=no+1))

done

For

Syntax for <variable> in <value1> <value2> <value3> ….

do

<actions>

done

for city in bombay delhi calcutta

do

echo $city is a metro city

done

Loop breaking statements

Statement

Significance

break Control comes out of the loop completely

continue Does not proceed with current record further and starts processing the next record from the start of the loopn=20

while [ $n -le 100 ]

do

echo $n

((n=n+1))

if test $n -eq 30

then

break

fi

done

n=0

while test $n -le 10

do

((n=n+1))

if test $n -eq 5

then

continue

fi

echo $n>>del

done

Sleep

Syntax: sleep < number of seconds>

Idly waits for number of seconds specified. Mainly used to display an event clearly to user.

Basename

Extracts the base filename from an absolute pathname

Example:-

$ basename /home/ems2000/shell_scripts/ppc1.sh

ppc1.sh

When used with second argument, it strips off the second argument from the first

Example:-

$ basename ppc1.sh .sh

ppc1

The here document (<<)

Shell offers a mechanism of data to be read by a shell script to be read from itself rather than from another file. The here operator(<<) comes handy.

echo "Enter city code:\c"

read city_code

grep "^$city_code" << stcity

01 Calcutta

02 Bombay

stcity

if [ $? -ne 0 ] ; then

echo invalid code

fi

E x a m p l e

Putting all the inputs at a time

A program may ask for different inputs several times and at several points of execution.Instead of putting the inputs when they are asked for by the program, one can put all of them at a time while calling the program for execution.

echo “First name:\c"

read fnameecho “Middle name:\c"read mnameecho “Last name:\c"read lnameecho your full name is $fname $mname $lname

$ prog8.sh<<end

> soma

> sen

> gupta

> end

First name:Middle name:Last name:your full name is soma sen gupta

Set -- : Help Command Substitution

Command ‘set’ is often required with command substitution.

If the output of the command executed begins with ‘-’( as with ls –l),

‘set’ interprets it as one of its options ,thus giving error messages.

By default, set displays all environment variables. This creates

problem when the argument in its command line evaluates to null

string ( for eg., in set `grep <srchstring> <filename(s)> ` statements.

IF the pattern cannot be located, set will operate with no arguments

and will display all the environment variables on the terminal.

To avoid this, use : set -- `command line` option.

echo "Enter city code:\c"

read city

set --`grep "^$city" <<tillend

01 Calcutta India WestBengal

02 Mumbai India Maharashtra

tillend`

if [ $? -eq 0 ] ; then

echo "\n\tCity code : $1"

echo "\n\tCity name : $2"

echo "\n\tCountry : $3"

echo "\n\tState : $4"

else

echo Invalid code

fi

Command grouping

Group

Significance

( ) Executes the commands placed inside ‘(‘ and ‘)’ separated by ‘;’ in a sub-shell under the current shell. They have the same PID

{ }

Same as above; but uses current shell only. The closing curly brace must be on a separate line. To have both the braces on the same line, terminate the last command with a ‘;’$pwd

/home/ems2000

$ ( cd scripts;pwd )

/home/ems2000/scripts

$pwd

/home/ems2000

$pwd

/home/ems2000

$ { cd scripts;pwd; }


$pwd


Conditional Execution

Group Significance

cmd1 && cmd2 Command cmd2 is executed only when command cmd1 succeeds

cmd1 || cmd2 Command cmd2 is executed only when command cmd1 fails

cmd1 || cmd2 && cmd3

Metacharacter coming first gets higher priority

$grep ems2000 prog.sh && echo “String found”

String found

$ grep ems99876 prog.sh || echo “String not found”

String not found

String Handling


$ expr “Training” : ‘.*’8

Finds the length of the string : Training

$ expr “Subhendu” : ‘…$..$’he

Extracts the fourth and fifth character from the string

$expr “Anamoly” : ‘[^d]*l’6

Locates position of character ‘l’ within string : Anamoly

Syntax Significance

$ expr “<string>” : ‘.*’ Finds the length of the string : <string>

$ expr “<string>” : ‘…$..$’ Extracts the fourth and fifth character from the string

$expr “<string>” : ‘[^d]*<char>’

Locates position of character <char> in the string <string>

Conditional Parameter Substitution

Syntax : {<variable> : <option> <stg>}

Options

Significance

+ Variable <var> evaluates to <stg> if <var> is defined and a non-null string is assigned to it. dir1=`ls` echo ${dir1:+”This directory is not empty”}

- Variable <var> is evaluated to <stg> if it is undefined or assigned a null stringEcho “Enter directory to be copied:\c” read dirto dir_child=${dirto:-/home/ems2000/scripts}IF dirto is null or not set, dir_child=/home/ems2000/scripts. The value of dirto is still null

Options

Significance

= If the variable <var> is null, it is assigned the string <stg>. echo “Enter filename:\c” read flname grep $pattern ${flname:=emp.lst}IF flname is null, flname=emp.lst

? Evaluates the parameter if the variable is assigned and non-null, otherwise it echoes the string following it. The shell is also terminated.This option is useful in terminating a script if the user fails to respond properly to shell directives.

echo “Enter pattern:\c”

read pattern${pattern:? “No pattern”}

Shell variables($1,$2 etc) can also be used instead of <var>

Merging Streams

Symbol

Significance

0 Standard input

1 Standard output

2 Standard error

Usually, the inputs required for a program are typed in by user in terminal, all the outputs in terms of messages and all the error messages ( shell genarated) are shown in the terminal .

But, one can make a program take inputs from a file, store outputs(messages) and / or error messages in a file, instead of displaying them in a terminal.


$ prog2.sh > sclist

All the output messages and prompts will be taken into file : sclist. However, the inputs to the program will be received from terminal.

$ cat prog2.sh

echo "Enter pattern:\c"

read pattern

echo “\nPattern is $pattern"

$ prog2.sh > outlist # Enter pressed

Patternfromterminal # written from terminal

$

$ cat outlist

Enter pattern:

Pattern is Patternfromterminal

$

$ prog1.sh < namelist

All the inputs prompted by program prog1.sh are taken from file : namelist

$ cat prog1.sh

echo "Enter first name:\c"

read fname

echo “\nEnter last name:\c"

read lname

echo “\nWelcome $fname $lname"

$ cat namelist

Suman

Das

$ prog1.sh <namelist

Enter first name:

Enter last name:

Welcome Suman Das

$ prog1.sh < namelist > sclist All the inputs prompted by program prog1.sh are taken from file : namelist and all the output messages are stored in file : sclist

$ cat prog1.sh

echo "Enter first name:\c"

read fname

echo “\nEnter last name:\c"

read lname

echo “\nWelcome $fname $lname"

$ cat namelist

Suman

Das

$ prog1.sh <namelist >sclist

$ cat sclist

Enter first name:

Enter last name:

Welcome Suman Das


$ 1>&2 Send the standard output to the destination of standard error

$ cat prog2.sh

ans=y

while [ "$ans" = "y" -o "$ans" = "Y" ]

do

echo "\n Enter city:\c" 1>&2

read city

echo "$city"

echo "\n More(y/n):\c" 1>&2

read ans

done

$prog2.sh > out.1

Enter city:calcutta

More(y/n):y

Enter city:bombay

More(y/n):n

$cat outlist

calcutta

bombay


$ 2>&1 Send the errors to the destination of standard output

$ cat prog3.sh

echo "Searching all the .kkl files"

ls -ltr *.kkl

$prog3.sh > out.2

*.kkl not found

$ prog3.sh>out.2 2>&1

$

$ prog3.sh

Searching all the .kkl files

*.kkl not found

$cat out.2


$cat out.2


*.kkl not found

Using subroutines

Using subroutines is one of the most popular practice in any programming language.

Codes in a subroutine does a piece of job. It can take inputs from an user, do some jobs and can return some value to the calling program.

Subroutines are used to:-

• Modularize program sections.

•Perform repeatitive jobs.

•Do Calculations.

Syntax:- <subroutine_name> {

<actions>

}

function <function_name> {

<actions>

}

Inputs to a subroutine are identified within the subroutine as : $1,$2,$3 …etc.

A subroutine is called from a program as :-

<subroutine_name> [ <value1>] [ <value2> …]

Subroutines are always declared at the top of a program

$ cat prog0.sh

show_name() {

case "$1" in

[bB]) tput bold

echo "$2";;

[lL]) tput rev

echo "$2";;

*) tput rmso

echo "$2"

esac

}

show_name b `whoami`

show_name l `hostname`

show_name n `uname -r`

show_name n “The End”

Passing values to subroutines

$ incr_val() {

sum=`expr $1 + 1`

echo $sum

}

no=12

no=`incr_val $no`

echo value is $no

Subroutine returning values

$ prog2.sh

value is 13

$ function show_name {grep "$1" *.kshif [ $? -eq 0 ] ; then return 1else return 2Fi }

echo "Enter pattern:\c"read patternshow_name "$pattern"if [ $? -eq 1 ] ; then echo "Pattern found..."elseecho "Not found"fi

Unix Overview This is not an exhaustive training on Unix, covering all the arenas with full details....

Documents

Transcript of Unix Overview This is not an exhaustive training on Unix, covering all the arenas with full details....