2013 Unix Wk2 Handout (1)

7/29/2019 2013 Unix Wk2 Handout (1)

1/16

3/11/13

1

FACULTY OFENGINEERING &

INFORMATIONTECHNOLOGIES

Introduction to Unix

COMP2129

Bob Kummerfeld

Files and Shells

COMMONWEALTH OF AUSTRALIACopyright Regulations 1969

WARNINGThis material has been reproduced and communicated to you by or on behalf ofthe University of Sydneypursuant to Part VB of the Copyright Act 1968 (the Act).The material in this communication may be subject to copyright under the Act. Anyfurther copying orcommunication of this material by you may be the subject ofcopyright protection under the Act.

Do not remove this notice.

The file system

root

bin

usr home

pub

tmpdev

ascii

etc

passwdken hrmt

Ass1 Erat

Erat.c a.out

doit

Paths and full path names

root

bin

usr home

pub

tmpdev

ascii

etc

passwdken hrmt

Ass1 Erat

Erat.c. a.out

doit

/


root

bin

usr home

pub

tmpdev

ascii

etc

passwdken hrmt

Ass1 Erat

Erat.c. a.out

doit

/

/usr/pub/ascii

Full path name


root

bin

usr home

pub

tmpdev

ascii

etc

passwdken hrmt

Ass1 Erat

Erat c. a.out

doit

/

/usr/pub/ascii

Full path name

Relative path name

.

..

7/29/2019 2013 Unix Wk2 Handout (1)

2/16

3/11/13

2


root

bin

usr home

pub

tmpdev

ascii

etc

passwdken hrmt

Ass1 Erat

Erat c. a.out

doit

/

/home/hrmt

/usr/pub/ascii

Full path name

Relative path name

.

..

Home directory ~

~hrmt

Files

Several types of files- Ordinary files

- Contain data, eg text, program- Directories

- Actually a special file that contains a list of files- Devices

- The file is a name that refers to a device such as a disk drive ornetwork interface

- special files- The content of these files is generated when the file is read

- Eg the files in /proc on a linux system

File command

The file command tries to work out what type of file you have:

bash$ file Sunix

Sunix directory

bash$ cd Sunix

bash$ file notes

notes: English text

File command and C files

bash$ file fperror.c

fperror.c: C program text

bash$ gcc fperror.c

bash$ file a.outa.out: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked

(uses shared libs), not stripped

bash$ cp fperror.c fperrorbash$ gcc fperrorfperror: file not recognized: File format not recognized

collect2: ld returned 1 exit status

At login

Shell reads from/etc/profile

Then it reads your own.bash_profile

.profile

Is your environment different?How can you see these files, starting with .

Access and permissions

bash$ ls -l

drwxr-x--- 6 judy judy 4096 Dec 14 2012 Cs12000

drwxrwxr-x 12 judy judy 4096 Jul 10 2012 Cs12001

drwxrwxr-x 6 judy judy 4096 Mar 15 12:29 EPS

drwxr-x--- 4 judy judy 4096 Feb 11 1999 IF

-rw-r----- 1 judy judy 4882 Feb 12 2001 Kay.ps

drwxrwxr-x 9 judy judy 4096 Mar 13 22:01 SDM

drwxr-x--- 7 judy judy 4096 Dec 17 11:06 Uidp

Directory/file etcuser,group,other

7/29/2019 2013 Unix Wk2 Handout (1)

3/16

3/11/13

3

Meanings of permissions

Read- File:- Directory

Write- File:- Directory

Execute- File:- Directory

Meanings of permissions

Read- File: program can read it- Directory: can ls it

Write- File: program can alter it- Directory: can add/remove files in it

Execute- File: can run the program- Directory: can read files in it, if name given

Some important cases

Your home directoryls ld .

Protecting files from yourselfchmod u-w precious_file.c

Making files executablechmod u+x doit

General form of chmod symbolic commands

chmod [u g o] [+ - ] [r w x] filenames

What the shell does

gcc -ansi -wall -pedantic *.c

first word is the command name -gccspaces are delimitersexpansions of metacharacters*look for this along $PATHexecute program with that namepass rest of line as parameters

programmer of command (gcc) decides what isallowed

Paths

default search paths - $PATH echo $PATH colon separated list of paths

Initial path

PATH=

/local/usr/bin:/gnu/usr/bin:/usr/bin:/bin:/usr/ccs/bin:/usr/ucb:/usr/openwin/bin:.:

7/29/2019 2013 Unix Wk2 Handout (1)

4/16

3/11/13

4

Paths searched by shell

root

bin

usr home

pub

tmpdev

ascii

etc

passwdken html

Ass1 Erat

Erat.c a.out

doit

/local/usr/bin

/gnu/usr/bin

/usr/bin

/bin

/usr/ccs/bin

/usr/ucb

/usr/openwin/bin

.

bin

Special characters

Selected shell short cuts* [ ] 0-9 a-z

Examplesls *.c

gcc *.c

ls -ld [0-9]*

ls [a-z]*.c

rm i *

Redirection and pipes

stdin

stderr

stdout

a.out

bash$a.out

0

2

1

Redirection

stdin

stderr

stdouta.out

bash$ a.out

bash$ a.out < data1 >res1 2>errors

data1 res1

errors

Redirection - appending to a file

stdin

stderr

stdouta.out

bash$ a.out

bash$ a.out < data1 res1 2>errors

data1 res1

errors

Redirection andpipes

stdin

stdouta.out more

stdin

stdout

a.out |more

pipe symbol

7/29/2019 2013 Unix Wk2 Handout (1)

5/16

3/11/13

5

Initial shell on lab machines

bash$ env

LOGNAME=demo

LD_LIBRARY_PATH=/gnu/usr/lib:/usr/openwin/lib:/local/usr/lib:/usr/local/openssl/lib

WWW_HOME=http://www.ug.cs.usyd.edu.au/

TERM=vt100

HOSTTYPE=sparc

PATH=/local/usr/bin:/gnu/usr/bin:/usr/bin:/bin:/usr/ccs/bin:/usr/ucb:/usr/openwin/bin:.:/usr/bags/stubin

HOME=/usr/cs2/demo

SHELL=/gnu/usr/bin/bash

PS1=bash\$

HZ=100

http_proxy=http://www-cache.cs.usyd.edu.au:8000/

ftp_proxy=http://www-cache.cs.usyd.edu.au:8000/

MANPATH=/local/usr/man:/gnu/usr/man:/usr/share/man:/usr/openwin/man

gopher_proxy=http://www-cache.cs.usyd.edu.au:8000/

OSTYPE=SunOS5

NNTPSERVER=news.cs.usyd.edu.au

OPENWINHOME=/usr/openwin

SHLVL=1

EDITOR=/local/usr/bin/red

TZ=Australia/NSW

WWW_http_GATEWAY=http://www-cache.cs.usyd.edu.au:8000/

_=/usr/bin/env

Shell command to save typing

Create file called doit, containing#!/bin/bash!echo "Compile the masterpiece"!gcc -W -Wall -pedantic -ansi *.c!Make executable and run itbash$ chmod u+x doit!bash$ ./doit!Compile the masterpiece!bash$ !

Your own commands

Create a command that will show all the details of the 5 mostrecently changed files in the current directory

ls ltr | tail 5

Need execute permission Need it on path

Save typing on common command sequences

Create a file with commands to- compile erat.c- run it, taking data from 01_data, saving the results in 01_res- ditto 02_data and 02_res- ditto 03_data and 03_res- then compare actual and expected results in (01_expected,

02_expected, 03_expected)

doit

gcc -W -Wall -pedantic -ansi erat.c

a.out < 01_data > 01_res



echo comparing first test results ..

diff 01_res 01_expected

echo comparing second test results ..


echo comparing third test results ..


doit

Then make doit executableIf . not in $PATH, need ./doit, else doitWhat about 50 tests?

7/29/2019 2013 Unix Wk2 Handout (1)

6/16

3/11/13

6

doit again

gcc -W -Wall -pedantic -ansi erat.c

echo Test on 01_data - normal > result

a.out < 01_data >> result

printf Test on 02_data - normal\n >> result


echo Test on 03_data - boundary >> result


doit so far

Saves typing Means you will be more likely to test systematically you know there has to be a better way to handle this, especially

with 50..100.. tests

Another tiny task

Afile has a list of people's last names several per line. We want toknow how many people with different names are in the file.

bash$ cat name_list

Wang Ng Smith Lee

Ng Howard Lee

bash$

bash$ fmt -1 name_list > names_split

bash$ sort -u names_split > names_sort

bash$



bash$ wc names_sort

the programmer might check as they go.


bash$ cat names_split

Wang

Ng

Smith

Lee

Ng

Howard

Lee

bash$ wc names_split

7 7 32 names_split


bash$ wc names_sort

5 5 25 names_sort

bash$ cat names_sort

Howard

Lee

Ng

Smith

Wang

7/29/2019 2013 Unix Wk2 Handout (1)

7/16

3/11/13

7

How do you get to know about fmt and all the othercommands?

Online manual

Practice to automate your knowledge

Summary

Shell- scripts a scripting language- Speed of creating solutions- Interpreted, slower- default search paths - $PATH

- colon separated list of paths

File System- Files- Directories- Paths

xkcd.org comment on kilobyte Shell commands and scripts

The Unix Shell

a shellis unix command interpreter the shell reads commands typed in a command window or from a

file, interprets them and runs the appropriate programs

simple shell commands such as:gcc myprog.c

will run the program (gcc) you specified

more complex commands may involve running several programsand connecting one to the next throughpipes

the shell also as simple control structures that dont involverunning a program but instead control how other programs are run

eg: if and for statement

7/29/2019 2013 Unix Wk2 Handout (1)

8/16

3/11/13

8

The Unix Shell: key concepts

I/O redirection and pipes scripts variables control structures

these shell features are supported by many simple commands thatcan be combined in scripts to provide powerful services

I/O redirection

when programs are run they have three standard files opened:- standard input (0)- standard output (1)- standard error (2)

these are normally all connected to the terminal window can be redirected using > or < or |!gcc myprog.c >output!

Pipes

pipes let you connect the standard output of one program to thestandard input of another program

myprog | hisprog! this has the same effect as

myprog >tempfile!! !hisprog

7/29/2019 2013 Unix Wk2 Handout (1)

9/16

3/11/13

9

Script Arguments

shell script arguments are available in specialvariables $1, $2 etc

for example, the script called compile:#!/bin/sh!

! !gcc $1.c -o $1!you invoke this like:

compile myprog!a list of all script arguments is available in theshell variable $*!

Control Structures

there are many different shell commandinterpreters available

-we use bashMost shells offer the control stuctures:

-if-for-case-while

these are implemented by the shell programitself, not by separate commands

if statement

The if statement is used to test a condition andcontrol flow of execution in a script

if command!!then!! !command2!! !command3!!fi!

if statement

example:if myprog

then

echo myprog ok

else

echo myprog failed

fi

then clause executed ifmyprogexits with result 0

else clause executed ifmyprog exitswith non-zero result

test command

there is a command called test that will testsome condition and exit with true or false (0 or 1)exit status

if test $1 == fred !! !then!! ! !echo first arg is fred!! !fi!test can test many conditions, especially relatingto files - see the manual entry (man test)

calling your own program test is a bad idea!

for statement

the for statement is used to iterate over a list,usually a list of files:

doit script:

for name in $*!! !do!! ! !gcc $name.c -o $name!! !done!this will compile each file name given when thescript is invoked, eg doit myprog hisprog

7/29/2019 2013 Unix Wk2 Handout (1)

10/16

3/11/13

10

while

the while statement executes a block of a shellscript while some condition is true, eg:

while test $result != 10!!do!! !!!done!

case

case is a selection statement (like switch in C)case $type in!

! !1) !myprog;;!! !2) !hisprog;;!! !pattern) otherprog;;!! !esac!

Useful Commands for handling text files

see the manual entries for details

sort!sorts lines of text in a file, veryflexible choice of key field to sorton, can remove duplicates,sends sorted lines to output

cut! cuts fields out lines of text from afile and sends the result to output

tr! transliterates or removescharacters from a fileeg remove \r from a file

comm!compares files and prints linesthat appear in only one or boththe files

Example

#!/bin/sh

tr A-Z a-z < $1 |

tr -cs a-z \n |

sort -u |comm -13 /usr/dict/words

Example

#!/bin/sh

tr A-Z a-z < $1 |

tr -cs a-z \n |

sort -u |

comm -13 /usr/dict/words

Read from first argument to script

Example

#!/bin/sh

tr A-Z a-z < $1 |

tr -cs a-z \n |

sort -u |


convert upper case to lower case

7/29/2019 2013 Unix Wk2 Handout (1)

11/16

3/11/13

11

Example

#!/bin/sh

tr A-Z a-z < $1 |

tr -cs a-z \n |

sort -u |



convert non-alphabetic to a newline

Example

#!/bin/sh

tr A-Z a-z < $1 |

tr -cs a-z \n |

sort -u |




sort and remove duplicates

Example

#!/bin/sh

tr A-Z a-z < $1 |

tr -cs a-z \n |




sort and remove duplicates

print lines that appear in input but notin /usr/dict/words

Example

#!/bin/sh

tr A-Z a-z < $1 |

tr -cs a-z \n |


What does it do?!

Shell scripts instead of programs

The combination of the shell scripting language (variables, pipes,control structures etc) and many general purpose text processingcommands can often be a more effective way of solving a problemthan writing a program in a conventional language (Java, C etc)

More useful shell features

changing your promptthe prompt is printed by the shell to indicate that it is waiting for you to

type a command

you can change the prompt to contain useful information such as currentmachine or directory. This is done using the PS1 shell variable

bash2.5$ PS1=\h \W

congo1 bob%

7/29/2019 2013 Unix Wk2 Handout (1)

12/16

3/11/13

12

command substitutionan argument enclosed in backquotes indicates that a command is to run

and the output used as the actual argument(s)

eg myprog `cat argfile`

will run the cat command, take the output (ie the contents of argfile) anduse it as the arguments for myprog

subshells

sometimes commands are run in another copy of the shell a subshell the environment is copied from the parent shell, the subshell may change

it but it reverts to the original when the subshell exits

tar cf MyDir | (cd Somewhere; tar xf -)

subshells

sometimes commands are run in another copy of the shell a subshell the environment is copied from the parent shell, the subshell may change

it but it reverts to the original when the subshell exits

tar cf MyDir | (cd Somewhere; tar xf -)

done in a subshell

also useful to collect output from several commands:(echo My Data; cat file1) >output!

this also works with for and whilefor X in $*!do!!echo $X!!cat $X!done > newfile!

take the collected output from the for loop andredirect to newfile

arithmeticthe expr command in standard shell will evaluate its arguments as an

expression

myprog `expr $count + 1`!this will run expr to calculate the value of the shell variable count, add

1 to it and send the answer to the output where it will be used as theargument to myprog

in the Bash shell you can use the command let for assignmentstatements of variables

! !let count = count +1!

the test command will evaluate boolean expressions. It will alsolet you test file characteristics

if test d name!! !then!! ! !echo name is a directory!! !fi!

-d name is a directory-s file exists and is non-zero size-r file exists and is readable-w file exists and is writableetc

7/29/2019 2013 Unix Wk2 Handout (1)

13/16

3/11/13

13

reading text from within a shell scriptread x!

will read a line from standard input and store it in x

read name!if [ $name == Fred ]!then!!echo name was Fred!fi!

input from within a script

also called a here document

This word can be anything, it isused to terminate the document

grep $1

7/29/2019 2013 Unix Wk2 Handout (1)

14/16

3/11/13

14

Comparing files

diff will show the differences between files sdiff is a side by side diff that shows both texts next to each other with

the differences indicated.

top

the top command will show running processes dynamically the displayis updated every second or so to show the details of processes

very flexible options, such as sorting on various process attributes

Strange file names

if you inadvertently (or on purpose) end up with a file with a strange nameit can be hard to work with it

eg a file called -x will often be confused with an argument -x (x can be almostany char)

so nano R will not try and open a file called

-R. To make this work use: nano ./-R

Common problems

forgetting to set execute permission forgetting that the current directory is not in $PATH using test as the name of a prog or script

End of Section

Example: generating web pages

web pages can be static files that contain a description of the pagewritten in the HTML markup language

web pages are often generatedby a program when you requestthe page

web page generation is carried out by programs that take currentinformation from many sources and write out the HTML for thepage

7/29/2019 2013 Unix Wk2 Handout (1)

15/16

3/11/13

15

How the Web works(the 2 minute version)

a user types a URL into a web browserprogram

the browser parses the URL and determinesthe machine to contact for the page

the browser makes a network connection to aweb serverrunning on the remote machine andrequests the web page specified by the URL

the web server copies the page from a file tothe browser

the browser displays the page

Web Pages

web pages are usually written in HTML (HyperText MarkupLanguage) eg:

Heading

This is bold text.

but can be images, sounds, plain text etc

Creating your own web pages

web pages are stored in files in (SIT servers):$HOME/lib/html

a file ending in .html is written in HTML a file ending in .txt will be displayed as plain text to allow the web server to access your files they need to be world

readable along with the directories

Creating your own web pages

setting permissions:chmod 711 $HOME

chmod 711 $HOME/lib

chmod 711 $HOME/lib/html

chmod 744 $HOME/lib/html/myfile.txt

CGI Scripts

generated web pages are often created by cgi scripts that run onthe web server machine in response to a browser asking for a URL

cgi scripts are executable programs or scripts in files ending in.cgi

could be written in C, java, perl, pythonorcould be written in shell!

Making a web pagewith a CGI script

Web Browser Web Server

url

web page

CGIscript

7/29/2019 2013 Unix Wk2 Handout (1)

16/16

3/11/13

16

CGI shell script

#!/bin/shecho "Content-type: text/plain"echoecho Testing my CGI script

tells the browser the type ofweb page

the blank line is essential

content of the page

CGI script

#!/bin/shecho "Content-type: text/plain"echocat example.txt

page content can come from anywhere

Count the visitors

#!/bin/shecho "Content-type: text/plain"echocat example.txtecho "Hello, today is " `date`

count=`cat counter`count=`expr $count + 1`echo $count >counterecho "There have been " $count " visits to this page"

use the outputof thecommandat that point

Whos logged on?

echo "There are "who | cut -c1-10|sort -u|wc -lecho "people logged on"echo "Friends logged in:"for name in `cat friends`do

who | grep $namedone | cut -c1-10 |sort -u

Output of the group ofcommands within the forloop is sent through the pipe

Entire web sites are generated from data held in databases Languages such as Perl, PHP or Python are commonly used COMP2129 class2go web site is generated by python scripts Shell can be used for simple web page generation

Summary

2013 Unix Wk2 Handout (1)

Documents

Transcript of 2013 Unix Wk2 Handout (1)