Post on 28-Aug-2019
Course Overview and Introduction to the UNIX ShellCSC209: Software Tools and Systems Programming (Winter 2019)
Furkan Alaca & Paul Vrbik
University of Toronto Mississauga
https://mcs.utm.utoronto.ca/~209/
Week 1
Alaca & Vrbik (UTM) 1-Shell Week 1 1 / 38
Overview
1 Course Objectives & Outcomes
2 Course Evaluation Scheme and Workflow
3 Academic Integrity and Effective Ways of Getting Help
4 Preparing for CSC209
5 What is UNIX?
6 Using a UNIX Shell to Run Programs and Access the File System
Acknowledgement: These slides are derived, in part, from content written byAndrew Petersen and Dan Zingaro.
Alaca & Vrbik (UTM) 1-Shell Week 1 2 / 38
What is this Course About?
Using a UNIX shellThe shell is an interface for interacting with the operating system(OS)It is programmable
Programming tools: Develop more efficiently
Debugging tools (gdb, valgrind)Build automation tools (make)Version control (git)
Systems programming: Interacting with the OS
The file systemSystem callsProcessesCommunication
The C programming language will be our tool of choice
Other systems-level languages exist, e.g., Go
Alaca & Vrbik (UTM) 1-Shell Week 1 3 / 38
Course Outcomes
After taking this course, you should be able to...
Diagram and explain the memory usage of an arbitrary C program
Read the man page of an unfamiliar system call or library functionand be able to understand and use it
Write command-line tools in adherence to the UNIX philosophy,which leverage system calls, inter-process communication, thestandard library (libc), and the network stack
Effectively use standard UNIX development tools (e.g., gdb, make,git, valgrind, gdb, bash) and command-line utilities (e.g., cat,less, grep)
Alaca & Vrbik (UTM) 1-Shell Week 1 4 / 38
Evaluation Scheme
5%: Weekly lecture prep (due 10pm on Sundays)
10%: Weekly tutorial exercises (due 10pm on Fridays)
5%: Assignment 1 (C Basics and Pointers)
10%: Assignment 2 (System Calls and Dynamic Memory)
10%: Assignment 3 (Processes)
10%: Assignment 4 (Communication)
10%: Midterm Exam (Fri. Feb. 15, 5pm)
40%: Final Exam (Min. 40% grade required to pass the course)
Alaca & Vrbik (UTM) 1-Shell Week 1 5 / 38
A Typical Week in CSC209
Prepare for class
Watch videos on PCRS to familiarize yourself with conceptsSolve simple exercises (worth marks)URL: https://pcrs.teach.cs.toronto.edu/UTM209-2019-01/
Attend class to consolidate and deepen your knowledge
Having prepared in advance, you can take better advantage of yourin-class hoursPractice by solving more advanced problemsAsk questions
Put your knowledge into action with assignments and weekly tutorials
Alaca & Vrbik (UTM) 1-Shell Week 1 6 / 38
Assignment and Tutorial Submission
Assignments will be submitted over git
Repositories will be managed with MarkUs:https://mcsmark.utm.utoronto.ca/csc209s19/
Each assignment or tutorial will have its own directory in yourrepositoryStarter code will be pushed to your repositoriesMarks will be made available through the MarkUs web interface
Marking will be automated
Be careful with required file names, directory structure, and outputformatEnsure that your code compiles without warnings or errors, and can berun without crashing the lab PCs
Alaca & Vrbik (UTM) 1-Shell Week 1 7 / 38
Assignment and Tutorial Submission (2)
Code submissions that do not compilewill receive a grade of zero
Alaca & Vrbik (UTM) 1-Shell Week 1 8 / 38
Assignment Resubmission
To encourage you to learn by looking at the test output and fixing theerrors, we will allow resubmissions
We will send you feedback in the form of test results soon after thesubmission deadline
The feedback is not your mark: Some tests are worth more than othersSome tests might not even be worth marksDo your best to fix the errors that we’ve identified
You will have 48 hours after the feedback is sent out to correct theerrors and resubmit
Your assignment will receive the higher of:
The grade on the original submissionThe grade on the resubmission, with a 10% deduction
Alaca & Vrbik (UTM) 1-Shell Week 1 9 / 38
Academic Integrity
The work you submit must be your own, done without participationby others. It is an academic offence to hand in anything writtenby someone else without acknowledgement.
You are not helping your friend when you give them a copy of yourassignment
You are hurting your friend when you ask them to give you a copy oftheir assignment
Things that you shouldn’t do:
Looking at another student’s assignmentUsing code that you haven’t written, without attributionAsking someone else to write your codeDiscussing the assignment implementation with other students
Alaca & Vrbik (UTM) 1-Shell Week 1 10 / 38
On the Other Hand...
Give a man a fish and feed him for a day. Teach a man to fishand you feed him for a lifetime. (Ancient Chinese proverb)
You are helping your friend when you:
Explain and/or clarify important conceptsReview/modify/practice exercises from PCRS, lectures, and previousweeks’ tutorials togetherHelp each other understand error messages, man pagesYou are encouraged to do the above, by, e.g., participating on thediscussion board (https://mcs.utm.utoronto.ca/forum/), formingan FSG (https://www.utm.utoronto.ca/asc/facilitated-study-groups-fsgs)
Your TAs are there to help you
Same guidelines apply: Don’t ask them “How do I do the assignment?”TAs won’t give you the solution, but they can help you understandtools/concepts that you need for completing the assignment, commonmistakes, helpful techniques for debugging your code, etc.
Alaca & Vrbik (UTM) 1-Shell Week 1 11 / 38
Linux
This course assumes some familiarity with Linux
Get accustomed to using a text editor on Linux
Familiarize yourself with basic shell commands/utilities
Learn to ssh into the lab machines to work remotely
Install a Linux virtual machine on your PC (even better: dual-boot)
This is how cool you will feel after taking CSC209.
Alaca & Vrbik (UTM) 1-Shell Week 1 12 / 38
C Programming
This isn’t a C programming course
We will use C, and cover some key concepts in C
But after taking this course, you should just as easily be able to learnanother language (like Go) and accomplish the same tasks (refer backto Course Objectives & Outcomes slides)
You have already learned at least two programming languages:Python (CSC108 and 148) and Java (CSC207)
Java syntax is heavily influenced by C
We are expecting you to quickly pick up C programming on your own
Alaca & Vrbik (UTM) 1-Shell Week 1 13 / 38
Preview of What’s Next
This week: UNIX shell and compiling C programs
Tutorial 1 is due on Friday (Jan. 11)
Next week: Arrays and pointers
Lecture prep is due on Sunday (Jan. 13)Tutorial will be due next Friday (Jan. 18)
Assignment 1 will be posted next week
Alaca & Vrbik (UTM) 1-Shell Week 1 14 / 38
The UNIX Philosophy
A simple summary of the UNIX philosophy (from A Quarter-Century ofUnix by Peter H. Salus, 1994):
Write programs that do one thing and do it well.
Write programs to work together.
Expect that the output from your program will be used as input foranother (e.g., by piping)Don’t require interactive input
Write programs that handle text streams, because that is a universalinterface.
More on Wikipedia:https://en.wikipedia.org/wiki/Unix_philosophy
Alaca & Vrbik (UTM) 1-Shell Week 1 15 / 38
UNIX vs. Linux vs. UNIX-like
UNIX is a proprietary OS developed by AT&T in 1969
UNIX quickly gave rise to both free and commercial imitations suchas BSD, Linux, Solaris...
The macOS kernel is a derivative of BSD
When we say UNIX we generally refer to these “UNIX-like” OSs,which are often referred to colloquially as “*nix”
We will use Linux in this course
Linux is the most widely-used OS in the UNIX-like family: It runs on allsorts of devices, e.g., PCs, smartphones, printers, security cameras,wireless routers...
More on Wikipedia:https://en.wikipedia.org/wiki/Unix_philosophy
Alaca & Vrbik (UTM) 1-Shell Week 1 16 / 38
The UNIX Timeline
System III & V family
BSD (Berkeley Software Distribution)
FreeBSD
NetBSD
OpenBSD
SunOS
Solaris
NextStep
Xenix OS
GNU
Linux
CommercialUNIX
HP-UX
AIX
UnixWare
IRIX
BSD family
1970 1980 1990 2000 Time
Microsoft/SCO
Richard Stallman
Darwin
GNU/Hurd
Linus Torvalds
Andrew S. Tanenbaum
Minix
3.3
6.5.30
4.1.4
Research UNIX 10.5
Bell Labs: Ken Thompson,Dennis Ritchie, et al.
Bill Joy
AT&T
IBM
SGI
Univel/SCO
2010
4.4
6.1
11.0
macOS 10.12
7.1
11.3
7.2
11i v 3
4.11
3.4
16.4
Sun/Oracle
Apple
Theo de Raadt
0.9
Matthew Dillon
DragonFly BSD 4.8
Timeline of UNIX-like operating systems.Alaca & Vrbik (UTM) 1-Shell Week 1 17 / 38
Examples of Common UNIX Tools
cd
mkdir
ls
cp, mv, rm
touch
who
diff
head, tail
cat
sort
grep
wc
tr
(The commands in
this column are for
text filtering)
stdin (standard input)
stdout (standard output)
stderr (standard error)
pipes
(All of the above in red have a man page: See next slide)
Alaca & Vrbik (UTM) 1-Shell Week 1 18 / 38
How to Learn Linux
Use it.
Don’t worry about memorizing stuff
Jump in and start working on your task(s) at hand, and look thingsup as needed
Excellent resources:Man pages: type man name-of-command to learn how a commandworks (Pro-tip: sometimes it’s easier to jump straight to the examples)Man pages also contain more general topics/tutorials, e.g., try man
intro or man gittutorial
Arch wiki: https://wiki.archlinux.org/ (very useful even forusers of other Linux flavours such as Ubuntu)Our ultimate goal is to teach you how to learn on your own—getting tothat stage will be the biggest achievement of your 4-year Universityeducation!
Common tasks will quickly start to become familiar
Alaca & Vrbik (UTM) 1-Shell Week 1 19 / 38
How to Learn Linux (2)
How you will feel when you find an answer in the man pages.
Alaca & Vrbik (UTM) 1-Shell Week 1 20 / 38
How to Learn Linux (2)
Even the man command has its own man page: Check man man.
Alaca & Vrbik (UTM) 1-Shell Week 1 21 / 38
The Shell: Commands
$ gcc -o hello hello.c
The $ is a prompt, i.e., prompting the user for input (commands)
We are using the bash shell, but there are others too
A command can be built-in to the shell (e.g., cd, ls, type)
See man builtins
A command can also be the name of an executable file
Can provide the path to the executable fileIf no path is provided, the shell searches for the executable file in alldirectories listed in the PATH environment variable
Alaca & Vrbik (UTM) 1-Shell Week 1 22 / 38
The Shell: Running a Program
$ gcc -o hello hello.c
$ ./hello
The first line compiles the C program hello.c into an executable filehello
The second line loads the hello program into memory and hands itto the OS to run the program
Recall from CSC207: Java programs are compiled into bytecode; Cprograms are compiled into native code
Alaca & Vrbik (UTM) 1-Shell Week 1 23 / 38
Memory Model
Memory is dividedinto segments
The program itselfis loaded into thebottom segments:
Read/writedataRead-only codeand data
Alaca & Vrbik (UTM) 1-Shell Week 1 24 / 38
Wait a Second...
$ gcc -o hello hello.c
$ ./hello
What’s with the ./? Why put it in front of hello but not gcc?
This is a good opportunity to discuss the importance of askingquestions if you spot something unfamiliarIn this course, a single missing or extra character can make a bigdifference (remember this when we introduce you to our good friends*, **, and & from the C language)
Can’t say “it’s just one character” if that character holds importantconceptual meaning, compared to perhaps a simple ; which justterminates a statement
The current directory is typically not in the PATH, so executing such acommand is done by inputing ./command, where . represents thecurrent directory
Type echo $PATH to output your PATH variable
Alaca & Vrbik (UTM) 1-Shell Week 1 25 / 38
The UNIX File System Hierarchy
/ "root"
"essential user command binaries"
bashcatchmodcpdateechogrepgunzipgziphostnamekilllesslnlsmkdirmoremountmvnanoopenpingpspwdrmshsutartouchumountuname
/bin/dev"device filesincl. /dev/null"
/home"user homedirectories"
/proc"process & kernelinformation files"
/lib"libraries &kernel modules"
/mnt"mount files fortemporary filesystems"
/usr"read-only user applicationsupport data & binaries"
"standard includefiles for 'C' code"
"obj, bin, libfiles for coding& packages"
/usr/bin"most usercommands"
/usr/include
/usr/lib
/usr/local"local software"
/usr/local/bin/usr/local/lib/usr/local/man/usr/local/sbin/usr/local/share
/usr/share"static data sharableaccross all architectures"
/usr/share/man"manual pages"
/etc"configuration files for the system"
crontabcupsfontsfstabhost.confhostnamehostshosts.allowhosts.denyinitinit.dissuemachine-idmtabmtools.confnanorcnetworkspasswdprofileprotocolsresolv.confrpcsecurettyservicesshellstimezone
/var"variable data files"
/var/cache"application cache data"
"data modified asprogrammes run"
/var/lib
"lock files to track resources in use"
/var/lock
/var/log"log files"
/var/spool"tasks waiting tobe processed"
/var/spool/cron/var/spool/cups/var/spool/mail
/var/opt"variable data for installed packages"
/var/tmp"temporary files saved between reboots"
/sbin"essential systembinaries"
fdiskfsckgettyhaltifconfiginitmkfsmkswaprebootroute
/opt"optional softwareapplications"
/root"home dir. forthe root user"
Alaca & Vrbik (UTM) 1-Shell Week 1 26 / 38
File System Hierarchy
/usr/bin/bash
The above is the path to the executable file bash (our shell)
The leading / represents the root directory
usr is a subdirectory of /
bin is a subdirectory of usr
bash is a file located in bin
A directory is a file that contains directory entries
Each directory entry maps a file name to an inode
An inode is a data structure that contains information about a file,such as its physical address on the disk
Stay tuned for CSC369
Alaca & Vrbik (UTM) 1-Shell Week 1 27 / 38
Directory Entries and inodes
12345 afile
sizeowner UID, GIDaccess timemodified timecreation timelink and block countspermissions
direct pointersto file blocks
single indirect pointer
double indirect pointer
triple indirect pointer
pointers to next fileblocks
12345Inode
Directory Entry
Alaca & Vrbik (UTM) 1-Shell Week 1 28 / 38
File System Hierarchy on the Lab PCs
On most UNIX systems, a user bob’s home directory is /home/bob
On the lab PCs, it is /student/bob
Your home directory is located on a networked file system that ismounted to the local directory /student on every lab PCType df to see a list of devices, network locations, etc. that aremounted to directories in your file system
Your home directory is also represented by the shortcut ~
cd ~ changes your current working directory to your home directoryType pwd to see what your current working directory is
Alaca & Vrbik (UTM) 1-Shell Week 1 29 / 38
Files in UNIX
“Everything is a file” is a defining feature of UNIX
It even has its own Wikipedia page:https://en.wikipedia.org/wiki/Everything_is_a_file
UNIX provides a file interface (i.e., open, read, write, and close) forall Input/Output (I/O)
Regular filesDirectoriesSpecial files (e.g., /dev/null, /dev/urandom)Devices (e.g., keyboard, mouse, printer, video output)
Turn your volume up and try running cat /dev/urandom | padsp
tee /dev/audio > /dev/null
Pipes for inter-process communication (can be named or unnamed*)Network sockets (sometimes)
Alaca & Vrbik (UTM) 1-Shell Week 1 30 / 38
Redirecting Output
By default, programs read from stdin (default: keyboard), writeresults to stdout (default: screen), and write errors to stderr
(default: screen)
Use > to redirect stdout, and 2> to redirect stderr
> overwrites the output file; use >> to append
1 and 2 are the file descriptors of stdout and stderr, respectively
Example:
ls >myfiles.txt: List the files in the current directory, saving theoutput in myfiles.txt (sample output for ls coming up on next slide)
More: https://www.tutorialspoint.com/unix/unix-io-
redirections.htm
Alaca & Vrbik (UTM) 1-Shell Week 1 31 / 38
Pipes and Process Substitution
Pipes transfer output from one command as input to anothercommand, e.g.,
ls | wc -l: Counts and outputs the number of lines of text that weregenerated by the ls command
You can also feed the output of one or more processes to stdin ofanother process with <; this is called process substitution
Try: ls >out.txt, followed by wc -l <out.txt
Now, try: wc -l <(ls) (and observe how the output differs)Whereas a pipe redirects stdin, process substitution saves the outputof a command into a temporary file, and passes the descriptor to theother process
The round brackets in the last command above tell the shell toevaluate the enclosed commandMore here: https://unix.stackexchange.com/questions/17107/
process-substitution-and-pipe
Alaca & Vrbik (UTM) 1-Shell Week 1 32 / 38
Directories and Links
$ ls
file1 file2 file3 test1 test2
$ ls -a
. .. file1 file2 file3 .hidden test1 test2
$ ls -la
total 16
drwxr -xr -x 4 bob staff 4096 Jan 6 20:18 .
drwxr -xr -x 3 bob staff 4096 Jan 6 20:18 ..
-rw-r--r-- 1 bob staff 0 Jan 6 20:16 file1
-rw-r--r-- 1 bob staff 0 Jan 6 20:17 file2
lrwxrwxrwx 1 bob staff 5 Jan 6 20:17 file3 -> file2
-rw-r--r-- 1 bob staff 0 Jan 6 20:18 .hidden
drwxr -xr -x 2 bob staff 4096 Jan 6 20:16 test1
drwxr -xr -x 2 bob staff 8192 Jan 6 20:16 test2
$
Alaca & Vrbik (UTM) 1-Shell Week 1 33 / 38
Directories and Links (2)
In the ls -la output from the previous slide:First column is a permission string, e.g., drwxr-xr-x
First character indicates whether it is a directory (d), link (l), or aregular file (-)A symbolic link is a file that contains a reference to another fileA hard link is a regular file that references an inode (doesn’t actuallyappear to ls as a “link”—having multiple hard links to an inodeessentially means referencing the same inode by multiple file names)
The rwx flags represent the read, write, and execute privileges on thefile possessed by the user (file owner), group, and other usersFor a directory, r allows you to list its contents (ls), w allows you tocreate and delete directory entries in it, and x allows you to enter thedirectory (cd)
Alaca & Vrbik (UTM) 1-Shell Week 1 34 / 38
Directories and Links (3)
In the ls -la output from the previous slide (continued):
Second column is the number of hard links to the inode referenced bythe file
Note that . and .. are hard links that are present in every directoryIf you rm a file name with only one hard link, and if the file is notcurrently opened by any running process, the file is deleted
Otherwise, only the specified file name is deleted, and the hard linkcount is decremented
Third and fourth columns are the user (owner) and group of the file
Fifth column is the size (in bytes) of the file
Sixth column is the last modified date of the file
Last column is the file name
Running ls -ali would also show you the inode number (everyinode has a unique numeric ID) referenced by each file
Alaca & Vrbik (UTM) 1-Shell Week 1 35 / 38
Changing File Permissions
If you are the file owner (or root), you can change a file’spermissions with chmod
Example: chmod 754 file.txt changes the permissions offile.txt to grant:
7: rwx to the user (owner)5: rx to the group4: r to all other users
Octal notation: For each subject (user, group, other), add up thevalues for r (4), w (2), and x (1)
Other ways of using chmod:
chmod u+rwx
chmod go-x
Exercise: See man chmod to find out what the above commands do
Alaca & Vrbik (UTM) 1-Shell Week 1 36 / 38
Globbing
Glob patterns are strings that expand to match multiple file names(similar, but simpler, than regular expressions)
Check man 7 glob (this displays the contents glob(7), i.e., the glob
entry from section 7 of the man-pages)
A ? matches any single character
A * matches any string, including the empty string
An expression [list of characters] matches a single characterinside the list, e.g., [abc], [a-z], [ax-z]
Usage examples:
rm *.log: Remove all files ending in .log
ls *.pdf: List all files in the current directory ending in .pdf
Alaca & Vrbik (UTM) 1-Shell Week 1 37 / 38
Job Control
A job is a program that the user has executed
When a program is running, hit ctrl+z to suspend it
jobs gives you the list of jobs, each with an associated job number
fg num brings job number num to the foreground and resumesexecution
Foreground job has control of the terminal, e.g., all user input iscaptured by that job
bg num resumes execution of job number num in the background
Background job does not control the terminal, but will be stopped ifthe terminal is closed
kill %num kills job number num
Any command can be started in the background by appending & tothe command (e.g., sleep 10&)
Alaca & Vrbik (UTM) 1-Shell Week 1 38 / 38