Linux System Fundamentals Lesson 3: Linux File Management Basics

Sadia Bashir

Contents Permissions

File permission Directory permission

Essential File Management Tasks Create/Move/Copy/Rename/Delete

Using Text File Editors Nano, Vim, Sublime, Emacs Working with Nano and Vim

More File operations stat, touch, file, wc, du, df, diff

Linux Wildcards & Pattern matching

Finding Files locate, find, whereis, which, whatis

I/O Redirection, Pipes and Filters

Using Common Text Manipulation Tools (tail, head, tr, tee)

Working with grep, egrep

Permissions Permissions are the rights to act on a file/directory

Owners assign permission on every file/directory

Permissions are important to create a secure environment

What they are : r : only open and view file/directory contents

w : modify file/directory contents

x : execute or run the file if it is a program or script

- : no read, write or execute permissions set on file/directory = : make it the only permission

Permissions are specified for three sets of people[identities]: owner [u] - a single person who owns the file. (typically the person who created the file but ownership may be

granted to some one else by certain users)

group [g] - every file belongs to a single group

others/all [o] [a] - everyone else who is not in the group or the owner

Note: Linux/UNIX OS has ability to multi-task and to have multiple users, who can have access to system simultaneously, a method to protect users from each other is required. This is where permissions come to play.

Format of setting Permissions

Octal Decimal Permission Representation

000 0(0+0+0) No Permission - - -

001 1(0+0+1) Execute - - x

010 2(0+2+0) Write - w -

011 3(0+2+1) Write + Execute - w x

100 4(4+0+0) Read r - -

101 5(4+0+1) Read + Execute r - x

110 6(4+2+0) Read + Write r w -

111 7(4+2+1) Read + Write + Execute r w x

drwxr-xr-x -rw-r--r--

Difference between a Directory and File Permissions: The first character identifies the file type : dash ( - ) indicates a normal file and d denotes it is a directory.

Working with Permissions

View Permissions – ls -l [path/to/directory/file] ls -l /home/juno/linuxtutorial/test.txt ls -ld /home/juno/linuxtutorial

Change permissions – chmod [permissions] [path] chmod : change file mode chmod has permission arguments that are made up of 3 components:

changing the permission for [ugoa] - user (or owner), group, others, all granting or revoking the permission - indicated with either a plus ( + ) or minus ( - ) Which permission are we setting? - read ( r ), write ( w ) or execute ( x )

Examples: chmod g+x test.txt chmod u-w test.txt chmod ugo+wx test.txt chmod 777 test.txt chmd 240 test.txt

Permissions for Directories

The same series of permissions may be used for directories but they have a slightly different behavior:

r - user has ability to read directory contents (i.e., do an ls)

w - user has ability to write into the directory (i.e., create files and directories)

x - user has ability to enter that directory (i.e., cd)

Essential File Management TasksCreating a new Directory using CLI

mkdir : command to creates directories as a superuser Syntax:

mkdir [option] [Directory_Name] sudo mkdir mydir (creates a new directory whose parent is the current directory) OR

mkdir [option] [/home/path/to/Directory_Name] sudo mkdir /home/juno/mydir

Options: -m : Creates the mydir directory, and set its permissions

sudo –m a=rwx mydir -p : mkdir will first create the parent directory /home/juno/a/b if it does not already

exists mkdir -p /home/juno/a/b/c

rmdir : Removes empty directories – sudo rmdir /home/juno/a

Creating a new File using CLI

touch : touch sample.txt

> sample.txt

cat > sample.txt

Naming [File/Directory] Conventions

A directory may have the same name as one of the items it contains

Different directories may contain files with same name

Some characters have special meanings to Unix hence it is best to avoid using these characters in file names:

/ \ " ' * | ! ? ~ $ < >

Unix is case-sensitive : so each of these is a unique file: myfile, Myfile, myFile, and MYFILE

Rename/copy/move/delete a File/Directory

RENAME : mv FILE_NAME.txt/DIRECTORY [oldname] [newname] FILE : mv sample.txt newsample.txt DIRECTORY : mv sample newsample

COPY : cp FILE_NAME.txt/DIRECTORY [source] [destination] FILE : cp sample.txt /home/juno/sample /home/juno/practice DIRECTORY : cp testd/home/juno/sample /home/juno/practice

MOVE : mv FILE_NAME.txt/DIRECTORY [source] [destination] FILE : mv sample.txt /home/juno/sample /home/juno/practice DIRECTORY : mv testd/home/juno/sample /home/juno/practice

DELETE : rm FILE_NAME.txt - rm -rf DIRECTORY FILE : rm sample.txt DIRECTORY : rm -rf test - -r : recursively Options : [-f: force, -i: iteratively ask before removing each sub-directory]

1. Using Text File Editors - Nano Nano:

-c : open file showing number of lines : nano -c test.txt File Controls

nano FILE.txt : Open or create the file Ctrl-o Y : Enter Save changes Ctrl-r Alt-f : Open a new file with a new buffer Alt-> : Switch to the next file buffer Alt-< : Switch to the previous file buffer Ctrl-x : Quit nano

Navigating through file contents Ctrl-a : Move to the beginning of the current line Ctrl-e : Move to the end of the current line Ctrl-v : Move down one page Ctrl-y : Move up one page Alt-\ : Go to the beginning of the file Alt-/ : Go to the end of the file Alt-g : Go to a target line number Alt-a Alt-} : Select a block and indent the block Alt-a Alt-{ : Select a block and outdent the block

1. Using Text File Editors - Nano (contd…)

Copy and Paste Alt-a : To select a block for copy or cut operation, do Alt-a again to unselect

Alt-a Alt-^ : Copy a highlighted block to the clipboard

Alt-a Ctrl-k : Cut a highlighted block to the clipboard

Ctrl-k : Cut from the current cursor position to the end of the current line

Ctrl-u : Paste the contents from the clipboard at the current cursor position

Search and Replace Ctrl-w : Search for a target string

Alt-w : Repeat the last search

Alt-r : Search and replace

2. Using Text File Editors - Vim Vim:

CLI Options : vim -R text.txt -R : Read-only mode (like "view") -M : Modifications in text not allowed -p and -o : open new tabs or windows at run time + : start at end of file +<lnum> : (+5) start at line <lnum>

File Control vim test.txt : Open the file specified with vim in command window vim File1.txt File2.txt : Open two files simultaneously ESC :next : go to the next file ESC :previous : go to the previous file ESC :tabe : file3.txt - Open a new file while vim is already opened with one file, if the file doesn’t exist, it will create it ESC :tabn : or ESC :tabp : go to the next file or previous file accordingly ESC :wqa : to save works in all tabs and quit vim ESC :bd : to close the file without quitting the vim ESC :w backup.txt : save the current file to a new file with the file name “backup.txt” ESC :w : Save changes ESC :wq: Save the changes and exit ESC :q! : Quit without save

Edit mode or Insert mode ESC i : Enter insert mode and start editing

2. Using Text File Editors - Vim (contd…) Line Control:

ESC 0 or ESC Home : move the cursor to the beginning of the current line

ESC $ or ESC End : move the cursor to the end of the current line

ESC :34 : go to line 34 ESC gg : go to the beginning of file

ESC G : go to the end of file

ESC ( : jump backward one sentence ESC ) : jump forward one sentence

ESC dd : delete the current line

Cursor Control h : Move Left l : Move Right j : Move Down k : Move Up

Screen Control ESC zt, zz, zb : move cursor to the Top, Middle, and Bottom of the screen respectively ESC ctrl-f : Scroll down one screen page ESC ctrl-b : Scroll up one screen page

Vim Undo and Redo ESC u : Undo the changes ESC Ctrl-r : Redo the changes

2. Using Text File Editors - Vim (contd…) Search and Replace

ESC /sometext and enter : Search for the word specified. n : To find the next occurrence N : the previous occurrence

ESC :s/foo/bar/g : search and replace “foo” with “bar” in the current line ESC :%s/foo/bar/gc : search and prompt to replace “foo” with “bar” ESC :%s/foo/bar/g : search and replaces all “foo” with bar in the file ESC :%s/[0-9]/a/g : find all digits and replaces it with “a”.

Copy and Paste ESC v : move the cursor to select the text to cut or copy Press d : to cut OR Press y : to copy Press p : Move cursor to the point to paste the text and Press p to paste it

Reveal hidden Characters :set list :set nolist

Help ESC :help : open help page ESC :q : exit help page

More File Operations

Viewing file timestamps : using touch command Different types of timestamps are associated with files e.g., access timestamp, modification

timestamp etc. state test.txt touch : command is used to change timestamps associated with a file :

touch -d "2013-01-10 10:00:07" -a test.txt : -d : use specified time instead of current time -a : change only the access time

touch -d "next saturday" -m test.txt - -m : changes file modification time touch -d “5 hours ago" -a test.txt

Determine file types : using file command file test.txt – view type of a specified file file * : view type of all files in current directory

Print the number of new lines, words, and bytes in files: using wc command wc test.txt OR wc -l -w -c test.txt where -l, -w, and -c represent line words and byte counts

respectively

More File Operations (contd….)

Display disk usage of files and directories: using du command du -h test.txt : -h : CLI option produces output in human readable format

du -s test.txt : -s : display the combined size of a directory and its subdirectories

du -S somedirectory : -S : display separate sizes

df -h test.txt : display the amount of disk space available on the file system containing a specific file or directory

df -h : shows disk usage for all the file systems

Compare two files: using diff command diff : The command requires two filenames as arguments

diff FILE1 FILE2 %< lines from FILE1 %> lines from FILE2 %= lines common to FILE1 and FILE2 c : changed, d : delete, a : add

The command examines both files and produces output in a particular format to let the user know what changes are required to be made for the files to match.

Linux Wildcards & Pattern matching : *?[ ] A wildcard is a character that can be used as a substitute for any class of

characters to create a “pattern” to be used in a search in order to greatly increase the flexibility and efficiency of search

Star/asterisk (*) Wildcard : It represents zero characters, all single characters or any string, in other words, it represents everything

ls *.txt : list down all files having .txt extension ls n* : list all files starting with lower case letter n rm *enw* : delete all files in a current directory having e n or w as part of their name file * : lists type of all file system objects in current directory

wildcards can be used at any point in the path ls -lh /home/*/.bash_history : list .bash_history file in every users home directory

Question-mark (?) Wildcard : represents exactly a single character ls *.??? file ?.* ls ?i* mv public_html/*.??g public_html/images/ : move all files of type either jpg/png into another directory

Linux Wildcards & Pattern matching (contd...)

Square Brackets Wildcard [] : represents a range of characters file *[xyz]* : lists all objects in the current directory that have an x, y and/or z in them

ls *.[xyz]* : list all files that have an extension that begins with x, y or z

file [a-f]* : hyphen indicates a range inclusive of these two characters - this command would provide information about all of the objects in the current directory that begin with any letter from a through f

file *[0-9]* : provide information about every object in the current directory whose name includes at least one numeral

file [a-cx-z]* : specifies multiple sets of ranges - this command would return information about all objects whose name begin with the first three or the final three lower case letters

ls jones[0-9][0-9][0-9] : succession of square bracket wildcards – this command would display all filenames in the current directory that consist of jones followed by a three-digit number

ls [^a-k]* : caret ( ^ ) reverses the range, which means look for any character which is not one of the range

Finding Files - Locate

Locate : the quickest way to find the location of files/directories locate [options] name(s)

When used without any options, locate displays every absolute pathname for which the user has access permission that contains any of the names of files and/or directories that are provided to it as arguments

locate sysctl.conf, locate updatedb.conf locate mlocate.db

Options : -o : display all the output in one line

-c : get the count of number of matching entries

-i : perform case insensitive search

-e : check if the file is physically present in the system

-l <num> : restrict search results

Note: The ‘locate’ command is fast as it searches for file paths in a database named “mlocate.db” rather than searching from the entire hard drive. This database is automatically updated on a regular basis by cron, which runs in background and updates this database file at a pre-configured frequency. Hence “updatedb” command is used to manually update mlocate.db file in order to get latest and reliable results.

Finding Files - Find Find : The find command searches out live file-system for files that match the search criteria

The find command syntax is: find /where/to/look/up criteria action OR

find /dir/path/look/up criteria action OR

find /dir/path/look/up -name "dir-name-here“ OR find /dir/path/look/up -name "pattern“ OR

find /dir/path/look/up -name "dir-name-here" -print OR

find /dir/path/look/up -name "dir-name-here“ OR find / -name "dir-name-here“

Searching through the root file system: sudo find / -name "apt“ Getting a detailed list of files/directories: sudo find / -name "apt" -ls Listing directories only: find / -type d -name "apt" -ls Performing a case insensitive search: find / -type d -iname "apt“ Finding a specified directory: find / -type d -name "project.images" Finding with wildcards: find . -name pupp\* find / -maxdepth 2 -name passwd: Find file under root and one level down find / -mindepth 3 -maxdepth 5 -name passwd: Find file between sub-directory level 2 and 4

Finding Files - Find (contd…) Find file based on file permissions irrespective of other permissions

find . -perm -g=r -type f -exec ls -l {} \; : Find files which have read permission to group find . -perm 040 -type f -exec ls -l {} \; : Find files which have read permission only to group [ search by octal ] find . -perm g=r -type f -exec ls -l {} \; : Find files which have read permission only to group

Find empty files find ~ -empty : find all empty files in home directory find . -maxdepth 1 –empty : List all the empty files only in home directory find . -maxdepth 1 -empty -not -name “\.*“ : List only the non-hidden empty files only in the current directory

Find Files Based on file-type find . -type d : Find all directories find . -type f : Find normal files only find . -type f -name ‘.*’ : Find hidden files find -type d -name ‘.*’ : Find hidden directories

Find Files by Size : - means less than the give size, + means more than the given size, and no symbol means exact given size

find ~ -size +100M find ~ -size -100M find ~ -size 100M

Finding Files - whereis, which, whatis whereis : command to locate the binary, source, and manual page files for a

command Usage: whereis [-options] whereis -b whereis : Locate binaries for a command whereis -m whereis : Locate man pages for a command whereis -B /bin -f chmod : Locate binaries in a specified path whereis -M /usr/share/man/man1 -f open : Locate man pages with limited scope

which : simple command to locate executables in the system. It allows user to pass several command names as arguments to get their paths in the system, searches the path of executable in system paths set in $PATH environment variable Usage: which [-option]

which ls open chmod grep which -a echo : display all paths

Whatis : displays brief information for command that matches the argument passed to it

whatis [-options] : whatis ls, whatis cat, whatis cd etc.

I/O Redirection >, <, >>, << Redirecting the source of an input to a command and the destination of the

output it produces to be the files

Input and output in the Linux environment is distributed across three streams: stdin : Standard Input - input to a program - input stream typically carries data from a user to a program

usually through keyboard

stdout : Standard Output - output from a program – output stream writes data that is generated by a program. When the standard output stream is not redirected, it will output text to the terminal

stderr : Standard Error - error output from a program - writes the errors generated by a program that has failed at some point in its execution. Default destination for this stream is the terminal display

Stream Redirection: these commands write standard output to a file Overwrite : Commands with a single bracket overwrite the destination's existing contents

> - standard output

< - standard input 2> - standard error

Append : Commands with a double bracket append to the destination's existing contents >> - standard output << - standard input

2>> - standard error

I/O Redirection (contd…) File Descriptors

standard streams have their default file descriptors 0 - stdin (Standard Input)

1 - stdout (Standard Output)

2 - stderr (Standard Error)

Examples: ls -lah . > lsoutput.txt

file >> lsoutput.txt

ls -la /nothing 2> error.txt : redirect stderr

ls -lah . /nothing &> all_output_v2.txt : to redirect both 1 and 2

ls -lah . /nothing &>> all_output_v2.txt : Appending both stdout (1) and stderr (2)

uniq < fruits.txt > uniq_fruits.txt : combining redirects

uniq < fruits.txt | sort -r > rev_sort.txt

sort < fruits.txt

Pipes - Joining programs together | With pipes, standard output of one command is fed into the standard input of another

ls | less : This takes the output of ls and pipes it to the less program. Less displays the data sent to it one line at a time.

Filters: One class of programs which can be used with pipes is called “Filters”. Filters take standard input, perform an operation upon it and send the results to standard output Some of the main CLI "filters“

cat : concatenate files and print on STDOUT less : displays data one line at a time head : output the first part of files tail : output the last part of files tee : read from STDIN and write to STDOUT and files wc : print the number of bytes, words, and lines in files grep : print lines matching a pattern sort : sort lines of text files uniq : remove duplicate lines from a sorted file cut : remove sections from each line of files tr : translates characters. Can be used to perform tasks such as upper/lowercase conversions or changing line

termination characters from one type to another

Using Common Text Manipulation Tools tail : output last few lines of a file

tail -n2 grocery.list : -n : represents number of lines tail -c12 grocery.list : -c : represents number of characters

head : output first few lines of a file head -n2 grocery.list : -n : represents number of lines head -c12 grocery.list : -c : represents number of characters More Usage:

to watch log files to track output of running processes to see if there are issues to note when a process finishes - The -f (tail –f) option output appended data even after the end-of-file

marker is reached, and continue displaying output when the stream grows

tee : reads from standard input, and writes to standard output and to files -duplicates its input, routing it to multiple outputs at once

ls | tee file1.txt file2.txt file3.txt ls | tee -a file.txt : append to the file tee command is used to store the output of a command to a file and redirect the same

output as an input to another command ls ~/ | tee pipe1.txt | grep ^s | tee pipe2.txt | sort -r

Using Common Text Manipulation Tools(contd….)

tr : The tr command is used to translate characters from stdin, displaying them on stdout. In its general form, tr takes two sets of characters, and replaces characters from the first set with characters from the second set

tr pre-defined character classes: alnum - alphanumeric characters alpha - alphabetic characters blank - whitespace characters cntrl - control characters digit - numeric characters graph - graphic characters lower - lower-case alphabetic characters print - printable characters punct - punctuation characters space - space characters upper - upper-case characters xdigit - hexadecimal characters

Examples: echo "Who is the standard text editor?" |tr [:lower:] [:upper:] - Upper-case a string echo 'ed, of course!' |tr -d aeiou - delete characters from a string echo 'The ed utility is the standard text editor.' |tr -s astu ' ‘ - translate characters to a space echo 'extra spaces – 5’ | tr -s [:blank:] - -s option flag is used to suppresses extra white space in a sting

Working with grep, egrep

grep : print lines matching a pattern grep "literal_string" filename : search for the given string in a single file

grep "string" FILE_PATTERN : search for the given string in multiple files

grep -i "string" FILE : case insensitive search

grep -r “paul" * : searching in all files recursively

grep -v "go" filename : invert match

grep -c "pattern" filename : counting number of matches

grep -n "go" filename : show line number while displaying the output

grep -l this *.txt : display only file names which match the given pattern

egrep: search for a pattern using extended regular expressions egrep "support|help|windows" myfile.txt

egrep -c '^begin|end$' myfile.txt