bash_day2

8/13/2019 bash_day2

1/21

Variable Substitution

# ! / b i n / b a s h# e xa mp le s ho wi ng th e us e of d ef au lt v ar ia bl e v al ue s

echo

{ S T R : - " H e l l o W o r l d ! " } " ( D e f a u l t i f N u l l ) "

echo

{ S T R - " H e l l o W o r l d ! " } " ( D e f a u l t i f n o t s e t ) "

read STR

echo { S T R : - " H e l l o W o r l d ! " } " ( D e f a u l t i f N u l l ) "echo

{ S T R - " H e l l o W o r l d ! " } " ( D e f a u l t i f n o t s e t ) "

${STR:-abc}Use default abc if null ${STR-abc}Use default abc if not set

${STR=abc}or ${STR:=abc}set STR to abc ${STR?}or ${STR:?}print error

# ! / b i n / b a s h

:

{1? " U s a g e :

0 A R G U M E N T " }

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, October 07 - 18, 2013 32

8/13/2019 bash_day2

2/21

Variable Substitution - String Chopping

# ! / b i n / b a s h

# e xa mp le s ho wi ng th e us e of s tr in g c ho pp in g

STR= " T h i s i s a S t r i n g "

echo { S T R : 0 : 1 0 } c h o p p e d { S T R : 1 0 }

${STR:x}All characters starting at character x

${STR:x:y}All characters starting at x up to (not including) y

${STR#pattern}delete pattern (smallest match from start)

${STR##pattern}delete pattern (largest match from start) ${STR%pattern}delete pattern (smallest match from end)

${STR%%pattern}delete pattern (largest match from end)



8/13/2019 bash_day2

3/21

Partial List of Commands/Variables/. . . so far

man mkdir bla $?$STR continue rmdir clear#! shell builtin ${STR:3:8} break 2let emacs ls $#$3 touch echo $STR ${#a}

$a -le $b chgrp if a = 3mv a=$((3+5)) expr test$$ ${STR%.txt} cp pwdscript suffix cd echo forchmod $* $PS3 $RANDOMesac $@ u+x${STR:-x} echo $STR infoapropos rm viread - - help a=$[3+5] shift



8/13/2019 bash_day2

4/21

Part III

Shell Config



8/13/2019 bash_day2

5/21

Config of PS1-PS4 (Prompt Statement Variables)

\dcurrent date

\hhostname \Hfully qualified domain name \lname of the terminal \nnewline \sname of the shell

\ttime 24-hour-format (try\T, \@) \uusername \wcurrent working directory (try \W) ...

PS1= " \ u , \ d , \ t , \ W / : "

PS1 - Default interaction prompt PS2 - Continuation interactive prompt PS3 - Prompt used by select inside shell script PS4 - Used by set -x to prefix tracing output



8/13/2019 bash_day2

6/21

Special Directories

.current directory ..parent directory

~own home directory

~userhome directory of user

~-previous directory

cd / u s r / local /bin

pwd

cd ~

pwd

cd ~ -pwd

cd ..

pwd



8/13/2019 bash_day2

7/21

Aliases and Variables

alias - short form for any command

alias

alias ll =

ls -l

environment variables Variables are not only within scripts, but also in the shell

By setting a variable, it is present in the current script/shell By exporting it (export VARIABLE=value), it is also present at all

child processes (NOT at the parent) Many environment variables are already set by default

P A T H = / h o m e / n e c k e l / b i n / :

P A T H ; echo

PATH

env

local variables setshows all local and global variables and functions unsetdeletes a variable



8/13/2019 bash_day2

8/21

Bash config

profile and rc

especially aliases are used in every session

should not be defined each time

You might even run arbitrary code on login or logout

e.g. clean up (kill jobs) on logout

Profile is used for complete session

bashrc is used for a single terminal

Files

/etc/profile

/etc/bash.bashrc

HOME/.bash profile

HOME/.bashrc

HOME/.bash logout



8/13/2019 bash_day2

9/21

Pipes

stdin/stdout: Standard input/output for programs

Most Linux programs read from stdin and write to stdout

Pipes are used to redirect input and output

cmd1 | cmd2connect the output of cmd1 with the input of cmd2

cmd > fileredirect the output of cmd to file

cmd 2> fileredirect stderr of cmd to file

cmd > file 2>&1redirect stdout to file and stderr to stdout

cmd &> fileredirect all output to file

cmd >> fileredirect the output of cmd and append it to file

cmd < fileuse the content of file as stdin for cmd cmd

8/13/2019 bash_day2

10/21

Wildcards

Wildcards are expanded by the shell

* zero or more characters

? exactly one character

[abcd] one of the characters a-d [a-d] same

[!a-d] any other character

{first,second} either firstor second

Similar pattern exist in other contexts as well (compare regularexpressions), but always a bit different. . . :-(



8/13/2019 bash_day2

11/21

Part IV

Bash Advanced: Regular Expressions and

More



8/13/2019 bash_day2

12/21

Regular Expressions???

The set of regular languages over an alphabet and the

corresponding regular expressions are defined recursively as follows: The empty language is a regular language, and the

corresponding regular expression is.

The empty string{} is a regular language, and thecorresponding regular expression is.

For each a in, the singleton language{ a } is a regularlanguage, and the corresponding regular expression is a.

If A and B are regular languages, and r1 and r2 are thecorresponding regular expressions,

Then A U B (union) is a regular language, and the

corresponding regular expression is (r1+r2) AB (concatenation) is a regular language, and the

corresponding regular expression is (r1r2) A* (Kleene star) is a regular language, and the

corresponding regular expression is (r1*)



8/13/2019 bash_day2

13/21

Regular Expressions!!!

What does all this mean to you, as a user? Absolutely nothing. As auser, you dont care if its regular, nonregular, unregular, irregular, orincontinent. So long as you know what you can expect from it, youknow all you need to care about.

Jeffrey Friedl, author of Mastering Regular Expressions

Did you ever. . .

. . . search for a character or string in a text file?

. . . use tab for auto-completion?

. . . use the * in a terminal for selecting a group of files?

. . .

Then youve already somehow used regular expressions.



8/13/2019 bash_day2

14/21

Regular Expressions

When searching for a string, exactly the given character

sequence is searched regular expressions get powerful as a tool to find patterns

Additionally to ordinary characters, which stand for themselvesspecial characters are used which are interpreted in a specialway.

The exact syntax for special characters differs between differentimplementations

Example to identify an email address:

[^@]\+@.\+\.[^.]\+

regular expressions usually look very cryptic! But imagine you would have to write a normal program to identifyan email address!

Regular expressions can be used to find/replace groups ofstrings which can be described by a pattern



8/13/2019 bash_day2

15/21

Special Characters for Regular Expressions

char A character maching itself

* matches zero or more occurences (greedy!) of the previousexpression

\+ matches one or more occurences (GNU extension) \? matches zero or one occurence (GNU extension) \{i\}matches i occurences

\{i,\}matches i or more occurences \{i,j\}matches i to j occurences . matches an arbitrary character matches the beginning of a string $ matches the end of a string

[list]matches a single character from the list ([ list]any characternot in the list)

\n matches newline  defines a group, reuse via\1 (1st group),\2 (2nd group) \|allows for a logical OR



8/13/2019 bash_day2

16/21

Character Classes

[:digit:]0 to 9 (alternative: [0-9]).

[:alnum:]alphanumeric character 0-9 or A-Z or a-z.

[:alpha:]character A-Z or a-z.

[:xdigit:]Hexadecimal notation 0-9, A-F, a-f. [:punct:]Punctuation symbols, e.g. . , ? ! ; : # $ % & ( )

[:print:]Any printable character.

[:space:]whitespace (space, tab, ...).

[:upper:]uppercase character A-Z (alternative: [A-Z]).

[:lower:]lowercase character a-z (alternative: [a-z]).



8/13/2019 bash_day2

17/21

programs using regular expressions

grepprint lines matching a pattern

grep -icase insensitive

grep -vinvert match

grep -nadditionally print line number

trtranslate: echo "lower/UPPER"| tr "A-Z""a-z"

sedstream editor

awkpattern scanning and processing language

findsearch for files in a directory hierarchy

...

most editors can handle regular expressions



8/13/2019 bash_day2

18/21

sed - Stream Editor

works (as most bash programs) on a stream of data

data is processed linewise no way of going back a line (efficient) apply some action on selected lines (using addresses to select)

Addresses

address: line number or regular expression

zero, one or two (comma-separated) addresses can be used

zero addresses: all lines are processed

one address: all lines matching the address are processed

two addresses: match from first to second address

nmatch only line number n n~stepstarting from line n, match every stepth line

/pattern/lines matching the given regular expression

i,jall lines from i to j (including both)

match the last line



8/13/2019 bash_day2

19/21

sed - Editing Commands

parameters -nsupress output (per default, all lines are printed)

-eediting command follows in command line

-fscript to be read from file

commands

pprint line

=print line number

i\textand a\textinsert text before/after matched line

c\textreplace matched line by text

sreplace pattern, e.g. sed -e s/search/replace/g (note: g at end optional, but replaces globally!)

wwrite line into given file, sed -n /patt/ w out.txt demo.txt



8/13/2019 bash_day2

20/21

Operating on Files

less is more than more

moredisplays the content of text files pagewise

only downward-scrolling is possible

lessis an extended and more flexible more

outputting files

catoutputs the content of a file to stdout (try tac) catconcatenates several files

headoutputs the first part of files

tailoutputs the last part of files

for i in

se q 30

; do e ch o

i >> temp . txt ; sleep 1; \

done &

t ai l - f t em p . tx t



8/13/2019 bash_day2

21/21

Operating on Files (2)

filefind out more about file type

cmpCompare files byte by byte

diffCompare files line by line (graphical: kdiff3)

patchapply patch (created by diff) to a file

tarpack and compress files (try tar -xzfand tar -czf)

sortsort lines of text files (and write to stdout)

uniqreport or omit repeated lines

wcprint newline, word, and byte counts for each file

cutremove sections from each line of files

echo " 1 ; 2 ; 3 ; 4 " | cut -b 4 -5

echo " 1 ; 2 ; 3 ; 4 " | cut -d " ; " -f 3

cut -d

- f1 , 2 / e t c / m ta b



bash_day2

Documents

Transcript of bash_day2