Post on 14-Dec-2015
Searching and Sorting
Why Use Data Files?
There are many cases where the input to the program may come from a data file.Using data files in your programs offer the following advantages.
Users do not have to input repetitive information.Several users and even programs can share common information.Programs are able to run on their own without waiting for a user to input anything.
Disadvantages of Data Files
The disadvantage to using data files is that it usually a little more involved to take input from a file, then directly from the user.
Searching and Sorting
The reason for this is that the data file may contain hundreds, or thousands Units of data. You must be able to search for the information you wish to find. In addition when writing to data files it would be preferable to have the file in some sort of order.
Searching/Sorting Commands
Unix comes with several command to facilitate searching and sorting including the following
grep(g)awksedsortcutuniqdiff
grep
grep is a command that will search a file for a certain “string” of information. When it finds a match it will show the whole line.
IE) grep ramadani /etc/passwdramadani:x:1711:100:,,,:/home_staff/ramadani:/
bin/bash
The password FileThe password file in Unix contains information about the users on the system. Every user on the system has an entry in the password fileThis file is consulted during login and to calculate file permissions.In most modern versions of Unix the password file doesn’t contain user passwords.
/etc/passwd file syntaxUSER:PASSWORD:UID:GID:COMMENT:HOME
DIR:SHELL
User- The user’s login namePassword- Where the password used to be (now in
shadow file)
UID/GID- The User’s ID number and Group ID number Comment-Stores address or other general info about
the userHOME DIR – Specifies the user’s home directorySHELL – Specifies the user’s shell
grep switches
grep –i RaMaDaNi /etc/passwd Case insensitive grep –v ramadani Search for everything but the
string (used to remove lines from files).
IE)grep –v –i test ~/data.txt >data.tempThis will remove all line(s) that contain the string test and re-direct the output to a temp file.
More Grep switches
There may be times where you will need to see what occurred above or below the line being searched for.
IE) grep –2 Jim ~/data.txt Will show the 2 lines above Jim and below Jim. Plus the line its self.
grep –c Jim Show how many lines contain the word Jim
diff
The diff command shows what's different between any 2 files. The diff command uses < > to indicate which file contains that line.
IE) diff file1 file2
<line in file1 but not file2>line in file2 but not file1
uniq
The uniq command is used to remove duplicate entries from a file. IE) uniq data.txt
daisyfredjohnFrank
data.txtdata.txtdaisyfreddaisyjohnFrank
The sort command
The sort command can be used to sort any file.
IE) sort /etc/passwd would put the file in order alphabetical order based on the first field.
Sorting by other fields
sort –t<file separator> +<file number> file
IE)sort –t: +2 /etc/passwd
sorts the passwd file based on UID (second field)
How sort number the fields
NAME:PASS:UID:GID: Fields 0 1 2 3 Field Number
sort <file><sorts by first field>sort +0 <sorts by first field>sort +1 <sorts by second field>
(g)awk
awk or gawk is more then just a simple comand. awk is a powerful programming language.Awk is a great prototyping languageStart with a few lines and keep adding until it does what you want
AWKA programming language for handling common data manipulation tasks with only a few lines of programAwk is a pattern action languageThe language looks a little like C but automatically handles input, field splitting, initialization, and memory management
Built-in string and number data typesNo variable type declarations
gawk general syntaxgawk ‘/pattern/ {output}’ file
pattern - is what is being searched for. output - what will the command output when the pattern is matched. file -the file being search
The quotes are the single quotes found next to the enter key.
Simple Output From AWK
If an action has no pattern, the action is performed for all input lines
gawk ‘{ print }’ filename gawk ‘{ print }’ filename will print all input lines on stdout
gawk ‘{ print $0 }’ filename will do the same thing
Printing specific fields
Multiple items can be printed on the same output line with a single print statement
gawk ‘ { print $1, $3 }’ fileThis will print the first and third fields in the file.commas are used in the print statement to indicate spaces.
Changing the field Separator
The default field separator in gawk is a space.
To change specify a different field simply use the field separator switch (-F)gawk –F: ‘{print $1,$7}’ /etc/passwd
would print the first and seventh fields (name and shell ) from the password file
Using gawk to search
By including a pattern in the gawk statement this will actually allow the gawk command to searchgawk ‘/root/ {print $1,$7}’ /etc/passwdThis will only print the login and shell of those lines that contain the string root
Consider the following text file
Joe,Smith,1234567fred,Sam,7654321Hank,Joe,9876543
Suppose you wanted only the people who’s last name are Joe. How would you structure a gawk command to accomplish that?
Solution
gawk –F, ‘$2~/joe/ {print }’ datafile
This gawk statement is read as followsUsing field separator of a comma search the second field for the string ‘joe’ and print the whole line using the datafile as input
Interactive exercise
Determine the commands that will accomplish the following;Sort the password file based on UID and save a copy of the file in you home directory called passwd.sorted