Map grep sort

map, grep, sort – slide 1

Streamlining and simplifying your Perl code using

Map, Grep, and Sort

Daina Pettit

dpettit@bluehost.com

daina@xmission.com

“Perl culture” sometimes gets shortened to “Perl cult”.*

Larry Wall

*Wall, Larry, Perl, the first postmodern computer language, Linux World [Conference], March 3, 1999

Overview● What are map, grep, & sort and why should I care?● map details● grep details● sort details● Combining map, grep, & sort● Advanced combinations

● Schwartzian Transform● Orcish Maneuver● Guttman-Rosler Transform● Alternatives

What are they?

map, grep, & sort are iterator functions that operate on lists or arrays.

1. map performs action on each element.

2. grep tests each element.

3. sort orders the elements.

General Form

All have similar forms.

@array = map { exp } @list;@array = grep { exp } @list;@array = sort { exp } @list;

@array = map exp, @list;@array = grep exp, @list;@array = sort @list;

General Form—code blocks

Damian Conway in Perl Best Practices* recommends:

“Always use a block with a map and grep”

This is a syntactic aid suggestion to help you prevent yourself from making an error with grouping arguments. Block enclosures actually incur more overhead. Not much, but some.

*Conway, Damian, Perl Best Practices, O'Reilly Media, Sebastopol, CA, 2005, pp 169-170.

@array = map { exp } @list;@array = grep { exp } @list;

What is map?

Map is essentially a loop that processes a list, much like a foreach loop.

foreach $line ( @lines ) {$line = uc $line;

What is map?

@lines = map uc, @lines;

What is map?

@lines = map { uc } @lines;

Aside—foreach inside-out

Alternate single line foreach is concise as map, and is slightly faster than map, but more cryptic.

$_ = uc foreach @lines;

foreach ( @lines ) { $_ = uc;}

What is the best way to use map?

● map is best for creating new lists. ● foreach is best for transforming a list.

@words = map { split } @lines;

foreach ( @lines ) {$_ = uc;

Dumping out a hash alternatives

foreach ( sort keys %h ) { print "$_ => $h{$_}\n";}

map { print "$_ => $h{$_}\n" } sort keys %h;

print "$_ => $h{$_}\n" foreach sort keys %h;

map {} is list context

Damian Conway in Perl Even-Better Practices* recommends:

"Use explicitly scalar map expressions"

*Thoughtstream Pty Ltd, 2013 pp 10-11

@dates = map { localtime $_ # Wrong! } @epoch_times;

@dates = map { scalar localtime $_ } @epoch_times;

@words = map { scalar split # Wrong!} @lines;

@words = map { split} @lines;

map {} confusion

How does perl know that { 6 } is a code block or a partial hash? Use +{ 6 }. + is required or you will get a syntax error.

map +{ 6 }, @stuff; # hashmap { 6 } @stuff; # code block

Using map in void context● Frowned upon. ● Incurs extra overhead.

Creating a hash in map

map { $age_of{$_} = M } @files;

foreach ( @files ) { $age_of{$_} = M;}

$age_of{$_} = M for @files;

Skipping in map● Drop an item using an empty list.● Do NOT use an explicit return.

@ones = map { $_ < 10 ? $_ : (); } @numbers;

What is grep?● Similar to Unix command-line utility grep● Given a list, grep returns only certain items

@ones = map { $_ < 10 ? $_ : (); } @numbers;

What is grep?● Similar to Unix command-line utility grep● Given a list, grep returns only certain items

@ones = map { $_ < 10 ? $_ : (); } @numbers;

@ones = grep { $_ < 10 } @numbers;

Boolean Scalar Context● Anywhere in perl where a true/false is expected

—if, while, and, or, not, &&, ||, !, etc.● Evaluation results in 0, “0”, 0.0, “”, or undef then

it is false. Everything else is true.

if ( 0 ) {} # Falseif ( 400 ) {} # Trueif ( 1 ) {} # Trueif ( "false" ) {} # True!if ( "00" ) {} # True!undef $x;if ( $x ) {} # False

Examples of grep● Expression can be any valid perl expression.● Expression is in scalar boolean context.

@ones = grep { $_ < 10 } @numbers;

@dirs = grep { d } @files;

@no_dup = grep { ! $h{$_}++ } @old;

@errors = grep { /error/i } @log;

@true = grep { $_ } @all;

Sorting Basics

Sort can be called in three ways:

1. With no comparison directives

2. With a subroutine that returns comparison directives

3. With a code block (an anonymous subroutine) that returns comparison directives

@sorted = sort @unsorted;@sorted = sort sub @unsorted;@sorted = sort { exp } @unsorted;

Sorting Basics

Sort requires the comparison directives value of -1, 0, or 1 to tell whether any two elements, $a and $b, are in order (-1), the same (0), or out of order (1).

cmp and <=> conveniently provide this for string or numeric comparisons, respectively.

We don't have to use cmp and <=>. We just have to return -1, 0, or 1.

$a <=> $b

Sorting Basics

Basic ASCII-betical sort:

Basic numeric sort:

@sorted = sort @list;

@sorted = sort { $a <=> $b } @list;

Sorting Basics

Basic ASCII-betical sort:

Basic numeric sort:

@sorted = sort { $a cmp $b } @list;

@sorted = sort { $a <=> $b } @list;

Sorting Basics--reverse

Reverse ASCII-betical sort:

Reverse numeric sort:

@sorted = sort { $b cmp $a } @list;

@sorted = sort { $b <=> $a } @list;

Sorting Basics--reverse

Or just use reverse function:

Reverse numeric sort:

@sorted = reverse sort @list;

@sorted = reverse sort { $a <=> $b } @list;

Sorting Basics--subroutine

Using a subroutine instead of a code block

You can also use anonymous subroutines.

These subroutines cannot be recursive!

sub compare {uc ( $a ) cmp uc ( $b );

$comp = \&compare;

@sorted = sort $comp @list;

Complicated Sorting

You can sort on anything you can get to through $a and $b.

@sorted = sort { @array_a = split / /, $a; @array_b = split / /, $b; $array_a[5] cmp $array_b[5];

} @lines;

Complicated Sorting

Sorting hash keys

Sorting hash keys by value

@sorted_keys = sort keys %hash;

@sorted_keys = sort { $hash{$a} cmp $hash{$b} } keys %hash;

Complicated Sorting

We can sorting with multiple keys such as sort by year, then by month, then by day even if the data is mm-dd-yyyy.

@sorted_dates = sort { ( $ma, $da, $ya ) = split //, $a; ( $mb, $db, $yb ) = split //, $b; $ya<=>$yb || $ma<=>$mb || $da<=>$db;} @dates;

Complicated Sorting

We don't have to always use the comparison operators. We can make up our own unique order.

@order = sort { return 1 if $a eq 'King' && $b ne 'King'; return 1 if $a ne 'King' && $b eq 'King';

return 0; } @cards; # King first, # the rest doesn't matter.

Combinations

Since map, grep, and sort both take and return lists, you can chain them together.

@pics = map { lc } grep { /\.jpe?g$/i } sort @list;

Optimizing sort

Given a list of files, sort by the age of the files.

chomp ( @files = `ls 1` );

“file1” “file7” “a.out” “x.pl” “5.dat”

Optimizing sort

Sorts by name, but not by age.

@sorted = sort @files; # by name

Optimizing sort

Sorts by date, but slow for large data sets.

M is called twice every time sort compares!

@sorted = sort { M $a <=> M $b } @files;

Optimizing sort

We want to call M once for each file, save that and use that each time sort needs to compare.

Map will do this for us!

@order = map { [ $_, M ] } @files;

1.2 2.9 3.1 1.1 2.9

Optimizing sort

Then we want to sort based on just the date part.

But now we need to get rid of the date part.

@order = sort { $a>[1] <=> $b>[1] } map { [ $_, M ] } @files;

“x.pl” “file1” “5.dat” “file7” “a.out”

1.1 1.2 2.9 2.9 3.1

Optimizing sortNow use map to extract just element 0 and we are back to the original list and sorted by date.

This is known as the Schwartzian Transform.**Perl idiom named for Randal Schwartz, author of Learning Perl, coined by Tom Christiansen.

@order = map { $_>[0] } sort { $a>[1] <=> $b>[1] } map { [ $_, M ] } @files;

“x.pl” “file1” “5.dat” “file7” “a.out”

Optimizing sort

Key points to remember for ST:● map sort map idiom

Optimizing sort

Key points to remember for ST:● map sort map idiom● Use proper comparison

Optimizing sort

Key points to remember for ST:● map sort map idiom● Use proper comparison● Extract value to compare

Optimizing sort

Key points to remember for ST:● map sort map idiom● Use proper comparison● Extract value to compare● Everything else stays the same.

Optimizing sort—Orcish Maneuver*

Uses “or” cache (in a hash) to remember values already computed: ||=

● Simpler than ST● Almost as fast as ST● Faster if list contains duplicates

*Term coined by Joseph Hall in Effective Perl Programming, Addison-Wesley Professional, Boston, MA, 1998.

@order = sort { ( $cache{$a} ||= M $a ) <=> ( $cache{$b} ||= M $b ) } @files;

Optimizing sort—Orcish Maneuver

Key points to remember for OM:● Only sort

Key points to remember for OM:● Only sort● Compute comparison data

Key points to remember for OM:● Only sort● Compute comparison data● Use proper comparison operator

Everything else stays the same.

Optimizing sort—Guttman-Rosler Transform*

This is a tweak on ST. Takes advantage of substr and sprintf being faster than array manipulation. Also uses default string sort which is slightly faster.

*A Fresh Look at Efficient Perl Sorting, Uri Guttman and Larry Rosler, approx. 1999.

@order = map { substr $_, 10 } sort map { m#(\d{4})/(\d+)/(\d+)#; sprintf "%d%02d%02d%s", $1, $2, $3, $_ } @dates;

Optimizing sort—Guttman-Rosler Transform

Faster than ST

Harder to code and less readable

Not suitable for all sorts

@order = map { substr $_, 10 } sort map { m#(\d{4})/(\d+)/(\d+)#; sprintf "%d%02d%02d%s", $1, $2, $3, $_ } @dates;

Further List & Sort Options

List::Util

shuffle, reduce, any, first, max, min, ...

List::MoreUtils

uniq, natatime, ...

Sort::Key

May be faster than ST or GRT

Sort::Naturally

Automatically sorts numeric when appropriate

Sort::Maker

Internally uses OM, ST, or GRT.

Comments?

Questions?

Resources

http://www.perlmonks.org

http://www.cpan.org

http://www.hidemail.de/blog/perl_tutor.shtml

http://perldoc.perl.org/

http://www.stonehenge.com/writing.html

For profiling:

perldoc Devel::NYTProf

perldoc Benchmark

Map grep sort

Software

Transcript of Map grep sort

Regular Expressions grep and egrep

Some announcements - eecs.yorku.ca file−Shell functionalities (common) −Bourn shell and shell script. 2 Last time Overview of UNIX ... Basic cat, rmdir .. Advanced grep, sort,

Doctor Demo - wiki.opnfv.orgJul 30, 2015 · 30 July 2015 Doctor demo, Santa Clara 37 tipica@d718:~$ ps aux | grep instance-00000001 | grep -v grep | sed -e 's/-smbios.*$//' libvirt+

GREP Q1 2013 Newsletter

1.1.2.8 – Intermediate Perl 2/1/20161.1.2.8.3 - Intermediate Perl - sort/grep/map 1 1.1.2.8.3 Intermediate Perl – Session 3 · map · transforming data ·

Linux intro 3 grep + Unix piping

Sort Value and Range Map

1.2.2.1.1.p1mkweb.bcgsc.ca/perlworkshop/data/courses/1.2.2.1/01/pdf/1.2.2.1.1.… · 6/23/2005 1.1.2.8.3 - Intermediate Perl - sort/grep/map 8 1.2.2.1 – sort/grep/map in Perl Map

Manual Grep 2 Grep

Lec 2 Grep Sed Intro

Find Grep and Vi in Aix

Case Study #2 - DCClblopes/aulas/ic/T10.pdf$ diff teste1.c teste2.c to ﬁnd text patterns within ﬁles we can use the “grep” and “ egrep” commands $ grep int test.c $ grep

Map/Reduce - Harvard Universitysites.fas.harvard.edu/~cs165/notes/mapreduce-full.pdf · • Should map/reduce and databases be in the same discussion? ... •Distributed Grep ...

GREP en InDesign

Map/Reduce on Lustre - cdn.opensfs.orgcdn.opensfs.org/wp-content/uploads/2012/12/1100-1130_Nathan_Rutm… · Map/Reduce on Lustre Hadoop Performance in ... Grep (64GB , 32 nodes,

GREP - Ghent University Repository

Clustering Lecture 8: MapReduce - csce.uark.eduxintaowu/BDAM/mapreduce.pdf · Distributed Grep . Very big data . Split data Split data Split data Split data . grep grep grep grep

Agenda User Profile File (.profile) –Keyword Shell Variables Linux (Unix) filters –Purpose –Commands: grep, sort, awk cut, tr, wc, spell.

gnu Grep: Print Lines Matching A Pattern - Del Maracademy.delmar.edu/Courses/ITSC1358/eBooks/Linux(grep).pdf · GNU Grep: Print lines matching a pattern ... Print the version number

AROMA: Automated Resource Allocation and …menasce/cs788/slides/wicke-d-AROMA.pdf · • Hadoop jobs: Sort, Wordcount and Grep (standard examples from Hadoop distribution).