Post on 12-Sep-2020
Map Reduce
Last Time โฆ
Parallel Algorithms
Work/Depth Model
Spark
Map Reduce
Assignment 1
Questions ?
Today โฆ
Map Reduce
Matrix multiplication
Similarity Join
Complexity theory
MapReduce โ word counting
Input set of documents
Map:
reads a document and breaks it into a sequence of words๐ค1, ๐ค2, โฆ ,๐ค๐
Generates (๐, ๐ฃ) pairs,๐ค1, 1 , ๐ค2, 1 , โฆ , (๐ค๐, 1)
System:
group all ๐, ๐ฃ by key
Given ๐ reduce tasks, assign keys to reduce tasks using a hash function
Reduce:
Combine the values associated with a given key
Add up all the values associated with the word total count for that word
Matrix-vector multiplication
๐ ร ๐ matrix ๐ with entries ๐๐๐
Vector ๐ of length ๐ with values ๐ฃ๐
We wish to compute
๐ฅ๐ =
๐=1
๐
๐๐๐๐ฃ๐
If ๐ can fit in memory
Map: generate (๐,๐๐๐๐ฃ๐)
Reduce: sum all values of ๐ to produce (๐, ๐ฅ๐)
If ๐ is too large to fit in memory? Stripes? Blocks?
What if we need to do this iteratively?
Matrix-Matrix Multiplication
๐ = ๐๐ ๐๐๐ = ๐๐๐๐๐๐๐
2 mapreduce operations
Map 1: produce ๐, ๐ฃ , ๐, ๐, ๐, ๐๐๐ and ๐, ๐, ๐, ๐๐๐
Reduce 1: for each ๐ ๐, ๐, ๐๐๐ ร ๐๐๐
Map 2: identity
Reduce 2: sum all values associated with key ๐, ๐
Matrix-Matrix multiplication
In one mapreduce step
Map:
generate ๐, ๐ฃ ๐, ๐ , ๐, ๐,๐๐๐ & ๐, ๐ , ๐, ๐, ๐๐๐
Reduce:
each key (๐, ๐) will have values ๐, ๐ , ๐, ๐,๐๐๐ & ๐, ๐ , ๐, ๐, ๐๐๐ โ๐
Sort all values by ๐
Extract ๐๐๐ & ๐๐๐ and multiply, accumulate the sum
Complexity Theory for mapreduce
Communication cost
Communication cost of a task is the size of the input to the task
We do not consider the amount of time it takes each task to
execute when estimating the running time of an algorithm
The algorithm output is rarely large compared with the input or the
intermediate data produced by the algorithm
Reducer size & Replication rate
Reducer size ๐
Upper bound on the number of values that are allowed to appear in the list associated with a single key
By making the reducer size small, we can force there to be many reducers
High parallelism low wall-clock time
By choosing a small ๐ we can perform the computation associated with a single reducer entirely in the main memory of the compute node
Low synchronization (Comm/IO) low wall clock time
Replication rate (๐)
number of (๐, ๐ฃ) pairs produced by all the Map tasks on all the inputs, divided by the number of inputs
๐ is the average communication from Map tasks to Reduce tasks
Example: one-pass matrix mult
Assume matrices are ๐ ร ๐
๐ โ replication rate
Each element ๐๐๐ produces ๐ keys
Similarly each ๐๐๐ produces ๐ keys
Each input produces exactly ๐ keys load balance
๐ โ reducer size
Each key has ๐ values from ๐ and ๐ values from ๐
2๐
Example: two-pass matrix mult
Assume matrices are ๐ ร ๐
๐ โ replication rate
Each element ๐๐๐ produces 1 key
Similarly each ๐๐๐ produces 1 key
Each input produces exactly 1 key (2nd pass)
๐ โ reducer size
Each key has ๐ values from ๐ and ๐ values from ๐
2๐ (1st pass), ๐ (2nd pass)
Real world example: Similarity Joins
Given a large set of elements ๐ and a similarity measure ๐ (๐ฅ, ๐ฆ)
Output: pairs whose similarity exceeds a given threshold ๐ก
Example: given a database of 106 images of size 1MB each, find pairs of images that are similar
Input: (๐, ๐๐), where ๐ is an ID for the picture and ๐๐ is the image
Output: ๐๐, ๐๐ or simply ๐, ๐ for those pairs where ๐ ๐๐ , ๐๐ > ๐ก
Approach 1
Map: generate (๐, ๐ฃ)
๐, ๐ , ๐๐ , ๐๐
Reduce:
Apply similarity function to each value (image pair)
Output pair if similarity above threshold ๐ก
Reducer size โ ๐ 2 (2MB)
Replication rate โ ๐ 106 โ 1
Total communication from mapreduce tasks?
106 ร 106 ร 106 bytes 1018 bytes 1 Exabyte (kB MB GB TB PB EB)
Communicate over GigE 1010 sec 300 years
Approach 2: group images
Group images into ๐ groups with 106
๐images each
Map: Take input element ๐, ๐๐ and generate
๐ โ 1 keys ๐ข, ๐ฃ | ๐๐ โ ๐ข ๐ข , ๐ฃ โ {1, โฆ , ๐} โ {๐ข}
Associated value is (๐, ๐๐)
Reduce: consider key (๐ข, ๐ฃ)
Associated list will have 2 ร106
๐elements ๐, ๐๐
Take each ๐, ๐๐ and ๐, ๐๐ where ๐, ๐ belong to different groups and
compute ๐ ๐๐ , ๐๐
Compare pictures belonging to the same group
heuristic for who does this, say reducer for key ๐ข, ๐ข + 1
Approach 2: group images
Replication rate: ๐ = ๐ โ 1
Reducer size: ๐ = 2 ร 106/๐
Input size: 2 ร 1012/๐ bytes
Say ๐ = 1000,
Input is 2GB
Total communication: 106 ร 999 ร 106 = 1015 bytes 1 petabyte
Graph model for mapreduce
problems Set of inputs
Set of outputs
many-many relationship between the inputs and outputs, which describes which inputs are necessary to produce which outputs.
Mapping schema
Given a reducer size ๐
No reducer is assigned more than ๐inputs
For every output, there is at least one reducer that is assigned all input related to that output
๐1
๐4
๐2
๐3
๐1, ๐2
๐1, ๐3
๐1, ๐4
๐2, ๐3
๐2, ๐4
๐3, ๐4
Grouping for Similarity Joins
Generalize the problem to ๐ images
๐ equal sized groups of ๐
๐images
Number of outputs is ๐2โ๐2
2
Each reducer receives 2๐
๐inputs (๐)
Replication rate ๐ = ๐ โ 1
๐ =2๐
๐
The smaller the reducer size, the larger the replication rate, and therefore higher the communication
communication โ reducer size
communication โ parallelism
Lower bounds on Replication rate
1. Prove an upper bound on how many outputs a reducer with ๐inputs can cover. Call this bound ๐(๐)
2. Determine the total number of outputs produced by the problem
3. Suppose that there are ๐ reducers, and the ๐๐กโ reducer has ๐๐ < ๐
inputs. Observe that ๐=1๐ ๐(๐๐)must be no less than the number of
outputs computed in step 2
4. Manipulate inequality in 3 to get a lower bound on ๐=1๐ ๐๐
5. 4 is the total communication from Map tasks to reduce tasks. Divide
by number of inputs to get the replication rate
Lower bounds on Replication rate
1. Prove an upper bound on how many outputs a reducer with ๐inputs can cover. Call this bound ๐(๐)
2. Determine the total number of outputs produced by the problem
3. Suppose that there are ๐ reducers, and the ๐๐กโ reducer has ๐๐ < ๐
inputs. Observe that ๐=1๐ ๐(๐๐)must be no less than the number of
outputs computed in step 2
4. Manipulate inequality in 3 to get a lower bound on ๐=1๐ ๐๐
5. 4 is the total communication from Map tasks to reduce tasks. Divide
by number of inputs to get the replication rate
๐2โ๐2
2
๐2โ๐2
2
๐=1
๐๐๐2
2โฅ๐2
2
๐
๐=1
๐
๐๐ โฅ ๐2
๐=1
๐
๐๐ โฅ๐2
๐๐ โฅ๐
๐
Matrix Multiplication
Consider the one-pass algorithm extreme case
Lets group rows/columns into bands ๐ groups ๐/๐ columns/rows
=
๐ ๐ ๐
๐ ร ๐
Matrix Multiplication
Map:
for each element of ๐,๐ generate ๐ (๐, ๐ฃ) pairs
Key is group paired with all groups
Value is ๐, ๐, ๐๐๐ or ๐, ๐, ๐๐๐
Reduce:
Reducer corresponds to key (๐, ๐)
All the elements in the ๐๐กโ band of ๐ and ๐๐กโ band of ๐
Each reducer gets ๐๐
๐elements from 2 matrices
๐ =2๐2
๐, ๐ = ๐ ๐ =
2๐2
๐
Lower bounds on Replication rate
1. Prove an upper bound on how many outputs a reducer with ๐inputs can cover. Call this bound ๐(๐)
2. Determine the total number of outputs produced by the problem
3. Suppose that there are ๐ reducers, and the ๐๐กโ reducer has ๐๐ < ๐inputs. Observe that ๐=1
๐ ๐(๐๐)must be no less than the number of outputs computed in step 2
4. Manipulate inequality in 3 to get a lower bound on ๐=1
๐ ๐๐
5. 4 is the total communication from Map tasks to reduce tasks. Divide by number of inputs to get the replication rate
Each reducer receives ๐ rows from ๐and ๐ ๐ = 2๐๐ and produces ๐2
outputs ๐ ๐ =๐2
4๐2
๐2
๐=1๐ ๐๐
2
4๐2โฅ ๐2
๐=1
๐
๐๐2 โฅ 4๐4
๐=1๐ ๐๐ โฅ
4๐4
๐
๐ =1
2๐2
๐=1
๐
๐๐ =2๐2
๐
Matrix Multiplication LET US REVISIT THE TWO-PASS APPROACH
Matrix-Matrix Multiplication
๐ = ๐๐ ๐๐๐ = ๐๐๐๐๐๐๐
2 mapreduce operations
Map 1: produce ๐, ๐ฃ , ๐, ๐, ๐, ๐๐๐ and ๐, ๐, ๐, ๐๐๐
Reduce 1: for each ๐ ๐, ๐, ๐๐๐ ร ๐๐๐
Map 2: identity
Reduce 2: sum all values associated with key ๐, ๐
Grouped two-pass approach
=
๐2 groups of ๐2
๐2elements each
First pass: compute products of square
(๐ผ, ๐ฝ) of ๐ with square (๐ฝ, ๐พ) of ๐
Second pass: โ๐ผ, ๐พ sum over all ๐ฝ
Grouped two-pass approach
Replication rate for map1 is ๐ 2๐๐2 total communication
Each reducer gets 2๐2
๐2 ๐ =
2๐2
๐2 ๐ = ๐
2
๐
Total communication 22๐3
๐
Assume map2 runs on same nodes as reduce1 no communication
Communication ๐๐2 2๐3
๐
Total communication 32๐3
๐
Comparison
๐4
๐
๐3
๐๐ < ๐2<
If ๐ is closer to the minimum of 2๐, two pass is better by a factor of ๐ช ๐