Main%Memory%Map%Reduce%(M3R)% - IBM...1 Main%Memory%Map%Reduce%(M3R)% PL%Day,%September%23,%2013% %...

1

Main Memory Map Reduce (M3R)

PL Day, September 23, 2013 Avi Shinnar, David Cunningham, Ben Herta, Vijay Saraswat, Wei Zhang In collaboraLon with: Yan Li*, Dave Grove, Mikio Takeuchi**, Salikh Zakirov**, Juemin Zhang * IBM China Research Lab ** IBM Tokyo Research Lab Otherwise IBM TJ Watson Research Lab, New York

© 2013 IBM CorporaLon

resilience performance

low latency

M3R (engine) Hadoop

X10 Java

out-‐of-‐core in-‐memory

scalability


•  “Java”-‐like language designed for performance and producLvity at scale

•  Asynchronous ParLLoned Global Address Space programming model –  async S: run S as a separate ac/vity –  at (P) S: run S at place P –  finish S: wait for termina/on of children ac/vi/es – MPI style barriers, local “atomic” synchronizaLon…

•  Advanced type system –  reified generics, closures, dependent types…

•  MulLple backends: Java, C++, CUDA hap://x10-‐lang.org/

X10


Main-‐Memory Map Reduce in X10

secondary sorLng (values)

sorLng (keys)

combiners

map-‐only jobs

mulL-‐threading

iteraLve jobs

user controlled serializaLon out of core shuffle

debugging

profiling


M3R/Hadoop Architecture Java Hadoop App Hadoop

Map Reduce Engine

HDFS data

M3R Engine

HDFS

X10 M3R jobs

M3R/Hadoop adaptor

X10 Java

mulLple jobs

mulLple jobs

HDFS

JVM only

JVM/Na/ve

Java M3R jobs


M3R / Hadoop

Performance

CompaLbility


Hadoop Job

Input (InputFormat/

RecordReader/

InputSplit)

File System

(HDFS)

Map (Mapper)

Reduce (Reducer)

Output (OutputFormat/

RecordWriter

OutputCommi:er)

Shuffle File

System

File

System


M3R/Hadoop Job: cache

Input (InputFormat/

RecordReader/

InputSplit)

File System

(HDFS)

Map (Mapper)

Reduce (Reducer)


RecordWriter

OutputCommi:er)

Shuffle File

System

File

System

Cache


M3R/Hadoop Job: in-‐memory

Input (InputFormat/

RecordReader/

InputSplit)

File System

(HDFS)

Map (Mapper)

Reduce (Reducer)


RecordWriter

OutputCommi:er)

Shuffle File

System

File

System

Cache


M3R/Hadoop Job: co-‐locaLon

Input (InputFormat/

RecordReader/

InputSplit)

File System

(HDFS)

Map (Mapper)

Reduce (Reducer)


RecordWriter

OutputCommi:er)

Shuffle File

System

File

System

Cache


Iterated Matrix Vector mulLplicaLon

•  Algorithm (“standard HPC”) –  Row block parLLon G –  Replicate V –  In parallel, at each place, mulLply each

row of G with V. –  In parallel, each place broadcasts its

segment of V to all others •  This reassembles V for next phase.

•  Performance key – Read the appropriate part of G once, never communicate it.

– Reassembly is local.

G V V


M3R/Hadoop Job: locality

Input (InputFormat/

RecordReader/

InputSplit)

File System

(HDFS)

Map (Mapper)

Reduce (Reducer)


RecordWriter

OutputCommi:er)

Cache

Shuffle ParLLoner


ParLLon Stability in M3R

•  The reducer associated with a given parLLon number will always be run at the same place – Assuming the number of reducers and the number of places remains the same,

•  The number of reducers is determined by the applicaLon.

•  The number of places is fixed for the duraLon of the M3R server.


Sparse Matrix Vector MulLplicaLon

0

200

400

600

800

1000

1200

1400

1600

1800

2000

0 200000 400000 600000 800000 1000000 1200000 1400000 1600000

Tim

e (s

)

Size M (G is an MxM matrix with sparsity 0.001)

Sparse Matrix Vector Multiplication

M3R/Hadoop Hadoop Expon. (M3R/Hadoop) Expon. (Hadoop)

50X 0

5

10

15

20

25

30

35

40

45

0 200000 400000 600000 800000 1000000 1200000 1400000 1600000

Tim

e (s

)

Size M (G is an MxM matrix of sparsity 0.001)

Sparse Matrix Vector Multiplication

M3R/Hadoop Expon. (M3R/Hadoop)


M3R / Hadoop

Performance

CompaLbility


JobTracker


M3R / Hadoop

Performance

CompaLbility


DML results (Nov. 2011)

PageRank Hadoop M3R Speedup 100K 880s 452s 1.9x 200K 885s 574s 1.5x 400K 872s 530s 1.7x

Linear Regression

Hadoop M3R Speedup

1000K 1272s 120s 10.6x 3000K 1438s 185s 7.8x 5000K 1473s 275s 5.4x

GNNMF Hadoop M3R Speedup 100K 1489s 115s 13x 200K 1492s 185s 8.1x 400K 1481s 300s 4.9x


Pig unit tests


Current Status / Future Work

•  VLDB ‘12 –  Avraham Shinnar, David Cunningham, Vijay Saraswat, and Benjamin

Herta. 2012. M3R: increased performance for in-‐memory Hadoop jobs. Proc. VLDB Endow. 5, 12 (August 2012), 1736-‐1747.

•  Things generally work quite well •  Working on out-‐of-‐core shuffle

– Performance degradaLon instead of crashing

•  Working on dynamic class loading


Main%Memory%Map%Reduce%(M3R)% - IBM...1 Main%Memory%Map%Reduce%(M3R)% PL%Day,%September%23,%2013% %...

Documents

Transcript of Main%Memory%Map%Reduce%(M3R)% - IBM...1 Main%Memory%Map%Reduce%(M3R)% PL%Day,%September%23,%2013% %...