pastiaro.wordpress.com

@rpastia

Building a connector – The Wrong Way

Mapper Reducer

Building a connector – The Right Way

Mapper ReducerPartitioner

InputSplit

InputFormat

RecordReader

RecordWriter

OutputFormat

The InputFormat: From Input to Mapper--range 2014-09-01;2014-09-20

--number_of_mappers 4

2014-09-01 2014-09-022014-09-03

2014-09-04

2014-09-05

… … …

2014-09-06

2014-09-20

2014-09-01

2014-09-02

2014-09-05

.

.

.

Input Split 1

(2014-09-01-A; record A)

(2014-09-01-B; record B)

(2014-09-01-…; record …)

(2014-09-02-A; record A)

(2014-09-02-B; record B)

(2014-09-02-…; record …)

(2014-09-05-A; record A)

(2014-09-05-B; record B)

(2014-09-05-…; record …)

Record Reader 1

Mapper

The InputFormat: From Input to Mapper

--range 2014-09-01;2014-09-20

--number_of_mappers 4

2014-09-01 2014-09-022014-09-03

2014-09-04

2014-09-05

… … …

2014-09-06

2014-09-20

2014-09-01

2014-09-02

2014-09-05

.

.

.

Input Split 1

(2014-09-01-A; record A)

(2014-09-01-B; record B)

(2014-09-01-…; record …)

(2014-09-02-A; record A)

(2014-09-02-B; record B)

(2014-09-02-…; record …)

(2014-09-05-A; record A)

(2014-09-05-B; record B)

(2014-09-05-…; record …)

Record Reader 1

Mapper