Distributed Computation to Sort massive data over

byParth Sawant , Suhas Pillai & Siddesh Pillai

0101001 10110100 10110010 00110010 00110010 10111001 101101000 01010011 01110101 01101000 01100001 01110011

& 01010000 01100001 01110010 011101000 1101000

Example Scenario• 8 MB data containing (char + int) strings and you have 4 slaves.

MASTER

SLAVESSLAVESSLAVESSLAVES

Master

SlaveSlaveSlaveSlave

CHUNKS CHUNKSCHUNKSCHUNKS

sort sortsortsort

Store file objects in a List

Sorted chunks Sorted

chunks

Sorted chunks

Sorted chunks

Sorted chunks

MASTER

File Objects

File Objects

File Objects

File Objects

File Objects

File Objects

File Objects

File Objects

TAKE 1000 elements from every file and do heap sort and continue till all are read

Sorted file

Fault Tolerance

SLAVE

SLAVE

SLAVE

SLAVE Heart beat

MASTER

Checks whether the slave is alive or dead or may be in coma

• Similar design pattern as task one.• Store the values in Hash Map (provenance and sum)• Reduction on master.

Task - 2

Questions