Hadoop and MapReduce
22
HADOOP Framework and Applications Prepared by: TEAM HADOOP slide1/22
-
Upload
abhishek-de -
Category
Education
-
view
308 -
download
1
Transcript of Hadoop and MapReduce
HADOOPFramework and Applications
Prepared by: TEAM HADOOP slide1/22
CONTENTS WHY HADOOP?
INTRODUCTION TO MapReduce
Prepared by: TEAM HADOOP slide 2/22
WHAT?“... to create building blocks for programmers who just happen to have lots of data to store, lots of data to analyze, or lots of machines to coordinate, and who don’t have the time, the skill, or the inclination to become distributed systems experts to build the infrastructure to handle it.” -Tom White Source: Hadoop: The Definitive Guide
Prepared by: TEAM HADOOP slide 3/22
WHAT? Hadoop contains many subprojects: Hadoop Common Chukwa HBase ZooKeeper Pig Zombie Hive MapReduce
We will focus on MapReduce
Prepared by: TEAM HADOOP slide 4/22
WHO & WHEN? Pre-2004 : Cutting and Cafarella develop
open source projects for web-scale indexing, crawling and search.
Prepared by: TEAM HADOOP slide 5/22
WHO & WHEN? 2004: Jeffrey Dean and Sanjay
Ghemawat introduce map reduce model used internally at Google.
Prepared by: TEAM HADOOP slide 6/22
WHO & WHEN? 2006: Hadoop becomes official Apache
project, Cutting joins Yahoo!Yahoo adopts Hadoop.
Prepared by: TEAM HADOOP slide 7/22
TRENDS
Prepared by: TEAM HADOOP slide 8/22
WHO USES IT?
Prepared by: TEAM HADOOP slide 9/22
Roughly how long to read 1TB from a commodity hard disk?
Prepared by: TEAM HADOOP slide 10/22
Roughly how long to read 1TB from a commodity hard disk?
Around 4 hours
62 seconds…
WITH HADOOP..
Prepared by: TEAM HADOOP slide 11/22
INTRODUCTION TO MapReduce
"Break large problem into smaller parts, solve in parallel, combine results."
Prepared by: TEAM HADOOP slide 12/22
Typical scenario How many times is the word ‘IT’
present? You’ll probably count but in a 30k paged document, can you??
Prepared by: TEAM HADOOP slide 13/22
Map Reduce Typical Illustration
Prepared by: TEAM HADOOP slide 14/22
Map Reduce paradigm
Input
Map
Shuffle/SortReduce
Output
Prepared by: TEAM HADOOP slide 15/22
Map Reduce paradigm Map: transforms input record to
intermediate (key, value) pair
Prepared by: TEAM HADOOP slide 16/22
Map Reduce paradigm Reduce: transforms all records for given
key to final output.
Prepared by: TEAM HADOOP slide 17/22
Map reduce principles
Move code to data (local
computation)
Allow programs to scale
transparently w.r.t size of input
Abstract away fault tolerance, synchronization,
etc.
Prepared by: TEAM HADOOP slide 18/22
Map Reduce: strengths
Batch, offline jobs
Write-once, read-many across full data set
Usually, though not always, simple computations
I/O bound by disk/network bandwidth
Prepared by: TEAM HADOOP slide 20/22
What it’s not!
What it’s not:
High-performance parallel computing, e.g. MPI
Low-latency random access relational database
Always the right solution
Prepared by: TEAM HADOOP slide 21/22
THANK YOU!
QUESTIONS?
Prepared by: TEAM HADOOP slide 22/22
