Hadoop: Distributed data processing

Hadoop: Distributed Data Processing

Amr AwadallahFounder/CTO, Cloudera, Inc.ACM Data Mining SIGThursday, January 25th, 2010

Wednesday, January 27, 2010

Amr Awadallah, Cloudera Inc 2

Outline

▪Scaling for Large Data Processing▪What is Hadoop?▪HDFS and MapReduce▪Hadoop Ecosystem▪Hadoop vs RDBMSes▪Conclusion

Current Storage Systems Can’t Compute

InstrumentationCollection

Storage Farm for Unstructured Data (20TB/day)

Mostly Append

RDBMS (200GB/day)Interactive Apps

Mostly Append

ETL Grid

Mostly Append

ETL Grid

Filer heads are a bottleneck

Mostly Append

Ad hoc Queries &Data Mining

ETL Grid Non-ConsumptionFiler heads are a bottleneck

The Solution: A Store-Compute Grid

Storage + Computation

Mostly Append

RDBMSInteractive Apps

Mostly Append

ETL and Aggregations

RDBMSInteractive Apps “Batch” Apps

Mostly Append

ETL and Aggregations

Ad hoc Queries& Data Mining

What is Hadoop?

▪A scalable fault-tolerant grid operating system for data storage and processing

What is Hadoop?

▪A scalable fault-tolerant grid operating system for data storage and processing▪ Its scalability comes from the marriage of:▪ HDFS: Self-Healing High-Bandwidth Clustered Storage▪ MapReduce: Fault-Tolerant Distributed Processing

What is Hadoop?

▪Operates on unstructured and structured data

What is Hadoop?

▪Operates on unstructured and structured data▪A large and active ecosystem (many developers and additions like HBase, Hive, Pig, …)

What is Hadoop?

▪Operates on unstructured and structured data▪A large and active ecosystem (many developers and additions like HBase, Hive, Pig, …)▪Open source under the friendly Apache License

What is Hadoop?

▪Operates on unstructured and structured data▪A large and active ecosystem (many developers and additions like HBase, Hive, Pig, …)▪Open source under the friendly Apache License▪http://wiki.apache.org/hadoop/

Hadoop History

Hadoop History▪ 2002-2004: Doug Cutting and Mike Cafarella started working

on Nutch

on Nutch▪ 2003-2004: Google publishes GFS and MapReduce papers

on Nutch▪ 2003-2004: Google publishes GFS and MapReduce papers ▪ 2004: Cutting adds DFS & MapReduce support to Nutch

on Nutch▪ 2003-2004: Google publishes GFS and MapReduce papers ▪ 2004: Cutting adds DFS & MapReduce support to Nutch▪ 2006: Yahoo! hires Cutting, Hadoop spins out of Nutch

on Nutch▪ 2003-2004: Google publishes GFS and MapReduce papers ▪ 2004: Cutting adds DFS & MapReduce support to Nutch▪ 2006: Yahoo! hires Cutting, Hadoop spins out of Nutch▪ 2007: NY Times converts 4TB of archives over 100 EC2s

on Nutch▪ 2003-2004: Google publishes GFS and MapReduce papers ▪ 2004: Cutting adds DFS & MapReduce support to Nutch▪ 2006: Yahoo! hires Cutting, Hadoop spins out of Nutch▪ 2007: NY Times converts 4TB of archives over 100 EC2s▪ 2008: Web-scale deployments at Y!, Facebook, Last.fm

on Nutch▪ 2003-2004: Google publishes GFS and MapReduce papers ▪ 2004: Cutting adds DFS & MapReduce support to Nutch▪ 2006: Yahoo! hires Cutting, Hadoop spins out of Nutch▪ 2007: NY Times converts 4TB of archives over 100 EC2s▪ 2008: Web-scale deployments at Y!, Facebook, Last.fm▪ April 2008: Yahoo does fastest sort of a TB, 3.5mins over 910

nodes▪ May 2009: ▪ Yahoo does fastest sort of a TB, 62secs over 1460 nodes▪ Yahoo sorts a PB in 16.25hours over 3658 nodes

▪ June 2009, Oct 2009: Hadoop Summit (750), Hadoop World (500)

▪ September 2009: Doug Cutting joins ClouderaWednesday, January 27, 2010

Hadoop Design Axioms

1. System Shall Manage and Heal Itself

1. System Shall Manage and Heal Itself2. Performance Shall Scale Linearly

1. System Shall Manage and Heal Itself2. Performance Shall Scale Linearly 3. Compute Should Move to Data

1. System Shall Manage and Heal Itself2. Performance Shall Scale Linearly 3. Compute Should Move to Data4. Simple Core, Modular and

Extensible

Block Size = 64MBReplication Factor = 3

HDFS: Hadoop Distributed File System

Cost/GB is a few ¢/month vs $/month

Block Size = 64MBReplication Factor = 3

HDFS: Hadoop Distributed File System

Cost/GB is a few ¢/month vs $/month

MapReduce: Distributed Processing

MapReduce Example for Word Count

Split 1

Split i

Split N

SELECT word, COUNT(1) FROM docs GROUP BY word;cat *.txt | mapper.pl | sort | reducer.pl > out.txt

Split 1

Split i

Split N

Map 1(docid, text)

(docid, text) Map i

(docid, text) Map M

(words, counts)

“To Be Or Not

To Be?”

Be, 12

Be, 7Be, 6

Split 1

Split i

Split N

Reduce 1

Reduce i

Reduce R

(sorted words, counts)

Shuffle(sorted words, counts)

Map 1(docid, text)

(docid, text) Map i

(docid, text) Map M

(words, counts)

“To Be Or Not

To Be?”

Be, 12

Be, 7Be, 6

Split 1

Split i

Split N

Reduce 1

Reduce i

Reduce R

(sorted words, counts)

Shuffle(sorted words, counts)

Map 1(docid, text)

(docid, text) Map i

(docid, text) Map M

(words, counts)

“To Be Or Not

To Be?”

Be, 12

Be, 7Be, 6

Output File 1(sorted words,

sum of counts)

Output File i(sorted words, sum of counts)

Output File R(sorted words,

sum of counts)

Be, 30

Hadoop High-Level Architecture

Name NodeMaintains mapping of file blocks

to data node slaves

Job TrackerSchedules jobs across

task tracker slaves

Data NodeStores and serves

blocks of data

Hadoop ClientContacts Name Node for data or Job Tracker to submit jobs

Task TrackerRuns tasks (work units)

within a jobShare Physical Node

Apache Hadoop Ecosystem

HDFS(Hadoop Distributed File System)

MapReduce (Job Scheduling/Execution System)

HBase (key-value store)

Pig (Data Flow) Hive (SQL)

BI ReportingETL Tools

n) Sqoop

(Streaming/Pipes APIs)

Relational Databases:Hadoop:

Use The Right Tool For The Right Job

When to use?• Affordable Storage/Compute

• Structured or Not (Agility)• Resilient Auto Scalability

When to use?• Interactive Reporting (<1sec)

• Multistep Transactions• Interoperability

Economics of Hadoop

Economics of Hadoop▪ Typical Hardware:▪ Two Quad Core Nehalems▪ 24GB RAM▪ 12 * 1TB SATA disks (JBOD mode, no need for RAID)▪ 1 Gigabit Ethernet card

▪ Cost/node: $5K/node

▪ Cost/node: $5K/node▪ Effective HDFS Space:▪ ¼ reserved for temp shuffle space, which leaves 9TB/node▪ 3 way replication leads to 3TB effective HDFS space/node▪ But assuming 7x compression that becomes ~ 20TB/node

Effective Cost per user TB: $250/TB

Effective Cost per user TB: $250/TBOther solutions cost in the range of $5K to $100K per user TB

Sample Talks from Hadoop World ‘09▪ VISA: Large Scale Transaction Analysis▪ JP Morgan Chase: Data Processing for Financial Services▪ China Mobile: Data Mining Platform for Telecom Industry▪ Rackspace: Cross Data Center Log Processing▪ Booz Allen Hamilton: Protein Alignment using Hadoop▪ eHarmony: Matchmaking in the Hadoop Cloud▪ General Sentiment: Understanding Natural Language▪ Yahoo!: Social Graph Analysis▪ Visible Technologies: Real-Time Business Intelligence▪ Facebook: Rethinking the Data Warehouse with Hadoop and

Slides and Videos at http://www.cloudera.com/hadoop-world-nyc

Cloudera Desktop

Conclusion

Hadoop is a data grid operating system which provides an economically

scalable solution for storing and processing large amounts of unstructured or structured

data over long periods of time.

Amr AwadallahCTO, Cloudera Inc.aaa@cloudera.com

http://twitter.com/awadallah

Online Training Videos and Info:http://cloudera.com/hadoop-

traininghttp://cloudera.com/blog

http://twitter.com/cloudera

Contact Information

Hadoop: Distributed data processing

Technology

Transcript of Hadoop: Distributed data processing

Distributed batch processing with Hadoop

Hadoop Distributed File System

An Efficient Implementation of Mobile Raspberry Pi Hadoop ... · deployment of the Mobile Raspberry Pi SBC with Hadoop clusters, which provides parallel and distributed processing

Hadoop Distributed File System - SNIA · Hadoop Distributed File System Dhruba Borthakur Apache Hadoop Project Management Committee dhruba@apache.org dhruba@facebook.com

Troubleshooting Hadoop: Distributed Debugging

AI and Machine Learning: Actionable Insights and Machine Learning...YARN (Hadoop operating system) Apache Hadoop on the data lake • Distributed processing • Diverse analytics •Batch/script

Hops - Distributed metadata for Hadoop

HDFS: Hadoop Distributed FS

The Hadoop Distributed File System

Hadoop Distributed File System

Real-Time Stream Processing CMSC 491 Hadoop-Based Distributed Computing Spring 2015 Adam Shook.

On the Energy (In)efficiency of Hadoop: Scale-down Efficiencycsl.stanford.edu/~christos/publications/2009.hadoopenergy.hotpowe… · Hadoop crash-course 4 Hadoop == Distributed Processing

Hadoop and Distributed Computing

Cloudera Academic · Cloudera Academic Cloudera Apache Hadoop Apache Hadoop is an open-source software framework for distributed storage and distributed processing of very large data

RESEARCH ARTICLE Big Data and Hadoop with … · In this paper, first of all, we ... B. Hadoop Distributed File System Hadoop distributed file system is a distributed, scalable and

Distributed and Parallel Processing Technology Chapter 4 Hadoop I/O Jaeyong Choi 1.

· Hadoop is an open source distributed processing framework that manages data processing and storage for big data applications running in clustered systems. Apache Hadoop is a collection

Brief on Hadoop - Arif Works · Hadoop is a distributed data storage and processing platform Stores massive amounts of data in a very resilient way Handles low-level distributed system

Snapshotting in Hadoop Distributed File System for Hadoop ...€¦ · Snapshotting in Hadoop Distributed File System for Hadoop Open Platform as Service ... 2.2 Hadoop Open Platform

HDFS: Hadoop Distributed File Systemcis.csuohio.edu/~sschung/cis612/LectureNotes_HadoopFinal_1.pdfWhat is Hadoop? • Framework for large-scale data processing • Inspired by Google’s