Hadoop

red red red red red red red red red red red red red red red red red red red red

CYS14011 - Rithu P Ravi

CYS14012 - Saumya K

red 1/1

Big Data Hadoop... HDFS Map Reduce

Why and What HADOOP?...

Apache Hadoop is an open-source software framework

A tool to process big data

Rithu P Ravi,SaumyaK HADOOP 2/30


Outline

1 Big Data

2 Hadoop...

3 HDFS

4 Map Reduce



Big Data

Data beyond storage and processing power

3 Vs

Volume

Velocity

Variety



Big Data

Exponential growth of data

Challenges to Google, Yahoo, Microsoft, Amazon

Need to go through TBs and PBs of data ?

Existing tools became inadequate to process such largedata sets.



Big ElephantNumerous small chicken..?



How to handle such BIG ?

Issues

How to handle a system up and downs ?

How to combine the data from all the systems ?



Problem1 : Systems Ups Downs

Commodity hardware for data storage and analysis

Chances of failure are very high

Replication of data across some machines

GFS (Google File System)

GFS

Divides data into chunks and stores in the file System

Can store data in ranges of PBs also



Problem 2 : How to combine the data ?

Analyze data across different machines .

Merge-, Data has to travel across network.

Doing this is notoriously challenging

Again GoogleMapReduce



Map Reduce

Provides a programming model

Abstracts disk reads and writes

Converts to (keys,values) pair

Two Phases

MapReduce



Outline

1 Big Data

2 Hadoop...

3 HDFS

4 Map Reduce



HADOOP

A reliable shared storage system

Analysis system



History

Google was the first to launch GFS and MapReduce

Published a paper 2004

A brand new technology

Was well proven in Google by 2004 itself



History

Doug Cutting

Open source version of MapReduce system called Hadoop

Yahoo and others rallied around to support this effort.

Now Hadoop is core part in : Facebook, Yahoo, LinkedIn,Twitter



Core Concepts

HDFS

Map Reduce



Outline

1 Big Data

2 Hadoop...

3 HDFS

4 Map Reduce



HDFS...Hadoop Distributed File System

Streaming very large files on commodity cluster

1 Very Large Files : MBs to PBs2 Streaming

Write once read many approachNo modifiationTime to read the whole data is more important

3 Commodity Cluster

No High end ServersYes, high chance of failure (But HDFS is tolerantenough)Replication is done



HDFSHadoop Distributed File System...

Services

Masters

Name Node

Secondary Name Node

Job Tracker

Slaves

Data Node

Task Tracker



HDFSHadoop Distributed File System...

Name Node

Master Node

Maintains Name System

Meta Data

Secondary Name Node

Periodically updating fsimage file

Data Node

Slaves

Actual Storage



HDFS Architecture



Outline

1 Big Data

2 Hadoop...

3 HDFS

4 Map Reduce



Map Reduce

Large scale data processing in parallel.

It provides

Automatic parallelization and distributionFault-tolerance

Two Phases in Map Reduce

MapReduce



Map Reduce

Job Tracker

Master

Manages the jobes in the cluster

Task Tracker

Slaves

Responsible for Map Reduce



Map Reduce



Map Reduce

Map Phase

map(inKey,invalue)-list(outKey, intermediateValue)

Processes input key/value pair

Produces set of intermediate pairs

Reduce Phase

reduce(outKey,list(intermediateValue))- list(outValue)

Combines all intermediate values for a particular key

Produces a set of merged output values (usually just one)



Map Reduce



ReferencesIf you want to improve this style

Hadoop Tutorial-Durga Softhttps://www.youtube.com/watch?v=DLutRT6K2rM/

Hadoop Official Sitehttp://hadoop.apache.org/index.html/

Processing Big Data using Hadoop FrameworkPrashant D. Londhe, Satish S. Kumbhar, Ramakant S.Sul, Amit J. Khadse



Happy Hadooping.... :)


Big DataHadoop...HDFSMap Reduce

Hadoop

Documents

Transcript of Hadoop