Data 360 Conference: Introduction to Big Data, Hadoop and Big Data Analytics

Apache Hadoop Training Series: Hadoop Introduction

10/23/14

Apache Hadoop Training Series: Hadoop Introduction 1

[email protected]

https://www.linkedin.com/in/avkashchauhan

Lets Start and Define Big

Data


10/23/14



10/23/14


Lets Start and

Define Big Data

How Hadoop

Fits in this scenario


10/23/14


http://www.packtpub.com/using-cloudera-impala/book http://www.amazon.com/Simplifying-Windows-Azure-HDInsight-Service/dp/0735673802

http://blogs.msdn.com/b/microsoft_press/archive/2014/05/27/free-ebook-introducing-microsoft-azure-hdinsight.aspx

https://www.linkedin.com/in/avkashchauhan


10/23/14


Hadoop is an Open Source (Java based), “Scalable”, “fault tolerant” platform for large amount of unstructured data storage

& processing, distributed across machines.


10/23/14


Flexibility A Single Repo for

storing and analyzing any kind of data not bounded by schema

Scalability Scale-out architecture

divides workload across multiple nodes using flexible

distributed file system

Low Cost Deployed on commodity

hardware & open source platform

Fault Tolerant Continue working event if node(s) go

down

A system to move computation, where the data is.


10/23/14


Lets Start and Define Big Data

How Hadoop

Fits in this scenario

Hadoop Landscape


10/23/14



10/23/14



How Hadoop Fits in this scenario

Hadoop Landscape

Hadoop Core

Components

Data Storage

Data Processing


10/23/14


Hadoop Common

HDFS MapReduce

/YARN


10/23/14


Cloud

Cloudera Impala Hortonworks Tez

Impala uses C++ based in-memory processing of HDFS data through SQL like statements to expedite the data processing

Use cases include user collaborative filtering, user recommendations, clustering and classification.


10/23/14



How Hadoop Fits

in this scenario

Hadoop Landscape

Hadoop Core

Components

Applying Hadoop to Save $$


10/23/14




Hadoop Landscape

Hadoop Core Components


Concept of Data Lake


10/23/14



10/23/14



How Hadoop Fits

in this scenario

Hadoop Landscape

Hadoop Core

Components



Hadoop in Cloud


10/23/14



10/23/14




Hadoop Landscape




Hadoop in Cloud

Big Data Analytics


10/23/14


EDW

OLAP

ODS


10/23/14



10/23/14




Hadoop Landscape




Hadoop in Cloud

Big Data Analytics

With Hadoop


10/23/14


Amazon HDInsight Directives Data Storage S3 Azure Blobs Direct access to compute

machine to super fast data delivery

Processing EC2

Azure Compute Dedicated Machines ready to turn with specific version of Hadoop runtime

Processing Libraries Java based or any other language supported through Hadoop Streaming

.Net based code User uploads their code processing binaries/ libraries

Results S3 Azure Blobs Once job is completed the results are stored back to specific data storage used as source

Visualization Custom Custom 3rd party application can connect to storage to perform visualization


10/23/14



10/23/14




Hadoop Landscape




Hadoop in Cloud

Big Data Analytics

With Hadoop


10/23/14



10/23/14


http://blogs.msdn.com/b/microsoft_press/archive/2014/05/27/free-ebook-introducing-microsoft-azure-hdinsight.aspx

Data 360 Conference: Introduction to Big Data, Hadoop and Big Data Analytics

Documents

Transcript of Data 360 Conference: Introduction to Big Data, Hadoop and Big Data Analytics