2014 july 24_what_ishadoop

EVERYONE LIKES ELEPHANTS

Adam Muiseamuise@hortonworks.comPrincipal ArchitectHortonworks

Who am I?

Who is ?

We do Hadoop

The leaders of Hadoop’s development

Community driven, Enterprise Focused

Drive Innovation in the platform – We lead the roadmap

100% Open Source – Democratized Access to Data

We do Hadoop successfully.

Support

Professional ServicesTraining

What is Hadoop? What is everyone talking about?

“Big Data” is the marketing term of the decade in IT

What lurks behind the hype is the democratization of Data, a move to aggregate disparate data silos

into one shiny pile of analytic gold

So what are the problems with Big Data?

Let’s talk challenges…

Volume

VolumeVolume

Volume

VolumeVolume Volume

Volume

VolumeVolume

Volume

VolumeVolume

Volume

VolumeVolume Volume

Volume

VolumeVolume

VolumeVolume Volume

Volume

VolumeVolume

Volume

VolumeVolume

Volume

VolumeVolume

Volume

VolumeVolume

Volume

VolumeVolume

Volume

VolumeVolume

Volume

VolumeVolume Volume

Volume

VolumeVolume

Volume

VolumeVolume

Volume

VolumeVolume

Volume

VolumeVolume

Volume

VolumeVolume

Volume

VolumeVolume

Volume

VolumeVolume

Volume

VolumeVolume

Volume

VolumeVolume

Volume

VolumeVolume

Volume

VolumeVolume

Volume

VolumeVolume

Volume

VolumeVolume

Volume

VolumeVolume

Volume

VolumeVolume

Volume

VolumeVolume

Volume

VolumeVolume

Volume

Storage, Management, Processing all become challenges with Data at Volume

Traditional technologies adopt a divide, drop, and conquer approach

The solution?EDW

DataDataData

DataData

Data DataData

Yet Another EDW

DataDataData

DataData

Data DataData

Analytical DB

DataDataData

DataData

Data DataData OLTP

DataDataData

DataData

Data DataData

Another EDW

DataDataData

DataData

Data DataData

Ummm…you dropped something

DataDataData

DataData

Data DataData

DataDataData

DataData

Data DataDataDataDataData

DataData

Data DataData

DataDataData

DataData

Data DataData

DataDataData

DataData

Data DataData

DataDataData

DataData

Data DataData

DataDataData

DataData

Data DataData

DataDataData

DataData

Data DataDataDataData

DataData

Data DataData

DataDataData

DataData

Data DataData

Yet Another EDW

DataDataData

DataData

Data DataData

Analytical DB

DataDataData

DataData

Data DataData

DataDataData

DataData

Data DataData

Another EDW

DataDataData

DataData

Data DataData

Analyzing the data usually raises more interesting questions…

…which leads to more data

Wait, you’ve seen this before.

DataDataData

DataData

Data DataData

DataDataData

DataData

Data DataDataDataData

DataData

Data DataData

DataDataData

DataData

Data DataData

DataDataData

DataData

Data DataData

DataDataData

DataData

Data DataData

Analytics Sausage Factory

Data DataData

DataData

Data DataData …Data

DataData…

DataData

Data begets Data.

What keeps us from our Data?

“Prices, Stupid passwords, and Boring Statistics.”

- Hans Rosling

http://www.youtube.com/watch?v=hVimVzgtD6w

Your data silos are lonely places.

DataDataData

DataData

Data DataData

Accounts

DataDataData

DataData

Data DataData

Customers

DataDataData

DataData

Data DataData

Web Properties

DataDataData

DataData

Data DataData

… Data likes to be together.

DataDataData

DataData

Data DataData

Accounts

DataDataData

DataData

Data DataData

Customers

DataDataData

DataData

Data DataData

Web Properties

DataDataData

DataData

Data DataData

Data likes to socialize too.EDW

DataDataData

DataData

Data DataData

Accounts

DataDataData

DataData

Data DataData

Customers

DataDataData

DataData

Data DataData

Web Properties

DataDataData

DataData

Data DataData

Machine Data

DataDataData

DataData

Data DataData

Twitter

DataDataData

DataData

Data DataData

Facebook

DataDataData

DataData

Data DataData

DataDataData

DataData

Data DataData

Weather Data

DataDataData

DataData

Data DataData

New types of data don’t quite fit into your pristine view of the world.

My Little Data Empire

DataDataData

DataData

Data DataData

DataDataDataData

DataData

Machine Data

DataDataDataData

DataData

To resolve this, some people take hints from Lord Of The Rings...

…and create One-Schema-To-Rule-Them-All…

DataDataData

DataData

Data DataDataSchema

…but that has its problems too.

DataDataData

DataData

Data DataDataSchemaData

DataData

ETL ETL

DataDataData

DataData

Data DataDataSchemaData

DataData

ETL ETL

What if the data was processed and stored centrally? What if you didn’t

need to force it into a single schema? We call it a Data Lake.

DataDataData

DataData

Schema

BI & Analytics Schema Schema

DataData

Data Lake

DataData

DataDataData

DataData

SchemaSchema

DataData

DataProcess Process

DataData

DataDataData Sources

Data Sources

A Data Lake Architecture enables:- Landing data without forcing a single schema- Landing a variety and large volume of data efficiently- Retaining data for a long period of time with a very low $/TB- A platform to feed other Analytical DBs- A platform to execute next gen data analytics and processing applications (SAS, Informatica,

Graph Analytics, Machine Learning, SAP, etc…)

In most cases, more data is better.Work with the population, not just a sample.

Your view of a client today.

Female

Age: 25-30

Town/City

Middle Income Band

Product Category Preferences

Your view with more data.

Female

Age: 27 but feels old

GPS coordinates

$65-68k per year

Product recommendations

Tea PartyHippie

Looking to start a business

Walking into Starbucks right now…

A depressed Toronto Maple Leaf’s Fan

Products left in basket indicate drunk amazon shopper

Gene Expression for Risk Taker

Thinking about a new house

Unhappy with his cell phone plan

Pregnant

Spent 25 minutes looking at tea cozies

So what is the answer?

Enter the Hadoop.

http://www.fabulouslybroke.com/2011/05/ninja-elephants-and-other-awesome-stories/

………

Hadoop was created because traditional technologies never cut it for the Internet properties like Google, Yahoo, Facebook, Twitter, and LinkedIn

Traditional architecture didn’t scale enough…

DB DBDB

AppApp AppApp

DB DBDB

AppApp AppApp DB DBDB

AppApp AppApp

Traditional architectures cost too much at that volume…

$pecial Hardware

$upercomputing

So what is the answer?

If you could design a system that would handle this, what would it look like?

It would probably need a highly resilient, self-healing, cost-efficient, distributed file system…

Storage Storage Storage

It would probably need a completely parallel processing framework that took tasks to the

data…

Storage Storage StorageProcessing Processing Processing

Processing Processing Processing

It would probably run on commodity hardware, virtualized machines, and common OS

platforms

It would probably be open source so innovation could happen as quickly as possible

It would need a critical mass of users

{Processing + Storage}=

{YARN + HDFS}

Want to get your hands dirty?

To do this, we need to install Hadoop right?

Enter the

Sandbox.

The Sandbox is ‘Hadoop in a Can’.It contains one copy of each of the Master and Worker node processes

used in a cluster, only in a single virtual node.

ProcessingStorage

Linux VM

Getting started with Sandbox VM:

- Pick your flavor of VM at…http://www.hortonworks.com/sandbox

- Start the sandbox VM- find the IP displayed - go to…

http://172.16.130.137

- Register- Click on ‘Start Tutorials’- On the left hand nav, click on ‘HCatalog, Basic Pig & Hive Commands’

http://hortonworks.com/hadoop-tutorial/how-to-use-hcatalog-basic-pig-hive-commands/

In this tutorial you can…- Land files in HDFS- Assign metadata with HCatalog- Use SQL with Hive- Learn to process data with Pig

Hadoop has other open source projects…

Apache Hadoop

FlumeAmbari

HBaseFalcon

MapReduceHDFS

SqoopHCatalog

StormYARN

Hortonworks Data Platform

FlumeAmbari

HBaseFalcon

MapReduceHDFS

SqoopHCatalog

Storm YARN

What else are we working on?

hortonworks.com/labs/

There is NO second place

HortonworksWe do Hadoop.

2014 july 24_what_ishadoop

Software

Transcript of 2014 july 24_what_ishadoop

July cs 2014

July 2014 Messenger

Photography July 2014

Handyman July 2014

Anuchit Jayapipat 3 July 2014 MSW Technology Anuchit Jayapipat 3 July 2014.

Publicity Report - July 3 - July 9, 2014

TMILLAN / JULY 2014 PAGE 1 / TMILLAN / 12th EDITION / JULY 2014

2014 RTP July 22, 2014

2014 july

July 2014

July 2014/ Julio 2014

July 17th 2014 Meeting Program: Texas Archeological Society …txhas.org/PDF/newsletters/2014/2014 July Profile.pdf · 1 July, 2014 Vol 3, Issue 6 July 17th 2014 Meeting Program:

Hhh July 2014

JULY 2014 - thecollegeclubofboston.com

Mastermind July-2014

July 2014 Presentation

July/August 2014 News ‘n Views - Clover Sitesstorage.cloversites.com/banwellcommunitychurch/documents/2014 July... · July/August 2014 News ‘n Views Banwell Community Church,

MasterLab 2014 - July 2, 2014

IC July 2014

July 2 to July 8, 2014