Apache Hadoop Crash Course

ApacheHadoopCrashCourseRafaelCossDataEvangelist@racoss#FutureOfData

2 ©HortonworksInc.2011–2016.AllRightsReserved

AgendaFutureofData

TraditionalDataArchitectures

What’sApacheHadoop?

DataAccesswithHadoop

LabIntro

CustomersarebuildingModernDataApplicationstotransformtheirindustries–renovatingtheirITarchitecturesandinnovatingwiththeirDatainMotionorDataatResttopoweractionableintelligence.

SocialMapping

PaymentTracking

FactoryYields

DefectDetection

CallAnalysis MachineData

ProductDesign M&A

DueDiligence

NextProductRecs

CyberSecurity

RiskModeling

AdPlacement

ProactiveRepair

DisasterMitigation

InvestmentPlanning

InventoryPredictions

CustomerSupport

SentimentAnalysis

SupplyChain

AdPlacement

BasketAnalysis Segments

Cross-Sell

CustomerRetention

VendorScorecards

OptimizeInventories

OPEXReduction

MainframeOffloads

HistoricalRecords

DataasaService

PublicData

Capture

FraudPrevention

DeviceDataIngest

RapidReporting

DigitalProtection

3 © HortonworksInc.2011–2016.AllRightsReserved

FutureofData

INTERNETOF

ANYTHING

TheFutureofDataisaboutactionableintelligencederivedfromaconstantlyconnectedsocietywitheasysecureaccesstorichdatasetscomingfromtheInternetofAnything

DataPowersHighwaySafety

TirePressure

Serverlog Mobile

Sensor

Location

Precipitation

Social

Click-stream

DataPowersHighwaySafety

NewDataParadigmOpensUpNewOpportunity

2.8zettabytesin2012

44zettabytesin2020

1 zettabyte (ZB) = 1 million petabytes (PB); Sources: IDC, IDG Enterprise, and AMR Research

Clickstream

ERP,CRM,SCM

Web&social

Geolocation

InternetofThings

Server logs

Files, emails

Transformeveryindustryviafullfidelityofdataandanalytics

Opportunity

T R A D I T I O N A L

LAGGARDS

LEADERS

AbilitytoConsumeData

EnterpriseBlindSpot

Whatdisruptedthedatacenter?

ModernDataApplications

Polygot Persistence

SQLNoSQL

NewSQLSearch

At-Rest In-Motion

AnalyticsDataVariety

Integration

DataLake Federation

OptimizationStorage,Compute

DistributedComputing

CommodityHardware

HybridDistributedComputing

TheFutureofDataActionableIntelligence

D AT A I N M O T I O N

STORAG

ESTORAG

GROUP2GROUP1

GROUP4GROUP3

D A T A A T R E S T

INTERNETOF

ANYTHING

ConnectedDataPlatformsarepoweringActionableIntelligence

Anyandalldatafromsensors,machines,

geolocation,clicks,files,social.

Securepoint-to-pointandbi-directionaldataflows

Collectandcuratealldata.

TraditionalDataArchitectures

SystemsofIntelligence

SystemsofEngagements

SystemsofInteractions

DataSystems

SystemsofRecord

SystemsofInsight

EventsInGray

AnalyticsIn

OperatorsDevelopers

Unstructured

Visualization&Dashboards

BusinessAnalytics

DataMarts

DataMarts Archive

StatisticsOLAP

FileServer

ClickstreamLogs

Web&SocialLogs

AudioVideo

LogsLogs

Geolocation

POS CRM ERP

Filter

AppServer

MessageBus

Documents

Unstructured

BusinessAnalytics

DataMarts

DataMarts Archive

StatisticsOLAP

FileServer

ClickstreamLogs

Web&SocialLogs

AudioVideo

LogsLogs

Geolocation

POS CRM ERP

Filter

AppServer

MessageBus

Documents

Ã Tooexpensiveandslowasdatagrowthkeepsaccelerating

Ã Tooslowtogetthedatapreparedforanalytics

Ã Analyticsisonlyleveragingalimiteddataset

Ã Colddatabecomesarchivedandisnolongerusableforanalytics

Ã DataingestisrigidandslowfornewIoAT datatypes

Ã Limitedrealtimeinsights

TraditionalDataArchitectureChallengeswithBigData

Unstructured

BusinessAnalytics

DataMarts

DataMarts Archive

StatisticsOLAP

FileServer

ClickstreamLogs

Web&SocialLogs

AudioVideo

LogsLogs

Geolocation

POS CRM ERP

Filter

AppServer

MessageBus

Documents

Next Generation AnalyticsIterative & ExploratoryData is the structure

IT TeamDelivers DataOn Flexible

Platform

BusinessUsers

Explore andAsk Any Question

Analyze ALL Available Information

Whole population analytics connects the dots

Traditional AnalyticsStructured & Repeatable

Structure built to store data

BusinessUsers

DetermineQuestions

IT TeamBuilds System

To AnswerKnown Questions

Available Information

AnalyzedInformation

Capacity constrained down sampling of available information

Carefully cleanse all information before any analysis

AnalyzedInformation

Analyze information as is & cleanse as needed

AnalyzedInformation

ModernDataApplications

Next Generation AnalyticsIterative & ExploratoryData is the structure

Traditional AnalyticsStructured & Repeatable

Structure built to store data

?AnalyzedInformation

Question

DataAnswer

Hypothesis

StartwithhypothesisTestagainstselecteddata

Data leads the way Explore all data, identify correlations

Correlation

All Information

Exploration

Actionable Insight

Analyzeafterlanding… Analyzeinmotion…

ModernDataApplicationsHasTwoThemes

What’sApacheHadoop?

HadoopArchitecture

DataAccessEngines

DistributedReliableStorage

DistributedComputeFrameworkResourceManagement,DataLocalityDataOperatingSystem

Batch Interactive Real-time

Governance&

IntegrationSecurity

Applications

DeployAnywhere

HadoopDataPlatformArchitecture

StoreandprocessallofyourCorporateDataAssets

YARN:DataOperatingSystem

DATA MANAGEMENT

Providelayeredapproachto

securitythroughAuthentication,Authorization,Accounting,andDataProtection

SECURITY

Access yourdatasimultaneously inmultiple ways(batch, interactive, real-time)

DATA ACCESS

Loaddataandmanage according

topolicy

GOVERNANCE & INTEGRATION

ENTERPRISEMGMT&SECURITY

Empowerexistingoperationsandsecuritytoolstomanage Hadoop

PRESENTATION&APPLICATION

Enablebothexistingandnewapplicationtoprovidevaluetotheorganization

Providedeploymentchoice acrosson-premise,appliance, virtualized,cloud

DEPLOYMENTOPTIONS

Deployandeffectivelymanage theplatform

OPERATIONS

22 ©HortonworksInc.2011–2016.AllRightsReserved runson

RDBMSImport/Export

DistributedStorage&ProcessingFramework

SecureNoSQL DB

SQLonHBase

NoSQL DB

WorkflowManagement

StreamingDataIngestion

ClusterSystemOperations

SecureGateway

DistributedRegistry

Search&Indexing

EvenFasterDataProcessing

DataManagement

MachineLearning

HadoopEcosystem

OpenEnterpriseHadoopCapabilities

YARN : Data Operating System

DATA ACCESS SECURITYGOVERNANCE & INTEGRATION OPERATIONS

1 ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

Data Lifecycle & Governance

FalconAtlas

AdministrationAuthenticationAuthorizationAuditingData Protection

RangerKnoxAtlasHDFSEncryptionData Workflow

SqoopFlumeKafkaNFSWebHDFS

Provisioning, Managing, & Monitoring

AmbariCloudbreakZookeeper

Scheduling

MapReduce

Script

Search

HBaseAccumuloPhoenix

Stream

In-memory

Others

ISV Engines

Tez Tez Slider Slider

DATA MANAGEMENT

HortonworksDataPlatform

DeploymentChoiceLinux Windows On-Premise Cloud

HDFS Hadoop Distributed File System

HORTONWORKS DATAPLATFORM

DATAMGMT

HDP2.2Dec2014

HDP2.1April2014

HDP2.0Oct2013

HDP2.2Dec2014

HDP2.1April2014

HDP2.0Oct2013

OngoingInnovationinApache

HDFSYARNMapReduceHadoopCore

WhatisApacheHadoop?

Yahoo!2006

HortonworksOct2011

Yahoo!startfocusonmultipleHadoopapps&clustersContributesHadooptoApache

HDP1.0Oct2012

ApacheHadoopv2YARN

GooglepublishesGFS&MapReduce papers2004-2005

HDP 2.4March2016

HDP2.2Dec2014HDP2.3July2015 2.7.1

+ /directory/structure/in/memory.txt

Resource management + schedulingDisk, CPU, Memory

CoreNameNode

ResourceManagerYARN

Hadoop daemon

User application

DataNodeHDFS

NodeManagerYARN

Worker Node

HDFS:Scalable,ReliableandSecureStoragePlatformTheStoragePlatformforHadoop2.0ScalableHorizontallygrowasdatavolumesgrow,addingoneormultiple nodesatatime

ReliableHighlyavailable(HA)andfaulttoleranttoprotectagainstdatalossandcorruption

CostEffectiveLeverageCommodityHardwareCrossworkloadaccess

SecureStrongaccess controls,integratedwithauthenticationmechanisms

Granulardataaccesscontrolstodatasets acrossusersandgroupsProtectsdataoverthewireandatrest

YARN: Data Operating System

C A B C B B A C

B A B A C A

Standards Based Data Interfaces

NFSSource /

Destination

Source / Destination

Ingestandstoreanydatainanyformat

Flexiblereadaccess enablesavarietyofworkloads

Heterogeneous Storage

Before• DataNodeisasinglestorage• Storageisuniform-OnlystoragetypeDisk• Storagetypeshiddenfromthefilesystem

New Architecture• DataNodeisacollectionofstorages• Supportdifferenttypesofstorages

– Disk,SSDs,Memory

Alldisksasasinglestorage

S3SwiftSANFilers

Collectionoftieredstorages

Hadoop Distributed File System (HDFS)

Fault Tolerant Distributed Storage• Dividefilesintobigblocksanddistribute3copiesrandomlyacrossthecluster• ProcessingDataLocality

• NotJuststoragebutcomputation

10110100101001001110011111100101001110100101001011001001010100110001010010111010111010111101101101010110100101010010101010101110010011010111010

Logical File

Blocks

Cluster

Batch Processing in HadoopMapReduceBatch Access to DataOriginal data access mechanism for Hadoop

• FrameworkMadefordevelopingdistributed applications toprocessvastamountsofdatain-parallelonlargeclusters

• ProvenReliable interfacetoHadoopwhichworksfromGBtoPB.But,batchoriented– Speedisnotit’sstrongpoint.

• EcosystemPortedtoHadoop2torunonYARN.Supportsoriginalinvestments inHadoopbycustomersandpartnerecosystem.

DataNode1

Mapper

Dataisshuffledacrossthenetwork

&sorted

MapPhase Shuffle/Sort ReducePhase

MapReduce JobLifecycle

SayingthatMapReduce isdeadispreposterous- Wouldlimitsustoonlynewworkloads- ALLHadoop clustersusemapreduce

- ProvenatEnterpriseScale

DataNode2

Mapper

DataNode3

Mapper

DataNode1

Reducer

DataNode2

Reducer

DataNode3

Reducer

Interactive Real-TimeBatch

What is MapReduce?Break a large problem into sub-solutionsMap

• Iterate over a large # of records

• Extract something of interest fromeach record

Shuffle

• Sort Intermediate results

Reduce

• Aggregate, summarize, filter or transform intermediate results

• Generate final output

MapProcess

DataData

DataData MapProcess

ReduceProcess

Read&ETL

Shuffle&Sort Aggregation

DataData

1st GenHadoop:CostEffectiveBatchatScale

HADOOP1.0BuiltforWeb-ScaleBatchApps

SingleAppBATCH

SingleAppINTERACTIVE

SingleAppBATCH

SiloscreatedfordistinctusecasesSingleApp

SingleAppONLINE

HadoopemergedasfoundationofnewdataarchitectureApacheHadoopisanopensourcedataplatformformanaginglargevolumesofhighvelocityandvarietyofdata

• BuiltbyYahoo!tobetheheartbeatofitsad&searchbusiness

• DonatedtoApacheSoftwareFoundationin2005withrapidadoptionbylargewebproperties&earlyadopterenterprises

• Incrediblydisruptivetocurrentplatformeconomics

TraditionalHadoopAdvantages

ü Managesnewdataparadigmü Handlesdataatscaleü Costeffectiveü Opensource

TraditionalHadoopHadLimitations

Batch-onlyarchitectureSinglepurposeclusters,specificdatasetsDifficulttointegratewithexistinginvestmentsNotenterprise-grade

Application

StorageHDFS

Batch ProcessingMapReduce

YARNextendsHadoopintodatacenterleaders

YARNThe Architectural Center of Hadoop

• Common data platform, many applications

• Support multi-tenant access & processing

• Batch, interactive & real-time use cases

• Supports 3rd-party ISV tools

(ex. SAS, Syncsort, Actian, etc.)

YARN Ready Applications Facilitates ongoing innovation and enterprise adoption via ecosystem of new and existing “YARN Ready” solutions

YARN : Data Operating System

BATCH, INTERACTIVE & REAL-TIME DATA ACCESS

1 ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

DATA MANAGEMENT

MapReduce

Script

Search

HBaseAccumuloPhoenix

Stream

In-memory

Others

ISV Engines

Tez Tez Slider Slider

WhatdoesiOS 6andWindows3.1haveincommon?

HadoopBeyondBatchwithYARN

SingleUseSysztemBatchApps

MultiUseDataPlatformBatch,Interactive,Online,Streaming,…

Ashiftfromtheoldtothenew…

HADOOP 1

MapReduce(cluster resource management

& data processing)

Data FlowPig

SQLHive

Others

API,Engine,

andSystem

YARN(Data Operating System: resource management, etc.)

Data FlowPig

SQLHive

OtherISV

Apache Yarn as a Base

System

Engine

API’s

1 ° ° ° ° °

° ° ° ° ° N

HDFS (redundant, reliable storage)

1 ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° N

BatchMapReduce

Tez Tez

MapReduce as the BaseHADOOP 2

ArchitectureEnabledbyYARNAsinglesetofdataacrosstheentireclusterwithmultipleaccessmethodsusing“zones”forprocessing

1 ° ° ° ° ° ° °

° ° ° ° ° ° ° °

° ° ° ° ° ° ° n

SQLHive

InteractiveSQLQueryforAnalytics

Script-basedETLAlgorithmexecutedinbatchtoreworkdatausedbyHiveandHBaseconsumers

• Maximize compute resources to lower TCO

• No standalone, silo’d clusters

• Simple management & operations

…all enabled by YARN

StreamProcessingStorm

Identify&actonreal-timeevents

NoSQLHbase

Accumulo

Low-latencyaccessservingupawebfrontend

HadoopWorkloadEvolution

SingleUseSystemBatchApps

MultiUseDataPlatformBatch,Interactive,Online,Streaming,…

Ashiftfromtheoldtothenew… MultiUsePlatformData&Beyond

HADOOP 1

HADOOP 2

1 ° ° ° °

° ° ° ° N

1 ° ° °

° ° ° N

MapReduce

HADOOP.Next

YARN ‘

1 ° ° ° ° ° °

° ° ° ° ° ° N

DATA ACCESS APPS

Docker

MySQLMR2 Others(ISV Engines)

Multiple(Script, SQL, NoSQL, …)

MR2 Others(ISV Engines)

Multiple(Script, SQL, NoSQL, …)

Docker

Tomcat

Docker

Gartner:WhatisHadoop?

Ã CommonApacheProjects– ALL=7(6)– Exceptfor1=3(5)– Exceptfor2=4(4)² About14CommonProjects

Ã UncommonProjects– Only1=9(1)– Only2=7 (2)– Only3=6 (3)² About22UncommonProjects

http://blogs.gartner.com/merv-adrian/2015/07/02/now-what-is-hadoop/

ODPi ODPi ODPi

HORTONWORKS DATA PLATFORM

HDP 2.3 is Apache Hadoop; not “based on” Hadoop

4.10.2

DATA MGMT DATA ACCESS GOVERNANCE & INTEGRATION OPERATIONS SECURITY

HDP 2.2Dec 2014

HDP 2.1April 2014

HDP 2.0Oct 2013

HDP 2.2Dec 2014

HDP 2.1April 2014

HDP 2.0Oct 2013 0.12.0 0.12.0

0.12.1 0.13.0 0.4.0

1.4.4 1.4.4 3.3.23.4.5

0.4.00.5.0

0.14.0 0.14.0 3.4.6 0.5.0 0.4.00.9.30.5.2

4.0.04.7.2

1.2.1 0.60.0 0.98.4 4.2.0 1.6.1 0.6.0 1.5.21.4.5 4.1.02.0.0

1.4.0 1.5.1 4.0.0

1.5.1 1.4.4 3.4.5

2.7.1 1.4.6 1.0.0 0.6.0 0.5.02.1.00.8.2 3.4.61.5.25.2.1 0.80.0 0.5.01.7.04.4.0 0.10.0 0.6.10.7.01.2.10.15.0HDP 2.3Oct 2015 4.2.0

0.96.1

0.98.0 0.9.1

1.4.1 1.1.2

2.7.1 1.4.6 1.3.0 0.9.0 0.6.02.4.00.10.0 3.4.61.5.25.5.1 0.80.0 0.7.01.7.04.7.0 1.0.1 0.10.00.7.01.2.10.16.0HDP 2.5*2H2016 4.2.01.6.2 1.1.2

2.7.1 1.4.6 1.1.0 0.6.0 0.5.02.2.10.9.0 3.4.61.5.25.2.1 0.80.0 0.5.01.7.04.4.0 0.10.0 0.6.10.7.01.2.10.15.0HDP 2.4Mar 2016 4.2.01.6.0 1.1.2

Ongoing Innovation in Apache

* HDP 2.5 – Shows current Apache branches being used. Final component version subject to change based on Apache release process.

NextGenerationDataVendorsInvestmentfortheEnterprise

VerticalIntegration with YARN and HDFSEnsure engines can run reliably and respectfully in a YARN based cluster

Provision, Manage & Monitor

AmbariZookeeper

Scheduling

Loaddataandmanageaccordingtopolicy

Providelayeredapproachto

securitythroughAuthentication,Authorization,Accounting,andDataProtection

SECURITYGOVERNANCE

Deployandeffectivelymanage theplatform

° ° ° ° ° ° ° ° ° ° ° ° ° ° °

Script

JavaScala

Cascading

Stream

Search

HBaseAccumulo

BATCH, INTERACTIVE & REAL-TIME DATA ACCESS

In-Memory

Others

ISV Engines

1 ° ° ° ° ° ° ° ° ° ° ° ° ° °

YARN: Data Operating System(ClusterResourceManagement)

HDFS (Hadoop Distributed File System)

Tez Slider SliderTez Tez

OPERATIONS

Horizontal Integration for Enterprise ServicesEnsure consistent enterprise services are applied across the Hadoop stack

Whatdodistributionsdo?

Ã Defineastackofcomponents• RichandlatestsetofApacheProjects(opensource&opencommunity)withoutlockin

Ã VerticalandHorizontalintegrationofcomponents• Vertical:BestSpeedandScale• Horizontal:OpenEnterpriseReady

Ã ProvisionandUpgradestack• Robust,EasyandAnywhere

Ã Acceleratetimetovalue(easyofuse)• NewFaceofHadoopwithUis fromAmbari,AmbariViews,Ranger,Falcon,Atlas

Ã PartnerEcosystem• RichandDeep

Ã Support• Industry’sbest,SmartSenseandinfluencecommunity

HadoopOperations&Tools

How Do You Operate a Hadoop Cluster?

Apache™ Ambari isaplatformtoprovision,manageandmonitorHadoopclusters

Ambari Core Features and Extensibility

Install&Configure

Operate,Manage&Administer

Develop

Optimize&Tune

Developer

DataArchitect

Ambariprovidescoreservicesforoperations,developmentandextensionspointsforboth

ExtensibilityFeatures

Stacks,Blueprints&RESTAPIs

CoreFeatures

InstallWizard&Web

Web,OperatorViews,Metrics&Alerts

UserViews

ViewsFramework&RESTAPIs

ViewsFramework

ClusterAdmin

Newuserinterfaceenablesfast&easySQLdefinitionandexecution.

New User Views for DevOps

CapacitySchedulerViewBrowseandmanageYARNqueues

Tez ViewViewinformationrelatedtoTez jobsthatareexecuting onthecluster

NewUserViewsforDevelopment

PigViewAuthorandexecute PigScripts.

HiveViewAuthor,execute anddebugHive

queries.

FilesViewBrowseHDFSfilesystem.

ApacheZeppelin

• Web-basednotebookfordataengineers,dataanalystsanddatascientists• Bringsinteractivedataingestion,data

exploration,visualization,sharingandcollaborationfeaturestoHadoopandSpark

• Moderndatasciencestudio• ScalawithSpark• PythonwithSpark• SparkSQL• ApacheHive,andmore.

HadoopDataAccess

Access patterns enabled by YARN

1 ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° °

Applications BatchNeeds to happen but, no timeframe limitations

InteractiveNeeds to happen at Human time

Real-Time Needs to happen at Machine Execution time.

Apache Hive: SQL in Hadoop

• Created by a team at Facebook

• Provides a standard SQL interface to data stored in Hadoop

• Quickly find value in raw data files

• Proven at petabyte scale

• Compatible with ALL major BI tools such as Tableau, Excel, MicroStrategy, Business Objects, etc…

SensorMobile

WeblogOperational

SQLQueries

Hive and the Stinger Initiative

BaseOptimizationsGeneratesimplifiedDAGsIn-memoryHashJoins

VectorQueryEngineOptimizedformodernprocessor

architectures

TezExpresstasksmoresimply

Eliminate diskwritesPre-warmedContainers

ORCFileColumnStore

HighCompressionPredicate/FilterPushdowns

YARNNext-genHadoopdataprocessing

framework

QueryPlannerIntelligentCost-BasedOptimizer

PerformanceOptimizations100x+fastertimetoinsightDeeperanalyticalcapabilities

Stinger.next andSub-SecondSQL

Emergence of LLAP brings Sub-Second SQL response times within reach with Hive.

BATCH & INTERACTIVE BATCH & INTERACTIVE BATCH, INTERACTIVE & SUB-SECONDSPEED

DELIVERY

UPDATES

ENGINES

STINGERD E L I V E R E D

PROGRESSD E L I V E R E D FINALVERSION

HDP 2.1VERSION

0.13VERSION

HDP 2.3VERSION

SQL:2003+ SQL:2011 SUBSET

READ-ONLY SQL INSERT/UPDATE/DELETE

MR, TEZ MR, TEZ

F U T U R ES T I N G E R N E X T

COMPLETE ACID SUPPORT INCLUDING MERGE

COMPREHENSIVE SQL:2011 BASED ANALYTICS

MR, TEZ, LLAP

DELIVERED IN DEVELOPMENT

TieredDataStorage

Stinger.next Phase3

YARN:ContainerizedApplications

DataTypes SQL Features File Formats Latest Additions…Numeric CoreSQLFeatures Columnar ScalableCrossProduct

FLOAT/DOUBLE Date,Time andArithmeticalFunctions ORCFile PrimaryKey/Foreign KeyDECIMAL INNER,OUTER,CROSSandSEMIJoins Parquet Non-EquijoinINT/TINYINT/SMALLINT/BIGINT DerivedTableSubqueries

TextTechPreview:Proc.Extensions(PL/SQL)

BOOLEAN Correlated+ UncorrelatedSubqueries CSV FutureString UNIONALL Logfile ACIDMERGE

CHAR/VARCHAR UDFs, UDAFs,UDTFs Nested/Complex MultiSubquerySTRING CommonTableExpressions Avro Comparison tosub-selectBINARY UNIONDISTINCT JSON INTERSECT andEXCEPT

Date, Time AdvancedAnalytics XMLDATE OLAPandWindowing Functions CustomFormatsTIMESTAMP CUBE andGrouping Sets OtherFeaturesIntervalTypes NestedDataAnalytics XPath Analytics

ComplexTypes NestedDataTraversalARRAY LateralViewsMAP ACIDTransactionsSTRUCT INSERT/UPDATE/DELETEUNION

ApacheHive:JourneytoSQL:2011Analytics

LegendExisting

Future

NewwithHive2.0

Columnar Storage

ORCFile Parquet

Unstructured Data

JSON CSV

Text Avro

Custom

Weblog

SQL Engines

RowEngine VectorEngineSQ

SQL Support

SQL:2011 Optimizer HCatalog HiveServer2

Block Cache

LinuxCache

Hadoop 1

MapReduce

Hadoop 2

Tez Spark

Vector Cache

Persistent Server

Historical

Current

In-development

Legend

Apache Hive: Modern Architecture

ApacheTezisacriticalinnovationoftheStingerInitiative.

• Along with YARN, Tez not only improves Hive, but improvesallthingsbatchand interactiveforHadoop;Pig,Cascading…

• More Efficient Processing than MapReduce

• Reduceoperationsandcomplexityofbackendprocessing• AllowsforMapReduceReducewhichsavesharddiskoperations• Implementsa“service”whichisalwayson,decreasingstarttimesofjobs• AllowsCachingofDatainMemory

Cascading/ Scalding

WhyisTez Important?

°1 ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° °

° ° ° ° ° ° °

° ° ° ° ° ° N

HDFS (Hadoop Distributed File

System)

Scripting

Tez Tez

Applications

ApacheTez

Hive– MapReduce Hive– Tez

SELECT a.state, COUNT(*), AVG(c.price) FROM a

JOIN b ON (a.id = b.id)JOIN c ON (a.itemId = c.itemId)

GROUP BY a.state

SELECTa.state

JOIN(a,c)SELECTc.price

SELECTb.id

JOIN(a, b)GROUPBYa.state

COUNT(*)AVG(c.price)

SELECTa.state,c.itemId

JOIN(a,c)

JOIN(a, b)GROUPBYa.state

COUNT(*)AVG(c.price)

SELECTb.id

Tez avoidsunneededwritestoHDFS

Scripting Data Pipeline & ETLApache Pig

• Dataflowengineandscriptinglanguage(PigLatin)• Allowsyoutotransformdataanddatasets

Advantages over MapReduce• Reducestimetowritejobs• Communitysupport• PiggybankhasasignificantnumberofUDF’stohelpadoption• TherearealargenumberofexistingshopsusingPIG

PigLatin

• Pigexecutesinauniquefashion:oDuringexecution,eachstatementisprocessedbythePiginterpretero Ifastatementisvalid,itgetsaddedtoalogicalplan builtbytheinterpreter

oThestepsinthelogicalplandonotactuallyexecuteuntilaDUMPorSTOREcommandisused

WhyusePig?

• Maybewewanttojointwodatasets,fromdifferentsources,onacommonvalue,andwanttofilter,andsort,andgettop5sites

ResourceManagement

Storage

Elegant Developer APIsDataFrames, Machine Learning, and SQL

Made for Data ScienceAll apps need to get predictive at scale and fine granularity

Democratize Machine LearningSpark is doing to ML on Hadoop what Hive did for SQL on Hadoop

CommunityBroad developer, customer and partner interest

Realize Value of Data Operating SystemA key tool in the Hadoop toolbox

ApacheSparkenthusiasm

Applications

SparkCoreEngine

ScalaJavaPythonlibraries

MLlib(Machinelearning)

SparkSQL*

SparkStreaming*

SparkCoreEngine

Apache Spark & Apache Hadoop Perfect Together

General Purpose Data Access Engineforfast,large-scaledataprocessing

Designed for Iterative, In-Memorycomputationsandinteractivedatamining

Expressive Multi-Language APIsforJava,Scala,PythonandR

Built-in LibrariesEnabledataworkerstorapidlyiterateoverdatafor:ETL,MachineLearning,SQLandStreamprocessing

ScalaJava

PythonR

Spark Core Engine

Spark SQL

Spark StreamingMLlib GraphX

1 ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

Apache Projects Enable Access Patterns

Various open source projects have incubated in order to meet these access pattern needs

Today, they can all run on a single cluster on a single set of data because of YARN

All powered by a broad open community

1 ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° °

InteractiveSolr

SparkHivePig

Real-TimeHBase

AccumuloStorm

BatchMapReduce

Applications

ConnectedDataPlatforms

ConnectedDataPlatformsEnableArchitecturalTransformations

DatainMotion(Cloud)

DatainMotion

(on-premises)

DataatRest

(on-premises)

EdgeData

DatainMotion

EdgeAnalytics

DataatRest(Cloud)

EdgeData

DataatRest

(on-premises)

ClosedLoop

Analytics

MachineLearning

DeepHistoricalAnalysis

Must-haveConsiderationsforTechnology

ContinuousDataLifeCycle

Real-timeinsightsfromorigintorest

EnterpriseReady

ManagementSecurity

Governance

DeploymentFlexibility

OnPremiseCloudHybrid

OpenInnovation

ArchitectureCommunityEcosystem

HandsonLabOverview

HDP2.4Sandbox

Ã ProvidesFreepreconfiguredHDP– RunsinaVirtualMachineor

AzureHortonworks.com/sandbox

Ã EasytoUse– Operations

• Ambari– DevandDevOps

• AmbariUserViews– WebNotebook

• Zeppelin

Ã Workswith60+FreetutorialHortonworks.com/tutorials

DataDiscoveryLab• ElefanteWineCompanyhasafleetofover100trucks.

• Thegeolocationdatacollectedfromthetruckscontainseventsgeneratedwhilethetruckdriversaredriving.

• Thecompany’sgoalwithHadoopistoMitigateRisk:o Understandcorrelationsbetweenmilesdrivenandeventso Computetheriskfactorforeachdriverbasedonmileage&events

o LabEnvo Sandbox2.4

o LabDoco URL:http://goo.gl/14OAato LoadDatao QueryDatao ProcessData

Elefante Wine Current Challenges

The CompanyElefante Wine is a boutique wine fulfillment company with a large fleet of trucks. It delivers wine in a highly-regulated industry with stringent transportation requirements.

The SituationRecently a number of driver violations led to fines and increased insurance rates

The Challenges• Rising Operational Costs• Driver Safety• Risk Management• Logistics Optimization

ElefanteWineCompanyhasalargefleetoftrucksinUSA

Atruckgeneratesmillionsofeventsforagivenroute;aneventcouldbe:

§ 'Normal'events:starting/stoppingofthevehicle

§ ‘Violation’events:speeding,excessiveaccelerationandbreaking,unsafetaildistance

Companyusesanapplicationthatmonitorstrucklocationsandviolationsfromthetruck/driverinreal-timetocalculaterisk

Route?Truck?Driver?

Analystsqueryabroadhistorytounderstandiftoday’sviolations arepartofalargerproblemwithspecificroutes,trucks,ordrivers

Elefante Wine Risk and Driver Safety Challenges

Trucksoutfittedwithnewsensorsgeneratinglargevolumesofnewdata:

• Location

• Speed

• DriverViolations

Needtobeintegratereal-time&historicaldata

Increase safety and reduce liabilitiesAnticipate driver violations BEFORE they happen and take precautionary actionsFindpredictivecorrelationsindriverbehavioroverlargevolumesofreal-timedata

Difficult to deliver timely insights to the right people and systems to take action

Data DiscoveryUncover new findings

Predictive Analytics Identify your next best action

Better Understandingof the Past

Better Prediction of the Future

What’sourgoal?

Ã Solution:– CollectadditionaldataviasensorsintruckstobetterunderstandRiskFactors

Ã How:– Quicklystorenewsensordatainacommonrepository– Preparethedataforanalysis– Explorethedata– CalculateRisk– Generateareport

Move Data Into Hadoop

Geolocation.csv

trucks.csv

Geolocation_stage Geolocation

Trucks_stage Trucks

csv ORC

ORCSQL

Geolocation

Trucks

PIGorSparkRisk Calculation

Truck_mileage

Avg_mileage

DriverMileage

RiskFactor

Events

Trucking Risk Analysis – Hadoop ELT

CalculateRisk

GettingStartedResources

developer.hortonworks.com

HortonworksNourishestheCommunityHOR TONWORKS

C OMMUN I TY C ONNE C T I ONHOR TONWOR KS PA R TN ERWORKS

https://community.hortonworks.com

Thankyou!

rafael@hortonworks.com

@racoss

Apache Hadoop Crash Course - HS16SJ

Technology

Transcript of Apache Hadoop Crash Course - HS16SJ

Intro to Spark with Zeppelin Crash Course Hadoop Summit SJ

New Map Reduce & Hadoop · 2018. 1. 24. · Hadoop Map Reduce Hadoop 2 TEZ Execution Engine DevelopmentSummary Hadoop File System Shell Overview Invoke via: hadoop fs

A Crash Course in Apache Hadoop - Blancoblanco.io/assets/hadoop-workshop/crash-course-1.pdf · A Crash Course in Apache Hadoop. Event Outline 1. What is Hadoop 2. Current data challenges

Hadoop Deployment Manual - Hyadespleiades.ucsc.edu/doc/bright/hadoop-deployment-manual.pdf2.2 Ncurses Installation Of Hadoop Using cm-hadoop-setup ... •The Hadoop Deployment Manual

Hadoop , Hadoop , Hadoop !!!

BIGDATA HADOOP COURSE CONTENT · Industries using Hadoop. Data Locality. Hadoop Architecture. Map Reduce & HDFS. Using the Hadoop single node image (Clone). The Hadoop Distributed

A Crash Course in Apache Hadoop - Blanco · Why Hadoop? Benefits of the Hadoop Architecture Consolidates Data Integrates with many existing platforms Scalable and Affordable Real-Time

Hadoop Crash Course Hadoop Summit SJ

Docker based Hadoop provisioning - Hadoop Summit 2014

Data Science with Apache Spark - Crash Course - HS16SJ

Configuración para Hadoop Configuración de WPS para Hadoop · Configuración para Hadoop Versión 4.2 Introducción ¿Qué es Hadoop? Hadoop esun marco de trabajo del software de

Apache NiFi Crash Course - San Jose Hadoop Summit

Introduction to Hadoop and HDFS. Table of Contents Hadoop – Overview Hadoop Cluster HDFS.

· (Page views ? Hourly? Monthly Hadoop Node Hadoop Node Hadoop Camus Node Hadoop Node Hadoop Node Hadoop Node Hadoop Node Hadoop Node Ad-Hoc Analysis External Datastores Trends

Hue: The Hadoop UI - Hadoop Singapore

Why use Hadoop?, Challenges / Learning Hadoop & Average Salary of Hadoop Professional

Data Science Crash Course Hadoop Summit SJ

Hadoop Hand-on Lab: Installing Hadoop 2

Hadoop Interview Questions Version 2.0.0 Author: Hadoop ...kpbigdata.com/img/Hadoop_Interview_question.pdf · Hadoop Interview Questions Version 2.0.0 Author: Hadoop Learning Resource