Introduction to HDF 3.0

1 ©HortonworksInc.2011– 2017AllRightsReserved

TimothySpann2017FutureofData– PrincetonMeetupJune20,2017HostedbyTRACIntermodal

Introduction to HDF 3.0


• Schema Registry – Milind Pandit• HDFStreamingUpdates– TimSpann

• EDWOptimizationwithHadoopandHDF- GregoryCKeys,PhD.


AmbariIntegration


FormatandSchemaAwareEfficientFlowManagement

Ã Provideprocessorsforschemaawarerecordstructureforcommonprocessingpatterns– Split,Enrich,Partition,Convert,Query (SQLqueriespoweredbyApacheCalcite)– Put/GetrecordsbetweenNiFi andKafka,ElasticSearch,RDMBS(moresoon)– Easybridgingto/fromColumnardataformatslikeORCorParquet

Ã Separateformat/schemaspecificlogicintoextensiblerecordreadersandwriters– Developerscanwritenewreaders/writers– Userscancreatenewreaders/writerswithscriptingliveinproduction!

Ã Sowhat?– Formatandschemaawareprocessing*with*genericreusablecomponents– Maintainsfullprovenance/lineagetrail– Dramaticspeed/efficiencyincreasepernode– IntegrationwithHortonworksSchemaRegistryandextensibleforothers


RecordReaderCS


RecordWriterCS


‘QueryRecord’Processor– Treatstreamingrecordsastables


ComponentVersioning


StreamProcessing– IntroducingStreamingAnalyticsManager(SAM)

StreamingAnalyticsManager

AbrandnewproductmoduleintheHDFstacktodesign,develop,deployandmanagestreaminganalyticsappwithadrag-and-dropuserexperience


SAM- WriteComplexStreamingApplicationsWithNoCode


Ã AbrandnewproductmoduleintheHDFstacktodesign,develop,deployandmanagestreaminganalyticsappwithdrag-and-dropparadigm– Buildstreaminganalyticsapplicationsthatdoeventcorrelation,contextenrichment,complex

patternmatching,analyticalaggregationsandcreationofalerts/notificationswheninsightsarediscovered.

– Givethecodersthepowertoaddkeyfunctionsandextendtheplatform (addcustomsinks,processors,spouts,etc..)


SAM’sValueProposition

Ã Buildanddeploycomplexstreamapplicationswithoutwritinganycode

Ã Onlyopensourcetoolinthemarket withgraphicalprogrammingparadigm

Ã Speedtime-to-markettobuildcomplexstreaminganalyticsapplications

Ã Buildstreaminganalyticsapplicationswithoutspecializedskillsets.

Ã Decoupledataformatfromthestreamingapplicationitselfwhilebeingschemaaware

Ã Supportmultipleunderliningstreamingengines


StreamBuilderModuleforAppDevelopers

Ã Buildercomponents,shownonthecanvaspalette,arethebuildingblocksusedbytheappdevelopertobuildstreamingapplications

Ã Draganddroptobuildaworkingstreamingapplicationwithoutwritingasinglelineofcode

Ã 4TypesofComponents:Sources,Processors,SinksandCustom


StreamInsightModuleforBusinessAnalysts

Ã Atooltocreatereal-timeanalyticsdashboards,chartsandgraphs

Ã 30+visualizationchartsoutoftheboxwithcustomizationcapability

Ã DruidistheAnalyticsEnginethatpowerstheStreamInsightModule.


StreamOpsModuleforITOperations

Ã Createandmanagedifferentenvironmentsinwhichindividualstreamingapplicationswillbebuilt

Ã EnvironmentsconsistsofservicessuchasHDFS,Kafka,Stormfromdifferentservicepools

Ã Savetimeandreduceoperationaloverheadwithsamedraganddropparadigmasthestreambuildmodule


StreamBuilderModuleforAppDevelopers

Ã Buildercomponents,shownonthecanvaspalette,arethebuildingblocksusedbytheappdevelopertobuildstreamingapps.

Ã Draganddroptobuildaworkingstreamingapplicationwithoutwritingasinglelineofcode.

Ã 4TypesofComponents:Sources,Processors,SinksandCustom


SAMisAllaboutDoingReal-TimeAnalyticsontheStream

Real-TimePrescriptiveAnalytics

Real-TimeAnalytics

Real-TimePredictiveAnalytics

Real-TimeDescriptiveAnalytics

Whatshouldwedorightnow?

Whatcouldhappennow/soon?

Whatishappeningrightnow?


Real-TimePrescriptiveAnalytics

Ã Question:Whatshouldwedorightnow?

Ã Context:Itisrainy,thedriverisbeenontheroadfor12hoursandhehas30highspeedingalertsovera3minutewindowinthelast2hours.

Ã Answer:DispatcharadiocalltotheDrivertoslowdown


Real-TimePredictiveAnalytics

Ã Question:NoviolationeventsbutwhatmighthappenthatIneedtobeworriedabout?

Ã Mydatascienceteamhasamodelthatcanpredictthatbasedon– Weather– Roads– DriverHRinfolikedrivercertificationstatus,wagePlan– Drivertimesheetinfolikehours,andmilesloggedoverthelastweek


BuildingthePredictiveModelonHDP

Exploresmallsubsetofeventstoidentifypredictivefeaturesandmakeahypothesis.E.g.hypothesis:“foggyweathercausesdriverviolations”

1

IdentifysuitableMLalgorithmstotrainamodel– wewilluseclassificationalgorithmsaswehavelabeledeventsdata

2

TransformenrichedeventsdatatoaformatthatisfriendlytoSparkMLlib – manyMLlibsexpecttrainingdatainacertainformat

3

TrainalogisticclassificationSparkmodelonYARN,withaboveeventsastraininginput,anditeratetofinetunegeneratedmodel

4


LogisticalRegressionModel


ScoringthePredictiveModelonHDF

UseSAM’senrich/customprocessorstoenrichtheeventwiththefeaturesrequiredforthemodel6

EnrichwithFeatures

UseSAM’sprojection/customprocessorstotransform/normalizethestreamingeventandthefeaturesrequiredforthemodel

7Transform/Normalize

UseSAM’sPMMLprocessortoscorethemodelforeachstreameventwithitsrequiredfeatures8

ScoreModel

UseSAM’sruleandnotificationprocessorstoalert,notifyandtakeactionusingtheresultsofthemodel9

Alert/Notify/Action

ExporttheSparkMllib modelandimportintotheHDF’sModelRegistry5 Model

Registry


SAM’sModelRegistryandPMMLProcessor

Ã ModelRegistry– Samhasrepositorytostore

andmanagePMMLbasedpredictivemodels

– Firstclassfeatureslikeversion,evolutionpolicies,etc,willbeaddedinfuturerelease

Ã PMMLProcessor– Processorthatcanusemodel

fromtheregistryandscorethemodelsbasedontheinputstreamofeventscomingin


SAMExtensibility:CustomProcessors,UDF,UDAFs

Ã CustomComponents– Mostuserswillwanttobuildcustomcomponentstomeet

certainrequirements.– SAMprovidestheabilitytoaddbuildcustomcomponent

usingtheSAMSDK– ThejarsthencanthenbeuploadedinSAMviatheUser

Interface

Ã 3TypesofCustomComponents– CustomProcessors– CustomUDF

• UserdefinedfunctionsthatareusedbytheProjectionprocessor

– CustomUDAFs• Userdefinedaggregatefunctionsthatareusedbythe

Aggregateprocessor.– SDKcanbeusedtocreatecustomUDFfunctionsfor

windowedaggregations


StreamingSplitJoinPattern

Ã 3Enrichmentshavetoperformedontheeventstreamtofeedintomodel:– FromLat,Longandtime,queryweather

conditions– FromdriverId,lookupinformationabout

driver’scertificationandwageplan– FromdriverId,lookupinformationabouthow

manymilesandhourswasonthedriverontheroadlastweek

Ã StreamingSplitJoinPattern– ComplexPatternthatallowsparallelprocessing

todecreaselatency(UsedbyApacheMetronextensively)

1. CreateasplitJoin Key2. Splitthestreamintonwherenisthenumber

ofdifferentenrichmentsyouwanttodo3. JointhenstreamsbasedonthesplitJoinKey

ComplexpatterntoimplementthatSAMallowstheusertodo

simplywithnocode!


StreamInsightModuleforBusinessAnalysts

Ã Atooltocreatetime-seriesandreal-timeanalyticsdashboards,chartsandgraphs

Ã 30+visualizationchartsoutoftheboxwithcustomizationcapability

Ã DruidistheAnalyticsEnginethatpowerstheStreamInsightModule.


SetUpAnEnvironmentforSAM


HortonworksSAMCanvastobuildtheStreamingAnalyticsAppwithoutwritingalineofcode


HortonworksSAMAppDashboard


SchemaRegistryDashboardandDetailsofOneSchema


Contact:

[email protected]/futureofdata-princeton

community.hortonworks.com/users/9304/tspann.html


HortonworksCommunityConnection

Read access for everyone, join to participate and be recognized

• FullQ&APlatform(likeStackOverflow)

• KnowledgeBaseArticles

• CodeSamplesandRepositories

Introduction to HDF 3.0

Technology

Transcript of Introduction to HDF 3.0