Hands-on with Apache NiFi and MiNiFi - Berlin Buzzwords...Apache NiFi / Integration, or ingestion,...

Hands-onwithApacheNiFiandMiNiFi

AndrewPsaltis- @itmdata

BerlinBuzzwords2017

2 ©HortonworksInc.2011– 2016.AllRightsReserved

Dataflow and the associated problems


SimplisticViewofDataflows:Easy,Definitive

AcquireData

StoreData

DataFlow

ProcessAnalyzeData


RealisticViewofDataflows:Complex,Convoluted

AcquireData

StoreData

AcquireData

StoreData

StoreData

StoreData

StoreData

Processand

AnalyzeData

DataFlow

AcquireData

AcquireData


Movingdataeffectivelyishard

Standards:http://xkcd.com/927/


Apache NiFi


• Web-based User Interface for creating, monitoring, & controlling data flows

• Directed graphs of data routing and transformation

• Highly configurable - modify data flow at runtime, dynamically prioritize data

• Easily extensible through development of custom components

• Data Provenance tracks data through entire system

[1]https://nifi.apache.org/

Apache NiFiDataflow


NiFi - TerminologyÃ FlowFile

• Unitofdatamovingthroughthesystem• Content+Attributes(key/valuepairs)

Ã Processor• Performs thework,canaccessFlowFiles

Ã Connection• Linksbetweenprocessors• Queuesthatcanbedynamicallyprioritized

Ã ProcessGroup• Setofprocessorsandtheirconnections• Receivedataviainputports,senddataviaoutputports


FlowFiles arelikeHTTPdataHTTPData FlowFile

HTTP/1.1200OKDate:Sun,10Oct201023:26:07GMTServer:Apache/2.2.8(CentOS)OpenSSL/0.9.8gLast-Modified:Sun,26Sep201022:04:35GMTETag:"45b6-834-49130cc1182c0"Accept-Ranges:bytesContent-Length:13Connection:closeContent-Type:text/html

Helloworld!

StandardFlowFile AttributesKey:'entryDate’ Value:'FriJun1717:15:04EDT2016'Key:'lineageStartDate’Value:'FriJun1717:15:04EDT2016'Key:'fileSize’ Value:'23609'FlowFile AttributeMapContentKey:'filename’ Value:'15650246997242'Key:'path’ Value:'./’

BinaryContent*

Header

Content


FlowFiles &DataAgnosticism

Ã NiFi isdataagnostic!

Ã But,NiFi wasdesignedunderstandingthatusers

cancareaboutspecificsandprovidestooling

tointeractwithspecificformats,protocols,etc.

ISO8601- http://xkcd.com/1179/

Robustnessprinciple

Beconservativeinwhatyoudo,beliberalinwhatyouacceptfromothers


NiFi - TerminologyÃ FlowFile

• Unitofdatamovingthroughthesystem• Content+Attributes(key/valuepairs)

Ã Processor• Performsthework,canaccessFlowFiles

Ã Connection• Linksbetweenprocessors• Queuesthatcanbedynamicallyprioritized

Ã ProcessGroup• Setofprocessorsandtheirconnections• Receivedataviainputports,senddataviaoutputports


NiFi isbasedonFlowBasedProgramming(FBP)

FBPTerm NiFi Term DescriptionInformationPacket

FlowFile Each objectmovingthroughthesystem.

Black Box FlowFileProcessor

Performsthework, doingsomecombinationofdatarouting,transformation,ormediationbetweensystems.

BoundedBuffer

Connection Thelinkage betweenprocessors, actingasqueuesandallowingvariousprocessestointeractatdifferingrates.

Scheduler FlowController

Maintainstheknowledgeofhowprocessesareconnected, andmanagesthethreadsandallocationsthereofwhichallprocessesuse.

Subnet ProcessGroup

Asetofprocessesandtheirconnections,whichcanreceiveandsenddataviaports.Aprocess groupallowscreationofentirelynewcomponentsimplybycompositionofits components.


ApacheNiFi

KeyFeaturesandPrinciples

• Guaranteeddelivery• Databuffering

- Backpressure- Pressurerelease

• Prioritizedqueuing• FlowspecificQoS

- Latencyvs.throughput- Losstolerance

• Dataprovenance

• Recovery/recordingarollinglogoffine-grainedhistory

• Visualcommandandcontrol

• Flowtemplates• Pluggable/multi-role

security• Designedforextension• Clustering


Theneedfordataprovenance

For Operators• Traceability, lineage• Recovery and replayFor Compliance• Audit trail• RemediationFor Business / Mission• Value sources • Value IT investment

BEGIN

ENDLINEAGE


DataProvenance(NotjustLineage)

• Viewattributesandcontentatgivenpointsintime(beforeandaftereachprocessor)!!!

• Records,indexes,andmakeseventsavailablefordisplay


Provenance

§ Constrained§ High-latency§ Localizedcontext

§ Hybrid– cloud/on-premises§ Low-latency§ Globalcontext

SOURCES REGIONALINFRASTRUCTURE

COREINFRASTRUCTURE

Origin– attributionReplay– recovery

EvolutionoftopologiesLongretention

TypesofLineage• Event(runtime)• Configuration(designtime)


Theneedforfine-grainedsecurityandcompliance

It’s not enough to say you have encrypted communications• Enterprise authorization

services –entitlements change often

• People and systems with different roles require difference access levels

• Tagged/classified data


Security

§ Constrained§ High-latency§ Localizedcontext

§ Hybrid– cloud/on-premises§ Low-latency§ Globalcontext

SOURCES REGIONALINFRASTRUCTURE

COREINFRASTRUCTURE

Sign,encrypt,static(dataandcontrol)

TLS,obfuscation,dynamicentitlementsKerberos,PKI,AD/DS,etc.


BackPressure

• Configureback-pressureperconnection• BasedonnumberofFlowFilesortotalsizeof

FlowFiles• Upstreamprocessornolongerscheduledtorun

untilbelowthreshold


Prioritization

• Configure a prioritizer per connection• Determine what is important for your

data – time based, arrival order, importance of a data set

• Funnel many connections down to a single connection to prioritize across data sets

• Develop your own prioritizer if needed


Latency vs. Throughput

• Choose between lower latency, or higher throughput on each processor• Higher throughput allows framework to batch together all operations for

the selected amount of time for improved performance• Processor developer determines whether to support this by using

@SupportsBatching annotation


Extension/IntegrationPoints

NiFi Term DescriptionFlow FileProcessor

Push/Pull behavior.CustomUI

ReportingTask

Used topushdatafromNiFi tosomeexternalservice(metrics,provenance,etc..)

ControllerService

Usedtoenablereusablecomponents/ sharedservicesthroughouttheflow

RESTAPI Allowsclientstoconnecttopullinformation,changebehavior,etc..


NiFi Positioning


NiFiPositioning

ApacheNiFi/MiNiFi

ETL(Informatica,etc.)

EnterpriseServiceBus(Fuse,Mule,etc.)

MessagingBus

(Kafka,MQ,etc.)

ProcessingFramework(Storm,Spark,etc.)


ApacheNiFi/ProcessingFrameworks

NiFiSimpleeventprocessing• Primarilyfeeddataintoprocessing

frameworks,canprocessdata,withafocusonsimpleeventprocessing

• Operateonasinglepieceofdata,orincorrelationwithanenrichmentdataset(enrichment,parsing,splitting,andtransformations)

• Canscaleout,butscaleupbettertotakefulladvantageofhardwareresources,runconcurrentprocessingtasks/threads(processingterabytesofdataperdayonasinglenode)

⚠ Notanotherdistributedprocessingframework,buttofeeddataintothose

ProcessingFrameworks(Storm,Spark,etc.)Complexanddistributedprocessing• Complexprocessingfrommultiplestreams(JOIN

operations)

• Analyzingdataacrosstimewindows(rollingwindowaggregation,standarddeviation,etc.)

• Scaleouttothousandsofnodesifneeded

⚠ Notdesignedtocollectdataormanagedataflow


ApacheNiFi/MessagingBusServices

NiFiProvidedataflowsolution• Centralizedmanagement,fromedgetocore

• Greattraceability,eventleveldataprovenancestartingwhendataisborn

• Interactivecommandandcontrol– realtimeoperationalvisibility

• Dataflowmanagement,includingprioritization,backpressure,andedgeintelligence

• Visualrepresentationofglobaldataflow

⚠ Notamessagingbus,flowmaintenanceneededwhenyouhavefrequentconsumersideupdates

MessagingBus(Kafka,JMS,etc.)Providemessagingbusservice• Lowlatency

• Greatdatadurability

• Decentralizedmanagement(producers&consumers)

• Lowbrokermaintenancefordynamicconsumersideupdates

⚠ Notdesignedtosolvedataflowproblems(prioritization,edgeintelligence,etc.)

⚠ Traceabilitylimitedtoin/outoftopics,nolineage

⚠ Lackofglobalviewofcomponents/connectivities


ApacheNiFi/Integration,oringestion,Frameworks

NiFiEnduserfacingdataflowmanagementtool• Outoftheboxsolutionfordataflow

management

• Interactivecommandandcontrolinthecore,designanddeployontheedge

• Flexiblefailurehandlingateachpointoftheflow

• Visualrepresentationofglobaldataflowandconnectivities

• Nativecrossdatacentercommunication

• Dataprovenancefortraceability

⚠ Notalibrarytobeembeddedinotherapplications

Integrationframework(SpringIntegration,Camel,etc),ingestionframework(Flume,etc)Developerfacingintegrationtoolwithafocusondataingestion• Asetoftoolstoorchestrateworkflow

• Afixeddesignanddeploypattern

• Leveragemessagingbusacrossdisconnectednetworks

⚠ Developerfacing,customcodingneededtooptimize

⚠ Pre-builtfailurehandling,lackofflexibility

⚠ Noholisticviewofglobaldataflow

⚠ Nobuilt-indatatraceability


ApacheNiFi/ETLTools

NiFiNOTschemadependent• Dataflowmanagementforbothstructuredand

unstructureddata,poweredbyseparationofmetadataandpayload

• Schemaisnotrequired,butyoucanhaveschema

• Minimummodelingeffort,justenoughtomanagedataflows

• Dotheplumbingjob,maximizedevelopers’brainpowerforcreativework

⚠ NotdesignedtodoheavyliftingtransformationworkforDBtables(JOINdatasets,etc.).Youcancreatecustomprocessorstodothat,butlongwaytogotocatchupwithexistingETLtoolsfromuserexperienceperspective(GUIfordatawrangling,cleansing,etc.)

ETL(Informatica,etc.)Schemadependent• TailoredforDatabases/WH

• ETLoperationsbasedonschema/datamodeling

• Highlyefficient,optimizedperformance

⚠ Mustpre-prepareyourdata,timeconsumingtobuilddatamodeling,andmaintainschemas

⚠ Notgearedtowardshandlingunstructureddata,PDF,Audio,Video,etc.

⚠ Notdesignedtosolvedataflowproblems


Apache MiNiFi


ApacheMiNiFi

Ã NiFi livesinthedatacenter.Giveitanenterpriseserveroraclusterofthem.

Ã MiNiFi livesasclosetowheredataisbornandisaguestonthatdeviceorsystem

“LetmegetthekeypartsofNiFi closetowheredatabeginsandprovidebidirectionaldatatransfer"


RealitiesofcomputingoutsidethecomfortsofthedatacenterÃ Limited

– computingcapability– power/network

Ã Restrictedsoftwarelibrary/platformavailability

Ã NoUI

Ã Physicallyinaccessible

Ã Notfrequentlyupdated

Ã Competingstandards/protocols

Ã Scalability

Ã Privacy&Security


ApacheNiFiMiNiFiKeyFeatures

• Guaranteeddelivery• Databuffering

- Backpressure- Pressurerelease

• Prioritizedqueuing• FlowspecificQoS

- Latencyvs.throughput- Losstolerance

• Dataprovenance

• Recovery/recordingarollinglogoffine-grainedhistory

• Designedforextension

• DesignandDeploy• Warmre-deploys


MiNiFi:PrecedentfromNiFi

Ã ProvidesthesemanticsbetweentwoNiFi componentsacrossnetworkboundaries– Acustomprotocolforinter-NiFi communication– Secure,Extensible,LoadBalanced&ScalableDeliverytoCluster

Ã ExtractedouttoaclientlibrarywhichpowersintegrationintopopularframeworkslikeApacheSpark,ApacheStorm,ApacheFlink,andApacheApex

Ã AttributesandtheFlowFile formatmaintained

AquicklookatNiFi SitetoSite

https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#site-to-site



Ã Fine-grained,eventlevelaccessofinteractionswithFlowFiles– CREATE,RECEIVE,FETCH,SEND,DOWNLOAD,DROP,EXPIRE,FORK,JOIN…

Ã Capturestheassociatedattributes/metadataatthetimeoftheevent

Ã AmapofaFlowFile’s journeyandhowtheyrelatetootherFlowFiles inasystem– MiNiFi enablesustogetmoreandfurtherilluminatethemapofdataprocessing

Adeeperdiveintoprovenance

http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#data-provenance



RECEIVEevent


ApacheMiNiFi

Ã Thefeedbackloopislongerandnotguaranteed– RemovalofWebServerandUI

Ã Declarativeconfiguration– LendsitselfwelltoCMprocesses– Extensibleinterfacetosupportvaryingformats

• CurrentlyprovidedinYAML

Ã Reducedsetofbundledcomponents

DeparturesfromNiFi ingettingtherightfit


ApacheMiNiFi:Scoping

Ã Gosmall:Java–Writeonce,runanywhere*– FeatureparityandreuseofcoreNiFi libraries

Ã Gosmaller:C++–Writeonce**,runanywhere

Ã Gosmallest:Writen-manytimes,runanywhereLanguagelibrariestosupporttagging,FlowFile format,SitetoSiteprotocol,andprovenancegenerationwithoutaprocessingframework– Mobile:Android&iOS– LanguageSDKs

ProvideallthekeyprinciplesofNiFi invarying,smallerfootprints


Ã ConstrainedÃHigh-latencyÃ Localizedcontext

ÃHybrid– cloud/on-premisesÃ Low-latencyÃGlobalcontext

CoreInfrastructure

HarnessingDatainMotion

RegionalInfrastructure

Sources


Demo


Learnmoreandjoinus!

Apache NiFi sitehttps://nifi.apache.org

Subproject MiNiFi sitehttps://nifi.apache.org/minifi/

Subscribe to and collaborate [email protected]@nifi.apache.org

Submit Ideas or Issueshttps://issues.apache.org/jira/browse/NIFIhttps://issues.apache.org/jira/browse/MINIFI

Follow on Twitter@apachenifi


ThankYou

Hands-on with Apache NiFi and MiNiFi - Berlin Buzzwords...Apache NiFi / Integration, or ingestion,...

Documents

Transcript of Hands-on with Apache NiFi and MiNiFi - Berlin Buzzwords...Apache NiFi / Integration, or ingestion,...