DI&A Slides: Data Insights and Analytics Frameworks

The First Step in Information Management

www.firstsanfranciscopartners.com

Producedby:

MONTHLY SERIES

Broughttoyouinpartnershipwith:

January 5, 2017Data Insights and Analytics Frameworks

Welcometothenewseries

§ Thepurposeofournewseriesisto:

− GrowunderstandingonDataInsightsandAnalytics

− CovertheinsandoutsoftheBigData,Analytics,BusinessIntelligenceand reportinguniverse

− Focusonpractical,realistic,value

− Wanttobypassfluff,hypeandfalsepromises

− Needyourfeedback− UseQ&A

pg 2© 2017 First San Francisco Partners www.firstsanfranciscopartners.com

DataLakevs.Data

Warehouse

Descriptive,Prescriptive

andPredictiveAnalytics

GoverningQualityAnalytics

TheRoleofaDataScientist(InterviewwithaCDS)

Analytics,BIandDataScience:

What’stheProgression?

Topicsfortoday’swebinar

FrameworksdefinedEnterpriseanalyticsarchitectureOverviewofstandarddatainsightsandanalyticscomponentsBigDataSandboxReal-timeAnalyticsFromLegacyarchitecturestodatainsightKeytakeawaysQ&A

pg 3

Frameworksdefined

Enterpriseanalyticsarchitecture

• BigData• Sandbox• Real-timeAnalytics• LegacyArchitectures

Overviewofstandarddatainsightsandanalyticscomponents

Keytakeaways

Q&A

© 2017 First San Francisco Partners www.firstsanfranciscopartners.com

Frameworksdefined

▪ Thestructurefordeliveringandgettingvalueoutofyourdataandanarchitecturefordecision-making,includingorganizationalmodels

▪ Yourenterpriseanalyticsarchitectureneedstoreflectholisticthinking▪ Yourstartingpointandbusinessneedsdeterminehowyouprogress,notapre-definedcurve

▪ Organizationsthatcanbarelydeliveranaccurateproductionreportaredoingpredictiveanalytics– Right?Wrong?

pg 4

Predictive

ManagingProactive

Operating

DataInsightandAnalyticsMaturity© 2017 First San Francisco Partners www.firstsanfranciscopartners.com

TheDataInsightandAnalyticsFramework

DataInsightandAnalyticsFramework

DataInsightandAnalyticsStrategy

TechnologyInfrastructure

DataInsightandAnalyticsOperatingModel

Compo

nents “BestFit”Data

Architecture DataQuality DemandManagement

Presentation DataWrangling MetadataManagement

GOVERNANCE

ORGANIZATIONAL ALIGNMENT

pg 5

Sampleenterpriseanalyticsarchitecture

ODS*

DM Big Data*

Sandbox

Securityandgovernance

Presentation(visualization,reports,algorithms,queries)

Dataingestion

Operationsscheduling,

managem

ent,DataQuality,ControlsSources

* = Includes Real Time ETLandMovement

DW*

Metadata

Data Lake


ODS*

DM

Big Data*

Sandbox

Securityandgovernance


Dataingestion

Operationsscheduling,m

anagement,DataQ

uality,Controls

Sources

ETLandMovement

DW*

Metadata

Data Lake

Enterpriseanalyticsarchitecture– BigData

§ MorethantheBigData“stack”

§ Nolongerlinear– ProductiontoAccess

§ Arrangedbylatency,access,intendedvalue,datavelocity,datavolumeanddatamovementcapacity1. Standard“BigData”2. Sandbox3. Realtime4. Heritage

pg 7

12*3

4


BigData

§ Components− DataSources− Ingestion− Structuring− AnalyticsandVisualization− Metadata

§ HighPriorityConcerns− Metadata– Technical,Business,Lineage,Meaning,Interpretation− Securityandprivacy- Accessandusagemustbemanaged

accordingtorisk,permissions,policy,contractualagreements− DataGovernance

§ Oversightofsemantics,lineage,quality

− Latency,access,usage§ Persistent§ Typeofdatastructure– Hive,Hbase

pg 8

DataSources IngestionandTransformation

StructuringData

AnalyticsandVisualization

MonitorTechnicalMetadataBusinessMetadata


Usecase– BigData

§ Telecom− Analyze500discretedataelementsincludingsupportcallpatterns,lateor

delinquentpaymentsandotherongoingvitalsignsviapredictiveanalytics− Identify“churn”prospectsandtakestepstopreventit

§ Results− 47%reductionincustomerchurn,protecting$15millioninrevenue− Predictiveanalyticshasspreadorganicallytootherpartsofthecompany,

includingcollections


Sandbox

§ Components– SimilartoBigData− Standalonesandboxisalsoarelevantcomponent− Ingestion– Batch− AnalyticsandVisualization- DataScientist/Analystonly

§ HighPriorityConcerns− Data

§ Discovery§ Understanding§ Standardization§ Usefulness

− SecurityandPrivacy§ Rawaspectimpliesnocontrols§ Controlthedata,nottheenvironment

− DataGovernancefocus,Usage,access− Metadata,provenanceandpedigree

§ Latency,access,usage− Sandboxequalsnon-persistent,non-production− Self-service− Housekeeping

pg 10

Stand Alone Sandbox

DataSources IngestionandTransformationAnalyticsandVisualization



Usecase– Sandbox

§ PredictiveMaintenance,Manufacturing− Collectedmachinesensordata− Createdasandboxenvironmenttocentralizethepartsfailureanalysis− Combinedsensordatawithoperationaldata− Afterfindinginsights,operationalizetheprocessesinthelineofbusiness

§ Results− Shortertime-to-insight;Ittookonlythreeweeks(from6months)todevelopapartsfailure

predictionalgorithm


Real-timeanalytics

§ Components- similartoregularBigData− Streamingprocessinganaddedcomponent− Realtimecapabilityonnon-BigDataaswell

§ HighPriorityConcerns− Speedofingestion

§ Real-timeDataIngestion− DataStreaming− DataMessaging− In-memorydatabaseforextremelowlatencyrequirements

− Securityandprivacy− DataGovernanceFocus

§ Metadata§ Compliance

− Latency,access,usage− Real-timeDataQuery

§ EnterpriseQueryandReporting§ FastQueryDatabasetostoreanalyticalresults§ Agents,messaging,newevents§ Flexiblepersistencyandaccessibility§ Verylowlatency,highperformance

pg 12

DataSources

IngestionandTransformation

StructuringData

AnalyticsandVisualization


StreamingData

Processing

Real time DW, or ODS


Usecase– Real-timeanalytics

§ Health− Automaticallysiftsthroughmillionsofpostsondozensofsocialmediasites,localnews

reports,medicalworkers’socialnetworksandgovernmentwebsitestotrackinstancesofdisease

− Continuallyplotsdiseasehotspotsonamap

§ Results− Identifiedaclusterof“mysteryhemorrhagicfever”inGuineaoveraweekbeforethe

MinistryofHealthofGuineanotifiedtheWorldHealthOrganization(WHO),thatadaylaterconfirmedtheEbola outbreak


Relevanceoflegacyortraditionaldatainsight

§ Legacystructuresstillhaverelevance− Reporting− StandardBI

§ Components− Familiarnames– ETL,DW,ODS,DM− ManyaspectsofBigDatatechnologyarenotrelevantto

manydatauses

§ TraditionalConcerns− ETLvs.webservicespipelinesviadatalayer− Understandingneedfortraditionaluses:

§ Departmentaluse§ Historicalreporting§ Operationalandad-hocreporting

− Supportofmulti-latency,historical,operational,etc.,requirements

pg 14

ODS*

DM


Sources

ETLandMovement

DW*


Usecase– Legacyortraditionaldatainsight

§ Healthcare− Adatawarehousetofacilitateinformationandbestpracticessharingbetweenthousandsof

providersandresearchprofessionals.− Alsodeployedpredictiveanalyticsandartificialintelligencetoderivebetterinsightsfrom

ElectronicHealthRecordsandimprovepatientoutcome.

§ Results− 400,000patientrecordscentralizedinasingledatawarehousewhichcanscaleupto20

millionrecords.− 42%anticipatedimprovementinpatientoutcomeswithArtificialIntelligence− 58%anticipatedreductionincostperunitofoutcomechange


Keytakeaways

§ Referencearchitecturesarejustthat−Youmaynotneedalake,datawarehouseorsandbox

§ Avoidcobblingtogethertechnicalcomponents

§ Plantomatchyourarchitecturetoneedsandusage,vs.existingcomponents

§ WebServicesareanimportanttool− Ifusingservices,pleaseconsideradistinctdatalayer


Q&A


pg 18

ThankyouandHappyNewYear!SeeyouThursday,February2forDataLakevs.DataWarehouse

JohnLadley@[email protected]

KelleO’Neal@[email protected]


DI&A Slides: Data Insights and Analytics Frameworks

Technology

Transcript of DI&A Slides: Data Insights and Analytics Frameworks