Making Big Data Work

Making Big Data workLewis CrawfordPrincipal Architect @ the DataShed

thedatashed.co.uk

[email protected]

©theDataShedLimited 2015

Who am I?

• Forthelast3years,theDataShed hasbeenprovidingconsultancyservicestoavastarrayoflargeclients.Ourprimaryfocusisensuringthattechnologyandanalyticalstrategiesaretrulyalignedsothatbusinessescanleveragethelatestandgreatestintechnologytomodel,mineanddescribetheirdataasset.

• WewereworkingwithBigDatatechnologybeforethetermwascoined,wehaveexperiencedeliveringanalyticalsystemsdrivenbyPetabytedatasets,andhavedesigned,implementedandsupportedoneofthelargestreal-timedataintegrationandpredictiveanalyticsplatformsintheaviationworld.

• Ourmodelisbasedonusingasmallnumberofexceptionallyhighlyskilledindividualstodeliverdisruptiveandinnovativesolutionsinanagileanddelivery-focusedmanner.


So what is ‘Big Data’?


Why do Big Data projects fail?

ToomanypeoplethinkthatBigDatais:

“Thebeliefthatthemoredatayouhave,themoreinsightsandanswerswillriseautomaticallyfromthepoolofonesandzeros.”

GillPress,Forbes.com


How to make Big Data work?

1. Understandyourproblem

2. Applyappropriatetools

3. Automateeverything.


Real-time data


Continuous Integration Demo


Little Big Data


A problem closer to home…

• Everybusinessneedstounderstand:• Theirpotentialcustomersandmarket• Currentcustomers• Theirproductsandsales• Howandwhentheyengageprospectsandcustomers

• Analyticsanddataareexpensive• Manyofthemandatoryelementsareverysimilarforeveryone• TheDataShedisAnalyticsasaServiceandSingleCustomerViewasaService.


The deduplication problem…

• SMEhas250,000customers(twosystemsofrecord)• Toidentifyduplicatesbruteforceapproach: 31,249,875,000comparisons• Buildingasystemtoprocessaminimumof100clientsaday…• 3.1trillionrecordstocompareusing>10differentalgorithms

• Traditionalscaleupapproachwouldbeexpensive,andmakeslargeassumptionsaroundblockingandpartitioningrules• Asmalldataproblembutabigdatasolution?

Title FirstName Surname Address 1 Address2 Address3

Dr RJ Smith TwoOaks 112OldSt. CountyDurham

Mrs Robyn Smith 112OldStreet Durham DH15YJ


The Shed demo


How to make Big Data work?1. Understandyourproblem

• ’BigData’challengesaren’tnecessarilynew,howevermuchofthetechnology is• Articulateandcommunicate– focusondistillingyourproblemdown• Incremental improvementnotwholesalereplacement

2. Applyappropriate tools• Understandtheeconomics aswellasthetechnology• Newtechnologiesneedtobeevaluatedwithinthecontextofyourproblemscope• Newtechnologiesareenablers notdeliverables(#datalake)• ’BigData’technologyshouldbeseenascomplementarytoexistingtechnology

3. Automateeverything• Continuousintegrationtoincludeall testing• Containerisewherepossible• Measureeverything


If you really want to get involved…


Get your hands dirty

Ifyou’reinterestedinlearningmore,we’llbehostingahands-onlabseventinthenearfuture.

Sendyourdetailsto:Email:[email protected]:@thedatashed


Any questions?


Lewis CrawfordPrincipal Architect @ the DataShed

thedatashed.co.uk

[email protected]

Making Big Data Work

Technology

Transcript of Making Big Data Work