NEARING THE EVENT HORIZON.HADOOP WAS PREDICTABLE, WHAT’S NEXT?

Mike Miller (UW)_mlmilleratmit

September 28, 2011

Mike Miller

What I Am

Assistant Professor, Particle Physics(UW)

Cloudant Founder, Chief Scientist

Background: machine learning, analysis, big data, globally distributed systems

2

Mike Miller

What I Am Not

3

didn’t see these comingSuper luminal neutrinosRed Sox blow 9 game lead in SeptemberAmazon Silk...

But here I go anyway

Mike Miller

My First Postulate of Big-Data

What matters for google...... matters for the internet......and therefore matters for the enterprise...... will therefore be re-architected by Apache...... and therefore matters to you.

4

Google Matters

Mike Miller

Evidence

5

Business Week, 12/24/2007

Mike Miller

Evidence

5

Business Week, 12/24/2007

Mike Miller

Evidence

5

Business Week, 12/24/2007

Mike Miller

The Old Canon

• Google File System (the important one)http://labs.google.com/papers/gfs.html

• MapReduce (the big one)http://labs.google.com/papers/mapreduce.html

• BigTable (clone me!)http://labs.google.com/papers/bigtable.html

• Dynamo (ok, AWS. but masterless quorum) http://s3.amazonaws.com/AllThingsDistributed/sosp/amazon-dynamo-sosp2007.pdf

6

copy these. use these. print $$$

Mike Miller

So... is that it?

7

http://gigaom.com/cloud/democratizing-big-data-is-hadoop-our-only-hope/

Mike Miller

What’s Painful about MapReduce?

• Processing latencyNon-incremental, must re-slurp entire dataset every pass

• Ad-Hoc queriesBare metal interface, data import

• GraphsOnly a handful of graph problems amenable to MRhttp://www.computer.org/portal/web/csdl/doi/10.1109/MCSE.2009.120

8

Mike Miller

Enter The New Canon• Percolator

incremental processinghttp://research.google.com/pubs/pub36726.html

• Dremelad-hoc analysis querieshttp://research.google.com/pubs/pub36632.html

• PregelBig graphshttp://dl.acm.org/citation.cfm?id=1807184

9

Scalable, Fault Tolerant, Approachable

Mike Miller

Percolator: incremental processing• Replaced MapReduce as the tool to build search index

“However, reprocessing the entire web discards the work done in earlier runs and makes latency proportional to the size of the repository, rather than the size of the update.”

• Bigtable alone can’t do it“BigTable scales...but doesn’t provide tools to help programmers maintain data invariants in the face of concurrent updates.”

• ApplicabilityIncrementally updating dataComputational output can be broken down into small piecesComputation large in some dimension (data size, cpu, etc)

• Does it matter?“...Converting the indexing system to an incremental system ... reduced the averaging document processing latency by a factor of 100...”

10

Mike Miller

Percolator: incremental processing

• BigTable plus...

Transactionssnapshot isolation, locks

Timestamps

Notifications

Observersyour code to be run upon notification of an update

11

Mike Miller

Dremel: ad-hoc Query• Scalable, interactive ad-hoc query system for read-only nested

data“...capable of running aggregation queries over trillion-row tables in seconds.”

• ... on nested data structures in situWeb and scientific data is often non-relationalnested data (protobu!s) underlies most structured data at Google

• UsageDEFINE TABLE t AS /path/to/data/*SELECT TOP(signal1,100), COUNT(*) FROM t

• ApplicabilityAnalysis of crawled documentsTracking of install data for apps on Android MarketCrash reportsSpam analysis...

12

dream BI tool

Mike Miller

Dremel: ad-hoc Query• Ingredients

In situ dataSQL like interfaceServing trees for query executionColumn striped data

13

Mike Miller

Dremel: ad-hoc Query• Ingredients

In situ dataSQL like interfaceServing trees for query executionColumn striped data

13

Mike Miller

Dremel: ad-hoc Query• Ingredients

In situ dataSQL like interfaceServing trees for query executionColumn striped data

13

Mike Miller

Pregel: Big Graphs• Massively parallel processing of big graphs

billions of vertices, trillions of edges

• Bulk synchronous parallel modelsequence of vertex oriented iterationssend/receive messages from other vertex computationsread/modify state of vertex, outgoing edges, graph topology

• Expressive, easy to programdistribution details hidden behind abstract API

• Iterativecomputation continues until each vertex votes to terminate

• In productionPageRank 15 lines of code

14Nothing like this exists in open source

Mike Miller

Pregel: Big Graphs• Master “Name” node

connects processes for messaging

• Message Passingno remote procedures, reads

• Graph hashed across nodesvertex, outgoing edges stored in RAM

• Aggregators global mechanism for aggregationall but final reduce computed on node local data

• Checkpointing configurable, enables automatic recovery

15

Mike Miller

Pregel: Big Graphs• Master “Name” node

connects processes for messaging

• Message Passingno remote procedures, reads

• Graph hashed across nodesvertex, outgoing edges stored in RAM

• Aggregators global mechanism for aggregationall but final reduce computed on node local data

• Checkpointing configurable, enables automatic recovery

15

Mike Miller

Pregel: Big Graphs• Master “Name” node

connects processes for messaging

• Message Passingno remote procedures, reads

• Graph hashed across nodesvertex, outgoing edges stored in RAM

• Aggregators global mechanism for aggregationall but final reduce computed on node local data

• Checkpointing configurable, enables automatic recovery

15

Mike Miller

Lessons Learned

• Hire Je! Dean and Sanjay Ghemawat

• GFS enables everything

• There is massive opportunity on the horizon

16