IG DATA SYSTEMSpages.cs.wisc.edu/~paris/cs564-f16/lectures/lecture-18.pdf– OLAP (decision support...

28
BIG D ATA SYSTEMS CS 564- Fall 2016 ACKs: Magda Balazinska

Transcript of IG DATA SYSTEMSpages.cs.wisc.edu/~paris/cs564-f16/lectures/lecture-18.pdf– OLAP (decision support...

Page 1: IG DATA SYSTEMSpages.cs.wisc.edu/~paris/cs564-f16/lectures/lecture-18.pdf– OLAP (decision support queries) • MapReduce – first developed by Google, published in 2004 – only

BIG DATA SYSTEMS

CS564- Fall2016

ACKs:MagdaBalazinska

Page 2: IG DATA SYSTEMSpages.cs.wisc.edu/~paris/cs564-f16/lectures/lecture-18.pdf– OLAP (decision support queries) • MapReduce – first developed by Google, published in 2004 – only

BIG DATA

Definitionfromindustry:• highvolume• highvariety• highvelocity

2CS564[Fall2015]- ParisKoutris

Page 3: IG DATA SYSTEMSpages.cs.wisc.edu/~paris/cs564-f16/lectures/lecture-18.pdf– OLAP (decision support queries) • MapReduce – first developed by Google, published in 2004 – only

VOLUME

• Databasesparallelizeeasily;techniquesavailablefromthe80’s(GAMMAproject)– datapartitioning– parallelqueryprocessing

• SQLisembarrassinglyparallel

3CS564[Fall2015]- ParisKoutris

Page 4: IG DATA SYSTEMSpages.cs.wisc.edu/~paris/cs564-f16/lectures/lecture-18.pdf– OLAP (decision support queries) • MapReduce – first developed by Google, published in 2004 – only

VARIETY

• complexworkloads:– MachineLearningtasks:e.g.clickprediction,topicmodeling,SVM,k-means

• varioustypesofdata:– textdata– semi-structureddata– graphdata– multimedia(video,photos)

4CS564[Fall2015]- ParisKoutris

Page 5: IG DATA SYSTEMSpages.cs.wisc.edu/~paris/cs564-f16/lectures/lecture-18.pdf– OLAP (decision support queries) • MapReduce – first developed by Google, published in 2004 – only

VELOCITY

• dataisgeneratedveryfastandneedstobeprocessedveryfast– realtimeanalytics– datastreaming(eachdataitemcanbeprocessedonlyonce!)

5CS564[Fall2015]- ParisKoutris

Page 6: IG DATA SYSTEMSpages.cs.wisc.edu/~paris/cs564-f16/lectures/lecture-18.pdf– OLAP (decision support queries) • MapReduce – first developed by Google, published in 2004 – only

ANOTHER V:VERACITY

Thedatacollectedisoftenuncertain• inconsistentdata• incompletedata• ambiguousdata

Example:sensordata

6CS564[Fall2015]- ParisKoutris

Page 7: IG DATA SYSTEMSpages.cs.wisc.edu/~paris/cs564-f16/lectures/lecture-18.pdf– OLAP (decision support queries) • MapReduce – first developed by Google, published in 2004 – only

DATA LANDSCAPE

7CS564[Fall2015]- ParisKoutris

Page 8: IG DATA SYSTEMSpages.cs.wisc.edu/~paris/cs564-f16/lectures/lecture-18.pdf– OLAP (decision support queries) • MapReduce – first developed by Google, published in 2004 – only

SOME EXAMPLES

• Greenplum:foundedin2003acquiredbyEMCin2010.Aparallelshared-nothingDBMS

• Vertica:foundedin2005andacquiredbyHPin2011.Aparallelcolumn-storeshared-nothingDBMS

• AsterData:foundedin2005acquiredbyTeradatain2011.Aparallel,shared-nothing,MapReduce-baseddataprocessingsystem

• Netezza:foundedin2000andacquiredbyIBMin2010.Aparallelshared-nothingDBMS

8CS564[Fall2015]- ParisKoutris

Page 9: IG DATA SYSTEMSpages.cs.wisc.edu/~paris/cs564-f16/lectures/lecture-18.pdf– OLAP (decision support queries) • MapReduce – first developed by Google, published in 2004 – only

WE WILL SEE 2APPROACHES

• Paralleldatabases,startedatthe80s– OLTP(transactionprocessing)– OLAP(decisionsupportqueries)

• MapReduce– firstdevelopedbyGoogle,publishedin2004– onlyfordecisionsupportqueries– ecosystemaroundit:Hadoop,PigLatin,Hive,…

9CS564[Fall2015]- ParisKoutris

Page 10: IG DATA SYSTEMSpages.cs.wisc.edu/~paris/cs564-f16/lectures/lecture-18.pdf– OLAP (decision support queries) • MapReduce – first developed by Google, published in 2004 – only

PARALLEL DBMS

• Thegoalistoimproveperformancebyexecutingmultipleoperationsinparallel(scale-out)

• Terminologytomeasureperformance:– Speed-up:usingmoreprocessors,howmuchfasterdoesthetaskrun(ifproblemsizeisfixed)?

– Scale-up:usingmoreprocessors,doesperformanceremainthesameasweincreasetheproblemsize?

10CS564[Fall2015]- ParisKoutris

Page 11: IG DATA SYSTEMSpages.cs.wisc.edu/~paris/cs564-f16/lectures/lecture-18.pdf– OLAP (decision support queries) • MapReduce – first developed by Google, published in 2004 – only

SCALE-UP VS SCALE-OUT

Scale-up• usingmorepowerfulmachines,moreprocessors/RAMpermachine

Scale-out• usingalargernumberofservers

11CS564[Fall2015]- ParisKoutris

Page 12: IG DATA SYSTEMSpages.cs.wisc.edu/~paris/cs564-f16/lectures/lecture-18.pdf– OLAP (decision support queries) • MapReduce – first developed by Google, published in 2004 – only

ARCHITECTURES

• Sharedmemory– nodesshareRAM+disk– easytoprogram,expensivetoscale

• Shareddisk– nodesaccessthesamedisk,hardtoscale

• Sharednothing– nodeshavetheirownRAM+disk– connectedthroughafastnetwork

12CS564[Fall2015]- ParisKoutris

Page 13: IG DATA SYSTEMSpages.cs.wisc.edu/~paris/cs564-f16/lectures/lecture-18.pdf– OLAP (decision support queries) • MapReduce – first developed by Google, published in 2004 – only

PARALLEL QUERY EVALUATION

• Inter-queryparallelism:– eachqueryrunsononeprocessor

• Inter-operatorparallelism:– eachqueryrunsonmultipleprocessors– anoperatorrunsononeprocessor

• Intra-operatorparallelism:– Anoperatorrunsonmultipleprocessors

13CS564[Fall2015]- ParisKoutris

Page 14: IG DATA SYSTEMSpages.cs.wisc.edu/~paris/cs564-f16/lectures/lecture-18.pdf– OLAP (decision support queries) • MapReduce – first developed by Google, published in 2004 – only

PARALLEL DATA STORAGE

Horizontaldatapartitioning• block partitioned• hash partitioned• rangepartitioned

Uniformvsskewedpartitioning

14CS564[Fall2015]- ParisKoutris

Page 15: IG DATA SYSTEMSpages.cs.wisc.edu/~paris/cs564-f16/lectures/lecture-18.pdf– OLAP (decision support queries) • MapReduce – first developed by Google, published in 2004 – only

PARALLEL QUERY EVALUATION

• ParallelSelection

• ParallelJoin– hashjoin– broadcastjoin

15CS564[Fall2015]- ParisKoutris

Page 16: IG DATA SYSTEMSpages.cs.wisc.edu/~paris/cs564-f16/lectures/lecture-18.pdf– OLAP (decision support queries) • MapReduce – first developed by Google, published in 2004 – only

MAPREDUCE

• Google[Dean2004]• Opensourceimplementation:Hadoop• MapReduce:– high-levelprogrammingmodelandimplementationforlarge-scaleparalleldataprocessing

– designedtosimplifytaskofwritingparallelprograms

16CS564[Fall2015]- ParisKoutris

Page 17: IG DATA SYSTEMSpages.cs.wisc.edu/~paris/cs564-f16/lectures/lecture-18.pdf– OLAP (decision support queries) • MapReduce – first developed by Google, published in 2004 – only

MAPREDUCE

• HidesmessydetailsinMapReduce runtimelibrary– automaticparallelization– loadbalancing– networkanddisktransferoptimizations– handlingoffailures– robustness

17CS564[Fall2015]- ParisKoutris

Page 18: IG DATA SYSTEMSpages.cs.wisc.edu/~paris/cs564-f16/lectures/lecture-18.pdf– OLAP (decision support queries) • MapReduce – first developed by Google, published in 2004 – only

MAPREDUCE PIPELINE

• readthepartitioneddata(HDFS,GFS)• Map:extractsomethingyoucareaboutfromeachrecord

• ShuffleandSort(donebythesystem)• Reduce:aggregate,summarize,filter,transform• writetheresults

18CS564[Fall2015]- ParisKoutris

Page 19: IG DATA SYSTEMSpages.cs.wisc.edu/~paris/cs564-f16/lectures/lecture-18.pdf– OLAP (decision support queries) • MapReduce – first developed by Google, published in 2004 – only

MAPREDUCE DATAFLOW

19CS564[Fall2015]- ParisKoutris

source:Hadoop– TheDefinitiveGuide,byTomWhite

Page 20: IG DATA SYSTEMSpages.cs.wisc.edu/~paris/cs564-f16/lectures/lecture-18.pdf– OLAP (decision support queries) • MapReduce – first developed by Google, published in 2004 – only

DATA MODEL

• Afile=abagof(key,value)pairs

• AMapReduce program:– Input:abagof(inputkey,value)pairs– Output:abagof(outputkey,value)pairs

20CS564[Fall2015]- ParisKoutris

Page 21: IG DATA SYSTEMSpages.cs.wisc.edu/~paris/cs564-f16/lectures/lecture-18.pdf– OLAP (decision support queries) • MapReduce – first developed by Google, published in 2004 – only

THE MAP FUNCTION

UserprovidestheMAP function:• Input:(inputkey,value)• Output:bagof(intermediatekey,value)

Thesystemappliesthemapfunctioninparalleltoall(inputkey,value)pairsintheinputfile

21CS564[Fall2015]- ParisKoutris

Page 22: IG DATA SYSTEMSpages.cs.wisc.edu/~paris/cs564-f16/lectures/lecture-18.pdf– OLAP (decision support queries) • MapReduce – first developed by Google, published in 2004 – only

THE REDUCE FUNCTION

UserprovidestheREDUCE function:• Input:(intermediatekey,bagofvalues)• Output:bagof(outputkey,values)

Thesystemgroupsallpairswiththesameintermediatekey,andpassesthebagofvaluestotheREDUCEfunction

22CS564[Fall2015]- ParisKoutris

Page 23: IG DATA SYSTEMSpages.cs.wisc.edu/~paris/cs564-f16/lectures/lecture-18.pdf– OLAP (decision support queries) • MapReduce – first developed by Google, published in 2004 – only

EXAMPLE:WORD COUNT

• Countthenumberofoccurrencesofeachwordinalargecollectionofdocuments

• EachDocument– key=documentid(did)– value=setofwords(word)

23CS564[Fall2015]- ParisKoutris

Page 24: IG DATA SYSTEMSpages.cs.wisc.edu/~paris/cs564-f16/lectures/lecture-18.pdf– OLAP (decision support queries) • MapReduce – first developed by Google, published in 2004 – only

MAPREDUCE JOBS

• AMapReduce jobconsistsofonesingle“query”– e.g.countthewordsinalldocs

• Morecomplexqueriesmayconsistofmultiplejobs

24CS564[Fall2015]- ParisKoutris

Page 25: IG DATA SYSTEMSpages.cs.wisc.edu/~paris/cs564-f16/lectures/lecture-18.pdf– OLAP (decision support queries) • MapReduce – first developed by Google, published in 2004 – only

MAPREDUCE ECOSYSTEM

Lotsofextensionstoaddresslimitations:• CapabilitiestowriteDAGsofMapReduce jobs• Declarativelanguages• Mostcompaniesusebothtypesofengines(MRandDBMS),withincreasedintegration

• PotentialreplacementtoMapReduce:Spark

25CS564[Fall2015]- ParisKoutris

Page 26: IG DATA SYSTEMSpages.cs.wisc.edu/~paris/cs564-f16/lectures/lecture-18.pdf– OLAP (decision support queries) • MapReduce – first developed by Google, published in 2004 – only

MAPREDUCE ECOSYSTEM

PIGLatin(Yahoo!)• Newlanguage,likeRelationalAlgebra• opensourceHive(Facebook)• SQL-likelanguage• opensourceSQL/Tenzing (Google)• SQLonMR• Proprietary– morphedintoBigQuery

26CS564[Fall2015]- ParisKoutris

Page 27: IG DATA SYSTEMSpages.cs.wisc.edu/~paris/cs564-f16/lectures/lecture-18.pdf– OLAP (decision support queries) • MapReduce – first developed by Google, published in 2004 – only

PARALLEL DBMSVS MAPREDUCE

ParallelDBMS:• Relationaldatamodelandschema• Declarativequerylanguage:SQL• Caneasilycombineoperatorsintocomplexqueries• Queryoptimization,indexing,andphysicaltuning• Streamsdatafromoneoperatortothenextwithoutblocking

27CS564[Fall2015]- ParisKoutris

Page 28: IG DATA SYSTEMSpages.cs.wisc.edu/~paris/cs564-f16/lectures/lecture-18.pdf– OLAP (decision support queries) • MapReduce – first developed by Google, published in 2004 – only

PARALLEL DBMSVS MAPREDUCE

MapReduce:• datamodelisafilewithkey-valuepairs• noneedto“loaddata”beforeprocessing• easytowriteuser-definedoperators• caneasilyaddnodestothecluster• intra-queryfault-tolerancethankstoresultsondisk• Arguablymorescalable,butalsoneedsmorenodes

28CS564[Fall2015]- ParisKoutris


Itti: CS564 - Brain Theory and Artificial Intelligence. Memory and Consciousness Itti: CS564 - Brain Theory and Artificial Intelligence University of Southern.

Itti: CS564 - Brain Theory and Artificial Intelligence. Memory and Consciousness Itti: CS564 - Brain Theory and Artificial Intelligence University of Southern.

HAF F16 Supplement

HAF F16 Supplement

SABMiller F16 Half Year

SABMiller F16 Half Year

NIFB F16 FINAL PAPER

NIFB F16 FINAL PAPER

Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 21. Reinforcement Learning 1 Michael Arbib: CS564 - Brain Theory.

Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 21. Reinforcement Learning 1 Michael Arbib: CS564 - Brain Theory.

CS564:DATABASE# MANAGEMENTSYSTEMSpages.cs.wisc.edu/~paris/cs564-f15/lectures/lecture-01.pdfABOUTME# • undergrad$in$Athens,$Greece$ • Ph.D.inUniversityofWashington(theotherUW) •

CS564:DATABASE# MANAGEMENTSYSTEMSpages.cs.wisc.edu/~paris/cs564-f15/lectures/lecture-01.pdfABOUTME# • undergrad$in$Athens,$Greece$ • Ph.D.inUniversityofWashington(theotherUW) •

F16 Fighting Falcon Facts

F16 Fighting Falcon Facts

Haf Flight Manual f16

Haf Flight Manual f16

F16 Catalog

F16 Catalog

1995 WC321RB F15 F16 - winnebago.com

1995 WC321RB F15 F16 - winnebago.com

F16 MLU M1 Pilot's Manual

F16 MLU M1 Pilot's Manual

F16 Brochure

F16 Brochure

SABMiller F16 Full Year

SABMiller F16 Full Year

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 7. Object Recognition CS564 – Lecture 7. Object Recognition and Scene.

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 7. Object Recognition CS564 – Lecture 7. Object Recognition and Scene.

Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 1 1 L. Itti: CS564 - Brain Theory and Artificial Intelligence University of Southern.

Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 1 1 L. Itti: CS564 - Brain Theory and Artificial Intelligence University of Southern.

F16 Worlds Travemünde Info

F16 Worlds Travemünde Info

Itti: CS564 - Brain Theory and Artificial Intelligence University of Southern California

Itti: CS564 - Brain Theory and Artificial Intelligence University of Southern California

F16-T Brochure_email_v11 (1)

F16-T Brochure_email_v11 (1)

Michael Arbib: CS564 - Brain Theory and Artificial Intelligence

Michael Arbib: CS564 - Brain Theory and Artificial Intelligence

Usaf Flight Manual f16

Usaf Flight Manual f16