A Robust Partitioning Scheme for Ad-Hoc Query...

19
A Robust Partitioning Scheme for Ad-Hoc Query Workloads ANIL SHANBHAG MIT J/W Alekh Jindal, Sam Madden, Jorge Quiane, Aaron J. Elmore Microsoft MIT QCRI Univ. Chicago

Transcript of A Robust Partitioning Scheme for Ad-Hoc Query...

  • ARobustPartitioningSchemeforAd-HocQueryWorkloads

    ANILSHANBHAGMIT

    J/WAlekh Jindal,SamMadden, JorgeQuiane, AaronJ.ElmoreMicrosoftMIT QCRI Univ.Chicago

  • Today

    Datacollectionischeap=>Lotsofdata!

  • DataPartitioning

    FindaverageordersizeforallordersbetweenSept10andSept11,2017

    DataSkipping - Skipdatablocksnotnecessary

    10%selectivityquery=>10xfasterifdatapartitionedonselectionpredicate

    Orderdate

  • TheProblem

    Analytics

    Ad-Hoc/ExploratoryAnalysis

    RecurringWorkloads

    +

    Focusofexistingwork

    Giveworkload=>Returnpartitioninglayout

    Problems:1. Tedioustocollectworkload2. Maynotbeknownupfront3. Changesovertime

    Howtogetbenefitsofpartitioninginthiscase?

  • OurApproach

    Doeverythingadaptively!

    Twostepprocess:1. Upfrontloadthedatasetpartitioned2. Asusersquery,incrementallyimprovethe

    partitioningofthedata

  • DistributedstoragesystemslikeHDFS,filesbrokenintoblocks(128MBchunks)

    ASamenumberofblockscreatedasinHDFS.Eachblocknowhasadditionalmetadata

  • AdaptiveRe-Partitioning

    Whenusersubmitsaquery,optimizertriestoimprovethepartitioningbyreorganizingthepartitioningtree

    HereifqueriesaskA

  • SystemArchitecturePredicatedScanQueryExample:

    FINDemployeesWITHAge<30AND20k<Salary<40k1

    2

  • 1.UpfrontPartitionerGoal:Generateapartitioningtree

    WITHOUTanupfrontqueryworkload

    >Generatesatreewithheterogeneousbranching

    >Balancethepartitioningbenefitacrossallattributes

    !

    " #

    $

    ! " !

  • AllocationGoal: Balancepartitioningbenefitacrossattributes

    Allocationofattributei ~averagepartitioningofanattributej

    = 𝛴all nodes i nij cij

    UpfrontPartitioningAlgorithm

    AttributeAllocations

    PartitioningTree

    UniformifnoworkloadinformationWeightedifwehavepriorworkloadinformation

  • 2.AdaptiveQueryExecutorGoal:Returnmatchingtuples+checkifpartitioninglayoutcanbeimproved

    Alternativesfoundviatransformationsonthepartitioningtree

    1.SwapRule

    2.PushupRule 3.RotateRule

  • Gettingaplan

  • CostModelThesystemmaintainawindowWofpastqueries

    ComputeBenefitandRepartitioningCostforthebestplan

    RepartitioningONLY happenswhenreductioninthetotalcostofthequeryworkloadisgreaterthanre-partitioningcost.

    Solvesconstantre-partitioningduetorandomquerysequencesandboundstheworsecaseimpact.

  • Performance

    4metrics

    1)Loadtime

    2)Timetakenbyfirstquery

    3)Aggregateruntimeoveraworkload

    4)Incrementalimprovementwithworkloadhints

  • LoadTimeTPC-H:ScaleFactor200+De-normalized.Datasize:1.4TB

    Loadingperformance: 1.38timesslowerthanHDFS

    Loadtimescalesalmostlinearlywithdatasizeandindependentofnumberofcolumns

  • Timetakenbyfirstquery

    OnAverage:45%betterthanfullscan20%betterthank-dtree

  • AggregateWorkloadRuntime

    0400800

    120016002000

    0400800

    120016002000

    0400800

    120016002000

    0 25 50 75 100 125 150 175 2004uery 1o

    0400800

    1200160020007

    ime

    7aNe

    Q (iQ

    s)

    full scaQ raQge raQge2 AmoebaWorkload:200Queriesgeneratedfromrandominitializationof8querytemplatesofTPC-Hbenchmark

    fullscan – Baseline

    range – partitionsonorderdate (1perdate)1.88xbetter

    range2– partitionsonorderdate(64),r_name(4),c_mktsegment(4),quantity(8)3.48xbetter

    Amoeba– 3.84xbetterthanbaseline

  • WorkloadHints

    0400800

    120016002000

    0 25 50 75 100 125 150 175 2004uery 1o

    0400800

    120016002000

    7im

    e 7a

    NeQ

    (iQ s

    )

    default better iQitBetterInit:Startswithcustomallocationtomimicrange2

    6.67xbetterthan fullscan

    Filteringratio:default:0.81betterinit :0.9

  • Conclusion•Amoeba isadistributedstoragesystembasedonanadaptivedatapartitioningscheme• Lowloadingoverhead• Improvedfirstqueryperformance• Adapttochangesandsignificantlyimprovementtoworkloadruntime• Canexploitworkloadhints

    •Allowsanalyststogetstartedrightawayandreapbenefitsofpartitioningwithoutanupfrontworkload


Massively Distributed Database Systems In-Network Query Processing (Ad-Hoc Sensor Network)

Massively Distributed Database Systems In-Network Query Processing (Ad-Hoc Sensor Network)

Requested by BG EMN NCP on 11th April 2018...EMN Ad-Hoc Query on Judicial review of appeals against international protection decisions EMN Ad-Hoc Query on Judicial review of appeals

Requested by BG EMN NCP on 11th April 2018...EMN Ad-Hoc Query on Judicial review of appeals against international protection decisions EMN Ad-Hoc Query on Judicial review of appeals

Ad-Hoc Query on the Recognition of Professional Qualifications obtained outside … · 2017-01-04 · 1 Ad-Hoc Query on the Recognition of Professional Qualifications obtained outside

Ad-Hoc Query on the Recognition of Professional Qualifications obtained outside … · 2017-01-04 · 1 Ad-Hoc Query on the Recognition of Professional Qualifications obtained outside

Ad-Hoc Query on integration programme for beneficiaries of … · 2016-02-08 · 1 of 18 Ad-Hoc Query on integration programme for beneficiaries of international protection Requested

Ad-Hoc Query on integration programme for beneficiaries of … · 2016-02-08 · 1 of 18 Ad-Hoc Query on integration programme for beneficiaries of international protection Requested

Partitioning Defaults 1 - Columbia Business School€¦ · Partitioning Defaults 4 4 generate these hypotheses in the theoretical framework of Query Theory (E. J. Johnson, Haubl,

Partitioning Defaults 1 - Columbia Business School€¦ · Partitioning Defaults 4 4 generate these hypotheses in the theoretical framework of Query Theory (E. J. Johnson, Haubl,

AESuniversity Ad hoc Reporting. Ad hoc Reports What are ad hoc reports? Why would you use ad hoc reports? Creating an ad hoc report from a query Building.

AESuniversity Ad hoc Reporting. Ad hoc Reports What are ad hoc reports? Why would you use ad hoc reports? Creating an ad hoc report from a query Building.

AQWA Adaptive Query-Workload-Aware Partitioning of Big Spatial Data Dimosthenis Stefanidis Stelios Nikolaou.

AQWA Adaptive Query-Workload-Aware Partitioning of Big Spatial Data Dimosthenis Stefanidis Stelios Nikolaou.

Apache Drill interactive, ad-hoc query at scale...Apache Drill interactive, ad-hoc query at scale Michael Hausenblas, Chief Data Engineer EMEA, MapR Hadoop ecosystem - Open Source

Apache Drill interactive, ad-hoc query at scale...Apache Drill interactive, ad-hoc query at scale Michael Hausenblas, Chief Data Engineer EMEA, MapR Hadoop ecosystem - Open Source

EMN Ad-Hoc Query on Ad-hoc query on recreational ... · EMN Ad-Hoc Query on Ad-hoc query on recreational activities and leisure equipment in detention Library (e.g magazines, books)

EMN Ad-Hoc Query on Ad-hoc query on recreational ... · EMN Ad-Hoc Query on Ad-hoc query on recreational activities and leisure equipment in detention Library (e.g magazines, books)

Ad-Hoc Query on - European Commission · PDF fileAd-Hoc Query on “the support ... bedridden, patient in ... EMN NL NCP Ad-Hoc Query the support/accommodation provided to irregularly

Ad-Hoc Query on - European Commission · PDF fileAd-Hoc Query on “the support ... bedridden, patient in ... EMN NL NCP Ad-Hoc Query the support/accommodation provided to irregularly

An Adaptive Partitioning Scheme for Ad-hoc and Time ... · storage system provides improved query performance for ad-hoc workloads, adapts to changes in the query workloads, and converges

An Adaptive Partitioning Scheme for Ad-hoc and Time ... · storage system provides improved query performance for ad-hoc workloads, adapts to changes in the query workloads, and converges

A Robust Partitioning Scheme for Ad-Hoc Query Workloads · Microsoft MIT QCRI Univ. Chicago. Today Data collection is cheap => Lots of data ! Data Partitioning Find average order

A Robust Partitioning Scheme for Ad-Hoc Query Workloads · Microsoft MIT QCRI Univ. Chicago. Today Data collection is cheap => Lots of data ! Data Partitioning Find average order

FI Exercise 1 Creating a Basic Ad Hoc Query

FI Exercise 1 Creating a Basic Ad Hoc Query

Ad-Hoc Query on support to apprehended foreigners for whom ......1 of 16 Ad-Hoc Query on support to apprehended foreigners for whom detention is not applicable Requested by PL EMN

Ad-Hoc Query on support to apprehended foreigners for whom ......1 of 16 Ad-Hoc Query on support to apprehended foreigners for whom detention is not applicable Requested by PL EMN

Query-Aware Partitioning for Monitoring Massive Network … ·  · 2015-01-21Query-Aware Partitioning for Monitoring Massive Network ... AT&T Labs-Research Rutgers University AT&T

Query-Aware Partitioning for Monitoring Massive Network … ·  · 2015-01-21Query-Aware Partitioning for Monitoring Massive Network ... AT&T Labs-Research Rutgers University AT&T

Exercise 3 – Modifying a Basic Ad Hoc Query (Restricted Key ...

Exercise 3 – Modifying a Basic Ad Hoc Query (Restricted Key ...

SAP BI Creating a Basic Ad Hoc Query |

SAP BI Creating a Basic Ad Hoc Query |

EMN FR NCP Ad-hoc query on verification procedure of the ...emn.lt/old/uploads/files/261_doc_2_101531.pdf1 of 24 AD-HOC QUERY ON VERIFICATION PROCEDURE OF THE REGULARITY OF A THIRD-COUNTRY

EMN FR NCP Ad-hoc query on verification procedure of the ...emn.lt/old/uploads/files/261_doc_2_101531.pdf1 of 24 AD-HOC QUERY ON VERIFICATION PROCEDURE OF THE REGULARITY OF A THIRD-COUNTRY

Ad-Hoc Query on Marriage rights to entry and permanent … · 2017-01-04 · 1 of 30 Ad-Hoc Query on Marriage – rights to entry and permanent residence Requested by UK EMN NCP on

Ad-Hoc Query on Marriage rights to entry and permanent … · 2017-01-04 · 1 of 30 Ad-Hoc Query on Marriage – rights to entry and permanent residence Requested by UK EMN NCP on

Ad-Hoc Query on Expulsion definition of manifestly ...emn.ee/wp-content/uploads/2015/11/607_emn_ahq... · Ad-Hoc Query on Expulsion definition of manifestly unfounded as related to

Ad-Hoc Query on Expulsion definition of manifestly ...emn.ee/wp-content/uploads/2015/11/607_emn_ahq... · Ad-Hoc Query on Expulsion definition of manifestly unfounded as related to