Generalized Sub-Query Fusion for Eliminating Redundant I/O ...

25
Generalized Sub-Query Fusion for Eliminating Redundant I/O from Big-Data Queries Kaushik Rajan, Partho Sarthi, Akash Lal (Microsoft Research) Abhishek Modi, Ashit Gosalia, Prakhar Jain, Mo Liu, (Microsoft) Saurabh Kalikar (Intel, interned at Microsoft Research)

Transcript of Generalized Sub-Query Fusion for Eliminating Redundant I/O ...

Page 1: Generalized Sub-Query Fusion for Eliminating Redundant I/O ...

Generalized Sub-Query Fusion for Eliminating Redundant I/O from Big-Data QueriesKaushik Rajan,

Partho Sarthi, Akash Lal (Microsoft Research)

Abhishek Modi, Ashit Gosalia, Prakhar Jain, Mo Liu, (Microsoft)

Saurabh Kalikar (Intel, interned at Microsoft Research)

Page 2: Generalized Sub-Query Fusion for Eliminating Redundant I/O ...

Big data query compilation

1 1 1 1

2 2 2 2

Query compilation

SQL Query

Data parallel execution

scheduling/execution exchange

Map 1

Reduce 2

Reduce 3 Execution Plan

Query compilation and execution in big-data systems (Spark, Hadoop, Snowflake, Amazon Redshift, Google

BigQuery, Azure Synapse)

Page 3: Generalized Sub-Query Fusion for Eliminating Redundant I/O ...

Big data query compilation

1 1 1 1

2 2 2 2

scheduling/execution exchange

Plans with fewer stages preferable

Map 1

Reduce 2

Reduce 3 Execution Plan

Exchanges expensive as they induce disk and network I/O

Data parallel execution

Query compilation

SQL Query

Page 4: Generalized Sub-Query Fusion for Eliminating Redundant I/O ...

Redundant stages of processing

β€’ TPCDS, 40% of queries have redundant I/O

β€’ 16% of all queries, High-impact spend at least 50% time on stages with redundant I/O

β€’ 9% medium impact, spend 10-50% time on stages with redundant I/O

Redundancy analysis in SPARK on TPCDS

Page 5: Generalized Sub-Query Fusion for Eliminating Redundant I/O ...

RESIN: MapReduce reasoning during optimization

State-of-the-art

Logical plan

Physical plan

Map 1

Reduce 2Produce plans with

fewer stages

SQL→SQL+MR

RESIN

MR ops

Logical plan

SQLSQL β†’ SQL rewrites SQL PhyOps,

exchanges Physical plan

Map 1

Reduce 2

Reduce 3

Page 6: Generalized Sub-Query Fusion for Eliminating Redundant I/O ...

RESIN: MapReduce reasoning during optimization

Logical plan

SQLSQL β†’ SQL rewrites SQL PhyOps,

exchanges Physical plan

State-of-the-art

Map 1

Reduce 2

Reduce 3

Logical plan

Physical plan

MR opsMap 1

Reduce 2Produce plans with

fewer stages

SQL→SQL+MR

RESIN

1. Subquery Fusion2. Binary elimination

ResinMapResinReduce

Page 7: Generalized Sub-Query Fusion for Eliminating Redundant I/O ...

Impact of RESIN on I/O and Memory

Page 8: Generalized Sub-Query Fusion for Eliminating Redundant I/O ...

Rest of the talk

1. ResinMap and ResinReduce

2. Generalized sub-query fusion

3. Implementation on Spark

4. Experimental evaluation

Page 9: Generalized Sub-Query Fusion for Eliminating Redundant I/O ...

T1 = SELECT 𝑖𝑑, β„Žπ‘Ÿ ← β„Žπ‘Ÿ1,π‘ π‘–π‘”π‘›π‘Žπ‘™ ← π‘ π‘–π‘”π‘›π‘Žπ‘™1

FROM rawLogsWHERE β„Žπ‘Ÿ1 β‰₯ 0 ∧ β„Žπ‘Ÿ1 <

24 ∧ π‘ π‘–π‘”π‘›π‘Žπ‘™1 β‰₯ 0

signals = SELECT * FROM T1UNION ALL

SELECT * FROM T2

Union

Project

iotLogs : (𝑖𝑑, β„Žπ‘Ÿ1, π‘ π‘–π‘”π‘›π‘Žπ‘™1, β„Žπ‘Ÿ2, π‘ π‘–π‘”π‘›π‘Žπ‘™2)

T2 = SELECT 𝑖𝑑, β„Žπ‘Ÿ ← β„Žπ‘Ÿ2,π‘ π‘–π‘”π‘›π‘Žπ‘™ ← π‘ π‘–π‘”π‘›π‘Žπ‘™2

FROM rawLogsWHERE β„Žπ‘Ÿ2 β‰₯ 0 ∧ β„Žπ‘Ÿ2 <

24 ∧ π‘ π‘–π‘”π‘›π‘Žπ‘™2 β‰₯ 0

ResinMap[/*1*/{Filter(β„Žπ‘Ÿ1 β‰₯ 0 ∧ β„Žπ‘Ÿ1 < 24 βˆ§π‘ π‘–π‘”π‘›π‘Žπ‘™1 β‰₯ 0)), Cols(id, β„Žπ‘Ÿ ← β„Žπ‘Ÿ1,

π‘ π‘–π‘”π‘›π‘Žπ‘™ ← π‘ π‘–π‘”π‘›π‘Žπ‘™1)}, /*2*/{Filter(β„Žπ‘Ÿ2 β‰₯ 0 ∧ β„Žπ‘Ÿ2 < 24 βˆ§π‘ π‘–π‘”π‘›π‘Žπ‘™2 β‰₯ 0), Cols(id, β„Žπ‘Ÿ ← β„Žπ‘Ÿ2,

π‘ π‘–π‘”π‘›π‘Žπ‘™ ← π‘ π‘–π‘”π‘›π‘Žπ‘™2)}

(a) SQL Query(b) Standardexecution plan

(c) Optimizedexecution plan

signals

signals

scaniotLogs

I scan iotLogs

FilterProjectFilter

scan iotLogsiotLogs :

(𝑖𝑑, β„Žπ‘Ÿ1, π‘ π‘–π‘”π‘›π‘Žπ‘™1, β„Žπ‘Ÿ2, π‘ π‘–π‘”π‘›π‘Žπ‘™2)

ResinMapA row-wise operator, can produce multiple output rows per input row

Page 10: Generalized Sub-Query Fusion for Eliminating Redundant I/O ...

T1 = SELECT 𝑖𝑑, β„Žπ‘Ÿ ← β„Žπ‘Ÿ1,π‘ π‘–π‘”π‘›π‘Žπ‘™ ← π‘ π‘–π‘”π‘›π‘Žπ‘™1

FROM rawLogsWHERE β„Žπ‘Ÿ1 β‰₯ 0 ∧ β„Žπ‘Ÿ1 <

24 ∧ π‘ π‘–π‘”π‘›π‘Žπ‘™1 β‰₯ 0

signals = SELECT * FROM T1UNION ALL

SELECT * FROM T2

Union

Project

iotLogs : (𝑖𝑑, β„Žπ‘Ÿ1, π‘ π‘–π‘”π‘›π‘Žπ‘™1, β„Žπ‘Ÿ2, π‘ π‘–π‘”π‘›π‘Žπ‘™2)

T2 = SELECT 𝑖𝑑, β„Žπ‘Ÿ ← β„Žπ‘Ÿ2,π‘ π‘–π‘”π‘›π‘Žπ‘™ ← π‘ π‘–π‘”π‘›π‘Žπ‘™2

FROM rawLogsWHERE β„Žπ‘Ÿ2 β‰₯ 0 ∧ β„Žπ‘Ÿ2 <

24 ∧ π‘ π‘–π‘”π‘›π‘Žπ‘™2 β‰₯ 0

ResinMap[/*1*/{Filter(β„Žπ‘Ÿ1 β‰₯ 0 ∧ β„Žπ‘Ÿ1 < 24 βˆ§π‘ π‘–π‘”π‘›π‘Žπ‘™1 β‰₯ 0)), Cols(id, β„Žπ‘Ÿ ← β„Žπ‘Ÿ1,

π‘ π‘–π‘”π‘›π‘Žπ‘™ ← π‘ π‘–π‘”π‘›π‘Žπ‘™1)}, /*2*/{Filter(β„Žπ‘Ÿ2 β‰₯ 0 ∧ β„Žπ‘Ÿ2 < 24 βˆ§π‘ π‘–π‘”π‘›π‘Žπ‘™2 β‰₯ 0), Cols(id, β„Žπ‘Ÿ ← β„Žπ‘Ÿ2,

π‘ π‘–π‘”π‘›π‘Žπ‘™ ← π‘ π‘–π‘”π‘›π‘Žπ‘™2)}

(a) SQL Query(b) Standardexecution plan

(c) Optimizedexecution plan

signals

signals

scaniotLogs

I scan iotLogs

FilterProjectFilter

scan iotLogsiotLogs :

(𝑖𝑑, β„Žπ‘Ÿ1, π‘ π‘–π‘”π‘›π‘Žπ‘™1, β„Žπ‘Ÿ2, π‘ π‘–π‘”π‘›π‘Žπ‘™2)

ResinMapA row-wise operator, can produce multiple output rows per input row

Page 11: Generalized Sub-Query Fusion for Eliminating Redundant I/O ...

ResinMap[/*1*/{Filter(β„Žπ‘Ÿ1 β‰₯ 0 ∧ β„Žπ‘Ÿ1 < 24 βˆ§π‘ π‘–π‘”π‘›π‘Žπ‘™1 β‰₯ 0)), Cols(id, β„Žπ‘Ÿ ← β„Žπ‘Ÿ1,

π‘ π‘–π‘”π‘›π‘Žπ‘™ ← π‘ π‘–π‘”π‘›π‘Žπ‘™1)}, /*2*/{Filter(β„Žπ‘Ÿ2 β‰₯ 0 ∧ β„Žπ‘Ÿ2 < 24 βˆ§π‘ π‘–π‘”π‘›π‘Žπ‘™2 β‰₯ 0), Cols(id, β„Žπ‘Ÿ ← β„Žπ‘Ÿ2,

π‘ π‘–π‘”π‘›π‘Žπ‘™ ← π‘ π‘–π‘”π‘›π‘Žπ‘™2)}

signals

scan iotLogs

ResinMapA row-wise operator, can produce multiple output rows per input row

//Each mapper m processes a partition rawlogs[m]Method ResinMap(m) {

foreach 𝑖𝑑, β„Žπ‘Ÿ1, π‘ π‘–π‘”π‘›π‘Žπ‘™1, β„Žπ‘Ÿ2, π‘ π‘–π‘”π‘›π‘Žπ‘™2 ∈ π‘–π‘œπ‘‘πΏπ‘œπ‘”π‘  π‘š {if(β„Žπ‘Ÿ1 β‰₯ 0 ∧ β„Žπ‘Ÿ1 < 24 ∧ π‘ π‘–π‘”π‘›π‘Žπ‘™1 β‰₯ 0) {

β„Žπ‘Ÿ = β„Žπ‘Ÿ1; π‘ π‘–π‘”π‘›π‘Žπ‘™ = π‘ π‘–π‘”π‘›π‘Žπ‘™1; π‘œπ‘’π‘‘π‘π‘’π‘‘ 𝑖𝑑, β„Žπ‘Ÿ, π‘ π‘–π‘”π‘›π‘Žπ‘™}if(β„Žπ‘Ÿ2 β‰₯ 0 ∧ β„Žπ‘Ÿ2 < 24 ∧ π‘ π‘–π‘”π‘›π‘Žπ‘™2 β‰₯ 0) {

β„Žπ‘Ÿ = β„Žπ‘Ÿ2; π‘ π‘–π‘”π‘›π‘Žπ‘™ = π‘ π‘–π‘”π‘›π‘Žπ‘™2; π‘œπ‘’π‘‘π‘π‘’π‘‘ 𝑖𝑑, β„Žπ‘Ÿ, π‘ π‘–π‘”π‘›π‘Žπ‘™}

}

Single table select, project, union queries in one stage

Page 12: Generalized Sub-Query Fusion for Eliminating Redundant I/O ...

ResinReduce

πΊπ‘Ÿπ‘œπ‘’π‘π΅π‘¦π‘–π‘‘,s1←max(π‘ π‘–π‘”π‘›π‘Žπ‘™1)πΊπ‘Ÿπ‘œπ‘’π‘π΅π‘¦π‘–π‘‘,𝑠2←m𝑖𝑛(π‘ π‘–π‘”π‘›π‘Žπ‘™2)

Join (𝑐1 = 𝑐2)

Project(c1 ← 𝑖𝑑, 𝑠1) Project(c2 ← 𝑖𝑑, 𝑠2)

πΉπ‘–π‘™π‘‘π‘’π‘Ÿ (β„Žπ‘Ÿ1 ≀ 12) πΉπ‘–π‘™π‘‘π‘’π‘Ÿ (β„Žπ‘Ÿ2 ≀ 18)

iotLogs : (𝑖𝑑, β„Žπ‘Ÿ1, π‘ π‘–π‘”π‘›π‘Žπ‘™1, β„Žπ‘Ÿ2, π‘ π‘–π‘”π‘›π‘Žπ‘™2)

iotLogs : (𝑖𝑑, β„Žπ‘Ÿ1, π‘ π‘–π‘”π‘›π‘Žπ‘™1, β„Žπ‘Ÿ2, π‘ π‘–π‘”π‘›π‘Žπ‘™2)

ResinReduce[(π‘˜π‘’π‘¦ = 𝑖𝑑),/*1*/{filter(β„Žπ‘Ÿ1 ≀ 12)), aggregate(

)𝑠1 ←

max(π‘ π‘–π‘”π‘›π‘Žπ‘™1 , π‘Ÿπ‘1 ← π‘π‘œπ‘’π‘›π‘‘ βˆ— )}/*2*/{ filter(β„Žπ‘Ÿ2 ≀ 18)aggregate (s2 β†π‘šπ‘–π‘› π‘ π‘–π‘”π‘›π‘Žπ‘™2 π‘Ÿπ‘2 ← π‘π‘œπ‘’π‘›π‘‘ βˆ— )}]

iotLogs : (𝑖𝑑, β„Žπ‘Ÿ1, π‘ π‘–π‘”π‘›π‘Žπ‘™1, β„Žπ‘Ÿ2, π‘ π‘–π‘”π‘›π‘Žπ‘™2)

(Filter(π‘Ÿπ‘1 > 0 ∧ π‘Ÿπ‘2 > 0)

Eliminate multiple shuffles from single table join queries

Filter( β„Žπ‘Ÿ1 ≀ 12 ∨ β„Žπ‘Ÿ2 ≀ 18)

Project(𝑖𝑑, π‘ π‘–π‘”π‘›π‘Žπ‘™1) Project(𝑖𝑑, π‘ π‘–π‘”π‘›π‘Žπ‘™2)

Key based operator, process rows sharing key, produce one row

Page 13: Generalized Sub-Query Fusion for Eliminating Redundant I/O ...

ResinReduce[(π‘˜π‘’π‘¦ = 𝑖𝑑),/*1*/{filter, aggregate}/*2*/{ filter,aggregate}]

Join (𝑑𝑖𝑑 = 𝑖𝑑)

Filter(πœ™4)(β„Žπ‘‘))

Project(𝑐𝑖𝑑𝑦, 𝑑𝑖𝑑)

πΊπ‘Ÿπ‘œπ‘’π‘π΅π‘¦π‘π‘–π‘‘π‘¦,s1←max(π‘ π‘–π‘”π‘›π‘Žπ‘™) πΊπ‘Ÿπ‘œπ‘’π‘π΅π‘¦π‘π‘–π‘‘π‘¦,𝑠2←max(π‘ π‘–π‘”π‘›π‘Žπ‘™)

Join (𝑐1 = 𝑐2)

Scan signals Scan signals Scan dInfo

Filter(πœ™1(β„Žπ‘Ÿ)) Filter(πœ™2(β„Žπ‘Ÿ))

Project(id, π‘ π‘–π‘”π‘›π‘Žπ‘™) Project(id, π‘ π‘–π‘”π‘›π‘Žπ‘™)

Project(𝑐𝑖𝑑𝑦, 𝑑𝑖𝑑)

Scan dInfo

Project(𝑐𝑖𝑑𝑦, π‘ π‘–π‘”π‘›π‘Žπ‘™)

Join (𝑑𝑖𝑑 = 𝑖𝑑)

Project(𝑐𝑖𝑑𝑦, π‘ π‘–π‘”π‘›π‘Žπ‘™)

Filter(πœ™3(β„Žπ‘‘))

S1 S2

S3

S4

S5 S6

S7 S8

S9

Project(c1 ← 𝑐𝑖𝑑𝑦, 𝑠1) Project(c2 ← 𝑐𝑖𝑑𝑦, 𝑠2)

Join (𝑑𝑖𝑑 = 𝑖𝑑)

Filter( πœ™1 ∧ πœ™3 ∨ (πœ™2 ∧ πœ™4)(β„Žπ‘Ÿ, β„Žπ‘‘)

Filter(πœ™1 ∨ πœ™2(β„Žπ‘Ÿ))

Project(id,signal,hr) Project(𝑐𝑖𝑑𝑦, 𝑑𝑖𝑑,ht)

Project(𝑐𝑖𝑑𝑦, π‘ π‘–π‘”π‘›π‘Žπ‘™, β„Žπ‘Ÿ, β„Žπ‘‘)

S10

Filter(πœ™3 ∨ πœ™4(β„Žπ‘‘))

Scan dInfoScan signals

S11

S12

S13

Sub-query fusion

Eliminate scans/shuffles from multi-table queries

Page 14: Generalized Sub-Query Fusion for Eliminating Redundant I/O ...

Join (𝑑𝑖𝑑 = 𝑖𝑑)

Filter(πœ™4(β„Žπ‘‘))

Project(𝑐𝑖𝑑𝑦, 𝑑𝑖𝑑)

Scan signals Scan signals Scan dInfo

Filter(πœ™1(β„Žπ‘Ÿ)) Filter(πœ™2(β„Žπ‘Ÿ))

Project(id, π‘ π‘–π‘”π‘›π‘Žπ‘™) Project(id, π‘ π‘–π‘”π‘›π‘Žπ‘™)

Project(𝑐𝑖𝑑𝑦, 𝑑𝑖𝑑)

Scan dInfo

Project(𝑐𝑖𝑑𝑦, π‘ π‘–π‘”π‘›π‘Žπ‘™)

Join (𝑑𝑖𝑑 = 𝑖𝑑)

Project(𝑐𝑖𝑑𝑦, π‘ π‘–π‘”π‘›π‘Žπ‘™)

Filter(πœ™3(β„Žπ‘‘))

S2

S3

S4

Filter(πœ™1 ∨ πœ™2(β„Žπ‘Ÿ))

Project(id,signal,hr) Project(𝑐𝑖𝑑𝑦, 𝑑𝑖𝑑,ht)

S10

Filter(πœ™3 ∨ πœ™4(β„Žπ‘‘))

Scan dInfoScan signals

S11

Sub-query fusion

Eliminate scans/shuffles from multi-stage queries

Filter(πœ™1(β„Žπ‘Ÿ)) Filter(πœ™2(β„Žπ‘Ÿ))

S1

Filter(πœ™1 ∧ πœ™3) Filter(πœ™2 ∧ πœ™4)

πΊπ‘Ÿπ‘œπ‘’π‘π΅π‘¦π‘π‘–π‘‘π‘¦,s1←max(π‘ π‘–π‘”π‘›π‘Žπ‘™) πΊπ‘Ÿπ‘œπ‘’π‘π΅π‘¦π‘π‘–π‘‘π‘¦,𝑠2←max(π‘ π‘–π‘”π‘›π‘Žπ‘™)

Join (𝑐1 = 𝑐2)

Project(c1 ← 𝑐𝑖𝑑𝑦, 𝑠1) Project(c2 ← 𝑐𝑖𝑑𝑦, 𝑠2)

Page 15: Generalized Sub-Query Fusion for Eliminating Redundant I/O ...

Join (𝑑𝑖𝑑 = 𝑖𝑑)

Project(𝑐𝑖𝑑𝑦, π‘ π‘–π‘”π‘›π‘Žπ‘™)

Join (𝑑𝑖𝑑 = 𝑖𝑑)

Project(𝑐𝑖𝑑𝑦, π‘ π‘–π‘”π‘›π‘Žπ‘™)

Sub-query fusion

Eliminate scans/shuffles from multi-stage queries

Filter(πœ™1 ∨ πœ™2(β„Žπ‘Ÿ))

Project(id,signal,hr) Project(𝑐𝑖𝑑𝑦, 𝑑𝑖𝑑,ht)

Filter(πœ™3 ∨ πœ™4(β„Žπ‘‘))

Scan dInfoScan signals

Filter(πœ™1(β„Žπ‘Ÿ)) Filter(πœ™2(β„Žπ‘Ÿ))Filter(πœ™1 ∧ πœ™3) Filter(πœ™2 ∧ πœ™4)

S5 S6

Join (𝑑𝑖𝑑 = 𝑖𝑑)Filter( πœ™1 ∧ πœ™3 ∨ (πœ™2 ∧ πœ™4)(β„Žπ‘Ÿ, β„Žπ‘‘)

Project(𝑐𝑖𝑑𝑦, π‘ π‘–π‘”π‘›π‘Žπ‘™, β„Žπ‘Ÿ, β„Žπ‘‘)

Filter(πœ™1 ∧ πœ™3) Filter(πœ™2 ∧ πœ™4)

S12

πΊπ‘Ÿπ‘œπ‘’π‘π΅π‘¦π‘π‘–π‘‘π‘¦,s1←max(π‘ π‘–π‘”π‘›π‘Žπ‘™) πΊπ‘Ÿπ‘œπ‘’π‘π΅π‘¦π‘π‘–π‘‘π‘¦,𝑠2←max(π‘ π‘–π‘”π‘›π‘Žπ‘™)

Join (𝑐1 = 𝑐2)

Project(c1 ← 𝑐𝑖𝑑𝑦, 𝑠1) Project(c2 ← 𝑐𝑖𝑑𝑦, 𝑠2)

Page 16: Generalized Sub-Query Fusion for Eliminating Redundant I/O ...

πΊπ‘Ÿπ‘œπ‘’π‘π΅π‘¦π‘π‘–π‘‘π‘¦,s1←max(π‘ π‘–π‘”π‘›π‘Žπ‘™) πΊπ‘Ÿπ‘œπ‘’π‘π΅π‘¦π‘π‘–π‘‘π‘¦,𝑠2←max(π‘ π‘–π‘”π‘›π‘Žπ‘™)

Join (𝑐1 = 𝑐2)

Project(c1 ← 𝑐𝑖𝑑𝑦, 𝑠1) Project(c2 ← 𝑐𝑖𝑑𝑦, 𝑠2)

Sub-query fusion

Eliminate scans/shuffles from multi-stage queries

Filter(πœ™1 ∨ πœ™2(β„Žπ‘Ÿ))

Project(id,signal,hr) Project(𝑐𝑖𝑑𝑦, 𝑑𝑖𝑑,ht)

Filter(πœ™3 ∨ πœ™4(β„Žπ‘‘))

Scan dInfoScan signals

Join (𝑑𝑖𝑑 = 𝑖𝑑)Filter( πœ™1 ∧ πœ™3 ∨ (πœ™2 ∧ πœ™4)(β„Žπ‘Ÿ, β„Žπ‘‘)

Project(𝑐𝑖𝑑𝑦, π‘ π‘–π‘”π‘›π‘Žπ‘™, β„Žπ‘Ÿ, β„Žπ‘‘)

Filter(πœ™1 ∧ πœ™3) Filter(πœ™2 ∧ πœ™4)

S7 S8

S9S13

ResinReduce[(π‘˜π‘’π‘¦ = 𝑐𝑖𝑑𝑦),/*1*/{filter(πœ™1 ∧ πœ™3), aggregate(

)𝑠1 ←

max(π‘ π‘–π‘”π‘›π‘Žπ‘™ , π‘Ÿπ‘1 ← π‘π‘œπ‘’π‘›π‘‘ βˆ— )}/*2*/{ filter(πœ™2 ∧ πœ™4)

aggregate (s2 ← π‘šπ‘Žπ‘₯ π‘ π‘–π‘”π‘›π‘Žπ‘™ π‘Ÿπ‘2 β†π‘π‘œπ‘’π‘›π‘‘ βˆ— )}]

(Filter(π‘Ÿπ‘1 > 0 ∧ π‘Ÿπ‘2 > 0)

Page 17: Generalized Sub-Query Fusion for Eliminating Redundant I/O ...

Join (𝑑𝑖𝑑 = 𝑖𝑑)

Filter(πœ™4)(β„Žπ‘‘))

Project(𝑐𝑖𝑑𝑦, 𝑑𝑖𝑑)

πΊπ‘Ÿπ‘œπ‘’π‘π΅π‘¦π‘π‘–π‘‘π‘¦,s1←max(π‘ π‘–π‘”π‘›π‘Žπ‘™) πΊπ‘Ÿπ‘œπ‘’π‘π΅π‘¦π‘π‘–π‘‘π‘¦,𝑠2←max(π‘ π‘–π‘”π‘›π‘Žπ‘™)

Join (𝑐1 = 𝑐2)

Scan signals Scan signals Scan dInfo

Filter(πœ™1(β„Žπ‘Ÿ)) Filter(πœ™2(β„Žπ‘Ÿ))

Project(id, π‘ π‘–π‘”π‘›π‘Žπ‘™) Project(id, π‘ π‘–π‘”π‘›π‘Žπ‘™)

Project(𝑐𝑖𝑑𝑦, 𝑑𝑖𝑑)

Scan dInfo

Project(𝑐𝑖𝑑𝑦, π‘ π‘–π‘”π‘›π‘Žπ‘™)

Join (𝑑𝑖𝑑 = 𝑖𝑑)

Project(𝑐𝑖𝑑𝑦, π‘ π‘–π‘”π‘›π‘Žπ‘™)

Filter(πœ™3(β„Žπ‘‘))

S1 S2

S3

S4

S5 S6

S7 S8

S9

Project(c1 ← 𝑐𝑖𝑑𝑦, 𝑠1) Project(c2 ← 𝑐𝑖𝑑𝑦, 𝑠2)

Join (𝑑𝑖𝑑 = 𝑖𝑑)

Filter( πœ™1 ∧ πœ™3 ∨ (πœ™2 ∧ πœ™4)(β„Žπ‘Ÿ, β„Žπ‘‘)

Filter(πœ™1 ∨ πœ™2(β„Žπ‘Ÿ))

Project(id,signal,hr) Project(𝑐𝑖𝑑𝑦, 𝑑𝑖𝑑,ht)

Project(𝑐𝑖𝑑𝑦, π‘ π‘–π‘”π‘›π‘Žπ‘™, β„Žπ‘Ÿ, β„Žπ‘‘)

S10

Filter(πœ™3 ∨ πœ™4(β„Žπ‘‘))

Scan dInfoScan signals

S11

S12

S13

Sub-query fusion

Eliminate scans/shuffles from multi-table queries

ResinReduce[(π‘˜π‘’π‘¦ = 𝑐𝑖𝑑𝑦),/*1*/{filter(πœ™1 ∧ πœ™3), aggregate(

)𝑠1 ←

max(π‘ π‘–π‘”π‘›π‘Žπ‘™ , π‘Ÿπ‘1 ← π‘π‘œπ‘’π‘›π‘‘ βˆ— )}/*2*/{ filter(πœ™2 ∧ πœ™4)

aggregate (s2 ← π‘šπ‘Žπ‘₯ π‘ π‘–π‘”π‘›π‘Žπ‘™ π‘Ÿπ‘2 β†π‘π‘œπ‘’π‘›π‘‘ βˆ— )}]

(Filter(π‘Ÿπ‘1 > 0 ∧ π‘Ÿπ‘2 > 0)S13

Page 18: Generalized Sub-Query Fusion for Eliminating Redundant I/O ...

In the paper

β€’Parameters for ResinMap and ResinReduceoperators, semantics and implementation

β€’ Fusing of operators without increasing the number of rows shuffled

β€’ Fusion rules for all sparkSQL operators, conditions under which fusion is possible

Page 19: Generalized Sub-Query Fusion for Eliminating Redundant I/O ...

Rest of the talk

1. ResinMap and ResinReduce

2. Generalized sub-query fusion

3. Implementation on Spark

4. Experimental evaluation

Page 20: Generalized Sub-Query Fusion for Eliminating Redundant I/O ...

Implementation

Implemented RESIN on catalyst optimizer in SPARK 2.4

1. Added logical and physical operators for ResinMap andResinReduce

2. Added a new batch of optimization rulesβ€’ Perform fusion in a single traversal of the tree

β€’ Perform Union and Join elimination by checking fused parent

β€’ Introduce exchanges if parent after fusion cannot be eliminated

3. Added implementations for our operators with codegen support

Page 21: Generalized Sub-Query Fusion for Eliminating Redundant I/O ...

Evaluation

β€’ Evaluated with TPCDS at 1TB and 10TB scale, data stored in parquet

β€’ Two different clusters <120 cores, 480 GB memory> and <480 cores, 1.6TB memory>

β€’ Detailed evaluation of 40 (out of 104) queries with redundant I/O

β€’ Note: baseline that already has basic I/O optimizations (predicate and project pushdown to store, exchange reuse)

Page 22: Generalized Sub-Query Fusion for Eliminating Redundant I/O ...

Speedup at 1TB

Page 23: Generalized Sub-Query Fusion for Eliminating Redundant I/O ...

Impact of RESIN on I/O and Memory

Page 24: Generalized Sub-Query Fusion for Eliminating Redundant I/O ...

Conclusions

β€’Big-data optimizers produce plans with redundant I/O and compute

β€’Proposed optimizer extensions to perform first class map-reduce reasoning

β€’Added generic map and reduce operators, rewrites that fuse stages and eliminate redundant I/O

β€’Demonstrated savings in terms of latency, disk and network I/O

Page 25: Generalized Sub-Query Fusion for Eliminating Redundant I/O ...

Thank You

Email [email protected] or any of the other authors to contact us

mailto:[email protected]

Eliminating Entries

Eliminating Entries

Eliminating redundant software

Eliminating redundant software

STATE-SPACE GENERALIZED PREDICTIVE …library.utia.cas.cz/separaty/historie/belda-0410813.pdfSTATE-SPACE GENERALIZED PREDICTIVE CONTROL FOR REDUNDANT PARALLEL ROBOTS KvΔ›toslav Belda,

STATE-SPACE GENERALIZED PREDICTIVE …library.utia.cas.cz/separaty/historie/belda-0410813.pdfSTATE-SPACE GENERALIZED PREDICTIVE CONTROL FOR REDUNDANT PARALLEL ROBOTS KvΔ›toslav Belda,

Redundant devops

Redundant devops

Data-Triggered Threads: Eliminating Redundant Computationcseweb.ucsd.edu/~tullsen/hpca2011tseng.pdfΒ Β· Silent stores [17,18] detects and removes store in-structions from the load/store

Data-Triggered Threads: Eliminating Redundant Computationcseweb.ucsd.edu/~tullsen/hpca2011tseng.pdfΒ Β· Silent stores [17,18] detects and removes store in-structions from the load/store

Eliminating Fears

Eliminating Fears

Sparse and Redundant

Sparse and Redundant

Parallel Redundant katalog - Deutsche Messe AGdonar.messe.de/.../parallel-redundant-voltage-stabilizer-eng-493215.pdfΒ Β· PARALLEL REDUNDANT VOLTAGE STABILIZER PARALLEL REDUNDANT VOLTAGE

Parallel Redundant katalog - Deutsche Messe AGdonar.messe.de/.../parallel-redundant-voltage-stabilizer-eng-493215.pdfΒ Β· PARALLEL REDUNDANT VOLTAGE STABILIZER PARALLEL REDUNDANT VOLTAGE

Guidelines on auditing a Safety Management System - … · Section 1 – Safety Management in ISAGO ... reducing delays, and eliminating redundant audits by airlines. The GOSM Ed

Guidelines on auditing a Safety Management System - … · Section 1 – Safety Management in ISAGO ... reducing delays, and eliminating redundant audits by airlines. The GOSM Ed

A Theoretical Investigation of Generalized Voters for Redundant Systems

A Theoretical Investigation of Generalized Voters for Redundant Systems

Generalized Redundant Calibration of Radio Inteferferometers

Generalized Redundant Calibration of Radio Inteferferometers

Redundant Number Systems

Redundant Number Systems

SSP: Eliminating Redundant Writes in Failure-Atomic NVRAMs ...jzhao/files/ssp-micro2019.pdfSSP maintains additional meta information that tracks state on a cache line basis within

SSP: Eliminating Redundant Writes in Failure-Atomic NVRAMs ...jzhao/files/ssp-micro2019.pdfSSP maintains additional meta information that tracks state on a cache line basis within

Redundant Systems

Redundant Systems

Redundant Controller Connection Reference For X28 … and X2… · Redundant Controller Connection Reference For X28-F16F and Falcon ... Redundant Controller Connection Reference

Redundant Controller Connection Reference For X28 … and X2… · Redundant Controller Connection Reference For X28-F16F and Falcon ... Redundant Controller Connection Reference

Reliable Control System For Future Particle …vxWorks operating system. I generalized the EPICS redundant IOC to other operating systems and made it available for wider range of applications[10].

Reliable Control System For Future Particle …vxWorks operating system. I generalized the EPICS redundant IOC to other operating systems and made it available for wider range of applications[10].

RAID Redundant Array of Independent Disks Redundant Array of Inexpensive Disks

RAID Redundant Array of Independent Disks Redundant Array of Inexpensive Disks

REDUNDANT POWER PACKAGES and MODULAR REDUNDANT …

REDUNDANT POWER PACKAGES and MODULAR REDUNDANT …

.High Performance. Accurate. .Redundant.Β Β· sales@masibus.com Features Redundant Master Time Sync Unit. MTS300R.High Performance. Accurate..Redundant. Masibus MTS300R is GPS based

.High Performance. Accurate. .Redundant.Β Β· [email protected] Features Redundant Master Time Sync Unit. MTS300R.High Performance. Accurate..Redundant. Masibus MTS300R is GPS based

Reliable Redundant Resilient

Reliable Redundant Resilient