Technische Universität München Hewlett-Packard Laboratories Managing Long-Running Queries Stefan...

Technische Universität MünchenHewlett-Packard Laboratories

Managing Long-Running Queries

Stefan KrompassTUM, Harumi KunoHPL, Janet WienerHPL,

Kevin WilkinsonHPL, Umeshwar DayalHPL, and Alfons KemperTUM

TUMTechnische Universität MünchenMunich, Germany

HPLHewlett-Packard LaboratoriesPalo Alto, CA, USA

2


Motivation

DBMS

Business analysis

Customer relations

Sales

Order entry

Administrator

Maintenance

3


Goals of this work

• Develop technology to study policies for mixed workloads• Initial study of managing mixed workloads, in particular: impact of

long-running queries on a workload– Unreliable cost estimates

under-informed admission control and scheduling decisions– Unobserved resource contention

monitored resource not the source of contention– System overload

4


Outline

• Workload management components &workload management policies

• Experiments• Conclusions

5


Workload management overview

Workload

Queries

Objectives

Client

DBMS

Workload Manager

Admission Controller

Scheduler

Execution Controller

Database Engine

6


Experimental approach

• Create workloads that inject “problem” queries (our workloads are derived from actual mixed workload queries)

• Develop a workload management software that implements admission control, scheduling, and execution control policies

• Workload manager feeds queries into database engine simulator– Investigate workloads that run for hours– Obtain reproducible results– Experiment with comprehensive set of workload management

policies– Inject problem queries

7


Experimental input: queries

query typesize of query pool

queries per workload

average elapsed

time

short 2807 400 30 sec

medium 247 23 10 min

long 48 3 1 hr

Problem queries

8


Taxonomy of long-running queries

no (< equal)nonostarving

yesnonooverload

no (> equal)yesnosurprise-hog

yesyesnosurprise-long

no (> equal)yesyesexpected-hog

yesyesyesexpected-long

Uses equal share of

resources

Query progress

reasonable

Query expected to be long

Query type

9


Experimental inputs

• Workload types: expected-long, surprise-long, surprise-hog

• Admission control– Policies: none, limit expected costs of a query– Thresholds: 0.2m, 0.5m, and 1.0m

• Scheduling– Queue: FIFO– Multiprogramming level (MPL)

• Execution control– Policies: none, kill, kill&requeue, suspend&resume– Thresholds: absolute 5000 (time units), absolute 12000,

absolute 5000 & progress < 30%, relative 1.2x

10


How did we choose the thresholds?

Expected = actual CPU

costs

Surprise-*

11


Experimental measure - weighted makespan

31

5

9

4

13

9

4.5Elapsed timeof queries

Q4

Q3

Q2

Q1

Makespan(Q4 filtered,as expected

no penalty)Makespan(Q2 and Q4

filtered)

Penalty fortreating Q2 asfalse positive

Weig

hted

makesp

an

12


Experimental measure - weighted makespan

1A1C

“one long query admitted” (1A)“one long query completed” (1C)

13


0

20

40

60

80

100

120

140

160W

eigh

ted

mak

esp

an

(in

sim

ula

tor

tim

e u

nit

s)

3A3C

0A0C

3A3C

2A2C

0A0C0A0C0A0C

2A2C

Workload type

surprise-long

expected-long

penalty

none 0.5m

1.0m0.2m

Can WM handle unreliable cost estimates?Admission control thresholds

14


none

Admission control threshold

0

20

40

60

80

100

120

140

160

Wei

ghte

d m

akes

pan

(

in s

imu

lato

r ti

me

un

its)

3A3C

0A0C

3A3C

3A2C3A0C

3A0C

0A0C0A0C0A0C

0A0C

1.0m

absolute 12000

penalty absolute 5000none absolute 5000,

progress <30%relative 1.2x

Can WM handle unreliable cost estimates?

Adm ctl + exec ctl with different kill thresholds

15


0

20

40

60

80

100

120

140

160

Wei

ghte

d m

akes

pan

(in

sim

ulat

or t

ime

unit

s)

penalty

2A2C 2A2C2A2C2A2C

2A2C

2A0C

2A0C

none killkill &

requeuesuspend&

resume

relative 1.2x

absolute 5000

Can WM handle unreliable cost estimates?

Execution control actions (w/ admission control 1.0m)

16


0

20

40

60

80

100

120

140

160

Wei

ghte

d m

akes

pan

(in

sim

ulat

or t

ime

unit

s)

2A2C 2A0C2A0C

2A0C2A0C 2A0C

2A0C

2A0C

2A0C

2A2C

MPL 4 MPL 10 (overload)

penalty

relative 1.2xabsolute 12000none

absolute 5000absolute 5000,progress <30%

Can workload management handle system overload?

17


Conclusion

• Systematic study of workload management policies to mitigate the impact of long-running queries

• Can workload management handle…– unreliable cost estimates– unobserved resource contention– system overload

• Value of this work: experimental framework for studying more challenging workload management problems

18


Related work (excerpt)

D. G. Benoit. Automated Diagnosis and Control of DBMS Resources. EDBT PhD. Workshop, 2000

S. Chaudhuri, R. Kaushik, and R. Ramamurthy. When Can We Trust Progress Estimators for SQL Queries? SIGMOD 2005

S. Chaudhuri, R. Kaushik, R. Ramamurthy, and A. Pol. Stop-and-Restart Style Execution for Long Running Decision Support Queries. VLDB 2007

S. Krompass, H. Kuno, U. Dayal, and A. Kemper. Dynamic Workload Management for Very Large Data Warehouses: Juggling Feathers and Bowling Balls. VLDB 2007

G. Luo, J. F. Naughton, and P. S. Yu. Multi-query SQL Progress Indicators, EDBT 2006

Workload management tools: HP Neoview, IBM Workload Manager for DB2, Microsoft SQL Server, Oracle Database Resource Manager, Teradata Dynamic Workload Manager

Technische Universität München Hewlett-Packard Laboratories Managing Long-Running Queries Stefan...

Documents

Transcript of Technische Universität München Hewlett-Packard Laboratories Managing Long-Running Queries Stefan...