A Really Bad Graph. For Discussion Today Project Proposal 1.Statement of hypothesis 2.Workload...

A Really Bad Graph

For Discussion TodayProject Proposal1. Statement of hypothesis2. Workload decisions3. Metrics to be used4. Method

© 1998, Geoff Kuenning

Designing Experiments

• Introduction• 2k factorial designs• 2kr factorial designs• 2k-p fractional factorial designs• One-factor experiments• Two-factor full factorial design without

replications• Two-factor full factorial design with

replications• General full factorial designs with k factors



Introduction To Experiment Design

• You know your metrics

• You know your factors

• You know your levels

• You’ve got your instrumentation and test loads

• Now what?



Goals in Experiment Design

• Obtain maximum information• With minimum work

– Typically meaning minimum number of experiments

• More experiments aren’t better if you’re the one who has to perform them

• Well-designed experiments are also easier to analyze



Experimental Replications

• The system under study will be run with varying levels of different factors, potentially with differing workloads

• A run with a particular set of levels and other inputs is a replication

• Often, you need to do multiple replications with a single set of levels and other inputs– For statistical validation



Interacting Factors

• Some factors have effects completely independent of each other– Double the factor’s level, halve the

response, regardless of other factors• But the effects of some factors depends

on the values of other factors– Interacting factors

• Presence of interacting factors complicates experimental design



Basic Problem in Designing

Experiments• You have chosen some number of factors• They may or may not interact• How can you design an experiment that

captures the full range of the levels?– With minimum amount of work

• Which combination or combinations of the levels of the factors do you measure?



Common Mistakes in Experimentation

• Ignoring experimental error

• Uncontrolled parameters

• Not isolating effects of different factors

• One-factor-at-a-time experiment designs

• Interactions ignored

• Designs require too many experiments



Types of Experimental Designs

• Simple designs

• Full factorial design

• Fractional factorial design

Experimental Design

(l1,0, l1,1, … , l1,n1-1) x (l2,0, l2,1, … , l2,n2-1) x …

x (lk,0, lk,1, … , lk,nk-1)

k different factors, each factor with ni levelsr replications

Factor 1

Factor k

Factor 2


Simple Designs

• Vary one factor at a time

• For k factors with ith factor having ni levels -

• Assumes factors don’t interact• Usually more effort than required• Don’t use it, usually

n nii

k

1 1

1

Simple Designs

(l1,0, l1,1, … , l1,n-1) x (l2,0, l2,1, … , l2,n-1) x …

x (lk,0, lk,1, … , lk,n-1)

Factor 1

Factor k

Factor 2

fix

vary

Simple Designs

(l1,0, l1,1, … , l1,n-1) x (l2,0, l2,1, … , l2,n-1) x …

x (lk,0, lk,1, … , lk,n-1)

Factor 1

Factor k

Factor 2


Full Factorial Designs

• For k factors with ith factor having ni levels -

• Test every possible combination of factors’ levels

• Captures full information about interaction• A hell of a lot of work, though

n nii

k

1

Full Factorial Designs

(l1,0, l1,1, … , l1,n-1) x (l2,0, l2,1, … , l2,n-1) x …

x (lk,0, lk,1, … , lk,n-1)

Factor 1

Factor k

Factor 2


Reducing the Work in Full Factorial Designs

• Reduce number of levels per factor– Generally a good choice– Especially if you know which factors are

most important - use more levels for them

• Reduce the number of factors– But don’t drop important ones

• Use fractional factorial designs


Fractional Factorial Designs

• Only measure some combination of the levels of the factors

• Must design carefully to best capture any possible interactions

• Less work, but more chance of inaccuracy

• Especially useful if some factors are known not to interact

(l1,0, l1,1, … , l1,n-1) x (l2,0, l2,1, … , l2,n-1) x …

x (lk,0, lk,1, … , lk,n-1)

Fractional Factorial Designs

Factor 1

Factor k

Factor 2


2k Factorial Designs

• Used to determine the effect of k factors– Each with two alternatives or levels

• Often used as a preliminary to a larger performance study– Each factor measured at its maximum and

minimum level– Perhaps offering insight on importance and

interaction of various factors


Unidirectional Effects

• Effects that only increase as the level of a factor increases– Or visa versa

• If this characteristic is known to apply, a 2k factorial design at minimum and maximum levels is useful

• Shows whether the factor has a significant effect


22 Factorial Designs

• Two factors with two levels each

• Simplest kind of factorial experiment design

• Concepts developed here generalize

• A form of regression can be easily used here

• Simplest to show with an example


22 Factorial Design Example

• The Time Warp Operating System• Designed to run discrete event simulations in

parallel• Using an optimistic method• Goal is fastest possible completion of a given

simulation• Usually quality is expressed in terms of speedup

• Here, the simpler metric of runtime is used


Factors and Levels for Time Warp Example

• First factor - number of nodes used to run the simulation– Vary between 8 and 64

• Second factor - whether or not dynamic load management is used– To migrate work from node to node as load in the

simulation changes

• Other factors exists, but ignore them for now


Defining Variables for the 22 Factorial TW

Example

xA 11 if 8 nodes

if 64 nodes

xB 11 if no dynamic load management

if dynamic load management used


Sample Data For Example

• Single runs of one benchmark simulation

DLM(+1)

NODLM(-1)

8 Nodes (-1) 64 Nodes (+1)

820

776 197

217


Regression Model for Example

• y = q0 + qAxA + qBxB + qABxAxB

• Note this is a nonlinear model

820 = q0 -qA - qB + qAB

217 = q0 +qA - qB - qAB

776 = q0 -qA + qB - qAB

197 = q0 +qA + qB + qAB


Regression Model, Con’t

• 4 equations in 4 unknowns

Another way to look at it shown in this table -Experiment A B y

1 -1 -1 y1

2 1 -1 y2

3 -1 1 y3

4 1 1 y4


Solving the Equations

q0 = 1/4(820 + 217 + 776 + 197) = 502.5

qA = 1/4(-820 + 217 - 776 + 197) = -295.5

qB = 1/4(-820 - 217 + 776 + 197) = -16

qAB = 1/4(820 - 217 - 776 + 197) = 6

So,

y = 502.5 - 295.5xA - 16xB + 6xAxB


The Sign Table Method

• Another way of looking at the problem in a tabular form

I A B AB y1 -1 -1 1 8201 1 -1 -1 2171 -1 1 -1 7761 1 1 1 1972010 -1182 -64 24 Total502.5 -295.5 -16 6

Total/4


Allocation of Variation for 22 Model

• Calculate the sample variance of y

Numerator is the SST - total variation

SST = 22qA2 + 22qB

2 + 22qAB2

• We can use this to explain what causes the variation in y

s

y yy

ii22

12

2

2

2 1


Terms in the SST

• 22qA2 is part of variation explained by

the effect of A - SSA

• 22qB2 is part of variation explained by

the effect of B - SSB

• 22qAB2 is part of variation explained by

the effect of the interaction of A and B - SSAB

SST = SSA + SSB + SSAB


Variations in Our Example

• SST = 350449

• SSA = 349281

• SSB = 1024

• SSAB = 144

• We can now calculate the fraction of the total variation caused by each effect


Fractions of Variation in Our Example

• Fraction explained by A is 99.67%• Fraction explained by B is 0.29%• Fraction explained by the interaction of

A and B is 0.04%• So almost all the variation comes from

the number of nodes• So if you want to run faster, apply more

nodes, don’t turn on dynamic load management


General 2k Factorial Designs

• Used to explain the effects of k factors, each with two alternatives or levels

• 22 factorial designs are a special case

• Methods developed there extend to the more general case

• But many more possible interactions between pairs (and trios, etc.) of factors

For Discussion Tuesday March 25

• Survey your proceedings for just one paper in which factorial design has been used or, if none, one in which it could have been used effectively.

© 2003, Carla Ellis

A Really Bad Graph. For Discussion Today Project Proposal 1.Statement of hypothesis 2.Workload...

Documents

Transcript of A Really Bad Graph. For Discussion Today Project Proposal 1.Statement of hypothesis 2.Workload...