Post on 01-Feb-2022
Introduction Experimental design Output analysis Conclusions References
Design of Computer Experiments
Christian DehlendorffDTU Informatics
Technical University of Denmark
November 10th 2010
cd@imm.dtu.dk
DTU Informatics 1/51
Introduction Experimental design Output analysis Conclusions References
Outline1 Introduction
IntroductionComputer experimentsCase-study
2 Experimental designIntroductionOptimal designsCrossed designsTop-down design
3 Output analysisKrigingExtension
4 Conclusionscd@imm.dtu.dk
DTU Informatics 2/51
Introduction Experimental design Output analysis Conclusions References
Introduction
Introduction
The overall topic of this presentation is computer experiments andit consists of two main parts
Design of experimentsAnalysis (Kriging)
cd@imm.dtu.dk
DTU Informatics 3/51
Introduction Experimental design Output analysis Conclusions References
Introduction
Introduction
Design and Analysis of Computer Experiments (DACE)In recent years received increasingly more attentionIdea: Replace physical experimentation with experiments on acomputer model that mimics the physical systemComputer models are used in many areasTime consuming and complex
cd@imm.dtu.dk
DTU Informatics 4/51
Introduction Experimental design Output analysis Conclusions References
Computer experiments
Physical experimentation
Important conceptsRandomizationBlockingReplication
cd@imm.dtu.dk
DTU Informatics 5/51
Introduction Experimental design Output analysis Conclusions References
Computer experiments
Computer experiments
Computer (simulation) experiments are experiments on acomputer code (simulation model)
study physical phenomena using simulationbypass constraints in physical experiments (costs etc.)
Computer experiments are different from physical experimentsno experimental error
or generated artificiallyno blocking neededno randomization needed
cd@imm.dtu.dk
DTU Informatics 6/51
Introduction Experimental design Output analysis Conclusions References
Computer experiments
Computer models
Applications: Hospital units, supermarkets, spread ofinfectious diseases, computational fluid dynamics and manymoreAllow experimentation in areas where physicalexperimentation is impossible, e.g.,
health care applications: patient safety, budget constraints
cd@imm.dtu.dk
DTU Informatics 7/51
Introduction Experimental design Output analysis Conclusions References
Case-study
Case-study
Simulation model for an orthopedic surgical unit at GentofteHospitalSimulates the patients’ route through the surgical unitIncludes the staff and other resources needed at the surgicalunitDiscrete event simulation modelAnimation
cd@imm.dtu.dk
DTU Informatics 8/51
Introduction Experimental design Output analysis Conclusions References
Case-study
Hospital unit model
cd@imm.dtu.dk
DTU Informatics 9/51
Introduction Experimental design Output analysis Conclusions References
Case-study
Hospital unit model
cd@imm.dtu.dk
DTU Informatics 9/51
Introduction Experimental design Output analysis Conclusions References
Case-study
Hospital unit model
cd@imm.dtu.dk
DTU Informatics 9/51
Introduction Experimental design Output analysis Conclusions References
Introduction
Design of Experiments
Design of experimentsChoose values for the input variables (settings)Choose the number of different combinations of input settingsto test (runs)Choose the sequence {x1, . . . , xn}Challenges:
limitation on the length of the sequencemany factorscoverage
cd@imm.dtu.dk
DTU Informatics 10/51
Introduction Experimental design Output analysis Conclusions References
Introduction
Random sampling
Input values x1, . . . , xn are independent identical samples fromU(C s) where
C s is the domain for the s input variables (usually [0, 1]s)U() is the uniform distribution
Example: x1 = [x11, . . . , x1s ] and xij ∼ U(0, 1)
cd@imm.dtu.dk
DTU Informatics 11/51
Introduction Experimental design Output analysis Conclusions References
Introduction
Latin Hypercube Design
Formal design algorithm for s design variables and n runs1 Take s independent permutations πj(1), . . . , πj(n) of the
integers 1, . . . , n, j = 1, . . . , s
cd@imm.dtu.dk
DTU Informatics 12/51
Introduction Experimental design Output analysis Conclusions References
Introduction
Latin Hypercube Design
Formal design algorithm for s design variables and n runs1 Take s independent permutations πj(1), . . . , πj(n) of the
integers 1, . . . , n, j = 1, . . . , s2 Take ns mutually independent random numbers U j
k ∼ U(0, 1)
cd@imm.dtu.dk
DTU Informatics 12/51
Introduction Experimental design Output analysis Conclusions References
Introduction
Latin Hypercube Design
Formal design algorithm for s design variables and n runs1 Take s independent permutations πj(1), . . . , πj(n) of the
integers 1, . . . , n, j = 1, . . . , s2 Take ns mutually independent random numbers U j
k ∼ U(0, 1)
3 Define the setting of the j factor in the kth run:x j
k =πj (k)−U j
kn
cd@imm.dtu.dk
DTU Informatics 12/51
Introduction Experimental design Output analysis Conclusions References
Introduction
Latin Hypercube Design
Formal design algorithm for s design variables and n runs(mid-point method)
1 Take s independent permutations πj(1), . . . , πj(n) of theintegers 1, . . . , n, j = 1, . . . , s
2 Define the setting of the j factor in the kth run:x j
k =πj (k)−0.5
n
cd@imm.dtu.dk
DTU Informatics 12/51
Introduction Experimental design Output analysis Conclusions References
Introduction
LHS - examplesRandom LHS
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
x1
x 2
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
cd@imm.dtu.dk
DTU Informatics 13/51
Introduction Experimental design Output analysis Conclusions References
Introduction
LHS - examplesMid-point LHS
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
x1
x 2
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
cd@imm.dtu.dk
DTU Informatics 13/51
Introduction Experimental design Output analysis Conclusions References
Introduction
LHS - examplesBoth
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
x1
x 2
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
cd@imm.dtu.dk
DTU Informatics 13/51
Introduction Experimental design Output analysis Conclusions References
Optimal designs
Optimal designs
Modification of LHD3
Orthogonal LHD: the columns of the design are orthogonalLHD under minimax criterion: minimize the maximumdistance from any point in the input domain to its closestdesign pointLHD under maximin criterion: maximize the minimumdistance between design pointsand many more (A, D, G, IMSE etc.)
cd@imm.dtu.dk
DTU Informatics 14/51
Introduction Experimental design Output analysis Conclusions References
Optimal designs
Illustration of maximin and minimax
●
●
●
●
●
●
●
●
●
●
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
x1
x 2
cd@imm.dtu.dk
DTU Informatics 15/51
Introduction Experimental design Output analysis Conclusions References
Optimal designs
Uniform design
A space-filling designIntuition: scatter design points such that they mimic theuniform distributionRobust against model specificationFlexible in terms of runs and levels of the factors
cd@imm.dtu.dk
DTU Informatics 16/51
Introduction Experimental design Output analysis Conclusions References
Optimal designs
Uniform designs
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
x1
x 2
cd@imm.dtu.dk
DTU Informatics 17/51
Introduction Experimental design Output analysis Conclusions References
Optimal designs
Uniform designs
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
x1
x 2
cd@imm.dtu.dk
DTU Informatics 17/51
Introduction Experimental design Output analysis Conclusions References
Crossed designs
Designing simulation experiments with controllable anduncontrollable factors
Experimental design to handle a model having bothControllable factors
controllable in computer model and physical systema setting of the controllable factors is called a whole plot
Uncontrollable factorscontrollable in computer model but not in the physical systemone setting of the uncontrollable factors is called a subplot
cd@imm.dtu.dk
DTU Informatics 18/51
Introduction Experimental design Output analysis Conclusions References
Crossed designs
Crossed designs
The standard method is a crossed design ofa sampling plan for the controllable factorsa sampling plan for the uncontrollable factors
cd@imm.dtu.dk
DTU Informatics 19/51
Introduction Experimental design Output analysis Conclusions References
Crossed designs
Crossed designs
Controllable Environmental settingsetting 1 2 3 4 5
1 xe1 xe2 xe3 xe4 xe52 xe1 xe2 xe3 xe4 xe53 xe1 xe2 xe3 xe4 xe54 xe1 xe2 xe3 xe4 xe55 xe1 xe2 xe3 xe4 xe5
cd@imm.dtu.dk
DTU Informatics 20/51
Introduction Experimental design Output analysis Conclusions References
Crossed designs
Crossed designs
Main problem is the replication of the settings of theuncontrollable factors
The actual subplots are not importantThe coverage of the design space for the uncontrollablefactors may be a problem
cd@imm.dtu.dk
DTU Informatics 21/51
Introduction Experimental design Output analysis Conclusions References
Top-down design
A different methodology
Use different settings of the uncontrollable factors for eachwhole plotFor m whole plots and q uncontrollable factor settings at eachwhole plot N = m × q different settings of the uncontrollablefactors are testedChallenge: How to select the different settings of theuncontrollable factors for each whole plot?
cd@imm.dtu.dk
DTU Informatics 22/51
Introduction Experimental design Output analysis Conclusions References
Top-down design
Bottom-up approach
Initial ideaDefine q areas (or hyper volumes)Construct a space-filling design in each hyper volumeAssign one sample point from each hyper volume to eachwhole plot
cd@imm.dtu.dk
DTU Informatics 23/51
Introduction Experimental design Output analysis Conclusions References
Top-down design
Illustration of bottom-up design
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
x1
x 2
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
cd@imm.dtu.dk
DTU Informatics 24/51
Introduction Experimental design Output analysis Conclusions References
Top-down design
Bottom-up design
●
●
● ●●
●
●
●
●●
●●
●●
●●
●●
● ●
0.0 0.5 1.0 1.5 2.0
12
510
2050
100
Ratio of minimum distance
Nor
mal
ized
WD
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
Uniform subdesignMaximin subdesign
Maximin full design
cd@imm.dtu.dk
DTU Informatics 25/51
Introduction Experimental design Output analysis Conclusions References
Top-down design
Construction of top-down designsTop-down design method1 Construct overall uniform
design (Ub)
2 Split overall design into qsub regions with m settings
3 Use uniform design of size qas center points (Us)
4 Apply exchange algorithm toget well defined regions
5 Assign one setting fromeach sub region to eachwhole-plot
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
x1
x 2
cd@imm.dtu.dk
DTU Informatics 26/51
Introduction Experimental design Output analysis Conclusions References
Top-down design
Construction of top-down designsTop-down design method1 Construct overall uniform
design (Ub)2 Split overall design into q
sub regions with m settings
3 Use uniform design of size qas center points (Us)
4 Apply exchange algorithm toget well defined regions
5 Assign one setting fromeach sub region to eachwhole-plot
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
x1
x 2
cd@imm.dtu.dk
DTU Informatics 26/51
Introduction Experimental design Output analysis Conclusions References
Top-down design
Construction of top-down designsTop-down design method1 Construct overall uniform
design (Ub)2 Split overall design into q
sub regions with m settings3 Use uniform design of size q
as center points (Us)
4 Apply exchange algorithm toget well defined regions
5 Assign one setting fromeach sub region to eachwhole-plot
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
x1
x 2
cd@imm.dtu.dk
DTU Informatics 26/51
Introduction Experimental design Output analysis Conclusions References
Top-down design
Construction of top-down designsTop-down design method1 Construct overall uniform
design (Ub)2 Split overall design into q
sub regions with m settings3 Use uniform design of size q
as center points (Us)4 Apply exchange algorithm to
get well defined regions
5 Assign one setting fromeach sub region to eachwhole-plot
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
x1
x 2
cd@imm.dtu.dk
DTU Informatics 26/51
Introduction Experimental design Output analysis Conclusions References
Top-down design
Construction of top-down designsTop-down design method1 Construct overall uniform
design (Ub)2 Split overall design into q
sub regions with m settings3 Use uniform design of size q
as center points (Us)4 Apply exchange algorithm to
get well defined regions5 Assign one setting from
each sub region to eachwhole-plot
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
x1
x 2
cd@imm.dtu.dk
DTU Informatics 26/51
Introduction Experimental design Output analysis Conclusions References
Top-down design
Exchange algorithm
Exchange the centers for the points xi and xj if
∆ij = [d(xi , c(xi )) + d(xj , c(xj))]− [d(xi , c(xj)) + d(xj , c(xi ))] > 0(1)
where c(xi ) is the location of xi ’s center and d() measures theEuclidean distance.
cd@imm.dtu.dk
DTU Informatics 27/51
Introduction Experimental design Output analysis Conclusions References
Top-down design
Sub designs
Table: Whole plot performance for different numbers of uncontrollablefactors (su) and different numbers of overall number of subplots (N).The whole plot size is kept fixed at k = 10.
su N max-min min-max N max-min min-max2 200 1.95 2.84 400 1.65 3.083 200 2.29 4.21 400 2.01 5.244 200 2.37 3.99 400 2.10 4.815 200 2.75 3.43 400 2.72 3.946 200 2.67 3.14 400 2.66 3.827 200 2.32 2.82 400 2.39 3.308 200 2.21 2.62 400 2.26 2.929 200 2.08 2.39 400 2.01 2.6910 200 1.82 2.08 400 1.97 2.5111 200 1.67 1.83 400 1.73 2.0912 200 1.58 1.71 400 1.58 1.9213 200 1.42 1.54 400 1.46 1.6914 200 1.41 1.53 400 1.41 1.6715 200 1.35 1.44 400 1.37 1.5416 200 1.30 1.38 400 1.29 1.5117 200 1.27 1.34 400 1.27 1.4118 200 1.22 1.27 400 1.24 1.3519 200 1.20 1.24 400 1.21 1.32
cd@imm.dtu.dk
DTU Informatics 28/51
Introduction Experimental design Output analysis Conclusions References
Top-down design
Overall uniformity ensured - good sub design coverage
Benefits of using the top-down constructionOverall uniformity ensuredGood sub design coverage
in the previous example 2-3 times higher uniformity criteria(smaller is better) compared to a uniform design of same size
Better understanding of changes in the uncontrollable factorsm times as many subplots as the crossed design
cd@imm.dtu.dk
DTU Informatics 29/51
Introduction Experimental design Output analysis Conclusions References
Top-down design
Example
Two designsCrossed designTop-down design
Analyzed with generalized additive models
cd@imm.dtu.dk
DTU Informatics 30/51
Introduction Experimental design Output analysis Conclusions References
Top-down design
16 controllable factors settings and 10 uncontrollable factorsettings
Table: Controllable factors for simulation experiment. Currentcorresponds to the current setting at the surgical unit
Factor Low High CurrentAnesthesiologists (A) 2 3 2Porters (B) 3 4 3Recovery beds (C) 6 8 6Operating days (D) 5 4 5
cd@imm.dtu.dk
DTU Informatics 31/51
Introduction Experimental design Output analysis Conclusions References
Top-down design
Example: Crossed vs. top-down design
Top-down design
Recovery bed occupancy(% increase)
Par
tial e
ffect
on
CV
aR (
min
utes
)
6 Recovery beds
−2
0
2
4
−20 −2 15 32 50
Top−down design
Recovery bed occupancy(% increase)
Par
tial e
ffect
on
CV
aR (
min
utes
)
8 Recovery beds
−2
0
2
4
−20 −2 15 32 50Anesthesiologists occupancy
(% increase)
Par
tial e
ffect
on
CV
aR (
min
utes
)
2 Anesthesiologists
−2
0
2
4
−20 −2 15 32 50Anesthesiologists occupancy
(% increase)
Par
tial e
ffect
on
CV
aR (
min
utes
)
3 Anesthesiologists
−2
0
2
4
−20 −2 15 32 50
Recovery bed occupancy(% increase)
Par
tial e
ffect
on
CV
aR (
min
utes
)
6 Recovery beds
−4
−2
0
2
4
−20 −2 15 32 50
Crossed design
Recovery bed occupancy(% increase)
Par
tial e
ffect
on
CV
aR (
min
utes
)
8 Recovery beds
−4
−2
0
2
4
−20 −2 15 32 50Anesthesiologists occupancy
(% increase)
Par
tial e
ffect
on
CV
aR (
min
utes
)
2 Anesthesiologists
−30
−20
−10
0
10
20
30
−20 −2 15 32 50Anesthesiologists occupancy
(% increase)
Par
tial e
ffect
on
CV
aR (
min
utes
)
3 Anesthesiologists
−30
−20
−10
0
10
20
30
−20 −2 15 32 50
Crossed design
Recovery bed occupancy(% increase)
Par
tial e
ffect
on
CV
aR (
min
utes
)
6 Recovery beds
−2
0
2
4
−20 −2 15 32 50
Top−down design
Recovery bed occupancy(% increase)
Par
tial e
ffect
on
CV
aR (
min
utes
)
8 Recovery beds
−2
0
2
4
−20 −2 15 32 50Anesthesiologists occupancy
(% increase)
Par
tial e
ffect
on
CV
aR (
min
utes
)
2 Anesthesiologists
−2
0
2
4
−20 −2 15 32 50Anesthesiologists occupancy
(% increase)
Par
tial e
ffect
on
CV
aR (
min
utes
)
3 Anesthesiologists
−2
0
2
4
−20 −2 15 32 50
Recovery bed occupancy(% increase)
Par
tial e
ffect
on
CV
aR (
min
utes
)
6 Recovery beds
−4
−2
0
2
4
−20 −2 15 32 50
Crossed design
Recovery bed occupancy(% increase)
Par
tial e
ffect
on
CV
aR (
min
utes
)
8 Recovery beds
−4
−2
0
2
4
−20 −2 15 32 50Anesthesiologists occupancy
(% increase)
Par
tial e
ffect
on
CV
aR (
min
utes
)
2 Anesthesiologists
−30
−20
−10
0
10
20
30
−20 −2 15 32 50Anesthesiologists occupancy
(% increase)
Par
tial e
ffect
on
CV
aR (
min
utes
)
3 Anesthesiologists
−30
−20
−10
0
10
20
30
−20 −2 15 32 50
cd@imm.dtu.dk
DTU Informatics 32/51
Introduction Experimental design Output analysis Conclusions References
Top-down design
Example: Crossed vs. top-down design
Top-down design
Recovery bed occupancy(% increase)
Par
tial e
ffect
on
CV
aR (
min
utes
)
6 Recovery beds
−2
0
2
4
−20 −2 15 32 50
Top−down design
Recovery bed occupancy(% increase)
Par
tial e
ffect
on
CV
aR (
min
utes
)
8 Recovery beds
−2
0
2
4
−20 −2 15 32 50Anesthesiologists occupancy
(% increase)
Par
tial e
ffect
on
CV
aR (
min
utes
)
2 Anesthesiologists
−2
0
2
4
−20 −2 15 32 50Anesthesiologists occupancy
(% increase)
Par
tial e
ffect
on
CV
aR (
min
utes
)
3 Anesthesiologists
−2
0
2
4
−20 −2 15 32 50
Recovery bed occupancy(% increase)
Par
tial e
ffect
on
CV
aR (
min
utes
)
6 Recovery beds
−4
−2
0
2
4
−20 −2 15 32 50
Crossed design
Recovery bed occupancy(% increase)
Par
tial e
ffect
on
CV
aR (
min
utes
)
8 Recovery beds
−4
−2
0
2
4
−20 −2 15 32 50Anesthesiologists occupancy
(% increase)
Par
tial e
ffect
on
CV
aR (
min
utes
)
2 Anesthesiologists
−30
−20
−10
0
10
20
30
−20 −2 15 32 50Anesthesiologists occupancy
(% increase)
Par
tial e
ffect
on
CV
aR (
min
utes
)
3 Anesthesiologists
−30
−20
−10
0
10
20
30
−20 −2 15 32 50
Crossed designRecovery bed occupancy
(% increase)
Par
tial e
ffect
on
CV
aR (
min
utes
)
6 Recovery beds
−2
0
2
4
−20 −2 15 32 50
Top−down design
Recovery bed occupancy(% increase)
Par
tial e
ffect
on
CV
aR (
min
utes
)
8 Recovery beds
−2
0
2
4
−20 −2 15 32 50Anesthesiologists occupancy
(% increase)
Par
tial e
ffect
on
CV
aR (
min
utes
)
2 Anesthesiologists
−2
0
2
4
−20 −2 15 32 50Anesthesiologists occupancy
(% increase)
Par
tial e
ffect
on
CV
aR (
min
utes
)
3 Anesthesiologists
−2
0
2
4
−20 −2 15 32 50
Recovery bed occupancy(% increase)
Par
tial e
ffect
on
CV
aR (
min
utes
)
6 Recovery beds
−4
−2
0
2
4
−20 −2 15 32 50
Crossed design
Recovery bed occupancy(% increase)
Par
tial e
ffect
on
CV
aR (
min
utes
)8 Recovery beds
−4
−2
0
2
4
−20 −2 15 32 50Anesthesiologists occupancy
(% increase)
Par
tial e
ffect
on
CV
aR (
min
utes
)
2 Anesthesiologists
−30
−20
−10
0
10
20
30
−20 −2 15 32 50Anesthesiologists occupancy
(% increase)
Par
tial e
ffect
on
CV
aR (
min
utes
)
3 Anesthesiologists
−30
−20
−10
0
10
20
30
−20 −2 15 32 50
cd@imm.dtu.dk
DTU Informatics 33/51
Introduction Experimental design Output analysis Conclusions References
Kriging
Kriging
Interpolation techniqueOriginates from geo-statisticsHeavily used in the analysis of simulation and computerexperiments for deterministic outputCan fit complex functions
cd@imm.dtu.dk
DTU Informatics 34/51
Introduction Experimental design Output analysis Conclusions References
Kriging
Basics of Kriging
Approximate deterministic function y(x) with
Y (x) = f (x)β + Z (x) (2)
where Z (x) is a random field with correlation function
R(x1, x2) = exp(−
s∑l=1
θl (x l1 − x l
2)2)
(3)
R() is the central part of the interpolator
cd@imm.dtu.dk
DTU Informatics 35/51
Introduction Experimental design Output analysis Conclusions References
Kriging
Kriging basics
β = (FT R(θ)−1F)−1FT R(θ)−1y (4)
σ2 =1
n − p (y− Fβ)T R(θ)−1(y− Fβ) (5)
θ = argminθ
[(n − p) log σ2 + log(|R(θ)|)
](6)
cd@imm.dtu.dk
DTU Informatics 36/51
Introduction Experimental design Output analysis Conclusions References
Kriging
Kriging basics
Kriging predictor
y(x) = f(x)T β + r(x)T R(θ)−1(y− Fβ) (7)
with
MSE (y(x)) = σ2[1−QT(FTR−1F)−1Q− r(x)TR−1r(x)] (8)
Q = FTR−1r(x)− f(x)
cd@imm.dtu.dk
DTU Informatics 37/51
Introduction Experimental design Output analysis Conclusions References
Kriging
Kriging example
●
●
●
●
●
0.0 0.2 0.4 0.6 0.8 1.0
−1.
0−
0.5
0.0
0.5
1.0
5 points
x
y
●
●
●
●
●
0.0 0.2 0.4 0.6 0.8 1.0
−1.
0−
0.5
0.0
0.5
1.0
6 points
x
y
x
●
●
●
●
●
0.0 0.2 0.4 0.6 0.8 1.0
−1.
0−
0.5
0.0
0.5
1.0
7 points
x
y
x
x
●
●
●
●
●
0.0 0.2 0.4 0.6 0.8 1.0
−1.
0−
0.5
0.0
0.5
1.0
8 points
x
y
x
x x
cd@imm.dtu.dk
DTU Informatics 38/51
Introduction Experimental design Output analysis Conclusions References
Extension
Extension
Consider the following simulation model
where x are qualitative factors and u are quantitative factorsm settings of the qualitative factors (stratas)q settings of the quantitative factors in each strata
cd@imm.dtu.dk
DTU Informatics 39/51
Introduction Experimental design Output analysis Conclusions References
Extension
Models
Consider each strata separatelym models based on q observations each
Include all observations in a combined modelone model based on m × q observationschallenge: how should the correlation between observationsfrom different stratas be handled
cd@imm.dtu.dk
DTU Informatics 40/51
Introduction Experimental design Output analysis Conclusions References
Extension
Correlation structure
The jth strata is run at q different settings of the quantitativefactors
uj1, . . . ,ujq j = 1, . . . ,m (9)
and the corresponding model output is
yj1, . . . , yjq j = 1, . . . ,m (10)
These observations are assumed correlated by
R(ujk ,ulm) = exp(−
s∑i=1
θl (uljk − ui
lm)2)
(11)
cd@imm.dtu.dk
DTU Informatics 41/51
Introduction Experimental design Output analysis Conclusions References
Extension
Correlation structure
For observations from different stratasWe expect R(uij ,uik) ≥ R(uij ,ui ′k) (same quantitative factorsettings)We assume that the quantitative factors behave similarly indifferent stratasR(uij ,ui ′k) = cii ′ exp
(−
s∑l=1
θl (ulij − ul
i ′l )2)
0 ≤ cii′ ≤ 1
cd@imm.dtu.dk
DTU Informatics 42/51
Introduction Experimental design Output analysis Conclusions References
Extension
Correlation structure
Two ways of choosing cii ′ considered1 Sample mean and standard deviation for each strata
cii′ = exp(−θµ(µi − µi′)
2 − θσ(log(σi )− log(σi′))2)
2 2-stage approachCorrelation parameters from m Kriging model fitted for eachstratacii′ = exp
(−∑s
l=1 θs+l (θli − θl
i′)2)
cd@imm.dtu.dk
DTU Informatics 43/51
Introduction Experimental design Output analysis Conclusions References
Extension
Correlation structure
Mean and varianceU =
[U M
](12)
where
M =
µ1 log(σ1)µ2 log(σ2)...
...µm log(σm)
⊗ 1q×1 (13)
cd@imm.dtu.dk
DTU Informatics 44/51
Introduction Experimental design Output analysis Conclusions References
Extension
Correlation structure
2-stage approachU =
[U C
](12)
whereC = C⊗ 1q×1 (13)
and
C =
θ11 · · · θ1s... . . . ...θm1 · · · θms
(14)
cd@imm.dtu.dk
DTU Informatics 44/51
Introduction Experimental design Output analysis Conclusions References
Extension
Case-study
We consider the hospital unit and two cases24 factorial design for the qualitative factors with 10quantitative factor settings each (top-down design)20 qualitative factor settings (predicted to have short CVaRwaiting times) with 20 quantitative factor settings each(top-down design)
cd@imm.dtu.dk
DTU Informatics 45/51
Introduction Experimental design Output analysis Conclusions References
Extension
Case-study
Three reference modelscii ′ = I(i = i ′) + I(i 6= i ′)θc
Hung et al. (HRM): cii ′ = exp(−
dx∑l=1
θxl I(x li 6= x l
i ′)
)Zhou et al. (ZQZ): hypersphere parameterization of thecorrelation between observations from different stratas
cd@imm.dtu.dk
DTU Informatics 46/51
Introduction Experimental design Output analysis Conclusions References
Extension
Zhou et al.
T = LLT is the penalty matrix and L is given as
Lrs =
1 r = s = 1cos(θr ,s) s = 1 (r > 1)
sin(θr ,1) · · · sin(θr ,s−1) cos(θr ,s) s = 2, . . . , r − 1 (r > 1)
sin(θr ,1) · · · sin(θr ,r−2) sin(θr ,r−1) r = s (r > 1)
(15)where Lrs is the (rs)th element of L and θr ,s ∈ [0, π].
cd@imm.dtu.dk
DTU Informatics 47/51
Introduction Experimental design Output analysis Conclusions References
Extension
Zhou et al.
With 3 different settings of the qualitative factors
T3 =
1 0 0l21 l22 0l31 l32 l33
1 l21 l310 l22 l320 0 l33
(l21, l22) = (cos(θ21), sin(θ21))
(l31, l32, l33) = (cos(θ31), sin(θ31) cos(θ32), sin(θ31) sin(θ32))
cd@imm.dtu.dk
DTU Informatics 48/51
Introduction Experimental design Output analysis Conclusions References
Extension
Case-study
Table: Performance measured in MSPE
Model Correlation structure Example 1 Example 2
Kriging
θc 16.72 1.78g(µi , σi ) 9.71 2.002-stage 9.04 1.68HRM 11.93 1.83ZQZ 9.54 1.75
GAM 12.08 1.27
cd@imm.dtu.dk
DTU Informatics 49/51
Introduction Experimental design Output analysis Conclusions References
Conclusion
Widely applicable methodThe top-down design is an efficient design for handlingsimulation models with controllable and uncontrollable factorsKriging can be extended to more general types of computermodels
cd@imm.dtu.dk
DTU Informatics 50/51
Introduction Experimental design Output analysis Conclusions References
References[1] C. Dehlendorff, M. Kulahci, and K. K. Andersen. Designing simulation experiments with controllable and
uncontrollable factors for applications in health care. Journal of Royal Statistical Society: Series C, 60(1),2011. DOI: 10.1111/j.1467-9876.2010.00724.x.
[2] Christian Dehlendorff, Murat Kulahci, Søren Merser, and Klaus Kaae Andersen. Conditional value at risk as ameasure for waiting time in simulations of hospital units. Quality Technology and Quantitative Management,7(3):321–336, 2010. ISSN 1684-3703.
[3] Kai-Tai Fang, Runze Li, and Agus Sudjianto. Design and Modeling for Computer Experiments. Chapman &Hall/CRC, 2006.
[4] Ying Hung, V. Roshan Joseph, and Shreyes N. Melkote. Design and analysis of computer experiments withbranching and nested factors. Technometrics, 51(4):354–365, 2009.
[5] Jack P.C. Kleijnen. Kriging metamodeling in simulation: A review. European Journal of OperationalResearch, 192(3):707–716, 2009. ISSN 03772217.
[6] Jack P.C. Kleijnen. Design and Analysis of Simulation Experiments. Springer, 2008.[7] S.N. Lophaven, H.B. Nielsen, and J. Søndergaard. Dace - a matlab kriging toolbox version 2.0. Technical
Report IMM-REP-2002-12, Informatics and Mathematical Modelling, Technical University of Denmark, 2002.http://www.imm.dtu.dk/~hbn/publ/TR0212.ps.
[8] S.N. Lophaven, H.B. Nielsen, and J. Søndergaard. Aspects of the matlab toolbox dace. Technical ReportIMM-REP-2002-13, Informatics and Mathematical Modelling, Technical University of Denmark, 2002.http://www.imm.dtu.dk/~hbn/publ/TR0213.ps.
[9] Hong Qin and Kai-Tai Fang. Discrete discrepancy in factorial designs. Metrika, 60(1):59–72, 2004. ISSN00261335.
[10] Qiang Zhou, Peter Z.G. Qian, and Shiyu Zhou. A simple approach to emulation for computer models withqualitative and quantitative factors. Working paper: http://www.stat.wisc.edu/~zhiguang/qpqq2.pdf,2010.
cd@imm.dtu.dk
DTU Informatics 51/51