1

Experimental Design, Response Surface Analysis, and Optimization

2

Outline

Motivation and Terminology Difficulties in Solving the Basic

Problem Examples of Factors and Responses Types/Examples of Experimental

Design Full Factorial Designs Randomness of Effects Example: Full Factorial Design Situations with Many Factors Response Surfaces and Metamodels Regression Analysis Response Surface Methodology

3

Motivation and Terminology

Useful when there are many alternatives to consider (e.g., numerous capacity levels of various types, numerous parameters for a proposed inventory system)

Two basic types of variables: factors and responses

Factors: input parameters:-controllable or uncontrollable-quantitative or qualitative

Responses: outputs from the simulation model:

-uncertain in nature Basic problem: find the best levels (or values

of the parameters) in terms of the responses Experimental Design can tell you which

alternatives to simulate so that you obtain the desired information with the least amount of simulation

4

Multiple Responses

Uncertain Responses

Difficulties in Solving the Basic Problem

5

Examples of Factors

1. Mean interarrival time (uncontrollable, quantitative)

2. Mean service time (controllable or uncontrollable, quantitative)

3. Number of servers (controllable, quantitative)

4. Queuing discipline (controllable, qualitative)

5. Reorder point (controllable, quantitative)6. Mean interdemand time (uncontrollable,

quantitative)7. Distribution of interdemand time

(uncontrollable, qualitative)

6

1. Mean daily production rate2. Mean time in the system for patients3. Mean inventory level4. Number of customers who wait for more

than 5 minutes

Examples of Responses

7

Types/Examples of Experimental Designs

Completely Randomized Designs

Randomized Complete Block Designs

Nested Factorial Designs

Split Plot Type Designs

Latin Square Type Designs

Full Factorial Designs

Fractional Factorial Designs

8

2k Factorial Designs

Suppose that we have k (k > 2) factors. A 2k factorial design would require that two levels be chosen for each factor, and that n simulation runs (replications) be made at each of the 2k possible factor-level combinations (design points). For 3 factors, this yields a Design Matrix:

9

Design Matrix for 3 Factors

Factor 1 Factor 2 Factor 3

Design Point Level Level Level Response 1 - - - O1

2 + - - O2

3 - + - O3

4 + + - O4

5 - - + O5

6 + - + O6

7 - + + O7

8 + + + O8

“+” refers to one level of a factor and “-” refers to the other level. Normally, for quantitative factors, the smallest and largest levels for each factor are chosen.

2k Factorial Designs

10

2k Factorial Designs: Estimating Main Effects

The main effect of factor 1 is the change in the response variable as a result of the change in the level of the factor, averaged over all levels of all of the other factors

If the effect of some factor depends on the level of another factor, these factors are said to interact.

The degree of interaction (two-factor interaction effect) between two factors i and j is defined as half the difference between the average effect of factor i when factor j is at its “+” level ( and all factors other than i and j are held constant) and the average effect of i when j is at its “-” level; for example,

e

O O O O O O O O1

2 1 4 3 6 5 8 7

4

eO O O O O O O O

124 3 8 7 2 1 6 51

2 2 2

( ) ( ) ( ) ( )

11

Example: Full Factorial (2k) Design

Consider a simulation model of reorder point, reorder quantity inventory system. The two decision variables, or factors, to consider are the reorder point (P) and the reorder quantity (Q) for the inventory system. The maximum and minimum allowable values for each are given below:

Suppose that the response variable output by the model is the long-run average monthly cost (composed of three components: holding cost, shortage costs, and ordering costs) in thousands of dollars.

Factor MinimumValue

MaximumValue

P 20 40

Q 15 50

12

Example: Full Factorial (2k) Design

A 22 factorial design matrix with simulation results (for 10 replications at each design point) might be given by:

where a factor level of “-” indicates the minimum possible value for that factor, and a factor level of “+” indicates the maximum possible level; for example, design point 2 has P=40 and Q=15.

Design Point Factor Level for P Q

Response

1234

- - + - - + + +

135.6128.2121.7131.5

13

The response given is the average cost over the 10 replications. Now, the main effects are given by:

The interaction effect (ePQ) is given by:

Therefore, the average effect of increasing P from 20 to 40 is to increase monthly cost by 1.2, and the average effect of increasing Q from 15 to 50 is to decrease monthly cost by 5.3. Hence, it would be advisable to set P as low as possible and set Q as high as possible.

eO O O O

Q

( ) ( ) ( . . ) ( . . )

.3 1 4 2

2

1217 135 6 1315 128 2

25 3

eO O O O

P

( ) ( ) ( . . ) ( . . )

.2 1 4 3

2

128 2 135 6 1315 1217

212

ePQ

135 6 128 2 1217 1315

28 6

. . . ..

Example: Full Factorial (2k) Design

14

Also since the interaction effect is positive, it would seem advisable to set P and Q at opposite levels. (Of course, all of the above could be inferred from a cursory analysis of the responses for the various design points.) Note also that the literal interpretation of main effects assumes no interaction effects (pages 669 and 670 of Law and Kelton, 1991).

Example: Full Factorial (2k) Design

15

Randomness of Effects

Note that the main and interaction effects computed in the previous examples are just random variables. To determine if the effects are “significant” or real, and not due to random fluctuations, one could compute values for the main and interaction effects 10 times (once for each replication) and form confidence intervals for each of the main effects, and the interactions effect. If the confidence interval contains 0, then the effect is not statistically significant. (Note that statistical significance does not necessarily imply practical significance).

16

Situations with Many Factors

When there are many factors to consider, full factorial, or even fractional factorial designs may not be feasible from a computational standpoint.

Other types of design (e.g., Plackett-Burman designs or “supersaturated designs”) may be appropriate (Mauro, 1986).

Another approach is to reduce the number of factors to consider via “factor screening” techniques, involving, for example, group screening in which a whole group is treated as a factor.

17

Response Surfaces and Metamodels

A response surface is a graph of a response variable as a function of the various factors.

A metamodel (literally, model of the simulation model), is an algebraic representation of the simulation model, with the factors as independent variables and a response as the dependent variable. It represents an approximation of the response surface.

The typical metamodel used in a simulation application is a regression model.

18

A metamodel through the use of response surface methodology can be used to find optimal values for a set of factors. It can also be used to answer “what if” questions. (Experimentation with a metamodel is typically much less expensive than using a simulation model directly).

An experimental design process assumes a particular metamodel, e.g., E C P Q B B P B Q B PQ[ ( , )] 0 1 2 12

Response Surfaces and Metamodels

19

Basic Concepts of Regression Analysis

Regression is used to determine the “best” functional relation among variables.

Suppose that the functional relationship is represented as:

E(Y) = f (X1, ..., Xp / B1, ..., BE)

where E(Y) is the expected value of the response variable Y; the X1, ..., Xp are factors; and the B1, ..., BE are function parameters; e.g.,

E(Y) = B1 + B2 X1 + B3 X2 + B4

X1 X2

20

The observed value for Y, for a given set of X ’s, is assumed to be a random variable, given by:

Y = f (X1, ..., Xp/B1, ..., BE) +

Where is a random variable with mean equal to 0 and variance . The values for B1,...,BE are obtained by minimizing the sum of squares of the deviations.

E

2

Basic Concepts of Regression Analysis

21

Response Surface Methodology

Source: (Fu, 1994) Response surface methodology

(RSM) involves a combination of metamodeling (i.e., regression) and sequential procedures (iterative optimization).

22

RSM involves two phases: Fit a linear regression model to some

initial data points in the search space (through replications of the simulation model). Estimate a steepest descent direction from the linear regressions model, and a step size to find a new (and better) solution in the search space. Repeat this process until the linear

regression model becomes inadequate (indicated by when the slope of the linear response surface is approximately 0; i.e.,

when the interaction effects become larger than than the main effects).

Fit a nonlinear quadratic regression equation to this new area of the search space. Then find the optimum of this equation.

Response Surface Methodology

23

Terminology in Experimental Design

Source:(Ostle, 1963) Replication - the repetition of the basic

experiment Treatment - a specific combination of

several factor levels Experimental Unit - the unit to which a

single treatment is applied to one replication of the basic experiment

Experimental Error - the failure of two identically treated experimental units to yield identical results

Confounding - the “mixing up” of two or more factors so that it’s impossible to separate the effects

24

Randomization - randomly assigning treatments to experimental units (assures independent distribution of errors)

Main Effect (of a factor) - a measure of the change in a response variable to changes in the level of the factor averaged over all levels of all the other factors

Interaction is an additional effect (on the response) due to the combined influence of two or more factors

Terminology in Experimental Design