Optimal Sampling in Design of Experiment for Simulation-based Stochastic Optimization

Mark W. Brantley, Loo H. Lee, Senior Member, IEEE, Chun-Hung Chen, Senior Member, IEEE, and Argon Chen, Member, IEEE

Abstract— Simulation can be a very powerful tool to help decision making in many applications such as semiconductor manufacturing or healthcare, but exploring multiple courses of actions can be time consuming. We propose an optimal computing budget allocation (OCBA) method to improve the efficiency of simulation optimization using parametric regression. The approach proposed here, called OCBA-DOE, incorporates information from across the domain into a regression equation in order to efficiently allocate the simulation replications to improve the decision process. Asymptotic convergence rates of the OCBA-DOE method indicate that it offers a significant improvement when compared to a naïve allocation scheme and the traditional OCBA method. Numerical experiments reinforce these results.

I. INTRODUCTION IMULATION is a popular tool for designing large,

complex, stochastic engineering systems, since closed-form analytical solutions generally do not exist for such problems. Simulation allows one to accurately specify a system through the use of logically complex, and often non-algebraic, variables and constraints. Detailed dynamics of complex, stochastic systems can therefore be modeled. This capability complements the inherent limitation of traditional optimization. Semiconductor manufacturing system is a good example which is characterized by complex and re-entrant production processes over many heterogeneous machine groups with stringent performance requirements. Semiconductor manufacturing faces stringent challenges of volatile product demands, very short time to market, complex but fast evolving process technology, sky-rocketing capital investment and highly cost-sensitive competition [1],[2]. The health care arena provides another example. Simulations provide the ability to analyze the complex decisions associated with patient flow and ambulance

planning [3]. As a final example, simulations can help determine how to procure military equipment. These simulations can provide a means to investigate the trade-offs of performance, cost, and reliability given environmental and operating scenarios [4]. For such complex systems as these examples, simulation can help design good systems for efficient operations.

Manuscript received February 29, 2008. This work has been supported in

part by NSF under Grants IIS-0325074, by NASA Ames Research Center under Grants NAG-2-1643 and NNA05CV26G, by FAA under Grant 00-G-016, and by AFOSR under Grant FA95500410210.

M. W. Brantley is with the Department of Systems Engineering and Operations Research, George Mason University, Fairfax, VA 22030 USA (phone: 703-426-5172; e-mail: mbrantl1@gmu.edu).

L. H. Lee., is with the Department of Industrial and Systems Engineering, National University of Singapore (e-mail: iseleelh@nus.edu.sg).

C-H. Chen is with the Department of Systems Engineering and Operations Research, George Mason University, Fairfax, VA 22030 USA (e-mail: cchen9@gmu.edu).

A. Chen is with the Institute of Industrial Engineering, National Taiwan University, (e-mail: achen@ntu.edu.tw).

However, the added flexibility of simulation often creates models that are computationally intractable. There are two major challenges for simulation optimization: i) uncertainty: a large number of simulation replications must be performed for each alternative design to have a good estimate of the system performance; and ii) combinations: the number of candidate designs grows in a combinatorial way. The simulation time required for such an enormous number of designs is usually costly formidable.

Simulation optimization is a method to find a design consisting of a combination of input decision variable values of a simulation system that optimizes a particular output performance measure of the system. The level of complexity associated with this task is dependent upon the nature of the input decision variables, the nature of the underlying function associated with the output performance measure of the system, and the resources available to solve the problem. In this paper, we propose to investigate stochastic problems on a discrete domain with a finite simulation budget consisting of runs conducted sequentially on a single computer.

When presented with a finite number of designs in the domain, the problem we consider is that of selecting the best design from among the finite number of choices. We seek to determine the best allocation of simulation runs from among the competing designs. This type of problem is typically solved using Ranking and Selection (R&S) procedures that stem from the one stage method of [5]. This paper focuses upon improving a highly efficient R&S technique developed in [6]-[8] called the Optimal Computing Budget Allocation (OCBA) method. Numerical comparisons have shown that OCBA can achieve a speedup factor of approximately 4 for a small number of competing designs and can be as much as 20 times faster than traditional approaches for a much larger number of designs.

This paper expands the OCBA method by incorporating information from neighboring design locations or information from estimates of the underlying function generating the data. The authors in [9] note that they appear

S

4th IEEE Conference on Automation Science and EngineeringKey Bridge Marriott, Washington DC, USAAugust 23-26, 2008

978-1-4244-2023-0/08/$25.00 ©2008 IEEE. 388

to be the first to introduce the concept of applying R&S techniques to the problem of selecting the largest regression value. We expand upon this concept by not only using the regression value to make decisions but also incorporating developments from the statistical field known as design of experiments (DOE).

A brief history of the developments in DOE can be found in [10]. The authors in [11]-[13] develop optimal designs for estimating the extreme point in quadratic regression. The concept of applying DOE to simulation optimization and developing a criterion that provides the best fitting polynomial for queuing models is introduced in [14]. This method is extended for a generalized regression metamodel and an optimal allocation for the goodness of fit criterion is derived in [15].

This paper integrates the concepts from both DOE and OCBA by using information from estimates of the underlying function in order to maximize the probability that we select the best design. The main contribution of this paper is providing a method to optimally allocate simulation replications among k design locations utilizing the structure of the underlying function. The rest of the paper is organized as follows. In Section II, we introduce the simulation optimization problem setting. Section III provides the development of the OCBA-DOE method while Section IV provides convergence results. Numerical experiments comparing the results using the new OCBA-DOE method and other methods are provided in Section V. Finally, Section VI provides the conclusions and suggestions for future work using the concepts introduced here.

II. PROBLEM SETTING AND BAYESIAN FRAMEWORK This paper explores a problem with the principle goal

of selecting the best of alternative design locations. Without loss of generality, we consider the minimization problem shown below where the “best” design location is the one with smallest expected performance measure

k

],,[);(min 21 kiix

xxxxxyi

L∈ . (1)

We consider the case of simulation output that is

produced by an unknown function of one variable at k design locations where . In this paper, we consider that the expectation of the unknown underlying function is quadratic or approximately quadratic in nature on the prescribed domain, i.e.,

kixx ii ,,1,1 K=< +

2

210)( iii xxxy βββ ++= . (2) For ease of notation, we define ][ 210 ββββ = . In (2),

the parameters β are unknown and we consider a common case where must be estimated via simulation with

noise. The simulation output is independent from replication to replication such that

)ix(y

)( ixf

,, k

,1;)()( ixyxf ii K=+= ε where . (3) ),0(~ 2σε N The parameters β are unknown so are also

unknown. However, we can find an estimated expected performance measure at , that we define as , by using a least squares estimate of the form shown in (4) below where , , and are the least squares parameter estimates for the corresponding parameters associated with the constant, linear, and quadratic terms in (3).

)( ixy

yix )( ix

0β 1β 2β

ˆ[ˆ0ββ =

2

210ˆˆˆ)(ˆ iii xxxy βββ ++= (4)

In a similar manner, we define . ]ˆˆ

21 ββIn order to obtain the least squares parameter estimates,

we take n samples on any choice of (on at least three design locations to avoid a singular solution). Given the n samples, we define

ix

F as the dimensional vector containing the replication output measures and

n)( ixf X

as the 3×n]2

i

f

matrix composed of rows consisting of with each row corresponding to its respective

entry of in 1[ i xx

)( ix F . Using the matrix notation, we determine the least squares estimate for the parameters β which minimize the sum of the squares of the error terms

. As shown in most regression texts such as [16], we obtain the least squares estimate for the parameters as shown below:

)β(F)β T( XF − X−

FXXX TT 1)(ˆ −=β .

Our problem is to select the design location associated

with the smallest mean performance measure from among the k design locations within the constraint of a computing budget with only T simulation replications. Given the least squares estimates for the parameters, we can use (4) to estimate the expected performance measure at each design location. We designate the design location with the smallest estimated mean performance measure as so that bx

)i(ˆmin)(ˆib xyxy = . Given the uncertainty of the estimate of

the underlying function, is a random variable and we define Correct Selection as the event where is indeed the best location. We define as the number of simulation replications conducted at design location . Since the simulation is expensive and the computing budget is

bx

iNbx

xi

for restricted, we seek to develop an allocation rule each iN

389

in order to provide as much information as possible for identification of the best design location. Our goal then is to determine the optimal allocations to the design locations that maximize the probability that we correctly select the best design (PCS). This Optimal Computing Budget Allocation (OCBA) problem is reflected in (5) below.

the

(5)

onstraint denotes the total

co putational cost and implicitly assumes that the sim ion times for one sam

he underlying function , we m

TNNNts

ixyxyPPCS

k

ibNN k

=+++

∀≤=

L

K

21

,,

..

})()({max1

The c TNNN k =+++ L21

mulation execut ple are constant

across the domain. The nature of this problem makes it extremely difficult to

solve. To understand t )( ixy

nding ust conduct simulation runs to obtain )( ixf , which is a

measure of the system performance. Compou this property is the fact that )( ixf is a funct the random variable

ion ofε . To even assess the performance at one point on

the domain, the uncert in the system performance measure requires multiple runs to obtain good approximations of the performance measure. Since the optimal allocation is dependent upon the uncertainty of the parameters, we can only compute the PCS after exhausting the total simulation budget

ainty

T . Incorporating the information from the underlying function adds an additional level of complexity to the deri ation of the optimal allocations; however, it is this concept that we aim to exploit in order to provide a significant improvement in the ability to maximize PCS.

In order to solve the problem in (5), we aim to find the posterior distributi

v

ons of β

as the simulation replications are conducted and use these distributions to update the posterior distribution of the performance measures for each design location. We can then perform the comparisons with the performance measure at design location bx as expressed

in (5). We will use β~ and )(~

ixy to denote the random variables whose probability distributions a the posterior distribution of

re β and ( ixy ditional on ) con F given

samples respectively. Therefore, the probability of correct selection from (5) can be expressed as

})(~)(~{ ixyxyPPCS bi ∀≥= . (6)

Given a set of initial n simucontained in vector

lation runs with the output

F and using a non-informative prior distribution, [17] shows that the asymptotic posterior distribution of β is th given by

(,)[(~

en

])~ 1−XX 21− XFXXN TTT σβ

)(~ixy the β

~ elements, is a linear combination of

. (7)

Since th m at asymptotically is eans th )(~

ixy has a Gaussian di

8)

an estimate nd late

stribution of the form

([~)(~ 1Tii XXXNxy −− ])(,) 21

iTT

iT XXXXFX σ

where ]1[ 2ii

Ti xxX = . (

We c from our least squares results acan calcu

2σ )(~

ixy using (8). We can then use Monte Carlo simulation with (6) in order to estimate the PCS [18]. H th

ious section demonstrated how we can utilize er to

s re across the en r

owever, estimating e PCS via Monte Carlo simulation can be time consuming. The next section reduces the number of comparisons required and presents a way to approximate the PCS without running Monte Carlo Simulations.

III. APPROXIMATE ASYMPTOTICALLY OPTIMAL ALLOCATION DERIVATION

The prevthe quadratic structure of the underlying function in ordprovide estimates of the performance mea u

ti e domain. This section demonstrates that we can also use this quadratic structure to reduce the number of comparisons required in (6). We then establish that it is sufficient to allocate simulation runs to only three design locations to estimate the parameters for the quadratic function. (For notation sake, we will refer to the three design locations receiving simulation runs as support points { }321 ,, sss .) We will then derive an optimal allocation of simulation runs to the three support points.

1: Given bx from the least squares results, the

assumption that our underlying function is quadratth we can reduce the required number of com

Theoremic means

parisons in ou

atr PCS equations from he 1−k comparisons expressed in

(6) to two comparisons. As such, we can express our OCBA equation as shown in (5) below or in its equivalent form using the parameters β

~ .

NNts

ixyxyPPCS biNN k

+++

ΖΑ=≥=K

..

},;)(

t

TNk =L21

,,~)(~{max

1 with (9)

Case 1 (Interior Case):

1;1;,1 +=Ζ−=Α≠ bbkb

ase 2 (Left Boundary Case): C kb =Ζ=Α= ;2;1 Case 3 (Right Boundary Case): 1;1; −=Ζ=Α= kkb

roof: See [19] . P

390

Given the comparisons from the PCS equation in (9),

w onferroni inequality to obtain the lower bound for our objective function expressed in (10) below. N e

e will apply the B

ot that the comparisons from the PCS equation are not independent since they are both obtained using β

~ . However, the Bonferroni inequality does not require independence of the comparisons and yields [20]

})(~)(~{)}(~)(~{1 bb xyxyPxyxyPPCS ≤−≤−≥ ΖΑ .(10)

Theorem 2: Given that we assume the expectation of our

underlying function is quadratic, we require only support points to obtain all of the information in the

three XX T

m

nt proves two of these pport points will be at the extreme design locations.

pport po s. Define

atrix. Two of these support points will be at the extreme design locations ( 1x and kx ).

Proof: Reference [21] establishes that we require only three support poi s. Reference [22]su

In the remainder of this section, we will develop a method

to optimally allocate simulation runs given a set of suint

)(~)(~)(~)(~)(~1

222 bibibii xxxxxyxyxd −+−=−≡ ββ .

This shows that is a linear combination the

elements so the

)(~ixd

)

β~

(~ix termd s are also normally distribut

U

ed.

sing the results of Section 2, ,)(ˆ[~)( ]~iii xdNxd ς

where )(ˆˆ)(ˆbi xyyxd = and

= 2 (bi Xxς

)i −(x

( )⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

−−−− −

22

1220

),,0

bi

biT

ibi

xxxxXxxxσ . (11)

Since our aim is to efficiently allocate the com

budget to the three support points, we will rewrite this va ance term so that it is expressed in terms of the pe

puting

rircentage of the simulation budget allocated to each

support point. For notation sake, we will use iα to denote the percentage of the simulation budget allocated to is .

We are only taking simulation replications at the support points so we can rewrite the XX T matrix in rms of the support points. Given the symmetric nature of this ma

te trix

an form

d a basic theorem where 1111)( −−−− = ABCABC [23], we may write (11) in the simplified :

⎥⎦

⎤⎢⎣

⎡++=

3

3,

2

2,

1

1,2

ααασς iii

iDDD

T, (12)

2

3121

32321, ))((

))(())((

⎭⎬⎫

⎩⎨⎧

−−−−−−−

=ssss

xsxsxsxsD bbiii

2

3212

31312, ))((

))(())((

⎭⎬⎫

⎩⎨⎧

−−−−−−−

=ssss

xsxsxsxsD bbiii

2

2313

21213, ))((

))(())((

⎭⎬⎫

⎩⎨⎧

−−−−−−−

=ssss

xsxsxsxsD bbiii .

Lagrangian relaxation of the total budget constraint

exApresses our objective function as shown in (13) below.

∑

∫ ∫

=

∞

−

∞

−

−−

−−

−−=

Α

Α

Ζ

Ζ

k

ii

xd xd

tt

TN

dtedteF

1

)(ˆ )(ˆ

22

)(

21

211

22

λ

ππς ς (13)

Investigating j

Fα∂

∂ to determine the allocations, we can

establish that

0)(2

)(ˆ))(ˆ

(

)(2)(ˆ

))(ˆ(

2,

2

2/3

2,

2

2/3

=−−

+

−=

∂∂

Ζ

Ζ

Ζ

Ζ

Ζ

Α

Α

Α

Α

Α

λα

σςς

φ

ασ

ςςφ

α

j

j

j

j

j

DT

xdxd

DT

xdxdF

Solving for iα yields:

3

3

2

2

1

1 )()()(α

αα

αα

α QQQ== , (14)

2/3,

2/3,

)(

)(ˆ))(ˆ

()(

)(ˆ))(ˆ

(Ζ

ΖΖ

Ζ

Ζ

Α

ΑΑ

Α

Α −+

−=

ςςφ

ςςφ jj

jDxdxdDxdxdQ .

4) is a nonlinear equation with respect toNote that (1 α .

heHowever, we can derive an asymptotic solution for t quadratic case by only comparing the design that has the smallest ratio of each iid ς/ˆ . For notation sake, we call this design Μx . The asy tic assumption results in the determin n that jj DQ ,Μ

mptoatio = which leads to the solution

expressed in Theore .

m 3 below

391

Theorem 3: The approximate PCS expressed in (10) can be asymptotically maximized with allocations that satisfy:

j

j

i

i DD

αα,, ΜΜ

= . (15)

IV. CONVERGENCE RATES Thi he convergence

ra s section provides a comparison of t

te of the OCBA-DOE method against two other methods. For brevity, we provide only the results. A complete derivation can be found in [19]. The simplest case is the naive method for allocating the number of runs, iN , to each design location. Given a simulation budget T an k design locations, we allocate the runs equally such that kTN i /

d =

for each i and use mean statistics to comperformance at each design location. We will refer to this method as the Equal Allocation (EA) method.

Instead of equally allocating, we can st

pare the

ill use mean sta

n use the OCBA-DOE method derived in Se t

tistics for comparisons but optimally allocate the simulation runs to the designs using OCBA. We are not able to obtain an explicit form for the allocations to estimate the convergence rate of OCBA for the quadratic case. However, we can establish an upper bound for any method that uses mean statistics. We will obtain the highest PCS using mean statistics if we know a priori the best and second best design locations and allocate half of the runs to each of these design locations. Although this scenario is unlikely for a real application, the results show that in comparison to EA, the best we can obtain is an improvement of ( )2/kO in the convergence rate.

Finally, we cac ion III to use the estimated response from the regression

equation instead of mean statistics and optimally allocate to { } { }kk xxxsss ,,,, 2/)1−= . This method provides (1321

improvements of ( )2/3kO when com

V. NUMERICAL TESTING AND RESULTS

A. Testing Framework scribe how we tested our new

ters for each of these methods, w

ses the following function to represent

order to test the methods against a diverse set of

pr

pared to the EA case.

In this section, we deOCBA-DOE method against the two other allocation procedures that were described in Section IV: EA and OCBA. Based upon the findings in [6], for OCBA we used an initial allocation of 5 runs for each design and then allocated 8 runs during additional iterations. We allocated 2 initial runs to each support point for OCBA-DOE and 8 runs during additional iterations.

Given the differing paramee constructed our experiments to provide a fair

comparison. Both of our experiments will incorporate a randomly selected optimal solution and our comparison metric is the probability of correctly selecting the location of the best design. We conducted each experiment using a total

computing budget of 1,000 runs and the results will show that this amount is sufficient to compare the performance of the methods. To mitigate the fact that the methods have varying fixed costs associated with them and in order to compare the performance of the methods using various simulation budgets, we calculate the PCS for each method during each iteration until the total simulation budget is exhausted. We repeat this whole procedure 10,000 times and then calculate the PCS obtained for each method during these 10,000 independent applications.

B. Experiment 1 This experiment u

the simulation output:

)1,0()( 012

2 Naxaxaxf iii +++= .

Inoblems, the parameters are randomly selected where

)10,10(~ −Ua j . The domain consists of 11 design

]1,8.0,1[ Llocations where −−∈x . The optimal esisolution is the d gn location closest to

21 /5.0 aa−=θ when 02 >a and the design location farthest from θ when 0 . Fig. 1 contains the simulation results for the EA, OCBA, and OCBA-DOE methods. These results are consistent with the results from the convergence rate analysis from Section IV. The OCBA-DOE method clearly performs the best. As a point of comparison, we obtain a PCS of about 90% after 14 runs with OCBA-DOE, 380 runs with OCBA, and 1,000 runs with EA.

2 <a

70%

75%

80%

85%

90%

95%

100%

0 100 200 300 400 500 600 700 800 900 1000

Total Number of Simulation Runs

Prob

abili

ty o

f Cor

rect

Sel

ectio

n, P

{CS}

OCBA-DOEOCBAEA

Fig. 1. Results of Experiment 1.

s identical to Experiment 1 except the do

C. Experiment 2 This experiment imain consists of 101 designs where ]1,98.0,1[ L−−∈x .

Fig. 2 contains the simulation results f , and OCBA-DOE methods with the OCBA-DOE method clearly performing the best again. This method has a clear advantage of not having to devote a large number of

or the EA, OCBA

392

simulation runs for the initial iteration. As a point of comparison, the results for the OCBA-DOE method in Fig. 2 (with 101 designs) are very similar to the results for the EA method in Fig. 1 (with 11 designs).

[5] R. E. Bechhofer, “A single-sample multiple decision procedure for ranking means of normal populations with known variances”, Annals of Mathematical Statistics, vol. 25, no. 1, pp. 16-39, Mar. 1954.

[6] C. H. Chen, J. Lin, E. Yücesan, and S. E. Chick, “Simulation budget allocation for further enhancing the efficiency of ordinal optimization”, Journal of Discrete Event Dynamic Systems: Theory and Applications, vol. 10 , no. 3, pp 251-270, July 2000.

20%

30%

40%

50%

60%

70%

80%

90%

[7] C. H. Chen., D. He, M. Fu, and L. H. Lee, "Efficient Simulation Budget Allocation for Selecting an Optimal Subset," to appear in Informs Journal on Computing, 2008.

0 100 200 300 400 500 600 700 800

Total Number of Simulation Runs

Prob

abili

ty o

f Cor

rect

Sel

ectio

n, P

{CS}

[8] C. H. Chen and E. Yücesan, "An Alternative Simulation Budget Allocation Scheme for Efficient Simulation," International Journal of Simulation and Process Modeling, vol. 1, nos. 1/2, pp. 49-57, 2005.

[9] J. D. Gibbons, I. Olkin, and M. Sobel, Selecting and Ordering Populations: A New Statistical Methodology, NY, NY: John Wiley and Sons, 1977, pp. 239-241.

[10] V. B. Melas, Functional Approach to Optimal Experimental Design. Lecture Notes in Statistics 184, NY, NY: Springer, 2005, pp. 1-3.

900 1000

OCBAOCBAEA

Fig. 2. Results of Experiment 2.

VI. CONCLUSIONS We have present hen the underl

fu

mprove upon the OCD

REFERENCES [1] B. W. Hsieh, C. H. . C. Chang, "Scheduling

-DOE

[12] R. C. H. Cheng, V. B. Melas, and A. N. Pepelyshev, “Optimal designs

for the evaluation of an extreme point” in Optimum Design 2000, A. Atkinson, B. Bogacka, and A. Zhigljavsky, Eds. Dordrecht, The Netherlands: Kluwer Academic Publishers, 2001, pp. 15-24.

[11] V. V. Fedorov and W. C. Muller, “A reparametrization view of optimal design for the extremal point in polynomial regression”, Metrika, vol. 46, pp. 147-157, 1997.

ed a method that, w ying [14] R. C. H. Cheng and J. P. C. Kleijnen, “Improved design of queueing simulation experiments with highly heteroscedastic responses”, Operations Research, vol. 47, no. 5, pp. 762-777, Sep. -Oct. 1999.

[13] V. B. Melas, A. Pepelyshev, and R. C. H. Cheng, “Designs for estimating an extremal point of quadratic regression models in a hyperball”, Metrika, vol. 58, pp. 193-208, 2003.

nction is quadratic, can further enhance the efficiency of the simulation run allocation. Convergence rate analysis and numerical experiments indicate that the OCBA-DOE method offers a significant improvement over the EA and OCBA methods. Future efforts will compare the OCBA-DOE method to other DOE methods.

There are certainly ways to i

[15] J. D. Lamb and R. C. H. Cheng, “Optimal allocation of runs in a simulation metamodel with several independent variables”, Operations Research Letters, vol. 30, pp. 189-194, 2002.

BA- [

[16] N. R. Draper and H. Smith, Applied Regression Analysis: Third Edition, NY, NY: Wiley Series in Probability and Statistics, 1998, pp. 135-136.

17] M. H. DeGroot, Optimal Statistical Decisions, NY, NY: McGraw Hill Inc. 1970, pp. 250-252. OE method. We believe that the method can be extended

to higher order polynomials and potentially improved by using different regression techniques such as incorporating orthogonal polynomials. Another is to address problems where quadratic is not a good assumption globally. We propose to partition the search space into many smaller domains in which the underlying function is approximately quadratic. We will develop a global search technique by incorporating the concepts from OCBA-DOE with methods to partition the domain and efficiently allocate runs between partitions.

[18] A. M. Law and W. D. Kelton, Simulation Modeling and Analysis, Third Edition, Boston, MA: McGraw-Hill, Inc, 2000, pp. 90-91.

[19] M. W. Brantley, L. H. Lee, C. H. Chen and R. K. Jana, “Development of an optimal computing budget allocation method for underlying quadratic functions”, working paper, Department of Systems Engineering and Operations Research, George Mason University, 2008..

[20] J. Banks, J. S. Carson, II, B. L. Nelson, and D. M. Nicol, Discrete-Event Simulation, Third Edition, Upper Saddle River, NJ: Prentice Hall International Series In Industrial and Systems Engineering, 2001, pp. 469.

[21] D. L. Garza, “A spacing of information in polynomial regression”, The Annals of Mathematical Statistics, vol. 25, no. 1, pp. 123-130, Mar. 1954.

[22] J. C. Kiefer, J.C. “Optimum experimental designs” Journal of the Royal Statistical Society, Series B, vol. 21, no. 2, pp. 272-319, 1959.

[23] J. Neter, M. H. Kutner, C. J. Nachtsheim, W. Wasserman, Applied Linear Regression Models, Third Edition, Chicago, IL: Irwin, 1996, p. 194.

Chen, and SSemiconductor Wafer Fabrication by Using Ordinal Optimization-Based Simulation," IEEE Trans. Robotics and Automation, vol. 17, no. 5, pp. 599-608, Oct. 2001.

[2] B. W. Hsieh, B. W., C. H. Chen, S. C. Chang, "Efficient Simulation-

based Composition of Dispatching Policies by Integrating Ordinal Optimization with Design of Experiment," IEEE Trans. Automation Science and Engineering, vol. 4, no. 4, pp. 553-568, Oct. 2007.

[3] J. Zeto, M. W. Brantley, and G. Collins, et al., “Air Ambulance Analysis-Iraq”, presented at the 2005 Military Operations Research Symposium, West Point, NY.

[4] M. W. Brantley, M., W. J. McFadden, and M. J. Davis, "Expanding the Trade Space: An Analysis of Requirements Tradeoffs Affecting System Design", Acquisition Review Quarterly, vol. 9, no. 1, pp. 1-16, Winter 2002.

393