Optimization with expensive models - BCAM · • Tuning needs strong optimization approach due to...

Dr. Ivan Voutchkov, [email protected]

Optimization with expensive models


SEARCH

RSM Optimization


DOE TUNE

TUNELikelihood

function

RSM Optimization


TUNELikelihoodfunction

SEARCHTUNEDOE

TUNEPredictor

Response surface model (RSM)

RSM Optimization


Response surface models

Process, structural model

(expensive)

Input sampling (DOE) Response surface

Response surface model

Continuous input


• Radial basis functions

• Kriging

Response surface models


SEARCHTUNE

TUNEPredictor


Radial basis functions

Euclidian distance between data point and centre point

weights

prediction

Gram matrix

TUNEFindweights

TUNE

SEARCH

TUNEPredictor



rrf )(

3)( rrf

)ln()( 2 rrrf

2

2

2)(

r

erf

22)( rrf

Linear

Cubic splines

Thin plate

Gaussian

Multiquadratic

22

1)(

rrf Inverse - multiquadratic

k

j

jp

jr

erf 1)(

Kriging



Prediction depends on the distance between points


Kriging

Find hyper parameters to maximize CLF:

hyper parameters

tuning

Observations

TUNEFind

hyperparamsto max(CLF)

TUNE

SEARCH

TUNEPredictor

CLF =

Concentrated likelihood function


• Tuning needs strong optimization approach due to high dimensionality and multimodality of CLF. Number of optimization variables is 2k+1, where k is the number of design variables.

• Numerical instabilities during matrix inversion

• Clustered data

• Computationally expensive with high number of variables (>20-25) and high number of data points (> 200)

Kriging – common difficulties


• Tuning search must handle multimodal and multivariable problems.

• Apply techniques for global optimization.

• Use numerically stable matrix inversion algorithms.

• Avoid data clustering.

• Use iterative update techniques.

Requirements for the Kriging tool

Requirements – not recommendations!


RSM optimization

SEARCH

SLOW


RSM optimization

SEARCH

TUNEPredictor

FAST

Can we trust it?


predicted

minimum

true

minimum

RBF / Kriging


predicted

minimum

true

minimum

Validation may drive the optimization to a local optimum

Where to update ?

Need points here


Root of the Mean Squared Error - max(RMSE)


Probability of improvement

Predicted distribution given mean & variance

Current optimum

Predicted mean

Predicted variance

Probability of improvement


Expected improvement

x


OPTIMAT v2


SEARCHTUNEDOE

TUNEPredictor

Calculate updates

Find updates

TUNEPredictor

= getUpdates(DOE, settings)


Single and multiobjective optimization using Response Surface models

Significantly improved Response surface models with hybrid Particle swarm optimization tuning

Kriging

Non-stationary kriging

Gradient enhanced kriging

Co-kriging for multifidelity problems

Radial basis functions (automatic base selection)

Portable predictor as an Excel spreadsheet

OPTIMATv2 – main features


Flexible update strategies

predict(n) – MIN(OBJ1, OBJ2, …)

rmse(n) – based on MAX (RMSE1, RMSE2, ...)

ei(n) – based on MAX (EI1, EI2, …)

spacefill(n) – points furthest from existing.

constraint(n) – probability of constraint feasibility – experimental

Any combination – predict(5), rmse(5,2), ei(3) …

Extensive RSM search – modified NSGA2, local search around best points, dynamic construction of local RSMs around best points



Three releases

MATLAB toolbox (Windows and Linux)

Isight components

Standalone (EXE) (Windows and Linux)

RSMTune and RSMEval are available as separate components, but use the same data structure

Extended settings file, restarts

Interactive visualization of the RSM.



OPTIMATv2 as an Isight plugin


… with ease only with OPTIMATv2

Multiobjective, multifidelity


26


Multiobjective optimization


28


29

Weight (min)Strength (max)Mass (min)Fuel consumption (min)Speed (max)Drag (min)Lift (max)Stresses (min)Reliability (max)Carbon footprint (min)Noise (min)Pay load (max)Runway length (min)Safety and backup systems (max)Maintenance costs (min)……………………………….…………………………….………………………….

Real world = many objectives


Q1

Q2

Objectives’ space

Pareto front – a set of non-dominated solutions

x

Q1,

Q2

F1

F2

Variables’ space

Q1 min Q2 min


Objective function 1

Obje

ctive f

unction 2

31

Dominancy


Objective function 1

Obje

ctive f

unction 2

32

Dominancy


Q1

Q2

rank = 1

rank = 2

rank = 3

rank = 4

rank = 5

In NSGA2 - Multiobjective problem is converted to single objective minimize rank, i.e. encourage non-dominant points


Objective 1

Ob

ject

ive

2

Points arranged in clusters

Quality of the Pareto Front


Objective 1

Objective 2

Lack of diversity


Objective 1

Objective 2

low number of points


Objective 1

Objective 2

Optimal pareto front not reached


Objective 1

Objective 2

Diversity, uniform distribution, good number of points


Direct NSGA2


Direct NSGA2

Soton NSGA2 Isight’s native NSGA2


Robust optimization


2F

1F

x

more robust

less robust

xx

F(x)

Equal variation of variables leads to minimum variation of performance

Robustness

Minimize mean of reaction forces

Minimize variance of reaction forces


Mean and Variance estimation

• Monte-Carlo simulations .. thousands of runs (expensive!)

• Taylor series expansion, first and second order .. relatively simple function shapes – (not always accurate!)

• Sparse quadrature .. works best for noise-free functions


100 Monte Carlo simulations

Mean Variance



Mean Variance


Variance estimation

• Taylor series expansion, first and second order.

)0(

1 1

)0(2

1

)0()0(

2

1ˆjj

n

i

n

j

ii

ji

n

i

ii

i

xxxxxx

Fxx

x

FxFxF

n

i

n

i

x

n

j

x

ji

x

i

F

x

n

i i

F

jii

i

xx

F

x

F

x

FxF

1 1

2

1

2

2

22

22

2

2

12

2)0(

2

1ˆ

2

1ˆˆ

Second order terms

… approximation


• Integrals of statistical moments

• The calculation of the integrals needs a high number of quadrature points,

high number of function evaluations

Uncertainty quantificationusing numerical quadrature


Sparse grids vs. full grids

Fewer quadrature points fewer function evaluations

Full grid 225 points

Sparse grid49 points

An example: Dimension = 2Level of accuracy = 3Quadrature rule = Gauss Patterson

(normal distribution)


Sparse quadrature

N Vars -> /

Level 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31

2 5 13 25 41 61 85 113 145 181 221 265 313 365 421 481

3 9 29 69 137 241 389 589 849 1177 1581 2069 2649 3329 4117 5021

4 17 65 177 401 801 1457 2465 3937 6001 8801 12497 17265 23297 30801 40001

5 33 145 441 1105 2433 4865 9017 15713 26017 41265 63097 93489 134785 189729 261497

6 65 321 1073 2929 6993 15121 30241 56737 100897 171425

7 129 705 2561 7537 19313 44689 95441

8 257 1537 6017 18945 51713

9 513 3329 13953 46721

10 1025 7169 32001

11 2049 15361 72705

12 4097 32769

N Vars -> /

Level 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31

2 7 17 31 49 71 97 127 161 199 241 287 337 391 449 511

3 15 49 111 209 351 545 799 1121 1519 2001 2575 3249 4031 4929 5951

4 31 129 351 769 1471 2561 4159 6401 9439 13441 18591 25089 33151 43009 54911

5 63 321 1023 2561 5503 10625 18943 31745 50623 77505 114687 164865 231167 317185

6 127 769 2815 7937 18943 40193 78079 141569

7 255 1793 7423 23297 61183 141569

SPQ GRID rule 1 - Clenshaw-Curtis, number of required design points

SPQ GRID rule 2 - Gauss-Patterson, number of required design points


Sparse quadrature

0

10

20

30

40

50

60

70

80

90

0

2

4

6

8

10

12

14

SPQL2(17) MC(1000) MC(17)

% C

os

t-S

tDe

v

% F

tar

ge

t-M

ea

n

Percentage difference to MC (50000)

Ftarget-Mean Cost-StDev


Reduced Sparse quadrature

1.Start with a lower level SPQ design, e.g. level 22.Build a Response surface model (RSM) and update using

points from the next level SPQ plan – only such that maximize the error – Root of the Mean Squared Error

3.Use the RSM to perform higher level SPQ designs



0

1

2

3

4

5

6

7

8

9

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

SPQL2(17) SPQL3(49) SPQL4(129) SPQL5(321)

% C

os

t-S

tDe

v

% F

tar

ge

t-M

ea

n





1.86

1.88

1.9

1.92

1.94

1.96

1.98

2

2.02

2.04

0.18

0.185

0.19

0.195

0.2

0.205

0.21

SPQL3(49) ReducedSPQL4 (70) SPQL4(129) ReducedSPQL5 (70) SPQL5(321)

% C

os

t-S

tDe

v

% F

tar

ge

t-M

ea

n




Reduced Monte-Carlo?

1.Start with a small DOE2.Build a Response surface model (RSM) and update using

points RMSE technique3.Use the RSM to perform higher full Monte-Carlo

Optimization with expensive models - BCAM · • Tuning needs strong optimization approach due to...

Documents

Transcript of Optimization with expensive models - BCAM · • Tuning needs strong optimization approach due to...