Design Optimization UtilizingDesign Optimization UtilizingGradient/Hessian Enhanced Surrogate ModelGradient/Hessian Enhanced Surrogate Model
Dept. of Mechanical Engineering,University of Wyoming, USA
Wataru YAMAZAKI,Markus P. RUMPFKEIL,Dimitri J. MAVRIPLIS
28th, June, 2010,28th AIAA Applied Aerodynamics Conference
Yamazaki, W., Dept. of Aero. Eng., Tohoku Univ.Wataru YAMAZAKI, Univ. of Wyoming-2-
Outline
*Background- Efficient CFD Gradient/Hessian calculations- Surrogate Model Enhanced by Gradient/Hessian- Uncertainty Analysis
*Objectives
*Surrogate Model Approaches- Kriging- Direct and Indirect Gradient-enhanced Kriging- Gradient/Hessian-enhanced Kriging Approaches
*Results & Discussion- Analytical Function Fitting- Aerodynamic Data Modeling- 2D Airfoil Drag Minimization- Uncertainty Analysis at Optimal Airfoil
*Conclusions
Yamazaki, W., Dept. of Aero. Eng., Tohoku Univ.Wataru YAMAZAKI, Univ. of Wyoming-3-
Background~ Efficient CFD Hessian Calculation
M.P. Rumpfkeil and D.J. Mavriplis, AIAA-2010-1268“Efficient Hessian Calculations using Automatic Differentiation and the Adjoint Method”
An efficient CFD Hessian calculation methodby Adjoint method and Automatic Differentiation (AD)
For steady flowi. Solutions for grid deformation / flow residual equations
ii. Adjoint solutions for flow / grid deformation equations
iii. Ndv linear solutions each for dx/dDj and dw/dDj
iv. Ndv(Ndv+1)/2 cheap evaluations for each Hessian component
0, DxDs 0,, DwDxDR
0
TT
F
ww
R0
T
TT
F
x
R
xx
s
sR jkT
jkT
jkkj
FdDdD
Fd
2
Yamazaki, W., Dept. of Aero. Eng., Tohoku Univ.Wataru YAMAZAKI, Univ. of Wyoming-4-
Background~ Efficient CFD Hessian Calculation
Grid Deformation
Flow Residual
Flow AdjointMesh Adjoint
dx/dD1
dw/dD1
dx/dD2
dw/dD2
dx/dDNdv
dw/dDNdv
......
Gradient and Hessian
An efficient CFD Hessian calculation methodby Adjoint method and Automatic Differentiation (AD)
M.P. Rumpfkeil and D.J. Mavriplis, AIAA-2010-1268“Efficient Hessian Calculations using Automatic Differentiation and the Adjoint Method”
Yamazaki, W., Dept. of Aero. Eng., Tohoku Univ.Wataru YAMAZAKI, Univ. of Wyoming-5-
Background~ Approximate CFD Hessian
For steady flow, a special form of objective function
K
kkkk FFwF
1
2target
2
1
2target2target
2
1.. DDDLLL CCwCCwFge
K
k j
k
T
i
kk
K
k ji
kkkk
K
k j
k
T
i
kk
ji
dD
dF
dD
dFw
dDdD
FdFFw
dD
dF
dD
dFw
dDdD
Fd
1
1
2target
1
2
Last approximation is accurate only nearly optimum Approximate Hessian only requires the first-order derivatives
M.P. Rumpfkeil and D.J. Mavriplis, AIAA-2010-1268“Efficient Hessian Calculations using Automatic Differentiation and the Adjoint Method”
0
Yamazaki, W., Dept. of Aero. Eng., Tohoku Univ.Wataru YAMAZAKI, Univ. of Wyoming-6-
Background~ Uncertainty Analysis
Uncertainty due to manufacturing tolerances in-service wear-and-tear etc
Analysis of mean/variance/PDF ofobjective function w.r.t. fluctuation of design variables
Full Monte-Carlo Simulation Thousands/Millions exact function calls Accurate and easy, but computationally expensive
Moment Method Taylor series expansion by grad/Hessian at the center No information about PDF
Inexpensive Monte-Carlo Simulation Thousands/Millions surrogate model function calls Much lower computational cost
Yamazaki, W., Dept. of Aero. Eng., Tohoku Univ.Wataru YAMAZAKI, Univ. of Wyoming-7-
Objectives
The efficient adjoint gradient/Hessian calculation methodswill be effective…
for more efficient global design optimizationwith G/H-enhanced surrogate model approach
for more accurate and cheaper uncertainty analysisby inexpensive Monte-Carlo simulation
with G/H-enhanced surrogate model
Development of gradient/Hessian-enhanced surrogate models Application to design optimization and uncertainty analysis
Yamazaki, W., Dept. of Aero. Eng., Tohoku Univ.Wataru YAMAZAKI, Univ. of Wyoming-8-
Kriging, Gradient-enhanced Kriging
Kriging model approach - originally in geological statistics
Two gradient-enhanced Kriging (cokriging or GEK)
Direct CokrigingGradient information is included in the formulation(correlation between func-grad and grad-grad)
Indirect CokrigingSame formulation as original KrigingAdditional samples are created by using the gradient infoKriging model by both real and additional pts
xx
xxx
xx
i
i
yyy T
add
iadd2D example
: Real Sample Point: Additional Sample Point
Yamazaki, W., Dept. of Aero. Eng., Tohoku Univ.Wataru YAMAZAKI, Univ. of Wyoming-9-
Gradient/Hessian-enhanced Kriging
Indirect Approach
xHxGx
xxx
x
TTadd
iadd
iyy
2
1
2D example: Real Sample Point: Additional Sample Point
Arrangements to Use Full Hessian / Diagonal Terms
Major parameters : distance between real / additional pts number of additional pts per real pt
Worse matrix conditioning with smaller distancelarger number of additional pts
Severe tradeoffs for these parameters
Yamazaki, W., Dept. of Aero. Eng., Tohoku Univ.Wataru YAMAZAKI, Univ. of Wyoming-10-
Gradient/Hessian-enhanced KrigingDirect ApproachConsider a random process model estimating a function valueby a linear combination of function/gradient/Hessian components
n
iii
n
iii
n
iii yyywy
1
''
1
'
1
ˆ x
Minimizing Mean-Squared-Error (MSE) between exact/estimated function
2
1
''
1
'
1
ˆ YyyywEyMSEn
iii
n
iii
n
iii x
with an unbiasedness constraint
11
n
iiw
Solving by using the Lagrange multiplier approach
iforJJJ
w
JwyMSEJ
iii
n
ii
01ˆ
1 x
Yamazaki, W., Dept. of Aero. Eng., Tohoku Univ.Wataru YAMAZAKI, Univ. of Wyoming-11-
Gradient/Hessian-enhanced KrigingDirect Approach
Introducing correlation function for covariance termsCorrelation is estimated by distance between two pts with radial basis function
kjji
k
ji
xRxZZCov
RZZCov
ji
ji
xx
xx
xx
xx
,,
,,
2
2
1~0
rx
F
FR
T
Unknown parameters are determined by the following system of equations
,,,,, 111 wT x
Final form of the gradient/Hessian-enhanced direct Kriging approach is
FYRr xx 1ˆ Ty
samplesgivenatninformatioexactyyy
dataobservedbetweennscorrelatio
dataobservedandbetweennscorrelatio
termconstantmean
T
TT
,,,,, ''1
'11
111
Y
R
xr
YRFFRF
x
Yamazaki, W., Dept. of Aero. Eng., Tohoku Univ.Wataru YAMAZAKI, Univ. of Wyoming-12-
Gradient/Hessian-enhanced KrigingDirect Approach
nnnnnn
Nn
Nn
Nn
Nn
Nn
Nn
Ndvn
Ndvn
Nn
Nn
Nn
Nn
Nn
Nn
Nn
Nn
Nn
Nn
Nn
Ndvn
Nn
Ndvn
Nn
dvdv
nn
dv
n
dvdv
nn
dv
nnnn
dv
n
dv
nn
dvdv
nn
dv
n
dv
nn
dv
n
dv
nn
dv
n
dv
n
dv
nn
nnn
dv
nnn
nnn
n
dv
n
n
xx
R
xx
R
xx
R
xx
R
x
R
x
R
xx
R
xx
R
xx
R
xx
R
x
R
x
Rxx
R
xx
R
x
R
xx
R
x
R
x
R
xx
R
xx
R
xx
R
x
R
x
R
x
Rx
R
x
R
x
R
x
RRR
x
R
x
R
x
R
x
RRR
22
,4
211
2
,4
2
,3
11
2
,3
2
,2
2
,2
11
22
,4
11
211
2
,4
11
2
,3
11
11
2
,3
11
2
,2
11
2
,2
2
,3
11
2
,3
2
,2
11
,2
,,
11
2
,3
11
11
2
,3
11
,2
11
2
,2
11
,
11
,
2
,2
11
2
,2
,
11
,,,
2
,2
11
2
,2
,
11
,,,
111
111111111
111
111111111
11
1
111111
111
xxxxxxxxxxxx
xxxxxxxxxxxx
xxxxxxxxxxxx
xxxxxxxxxxxx
xxxxxxxxxxxx
xxxxxxxxxxxx
R
Correlations between F-F, F-G, G-G, F-H, G-H and H-H Up to 4th order derivatives of correlation function Automatic Differentiation by TAPENADE No sensitive parameter Better matrix conditioning than indirect approach
FYRr xx 1ˆ Ty
Yamazaki, W., Dept. of Aero. Eng., Tohoku Univ.Wataru YAMAZAKI, Univ. of Wyoming-13-
0.0
0.2
0.4
0.6
0.8
1.0
0.0 0.2 0.4 0.6 0.8 1.0Design Variable
Fun
ctio
n / R
MS
E
0.0E+00
2.0E-03
4.0E-03
6.0E-03
8.0E-03
1.0E-02
EI
Exact Function Sample Points Kriging RMSE EI
Infill Sampling Criteria for Optimization How to find promising location on surrogate model ? Maximization of Expected Improvement (EI) value Potential of being smaller than current minimum (optimal) Consider both estimated function and uncertainty (RMSE)
s
yys
s
yyyyEI minmin
min
xxxx
00s
EI,
y
EI
Results & DiscussionResults & Discussion
Yamazaki, W., Dept. of Aero. Eng., Tohoku Univ.Wataru YAMAZAKI, Univ. of Wyoming-15-
2D Rastrigin Function Fitting
2122
21 2cos2cos1020 xxxxy x
80 samples by Latin Hypercube SamplingDirect Kriging approach
Exact Rastrigin Function Function-based KrigingGradient-enhancedGradient/Hessian-enhanced
Yamazaki, W., Dept. of Aero. Eng., Tohoku Univ.Wataru YAMAZAKI, Univ. of Wyoming-16-
5D Rosenbrock Function Fitting
F: Function-based KrigingFG: Gradient-enhancedFGHd: G/diag. Hess-enhancedFGH: G/full Hess-enhanced
RMSE .vs. Number of sample points Superiority in direct Kriging approaches
thanks to exact enforcement of derivative information better conditioning of correlation matrix
Yamazaki, W., Dept. of Aero. Eng., Tohoku Univ.Wataru YAMAZAKI, Univ. of Wyoming-17-
Validation on Rosenbrock Func.
Minimization of 20D Rosenbrock 30 initial sample points by LHS EI-based infill sampling criteria Faster convergence in
G/H-enhanced direct approach
1.E-03
1.E-02
1.E-01
1.E+00
1.E+01
1.E+02
1.E+03
1.E+04
0 100 200 300
Number of Sample Points
Obj
ecti
ve F
unct
ion
F
Direct_FG
Direct_FGH
Indirect_FGH
1
1
221
2 1001dvN
iiii xxxy x
Uncertainty analysis on 2D Rosenbrock 5 sample points for surrogate model
(No sample point on the center location) Superior performance in
G/H-enhanced Inexpensive MC (IMC)
CDFs of Full-MC and IMCOptimization History
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0 10 20 30 40 50 60 70 80 90
Function ValueC
DF
Full-MC
IMC_F
IMC_FG
IMC_FGH
Yamazaki, W., Dept. of Aero. Eng., Tohoku Univ.Wataru YAMAZAKI, Univ. of Wyoming-18-
Aerodynamic Data Modeling Unstructured mesh CFD Steady inviscid flow, NACA0012 20,000 triangle elements Mach Number [0.5, 1.5] Angle of Attack[deg] [0.0, 5.0] 21x21=441 validation data
Exact Hypersurface of Lift Coefficient Exact Hypersurface of Drag Coefficient
Yamazaki, W., Dept. of Aero. Eng., Tohoku Univ.Wataru YAMAZAKI, Univ. of Wyoming-19-
Aerodynamic Data Modeling
Adjoint gradient is helpful to construct accurate surrogate model CFD Hessian is not helpful due to noisy design space
Function-based KrigingGradient-enhancedExact
Cl
Cd
Yamazaki, W., Dept. of Aero. Eng., Tohoku Univ.Wataru YAMAZAKI, Univ. of Wyoming-20-
2D Airfoil Shape Optimization
Unstructured mesh CFD Steady inviscid flow, M=0.755 NACA0012, 16 DVs for Hicks-Henne function Objective function of inverse design form
Exact / Approximate CFD Hessian available Computational time of F : 2 min,
FG : 4 min, FGHapprox
. : 6 min, FGHexact : 36 min (4 min in parallel)
Geometrical constraint for sectional area
22
2target2target
000.02
100675.0
2
12
1
dl
dddlll
CC
CCwCCwF
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
x
H(x
)
Yamazaki, W., Dept. of Aero. Eng., Tohoku Univ.Wataru YAMAZAKI, Univ. of Wyoming-21-
2D Airfoil Shape Optimization
Start from 16 initial sample points which only have function info Gradient/Hessian evaluations only for new optimal designs
Faster convergence in derivative-enhanced surrogate model Best design in gradient/exact Hessian-enhanced model
Yamazaki, W., Dept. of Aero. Eng., Tohoku Univ.Wataru YAMAZAKI, Univ. of Wyoming-22-
2D Airfoil Shape Optimization
Towards supercritical airfoils Shock reduction on upper surface
NACA0012 (Baseline) Optimal by G/exact H-enhanced model
Yamazaki, W., Dept. of Aero. Eng., Tohoku Univ.Wataru YAMAZAKI, Univ. of Wyoming-23-
2D Airfoil Shape Uncertainty Analysis
Geometrical uncertainty analysis at optimal airfoil Center = optimal obtained by Grad/exact H model
Comparison between2nd order Moment Method (MM2)
using gradient/Hessian at the centerInexpensive Monte-Carlo (IMC1)
using final surrogate model obtained in optimizationInexpensive Monte-Carlo (IMC2)
using different G/H-enhanced model by 11 samplesFull Non-Linear Monte-Carlo (NLMC)
using 3,000 CFD function calls
optimal (center) ±0.1 airfoil
Yamazaki, W., Dept. of Aero. Eng., Tohoku Univ.Wataru YAMAZAKI, Univ. of Wyoming-24-
2D Airfoil Shape Uncertainty Analysis
Mean of objective w.r.t. standard deviation of all design variables IMC showed good agreement with NLMC at smaller st. devi. Necessity of additional sampling criteria for total model accuracy ? Promising IMC with much cheaper computational cost
MM2 using derivative at the centerIMC1 using G/H surrogate model obtained in optimizationIMC2 using different G/H model by 11 samples (for st.devi.=0.01)NLMC using 3,000 CFD function calls
Yamazaki, W., Dept. of Aero. Eng., Tohoku Univ.Wataru YAMAZAKI, Univ. of Wyoming-25-
Concluding Remarks / Future Works
Development of gradient/Hessian-enhanced Kriging models Application to design optimization and uncertainty analysis
Direct Kriging approach is superior to indirect approach More accurate fitting on exact function Faster convergence towards global optimal design Promising inexpensive Monte-Carlo simulation at much lower cost
Application to higher-dimensional / complicated design problem Robust design with inexpensive Monte-Carlo simulation Gradient/Hessian vector product-enhanced approach
Thank you for your attention !!
Appendix
Yamazaki, W., Dept. of Aero. Eng., Tohoku Univ.Wataru YAMAZAKI, Univ. of Wyoming-27-
Moment Method
Taylor series expansion by grad/Hessian at the center No information about PDF
1st order Moment Method
2nd order Moment Method
dv
i
c
N
iD
iMM
cMM
dD
dF
F
1
2
21
1
x
x
dv dv
ji
c
dv
i
c
N
i
N
jDD
jiMMMM
N
iD
iMMMM
dDdD
Fd
dD
Fd
1 1
22
21
22
1
22
2
12
2
1
2
1
x
x
Yamazaki, W., Dept. of Aero. Eng., Tohoku Univ.Wataru YAMAZAKI, Univ. of Wyoming-28-
Gradient/Hessian-enhanced KrigingImplementation Details
Correlation function of a RBF
Estimation of hyper parameters by maximizing likelihood function with GA
Correlation matrix inversion by Cholesky decomposition
Search of new sample point location by maximizing Expected Improvement (EI) value with GA
else
hforhhhhscf
0
13183513
1,
226
Yamazaki, W., Dept. of Aero. Eng., Tohoku Univ.Wataru YAMAZAKI, Univ. of Wyoming-29-
Infill Sampling Criteria for Optimization How to find promising location on surrogate model ? Expected Improvement (EI) value Potential of being smaller than current minimum (optimal) Consider both estimated function and uncertainty (RMSE)
s
yys
s
yyyyEI minmin
min
xxxx
00s
EI,
y
EI
0.0
0.2
0.4
0.6
0.8
1.0
0.0 0.2 0.4 0.6 0.8 1.0Design Variable
Fun
ctio
n / R
MS
E
0.0E+00
2.0E-03
4.0E-03
6.0E-03
8.0E-03
1.0E-02
EI
Exact Function Sample Points Kriging RMSE EI
0.0
0.2
0.4
0.6
0.8
1.0
0.0 0.2 0.4 0.6 0.8 1.0Design Variable
Fun
ctio
n / R
MS
E
0.0E+00
2.0E-03
4.0E-03
6.0E-03
8.0E-03
1.0E-02
EI
Exact Function Sample Points Kriging RMSE EI
0.0
0.2
0.4
0.6
0.8
1.0
0.0 0.2 0.4 0.6 0.8 1.0Design Variable
Fun
ctio
n / R
MS
E
0.0E+00
2.0E-03
4.0E-03
6.0E-03
8.0E-03
1.0E-02
EI
Exact Function Sample Points Kriging RMSE EI
0.0
0.2
0.4
0.6
0.8
1.0
0.0 0.2 0.4 0.6 0.8 1.0Design Variable
Fun
ctio
n / R
MS
E
0.0E+00
2.0E-03
4.0E-03
6.0E-03
8.0E-03
1.0E-02
EI
Exact Function Sample Points Kriging RMSE EI
0.0
0.2
0.4
0.6
0.8
1.0
0.0 0.2 0.4 0.6 0.8 1.0Design Variable
Fun
ctio
n / R
MS
E
0.0E+00
2.0E-03
4.0E-03
6.0E-03
8.0E-03
1.0E-02
EI
Exact Function Sample Points Kriging RMSE EI
0.0
0.2
0.4
0.6
0.8
1.0
0.0 0.2 0.4 0.6 0.8 1.0Design Variable
Fun
ctio
n / R
MS
E
0.0E+00
2.0E-03
4.0E-03
6.0E-03
8.0E-03
1.0E-02
EI
Exact Function Sample Points Kriging RMSE EI
0.0
0.2
0.4
0.6
0.8
1.0
0.0 0.2 0.4 0.6 0.8 1.0Design Variable
Fun
ctio
n / R
MS
E
0.0E+00
2.0E-03
4.0E-03
6.0E-03
8.0E-03
1.0E-02
EI
Exact Function Sample Points Kriging RMSE EI
EI-based criteria have good balancebetween global/local searching
Yamazaki, W., Dept. of Aero. Eng., Tohoku Univ.Wataru YAMAZAKI, Univ. of Wyoming-30-
5D Rosenbrock Function Fitting
# of pieces of information = sum of # of F/G/H net components To scatter samples is better than concentration at limited samples Approximated computational time factor
G/H-enhanced surrogate model provides better performancewith efficient Gradient/Hessian calculation methods
FGHFGFhasiifTTTF i
N
ii
sample
//,3/2/1,1
Yamazaki, W., Dept. of Aero. Eng., Tohoku Univ.Wataru YAMAZAKI, Univ. of Wyoming-31-
1D Step Function Fitting
-0.5
0.0
0.5
1.0
1.5
0.0 0.2 0.4 0.6 0.8 1.0
Design Variable
Fun
ctio
n V
alue Exact
Samples
F
FG
FGH
Much better fit by G/H-enhanced direct Kriging
Yamazaki, W., Dept. of Aero. Eng., Tohoku Univ.Wataru YAMAZAKI, Univ. of Wyoming-32-
Minimization of 20D Rosenbrock Func.
Minimization of 20 dimensional Rosenbrock function No computational cost for Func/Grad/Hess evaluation Expensive for construction - likelihood function maximization
- inversion of correlation matrix Parallel computation for the likelihood maximization problem
1.E-02
1.E-01
1.E+00
1.E+01
1.E+02
1.E+03
1.E+04
0 3000 6000 9000 12000 15000
Computational Time [sec]
Obj
ecti
ve F
unct
ion
F
FG
FGH
Yamazaki, W., Dept. of Aero. Eng., Tohoku Univ.Wataru YAMAZAKI, Univ. of Wyoming-33-
Uncertainty Analysis
Uncertainty analysis at (1.0,1.0) on 2D Rosenbrock 5 sample points for surrogate model approaches
(No sample point on the center location) 2nd order Moment Method (MM2) by G/H at the center Superior results in G/H-enhanced Inexpensive MC (IMC)
St. Devi. = 0.15 means the possibility within -0.15<dx<0.15 is about 70%
-10
10
30
50
70
90
110
130
150
0.0 0.1 0.2 0.3 0.4 0.5
Standard Deviation of DVs
Mea
n of
Fun
ctio
n
Full-MC
MM2
IMC_F
IMC_FG
IMC_FGH
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0 10 20 30 40 50 60 70 80 90
Function Value
CD
F
CDF at St. Devi.=0.15
Yamazaki, W., Dept. of Aero. Eng., Tohoku Univ.Wataru YAMAZAKI, Univ. of Wyoming-34-
Aerodynamic Data Modeling
0.208
0.209
0.210
0.211
0.212
1.390 1.395 1.400 1.405 1.410Mach Number
CL
CFD Data
Linear by Adj_Grad
Quadratic by Adj_G/H
0.1080
0.1082
0.1084
0.1086
0.1088
0.1090
1.390 1.395 1.400 1.405 1.410
Mach Number
CD
CFD Data
Linear by Adj_Grad
Quadratic by Adj_G/H
Cl Cd
NACA0012 M=1.4 AoA=3.5[deg] Noisy in Mach number direction
Yamazaki, W., Dept. of Aero. Eng., Tohoku Univ.Wataru YAMAZAKI, Univ. of Wyoming-35-
2D Airfoil Shape Uncertainty Analysis
Cumulative Density Function at St. Devi. of 0.01 Quadratic model only by using gradient/Hessian at optimal Additional sampling criteria to increase total model accuracy
Top Related