Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical...

36
Innovative Numerical Technologies Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005 Nonlinear Mathematical Programming - From Small to Very Large Scale Applications - Nonlinear Mathematical Programming - From Small to Very Large Scale Applications - Frank Vogel, inuTech GmbH Klaus Schittkowski, Universität Bayreuth

Transcript of Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical...

Page 1: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

Nonlinear Mathematical Programming - From Small to Very Large

Scale Applications -

Nonlinear Mathematical Programming - From Small to Very Large

Scale Applications -

Frank Vogel, inuTech GmbHKlaus Schittkowski, Universität Bayreuth

Page 2: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

• We offer mathematical software and R&D servicesin areas such as

•Mathematical optimization

•Differential equations

•Further areas that require a thoroughknowledge of mathematical foundations

• with a staff of currently 12 people

• to close the gap between Engineering and Mathematics

• for customers such as:

inuTech – One Slide Summary

Page 3: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

• SQP and SCP methods for nonlinear programming

• Comparative numerical tests

• Efficient sensitivity calculation

• Small to very large scale industrial appilications

• Conclusions

Contents

Page 4: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

Nonlinear programming problem (NLP):

Problem Formulation

mjxgx

xf

jn ,...,1,0)(:

)( min

=≤ℜ∈

x - Design Vector

f - Objective Function

gj – Constraint Functions

Assumptions:

• All model functions are smooth (differentiable)

• The problem may be highly nonlinear

• The problem may become very large scale

Page 5: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

General Procedure

mjxgx

xfkj

n

k

,...,1,0)(:

)( min

=≤ℜ∈

Search direction: Formulate and solve a ”simpler” subproblem

called NLPk :

Goal: Given an current iterate , determine a search

direction and a step length to calculate the

next iterate

nkx ℜ∈

kkk dxx 1 α+=+

nkd ℜ∈ ℜ∈α

Page 6: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

• NLPk is strictly convex and smooth, i.e. NLPk has a unique solution

• NLPk is a first order approximation, i.e.

• The search direction is a descent direction forthe augmented Lagrangian merit function (Schittkowski 1982)

Requirements

General Procedure (cont.)

nkd ℜ∈

)()()()(

)()()()(

kkjkjk

kjkj

kk

kkk

k

xgxgxgxg

xfxfxfxf

∇=∇=∇=∇=

nkd ℜ∈

Page 7: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

New Iterate: Search for such that becomes

acceptable (”principle of sufficient descent”)

Line Search

10 ≤< kα

( )kkrk dxk

)( αα +Φ=Ψ

)(αkΨ

i.e., L2-Merit Function:

∑∈

+=ΦJj

jjr xgrxfx 2)(2

1)()(

Page 8: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

Goal: Fast local convergence speed based on local quadratic approximation of the Lagrangian function and linearization of constraints (Wilson 1963, Han 1976, Powell 1978)

SQP Methods (Sequential Quadratic Programming)

( ) ( ) ( )( ) mjxxxgxgxg

xxBxxxxxfxf

kT

kjkjkj

kkT

kkT

kk

,...,1 ,)()()(2

1)()(

=−∇+=

−−+−∇=

Motivation: In case of equality constraints only, the optimal solution

represents one Newton step for solving the Karush-Kuhn-

Tucker optimality conditions, if is the Hessian matrix of the

Lagrangian function

kd

kB

Page 9: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

• In case of n active constraints, the SQP method behaves like Newton's method for solving the corresponding system of equations.

• The algorithm is globally convergent.

• The local convergence speed is quadratic if is exact and superlinear if is an approximation of the Hessian (i.e. BFGS):

Properties of SQP Methods

0 with**1 →−<−+ kkkk xxxx γγ

kBkB

Page 10: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

Goal: Convex, 1st order approximation of model functions.

SCP Methods (Sequential Convex Programming)

mjLxxU

xg

LxxUxf

i iii

kji

ii

kjik

jkj

i iii

ki

ii

kikk

,...,1 ,)(

)(

,,

0,0,0

=−

−−

+=

−−

−+=

∑ ∑

∑ ∑

+ −

+ −

ββα

ββα

Idea: Insert inverse variables (with or without asymptotes) dependingon sign of partial derivative and linearize

summation over all indices with positiv partial derivative

( vice versa)

∑+

i:

∑−

i

Motivation: Mechanical engineering, i.e. statically determinatedstructures are linear in reciprocal variables (Fleury 1989, Svanberg1987, Zillober 1994)

Page 11: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

Subproblem solution

SCP Methods (cont.)

• Regularization of objective function leads to strictly convex, separable, often very large NLP's (diagonal and positive definite Hessian)

• Numerical solution by interior point method possible with n x n or m x m systems of equations

• Exploiting sparsity patterns in systems of linear equations

Page 12: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

Numerical Tests: SQP versus SCP

306 standard (medium sized) test problems: Percentage of successful runs (SUCC), avg. number of function evaluations (NF), and avg. number of iterations (NIT) for SQP code NLPQLP (Schittkowski) and SCP code SCPIP001 (Zillober)

427493SCPIP

2539100NLPQLP

NITNFSUCC in %code

Page 13: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

Sensitivity Analysis

Both SQP and SCP require the sensitivity information of the objective and constraint functions wrt. the optimization variables:

Let g be a function which depends on x (space), p (optimization variable),

t (time) and , , the solution of:

),,,,,( tuuupxgg⋅⋅⋅

=

⋅uu

⋅⋅u

(PDE) RuKuCuM =++⋅⋅⋅

Page 14: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

Sensitivity Analysis – numerical

Problems:

• (n+1) solutions of (PDE), i.e. computationally very expensive if

– Solution of (PDE) is time consuming

– Number of optimization parameters (n) is large

nip

tuuupxgtuuupepxgg

p i

ii

i

,...,1 ,),,,,,(),,,,,( =

∆−∆+=

∂∂

⋅⋅⋅⋅⋅⋅

Page 15: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

Sensitivity Analysis – analytical

Formal differentiation of (PDE) wrt. p yields:

(1) up

Ku

p

Cu

p

M

p

R

p

uK

p

uC

p

uM

∂∂−

∂∂−

∂∂−

∂∂=

∂∂+

∂∂+

∂∂ ⋅⋅⋅

⋅⋅⋅

i.e. structural mechanics (linear):

(2) up

K

p

R

p

uK

∂∂−

∂∂=

∂∂

Solution of (1), (2) resp., yields sensitivities of u and thus of g in one shot!

Page 16: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

Small to very large scale industrial Applications

Page 17: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

Optimization of airplanes (AIRBUS)

Physical problem:

• Linear structural mechanics

• 15 load cases fortwo different models(with and without ramp)

Page 18: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

Optimization of airplanes (cont.)

Optimization problem:

• Optimization variables: Beam cross sections and shell thicknesses

• Objective: Minimize weight

• Constraints: stresses, skin buckling of shells, euler buckling of beams, displacements, for all load cases simultaneously.

• Very large scale problem with > 2000 optimization variables and > 2.000.000 constraints

• Gradients are calculated semi-analytically

• Numerical solution using SCPIP in combination with an efficientactive grouping / set logic

Page 19: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

Topology Optimization

K: penalized global stiffness matrix u: nodal displacementsV: volume p: external loadxi,j: relative density in element i,j nx: number of finite elements (x-direction)ny: number of finite elements (y-direction)

(compliance)

(equilibrium condition)

(volume)

Page 20: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

Topology Optimization (cont.)

Optimization problem:

• Optimization variables: Pseudo Element Densities

• Objective: Minimize Compliance

• Constraints: Volume Constraints

• Very large scale problem with n (number of elements) optimization variables

• Gradients are calculated analytically

• Numerical solution using SCPIP

∑ =⋅= n

i ii kxxK1

03)(

Page 21: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

Topology Optimization – Case Studie (Half Beam)

Increasing Volume

De

cre

asi

ng

Filt

er

Ra

diu

s

Page 22: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

Optimal control of Elliptic PDEs

Semi-linear elliptic control problem on (Maurer, Mittelmann 2001)

( ) ( )1,01,0 ×=Ω

( )

9u8-

on 371.0

)( on 0

on

)() 2sin() 2sin()(2

1min 22

21

≤≤Ω≤

ΩΓ=+∂Ω=−∆−∈

+−

∞Ω∫

y

yy

ueyLu

dxxuxxxy

y

ν

ππ

Page 23: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

Optimal control of Elliptic PDEs (cont.)

Discretization: uniform square grid of size N

• Full discretization subject to state and control variables

• 5-star-formula for second derivatives

• Analytical calculation of gradients

• Very large scale problem solved by SCPIP

Page 24: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

Optimal control of Elliptic PDEs (cont.)

Numerical results: SCPIP (Acc=1.0E-7)

0.053616250,997499,998500

0.05302140,39779,998200

0.05351690,597179,998300

0.053418160,797319,998400

12

24

nit

0.0548312,197719,998600

0.052710,19719,998100

f(x)mnN+1

Page 25: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

Optimal Design of Capacitors

Physical problem:

• Capacitor with 2 contacts

• Voltage: 0V at top, 220V at bottom, symm. conditions left and right

• Stationary model:

u – electrical potential, px ,py – material conductivity

• Model solved employing a Diffpack – Simulator (based on FEM)

0=

∂∂

∂∂+

∂∂

∂∂

y

up

yx

up

x yx

0

0)(

220)(

=∂∂

==

nu

Vxu

Vxu

3

2

1

Γ∈Γ∈Γ∈

x

x

x

Page 26: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

Optimal Design of Capacitors (cont.)

Optimization problem:

• Optimization variables: pi – material conductivity for element i.

• Objective: Minimize the flow resistance on potential field

• Constraints on: Current densities for all elements, Current on bottom boundary

• Large scale problem: n (number of elements) variables, n+2 constraints

• A direct method has been implemented using Diffpack to calculatethe gradients analytically:

s.t. , 1

2

∑=

∇n

iiii

pAupMin

i

ZielZiel

ii

IdxnupI

nijup

1.1 9.0

,...,1,2max

22

≤∇≤

=≤∇

∫r

udp

dK

dp

df

dp

duK

iii

−=

Page 27: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

Optimal Design of Capacitors (cont.)

Optimization problem:

• Numerical solution using SCPIP

Current DensitiesOptimal conductivity distribution Potential u

Page 28: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

Optimization of MRI Scanner (SIEMENS)main magnet

gradient coil

Page 29: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

Physical problem:

• Coupled physical effects (solved using CAPA and Siemens in-housesolver)

Optimization of MRI Scanner (cont.)

Page 30: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

Optimization of MRI Scanner (cont.)

Optimization problem:

• Optimization variables: currents of prim. and second. gradient coils

• Objective function: Eddy current losses in frequency range (calculated by CAPA)

• Constraints: inductance, linearity in a given field-of-view, shielding, power dissipation, etc. (calculated by SIEMENS in-house tools)

• Gradients are calculated semi-analytically with CAPA

• Numerical solution using SQP

fQAfz

pf

f

ett dd)()(2

1

arg2

∫ ∫

−Ω=Φ

Ω

rρω

Eddy current losses

Frequency (Hz)

original state

automated optimisation

Manual optimisation

Page 31: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

Electrodynamic Loudspeaker (HARMAN/KARDON)

Physical problem:

• Coupled simulation of magnetic, mechanic and acoustic domain(solved with CAPA)

• Nonlinear magnetics and coupling effects

• Time domain simulation with 2.000 time stepsand more

• Single simulation run takes0.5 – 2 h cpu time

Page 32: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

Electrodynamic Loudspeaker (cont.)

Optimization problem:

• Optimization variables: Material parameters of Loudspeaker

• Objective function: „Constant“ SPL (sound pressure level) in givenfrequency domain:

• Semi-analytical calculation of sensitivities within CAPA

• Numerical solution using SQP

( ) ∑

Φ−Φ=2500,...,760,750

2arg )(),()(

ifietTi fxfxF

Vergleich Start / Target / Optimierter Schalldruck

5,006,508,009,50

11,0012,5014,0015,5017,0018,5020,0021,5023,0024,5026,0027,5029,00

750,

0085

0,00

950,

0010

50,00

1150

,0012

50,00

1350

,0014

50,00

1550

,0016

50,00

1750

,0018

50,00

1950

,0020

50,00

2150

,0022

50,00

2350

,0024

50,00

Frequenz

Sch

alld

ruck

TargetStartlösungOptimierte Lösung

Page 33: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

Conclusions

• SQP methods represent the state-of-the-art in nonlinear programming for smooth, small to medium sized and highly nonlinear optimization problems

• SQP methods are widely used and accepted in many engineering sciences, academia and commercial

• SCP methods have been invented by engineers, tuned tostructural design optimization

• SCP methods are sometimes even more efficient than SQP methods in case of lower accuracy requirements

• SCP methods are applicable for solving very large scaleoptimization problems

• Efficient Sensitivity Analysis is always beneficial (especially for large scale problems)

Page 34: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

Advantages and disadvantages

Advantages of SQP Methods

• Robust

• Highly accurate solutions with fast final convergence speed

• Highly nonlinear problems

• General purpose tools

Disadvantages of SQP Methods

• Fast convergence only in case of accurate gradients

• Large storage requirements

Page 35: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

Advantages and disadvantages (cont.)

Advantages of SCP Methods

• Tuned for structural optimization

• No inheritance of round-off errors in function and gradient calculations

• Excellent convergence speed in special situations

• Able to solve very large scale problems

Disadvantages of SCP Methods

• Slow convergence possible

Page 36: Nonlinear Mathematical Programming -From Small to Very ...€¦ · Nonlinear Mathematical Programming - From Small to Very Large Scale Applications --From Small to Very Large Frank

Innovative Numerical

Technologies

Weimarer Optimierungs- und Stochastiktage 2.0, December 1-2, 2005

Thank you for your attention !