Practice Problems 1 2 3 4 5 6 7 8 9 Return to MENU Practice Problems 123456789123456789.
Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical...
Transcript of Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical...
1© 2016 The MathWorks, Inc.
Solving Optimization Problems with MATLAB
Loren Shure
2
Introduction
Least-squares minimization
Nonlinear optimization
Mixed-integer programming
Global optimization
Topics
3
Optimization Problems
Minimize Risk
Maximize Profits
Maximize Fuel Efficiency
4
Design Process
Initial
Design
Variables
System
Modify
Design
Variables
Optimal
DesignObjectives
met?
No
Yes
5
Manually (trial-and-error or iteratively)
Why use Optimization?
Initial
Guess
6
Automatically (using optimization techniques)
Initial
Guess
Why use Optimization?
7
Why use Optimization?
Finding better (optimal) designs and decisions
Faster design and decision evaluations
Automate routine decisions
Useful for trade-off analysis
Non-intuitive designs may be found
Antenna Design Using Genetic Algorithmhttp://ic.arc.nasa.gov/projects/esg/research/antenna.htm
8
Introduction
Least-squares minimization
Nonlinear optimization
Mixed-integer programming
Global optimization
Topics
9
Curve Fitting Demo
Given some data:
Fit a curve of the form:
teccty 21
)(
t = [0 .3 .8 1.1 1.6 2.3];
y = [.82 .72 .63 .60 .55 .50];
10
How to solve?
Ecc
cey
eccty
t
t
2
1
21
1
)(
As a linear system of equations:
2
2min yEc
Ecy
c
Can’t solve this exactly
(6 eqns, 2 unknowns)
An optimization problem!
2
1
3.2
6.1
1.1
8.0
3.0
0
1
1
1
1
1
1
50.0
55.0
60.0
63.0
72.0
82.0
c
c
e
e
e
e
e
e
11
Introduction
Least-squares minimization
Nonlinear optimization
Mixed-integer programming
Global optimization
Topics
12
Nonlinear Optimization
min𝑥
log 1 + 𝑥1 −4
3
2
+ 3 𝑥1 + 𝑥2 − 𝑥13 2
13
Nonlinear Optimization - Modeling Gantry Crane
Determine acceleration profile that
minimizes payload swing
s25s4
s20s1
s20s1
:sConstraint
2
1
21
f
p
p
ppf
t
t
t
ttt
14
Symbolic Math ToolboxFunctions for analytical computations
Conveniently manage & document symbolic
computations in Live Editor
• Math notation, embedded text, graphics
• Share work as pdf or html
Perform exact computations using familiar
MATLAB syntax in MATLAB
Integration
Differentiation
Solving equations
Transforms
Simplification
Integrate with numeric computing – MATLAB,
Simulink and Simscape language
Perform Variable-precision arithmetic
15
Introduction
Least-squares minimization
Nonlinear optimization
Mixed-integer programming
Global optimization
Topics
16
Mixed-Integer Programming
Many things exist in discrete amounts:
– Shares of stock
– Number of cars a factory produces
– Number of cows on a farm
Often have binary decisions:
– On/off
– Buy/don’t buy
Mixed-integer linear programming:
– Solve optimization problem while enforcing that certain variables need to be integer
17
Continuous and integer variables
𝑥1 ∈ 0, 100 𝑥2 ∈ 1,2,3,4,5
Linear objective and constraints
min𝑥
−𝑥1 − 2𝑥2
𝑥1 + 4𝑥2 ≤ 20𝑥1 + 𝑥2 = 10
such that
Mixed-Integer Linear Programming
18
Traveling Salesman Problem
Problem
How to find the shortest path through a series of points?
Solution
Calculate distances between all combinations of points
Solve an optimization problem where variables correspond to trips between two points
1
11
0
1
1
0
0
00
19
Introduction
Least-squares minimization
Nonlinear optimization
Mixed-integer programming
Global optimization
Topics
20
Example Global Optimization Problems
Why does fmincon have a hard time finding the
function minimum?
0 5 10
-10
-5
0
5
10
x
Starting at 10
0 5 10
-10
-5
0
5
10
x
Starting at 8
0 5 10
-10
-5
0
5
10
x
Starting at 6
x s
in(x
) +
x c
os(2
x)
0 5 10
-10
-5
0
5
10
x
Starting at 3
0 5 10
-10
-5
0
5
10
x
Starting at 1
0 5 10
-10
-5
0
5
10
x
Starting at 0
x s
in(x
) +
x c
os(2
x)
21
Initial
Guess
Example Global Optimization Problems
Why didn’t fminunc find the maximum efficiency?
22
Example Global Optimization Problems
Why didn’t nonlinear regression find a good fit?
0 20 40 60 80 100 1200
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
t
c
c=b1e-b
4t+b
2e-b
5t+b
3e-b
6t
23
Global Optimization
Goal:
Want to find the lowest/largest value of
the nonlinear function that has many local
minima/maxima
Problem:
Traditional solvers often return one of the
local minima (not the global)
Solution:
A solver that locates globally optimal
solutions
Global Minimum at [0 0]
Rastrigin’s Function
24
Global Optimization Solvers Covered Today
Multi Start
Global Search
Simulated Annealing
Pattern Search
Particle Swarm
Genetic Algorithm
25
MultiStart Demo – Nonlinear Regression
lsqcurvefit solution MultiStart solution
26
MULTISTART
27
What is MultiStart?
Run a local solver from each set
of start points
Option to filter starting points
based on feasibility
Supports parallel computing
28
MultiStart Demo – Peaks Function
29
GLOBAL SEARCH
30
What is GlobalSearch?
Multistart heuristic algorithm
Calls fmincon from multiple
start points to try and find a
global minimum
Filters/removes non-promising
start points
31
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
GlobalSearch Overview Schematic Problem
Peaks function
Three minima
Green, z = -0.065
Red, z= -3.05
Blue, z = -6.55
x
y
32
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
GlobalSearch Overview – Stage 0Run from specified x0
x
y
33
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
GlobalSearch Overview – Stage 1
3
6
0
00
4
0
-2
x
y
34
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
GlobalSearch Overview – Stage 1
3
6
0
00
4
0
-2
x
y
35
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
GlobalSearch Overview – Stage 1
x
y
36
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
GlobalSearch Overview – Stage 2
x
y
37
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
GlobalSearch Overview – Stage 2
x
y
38
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
GlobalSearch Overview – Stage 2
x
y
39
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
GlobalSearch Overview – Stage 2
x
y
40
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
GlobalSearch Overview – Stage 2
6
Current penalty
threshold value : 4
x
y
41
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
GlobalSearch Overview – Stage 2
x
y
42
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
GlobalSearch Overview – Stage 2
Current penalty
threshold value : 4
-3
x
y
43
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
GlobalSearch Overview – Stage 2Expand basin of attraction if minimum already found
Current penalty threshold value : 2
-0.1
x
y Basins can overlap
44
GlobalSearch Demo – Peaks Function
45
SIMULATED ANNEALING
46
What is Simulated Annealing?
A probabilistic metaheuristic
approach based upon the physical
process of annealing in
metallurgy.
Controlled cooling of a metal
allows atoms to realign from a
random higher energy state to an
ordered crystalline (globally) lower
energy state
47
Simulated Annealing Overview – Iteration 1Run from specified x0
x
y
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
0.9
48
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Simulated Annealing Overview – Iteration 1
3
x
y 0.9
Possible New Points:
Standard Normal N(0,1) * Temperature
Temperature = 1
49
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Simulated Annealing Overview – Iteration 1
3
x
y 0.9
Temperature = 1
11.01
1/)9.03(
Taccepte
P
50
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Simulated Annealing Overview – Iteration 1
3
x
y 0.9
Temperature = 1
0.3
51
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Simulated Annealing Overview – Iteration 1
3
x
y 0.9
Temperature = 1
0.3
52
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Simulated Annealing Overview – Iteration 2
3
x
y 0.9
Temperature = 1
0.3
53
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Simulated Annealing Overview – Iteration 2
3
x
y 0.9
Temperature = 0.75
0.3
-1.2
54
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Simulated Annealing Overview – Iteration N-1
3
x
y 0.9
Temperature = 0.1
0.3
-1.2
-3
55
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Simulated Annealing Overview – Iteration NReannealing
3
x
y 0.9
Temperature = 1
0.3
-1.2
-3
-2
56
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Simulated Annealing Overview – Iteration NReannealing
3
x
y 0.9
Temperature = 1
0.3
-1.2
-3
-2
27.01
1/))3(2(
Taccepte
P
57
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Simulated Annealing Overview – Iteration NReannealing
3
x
y 0.9
Temperature = 1
0.3
-1.2
-3
-2
58
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Simulated Annealing Overview – Iteration N+1
3
x
y 0.9
Temperature = 0.75
0.3
-1.2
-3
-2
-3
59
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Simulated Annealing Overview – Iteration N+1
3
x
y 0.9
Temperature = 0.75
0.3
-1.2
-3
-2
-3
60
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Simulated Annealing Overview – Iteration …
3
x
y 0.9
Temperature = 0.75
0.3
-1.2
-3
-2
-3
-6.5
61
Simulated Annealing – Peaks Function
62
PATTERN SEARCH
(DIRECT SEARCH)
63
What is Pattern Search?
An approach that uses a pattern
of search directions around the
existing points
Expands/contracts around the
current point when a solution is
not found
Does not rely on gradients: works
on smooth and nonsmooth
problems
64
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Pattern Search Overview – Iteration 1Run from specified x0
x
y
3
65
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Pattern Search Overview – Iteration 1Apply pattern vector, poll new points for improvement
x
y
3
Mesh size = 1
Pattern vectors = [1,0], [0,1], [-1,0], [0,-1]
0_*_ xvectorpatternsizemeshPnew
0]0,1[*1 x1.6
0.4
4.6
2.8
First poll successful
Complete Poll (not default)
66
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Pattern Search Overview – Iteration 2
x
y
3
Mesh size = 2
Pattern vectors = [1,0], [0,1], [-1,0], [0,-1]
1.6
0.4
4.6
2.8
-4
0.3-2.8
Complete Poll
67
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Pattern Search Overview – Iteration 3
x
y
3
Mesh size = 4
Pattern vectors = [1,0], [0,1], [-1,0], [0,-1]
1.6
0.4
4.6
2.8
-4
0.3-2.8
68
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Pattern Search Overview – Iteration 4
x
y
3
Mesh size = 4*0.5 = 2
Pattern vectors = [1,0], [0,1], [-1,0], [0,-1]
1.6
0.4
4.6
2.8
-4
0.3-2.8
69
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Pattern Search Overview – Iteration NContinue expansion/contraction until convergence…
x
y
31.6
0.4
4.6
2.8
-4
0.3-2.8
-6.5
70
Pattern Search – Peaks Function
71
Pattern Search Climbs Mount Washington
72
PARTICLE SWARM
73
What is Particle Swarm Optimization?
A collection of particles move
throughout the region
Particles have velocity and are
affected by the other particles in
the swarm
Does not rely on gradients: works
on smooth and nonsmooth
problems
74
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Particle Swarm Overview – Iteration 1Initialize particle locations and velocities, evaluate all
locations
x
y
75
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Particle Swarm Overview – Iteration NUpdate velocities for each particle
x
yPrevious
Velocity
Best Location
for this Particle
Best Location for
Neighbor Particles
76
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Particle Swarm Overview – Iteration NUpdate velocities for each particle
x
y
New
Velocity
77
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Particle Swarm Overview – Iteration NMove particles based on new velocities
x
y
78
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Particle Swarm Overview – Iteration NContinue swarming until convergence
x
y
79
Particle Swarm – Peaks Function
80
GENETIC ALGORITHM
81
What is a Genetic Algorithm?
Uses concepts from evolutionary
biology
Start with an initial generation of
candidate solutions that are tested
against the objective function
Subsequent generations evolve
from the 1st through selection,
crossover and mutation
82
How Evolution Works – Binary Case
Selection
– Retain the best performing bit strings from one generation to the next. Favor these for
reproduction
– parent1 = [ 1 0 1 0 0 1 1 0 0 0 ]
– parent2 = [ 1 0 0 1 0 0 1 0 1 0 ]
Crossover
– parent1 = [ 1 0 1 0 0 1 1 0 0 0 ]
– parent2 = [ 1 0 0 1 0 0 1 0 1 0 ]
– child = [ 1 0 0 0 0 1 1 0 1 0 ]
Mutation
– parent = [ 1 0 1 0 0 1 1 0 0 0 ]
– child = [ 0 1 0 1 0 1 0 0 0 1 ]
83
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Genetic Algorithm – Iteration 1Evaluate initial population
x
y
84
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Genetic Algorithm – Iteration 1Select a few good solutions for reproduction
x
y
85
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Genetic Algorithm – Iteration 2Generate new population and evaluate
x
y
86
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Genetic Algorithm – Iteration 2
x
y
87
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Genetic Algorithm – Iteration 3
x
y
88
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Genetic Algorithm – Iteration 3
x
y
89
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
Genetic Algorithm – Iteration NContinue process until stopping criteria are met
x
y
Solution found
90
Genetic Algorithm – Peaks Function
91
Genetic Algorithm – Integer Constraints
Mixed Integer Optimization
s.t. some x are integers
Examples
Only certain sizes of components
available
Can only purchase whole shares of stock
)(min xfx
92
Application: Circuit Component Selection
6 components to size
Only certain sizes available
Objective:
– Match Voltage vs. Temperature
curve
Thermistor Circuit
300 Ω
330 Ω
360 Ω
…
180k Ω
200k Ω
220k Ω
Thermistors:
Resistance varies
nonlinearly with
temperature
?
𝑅𝑇𝐻 =𝑅𝑇𝐻,𝑁𝑜𝑚
𝑒𝛽𝑇−𝑇𝑁𝑜𝑚𝑇∗𝑇𝑁𝑜𝑚
93
Global Optimization Toolbox Solvers
GlobalSearch, MultiStart
– Well suited for smooth objective and constraints
– Return the location of local and global minima
ga, gamultiobj, simulannealbnd, particleswarm
– Many function evaluations to sample the search space
– Work on both smooth and nonsmooth problems
patternsearch
– Fewer function evaluations than ga, simulannealbnd, particlewarm
– Does not rely on gradient calculation like GlobalSearch and MultiStart
– Works on both smooth and nonsmooth problems
94
Optimization Toolbox Solvers
fmincon, fminbnd, fminunc, fgoalattain, fminimax
– Nonlinear constraints and objectives
– Gradient-based methods for smooth objectives and constraints
quadprog, linprog
– Linear constraints and quadratic or linear objective, respectively
intlinprog
– Linear constraints and objective and integer variables
lsqlin, lsqnonneg
– Constrained linear least squares
lsqnonlin, lsqcurvefit
– Nonlinear least squares
fsolve
– Nonlinear equations
95
Speeding-up with Parallel Computing
Global Optimization solvers that support Parallel
Computing:– ga, gamultiobj: Members of population evaluated in parallel
at each iteration
– patternsearch: Poll points evaluated in parallel at each
iteration
– particleswarm: Population evaluated in parallel at each
iteration
– MultiStart: Start points evaluated in parallel
Optimization solvers that support Parallel Computing:– fmincon: parallel evaluation of objective function for finite
differences
– fminunc, fminimax, fgoalattain,fsolve,
lsqcurvefit, lsqnonlin: same as fmincon
Parallel Computing can also be used in the Objective
Function– parfor
96
Speed up parallel applications
Take advantage of GPUs
Prototype code for your cluster
Parallel Computing Toolbox for the Desktop
Simulink, Blocksets,
and Other Toolboxes
Local
Desktop Computer
MATLAB
……
97
Scale Up to Clusters and Clouds
Cluster
Scheduler
Computer Cluster
…
…
…
…
…
…
… … …
Simulink, Blocksets,
and Other Toolboxes
Local
Desktop Computer
MATLAB
……
98
Learn More about Optimization with MATLAB
Recorded webinar: Optimization in
MATLAB: An Introduction to Quadratic
Programming
Optimization Toolbox Web demo:
Finding an Optimal Path using MATLAB
and Optimization Toolbox
MATLAB Digest: Improving
Optimization Performance with Parallel
Computing
MATLAB Digest: Using Symbolic
Gradients for OptimizationRecorded webinar: Mixed Integer
Linear Programming in MATLAB
Recorded webinar: Optimization in
MATLAB for Financial Applications
1 2 3 4 50
500
1000
1500
2000
2500
3000
Bonds
# P
urc
ha
se
d
Cash Flow Matching Example
99
Key Takeaways
Solve a wide variety of optimization problems in MATLAB
– Linear and Nonlinear
– Continuous and mixed-integer
– Smooth and Nonsmooth
Find better solutions to multiple minima and non-smooth problems using global
optimization
Use symbolic math for setting up problems and automatically calculating gradients
Using parallel computing to speed up optimization problems
100© 2016 The MathWorks, Inc.
Questions?