22.04.2013 Dr. Timm Faulwasser, Laboratoire dAutomatique...

Optimal Control: A Primer on Direct Solutions Methods Dr. Timm Faulwasser, Laboratoire d‘Automatique, EPFL

22.04.2013

Part III – Optimal Control

Timm Faulwasser <[email protected]>

Laboratoire d’Automatique

Course Overview (Optimal Control Part)

Introduction

Direct Solution Approaches to Optimal Control

• Direct Collocation

• Direct Single Shooting

• Direct Multiple Shooting

Variational Approach to Optimal Control

• Euler-Lagrange Equations

• Sufficient Optimality Conditions

Pontryargin’s Maximum Principle • Input and State Constraints • Time-optimal Control • Singular Problems Indirect Solution Approaches to Optimal Control

Summary & Research Outlook

Overview of Numerical Methods for Optimal Control

Continuous Time Optimal Control

Hamilton-Jacobi-Bellman-Equation

• tabulation in state space • closed-loop opt. control

Indirect Methods

• Pontryargin’s Maximum

Principle (NCO of OCP)

Direct Solution Methods

• convert OCP into NLP

Direct Single Shooting

• only discretized controls in NLP

• sequential approach

Direct Collocation

• discretized states and controls

in NLP (collocation) • simultaneous approach

Direct Multiple Shooting

• discretized controls and node

start values in NLP • simultaneous approach

M. Diehl, H.G. Bock, H. Diedam, and P.B. Wieber. Fast Direct Multiple Shooting Algorithms for Optimal Robot Control. Lecture Notes in Control and Information Sciences, 340, 2006.

Optimal Control: A Primer on Direct Solutions Methods to Optimal Control Problems

Timm Faulwasser <[email protected]>

Laboratoire d’Automatique


22.04.2013

Overview

Computation of Function Derivatives • finite differences

• symbolic derivatives

• algorithmic differentiation

Computation of Functional Derivatives • finite differences

• sensitivity equations

• adjoint sensitivities

Direct Solution Approaches to Optimal Control • direct collocation

• direct single shooting

• direct multiple shooting

Summary

General OCP

Optimal Control Problem

subject to

Numerical solution procedures for OCPs?

content of this lecture

Overview of Numerical Methods for Optimal Control

Continuous Time Optimal Control

Hamilton-Jacobi-Bellman-Equation

• tabulation in state space • closed-loop opt. control

Indirect Methods

• Pontryargin’s Maximum

Principle (NCO of OCP)


• convert OCP into NLP




Direct Collocation







Main Idea:

First discretize then optimize!

• studied extensively over the last decades (30 years)

• successfully applied to many applications

• take advantage of powerful & fast NLP solvers

• applicable to ODEs, DAEs, and PDEs




Direct Collocation







22.04.2013

General OCP

Optimal Control Formulation

subject to subject to

Numerical solution procedures for OCPs?

Essential Tricks for Numerical Optimal Control

Free end time problems

use time transformation

end time is decision variable

Time-varying systems

extended state variable

Try to exploit problem structure!

Overview










Summary

Parametrization of Controls

Main Idea: describe control trajectory by finitely many parameters

Lagrange Interpolating Polynomials

• split time horizon

• in each interval approximate by polynomial function

• number of optimization variables on

• collocation points


22.04.2013

• Control constraints via bounds on

• Continuity of control can be enforced via inequality constraints

Parametrization of Controls Direct Collocation (Direct Simultaneous Method)

Main Idea:

OCP finite dimensional NLP via discretization of control and state variables

State Collocation

• via Lagrange Interpolation Polynomials

• time derivative

• discretized ODE

Direct Collocation (Direct Simultaneous Method)

Continuous Time OCP

Fully-discretized Problem

subject to

subject to


Exercise Discretize the following OCP into an NLP via collocation:

Use affine functions for state and inputs and 1 stage.


22.04.2013


Advantages and Disadvantages

• very large scale NLPs

• number of stages and collocation points has to be chosen apriori

• infeasible path method: – ODEs satisfied at convergence only computationally efficient, unstable systems doable

• stage times can be optimized too

• path constraints via inequality constraints at collocation points

Pseudospectral Methods:

• variant of orthogonal collocation with 1 stage and high-order polynomials

• used in software tools DIDO, PROPT-TOMLAB



• very large scale NLPs

• number of stages and collocation points has to be chosen apriori

• infeasible path method: – ODEs satisfied at convergence only computationally efficient, unstable systems doable

• stage times can be optimized too


Exercise: Sufficient conditions for the existence of optimal solution?

Overview










Summary

Direct Single Shooting (Direct Sequential Approach)

Idea:

discretize only the control variables

Original OCP

subject to


22.04.2013


OCP finite dimensional NLP with embedded ODEs

Idea:

discretize only the control variables

Original OCP

subject to

Partially Discretized Problem

subject to


Exercise Discretize the following OCP into an NLP via the direct sequential approach:

Use a piecewise-constant approximation for the controls.


Sketch of Numerical Solution Approach

Issues:

1. How to compute cost and constraint derivatives? ( Part III of this lecture)

2. How to handle constraints?

IN: OUT:

0.

i. Integrate ODEs

(system & sensitivities).

ii. Compute gradients.

iii. Solve NLP

iv. IF BREAK_COND = FALSE

GOTO i.

ELSE

How to handle path constraints?

reformulation as integral constraints

discretization as interior point constraints

Typically reformulation and discretization are combined/recommended.

Relaxation needed:

at a given set of points


22.04.2013

Direct Single Shooting (Direct Sequential Method)


• small scale NLPs (compared to collocation)

• the accuracy of state variable enforced by error-control of state-of-the art integrators

• feasible path method: – ODEs satisfied at each NLP iteration computationally more demanding, unstable systems can be

difficult


Exercise: Sufficient conditions for the existence of optimal solution?

Example: Dynamic Optimization

A Scalar Example • direct single shooting • piecewise constant controls • NLP solved with SQP methods • path constraint reformulated as






22.04.2013



Overview










Summary


Original OCP

subject to

Idea: • discretize the control variables • split integration into multiple stages with discontinuous initial conditions


OCP finite dimensional NLP with (multiple) embedded ODEs

Original OCP

subject to

Partially Discretized Problem

subject to

Idea: • discretize the control variables • split integration into multiple stages with discontinuous initial conditions

H.G. Bock and K.J. Plitt. A multiple shooting algorithm for direct solution of optimal control problems. In Proceedings 9th IFAC World Congress Budapest,

pages 243–247. Pergamon Press, 1984.


22.04.2013


Sketch of Numerical Solution Approach

Same procedure as single shooting but with multiple independent ODEs.

IN: OUT:

0.

i. Integrate ODEs

(system & sensitivities).

ii. Compute gradients.

iii. Solve NLP

iv. IF BREAK_COND = FALSE

GOTO i.

ELSE



• between sequential and simultaneous methods: sequential infeasible path methods

• the accuracy of state variable enforced by error-control of state-of-the art integrators

• can deal with unstable systems

• structure of the NLP should be exploited (special SQP solvers)

• can be regarded as a generalization of single shooting

Overview










Summary

Computation of Function Derivatives

Main Possibilities

• numeric differentiation

• symbolic differentiation

• algorithmic differentiation (automatic differentiation)

Motivation

• derivatives of objectives and constraints (gradients)

• sensitivities of ODE or DAEs

• Jacobians to solve implicit equations (e.g. implicit ODE solvers, DAEs, …)

x … states u … inputs


22.04.2013

Numerical Differentiation

Finite Differences • scalar function • finite difference approximation

computationally cheap How to choose increment h? • typically accuracy

Complex Step Derivative Approximation • scalar analytic function • finite difference approximation using complex increment

computationally cheap + precise (increments can be extremely small) • easy to implement in MATLAB (“imaginary trick”) • use transp instead of ‘ for transposed matrices and vectors J.R.R.A. Martins, P.Sturdza and J.J. Alonso. The Complex-Step Derivative Approximation. ACM Transactions on Mathematical Software (TOMS),

pages 245—262, 2003.

Symbolic Differentiation

Calculate derivatives symbolically • via computer algebra tools (Maple, Mathematica, Matlab). usually high precision easy to use often complex formulas for derivatives computationally expensive analytic formula needed

Example • Speelpennings function • gradient

needs n(n-1) operations

Algorithmic Differentiation (Automatic Differentiation)

Example (Forward mode) •

Main Idea • many functions = compositions of elementary functions (sin, cos, …) • derivatives of elementary functions are known

• chain rule, product rule, , … differentiation of computer code; efficient and accurate computation of derivatives

Details: • A. Griewank and A. Walther. Evaluating Derivatives. SIAM 2008.

• AutoDiff homepage www.autodiff.org

Overview










Summary


22.04.2013

Derivatives of Functionals

Functional in Mayer form subject to ODE

• How to compute value of functional?

use state-of-the-art ODE solvers

• How to compute the functional derivatives?

possible solutions

• finite differences

• forward sensitivity analysis

• adjoint sensitivity analysis

• algorithmic differentiation, …. (not covered here)

Derivatives of Functionals


Finite Differences

1. Solve ODEs for nominal parameter value 2. Loop for solve ODEs for perturbed parameter

Trade-off: Choose small but not too small.

Derivatives of Functionals: Forward Sensitivities

Forward Sensitivities •formal derivation state sensitivities needed

• 1 set of ODEs per parameter ODEs to be solved • independent of number of functionals


Derivatives of Functionals: Adjoint Sensitivities

Adjoint (Reverse) Sensitivities

• adjoint variables needed (backward integration )

• interpolation of states between mesh points • ODEs


22.04.2013

Derivatives of Functionals: Adjoint Sensitivities

Adjoint (Reverse) Sensitivities

• adjoint variables needed (backward integration )

• interpolation of states between mesh points • ODEs

Overview










Summary

Summary

Approaches to numerical optimal control:

Direct Methods

• direct collocation



Indirect Methods

• based on necessary conditions of optimality

later in the course

Hamilton-Jacobi-Bellman-Theory

• not covered

Software Tools for Numerical Optimal Control

Final question: How to solve your problems? • large variety of powerful software (free and commercial) to solve NLPs and optimal control

problems • useful web resource on optimization software http://plato.asu.edu/guide.html • web interface to NLP solvers http://www.neos-server.org/neos/ • understanding of underlying concepts is necessary to use these tools; especially if fast

solutions are required (predictive control) • indirect solutions methods are covered later in the course • further reading:

• script of this course (B. Chachuat) & script on numerical optimal control http://homes.esat.kuleuven.be/~mdiehl/TRENTO/numopticon.pdf

Toolboxes for Dynamic Optimization (Direct Solution Methods) • ACADO: direct multiple and single shooting, www.acadotoolkit.org/ • BOCOP : www.bocop.org • DIDO: pseudo-spectral methods http://www.elissarglobal.com/academic/products/ • HQP: direct multiple shooting http://hqp.sourceforge.net/ • MUSCOD-II : direct multiple shooting http://www.iwr.uni-heidelberg.de/~agbock/RESEARCH/muscod.php • PROPT TOMLAB: pseudo-spectral, commercial package • ….

22.04.2013 Dr. Timm Faulwasser, Laboratoire dAutomatique...

Documents

Transcript of 22.04.2013 Dr. Timm Faulwasser, Laboratoire dAutomatique...