LMI BMI UFO Presentation 2013

Seminar:LP, QP, SOCP, SDP, LMI, BMI and other UFOs

(not-Dr.-yet) Mikhail V. KonnikSchool of Electrical Engineering and Computer Science

The University of NewcastleAustralia

December 13, 2013

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle Australia ()Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOsDecember 13, 2013 1 / 42

Disclaimer

the Author is not an expert (yet) in LMI/BMI/SDP (but he is doing hisbest to become one), and therefore:

the whole presentation is just an overview of LMI/BMI from the point ofview of the Author's (severely incomplete) understanding of optimisation;the Author is not ready to solve complicated BMI/UFO problems thatthe people in the audience surely have :-)

the problem of solving even LMI (let alone BMI) is much more complicatedthan many people in the audience think;

software packages for LMI/BMI are not as mature as for e.g. QP;the Author is known for his a�nity to �reinvent the wheel� to betterunderstand how stu� works.

The wheel that is being inventing by the Author is still in the blueprintstage, is not round, and cannot be used for anything remotely serious.

the Author is not liable for any loss, damage, illness, injury, headache,crash, collapse of the Universe or anything else caused by this presentation.

proceed at your own risk!


Introduction to Linear Matrix Inequalities

Part I ::A short introduction to Linear Matrix Inequalities(LMI) and Semide�nite Programming (SDP)


Introduction to Linear Matrix Inequalities LMI and Control

Linear Matrix Inequalities and Control

The story of LMI begins in about 1890, when Lyapunov published his seminalwork, where he showed that the di�erential equation

d

dxx(t) = Ax(t)

is stable (i.e., all trajectories converge to zero) if and only if there exists apositive-de�nite matrix P such that[1]

ATP + PA ≺ 0

The requirement P � 0 is what we now call a Lyapunov inequality on P .


Introduction to Linear Matrix Inequalities LMI and Control

Linear Matrix Inequalities and Control

The Lyapunov inequality ATP +PA < 0 is not the only example of how onecan convert control problems into LMI: Riccati inequality can be convertedinto an LMI, too:

ATP + PA+ PBR−1BTP + Q ≺ 0,

is equivalent to the following LMI (via Schur complement):[−ATP − PA− Q PB

BTP R

]≺ 0 (1)

The requirement P � 0 is what we now call a Lyapunov inequality on P .


Introduction to Linear Matrix Inequalities What is an LMI?

What is an LMI?

A (strict) linear matrix inequality (LMI) is a convex constraint in the form:

F (x) = F0 + x1F1 + · · ·+ xNFN = F0 +N∑i=1

xiFi � 0, (2)

where:

x = [x1, . . . , xN ] is a vector of unknown scalars (the decisionor optimisation variables) x1, x2 and so on;

F0,F1,F2, . . . ,FN are n × n symmetric matrices, they are given.

The sign F ≺ 0 is a generalised inequality meaning F is a negative de�nite

matrix (i.e., the largest eigenvalue of F (x) is negative), F � 0 is positivede�nite, and F � 0 is positive semi-de�nite.


Introduction to Linear Matrix Inequalities What is an LMI?

What is an LMI? - A set of CONVEX constraints!

Figure 1: Supporting hyperplanes for LMI.

Note that the LMI:

F (x) = F0 + x1F1 + . . .

+xNFN = F0 +N∑i=1

xiFi � 0,

is just a convex constraint

on x , and therefore:

Its solution set, calledthe feasible set, is aconvex subset of RN

Finding a solution x ofLMI, if any, is a convexproblem.

Convexity has an important consequence: even though LMI has no analyticalsolution in general, it can be solved numerically.


Introduction to Linear Matrix Inequalities Solving Linear Matrix Inequalities in a brute-force way

Solving LMI in a brute-force way


Introduction to Linear Matrix Inequalities Solving Linear Matrix Inequalities in a brute-force way

Solving LMI in a brute-force way

Figure 2: The corresponding set from the LMIF (x) := x1F1 + x2F2 � I .

The LMI speci�es a convexconstraint on x , and to�solve� an LMI means thatwe can �nd x that satis�esthe LMI.

Brute-force: we can try thevariables in

X =

[x1 x2x2 x3

]computeZ = ATX + XA

check eigenvaluesE = eig(Z )


Semide�nite Programming

We can do better: solve LMI via SDP

A problem which subsumes linear, quadratic,geometric and second-order cone programmingis called a semide�nite program (SDP):

minimizeX

CTX <- - this is optimality metric!

subject to F0 +N∑i=1

xiFi � 0, <- - this is LMI!

Ax = b

where the matrices F0,F1, . . .Fn ∈ Sk , and A ∈ Rp×n.

That is: LMI cuts a set of solutions via convex set of constraints, and SDP

allows to pick up an �optimal� solution using some optimality metric.

To solve LMI ⇔ solve a Semide�nite Programming (SDP) problem.(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle Australia ()Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOsDecember 13, 2013 10 / 42


What is Semide�nite?

The set of real symmetric n×n matricesis denoted Sn.

A matrix Z ∈ Sn is called positivesemide�nite if xTZx ≥ 0 for all x ∈Rn. On the other hand, a matrix Z iscalled positive de�nite if xTZx > 0for all nonzero x ∈ Rn.

The set of positive semide�nite matricesis denoted Sn+. In other words, the eigenvalues λ1 . . . λn of a positivesemide�nite matrix Z+ are nonnegative (i.e., can be zeros or less thanmachine epsilon ε).

The set Sn+ is a convex cone.



What is Semide�nite? (continued)

A semide�nite program (SDP) is a generalization of a linear program (LP),where the inequality constraints are replaced by matrix inequalities correspondingto the cone of positive semide�nite matrices.An SDP in the pure primal form is de�ned as:

minimiseX

trace(CX )

subject to trace(AiX ) = bi ∀i = 1, . . . ,m

X � 0.

(3)

where X ∈ Sn+ is the decision variable (can be a vector or a matrix), b ∈ Rm

and C ,A1 . . .Am ∈ Sn+ are given symmetric matrices. There are m a�neconstraints among the entries of the positive semide�nite matrix X .

Unlike LP, where the feasible sets are polyhedra, the feasible sets of SDPscan have curved faces, together with sharp corners where faces intersect.


Semide�nite Programming Scope of Semide�nite Programming

Scope of SDP

SDP:

minimiseX

CTX

subject to F0 +N∑i=1

xiFi ≺ 0 <- - - LMI

SOCP:

minimisex

cT x

subject to ‖Aix + bi‖2 < cTi x + di

Fx = g

QP:minimise

x

1

2xTHx + cT x + d

subject to Ax < b

LP:minimise

xcT x + d

subject to Ax < b


Semide�nite Programming SDP: One Ring to Rule them All

Semide�nite Programming SDP: One Ring to Rule them All

Using the power of the Sauron's One Ring SDP, re-cast a QP problem intoan SDP problem. Use Cholesky factor H = QTQ of the Hessian matrix H.

minimisex

xTQTQx + cT x + r

subject to Ax ≥ b

Equivalent SDP:minimise

XCTX

subject to F (x) � 0

where F (x) := blockdiag(FQ(x),Aix − bi ),

Ai is the ith row of A, and we use Schur complement r ≥ xTQTQx + cT x ,

FQ(x) =

[r − cT x xTQT

Qx I

]� 0.


Summary for SDP and LMI

Summary for SDP and LMI

1 A (strict) linear matrix inequality (LMI) is a convex constraint in theform:

F0 +N∑i=1

xiFi � 0.

2 Finding a solution x of LMI, if any, is a convex problem.

3 To solve LMI ⇔ solve a Semide�nite Programming (SDP):

minimizex

CTX <- - this is an optimality metric!

subject to x1F1 + · · ·+ xnFn + F0 � 0 <- - this is LMI!

Ax = b


4 LMI cuts a set of solutions via convex set of constraints, and SDP allowsto pick up an �optimal� solution using some optimality metric.


Solving convex SDPs and LMIs

Part II ::Solving Semide�nite Problems (SDP)


Solving convex SDPs and LMIs Analytical techniques and numerical algorithms

Solving convex SDPs and LMIs : analytic techniques andnumerical algorithms

Even small Semide�nite problems (and LMIs) are too di�cult to solveanalytically - use numerical methods!

1 Primal-dual Interior point methods

2 Logarithmic barrier methods

3 Augmented Lagrangian

There are some analytical methods that can help/simplify/relax the problem:

1 Schur complement - convert an LMI into a matrix

2 S-Procedure - a Lagrange relaxation technique for problems with quadraticconstraints (solves a system of quadratic inequalities via LMI).

3 The Elimination Lemma - eliminates some variables in LMI.


Solving convex SDPs and LMIs Schur complement

Analytical techniques: Schur complement

Schur complement is an indispensable tool for transforming non-linearconstraints into convex LMI. A symmetric matrix X can be decomposed:

X =

[A B

BT C

](4)

The matrix S , C −BTA−1B is called the Schur complement of A in X .

In other words, for all X ∈ Sn, Y ∈ Rm×n , Z ∈ Sm , the followingstatements are equivalent:

a) Z � 0;X − Y TZ−1Y � 0.

b)

[X Y T

Y Z

]� 0.

a) Z � 0;X − Y TZ−1Y � 0.

b) Z � 0 ;

[X Y T

Y Z

]� 0.


Solving convex SDPs and LMIs S-procedure

Analytical techniques: S-procedure for Quadratic Forms andStrict Inequalities

In some problems, we �nd that some quadratic function must be negativewhenever some other quadratic functions are all negative.

With the S-procedure, we can replace this problem by one inequalityto be satis�ed by introducing some positive scalars to be determined.

Motivation: The fundamental question of the theory of the S-lemma is thefollowing: ([2], page: 371).

When is a quadratic inequality a consequence of other quadratic

inequalities?

In short terms, the S-procedure is a Lagrange relaxation method; it triesto solve a system of quadratic inequalities via LMI.



Analytical techniques: S-procedure for Quadratic Forms andStrict Inequalities

Let T0 . . .Tp ∈ Rn×n be symmetric matrices. We consider the followingcondition on T0 . . .Tp:

ξTT0ξ > 0 for all ξ 6= 0 such that ξTTiξ ≥ 0 (5)

It is obvious that if there exists τ1 ≥ 0, . . . τp ≥ 0 such that:

T0 −p∑i=1

τiTi � 0,

then (5) holds.For further details and bibliography comments, the reader is referred to [3].*Please do NOT embarrass the Author with questions :-)



Analytical techniques: S-procedure, continued - anillustrative example

The following constraint on the variable P : ([3], page: 36)

∀ξ 6= 0 and π satisfying πTπ ≤ ξTCTCξ, (6)[ξπ

]T [ATP + PA PB

BTP 0

] [ξπ

]< 0

Applying the S-procedure, (6) is equivalent to:[ATP + PA+ τCTC PB

BTP −τ I

]< 0

Thus the problem of �nding P > 0 such that (6) holds can be expressed asan LMI in P and the scalar variable τ . ([3], page: 36)*Please do NOT embarrass the Author with questions here ;-)


Solving convex SDPs and LMIs The Elimination Lemma

Analytical techniques: S-procedure, continued - anillustrative example

The Elimination Lemma [3] allows to eliminate some variables appearing inLMI leading to inequalities without the eliminated variable. Consider:

G (z) + U(z)XV (z)T + V (z)XTU(z)T > 0, (7)

Suppose that for every z , U(z) and V (z) are orthogonal complements ofU(z) and V (z) respectively. Then (7) holds for some X and z = z0 if andonly if the inequalities

U(z)TG (z)U(z) > 0, V (z)TG (z)V (z) > 0, (8)

hold with z = z0.

In other words, feasibility of the matrix inequality (7) with variables X andz is equivalent to the feasibility of (8) with variable z .


Solving convex SDPs and LMIs Numerical algorithms for convex SDP and LMI

Numerical algorithms for convex SDP and LMI

Schur complement, S-procedure, Elimination Lemma and other analyticaltricks will NOT solve LMIs - use numerical methods!

1 Primal-dual Interior point methods

2 Logarithmic barrier methods

3 Augmented Lagrangian

LMIs are usually solved as SDP problems; therefore it is bene�cial to studyhow the SDP solvers work.*The Author promises NOT to embarrass the Audience with details here ;-)


Solving convex SDPs and LMIs Canonical and matrix forms of LMIs

Numerical algorithms for convex SDP and LMI: Canonicaland matrix forms of LMIs

Generally, we do not encounter the LMI in the canonical form in controltheory but rather in the form of matrix variables. Thus, before the solutionof LMI commences via SDP, the LMI must be pre-processed, or parsed.

For example, the Lyapunov's inequality:

ATP + PA ≺ 0 P = PT � 0 (9)

can be written in the canonical form:

F (x) = F0 +m∑i=1

xiFi � 0, (10)

where F0 = 0 and Fi = −ATBi − BiA, and where Bi , i = 1, . . . , n(n+ 1)/2are matrix bases for symmetric matrices of size n.


Solving convex SDPs and LMIs Canonical and matrix forms of LMIs

Canonical form of LMI

Consider the Lyapunov inequality in matrix form: ATX + XA � 0

where A =

[−1 20 −2

]and X =

[x1 x2x2 x3

]. The decision variables are

scalars x1 , x2 , and x3 of the matrix X .Convert to the canonical form that is F (x) = F0 + x1F1 + · · ·+ xNFN :

ATX + XA→ · · · →[−2x1 2x1 − 3x2

2x1 − 3x2 4x2 − 4x3

]� 0

Now we extract the coe�cients for each variable x1, x2 and x3:

[−2x1 2x1 − 3x2

2x1 − 3x2 4x2 − 4x3

]→ x1·

[−2 22 0

]︸︷︷︸matrix F1

+x2·[

0 −3−3 4


+x3·[0 00 −4


Therefore, we converted the problem into LMI x1F1 + x2F2 + x3F3 � 0.


Solving convex SDPs and LMIs Software toolboxes for numerical solution of SDP and LMI

Software toolboxes for numerical solution of SDP and LMI

Parsers:

1 YALMIP - yet anotherLMI parser

2 CVX - supports twosolvers: SeDuMi andSDPT3

Please note that both YALMIPand CVX are parsers, thatis, they don't solve the actualproblem; instead, they rely onother solvers such as SeDuMi,SDP3 and PENSDP.

Solvers:

1 SDPA - Mehrotra-typepredictor-corrector infeasibleprimal-dual interior-point

2 SDPT3 - infeasible path-following algorithm

3 SeDuMi - Self-Dual-Minimization, self-dualembedding technique, InteriorPoint

4 LMI Control Toolbox - LMILab, Nemirovskii's projectivealgorithm



Author's success story: Kalman gains via LMI

Denote the state estimation error as x = x − x and the estimation errore = z − z . The overall system's dynamics:

x(k + 1) = (A− KCy )x(k) +(Bw − KDyw

)w(k) (11)

e(k) = Cz x(k) + Dzww(k)

Write the SDP for the state estimator synthesis:

minY ,X ,P

trace(X )

subject to

P PA− YCy PB − YDyw

(PA− YCy )T P 0

(PB − YDyw )T 0 Im

� 0

[X CzCTz P

]� 0

P � 0 and Y = PK , X = P−1



Author's success story: Kalman gains via LMI

An LMI for Kalman gains for a small 2 × 2 system was solved using theSeDuMi solver and the YALMIP parser interface:

X = sdpvar(2,2); Dyw = [0 0 1 0; 0 0 0 1];

Y = sdpvar(2,2); Cz = [1 0; 0 1];

P = sdpvar(2,2); Dzw = 0; Cy = C_k;

Bw = [0 0 0 0; ...

0 0.01 0 0];

Fset=set(P>0) + set( [P, P*A_k-Y*Cy, P*Bw - Y*Dyw;...

(P*A_k-Y*Cy)', P, zeros(2,4);...

(P*Bw - Y*Dyw)', zeros(4,2), eye(4)]>0 ) +...

set([X,Cz; Cz', P]>0);

sol = solvesdp(Fset, trace(X));

The matrix of gains found via LMI is, of course, the same (up to numericalerrors) as the one found via Riccati equations.K = new_X*new_Y


Summary for methods of solving SDPs

Summary for methods of solving SDPs

1 There are analytical tools that help to simplify an SDP/LMI problem:Schur complement, S-Procedure, Elimination Lemma.

2 Even small Semide�nite problems (and LMIs) are too di�cult to solveanalytically - use numerical methods!

3 Good news: convex SDPs (LMIs) are solvable within reasonable timeusing numerical methods (Interior point, Logarithmic barrier, AugmentedLagrangian).

4 To convert an LMI from the matrix form into semide�nite (canonical)form, we need parsers (YALMIP, CVX).

5 To solve an LMI/SDP, we need solvers (SDPA, SDPT3, SeDuMi, LMIControl Toolbox).

6 Author's success story: using YALMIP and SeDuMi, he solved an LMIfor Kalman gains for a small 2× 2 system.


Solving non-convex SDPs and BMIs

Part III ::Solving non-convex Semide�nite Problems and

Bilinear Matrix Inequalities (BMIs)


Introduction to Bilinear Matrix Inequalities

Bilinear Matrix Inequalities

While a Linear matrix inequality (LMI) denotes a constraint of the form:

F (x) = F0 +n∑i=1

xiFi � 0, (12)

where Fi are �xed symmetric matrices and x ∈ Rn is the decision variable,a Bilinear Matrix Inequality constraints are in the form:

F (x) = F0 +n∑i=1

xiFi +n∑j=1

n∑j=1

Fi ,jxixj � 0, (13)

are denoted BMIs (bilinear matrix inequalities).

Bilinear Matrix Inequalities are in the realm of non-convex Semide�niteProgramming.



Bilinear Matrix Inequalities: a Challenge

Optimisation problems with BMIs are known to be non-convex andNP-hard in general[4], hence intractable in theory.

BMIs correspond to non-convex and (possibly) non-smooth Semide�niteProgramming - this is global optimisation (i.e., exponential time, localsolutions...);

no reliable solver exists for non-convex SDP problems: branch-and-bound, genetic algorithms, spectral bundle method, barrier methods -no guarantee to �nd a (global) solution within reasonable time.

numerical algorithms are inhumanly di�cult to devise: only few exist.



Bilinear Matrix Inequalities: an Illustration

Below is an illustration of the non-convex quadratic problem with linearobjective function, which is an example of a BMI problem.

Figure 3: Illustration of a BMI: a non-convex feasible set delimited by circularand hyperbolic arcs (adapted from[5]).

maxx

x2

subject to 3− 2x2 − x21 − x22 ≥ 0

− x1 − x2 − x1x2 ≥ 0

1+ x2x1 ≥ 0


Introduction to Bilinear Matrix Inequalities Numerical algorithms for non-convex SDPs and BMIs

Numerical algorithms for non-convex SDPs and BMIs

Don't even try to solve BMIs analytically - it is di�cult even for numericalmethods of global optimisation (non-convex SDPs):

1 Genetic algorithms: Read-coded Genetic Algorithms, Extrapolation-directed Crossover (EDX), Minimal Generation Gap (MGG);

2 Branch-and-Bound, Branch-and-Cut methods (have troubles with solvingmedium/large problems due to loose lower-bound approximations);

3 Coordinate-descent method (BMI problem is solved independently foreach coordinate at each step using a LMI optimisation)

4 spectral Bundle method;

5 attempts to use Augmented Lagrangian / Barrier methods.

*The Author promises NOT to embarrass the Audience with details here ;-)


Introduction to Bilinear Matrix InequalitiesSoftware toolboxes for numerical solution of non-convex

SDPs and BMIs

Software toolboxes for numerical solution of non-convexSDPs and BMIs

Parsers:

1 YALMIP - yet anotherLMI parser

Solvers:

1 PENBMI - penalty/barrierfunction (AugmentedLagrangian?)

2 HIFOO - hybrid method(quasi-Newton updating,bundling and gradientsampling)


Summary for methods of solving non-convex SDPs andBMIs

Summary for methods of solving non-convexSDPs and BMIs

1 Bilinear Matrix Inequalities are in the realm of non-convex Semide�niteProgramming = Global Optimisation.

2 BMIs are known to be non-convex and NP-hard in general, henceintractable in theory.

3 no reliable solver exists for non-convex SDP problems: branch-and-bound, simulated annealing, genetic algorithms, spectral bundle method,barrier methods - no guarantee to �nd a (global) solution within reasonabletime.

4 numerical algorithms are inhumanly di�cult to devise, only few exist:PENBMI and HIFOO (current on 2013).


Conclusion and Summary

Part IV ::Conclusion and Summary


Conclusion: Linear Matrix Inequalities / convex SDPs

Conclusion: Bilinear Matrix Inequalities /non-convex SDPs

1 a Linear Matrix Inequality is a convex constraint in the form:

F0 +N∑i=1

xiFi � 0.

2 In mathematical programming terminology, to solve LMI means tosolve a Semide�nite Programming (SDP) problem:

minimizex

CTX <- - this is an optimality metric!

subject to x1F1 + · · ·+ xnFn + F0 � 0 <- - this is LMI!

Ax = b



Conclusion: Linear Matrix Inequalities / convex SDPs

Conclusion: Bilinear Matrix Inequalities /non-convex SDPs, continued

1 There are analytical tools that help to simplify an SDP/LMI problem:Schur complement (most useful), S-Procedure, Elimination Lemma.

2 convex SDPs (LMIs) can be solved within reasonable time using well-developed numerical methods (Primal-dual Interior point, Logarithmicbarrier, Augmented Lagrangian).

3 To convert an LMI from the matrix form into semide�nite (canonical)form, we need parsers (YALMIP, CVX).

4 To solve an LMI/SDP, we need solvers (SDPA, SDPT3, SeDuMi, LMIControl Toolbox).

5 Author's success story: using YALMIP and SeDuMi, he solved an LMIfor Kalman gains for a small 2× 2 system.


Conclusion: Bilinear Matrix Inequalities / non-convex SDPs

Conclusion: Bilinear Matrix Inequalities /non-convex SDPs

1 a Bilinear Matrix Inequality constraints are in the form:

F (x) = F0 +n∑i=1

xiFi +n∑j=1

n∑j=1

Fi ,jxixj � 0,

2 BMIs are in the realm of non-convex Semide�nite Programming.

3 BMIs = non-convex SDPs, which are known to beNP-hard in general,hence intractable in theory.

4 no reliable solver exists for non-convex SDPs: branch-and-bound, geneticalgorithms, barrier methods - no guarantee to �nd a (global) solutionwithin reasonable time.

5 numerical algorithms are di�cult to devise, only few toolboxes exist.


Conclusion: Bilinear Matrix Inequalities / non-convex SDPs

The End

- [Frodo:] I will take the Ring to Mordor![pause]- [Frodo:] Though... I do not know the way.

Frodo Baggins

from The Lord of the Rings: The Fellowship of the Ring


Conclusion: Bilinear Matrix Inequalities / non-convex SDPs Convex set of LMI constraints

Convex set of LMI constraints

Figure 4: Supporting hyperplanes for LMI.

Let us denote by Xthe set of points x ∈Rm that satisfy:

X :=

{x : F0 +

m∑i=1

xiFi � 0

}.

The set X is convexsince we haveF (x) � 0 if and onlyif ∀ : z ∈ Rn :zTF (x)z ≥ 0.


Conclusion: Bilinear Matrix Inequalities / non-convex SDPs Convex set of LMI constraints

Stephen Boyd, V Balakrishnan, E Feron, and Laurent El Ghaoui.

History of linear matrix inequalities in control theory.In American Control Conference, 1994, volume 1, pages 31�34. IEEE, 1994.

Imre Pólik and Tamás Terlaky.

A survey of the s-lemma.SIAM review, 49(3):371�418, 2007.

Stephen Boyd, Laurent El Ghaoui, Eric Feron, and Venkataramanan Balakrishnan.

Linear matrix inequalities in system and control theory, volume 15.SIAM, 1994.

Onur Toker and Hitay Ozbay.

On the np-hardness of solving bilinear matrix inequalities and simultaneous stabilization with static outputfeedback.In American Control Conference, 1995. Proceedings of the, volume 4, pages 2525�2526. IEEE, 1995.

Didier Henrion.

Course on lmi: What is an lmi?Technical report, www.laas.fr/~henrion, October 2006.


LMI BMI UFO Presentation 2013

Education

Transcript of LMI BMI UFO Presentation 2013