IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense:...
Transcript of IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense:...
![Page 1: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/1.jpg)
IFT 6112BACKGROUND: OPTIMIZATION
http://www-labs.iro.umontreal.ca/~bmpix/teaching/6112/2018/
Mikhail Bessmeltsev
![Page 2: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/2.jpg)
Motivation
Quick summary!Mostly for common ground: You may already know this material.
First half is important; remainder summarizes interesting recent tools.
Numerical problems are everywhere in geometric modeling!
![Page 3: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/3.jpg)
Our Bias
Numerical analysis is a huge field.
Patterns, algorithms, & examples common in geometry.
![Page 4: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/4.jpg)
Rough Plan
• Linear problems
• Unconstrained optimization
• Equality-constrained optimization
• Variational problems
![Page 5: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/5.jpg)
Rough Plan
• Linear problems
• Unconstrained optimization
• Equality-constrained optimization
• Variational problems
![Page 6: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/6.jpg)
Vector Spaces and Linear Operators
![Page 7: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/7.jpg)
Abstract Example
Eigenvectors?
![Page 8: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/8.jpg)
In Finite Dimensions
![Page 9: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/9.jpg)
Linear System of Equations
Simple “inverse problem”
![Page 10: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/10.jpg)
Common Strategies
• Gaussian elimination– O(n3) time to solve Ax=b or to invert
• But: Inversion is unstable and slower!
• Never ever compute A-1 if you can avoid it.
![Page 11: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/11.jpg)
Simple Example
![Page 12: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/12.jpg)
Structure?
![Page 13: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/13.jpg)
Linear Solver Considerations
• Never construct 𝑨−𝟏 explicitly(if you can avoid it)
• Added structure helpsSparsity, symmetry, positive definiteness, bandedness
![Page 14: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/14.jpg)
Two Classes of Solvers
• Direct (explicit matrix)– Dense: Gaussian elimination/LU, QR for least-
squares– Sparse: Reordering (SuiteSparse, Eigen)
• Iterative (apply matrix repeatedly)– Positive definite: Conjugate gradients– Symmetric: MINRES, GMRES– Generic: LSQR
![Page 15: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/15.jpg)
Very Common: Sparsity
Induced by the connectivity of the triangle mesh.
Iteration of CG has local effect⇒ Precondition!
![Page 16: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/16.jpg)
For IFT 6112
• No need to implement a linear solver
• If a matrix is sparse, your code should store it as a sparse matrix!
https://eigen.tuxfamily.org/dox/group__TutorialSparse.html
![Page 17: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/17.jpg)
Optimization Terminology
Objective (“Energy Function”)
![Page 18: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/18.jpg)
Optimization Terminology
Equality Constraints
![Page 19: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/19.jpg)
Optimization Terminology
Inequality Constraints
![Page 20: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/20.jpg)
Encapsulates Many Problems
![Page 21: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/21.jpg)
Notions from Calculus
Gradienthttps://en.wikipedia.org/?title=Gradient
![Page 22: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/22.jpg)
Notions from Calculus
Gradienthttps://en.wikipedia.org/?title=Gradient
![Page 23: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/23.jpg)
Notions from Calculus
Jacobianhttps://en.wikipedia.org/wiki/Jacobian_matrix_and_determinant
![Page 24: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/24.jpg)
Notions from Calculus
Hessianhttp://math.etsu.edu/multicalc/prealpha/Chap2/Chap2-5/10-3a-t3.gif
![Page 25: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/25.jpg)
Optimization to Root-Finding
Critical point
(unconstrained)
Saddle point
Local min
Local max
![Page 26: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/26.jpg)
Convex Functions
https://en.wikipedia.org/wiki/Convex_function
𝑓′′ 𝑥 > 0
![Page 27: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/27.jpg)
Convex Functions
https://en.wikipedia.org/wiki/Convex_function
𝐻 𝑥 ≽0
![Page 28: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/28.jpg)
Generic tools are often not too effective!
![Page 29: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/29.jpg)
Generic Advice
Try the
simplest solver first.
![Page 30: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/30.jpg)
Rough Plan
• Linear problems
• Unconstrained optimization
• Equality-constrained optimization
• Variational problems
![Page 31: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/31.jpg)
Unconstrained optimization
min𝑥∈ℝ𝑛
𝑓(𝑥)
Trivial when 𝑓 𝑥 is linear
Easy when 𝑓(𝑥) is quadratic
Hard in case of generic non-linear.
![Page 32: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/32.jpg)
Special Case: Least-Squares
Normal equations(better solvers for this case!)
![Page 33: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/33.jpg)
Useful Document
http://www2.imm.dtu.dk/pubdb/views/edoc_download.php/3274/pdf/imm3274.pdf
The Matrix CookbookPetersen and Pedersen
![Page 34: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/34.jpg)
Example: Mesh Embedding
G. Peyré, mesh processing course slides
![Page 35: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/35.jpg)
Linear Solve for Embedding
• 𝒘𝒊𝒋 ≡ 𝟏: Tutte embedding• 𝒘𝒊𝒋 from mesh: Harmonic embedding
Assumption: 𝒘 symmetric.
![Page 36: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/36.jpg)
Rough Plan
• Linear problems
• Unconstrained optimization
• Equality-constrained optimization
• Variational problems
![Page 37: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/37.jpg)
Unconstrained Optimization
Unstructured.
![Page 38: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/38.jpg)
Basic Algorithms
Gradient descent
Line search
Multiple optima!
![Page 39: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/39.jpg)
Basic Algorithms
Accelerated gradient descent
Quadratic convergence on convex problems!(Nesterov 1983)
![Page 40: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/40.jpg)
Basic Algorithms
Newton’s Method
1
2
3
Line search for stability
![Page 41: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/41.jpg)
Basic Algorithms
Quasi-Newton: BFGS and friends
Hessian approximation
• (Often sparse) approximation from previous samples and gradients
• Inverse in closed form!
![Page 42: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/42.jpg)
Example: Shape Interpolation
Fröhlich and Botsch. “Example-Driven Deformations Based on Discrete Shells.” CGF 2011.
![Page 43: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/43.jpg)
Interpolation Pipeline
Roughly:
1. Linearly interpolate edge lengths and dihedral angles.
2. Nonlinear optimization for vertex positions.
Sum of squares: Gauss-Newton
![Page 44: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/44.jpg)
Software
• Matlab: fminunc or minfunc• C++: libLBFGS, dlib, others
Typically provide functions for functionand gradient (and optionally, Hessian).
Try several!
![Page 45: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/45.jpg)
Some Tricks
Regularization
![Page 46: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/46.jpg)
Some Tricks
Multiscale/graduated optimization
![Page 47: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/47.jpg)
Rough Plan
• Linear problems
• Unconstrained optimization
• Equality-constrained optimization
• Variational problems
![Page 48: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/48.jpg)
Lagrange Multipliers: Idea
![Page 49: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/49.jpg)
Lagrange Multipliers: Idea
- Decrease f: −𝛁𝒇- Violate constraint: ±𝛁𝒈
![Page 50: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/50.jpg)
Lagrange Multipliers: Idea
Want:
![Page 51: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/51.jpg)
Use of Lagrange Multipliers
Turns constrained optimization into
unconstrained root-finding.
![Page 52: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/52.jpg)
Quadratic with Linear Equality
(assume A is symmetric and positive definite)
![Page 53: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/53.jpg)
Many Options
• ReparameterizationEliminate constraints to reduce to unconstrained case
• Newton’s methodApproximation: quadratic function with linear constraint
• Penalty methodAugment objective with barrier term, e.g. 𝒇 𝒙 + 𝝆|𝒈 𝒙 |
![Page 54: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/54.jpg)
Example: Symmetric Eigenvectors
![Page 55: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/55.jpg)
Returning to Parameterization
What if 𝑽𝟎 = {}?
![Page 56: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/56.jpg)
Nontriviality Constraint
Prevents trivial solution 𝒙 ≡ 𝟎.
Extract the smallest eigenvalue.
![Page 57: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/57.jpg)
Back to Parameterization
Mullen et al. “Spectral Conformal Parameterization.” SGP 2008.
Easy fix
![Page 58: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/58.jpg)
Basic Idea of Eigenalgorithms
![Page 59: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/59.jpg)
Trust Region Methods
Example: Levenberg-Marquardt
Fix (or adjust) damping
parameter 𝝀 > 𝟎.
![Page 60: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/60.jpg)
Example: Polycube Maps
Huang et al. “L1-Based Construction of Polycube Maps from Complex Shapes.” TOG 2014.
Align with coordinate axes
Preserve areaNote: Final method includes more terms!
![Page 61: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/61.jpg)
Convex Optimization Tools
Try lightweight options
versus
![Page 62: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/62.jpg)
Iteratively Reweighted Least Squares
Repeatedly solve linear systems
“Geometric median”
![Page 63: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/63.jpg)
Alternating Projection
d can be aBregman
divergence
![Page 64: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/64.jpg)
Augmented Lagrangians
Add constraint to objective
Does nothing when constraint is satisfied
![Page 65: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/65.jpg)
Alternating DirectionMethod of Multipliers (ADMM)
https://web.stanford.edu/~boyd/papers/pdf/admm_slides.pdf
![Page 66: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/66.jpg)
Frank-Wolfe
https://en.wikipedia.org/wiki/Frank%E2%80%93Wolfe_algorithm
Linearize objective, preserve constraints
![Page 67: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/67.jpg)
Rough Plan
• Linear problems
• Unconstrained optimization
• Equality-constrained optimization
• Variational problems
![Page 68: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/68.jpg)
Variational Calculus: Big Idea
Sometimes your unknowns
are not numbers!Can we use calculus to optimize anyway?
![Page 69: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/69.jpg)
On the Board
![Page 70: IFT 6112 BACKGROUND: OPTIMIZATION · Two Classes of Solvers •Direct (explicit matrix) –Dense: Gaussian elimination/LU, QR for least- squares –Sparse: Reordering (SuiteSparse,](https://reader035.fdocuments.in/reader035/viewer/2022071006/5fc3afeb32ebca6c225a6491/html5/thumbnails/70.jpg)
Gâteaux Derivative
Analog of derivative at u in ψ direction
Vanishes for all ψ at a critical point!