Method of Lagrange Multipliersmgormley/courses/606-607-f18/slides606/... · 2019-01-11 · Method...
18
Method of Lagrange Multipliers 1 10-606 Mathematical Foundations for Machine Learning Matt Gormley Lecture 10 Oct. 1, 2018 Machine Learning Department School of Computer Science Carnegie Mellon University
Transcript of Method of Lagrange Multipliersmgormley/courses/606-607-f18/slides606/... · 2019-01-11 · Method...
Method of Lagrange Multipliers
1
10-606 Mathematical Foundations for Machine Learning
Matt Gormley
Lecture 10
Oct. 1, 2018
Machine Learning Department
School of Computer Science
Carnegie Mellon University
Reminders
• Homework 2: Linear Algebra + Calculus
– Out: Tue, Sep. 25
– Due: Tue, Oct. 2 at 11:59pm
• Quiz 1: Linear Algebra
– In-class, Wed, Oct. 3
• Homework 3: Calculus + Probability
– Out: Wed, Oct. 3
– Due: Wed, Oct. 10 at 11:59pm
• Quiz 2: Matrix Calculus + Probability
– In-class, Wed, Oct. 10
3
Q&A
4
MATRIX CALCULUS
6
Method of Lagrange Multipliers
Chalkboard– Motivation: Constrained Optimization
– Method of Lagrange Multipliers
– Extending to multiple constraints
– Extending to inequality constraints
7
Method of Lagrange Multipliers
8
Method of Lagrange Multipliers
9
Method of Lagrange Multipliers
10
Method of Lagrange Multipliers
11
Method of Lagrange Multipliers
12
Method of Lagrange Multipliers
13
Figure from http://tutorial.math.lamar.edu/Classes/CalcIII/LagrangeMultipliers.aspx
Method of Lagrange Multipliers
14
Figure from http://tutorial.math.lamar.edu/Classes/CalcIII/LagrangeMultipliers.aspx
DIMENSIONALITY REDUCTION
15
PCA Outline
• Dimensionality Reduction– High-dimensional data
– Learning (low dimensional) representations
• Principal Component Analysis (PCA)– Examples: 2D and 3D
– Data for PCA
– PCA Definition
– Objective functions for PCA
– PCA, Eigenvectors, and Eigenvalues
– Algorithms for finding Eigenvectors /
Eigenvalues
• PCA Examples– Face Recognition
– Image Compression
16
High Dimension Data
Examples of high dimensional data:
– High resolution images (millions of pixels)
17
High Dimension Data
Examples of high dimensional data:
– Multilingual News Stories
(vocabulary of hundreds of thousands of words)
18
High Dimension Data
Examples of high dimensional data:
– Brain Imaging Data (100s of MBs per scan)
19
Image from https://pixabay.com/en/brain-mrt-magnetic-resonance-imaging-1728449/
Image from (Wehbe et al., 2014)
High Dimension Data
Examples of high dimensional data:
– Customer Purchase Data
20
