Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for...

33
Lecture 08 Subgradients & the subgradient method

Transcript of Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for...

Page 1: Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for convex and differentiable f, f(y) f(x) for all I.e., linear approximation always

Lecture 08Subgradients &

the subgradient method

Page 2: Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for convex and differentiable f, f(y) f(x) for all I.e., linear approximation always

2

Page 3: Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for convex and differentiable f, f(y) f(x) for all I.e., linear approximation always

3

Page 4: Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for convex and differentiable f, f(y) f(x) for all I.e., linear approximation always

4

Page 5: Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for convex and differentiable f, f(y) f(x) for all I.e., linear approximation always

5

𝑥2𝑥1

Page 6: Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for convex and differentiable f, f(y) f(x) for all I.e., linear approximation always

6

Page 7: Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for convex and differentiable f, f(y) f(x) for all I.e., linear approximation always

7

Page 8: Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for convex and differentiable f, f(y) f(x) for all I.e., linear approximation always

8

Page 9: Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for convex and differentiable f, f(y) f(x) for all I.e., linear approximation always

9

Page 10: Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for convex and differentiable f, f(y) f(x) for all I.e., linear approximation always

10

Page 11: Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for convex and differentiable f, f(y) f(x) for all I.e., linear approximation always

11

Page 12: Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for convex and differentiable f, f(y) f(x) for all I.e., linear approximation always

12

Page 13: Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for convex and differentiable f, f(y) f(x) for all I.e., linear approximation always

13

Page 14: Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for convex and differentiable f, f(y) f(x) for all I.e., linear approximation always

14

Page 15: Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for convex and differentiable f, f(y) f(x) for all I.e., linear approximation always

15

Page 16: Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for convex and differentiable f, f(y) f(x) for all I.e., linear approximation always

16

Page 17: Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for convex and differentiable f, f(y) f(x) for all I.e., linear approximation always

17

Page 18: Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for convex and differentiable f, f(y) f(x) for all I.e., linear approximation always

18

𝜃

Page 19: Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for convex and differentiable f, f(y) f(x) for all I.e., linear approximation always

19

Page 20: Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for convex and differentiable f, f(y) f(x) for all I.e., linear approximation always

20

Page 21: Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for convex and differentiable f, f(y) f(x) for all I.e., linear approximation always

21

The subgradient method

Page 22: Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for convex and differentiable f, f(y) f(x) for all I.e., linear approximation always

22

Page 23: Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for convex and differentiable f, f(y) f(x) for all I.e., linear approximation always

23

Page 24: Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for convex and differentiable f, f(y) f(x) for all I.e., linear approximation always

24

Page 25: Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for convex and differentiable f, f(y) f(x) for all I.e., linear approximation always

25

Page 26: Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for convex and differentiable f, f(y) f(x) for all I.e., linear approximation always

26

Page 27: Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for convex and differentiable f, f(y) f(x) for all I.e., linear approximation always

27

Page 28: Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for convex and differentiable f, f(y) f(x) for all I.e., linear approximation always

28

Page 29: Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for convex and differentiable f, f(y) f(x) for all I.e., linear approximation always

29

Page 30: Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for convex and differentiable f, f(y) f(x) for all I.e., linear approximation always

30

Page 31: Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for convex and differentiable f, f(y) f(x) for all I.e., linear approximation always

31

Page 32: Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for convex and differentiable f, f(y) f(x) for all I.e., linear approximation always

32

AppendixSome notes from convex geometry

Page 33: Lecture 08webpages.iust.ac.ir/vhakami/files/convex/Lecture 08.pdf · Subgradients Recall that for convex and differentiable f, f(y) f(x) for all I.e., linear approximation always

33

“Normal cone” of an arbitrary set


Convex Sets and Convex Functions 1 Convex Sets,

Convex Sets and Convex Functions 1 Convex Sets,

Lecture 4 Convex Programming and Lagrange Duality...Lecture 4 Convex Programming and Lagrange Duality (Convex Programming program, Convex Theorem on Alternative, convex duality Optimality

Lecture 4 Convex Programming and Lagrange Duality...Lecture 4 Convex Programming and Lagrange Duality (Convex Programming program, Convex Theorem on Alternative, convex duality Optimality

1 Outline 2 Operations that preserve convexityaaa/Public/Teaching/ORF523/S16/... · The convex envelope (or convex hull) conv D f of a function f : Rn!R over a convex set D Rn is

1 Outline 2 Operations that preserve convexityaaa/Public/Teaching/ORF523/S16/... · The convex envelope (or convex hull) conv D f of a function f : Rn!R over a convex set D Rn is

Images formed by lenses. Convex (converging) lenses, f>0.

Images formed by lenses. Convex (converging) lenses, f>0.

Polygons - Concave or convexPolygons - Concave & convex Write down if polygon is convex or concave: 1) convex 2) convex 3) convex 4) convex 5) convex 6) convex 1 7) convex 8) concave

Polygons - Concave or convexPolygons - Concave & convex Write down if polygon is convex or concave: 1) convex 2) convex 3) convex 4) convex 5) convex 6) convex 1 7) convex 8) concave

Understanding Non-convex Optimization - Praneeth Netrapalli€¦ · •Convex optimization ()is a convex function, 𝒞is convex set •ut “today’s problems”, and this tutorial,

Understanding Non-convex Optimization - Praneeth Netrapalli€¦ · •Convex optimization ()is a convex function, 𝒞is convex set •ut “today’s problems”, and this tutorial,

Learning with Submodular Functions: A Convex Optimization ...fbach/2200000039-Bach-Vol6-MAL-039.pdf · Learning with Submodular Functions: A Convex Optimization Perspective ... F.

Learning with Submodular Functions: A Convex Optimization ...fbach/2200000039-Bach-Vol6-MAL-039.pdf · Learning with Submodular Functions: A Convex Optimization Perspective ... F.

Lecture 2: Convex Setsxwu/class/ELEG667/Lecture2.pdf · 2 Recap: Convex Optimization Problems minimize f 0(x) subject to f i(x) b i,i=1, 2,...,m where objective and constraint functions

Lecture 2: Convex Setsxwu/class/ELEG667/Lecture2.pdf · 2 Recap: Convex Optimization Problems minimize f 0(x) subject to f i(x) b i,i=1, 2,...,m where objective and constraint functions

Lecture 01 - Iran University of Science and Technologywebpages.iust.ac.ir/vhakami/files/convex/Lecture 01.pdf · •“Utility”refers to an assignment of value to each possible

Lecture 01 - Iran University of Science and Technologywebpages.iust.ac.ir/vhakami/files/convex/Lecture 01.pdf · •“Utility”refers to an assignment of value to each possible

Convex Optimization - Tullotullo.ch/static/cambridge/ConvexOptimization-LectureNotes.pdf · convex optimization 11 (i) argmin f is convex (ii)x is a local minimizer of f implies x

Convex Optimization - Tullotullo.ch/static/cambridge/ConvexOptimization-LectureNotes.pdf · convex optimization 11 (i) argmin f is convex (ii)x is a local minimizer of f implies x

Machine Learning 8: Convex Optimization for Machine ...agarivie/sites/... · Convex functions and optimization Proposition Let f be convex. Then x is local minimum of f i 0 2@f(x),

Machine Learning 8: Convex Optimization for Machine ...agarivie/sites/... · Convex functions and optimization Proposition Let f be convex. Then x is local minimum of f i 0 2@f(x),

Linear Programmingocw.snu.ac.kr/sites/default/files/NOTE/10130.pdf · 2018-01-30 · • Convex programming problem – f : convex g i: concave h j: linear – local optimum = global

Linear Programmingocw.snu.ac.kr/sites/default/files/NOTE/10130.pdf · 2018-01-30 · • Convex programming problem – f : convex g i: concave h j: linear – local optimum = global

New Convex Optimization - Nanjing University · 2020. 9. 3. · Introduction Problems Duality Methods Convex Functions Convex Functions A function f : Rd → Ris convex if domf is

New Convex Optimization - Nanjing University · 2020. 9. 3. · Introduction Problems Duality Methods Convex Functions Convex Functions A function f : Rd → Ris convex if domf is

Canonical barriers on convex cones · Let K ˆRn be a regular convex cone. Then the convex solution of the Monge-Ampère equation logdet F00= 2 F with boundary condition Fj @K = +1is

Canonical barriers on convex cones · Let K ˆRn be a regular convex cone. Then the convex solution of the Monge-Ampère equation logdet F00= 2 F with boundary condition Fj @K = +1is

Lecture 3: A brief overview on convex analysis and duality€¦ · Lecture 3: A brief overview on convex analysis and duality Index Terms: Convex sets, convex cones, convex functions,

Lecture 3: A brief overview on convex analysis and duality€¦ · Lecture 3: A brief overview on convex analysis and duality Index Terms: Convex sets, convex cones, convex functions,

CONVEX BODIES ASSOCIATED WITH A CONVEX BODY 781

CONVEX BODIES ASSOCIATED WITH A CONVEX BODY 781

Convex and non-convex worlds in machine learning · Intro Convex solver: bound majorization (Convex) objective: multi-classi cation Non-convexity: deep learning Summary Convex and

Convex and non-convex worlds in machine learning · Intro Convex solver: bound majorization (Convex) objective: multi-classi cation Non-convexity: deep learning Summary Convex and

Convex Optimization · 2021. 1. 16. · Convex Vector Optimization Problem Let f 0: Rn!Rq, K Rq a proper cone. minimize (with respect to K) f 0(x) subject to f i(x) 6 0 h i(x) = 0

Convex Optimization · 2021. 1. 16. · Convex Vector Optimization Problem Let f 0: Rn!Rq, K Rq a proper cone. minimize (with respect to K) f 0(x) subject to f i(x) 6 0 h i(x) = 0

Variational Methods in Convex and Non-Convex Plasticity

Variational Methods in Convex and Non-Convex Plasticity

Convex Optimization M2aspremon/PDF/MVA/UnconstrainedEquality.pdf · Convex Optimization M2. 3/53. Unconstrained minimization minimize f(x) fconvex, twice continuously di erentiable

Convex Optimization M2aspremon/PDF/MVA/UnconstrainedEquality.pdf · Convex Optimization M2. 3/53. Unconstrained minimization minimize f(x) fconvex, twice continuously di erentiable