Total Variation and Euler's Elastica for Supervised Learning Tong Lin, Hanlin Xue, Ling Wang,...
-
Upload
reynard-reeves -
Category
Documents
-
view
218 -
download
2
Transcript of Total Variation and Euler's Elastica for Supervised Learning Tong Lin, Hanlin Xue, Ling Wang,...
![Page 1: Total Variation and Euler's Elastica for Supervised Learning Tong Lin, Hanlin Xue, Ling Wang, Hongbin Zha Contact: tonglin123@gmail.com Peking University,](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2002/html5/thumbnails/1.jpg)
Total Variation and Euler's Elastica for Supervised Learning
Tong Lin, Hanlin Xue, Ling Wang, Hongbin Zha
Contact: [email protected]
Peking University, China
2012-6-29
1Key Lab. Of Machine Perception, School of EECS,
Peking University, China
![Page 2: Total Variation and Euler's Elastica for Supervised Learning Tong Lin, Hanlin Xue, Ling Wang, Hongbin Zha Contact: tonglin123@gmail.com Peking University,](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2002/html5/thumbnails/2.jpg)
Background• Supervised Learning:
• Definition: Predict u: x -> y, with training data (x1, y1), …, (xN, yN)
• Two tasks: Classification and Regression
• Prior Work:• SVM:
• RLS: Regularized Least Squares, Rifkin, 2002
2
Hinge loss:
Squared loss:
![Page 3: Total Variation and Euler's Elastica for Supervised Learning Tong Lin, Hanlin Xue, Ling Wang, Hongbin Zha Contact: tonglin123@gmail.com Peking University,](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2002/html5/thumbnails/3.jpg)
3
Background• Prior Work (Cont.):
• Laplacian Energy: “Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples,” Belkin et al., JMLR 7:2399-2434, 2006
• Hessian Energy: “Semi-supervised Regression using Hessian Energy with an Application to Semi-supervised Dimensionality Reduction,” K.I. Kim, F. Steinke, M. Hein, NIPS 2009
• GLS: “Classification using geometric level sets,” Varshney & Willsky, JMLR 11:491-516, 2010
![Page 4: Total Variation and Euler's Elastica for Supervised Learning Tong Lin, Hanlin Xue, Ling Wang, Hongbin Zha Contact: tonglin123@gmail.com Peking University,](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2002/html5/thumbnails/4.jpg)
4
Motivation
SVM Our Proposed Method
![Page 5: Total Variation and Euler's Elastica for Supervised Learning Tong Lin, Hanlin Xue, Ling Wang, Hongbin Zha Contact: tonglin123@gmail.com Peking University,](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2002/html5/thumbnails/5.jpg)
5
Large margin should not be the sole criterion; we argue sharper edges and smoother boundaries can play significant roles.
3D display of the output classification function u(x) by the proposed EE model
![Page 6: Total Variation and Euler's Elastica for Supervised Learning Tong Lin, Hanlin Xue, Ling Wang, Hongbin Zha Contact: tonglin123@gmail.com Peking University,](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2002/html5/thumbnails/6.jpg)
6
• General:
• Laplacian Regularization (LR):
• Total Variation (TV):
• Euler’s Elastica (EE):
1min ( ( ), ) ( )
n
i iiuL u x y S u
2 2min ( ) | |u
u y dx u dx
2 2min ( ) ( ) | |u
u y dx a b u dx
2min ( ) | |u
u y dx u dx
| |
u
u
Models
![Page 7: Total Variation and Euler's Elastica for Supervised Learning Tong Lin, Hanlin Xue, Ling Wang, Hongbin Zha Contact: tonglin123@gmail.com Peking University,](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2002/html5/thumbnails/7.jpg)
7
TV&EE in Image Processing• TV: a measure of total quantity of the value change• Image denoising (Rudin, Osher, Fatemi, 1992)
• Elastica was introduced by Euler in 1744 on modeling torsion-free elastic rods
• Image inpainting (Chan et al., 2002)
![Page 8: Total Variation and Euler's Elastica for Supervised Learning Tong Lin, Hanlin Xue, Ling Wang, Hongbin Zha Contact: tonglin123@gmail.com Peking University,](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2002/html5/thumbnails/8.jpg)
8
• TV can preserve sharp edges, while EE can produce smooth boundaries
• For details, see T. Chan & J. Shen’s textbook: Image Processing and Analysis: Variational, PDE, Wavelet, and Stochastic Methods, SIAM, 2005
![Page 9: Total Variation and Euler's Elastica for Supervised Learning Tong Lin, Hanlin Xue, Ling Wang, Hongbin Zha Contact: tonglin123@gmail.com Peking University,](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2002/html5/thumbnails/9.jpg)
9
Decision boundary
The mean curvature k in high dimensional space can have same expression except the constant 1/(d-1).
![Page 10: Total Variation and Euler's Elastica for Supervised Learning Tong Lin, Hanlin Xue, Ling Wang, Hongbin Zha Contact: tonglin123@gmail.com Peking University,](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2002/html5/thumbnails/10.jpg)
Framework
10
![Page 11: Total Variation and Euler's Elastica for Supervised Learning Tong Lin, Hanlin Xue, Ling Wang, Hongbin Zha Contact: tonglin123@gmail.com Peking University,](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2002/html5/thumbnails/11.jpg)
11
• The calculus of variations → Euler-Lagrange PDE
3
1 1( ) ( '( ) | |) ( ( '( ) | |))
| | | |V n u u u u
u u
2min [ ] ( ) ( )J u u y dx S u
2( ) | |LRS u u dx
( ) | |TVS u u dx
2( ) ( ) | |EES u a b u dx
2( ) 0 (#)u u y 2( ) 0| |
uu y
u
2( ) 0V u y
2( ) a b
Energy Functional Minimization
![Page 12: Total Variation and Euler's Elastica for Supervised Learning Tong Lin, Hanlin Xue, Ling Wang, Hongbin Zha Contact: tonglin123@gmail.com Peking University,](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2002/html5/thumbnails/12.jpg)
12
Solutions
a. Laplacian Regularization (LR)
Radial Basis Function Approximation
b. TV & EE: We develop two solutions• Gradient descent time marching (GD)• Lagged linear equation iteration (LagLE)
![Page 13: Total Variation and Euler's Elastica for Supervised Learning Tong Lin, Hanlin Xue, Ling Wang, Hongbin Zha Contact: tonglin123@gmail.com Peking University,](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2002/html5/thumbnails/13.jpg)
13
Experiments: Two-Moon Data
SVM
EE
Both methods can achieve 100% accuracies with different parameter combinations
![Page 14: Total Variation and Euler's Elastica for Supervised Learning Tong Lin, Hanlin Xue, Ling Wang, Hongbin Zha Contact: tonglin123@gmail.com Peking University,](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2002/html5/thumbnails/14.jpg)
14
Experiments: Binary Classification
![Page 15: Total Variation and Euler's Elastica for Supervised Learning Tong Lin, Hanlin Xue, Ling Wang, Hongbin Zha Contact: tonglin123@gmail.com Peking University,](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2002/html5/thumbnails/15.jpg)
15
Experiments: Multi-class Classification
![Page 16: Total Variation and Euler's Elastica for Supervised Learning Tong Lin, Hanlin Xue, Ling Wang, Hongbin Zha Contact: tonglin123@gmail.com Peking University,](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2002/html5/thumbnails/16.jpg)
16
Experiments: Multi-class Classification
Note: Results of TV and EE are computed by the LagLE method.
![Page 17: Total Variation and Euler's Elastica for Supervised Learning Tong Lin, Hanlin Xue, Ling Wang, Hongbin Zha Contact: tonglin123@gmail.com Peking University,](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2002/html5/thumbnails/17.jpg)
17
Experiments: Regression
![Page 18: Total Variation and Euler's Elastica for Supervised Learning Tong Lin, Hanlin Xue, Ling Wang, Hongbin Zha Contact: tonglin123@gmail.com Peking University,](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2002/html5/thumbnails/18.jpg)
18
Conclusions• Contributions:
• Introduce TV&EE to the ML community
• Demonstrate the significance of curvature and gradient empirically
• Achieve superior performance for classification and regression
• Future Work:• Hinge loss
• Other basis functions
• Extension to semi-supervised setting
• Existence and uniqueness of the PDE solutions
• Fast algorithm to reduce the running time
End, thank you!