Regularized Least-Squares. Outline Why regularization? Truncated Singular Value Decomposition Damped...

Regularized Least-Squares

Outline

• Why regularization?• Truncated Singular Value Decomposition• Damped least-squares• Quadratic constraints

Why regularization?

• We have seen that

systemt independen an form of columns the

solution unique a has

Why regularization?

• We have seen that

• But what happens if the system is almost dependent?– The solution becomes very sensitive to the data– Poor conditioning

systemt independen an form of columns the

solution unique a has

The 1-dimensional case

• The 1-dimensional normal equation

baaa. TT x ˆ

troubleif

Why regularization

• Contradiction between data and model

A more interesting example:scattered data interpolation

)( ipf

“True” curve

Radial basis functions

ii p-pλpf

Rbf are popular

• Modeling– J. C. Carr, R. K. Beatson, J. B. Cherrie, T. J. Mitchell,W. R. Fright, B. C. McCallum,

and T. R. Evans. Reconstruction and representation of 3d objects with radial basis functions. In Proceedings of ACM SIGGRAPH 2001, Computer Graphics Proceedings, Annual Conference Series, pages 67–76, August 2001.

– G. Turk and J. F. O’Brien. Modelling with implicit surfaces that interpolate. ACM Transactions on Graphics, 21(4):855–873, October 2002.

• Animation– J. Noh and U. Neumann. Expression cloning. In Proceedings of ACMSIGGRAPH

2001, Computer Graphics Proceedings, Annual Conference Series, pages 277–288, August 2001.

– F. Pighin, J. Hecker, D. Lischinski, R. Szeliski, and D. H. Salesin. Synthesizing realistic facial expressions from photographs. In Proceedings of SIGGRAPH 98, Computer Graphics Proceedings, Annual Conference Series, pages 75–84, July 1998.

Radial basis functions• At every point )()(

~ii pfpf

Radial basis functions• At every point

• Solve the least-squares problem

ii pfpf

)(min)(~

jjj ii

p-pλpfpfpf λλ

ii pfpf

)(min)(~

jjj ii

p-pλpfpfpf λλ

Rbf results

ii p-pλpf

pi0 close to

ii pfpf

)(min)(~

jjj ii

p-pλpfpfpf λλ

• If pi0 close to

, A is near singular

ii pfpf

)(min)(~

jjj ii

p-pλpfpfpf λλ

pi0 close to

Rbf results with noise

The Singular Value Decomposition• Every matrix A (nxm) can be decomposed into:

– where•U is an nxn orthogonal matrix•V is an mxm orthogonal matrix•D is an nxm diagonal matrix

The Singular Value Decomposition• Every matrix A (nxm) can be decomposed into:

– where•U is an nxn orthogonal matrix•V is an mxm orthogonal matrix•D is an nxm diagonal matrix

Geometric interpretation

Solving with the SVDTTT UDVAbAxAA.. and ˆ

Solving with the SVD

UDVAbAxAA..T

ˆ and

UDVAbAxAA..T

ˆ and

,11,11

...ˆ1

UDVAbAxAA..T

ˆ and

1ˆ buvx

...ˆ1

UDVAbAxAA..T

ˆ and

A is nearly rank defficient

• A is nearly rank defficient =>some of the are close to 0iσ

• A is nearly rank defficient =>some of the are close to 0=>some of the are close to

• Problem with

1ˆ buvx

• Problem with

• Truncate the SVD

1ˆ buvx

mkσσ1

1ˆ1 buvx , and with

pi0 close to

Rbf fit with truncated SVD

Choosing cutoff value k

• The first k such as kσ

Example: inverse skinning“Skinning Mesh Animations”, James and Twigg, siggraph

• Skinning

)(tivbT )()( , i

Bbbbii wt vTv

• Skinning

• Inverse skinning– Let be a set of pairs of geometry and skeleton

configurations

)()( , iBb

bbii wt vTv

sb vT ,

• Skinning

configurations

)()( , iBb

bbii wt vTv

sb vT ,

sbbiw vvT

• Skinning

configurations

)()( , iBb

bbii wt vTv

sb vT ,

• Skinning

configurations

)()( , iBb

bbii wt vTv

sbbiw w

,,min vvT

sb vT ,

“Skinning Mesh Animations”, James and Twigg, siggraph

Problem with the TSVD

• We have to compute the SVD of A, and O() process:impractical for large marices

• Little control over regularization

Damped least-squares

• Replace

where is a scalar and L is a matrix

2min Axbx

222min Lx Axbx

Damped least-squares

• Replace

where is a scalar and L is a matrix

The solution verifies

2min Axbx

222min Lx Axbx

bAxLLAA TTT ˆ2

Examples of L

Diagonal Differential

Limit scale Enforce smoothness

ˆ buvx

Example: “Least-Squares Meshes”, Sorkin and Cohen-Or, siggaph

• Reconstruct a mesh given– Control points– Connectivity (planar mesh)

• Smooth reconstruction

• In matrix form

, iji dL

otherwise

E(i,j)

0LzLyLx

Reconstruction

• Minimize reconstruction error

ss xx 22min Lxx

point control the of position desired the is

mesh tedreconstruc

the inpoint control the of position the is

indicespoint control ofset the is

“Least-Squares Meshes”, Sorkine and Cohen-Or, siggraph

Quadratic constraints

• Solve

Lx min2

tosubject Axbx

d-Lx min2

tosubject Axbx

Quadratic constraints

• Solve

Lx min2

tosubject Axbx

d-Lx min2

tosubject Axbx

,particular In minimized. be to is

which inset feasible a defines Axb

Example

,3 dLx and R

Example

,3 dLx and R

21 xxd-Lx

Example

,3 dLx and R

21 xxd-Lx

Discussion

• If , there is no solution (since there is no x for which )

d-Lxxmin d-Lx

Discussion

• If , the solution exists and is unique

d-Lxxmin d-Lx

d-Lxxmin

Discussion

– Either the solution of is in the feasible set

d-Lxxmin d-Lx

d-Lxxmin

2min Axb x

Discussion

– Either the solution of is in the feasible set

– or the solution is at the boundarySolve

d-Lxxmin d-Lx

d-Lxxmin

2min Axb x

d-LxAxb min2

tosubject x

Discussion

where is a Lagrange multiplier

dLbAxLLAA TTTT ˆ

Conclusion

• TSVD really useful if you need an SVD

Conclusion

• Regularization constrains the solution:– Value, differential operator, other properties– Soft (damping) or hard constraint (quadratic)– Linear or non-linear

Conclusion

• Regularization constrains the solution:– Value, differential operator, other properties– Soft (damping) or hard constraint (quadratic)– Linear or non-linear

• Danger of over-damping or constraining

Example: inverse kinematic

• Problem: solve for joint angles given end-effector positions

Regularized Least-Squares. Outline Why regularization? Truncated Singular Value Decomposition Damped...

Documents

Transcript of Regularized Least-Squares. Outline Why regularization? Truncated Singular Value Decomposition Damped...

Ordinary Least Squares

LEAST-SQUARES REGRESSION 3.2 Least Squares Regression Line and Residuals.

4.3 Least-Squares Regression

Overview of total least squares methods - Welcome to …eprints.soton.ac.uk/263855/1/tls_overview.pdfOverview of total least squares methods 2 The total least squares approximate solution

Asymmetric Least Squares Estimation and Testingjgxchen/Spring2012/Newey-Powell-1987.pdf · Asymmetric Least Squares Estimation and Testing ... ASYMMETRIC LEAST SQUARES ESTIMATION

Design of a Gradient-index Beam Shaping System Via a Genetic …dls/presentations/spie4095... · 2013. 1. 7. · •Damped least-squares •Expert Systems –ZEMAX –SYNOPSYS –Optics

6 6.5 © 2012 Pearson Education, Inc. Orthogonality and Least Squares LEAST-SQUARES PROBLEMS.

Least Squares Method

Discrete Least-squares Approximationsdattab/math435.2009/CHAPTER_LEASTSQURES.pdf · Least-Squares Approximation of a Function We have described least-squares approximation to ﬁt

Least squares CS1114 .

1 Least squares procedure Inference for least squares lines Simple Linear Regression.

Least Squares Migration

Restricted Least Squares, Hypothesis Testing, and ... · Restricted Least Squares, Hypothesis Testing, ... The ordinary least squares estimator is obtained be minimizing the sum of

Simple Linear Regression 1. the least squares procedure 2. inference for least squares lines

Lecture 6 Least-squares applications - Stanford …see.stanford.edu/materials/lsoeldsee263/06-ls-app.pdfEE263 Autumn 2007-08 Stephen Boyd Lecture 6 Least-squares applications • least-squares

Basics of Least Squares Adjustment Computation in Surveying.pdf · Keywords: Least squares, least squares collocation, Kalman filter, total least squares, adjustment computation 1.

Recursive Least-Squares Adaptive Filtersyoga/courses/Adapt/P9... · Recursive Least-Squares Adaptive Filters Dr. Yogananda Isukapalli. 2 –Introduction –Least-Squares problem –Derivation

Fuzzy Least Squares

Regularized Least Squares

Lecture 6 Least-squares applications · Lecture 6 Least-squares applications • least-squares data ﬁtting • growing sets of regressors • system identiﬁcation • growing