Regularized Least-Squares. Outline Why regularization? Truncated Singular Value Decomposition Damped...

Regularized Least-Squares



Outline

• Why regularization?• Truncated Singular Value Decomposition• Damped least-squares• Quadratic constraints


Why regularization?

• We have seen that

systemt independen an form of columns the

solution unique a has

A

Axbx

2

min


Why regularization?

• We have seen that

• But what happens if the system is almost dependent?– The solution becomes very sensitive to the data– Poor conditioning

systemt independen an form of columns the

solution unique a has

A

Axbx

2

min


The 1-dimensional case

• The 1-dimensional normal equation

baaa. TT x ˆ




aa

ba0 a

baaa.

T

T

TT

x

x

ˆ

ˆ

.if




troubleif

if

.

.

ˆ

ˆ

0 a

aa

ba0 a

baaa.

T

T

TT

x

x


Why regularization

• Contradiction between data and model


A more interesting example:scattered data interpolation

)( ipf

ip


“True” curve


Radial basis functions


Radial basis functions

i

n

ii p-pλpf

1

)(~


Rbf are popular

• Modeling– J. C. Carr, R. K. Beatson, J. B. Cherrie, T. J. Mitchell,W. R. Fright, B. C. McCallum,

and T. R. Evans. Reconstruction and representation of 3d objects with radial basis functions. In Proceedings of ACM SIGGRAPH 2001, Computer Graphics Proceedings, Annual Conference Series, pages 67–76, August 2001.

– G. Turk and J. F. O’Brien. Modelling with implicit surfaces that interpolate. ACM Transactions on Graphics, 21(4):855–873, October 2002.

• Animation– J. Noh and U. Neumann. Expression cloning. In Proceedings of ACMSIGGRAPH

2001, Computer Graphics Proceedings, Annual Conference Series, pages 277–288, August 2001.

– F. Pighin, J. Hecker, D. Lischinski, R. Szeliski, and D. H. Salesin. Synthesizing realistic facial expressions from photographs. In Proceedings of SIGGRAPH 98, Computer Graphics Proceedings, Annual Conference Series, pages 75–84, July 1998.


Radial basis functions• At every point )()(

~ii pfpf


Radial basis functions• At every point

• Solve the least-squares problem

)()(~

ii pfpf

2

1 1

2

1

)(min)(~

)(min

n

j

n

iiij

n

jjj ii

p-pλpfpfpf λλ




)()(~

ii pfpf

2

1 1

2

1

)(min)(~

)(min

n

j

n

iiij

n

jjj ii

p-pλpfpfpf λλ

nnnn

n

n λ

λ

pppp

pppp

pf

pf

:.

..

::

..

:1

1

1111

Axb


Rbf results

i

n

ii p-pλpf

1

)(~


pi0 close to

pi1




)()(~

ii pfpf

2

1 1

2

1

)(min)(~

)(min

n

j

n

iiij

n

jjj ii

p-pλpfpfpf λλ

nnnn

n

n λ

λ

pppp

pppp

pf

pf

:.

..

::

..

:1

1

1111

Axb




• If pi0 close to

pi1

, A is near singular

)()(~

ii pfpf

2

1 1

2

1

)(min)(~

)(min

n

j

n

iiij

n

jjj ii

p-pλpfpfpf λλ

nnnn

n

n λ

λ

pppp

pppp

pf

pf

:.

..

::

..

:1

1

1111

Axb


pi0 close to

pi1


Rbf results with noise


The Singular Value Decomposition• Every matrix A (nxm) can be decomposed into:

– where•U is an nxn orthogonal matrix•V is an mxm orthogonal matrix•D is an nxm diagonal matrix

TUDVA


The Singular Value Decomposition• Every matrix A (nxm) can be decomposed into:

– where•U is an nxn orthogonal matrix•V is an mxm orthogonal matrix•D is an nxm diagonal matrix

TUDVA

T

mmm

m

m

nnn

n

vv

vv

uu

uu

,1,

,11,1

1

,1,

,11,1

..

::

..

.

0..0

::

..0

::

0..

.

..

::

..

A


Geometric interpretation

T

mmm

m

m

nnn

n

vv

vv

uu

uu

,1,

,11,1

1

,1,

,11,1

..

::

..

.

0..0

::

..0

::

0..

.

..

::

..

A


Solving with the SVDTTT UDVAbAxAA.. and ˆ


Solving with the SVD

bUVDx

UDVAbAxAA..T

TTT

ˆ

ˆ and



bUVDx

UDVAbAxAA..T

TTT

ˆ

ˆ and

bx

T

nnn

n

mmmm

m

uu

uu

vv

vv

,1,

,11,11

,1,

,11,1

..

::

..

.

0..0

::

1..0

::

0..1

.

..

::

..ˆ



b

u

u

vvx

nm

m :.

0..0

::

1..0

::

0..1

...ˆ1

1

1

bUVDx

UDVAbAxAA..T

TTT

ˆ

ˆ and



m

i

Tii

iσ1

1ˆ buvx

b

u

u

vvx

nm

m :.

0..0

::

1..0

::

0..1

...ˆ1

1

1

bUVDx

UDVAbAxAA..T

TTT

ˆ

ˆ and


A is nearly rank defficient



T

mmm

m

m

nnn

n

vv

vv

uu

uu

,1,

,11,1

1

,1,

,11,1

..

::

..

.

0..0

::

..0

::

0..

.

..

::

..

A



T

mmm

m

nnn

n

vv

vv

uu

uu

,1,

,11,1

1

,1,

,11,1

..

::

..

.

0..0

::

0..0

::

0..

.

..

::

..

A



• A is nearly rank defficient =>some of the are close to 0iσ



• A is nearly rank defficient =>some of the are close to 0=>some of the are close to

iσ

iσ

1




• Problem with

m

i

Tii

iσ1

1ˆ buvx

iσ

iσ

1




• Problem with

• Truncate the SVD

n

i

Tii

iσ1

1ˆ buvx

iσ

k

i

Tii

im σ

mkσσ1

1

1ˆ1 buvx , and with

iσ

1


pi0 close to

pi1


Rbf fit with truncated SVD


Choosing cutoff value k

• The first k such as kσ


Example: inverse skinning“Skinning Mesh Animations”, James and Twigg, siggraph

• Skinning



• Skinning

?



• Skinning

)(tivbT )()( , i

Bbbbii wt vTv



• Skinning

• Inverse skinning– Let be a set of pairs of geometry and skeleton

configurations

)()( , iBb

bbii wt vTv

)(tiv

bT

si

sb vT ,



• Skinning


configurations

)()( , iBb

bbii wt vTv

)(tiv

bT

si

sb vT ,

2

,sii

Bb

sbbiw vvT



• Skinning


configurations

)()( , iBb

bbii wt vTv

)(tiv

bT

si

sb vT ,

s

sii

Bb

sbbiw

2

, vvT



• Skinning


configurations

)()( , iBb

bbii wt vTv

)(tiv

bT

s

sii

Bb

sbbiw w

bi

2

,,min vvT

si

sb vT ,


“Skinning Mesh Animations”, James and Twigg, siggraph


Problem with the TSVD

• We have to compute the SVD of A, and O() process:impractical for large marices

• Little control over regularization


Damped least-squares

• Replace

by

where is a scalar and L is a matrix

2min Axbx

222min Lx Axbx


Damped least-squares

• Replace

by

where is a scalar and L is a matrix

The solution verifies

2min Axbx

222min Lx Axbx

x̂

bAxLLAA TTT ˆ2


Examples of L

nl

l

0

01

L

11

11

L

Diagonal Differential

Limit scale Enforce smoothness

n

i

Tii

ii

i

lσ

σ

122

ˆ buvx


810


710


410


Example: “Least-Squares Meshes”, Sorkin and Cohen-Or, siggaph

• Reconstruct a mesh given– Control points– Connectivity (planar mesh)




• Smooth reconstruction

Eji

jiid,

vv




• Smooth reconstruction

• In matrix form

Eji

jiid,

vv

0

/1

1

, iji dL

otherwise

if

if

E(i,j)

ji

0LzLyLx

x

x

n

i

.

.

v

v

x and


Reconstruction

• Minimize reconstruction error

where

Cs

ss xx 22min Lxx

point control the of position desired the is

mesh tedreconstruc

the inpoint control the of position the is

indicespoint control ofset the is

s

s

x

x

C


“Least-Squares Meshes”, Sorkine and Cohen-Or, siggraph


Quadratic constraints

• Solve

or

Lx min2

tosubject Axbx

d-Lx min2

tosubject Axbx


Quadratic constraints

• Solve

or

Lx min2

tosubject Axbx

d-Lx min2

tosubject Axbx

d-xL

d-Lx

ˆ

2

,particular In minimized. be to is

which inset feasible a defines Axb


Example

1

0

0

000

010

001

,3 dLx and R


Example

1

0

0

000

010

001

,3 dLx and R

222

21

21 xxd-Lx


Example

1

0

0

000

010

001

,3 dLx and R

222

21

21 xxd-Lx

12

1x2x


Discussion

• If , there is no solution (since there is no x for which )

d-Lxxmin d-Lx


Discussion


• If , the solution exists and is unique

d-Lxxmin d-Lx

d-Lxxmin


Discussion



– Either the solution of is in the feasible set

d-Lxxmin d-Lx

d-Lxxmin

2min Axb x


Discussion



– Either the solution of is in the feasible set

– or the solution is at the boundarySolve

d-Lxxmin d-Lx

d-Lxxmin

2min Axb x

d-Lx

d-LxAxb min2

tosubject x


Discussion

Solve

where is a Lagrange multiplier

dLbAxLLAA TTTT ˆ


Conclusion

• TSVD really useful if you need an SVD


Conclusion


• Regularization constrains the solution:– Value, differential operator, other properties– Soft (damping) or hard constraint (quadratic)– Linear or non-linear


Conclusion


• Regularization constrains the solution:– Value, differential operator, other properties– Soft (damping) or hard constraint (quadratic)– Linear or non-linear

• Danger of over-damping or constraining


Example: inverse kinematic

• Problem: solve for joint angles given end-effector positions




?

Regularized Least-Squares. Outline Why regularization? Truncated Singular Value Decomposition Damped...

Documents

Transcript of Regularized Least-Squares. Outline Why regularization? Truncated Singular Value Decomposition Damped...