7/30/2019 Singular Value Decomposition Example

1/5

Singular Value Decomposition Example

Paul Skoufranis

March 11, 2010

We shall explore a concept known as the singular value decomposition of a matrix.

Definition Let A be an nm matrix. The singular values ofA are the square roots of the eigenvalues ofmm matrix ATA listed according to their algebraic multiplicity. We should note that the eigenvalues ofATA will always be positive numbers so it makes sense to take the square root. It is customary to denotethe singular values by 1, 2, . . ., m with 1 2 m.

Theorem Any n

m matrix A can be written in the form A = UVT where U is an orthogonal n

n

matrix, V is an orthogonal mm matrix, and is an nm matrix whose first r diagonal entries are thenon-zero singular values 1, 2, . . ., r ofA and whose other entries are all zeros. The expression UV

T isknown as the Singular Value Decomposition ofA.

Now we shall go through an example of computing the Singular Value Decomposition (SVD) of a ma-trix A, why it works, and how we can use the SVD to determine the action ofA on the unit circle x2+y2 = 1.

Example Compute the SVD of the matrix

A =

1 11 1

3 0

and use this decomposition to describe the action ofA on the unit circle.

Solution: The first step in finding the SVD ofA is to find the singular values of A. To do this, wesimply need to compute the eigenvalues ofATA. Thus let us first compute ATA;

ATA =

1 1

3

1 1 0 1 11 1

3 0

= 5 22 2

Therefore the characteristic polynomial ofATA is

fATA() = det(ATA I) = (5 )(2 ) 4 = 2 7+ 6 = ( 1)( 6)

Therefore the eigenvalues ofATA are 1 and 6. Thus the singular values ofA are 1 = 1 and 2 =

6 (ingeneral, at this step you would take the square root of each eigenvalue to get a singular value and includeit am() times). This tells us that the matrix will be the 3 2 matrix

=

6 0

0 10 0

7/30/2019 Singular Value Decomposition Example

2/5

where we place 1 =

6 in the first column and 2 = 1 in the second column since

6 > 1 (normally, if1 2 r > 0 are your non-zero singular values (including repetitions), we would place theseentries along the diagonal of and place zeros elsewhere).

Our next goal is to determine the matrix V. We know that since ATA is a symmetric matrix, ATA willhave an orthonormal basis of eigenvectors. It turns out that the matrix Vwill be 22 matrix whose columnsare an orthonormal eigenbasis for ATA. Therefore we compute the eigenspaces ofATA. We notice that

E6 = ker(ATA 6I) = ker 1 2

2 4

= ker

1 20 0

= span

1

5

2

1

and

E1 = ker(ATA I) = ker

4 22 1

= ker

1 1

2

0 0

= span

1

5

12

Let

v1 =1

5

2

1

and v2 =1

5

12

Then we let V be the 2 2 matrix

V= v1 v2 = 25

15

1

52

5 where we place the eigenvector for the eigenvalue 6 in the first column and the eigenvector for the eigenvalue1 in the second column since we placed

6 in first column of and

1 in the second column of .

Now our question is how do we find the orthogonal 3 3 matrix U? Well, if we could find an orthogonal3 3 matrix U such that AV= U, we would be done since V1 = VT as V is an orthogonal matrix. Letus consider the actions ofAV and on the standard basis e1 and e2 ofR

2. Well

10

= 1

10

0

0

1

= 2

01

0

and

AV10 = Av1 AV

01 = Av2

Therefore, ifU were an orthogonal 3 3 matrix such that AV= U and u1, u2, and u3 were the columnsofU, then {u1, u2, u3} would be an orthonormal basis ofR3 and

Av1 = AV

10

= U

10

=

6U

10

0

= 1u1

and

Av2 = AV

01

= U

01

=

1U

01

0

= 2u2

Thus we should let

u1 =1

1Av1 =

1

6

1 11 13 0

15

2

1

=130

332

3

and

u2 =1

2Av2 =

1

1

1 11 1

3 0

1

5

12

=

15

11

3

2

7/30/2019 Singular Value Decomposition Example

3/5

If we check, we see that {u1, u2} are indeed orthonormal vectors. The question is, why does this work?Well, just using the facts that u1 =

1

1Av1, u2 =

1

2Av2, A

TAv1 = 2

1v1 (since v1 is an eigenvector for ATA

with eigenvalue 21

), ATAv2 = 2

2v2 (v2 is an eigenvector for A

TA with eigenvalue 22

), and {v1, v2} is anorthonormal set, we see that

u1

u1 = 11Av1

1

1Av1

=

1

1Av1

T1

1Av1

=1

21

vT1 ATAv1

=1

21

vT1 (ATAv1)

=1

21

vT1

(21v1)

= vT1 v1

= v1 v1 = 1and

u2 u2 =

1

2Av2

1

2Av2

=

1

2Av2

T1

2Av2

=1

22

vT2ATAv2

=1

22

vT2 (ATAv2)

=1

22

vT2 (2

2v2)

= vT2 v2

= v2 v2 = 1and

u1 u2 =

1

1Av1

1

2Av2

=

1

1Av1

T1

2Av2

=1

12vT1 A

TAv2

=1

12vT

1 (AT

Av2)

=1

22vT1

(22v2)

=2

1vT1v2

=2

1v1 v2 = 0

3

7/30/2019 Singular Value Decomposition Example

4/5

which is want we wanted.However, to find an orthogonal matrix using u1 and u2, we need a third orthonormal vector as we needU to be a 3 3 matrix and we only have two orthonormal vectors (in some cases, you will not need thisstep). To find u3, write

u3 =

a

b

c

Then we want a2 + b2 + c2 = 1 so u3 is unit length, and we want

0 = u1 u3 = 130

(3a+ 3b+ 2

3c)

and

0 = u2 u3 = 15

(a b+

3c)

By solving a system of linear equations, we obtain that

110

where R is the set of all solutions to the equations 0 = 130

(3a+ 3b+ 2

3c) and 0 = 15

(a b+ 3c).

Therefore, if we let u3 =12

11

0

, u3 is a normal vector that is orthogonal to u1 and u2. Thus {u1, u2, u3}

is an orthonormal basis for R3. Then, if we let

U=u1 u2 u3

=

330

15

12

330

15

12

210

35

0

then U is an orthogonal matrix and A = UVT

as desired. If you are not confident in your computations,you could easily check your answer by multiplying this out.Now we desire to determine the action ofA on the unit circle. To see this, we will first apply VT to the

unit circle, then to that image, and then we will apply U. The following is a diagram representing theactions of each map:

4

7/30/2019 Singular Value Decomposition Example

5/5

In the diagram, we start with the top left diagram. The vector v1 and its images are shown in blue and thevector v2 and its images are shown in red. First we know that V take e1 to v1 and e2 to v2. Since V

T = V1

as V is an orthogonal matrix, VT takes v1 to e1 and v2 to e2. Then adds a z-axis and dilates the x-axis by6. Then U rotates the whole diagram in an awkward manner (you probably do not need to know exactly

what this rotation is but, it takes the x-axis to the line through u1, it takes the y-axis to the line throughu2, and it takes the z-axis to the line through u3). It is hard to see what the curve in the final diagram looks

like. The blue line goes to the point630

(3, 3, 2

3) and the red line goes to 15

(1,1,3).

5