Linear Algebra Review -...

Linear Algebra Review 9/4/07

Tim Marks 1

Linear Algebra Review

By Tim K. Marks

UCSD

Borrows heavily from:

Jana Koseckahttp://cs.gmu.edu/~kosecka/cs682.html

Virginia de Sa (UCSD)Cogsci 108F Linear Algebra review

Vectors

The length of x, a.k.a. the norm or 2-norm of x, is

e.g.,

!

x = x1

2+ x

2

2+L+ x

n

2

!

x = 32

+ 22

+ 52

= 38


Tim Marks 2

Good Review Materials

http://www.imageprocessingbook.com/DIP2E/dip2e_downloads/review_material_downloads.htm

(Gonzales & Woods review materials)

Chapt. 1: Linear Algebra Review

Chapt. 2: Probability, Random Variables, Random Vectors

Online vector addition demo:

http://www.pa.uky.edu/~phy211/VecArith/index.html


Tim Marks 3

Vector Addition

vvuu

u+vu+v

Vector Subtraction

uuvv

u-vu-v


Tim Marks 4

Example (on board)

Inner product (dot product) oftwo vectors

!

a =

6

2

"3

#

$

% % %

&

'

( ( (

!

b =

4

1

5

"

#

$ $ $

%

&

' ' '

!

a " b = aTb

!

= 6 2 "3[ ]

4

1

5

#

$

% % %

&

'

( ( (

!

= 6 " 4 + 2 "1+ (#3) " 5

!

=11


Tim Marks 5

Inner (dot) Product

vv

uuαα

The inner product is a The inner product is a SCALAR.SCALAR.

!

uTv = 0 " u # v


Tim Marks 6

Transpose of a Matrix

Transpose:Transpose:

Example of symmetric matrixExample of symmetric matrix

!

Cm"n = A

Tn"m

!

cij = a ji

!

(A + B)T

= AT

+ BT

!

(AB)T

= BTAT

Examples:Examples:

!

6 2

1 5

"

# $

%

& '

T

=6 1

2 5

"

# $

%

& '

!

6 2

1 5

3 8

"

#

$ $ $

%

&

' ' '

T

=6 1 3

2 5 8

"

# $

%

& '

If If , we say , we say AA is is symmetricsymmetric..

!

AT

= A


Tim Marks 7


Tim Marks 8


Tim Marks 9


Tim Marks 10

!

6 2

1 5

"

# $

%

& ' (2 5

3 1

"

# $

%

& ' =

18 32

17 10

"

# $

%

& '

Matrix ProductProduct:Product:

Examples:Examples:

A and B must haveA and B must havecompatible dimensionscompatible dimensions

In Matlab: >> A*B!

Cn" p = An"mBm" p

!

cij = aikbkjk=1

m

"

Matrix Multiplication is not commutative:Matrix Multiplication is not commutative:

!

An"nBn"n # Bn"nAn"n!

2 5

3 1

"

# $

%

& ' (6 2

1 5

"

# $

%

& ' =

17 29

19 11

"

# $

%

& '


Tim Marks 11

!

cij = aij + bij

!

Cn"m = A

n"m + Bn"m

Matrix Sum

Sum:Sum:

A and B must have the sameA and B must have the samedimensionsdimensions

Example:Example:

!

2 5

3 1

"

# $

%

& ' +

6 2

1 5

"

# $

%

& ' =

8 7

4 6

"

# $

%

& '

!

deta11

a12

a21

a22

"

# $

%

& ' =

a11

a12

a21

a22

= a11a22( a

21a12

!

det

a11

a12

a13

a21

a22

a23

a31

a32

a33

"

#

$ $ $

%

&

' ' '

= a11

a22

a23

a32

a33

( a12

a21

a23

a31

a33

+ a13

a21

a22

a31

a32

Determinant of a Matrix

Determinant:Determinant:

!

det2 5

3 1

"

# $

%

& ' = 2 (15 = (13Example:Example:

A must be squareA must be square


Tim Marks 12

Determinant in Matlab

Inverse of a Matrix

If A is a square matrix, the inverse of A, called A-1,satisfies

AA-1 = I and A-1A = I,

Where I, the identity matrix, is a diagonal matrixwith all 1’s on the diagonal.

!

I2

=1 0

0 1

"

# $

%

& '

!

I3

=

1 0 0

0 1 0

0 0 1

"

#

$ $ $

%

&

' ' '


Tim Marks 13

Inverse of a 2D Matrix

!

6 2

1 5

"

# $

%

& '

(1

)6 2

1 5

"

# $

%

& ' =

1

28

5 (2

(1 6

"

# $

%

& ' )6 2

1 5

"

# $

%

& ' =

1

28

28 0

0 28

"

# $

%

& ' =

1 0

0 1

"

# $

%

& '

!

6 2

1 5

"

# $

%

& '

(1

=1

28

5 (2

(1 6

"

# $

%

& '

Example:Example:

Inverses in Matlab


Tim Marks 14

Trace of a matrix

Example (on board)

Matrix Transformation: Scale

A square, diagonal matrix scales each dimension bythe corresponding diagonal element.

Example:

!

2 0 0

0 .5 0

0 0 3

"

#

$ $ $

%

&

' ' '

6

8

10

"

#

$ $ $

%

&

' ' '

=

12

4

30

"

#

$ $ $

%

&

' ' '


Tim Marks 15

Linear Independence• A set of vectors is linearly dependent if one of

the vectors can be expressed as a linearcombination of the other vectors.

Example:

!

1

0

0

"

#

$ $ $

%

&

' ' '

,

0

1

0

"

#

$ $ $

%

&

' ' '

,

2

1

0

"

#

$ $ $

%

&

' ' '

• A set of vectors is linearly independent if noneof the vectors can be expressed as a linearcombination of the other vectors.

Example:

!

1

0

0

"

#

$ $ $

%

&

' ' '

,

0

1

0

"

#

$ $ $

%

&

' ' '

,

2

1

3

"

#

$ $ $

%

&

' ' '

Rank of a matrix• The rank of a matrix is the number of linearly

independent columns of the matrix.Examples:

has rank 2

!

1 0 2

0 1 1

0 0 0

"

#

$ $ $

%

&

' ' '

• Note: the rank of a matrix is also the number oflinearly independent rows of the matrix.

has rank 3

!

1 0 2

0 1 0

0 0 1

"

#

$ $ $

%

&

' ' '


Tim Marks 16

Singular Matrix

All of the following conditions are equivalent. Wesay a square (n × n) matrix is singular if any oneof these conditions (and hence all of them) issatisfied.– The columns are linearly dependent

– The rows are linearly dependent

– The determinant = 0

– The matrix is not invertible

– The matrix is not full rank (i.e., rank < n)

Linear Spaces

A linear space is the set of all vectors that can beexpressed as a linear combination of a set of basisvectors. We say this space is the span of the basisvectors.– Example: R3, 3-dimensional Euclidean space, is

spanned by each of the following two bases:

!

1

0

0

"

#

$ $ $

%

&

' ' '

,

0

1

0

"

#

$ $ $

%

&

' ' '

,

0

0

1

"

#

$ $ $

%

&

' ' '

!

1

0

0

"

#

$ $ $

%

&

' ' '

,

0

1

2

"

#

$ $ $

%

&

' ' '

,

0

0

1

"

#

$ $ $

%

&

' ' '


Tim Marks 17

Linear Subspaces

A linear subspace is the space spanned by a subsetof the vectors in a linear space.– The space spanned by the following vectors is a

two-dimensional subspace of R3.

!

1

0

0

"

#

$ $ $

%

&

' ' '

,

0

1

0

"

#

$ $ $

%

&

' ' '

!

1

1

0

"

#

$ $ $

%

&

' ' '

,

0

0

1

"

#

$ $ $

%

&

' ' '

What does it look like?

What does it look like?

– The space spanned by the following vectors is atwo-dimensional subspace of R3.

Orthogonal and OrthonormalBases

n linearly independent real vectorsspan Rn, n-dimensional Euclidean space

• They form a basis for the space.

– An orthogonal basis, a1, …, an satisfies

ai ⋅ aj = 0 if i ≠ j– An orthonormal basis, a1, …, an satisfies

ai ⋅ aj = 0 if i ≠ jai ⋅ aj = 1 if i = j

– Examples.


Tim Marks 18

Orthonormal Matrices

A square matrix is orthonormal (also calledunitary) if its columns are orthonormal vectors.– A matrix A is orthonormal iff AAT = I.

• If A is orthonormal, A-1 = AT

AAT = ATA = I.

– A rotation matrix is an orthonormal matrix withdeterminant = 1.

• It is also possible for an orthonormal matrix to havedeterminant = –1. This is a rotation plus a flip (reflection).

Matrix Transformation:Rotation by an Angle θ

2D rotation matrix:


Tim Marks 19

http://www.math.ubc.ca/~cass/courses/m309-8a/java/m309gfx/eigen.html


Tim Marks 20

Some Properties ofEigenvalues and Eigenvectors

– If λ1, …, λn are distinct eigenvalues of a matrix, thenthe corresponding eigenvectors e1, …, en are linearlyindependent.

– If e1 is an eigenvector of a matrix with correspondingeigenvalue λ1, then any nonzero scalar multiple of e1 isalso an eigenvector with eigenvalue λ1.

– A real, symmetric square matrix has real eigenvalues,with orthogonal eigenvectors (can be chosen to beorthonormal).

SVD: Singular Value DecompositionAny matrix A (m × n) can be written as the product of three

matrices:A = UDV T

where

– U is an m × m orthonormal matrix

– D is an m × n diagonal matrix. Its diagonal elements, σ1, σ2, …, arecalled the singular values of A, and satisfy σ1 ≥ σ2 ≥ … ≥ 0.

– V is an n × n orthonormal matrix

Example: if m > n

!

• • •

• • •

• • •

• • •

• • •

"

#

$ $ $ $ $ $

%

&

' ' ' ' ' '

=

( ( ( (

| | | |

u1

u2

u3

L um

| | | |

) ) ) )

"

#

$ $ $ $ $ $

%

&

' ' ' ' ' '

*1

0 0

0 *2

0

0 0 *n

0 0 0

0 0 0

"

#

$ $ $ $ $ $

%

&

' ' ' ' ' '

+ v1

T ,

M M M

+ vn

T ,

"

#

$ $ $

%

&

' ' '

A U D V T


Tim Marks 21

SVD in Matlab>> a = [1 2 3; 2 7 4; -3 0 6; 2 4 9; 5 -8 0]a = 1 2 3 2 7 4 -3 0 6 2 4 9 5 -8 0

>> [u,d,v] = svd(a)u = -0.24538 0.11781 -0.11291 -0.47421 -0.82963 -0.53253 -0.11684 -0.52806 -0.45036 0.4702 -0.30668 0.24939 0.79767 -0.38766 0.23915 -0.64223 0.44212 -0.057905 0.61667 -0.091874 0.38691 0.84546 -0.26226 -0.20428 0.15809

d =

14.412 0 0

0 8.8258 0

0 0 5.6928

0 0 0

0 0 0

v =

0.01802 0.48126 -0.87639

-0.68573 -0.63195 -0.36112

-0.72763 0.60748 0.31863

Some Properties of SVD

– The rank of matrix A is equal to the number of nonzerosingular values σi

– A square (n × n) matrix A is singularif and only if

at least one of its singular values σ1, …, σn is zero.


Tim Marks 22

Geometric Interpretation of SVDIf A is a square (n × n) matrix,

– U is a unitary matrix: rotation (possibly plus flip)– D is a scale matrix– V (and thus V T) is a unitary matrix

Punchline: An arbitrary n-D linear transformation isequivalent to a rotation (plus perhaps a flip), followed by ascale transformation, followed by a rotationAdvanced: y = Ax = UDV Tx– V T expresses x in terms of the basis V.– D rescales each coordinate (each dimension)– The new coordinates are the coordinates of y in terms of the basis U

!

• • •

• • •

• • •

"

#

$ $ $

%

&

' ' '

=

( L (

u1

L un

) L )

"

#

$ $ $

%

&

' ' '

*1

0 0

0 *2

0

0 0 *n

"

#

$ $ $

%

&

' ' '

+ v1

T ,

M M M

+ vn

T ,

"

#

$ $ $

%

&

' ' '

A U D V T

Linear Algebra Review -...

Documents

Transcript of Linear Algebra Review -...