LINEAR ALGEBRASourav Sen Gupta

PLAN OF ACTION

Part I — Geometry of Multivariate Data

1. Vector space modelling of multivariate data

2. Understanding Matrices as linear operators

3. Subspaces and Singular Value Decomposition

Part II — Applications in Multivariate Analysis

1. SVD and Principal Component Analysis

2. Eigenvalues and Eigenvectors in PageRank

3. Factor Analysis and SVD in Recommenders

MULTIVARIATE DATA

MULTIVARIATE DATA

p variables/features

n samples/observations

X1 X2 X3 X4S1 5.1 3.5 1.4 0.2S2 4.9 3 1.4 0.2S3 4.7 3.2 1.3 0.2S4 4.6 3.1 1.5 0.2S5 5 3.6 1.4 0.2

S146 6.7 3 5.2 2.3

S147 6.3 2.5 5 1.9

S148 6.5 3 5.2 2

S149 6.2 3.4 5.4 2.3

S150 5.9 3 5.1 1.8

MULTIVARIATE DATA — HISTOGRAMS

MULTIVARIATE DATA — BOXPLOTS

MULTIVARIATE DATA — CORRPLOT

MULTIVARIATE DATA — SCATTERPLOT

MULTIVARIATE DATA — SCATTERPLOT

MULTIVARIATE DATA — SCATTERPLOT

MULTIVARIATE DATA — SCATTERPLOT

MULTIVARIATE DATA — SCATTERPLOTS

MULTIVARIATE DATA — SCATTERPLOTS

MULTIVARIATE DATA — 3D SCATTERPLOT

MULTIVARIATE DATA — 3D SCATTERPLOT

LET’S START WITH 2D

2D GEOMETRY

2D GEOMETRY

(x,y)

x - units

y - units

X

Y

SIZE

X

Y

(x,y)

SIZE

X

Y

(x,y)

size of (x,y) = sqrt(x2 + y2)L2 Norm

SIZE

X

Y

(x,y)

size of (x,y) = abs(x)+ abs(y)L1 Norm

SIZE

X

Y

(x,y)

size of (x,y) = max(abs(x), abs(y))L∞ Norm

UNIT CIRCLE

1

2

∞

ANGLE

X

Y

(x,y)

(u,v)

ANGLE

X

Y

(x,y)

(u,v)

(x,y).(u,v) = |(x,y)| |(u,v)| cos ADot Product

A

ANGLE

X

Y

(x,y)

(u,v)

(x,y).(u,v) = |(x,y)| |(u,v)|Dot Product

ANGLE

X

Y

(x,y)

(u,v)

(x,y).(u,v) = 0Dot Product

SIZE AND ANGLE CHEATSHEET

Notation for dot product : or =

L2 norm of a vector :

Distance between two vectors :

Do you see any connection with Statistics?

v · w vTwnX

i=1

viwi

vT v =nX

i=1

v2i = ||v||22

||v � w||2 =

vuutnX

i=1

(vi � wi)2

ANOTHER LOOK AT 2D

2D GEOMETRY

(x,y)

x - units

y - units

X

Y

2D GEOMETRY

(u,v)

u - units

v - units

U

V

2D GEOMETRY

U

V

X

Y

U = aX + bY

V = cX + dY

2D GEOMETRY

U

V

X

Y

UV

�=

a bc d

� XY

�

TRANSFORMATIONS

TRANSLATIONpq

�

ROTATIONcos ✓ sin ✓� sin ✓ cos ✓

�

SCALING↵ 00 �

�

GENERICa bc d

�

ACTION OF A MATRIX

MATRIX-VECTOR MULTIPLICATION

x =

M v w

LINEAR MAP

Mv w

LINEAR MAP

Mv w

R5 R4

Rn RmMm⇥n

Domain of the function? — Where does the input originate from?

Codomain of the function? — What are possible values of output?

Range of the function? — What are the actual values of output?

Kernel of the function? — Which inputs map to zero in the range?

SUBSPACES

RANGE

+ + + + =

Must be the linear combination of Columns

KERNEL

=

Must be the orthogonal to all Rows

Rn RmMm⇥n

Column Space of the Matrix = Linear span of all Columns = Range

Row Space of the Matrix = Linear span of Rows = Non-zero output

Null Space of the Matrix = Map to zero = Orthogonal to Row Space

Rn RmMm⇥n

ColSpaceRowSpace

Null Space

LINEAR EQUATIONS

SOLVABILITY OF LINEAR EQUATION Mx = b

x b

SOLVABILITY OF LINEAR EQUATION Mx = b

x bSolution always exists.

Solution is unique as Mx = b and My = b implies (x-y) = 0.

SOLVABILITY OF LINEAR EQUATION Mx = b

xb

SOLVABILITY OF LINEAR EQUATION Mx = b

xbu

v

Infinitely many solutions.

If Mu = b, then M(u+v) = b, for any v from Null Space.

SOLVABILITY OF LINEAR EQUATION Mx = b

xb

SOLVABILITY OF LINEAR EQUATION Mx = b

xb

y

e

If Mx = y, then Mx ~ b + e, with irreducible error e.

Solution does not exist!

xb

y

e

Rn RmMm⇥n

T

xb

y

e

Rn RmMm⇥n

T

MTMx = MTb = MTy provides solution x = (MTM)-1MTy

SOLVABILITY OF LINEAR EQUATION Mx = b

Unique Infinite

Infinite None

SINGULAR VALUES

ACTION OF A MATRIX

M

ACTION OF A MATRIX

M

ACTION OF A MATRIX

M

u1v1

M v1 = d1 u1

ACTION OF A MATRIX

M

u1v1

u2

v2

M v1 = d1 u1

M v2 = d2 u2

SINGULAR VECTORS

M v1 u1v2 v3 v4 v5 u2 u3 u4 u5

d1

d2

d3

d4

d5

x x=

SINGULAR VECTORS

M

v1

u1

v2

v3

v4

v5

u2 u3 u4 u5

d1

d2

d3

d4

d5

xx=

SINGULAR VALUE DECOMPOSITION

M

v1

u1

v2

v3

v4

v5

u2 u3 u4

d1

d2 xx=

M = U⌃V T

M

v1

u1

v2

v3

v4

v5

u2 u3 u4

d1

d2 xx=

ColSpaceRowSpace

v1

v2

v3

v4

v5

u1

u2

u3

u4

M = U⌃V T

M

v1

u1

v2

v3

v4

v5

u2 u3 u4

d1

d2 xx=

M

M

v1

u1

v2

v3

v4

v5

u2 u3 u4

d1

d2 xx=

VT

rotation

M

v1

v2

e1

e2

M

v1

u1

v2

v3

v4

v5

u2 u3 u4

d1

d2 xx=

VT ⌃

rotation scaling

M

e1

e2

d1e1

d2e2

M

v1

u1

v2

v3

v4

v5

u2 u3 u4

d1

d2 xx=

VT U⌃

rotation rotationscaling

M

d1e1

d2e2

d1u1

d2u2

M

v1

u1

v2

v3

v4

v5

u2 u3 u4

d1

d2 xx=

VT U⌃

rotation rotationscaling

M

VT U⌃

rotation rotationscaling

M

REFERENCESGilbert Strang — Lectures in Linear Algebra — MIT OCW 18.06 (or YouTube)Gilbert Strang — The Fundamental Theorem of Linear Algebra — AMM 1993Lloyd Trefethen and David Bau — Numerical Linear Algebra — SIAM 1997