1

Introduction to MatricesDouglas N. Greve

greve@nmr.mgh.harvard.edu

2

• Simplifies notation

• Simplifies concepts

• Grounding for general linear model (GLM)

• Simplifies implementation (eg, matlab, octave)

• Optimization

Why Matrices?

3

• A matrix is a table of numbers, like a spread sheet

• Each entry is called an “element”

• There is a value in each element (no empty cells)

• Elements can be whole numbers, positive, negative, fractions, real, imaginary, symbolic variables

What is the matrix?

2 4 y

3.3i -7.4M =

M

M

4

Parts of Matrices• Size/Dimensions• Elements• Groupings of Elements• Diagonals• Triangles• Vectors

5

• Number of Rows• Number of Columns • Rows-by-Columns (RowsxCols)

Matrix Size/Dimension

2x3

2 4 y

3.3i -7.4

1x32 4 y

2 2x1

1x1

6

• Scalar is just a number or value• No dimension or size

Scalar versus Matrix

= -1

No bracketsNo dimensionNot a 1x1

= [-1]1x1

7

• Indicated by its row and column number• Scalar value

Matrix Element

2x3

(1,1) (1,2) (1,3)

(2,2) (2,3)

M =

M(2,3) = -7.4

M(i,j)

M(row,col)

Mij

2 4 y

3.3i -7.4

8

Row = Column + Offset

Diagonals

2 4 6

1 3 5

7 8 9

3x3

2 4 6

1 3 5

7 8 9

Row = Column + 0 (Main Diagonal)

Row = Column - 1 (First Upper Diagonal)

Row = Column - 2 (Second Upper Diagonal)

Row = Column - 1 (First Lower Diagonal)

Row = Column - 2 (Second Lower Diagonal)} Off-Diagonals

}

9

Triangles

2 4 6

1 3 5

7 8 93x3

Upper Triangle

Lower Triangle

10

• Special Type of Matrix • Number of Rows or Columns = 1• Often given lower-case variable names

Vector

1x3 Row Vector

2

2 4 y

2x1 Column Vector

11

A matrix consists of vectors

2 4 x

3.3i -7.4

2 4 x

3.3i -7.4

2 4 y

3.3i -7.4

Two 1x3 row vectors Three 2x1 column vectors

2x3

12

Special Matrices• Square• Diagonal• Identity• Triangular• Symmetric• Toeplitz

13

•Rows = Columns

Square Matrix

2x2

2 4

3.3i

1x1

14

• All off-diagonal elements = 0

• M(i,j)=0 if i != j

Diagonal Matrix

3x3

2 0 0

0 3 0

0 0 9

15

• Main diagonal has all 1s • All off-diagonal elements = 0• Square• Symbol “I”

Identity Matrix

3x3

1 0 0

0 1 0

0 0 1

I =

• Any matrix multiplied by the identity is itself

16

Triangular Matrices

2 4 6

0 3 5

0 0 9 3x3

Upper Triangular – all values below the main diagonal are 0

2 0 0

1 3 0

7 8 93x3

Lower Triangular – all values above the main diagonal are 0

17

Symmetric Matrix

2 4 6

4 3 5

6 5 93x3

• M(row,col) = M(col,row)

• Reflect across main diagonal

• Square

18

All elements on a diagonal are the same

Toeplitz Matrix

2 4 6

1 2 4

7 1 2 3x3

2 4

1 2

7 1 3x2

A causal, time invariant linear system is represented by an upper triangular Toeplitz matrix

19

• All elements are 1

• Any number of rows or columns

Ones Matrix

2x3

1 1 1

1 1 1

1x31 1 1

11 2x1

1x1

20

• All elements are 0

• Any number of rows or columns

Zeros Matrix

2x3

0 0 0

0 0 0

1x30 0 0

00 2x1

1x1

21

Matrix Operations• Addition/Subtraction• Vector Multiplication• Matrix Multiplication• Multiplication by scalar• Transpose• Trace• Inverse• Pseudo Inverse

22

• Element-by-Element Addition/Subtraction• Size of two matrices must be the same• C(row,col) = A(row,col) +/- B(row,col)

Matrix Addition/Subtraction

2 4 6

1 3 5A = 0 7 9

1 -2 3.3B =

2 11 15

2 1 8.3C =

2x3

2x3

2x3

C = A + B (your first matrix equation!)

23

• One Row Vector and One Column Vector• Same length, Results in 1x1• Inner Product, Dot Product, “Scalar” Product• Like a correlation• “Multiply and accumulate”

Vector Multiplication

2 4 6x =

0

71

y =

1x3

3x1

z = x.y= x*y = xi*yi

2 4 6* * *0 1 70+4+42 = 46

24

• Series of vector multiplications• Same “inner dimension”

Matrix Multiplication

2 4 6

1 3 5A =

0

1B =

2x3

3x43

4

5

7

2

6

9

8

2

3

cij=Arowi*Bcoljc11

2x4

C=A*B = c12 c13 c14

c21 c22 c23 c24

Dimensions: (2x3) *(3x4)=(2x4)

25

• Series of vector multiplications• Same “inner dimension”

Matrix Multiplication

2 4 6

1 3 5A =

0

1B =

2x3

3x43

4

5

7

2

6

9

8

2

3

c11

2x4

C=A*B = c12 c13 c14

c21 c22 c23 c24

2 4 6 013

c11=

2*0+4*1+6*3= 22

26

• Series of vector multiplications• Same “inner dimension”

Matrix Multiplication

2 4 6

1 3 5A =

0

1B =

2x3

3x43

4

5

7

2

6

9

8

2

3

22

2x4

C=A*B = c12 c13 c14

c21 c22 c23 c24

2 4 6 457

c12=

2*4+4*5+6*7= 70

27

• Series of vector multiplications• Same “inner dimension”

Matrix Multiplication

2 4 6

1 3 5A =

0

1B =

2x3

3x43

4

5

7

2

6

9

8

2

3

22

2x4

C=A*B = 70 c14

c21 c22 c23 c24

2 4 6 269

c13=

2*2+4*6+6*9= 82

c13

28

• Series of vector multiplications• Same “inner dimension”

Matrix Multiplication

2 4 6

1 3 5A =

0

1B =

2x3

3x43

4

5

7

2

6

9

8

2

3

22

2x4

C=A*B = 70 c14

c21 c22 c23 c24

2 4 6 823

c14=

2*8+4*2+6*3= 42

82

29

• Series of vector multiplications• Same “inner dimension”

Matrix Multiplication

2 4 6

1 3 5A =

0

1B =

2x3

3x43

4

5

7

2

6

9

8

2

3

22

2x4

C=A*B = 70 42

c21 c22 c23 c24

1 3 5 013

c21=

1*0+4*1+5*3= 18

82

30

• Series of vector multiplications• Same “inner dimension”

Matrix Multiplication

2 4 6

1 3 5A =

0

1B =

2x3

3x43

4

5

7

2

6

9

8

2

3

22

2x4

C=A*B = 70 42

18 c22 c23 c24

1 3 5 457

c22=

1*4+4*5+5*7= 54

82

31

• One Row Vector one Column Vector• Same length (N)• Result is NxN matrix• Special case of Matrix Multiplication

Outer Product (Vector Multiplication)

2 4 6x =

0

71

y =

1x3

3x1

z = y*x

0

71

2 4 6 1x3

3x1

0

143x3

0 0

2 4 628 42

32

• Multiply each element by scalar value

Multiplication with a scalar

2 4 6

1 3 5A = = -1

2x3

=

No bracketsNo dimensionNot a 1x1

-2 -4 -6

-1 -3 -52x3

33

• “Reflect” across diagonal• B(row,col) = A(col,row)

Matrix Transpose

2 4 6

1 3 5A =

2

4B=AT =At =A’=

2x3 3x26

1

3

5

If A=A’, then A is symmetric (must be square)

34

Sum of diagonal Elements

Trace = M(i,i)

Trace

2 4 6

1 3 5

7 8 9

M =

Trace = 2 + 3 + 9 = 14

35

Determinant

c11

2x2

C = c12

c21 c22

= c11* c22 - c12* c21

• Property of a square matrix • Complicated in general• Simple for a 2x2

36

• C*A = I, then A=C-1

• Must be square • Complicated in general• Simple for a 2x2

Matrix Inverse

c11

2x2

C = c12

c21 c22

= c11* c22 - c12* c21

c22

2x2

C-1 =-c12

-c21 c11

37

Invertibility

1.0

2x2

C = 2.0

0.5 1.0

= 1.0*1.0 - 2.0*0.5 = 1-1 = 0

1.0

2x2

C-1 =-2.0

-0.5 1.0

IMPORTANT!!!• Not all matrices are invertible• =0• “Singular”

38

Singularity and “Ill-Conditioned”

1.0

2x2

C = 2.0

0.5 1.0

= 1.0*1.0 - 2.0*0.5 = 1-1 = 0

• Column 2 = twice Column 1• Linear Dependence

Ill-Conditioned: is “close” to 0Relates to efficiency of a GLM.

39

• Can be used for non-square• Same rules on invertibility apply• Least-mean-square (LMS) for over-determined system

Pseudo-Inverse

If X has more columns than rows X+ = (X’*X)-1*X’ Check: X+*X = (X’*X)-1*X’*X=I If X has more rows than columns X+ = X’ *(X*X’)-1*

Check: X*X+ = X*X’ *(X*X’)-1=I

40

Matrix Applications

41

Sum/Length/Average of a List

length(y) = x’*x = 1*1 + 1*1 + 1*1 = 3

sum(y) = x’*y = 1*1 + 1*2 + 1*3 = 6

x =

3x1

111

1x3

x’ = [1 1 1]1x3

“Ones vector”

3x1

Average = (x’*x)-1(x’*y) [Note: GLM] = (3-1)*6 = 6/3 = 2

y =

3x1

123

42

Selective Average

length(y) = x’*x = 0*0+ 1*1 + 1*1 = 2

sum(y) = x’*y = 0*1 + 1*2 + 1*3 = 5

x =

3x1

011

1x3

x’ = [0 1 1]1x3

“Ones vector”

3x1

Average = (x’*x)-1(x’*y) = (2-1)*5 = 5/2 = 2.5

y =

3x1

123

43

Sum of the Squares of a List

y’ = [1 2 3]1x3

SS(y) = y’*y = 1*1 + 2*2 + 3*3 = 14

y =

3x1

123

SS =y2)

44

Variance of a List

Residual e = y – *x2 = sqrt(e’*e)/DOFDOF = x’*x - 1 = #RowsX-#ColsX

y =

3x1

123

x =

3x1

111

2 =(y-)2

(N-1)

45

• Vector product is 0• Property of two vectors• Uncorrelated/Independence - statistical property

Orthogonality

1 -1 5x =

2

.23

y =

1x3

3x1

z = x*y = xi*yi

1 * 2 +-1 * 3 + 5 * .2 = [0]

1x1

46

Fitting a Line

HRF AmplitudeIQ, Height, Weight

Independent Variable

DependentVariable,Measurement

x1 x2

y2

y1

Subject 1

Subject 2

Thickness

Age

47

Linear ModelIntercept: b

Slope: m

Age

x1 x2

y2

y1

System of Linear Equationsy1 = 1*b + x1*my2 = 1*b + x2*mTwo Eqns, Two Unknowns

“Intercept” = “Offset”

m= (y2-y1) (x2-x1)

b= (x2*y1-x1*y2) (x2-x1)

1. Solve 1st Eq for b2. Substitute into 2nd Eq3. Solve 2nd equation for m4. Substitute m back into Eq for b

48

Matrix Formulation/GLM

Intercept: b

Slope: m

Age

x1 x2

y2

y1

y1 = 1*b + x1*my2 = 1*b + x2*m

- parameter vectorX - design matrix

y1y2

1 x11 x2

bm= *

y = X*2x2

2x12x1

49

Matrix Formulation/GLMIntercept: b

Slope: m

Age

x1 x2

y2

y1

y = X*

y1y2

1 x11 x2

bm= *

1X =

x1

1 x2

•# of Cols of X = # Parameters•Each parameter has a column•Each column means something•Parameter meaning from column

50

GLM SolutionIntercept: b

Slope: m

Age

x1 x2

y2

y1

y = X*ultiply both sides by X-1

=X-1*y

y1y2

1 x11 x2

bm= *

1X =

x1

1 x2

x2X-1 =

-x1

-1 11

= x2-x1Non-invertible if x1=x2Ill-conditioned if x1 near x2 -- Sensitive to noise

51

Non-invertibility

Intercept: b

Slope: m

Age

x1 x2

y2

y1

y1y2

1 x11 x2

bm= *

1X =

.1

1 .1

y = X*1.2

y = 1.2

1 = 2

.2 = 10

1.2y = 1.2

x1=x2=0.1 Want to find but …

is not unique!

52

More than Two Pointsy1 = 1*b + x1*my2 = 1*b + x2*my3 = 1*b + x3*m

Intercept: b

Slope: m

y = X*ultiply both sides by X+

=X+y=(X’X)-1X’y• Three Eqns, Two Unknowns• Over-determined• Still two columns in X, same meaning• X is not square (use pseudo-inv X+; LMS)• Same invertibility concerns• Measure noise (DOF != 0)

53

Matrix Decomposition/Factorization

Break up a single matrix into several matrices that are multiplied together:• Eigen System• Singular Value Decomposition (SVD)

• Principle Component Analysis (PCA)• Cholesky Decomposition - “square root” of a matrix

54

Eigen Decomposition

A = Q**Q-1, A*v = *v

A – square matrixv – eigenvector- eigenvalueQ – matrix of eigenvectors – diagonal matrix of eigenvalues

55