Matrix Algebra (I)

Matrix Algebra

Matrix algebra is a means of efficiently expressing large numbers of calculations to be made upon ordered sets of numbers

Also referred to as Linear Algebra

Definitions - Scalars

scalar - a single number

What is a Vector ?

Vector - a single row or column of numbers Each number is called a component or entry denoted with bold small letters row vector

column vector

1234

c

1 2 3 4r

What are Matrices ?

An nxm matrix is a rectangular array of numbers (called entries) arranged in n rows and m columns, e.g.

11 12 13

21 22 23

a a aa a aA

is a 2×3 matrix

Properties of Matrices

Matrices are denoted by bold capital letters

All matrices (and vectors) have an order or dimension - that is the number of rows ×the number of columns. Often a matrix A of dimension n × m is

denoted Anxm

Equality of Matrices

Two matrices are equal if all of their corresponding entries are equal

Statistical data sets are matrices (usually observations in rows and variables in columns)

11 12 1m

21 22 2m

n1 n2 nm

Variable 1 Variable 2 Variable mObservation 1 a a aObservation 2 a a a

Observation n a a a

Basic Matrix Operations

Transposition

Sum and Difference

Product

Inversion

Transpose of a Matrix

The transpose A’ of a matrix A is the matrix such that the ith row of A is the ith

column of A’ and vice versa.

Transpose of a Matrix : Example

If

A' 1 2 3

4 5 6

1 42 53 6

A

then

More on Transpose

(A’)’ = A

If A = A', then A is said to be symmetric

Sum and Difference

Two matrices may be added (subtracted) if they have the same order

Simply add (subtract) entries from corresponding locations

11 12 11 12 11 12

21 22 21 22 21 22

31 32 31 32 31 32

a a b b c ca a + b b = c ca a b b c c

11 11 11 12 12 12

21 21 21 22 22 22

31 31 31 32 32 32

a +b = c , a +b = c ,a +b = c , a +b = c ,a +b = c , a +b = c

where

Sum and Difference - Example

If we have

1 2 7 10 8 12+ = 3 4 + 8 11 = 11 15

5 6 9 12 14 18C A B

1 2 7 103 4 and = 8 115 6 9 12

A B

then C = A + B is given by

Sum and Difference - Example

Similarly, if we have

1 2 7 10 -6 -8- = 3 4 - 8 11 = -5 -7

5 6 9 12 -4 -6C A B

1 2 7 103 4 and = 8 115 6 9 12

A B

then C = A – B is given by

Properties of Sum / Difference

The transpose of a sum = sum of transposes (A+B+C)’ = A’+B’+C’

A+B = B+A (commutative)

A+(B+C) = (A+B)+C (associative)

Product of a Scalar and a Matrix

To multiply a matrix by a scalar, simply multiply each entry of the matrix by the scalar quantity

11 12 11 12

21 22 21 22

a a a a=a a a a

Product of a Scalar & a Matrix -Example

If we have

A1 2 3.5 7.0

3.5 3 4 = 10.5 14.05 6 17.5 21.0

A1 23 4 and = 3.55 6

then we can calculate λA by

• Note that λA = A λ if λ is a scalar

Product of Matrices We write the multiplication of two

matrices A and B as AB

This is referred to either as pre-multiplying B by A

or post-multiplying A by B

So for matrix multiplication AB, A is referred to as the premultiplier and B is referred to as the postmultiplier

Special Matrix Multiplication

Premultiplication of a column vector a by conformable row vector b yields a single value called the dot product If

53 4 6 and 2

8a b

53 4 6 2 = 3 5 +4 2 +6 8 = 71

8ab

then ab is defined by

Product of Matrices

In order to multiply matrices, they must be conformable (number of columns in the premultiplier = number of rows in postmultiplier)

Note that (m × n) × (n × p) = (m × p) (m × n) × (p × n) impossible (1 × n) × (n × 1) = (1 × 1)

Product of Matrices

If we have A3x3 and B3x2 then

11 12 13 11 12 11 12

21 22 23 21 22 21 22

31 32 31 3231 32 33

a a a b b c ca a a b b = c c

b b c ca a aAB C

3i1 1j i2 2j i3ij i3j k k1

32 31 12 32 22 33 32

j c = a b + a b + a b a b

e.g. c = a b + a b + a b

where

Product of Matrices

If we have A3x3 and B3x2 then

11 12 1311 12

21 22 21 22 23

31 32 31 32 33

a a ab bb b a a a is undefinedb b a a a

BA

i.e., matrix multiplication is not commutative(why ?)

Matrix Multiplication : An Example

If we have

11 12

21 22

31 32

c c1 4 7 1 4 30 662 5 8 2 5 = c c = 36 813 6 9 3 6 42 96c c

AB

11 11 11 12 21 13 31

12 11 12 12 22 13 32

21 21 11 22 21 23 31

22 21 12 22 22 23 32

31 31 11 32 21 33 31

32 31 12 32 22 3

c = a b + a b + a b =1 1 + 4 2 + 7 3 = 30

c = a b + a b + a b =1 4 + 4 5 + 7 6 = 66

c = a b + a b + a b = 2 1 +5 2 + 8 3 = 36

c = a b + a b + a b = 2 4 +5 5 + 8 6 = 81

c = a b + a b + a b = 3 1 + 6 2 + 9 3 = 42

c = a b + a b + a 3 32b = 3 4 + 6 5 + 9 6 = 96

1 4 7 1 42 5 8 and = 2 53 6 9 3 6

A B

then

where

Properties of Matrix Multiplication

Even if conformable, AB does not necessarily equal BA (i.e., matrix multiplication is not commutative) Multiplication is associative, i.e., A(BC) = (AB)C (AB)’ = B’A’

Special Uses of Matrix Multiplication

Suppose we have :a11x1 + a12x2 = b1

a21x1 + a22x2 = b2

If we let

then we can rewrite the system as Ax = b.

11 12 1 1

21 22 2 2

a a x b, = , = a a x bA x b

Special Matrices

There are a number of special matrices.

Zero Matrix Identity Matrix Diagonal Matrix

Zero MatrixA square matrix whose elements all equal 0

0 0 0 00 0 0 00 0 0 00 0 0 0

Note : O+A = A+O = A for any A

Identity MatrixAn identity matrix is a diagonal matrix where the diagonal elements all equal 1

Note : IA = AI = A for any A

1 0 0 00 1 0 00 0 1 00 0 0 1

I

Diagonal Matrix

A diagonal matrix is a square matrix with all off-diagonal entries being zero.

11

22

33

44

a 0 0 00 a 0 00 0 a 00 0 0 a

Prelude to Inverse Matrix

Consider the system of equations:

a11x1 + a12x2 = b1a21x1 + a22x2 = b2

which can be represented by:

Ax = b

where

11 12 1 1

21 22 2 2

a a x b= , = , and =a a x bA x b

Prelude to Inverse Matrix

If we were to solve this system of equations simultaneously for x2 we would have:

a21(a11x1 + a12x2 = b1)-a11(a21x1 + a22x2 = b2)

which yields (through cancellation & rearranging):

a21a11x1 + a21a12x2 - a11a21x1 - a11a22x2

= a21b1 - a11b2

Prelude to Inverse Matrix

This gives

11 2 21 1

211 22 12 21

a b a bx =

a a a a

The denominator is called the determinant of A,|A| or det(A) :

11 22 12 21

= a a a aA

Thus if |A|= 0 then either

i) no solution orii) many solutions

Determinant of a 2x2 Matrix If we have a 2x2 matrix A such that

then

For example if

11 22 12 21= a a - a aA

11 12

21 22

a aa aA

1 23 4A

11 22 12 211 2= = a a - a a 1 4 - 2 3 = -23 4A

Some Properties of Determinants

Determinants have several mathematical properties useful in matrix manipulations: |A|=|A'| |AB|= |A||B|

(determinant of product = product ofdeterminants)

Inverse of Matrices of Order 2

Suppose a bc dA

Check : AA-1 = A-1A = I

1

d bc adet A

A

then

Inverse of Matrices of Order 2 -Example

Suppose 1 22 5A

1

5 22 1 5 2

2 11A

then

Inverse of a Matrix

The inverse of a matrix A, denoted by A-1 , is such that AA-1 = A-1A = I

An inverse of A exists if only if |A| 0 A matrix which has no inverse is said to

be singular (AB)-1 = B-1A-1

Inverse Using EXCEL

• Enter the matrix entries, e.g. A1:D4

• Highlight the cells where you want to place the inverse matrix

• While still highlighted, enter

= MINVERSE(A1:D4)

• Press Ctrl Shift Enter together

Solving Linear Equations

Consider the system again :

Ax = b

where

If A-1 exists then

A-1Ax = A-1b

or x = A-1b

11 12 1 1

21 22 2 2

a a x b= , = , and =a a x bA x b

Trace of a Matrix

The trace of a square matrix A is the sum of the leading diagonal elements

Denoted tr(A)For example, the trace of

is

n

iii=1

tr = a =1+4 =5A

1 23 4A

Some Properties of Trace

For any scalar c, tr(cA) = c[tr(A)] tr(A B) = tr(A) tr(B) tr(AB) = tr(BA)

Quadratic Forms

A quadratic form is a function

Q(x) = x’Ax

in k variables x1,…,xk where

and A is a kxk symmetric matrix.

x1

2

k

xx

x

Quadratic Forms

A quadratic form has only squared terms and cross-product terms.

then

1

2

x 1 2 and x 2 2x A

2 21 1 2 2Q( ) = = x + 4x x 2xx x'Ax

Suppose

Positive Definite Matrix/Quadratic Form

If x’Ax > 0

x’ =[x1 x2 … xk] [0 0 … 0]

the matrix A and the quadratic form are said to be positive definite.

Quadratic Forms with 2 Unknowns

x 2 21

1 2 1 1 2 22

xa bQ( ) = x x ax 2bx x cxb c x

The quadratic form

can be re-written as

where

which is positive definite when

2a 0 and ac b 0

1 2z x / x

2 22x az 2bz c

Quadratic Form : Examples

The quadratic form 2 21 2 1 24x + 9x - 6x x

having matrix 4 -3-3 9

is positive definite. Why ?

How about ?2 21 2 1 24x + 9x -14x x

Comment

There are many other matrix algebra results that will be important to statistics. This file provides only a glimpse to the totality.