MA2213 Lecture 5

Linear Equations

(Direct Solvers)

Systems of Linear Equations p. 243-248

Occur in a wide variety of disciplines

Mathematics

Statistics

Physics

Chemistry Biology

Economics

Sociology

Psychology

Archaeology

Geology

AstronomyAnthropology

EngineeringManagement

Business

Medicine

Finance

Matrix Form

for a system of linear equations bxA

nnRA nRbnRx

coefficient matrix

(solution) column vector

(right) column vector

Linear Equations in Mathematics

Numerical Analysis

Geometry

Interpolation

Least Squares

Quadrature

Algebra

find intersection of lines or planes

partial fractions 111

22

x

b

x

a

x

0ba2 ba 1a

1b

Coefficient Matrix

Vandermonde (for polyn. interp.)

or Gramm

Transpose of VandermondeLec 4 vufoil 13 (to compute weights)

BBT

2

0

11

11

b

a

Matrix Arithmetic p. 248-264

Matrix Inverse

Matrix MultiplicationnmRA

100

010

001

3IIdentity Matrix

pnRB pmRAB

10

012I

ac

bd

bcaddc

ba 11

Theorem 6.2.6 p. 255 A square matrix has an inverse iff (if and only if) its determinant is not equal to zero.

Solution of(this means )0det A

exists and is unique.

bAxAAbAx 11 )(

bAxAA 11 )(

bAxbAxI 11

multiplication is associative

for nonsingular

bxA

Proof

A

Remark In MATLAB use: x = A \ b;

Column Rank of a Matrix

Definition The column rank of a matrix

},...,1,0{cr , mMRM nm

dimension of the subspace of 1 mm RR

?021

242cr

spanned by the column vectors of

Mcr Remark

Mmaximal number of

linearly independent column vectors

M

Question

is the

of

Row Rank of a Matrix

Definition The row rank of a matrix},...,1,0{rr , nMRM nm

is the dimension of the subspace of 1nR

?021

242rr

spanned by the row vectors of

Mrr Remark

Mmaximal number of

linearly independent row vectors of M

Question

A Matrix Times a Vector

nnnnnn

n

n

b

b

b

x

x

x

aaa

aaa

aaa

bxA

2

1

2

1

21

22221

11211

has solution iff b is a linear combination of columns of A

nnn

n

n

n

nn b

b

b

a

a

a

x

a

a

a

x

a

a

a

x

2

1

2

1

2

22

12

2

1

21

11

1

The equation

Existence of Solution in General

The linear equation bxA has a solution if and only if

AbA cr][cr A IS SINGULAR!EVEN IF Example

2

1

2

1

21

42

b

b

x

xthis has a solution iff

21 2bb then it has an infinite number of solutions

called Augmentedmatrix p. 265

Computing the Column and Row RanksThe ranks of a matrix nmRM can be computed using a sequence of elementary row operations p. 253-254.

i. Interchange two rowsii. Multiply a row by a nonzero scalariii. Add a nonzero multiple of one row to another rowQuestion Show that each of the ERO i, ii, iii has an inverse ERO i, ii, iii.

Elementary Row Operations

00010

01000

00100

10000

00001

iE

can be performed

on the left by by multiplying

nmRM on a matrix

Mnonsingular matrices

mmRE

MEM i

10000

05000

00100

00010

00001

iiE

MEM ii MEM ii

10000

01000

00100

02010

00001

iiiE

Invariance of Row Rank Under EROTheorem 1. Ifis an ERO matrix, thenProof Clearly, interchanging two rowsand multiplying a row by a nonzero scalar does not change the row rank. Finish the proof by showing that adding a multiple of any row to another row does not change the row rank.

nmRM and mmRE .rr rr MME

Remark Clearly the row rank of a matrix is invariant under sequence of ERO’s.

Matrix Multiplication

nmn RvvvM 21

mnn Rvvvvv ,,...,,, 1321

mmRE nm

n REvEvEvEM 21

Invariance of Column Rank under EROnmRM Theorem 2 If

is nonsingular then .cr cr MME Proof It suffices to show that for a set

are linearly dependent iff the set ofMofof column vectors

rkkk vvv ,...,,21

rkkk EvEvEv ,...,,21

are linearly dependent. Show why it suffices and then show it. Hint: prove

mmRE and

column vectors of ME

0011 11

rr krkkrk vcvcEvcEvc

Row Echelon Matrices

Definition A matrix

an row echelon matrix if

i. the nonzero rows come first

nmRM

ii. the first nonzero element in each row =1 (called a pivot) has all zeros below it

is called

iii. each pivot lies to the right of the pivot in the row above

Row Echelon MatricesThese three properties produce a staircase pattern in the matrix below

000000000

610000000

075231000

4407.931610

13541506.831

Question Where are the pivots ?

Row Rank of an Row Echelon Matrixequals the number of nonzero rows.

000000000

610000000

075231000

4407.931610

13541506.831

Question What is the rank of this matrix ?

Prove this by showing that the rows must be linearly independent. Hint : use pivots.

Col. Rank of a Row Echelon Matrixequals the number of nonzero rows.

000000000

610000000

075231000

4407.931610

13541506.831

Question Show this by showing that the col. vectors that contain pivots form a basis for the space spanned by col. vectors. Hint: do elem. col. operations on the matrix above.

Reduction to Row Echelon FormTheorem 3 For every matrix

nmRM there exists a nonsingular matrix

is an echelon matrix. such that

kEEEEEE 4321

mmRE ME

Furthermore, the matrix E is a product where each

kjjE 1, is an ERO matrix.

Application of the sequence of ERO’s is called reduction to row echelon form.

Proof Based on Gaussian elimination.

Row Rank = Column RankTheorem 4 For every matrix

nmRM

Proof. Theorem 3 implies that there exists

kEEEEEE 4321

EMM rr rr

MM rr cr

MEa product

of ERO matrices such that is a rowechelon matrix. Theorems 1 implies that

and theorems 2 implies that

.cr cr EMM Since ME is a row echelon

matrix, EMME cr rr hence .cr rr MM

Applications of Row Echelon Reduction

The linear equation bxA iff the last nonzero row of the reduced ][ bAE

Example

22

2

2

2

2

1

1

11

00

21

21

21

21

42b

bb

bbb

b

has a solution

has its pivot NOT in the last column.

Hence the condition above is satisfied iff .0221 bb

Applications of Row Echelon Reduction

A basis of column vectors for a matrix

],...,,[ 21 nvvvME can be obtained by first computing the reduction

nmRM

then choosing the column vectors

that form a basis for the space spanned by the column

rkkk vvv ,...,,21

that contain the pivots. Then the vectors

rkkk vEvEvE 111 ,...,,21

are column vectors of

vectors of

.M

M

Generalities on Gaussian EliminationGaussian elimination is the process of reducing a matrix to row echelon form through a sequence of ERO’s.

It can also be used to solve a system of linear equations

The final step of solving a system of equations after the augmented matrix has been reduced is called back substitution, this process is related to elementary column operations and will be addressed in the homework.

It is ‘best’ taught through showing examples.

We will show how to solve a system of linear equationsusing Gaussian elimination, it will become obvious how to use Gaussian elimination for reduction.

Gaussian Elimination (p. 264-269)Case 1.

nna

a

a

A

00

00

00

22

11

nnnn bxabxabxa ,...,, 22221111

The equations for this matrix are

Question How do we use the nonsingular assumption?

therefore, if A is nonsingular then

nn

nn a

bx

a

bx

a

bx ,...,,

22

22

11

11

Question What type of matrix is this ?

Back SubstitutionCase 2.

nn

n

n

a

aa

aaa

A

00

0 222

11211

A nonsingular solution by back-substitution p. 265

Question How do we use the nonsingular assumption?

nnnn bax 1

Question What is this matrix called ?

Question What are the associated equations ?

][ ,111

1,11 nnnnnnn xabax

][ 131321211

111 nnxaxaxabax

Question Why is this method called back-substitution ?

Gaussian Elimination on EquationsCase 3. Apply elementary row operations on equations to to obtain equations with an upper triangular matrix

22 321 xxx 1r

122 3rrr

133 321 xxx 2r421 xx 3r

2/,5/ 3322 rrrr

22 321 xxx55 3 x

133 rrr 222 32 xx22 321 xxx132 xx13 x 32 rr

Question How can we solve these equations ?

Gaussian Elimination on Augmented Matrix

nnnnn

n

n

b

b

b

aaa

aaa

aaa

2

1

21

22221

11211

11rarr iii

1

12

1

112

12

122

11211

0

0

nnnn

n

n

b

b

b

aa

aa

aaa

jjiii rarr

nj ,...,2for

loop) ( end i

1111 / arr

loop) ( end j

0,1, 11 kk akrr0 If 11 a

ni ,...,2for

)loop ( end i

0 If jja 0,, jkkj ajkrr

jjjj arr /nji ,...,1for

Gaussian Elimination

122 3rrr

133 rrr

4011

1133

2211

Question What is the solution ?

2220

5500

2211

1100

1110

2211

251

2 Rr

32 rr 32

13 rr

122 3rrr

bAx

EbEAx

Partial Pivoting p. 270-273

the integer ||max1

kjnk

j aS

that gives

kj Rr

For the j-th column in Gaussian elimination compute

nkj

then perform the row interchange

Read p. 273-276 about how Gaussian elimination can be used to compute the inverse of a matrix.

LU Decomposition p. 283-285

ULATo solve

where

1

01

001

21

21

nn

L

nn

n

n

u

uu

uuu

U

00

0 222

11211

Then for each b use forward substitution to solve L y = b then use backward substitution to solve U x = y.

first compute the factorization

bAx for many values of b with same A

LU Decomposition AlgorithmAlgorithm

Step 1 njau jj ,...,1,11 niuaii ,...,2,/ 1111

Step 2 for r = 2,…,n do

njruau jk

r

kkrjrjr

,1

1

niruau rk

r

kkirirrri

1,1

1

1

Question How many operations does this require ?

Homework Due Tutorial 3Question 1. Prove that the row rank of an row echelon matrix equals the number of nonzero rows.Question 2. Prove that the column rank of an row echelon matrix equals the number of nonzero rows by showing that the set of its column vectors having pivots is a maximal set of linearly independent column vectors. Question 3. Use Gaussian elimination to solve

901565,54963,2842 32132132 uuuuuuuuQuestion 4. Derive expressions for the entries of the Land U in the LU decomposition of a 3 x 3 matrix A.Question 5. Show how elementary column operations can be applied to a row echelon matrix M to obtain a row echelon matrix with exactly one 1 in each nonzero row. Use this to determine a basis for the space { x : Mx = 0 }.