SE301: Numerical Methods Topic 3: Solution of Systems of Linear Equations Lectures 12-17:

CISE301_Topic3 KFUPM 1

CISE301: Numerical Methods

Topic 3: Solution of Systems of Linear

Equations Lectures 12-17:

KFUPM

Read Chapter 9 of the textbook


Lecture 12Vector, Matrices, and

Linear Equations


VECTORS

1000

,

0100

,

0010

,

0001

12

tor column vec 241 vector row

:Examplesnumbers ofarray ldimensiona one a:Vector

4321 eeeevectorsIdentity


MATRICES

1200141001430021

lTridiagona,

6000000000400001

diagonal

1001

matrix identity 00

0000

matrix zero

:Examplesnumbers ofarray ldimensiona twoa:Matrix


MATRICES

1000140001403121

ngular upper tria,451501112

symmetric

:Examples


Determinant of a MATRICES

82)015(1)512(1)25(2

501-3

1-451-3

1-4550

2451501132

det

:Examplesonly matrices squarefor Defined


Adding and Multiplying Matrices

ji

ji

m

k

,bacBAC *

pm ifonly defined is ABCproduct The * q)B(p and m)(nA matrices twooftion Multiplica

,bacBAC *size same thehave they ifonly Defined * B andA matrices twoofaddition The

1kjikij

ijijij


Systems of Linear Equations

formMatrix form Standard

753

601315.2342

76535.23342

formsdifferent in presented becan equationslinear of systemA

3

2

1

31

321

321

xxx

xxxxxxxx


Solutions of Linear Equations

523

:equations following theosolution t a is 21

21

21

2

1

xxxx

xx


Solutions of Linear Equations A set of equations is inconsistent if there

exists no solution to the system of equations:

ntinconsiste are equations These542

32

21

21

xxxx


Solutions of Linear Equations Some systems of equations may have infinite

number of solutions

allfor solution ais)3(5.0

solutions ofnumber infinite have 642

32

2

1

21

21

aa

axx

xxxx


Graphical Solution of Systems ofLinear Equations

523

21

21

xxxx

solution

x1

x2


Cramer’s Rule is Not Practical

way efficient in computed are tsdeterminan theif used becan It system. 30by 30 a solve toyears10 needscomputer super A

. systems largefor practicalnot is Rule sCramer'

2

21115131

,1

21112513

system thesolve toused becan Rule sCramer'

17

21 xx


Naive Gaussian Elimination Examples

Lecture 13 Naive Gaussian

Elimination


Naive Gaussian Elimination The method consists of two steps:

Forward Elimination: the system is reduced to upper triangular form. A sequence of elementary operations is used.

Backward Substitution: Solve the system starting from the last variable.

''

'00''0

3

2

1

3

2

1

33

2322

131211

3

2

1

3

2

1

333231

232221

131211

bbb

xxx

aaaaaa

bbb

xxx

aaaaaaaaa

Not to be confused withthe derivative operation!


Elementary Row Operations

Adding a multiple of one row to another

Multiply any row by a non-zero constant


ExampleForward Elimination

1

2

3

4

1

1

2

3

4

6 2 2 4 1612 8 6 10 26

3 13 9 3 196 4 1 18 34

Part 1: Forward Elimination Step1: Eliminate from equations 2, 3, 4

6 2 2 40 4 2 20 12 8 10 2 3 14

xxxx

x

xxxx

166

2718



396

16

30005200

22404226

4equation from Eliminate:Step3 2196

16

134005200

22404226

4 ,3equationsfrom Eliminate:Step2

4

3

2

1

3

4

3

2

1

2

xxxx

xxxxx

x



396

16

30005200

22404226

3419

2616

1814639133

1068124226

:nEliminatio Forward theofSummary

4

3

2

1

4

3

2

1

xxxx

xxxx


ExampleBackward Substitution

36

)1(4)2(2)1(216,14

)1(2)2(26

22

59,133

for solve ,...for solvethen ,for Solve396

16

30005200

22404226

12

34

134

4

3

2

1

xx

xx

xxxxxxx


Forward Elimination

ni

baabb

njaaaaa

x

ni

baabb

njaaaaa

x

ijj

ji

ijij

ijj

ji

ijij

3

)2(

eliminate To

2

)1(

eliminate To

222

2

222

2

2

111

1

111

1

1


Forward Elimination

.eliminated is until Continue

1

)(

eliminate To

1

n

mmm

imjj

mjmm

imijij

m

x

nim

baabb

njmaaaaa

x


Backward Substitution

mm

n

mjjjmm

m

nn

nnnnnnnn

nn

nnnnn

nn

nn

a

xabx

axaxab

x

axab

x

abx

,

1,

2,2

11,2,222

1,1

,111


Summary of the Naive Gaussian Elimination Example How to check a solution Problems with Naive Gaussian Elimination

Failure due to zero pivot element Error

Lecture 14Naive Gaussian

Elimination


Naive Gaussian Eliminationo The method consists of two steps

o Forward Elimination: the system is reduced to upper triangular form. A sequence of elementary operations is used.

o Backward Substitution: Solve the system starting from the last variable. Solve for xn ,xn-1,…x1.

''

'00''0

3

2

1

3

2

1

33

2322

131211

3

2

1

3

2

1

333231

232221

131211

bbb

xxx

aaaaaa

bbb

xxx

aaaaaaaaa


Example 1

177564

832

11333723

1122210232

)(18323 ,2equations from Eliminate:Step1___nEliminatio Forward:1Part

:nEliminatioGaussian Naive using Solve

32

32

321

321

321

321

1

xxxxxxx

eqeqeqxxx

eqeqeqxxx

equationpivotunchangedeqxxxx


Example 1

131364

832

215331775

)(2641832

3equationfrom Eliminate:Step2nEliminatio Forward:1Part

3

32

321

32

32

321

2

xxxxxx

eqeqeqxx

equationpivotunchangedeqxxunchangedeqxxx

x


Example 1Backward Substitution

121

issolution The

1328

2146

11313

3

2

1

1,1

32

1,1

33,122,111

3

2,2

33,222

3,3

33

xxx

axx

axaxab

x

xa

xabx

abx


How Do We Know If a Solution is Good or Not Given AX=B

X is a solution if AX-B=0 Due to computation error AX-B may not be zero Compute the residuals R=|AX-B|

One possible test is ?????

ii

rmax if acceptable issolution The


Determinant

13detdet

1300410

321A'

213232321

A

:Exampletdeterminan affect thenot do operations elementary The

operations Elementary

(A')(A)


How Many Solutions Does a System of Equations AX=B Have?

0 elements0 elements B ingcorrespondB ingcorrespond

rows zerorows zeromoreor one has moreor one hasrows zero no has

matrix reducedmatrix reducedmatrix reduced0det(A)0det(A)0det(A)

Infinitesolution NoUnique


Examples

5.1

!105.0

0#:

02

0021

12

0021

11

2021

42

4221

32

4221

21

4321

solutions of # infintesolution NoUnique

XimpossibleX

solutionsInfinitesolutionNosolution

XXX

XXX


Lectures 15-16:Gaussian Elimination

with Scaled Partial Pivoting

Problems with Naive Gaussian Elimination Definitions and Initial step Forward Elimination Backward substitution Example


Problems with Naive Gaussian Elimination

o The Naive Gaussian Elimination may fail for very simple cases. (The pivoting element is zero).

o Very small pivoting element may result in serious computation errors

21

1110

2

1

xx

21

11110

2

110

xx


Example 2

1111

3524368541231211

:Pivoting Partial Scaledwith nEliminatioGaussian using sytstem following theSolve

4

3

2

1

xxxx


Example 2Initialization step

4321LVectorIndex

5842S vectorScale1

111

3524368541231211

4

3

2

1

xxxx

Scale vector:disregard sign find largest in magnitude in each row


Why Index Vector? Index vectors are used because it is much

easier to exchange a single index element compared to exchanging the values of a complete row.

In practical problems with very large N, exchanging the contents of rows may not be practical since they could be stored at different locations.


Example 2Forward Elimination-- Step 1: eliminate x1

]1324[ Exchangeequation pivot first theis 4equation

toscorrespondmax 54

,85

,43

,21

4,3,2,1

]4321[]5842[

1111

3524368541231211

equationpivot theofSelection

14

41,

4

3

2

1

Llandl

liS

aRatios

LS

xxxx

i

i

l

l



125.275.125.1

352475.025.05.50

75.175.25.0025.075.05.10

1111

3524368541231211

B andA Update

4

3

2

1

4

3

2

1

xxxx

xxxx

First pivot equation



]2314[25.1

85.5

45.04,3,2:Ratios

]1324[]5842[125.275.125.1

352475.025.05.50

75.175.25.0025.075.05.10

equationpivot second theofSelection

2,

4

3

2

1

LiS

a

LSxxxx

i

i

l

l



19

1667.225.1

35242000

8333.15.20025.075.05.10

]3214[1

8333.61667.225.1

35246667.125.0008333.15.200

25.075.05.10

4

3

2

1

4

3

2

1

xxxx

Lxxxx

Third pivot equation



7.23334

2531

1.13335.1

75.025.025.1

2.43275.28333.11667.2,5.4

29

]3214[

19

1667.225.1

35242000

8333.15.20025.075.05.10

234

1,

22,33,44,1

34

2,

33,44,2

4

3,

44,3

4,4

4

3

2

1

1

1111

2

222

3

33

4

4

xxxa

xaxaxabx

xxa

xaxabx

xa

xabx

ab

x

L

xxxx

l

llll

l

lll

l

ll

l

l


Example 3

1111

3524368541231211

Pivoting Partial Scaledwith nEliminatioGaussian using sytstem following theSolve

4

3

2

1

xxxx


Example 3Initialization step

4321LVectorIndex

5842S vectorScale1

111

3524368541231211

4

3

2

1

xxxx



]1324[ Exchangeequation pivot first theis 4equation

toscorrespondmax 54

,85

,43

,21

4,3,2,1

]4321[]5842[

1111

3524368541231211

equationpivot theofSelection

14

41,

4

3

2

1

Llandl

liS

aRatios

LS

xxxx

i

i

l

l



125.275.125.1

352475.025.05.100

75.175.25.0025.075.05.10

1111

3524368541331211

B andA Update

4

3

2

1

4

3

2

1

xxxx

xxxx



]1234[25.1

85.10

45.0

25.1

85.10

45.0:Ratios

125.275.125.1

352475.025.05.100

75.175.25.0025.075.05.10


4

3

2

1

L

xxxx



]1234[25.1

85.10

45.04,3,2:Ratios

]1324[]5842[125.275.125.1

352475.025.05.100

75.175.25.0025.075.05.10


2,

4

3

2

1

LiS

a

LS

xxxx

i

i

l

l



12.25

1.85710.9286

352475.025.05.100

1.71432.7619-000.35710.785700

]2314[125.275.125.1

352475.025.05.100

75.175.25.0025.075.05.10

B andA Updating

4

3

2

1

4

3

2

1

xxxx

Lxxxx



]1234[2

0.78574

2.76194,3:Ratios

]1234[]5842[1

2.251.85710.9286

352475.025.05.100

1.71432.7619000.35710.785700

equationpivot third theofSelection

3,

4

3

2

1

LiS

a

LS

xxxx

i

i

l

l



12.25

1.85711.4569

352475.025.05.100

1.71432.7619000.8448000

]1234[1

2.251.85710.9286

352475.025.05.100

1.71432.7619000.35710.785700

4

3

2

1

4

3

2

1

xxxx

Lxxxx



1.86734

2531

0.3469

0.39802.7619

1.71431.8571,1.72450.84481.4569

]1234[

12.25

1.85711.4569

352475.025.05.100

1.71432.7619000.8448000

234

1,

22,33,44,1

2,

33,44,2

4

3,

44,3

4,4

4

3

2

1

1

1111

2

222

3

33

4

4

xxxa

xaxaxabx

axaxab

x

xa

xabx

ab

x

L

xxxx

l

llll

l

lll

l

ll

l

l


How Good is the Solution?

0.0010.0030.0020.005

:Residues

1.72450.39800.34691.8673

1111

3524368541231211

4

3

2

1

4

3

2

1

R

xxxx

solution

xxxx


Remarks: We use index vector to avoid the need to move

the rows which may not be practical for large problems.

If we order the equation as in the last value of the index vector, we have a triangular form.

Scale vector is formed by taking maximum in magnitude in each row.

Scale vector do not change. The original matrices A and B are used in

checking the residuals.


Lecture 17 Tridiagonal & Banded

Systems and Gauss-Jordan

Method Tridiagonal Systems Diagonal Dominance Tridiagonal Algorithm Examples Gauss-Jordan Algorithm*


Tridiagonal Systems: The non-zero elements are

in the main diagonal, super diagonal and subdiagonal.

aij=0 if |i-j| > 1

5

4

3

2

1

5

4

3

2

1

6100014100026200014300015

bbbbb

xxxxx

Tridiagonal Systems


Occur in many applications Needs less storage (4n-2 compared to n2 +n for the general cases) Selection of pivoting rows is unnecessary

(under some conditions) Efficiently solved by Gaussian elimination

Tridiagonal Systems


Based on Naive Gaussian elimination. As in previous Gaussian elimination algorithms

Forward elimination step Backward substitution step

Elements in the super diagonal are not affected. Elements in the main diagonal, and B need

updating

Algorithm to Solve Tridiagonal Systems


Tridiagonal System

nnn

n

nnnn

n

b

bbb

x

xxx

dc

dcd

cd

b

bbb

x

xxx

dac

dacda

cdb a, d

3

2

1

3

2

1

1

3

22

11

3

2

1

3

2

1

1

1

32

221

11

and and d update toneed We


Diagonal Dominance

row. ingcorrespond in the elements of sum thenlarger tha iselement diagonaleach of magnitude The

)1(aa

ifdominant diagonally is matrix A

,1ijii nifor

An

ijj


Diagonal Dominance

dominant DiagonallyNot dominant Diagonally

121232103

521161103

:Examples


Solving Tridiagonal System

1,...,2,1for 1

onSubstituti Backward

2

nEliminatio Forward

1

11

1

11

1

nnixcbd

x

dbx

nibdabb

cdadd

iiii

i

n

nn

ii

iii

ii

iii


Example

1,2,3for 1,

onSubstituti Backward

42,

nEliminatio Forward689

12

,222

,111

,

5555

689

12

51251

25125

Solve

1

11

11

1

1

4

3

2

1

ixcbd

xdbx

ibdabbc

dadd

BCAD

xxxx

iiii

in

nn

ii

iiii

i

iii


Example

5619.45652.4

5652.616,5619.45652.4

215

5652.66.4

6.618,5652.46.4215

6.651219,6.4

5215

nEliminatio Forward689

12

,222

,111

,

5555

33

3443

3

344

22

2332

2

233

11

1221

1

122

bdabbc

dadd

bdabbc

dadd

bdabbc

dadd

BCAD


ExampleBackward Substitution After the Forward Elimination:

Backward Substitution:

25

1212

16.4

126.6

15652.4

125652.6

,15619.45619.4

5619.45652.66.612,5619.45652.46.45

1

2111

2

3222

3

4333

4

44

dxcbx

dxcbx

dxcbx

dbx

BD TT


Gauss-Jordan Method The method reduces the general system of

equations AX=B to IX=B where I is an identity matrix.

Only Forward elimination is done and no substitution is needed.

It has the same problems as Naive Gaussian elimination and can be modified to do partial scaled pivoting.

It takes 50% more time than Naive Gaussian method.


Gauss-Jordan MethodExample

270

200560

111

11233

11422

2/11

32fromxEleminate1

270

422124

222

3

2

1

1

3

2

1

xxx

eqeqeq

eqeqeq

eqeq

andequationsStep

xxx



21.16671.1667

2000.8333101667.001

21033

21111

6/22

31fromxEleminate2

270

200560

111

3

2

1

2

3

2

1

xxx

eqeqeq

eqeqeq

eqeq

andequationsStep

xxx



121

100010001

31

0.833322

31

1667.011

2/33

21fromxEleminate3

21.16671.1667

2000.8333101667.001

3

2

1

3

3

2

1

xxx

eqeqeq

eqeqeq

eqeq

andequationsStep

xxx



121

121

100010001

270

422124

222

3

2

1

3

2

1

3

2

1

xxx

issolutionxxx

todtransformeis

xxx

SE301: Numerical Methods Topic 3: Solution of Systems of Linear Equations Lectures 12-17:

Documents

Transcript of SE301: Numerical Methods Topic 3: Solution of Systems of Linear Equations Lectures 12-17: