Linear System - Classic

7/29/2019 Linear System - Classic

1/41

Linear SystemsIrwan Ary Dharmawan


2/41

Linear Systems

3333232131

2323222121

1313212111

bxaxaxa

bxaxaxa

bxaxaxa

3

2

1

3

2

1

333231

232221

131211

b

b

b

x

x

x

aaa

aaa

aaa


3/41

Linear Systems Solve Ax=b, where A is an nn matrix and

b is an n1 column vector

Can also talk about non-square systems where

A is mn, b is m1, and x is n1

Overdeterminedifm>n:

more equations than unknowns

Underdeterminedifn>m:

more unknowns than equations

Can look for best solution using least squares


4/41

Singular SystemsA is singular if some row is

linear combination of other rows

Singular systems can be underdetermined:

or inconsistent:

1064

532

21

21

xx

xx

1164

532

21

21

xx

xx


5/41

Inverting a Matrix Usually not a good idea to compute x=A-1b

Inefficient

Prone to roundoff error

In fact, compute inverse using linear solver

Solve Axi=bi where bi are columns of identity,

xi are columns of inverse

Many solvers can solve several R.H.S. at once


6/41

Gauss-Jordan Elimination

Fundamental operations:

1. Replace one equation with linear combination

of other equations

2. Interchange two equations

3. Re-label two variables

Combine to reduce to trivial system

Simplest variant only uses #1 operations,

but get better stability by adding

#2 (partial pivoting) or #2 and #3 (full pivoting)


7/41


Solve:

Only care about numbers form tableau oraugmented

matrix:

1354

732

21

21

xx

xx

13

7

54

32


8/41


Given:

Goal: reduce this to trivial system

and read off answer from right column

13

7

54

32

?

?

10

01


9/41


Basic operation 1: replace any row by

linear combination with any other row

Here, replace row1 with 1/2 * row1 + 0 * row2

13

7

54

32

1354

1 27

23


10/41

Gauss-Jordan Elimination Replace row2 with row2 4 * row1

Negate row2

1354

1 27

23

110

1 27

23

110

1 27

23


11/41


Replace row1 with row1 3/2 * row2

Read off solution: x1 = 2, x2 = 1

110

1 27

23

1

2

10

01


12/41

Gauss-Jordan Elimination For each row i:

Multiply row i by 1/aii

For each other row j:

Addaji times row i to row j

At the end, left part of matrix is identity,

answer in right part

Can solve any number of R.H.S. simultaneously


13/41

Pivoting Consider this system:

Immediately run into problem:

algorithm wants us to divide by zero!

More subtle version:

8

2

32

10

8

2

32

1001.0


14/41

Pivoting Conclusion: small diagonal elements bad

Remedy: swap in larger element from somewhere else


15/41

Partial Pivoting

Swap rows 1 and 2:

Now continue:

8

2

32

10

2

8

10

32

2

1

10

01

2

4

10

1 23


16/41

Full Pivoting

Swap largest element onto diagonal by swapping rows 1 and

2 and columns 1 and 2:

Critical: when swapping columns, must remember to swap

results!

8

2

32

10

2

8

01

23


17/41

Full Pivoting

Full pivoting more stable, but only slightly

2

8

01

23

323

8

32

32

0

1

1

2

10

01

* Swap results

1 and 2


18/41

Operation Count For one R.H.S., how many operations?

For each of n rows:

Do n times:

For each of n+1 columns:

One add, one multiply

Total = n3+n2 multiplies, same # of adds

Asymptotic behavior: when n is large, dominated by n3


19/41

Faster Algorithms Our goal is an algorithm that does this in

1/3 n3 operations, and does not requireall R.H.S. to be known at beginning

Before we see that, lets look at a few

special cases that are even faster


20/41

Tridiagonal Systems

Common special case:

Only main diagonal + 1 above and 1 below

4

3

2

1

4443

343332

232221

1211

00

0

0

00

b

bb

b

aa

aaaaaa

aa


21/41

Solving Tridiagonal Systems

When solving using Gauss-Jordan:

Constant # of multiplies/adds in each row

Each row only affects 2 others

4

3

2

1

4443

343332

232221

1211

00

0

0

00

b

b

b

b

aa

aaa

aaa

aa


22/41

Running Time 2n loops, 4 multiply/adds per loop

(assuming correct bookkeeping)

This running time has a fundamentally different dependence

on n: linear instead of cubic

Can say that tridiagonal algorithm is O(n) while

Gauss-Jordan is O(n3)


23/41

Big-O Notation Informally, O(n3) means that the dominant term for large n is

cubic

More precisely, there exist a c and n0 such that

running time c n3

if

n > n0

This type ofasymptotic analysis is often usedto characterize different algorithms


24/41

Triangular SystemsAnother special case: A is lower-triangular

4

3

2

1

44434241

333231

2221

11

0

00

000

b

b

b

b

aaaa

aaa

aa

a


25/41

Triangular Systems Solve by forward substitution

4

3

2

1

44434241

333231

2221

11

0

00

000

b

b

b

b

aaaa

aaa

aa

a

11

1

1

a

bx


26/41

Triangular Systems

Solve by forward substitution

4

3

2

1

44434241

333231

2221

11

0

00

000

b

b

b

b

aaaa

aaa

aa

a

22

1212

2

a

xabx


27/41

Triangular Systems Solve by forward substitution

4

3

2

1

44434241

333231

2221

11

0

00

000

b

bb

b

aaaa

aaaaa

a

33

2321313

3

a

xaxabx


28/41

Triangular Systems

If A is upper triangular, solve by backsubstitution

5

4

3

2

1

55

4544

353433

25242322

1514131211

0000

000

00

0

b

b

b

b

b

a

aa

aaa

aaaa

aaaaa

55

5

5

a

bx


29/41

Triangular Systems If A is upper triangular, solve by backsubstitution

5

4

3

2

1

55

4544

353433

25242322

1514131211

0000

000

00

0

b

b

b

b

b

a

aa

aaa

aaaa

aaaaa

44

5454

4

a

xabx


30/41

Triangular Systems Both of these special cases can be solved in

O(n2) time This motivates a factorization approach to solving arbitrary

systems:

Find a way of writing A as LU, where L and U are both triangular

Ax=b LUx=b Ly=b Ux=y

Time for factoring matrix dominates computation


31/41

Cholesky Decomposition

For symmetric matrices, choose U=LT

Perform decomposition

Ax=b LLTx=b Ly=b LTx=y

33

3222

312111

333231

2221

11

332313

232212

131211

00

00

00

l

ll

lll

lll

ll

l

aaa

aaa

aaa


32/41


22

312123

322332223121

2

21222222

2

22

2

21

11

13

31133111

11

12

21122111

111111

2

11

l

lla

lallll

lalall

lalall

l

alall

alal

33

3222

312111

333231

2221

11

332313

232212

131211

00

00

00

l

ll

lll

lll

ll

l

aaa

aaa

aaa


33/41


ii

i

k

jkikij

ji

i

k

ikiiii

l

lla

l

lal

1

1

1

1

2

33

3222

312111

333231

2221

11

332313

232212

131211

00

00

00

l

ll

lll

lll

ll

l

aaa

aaa

aaa


34/41

Cholesky Decomposition This fails if it requires taking square root of a negative

number Need another condition on A: positive definite

For any v, vTA v > 0

(Equivalently, all positive eigenvalues)


35/41

Cholesky Decomposition Running time turns out to be 1/6n

3

Still cubic, but much lower constant

Result: this is preferred method for solving

symmetric positive definite systems


36/41

LU Decomposition

Again, factor A into LU, where

L is lower triangular and U is upper triangular

Last 2 steps in O(n2) time, so total time dominated by

decomposition

Ax=b

LUx=b

Ly=b

Ux=y


37/41

Crouts Method

More unknowns than equations!

Let all lii=1

33

2322

131211

333231

2221

11

333231

232221

131211

00

00

00

u

uu

uuu

lll

ll

l

aaa

aaa

aaa


38/41

Crouts Method

33

2322

131211

3231

21

333231

232221

131211

00

0

1

01

001

u

uu

uuu

ll

l

aaa

aaa

aaa

22

123132

323222321231

1221222222221221

1212

11

31

31311131

11

21

21211121

1111

u

ula

laulul

ulauauul

au

u

a

laul

u

alaul

au


39/41

Crouts Method

For i = 1..n

For j = 1..i

For j = i+1..n

33

2322

131211

3231

21

333231

232221

131211

00

0

1

01

001

u

uu

uuu

ll

l

aaa

aaa

aaa

1

1

j

k

kijkjiji ulau

ii

i

k

kijkji

ji

u

ula

l

1

1


40/41

Crouts Method Interesting note: # of outputs = # of inputs,

algorithm only refers to elements not output yet

Can do this in-place!

Algorithm replaces A with matrix

of l and u values, 1s are implied Resulting matrix must be interpreted in a special way: not a regular

matrix

Can rewrite forward/backsubstitution routines to use this packed l-u

matrix

333231

232221

131211

ull

uul

uuu


41/41

LU Decomposition Running time is 1/3n

3

Only a factor of 2 slower than symmetric case

This is the preferred general method for

solving linear equations

Pivoting very important

Partial pivoting is sufficient, and widely implemented

Linear System - Classic

Documents

Transcript of Linear System - Classic