7.4 – SOLVING SYSTEMS OF LINEAR EQUATIONS USING A SUBSTITUTION STRATEGY SYSTEMS OF LINEAR EQUATIONS.
Systems of Linear Equations
-
Upload
emmanuel-garrett -
Category
Documents
-
view
68 -
download
2
description
Transcript of Systems of Linear Equations
![Page 1: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/1.jpg)
1
Systems of Linear Equations
Gauss-Jordan Elimination and
LU Decomposition
![Page 2: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/2.jpg)
2
Direct Methods
1. Gauss Eliminationa. Naïve Gauss Eliminiation
b. Pivoting Strategies
c. Scaling
2. Gauss-Jordan Elimination
3. LU Decomposition
![Page 3: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/3.jpg)
3
Gauss-Jordan Elimination
)3(3
)3(2
)3(1
)2(33
)2(3
)1(22
)2(2
11)2(
1
)2(3
)2(33
)2(2
)1(22
)2(111
)2(3
)2(33
)1(2
)1(23
)1(22
)1(1
)1(1311
)1(3
)1(33
)1(32
)1(2
)1(23
)1(22
1131211
3333231
2232221
1131211
1
1
1
1
1
1
c
c
c
bb
ab
ab
ba
ba
ba
ba
baa
baa
baa
baa
baaa
baaa
baaa
baaa
Similar to the Gauss elimination except
1. Elimination is applied to all equations (excluding the pivot equation) instead of just the subsequent equations.
2. All rows are normalized by dividing them by their pivot elements.
3. No back substitution is required.
![Page 4: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/4.jpg)
4
Pitfalls and Improvements
• Pitfalls: Same as those found in the Gauss elimination.– Division-by-zero, round-off error, and ill-
conditioned systems.
• Improvement strategies: Same as those used in the Gauss elimination – Use pivoting and scaling to avoid division-by-
zero and to reduce round-off error.
![Page 5: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/5.jpg)
5
Computing CostWhich of Gauss elimination and Gauss-Jordan elimination involves more FLOPS?
Gauss Elimination Gauss-Jordan Elimination
Elimination Step
Forward Elimination – only needs to eliminate the coefficients below the diagonal.
Cost ~ 2n3/3
Needs to eliminate coefficients below and above the diagonal.
Cost ~ 2 * (2n3/3)
Substitution Step
Back Substitution
Cost ~ O(n2)
No substitution step
Total 2n3/3 + O(n2) 4n3/3
(More costly when n is big)
![Page 6: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/6.jpg)
6
Question
What is
0
0
11A
333231
232221
1312111
333231
232221
131211
ˆˆˆ
ˆˆˆ
ˆˆˆ
aaa
aaa
aaa
A
aaa
aaa
aaa
A
0
0
1
ˆ
ˆ
ˆ
ˆ
ˆ
ˆ
0
0
1
ˆ
ˆ
ˆ
0
0
1
ˆ
ˆ
ˆ
0
0
1
31
21
11
31
21
11
31
21
111
31
21
111
a
a
a
A
a
a
a
AI
a
a
a
AAA
a
a
a
A
31
21
11
3
2
1
1
3
2
1
ˆ
ˆ
ˆ
ofcolumnfirstthegivessystemthisSolving
0
0
1
a
a
a
x
x
x
A
x
x
x
A
?
![Page 7: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/7.jpg)
7
Computing Matrix Inverse
• Gauss-Jordan
333231
232221
131211
333231
232221
131211
ˆˆˆ1
ˆˆˆ1
ˆˆˆ1
1
1
1
aaa
aaa
aaa
aaa
aaa
aaa
333231
232221
1312111
333231
232221
131211
ˆˆˆ
ˆˆˆ
ˆˆˆ
aaa
aaa
aaa
A
aaa
aaa
aaa
A
• Gauss Elimination
1
0
0
ˆ
ˆ
ˆ
0
1
0
ˆ
ˆ
ˆ
0
0
1
ˆ
ˆ
ˆ
33
23
13
32
22
12
31
21
11
a
a
a
A
a
a
a
A
a
a
a
A
Solve
Let
![Page 8: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/8.jpg)
8
Summary• Algorithms
– Gauss elimination• Forward elimination + back substitution
– Gauss-Jordan elimination• Calculate the computing cost of Gauss Elimination in
terms of FLOPS
• To avoid division-by-zero, use pivoting (partial or complete).
• To reduce round-off error, use pivoting and scaling.
• If a system is ill-conditioned, then substituting solution back to the system to check if the solution is accurate is not reliable.– How to determine if a matrix is ill-conditioned?
![Page 9: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/9.jpg)
9
Direct Methods
1. Gauss Eliminationa. Naïve Gauss Eliminiation
b. Pivoting Strategies
c. Scaling
2. Gauss-Jordan Elimination
3. LU Decomposition
![Page 10: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/10.jpg)
10
3x1 + 2x2 + x3 = 6
2x2 + 3x3 = 7
2x3 = 4
Step 1: Solve for x3
2x3 = 4 => x3 = 4/2 = 2
Step 2: Solve for x2
2x2 + 3(2) = 7
=> x2 = [7 – 3(2)] / 2 = 0.5
Step 3: Solve for x1
3x1 + 2(0.5) + (2) = 6
=> x1 = [6 – 2(0.5) – 2] / 3 = 1
Use to solve the system Ux = b where U is an upper triangular matrix.
33
2322
131211
00
0
u
uu
uuu
U
Back SubstitutionExample:
![Page 11: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/11.jpg)
11
2x1 = 4
3x1 + 2x2 = 7
x1 + 2x2 + 3x3 = 6
Step 1: Solve for x1
2x1 = 4 => x1 = 4/2 = 2
Step 2: Solve for x2
3(2) + 2x2 = 7
=> x2 = [7 – 3(2)] / 2 = 0.5
Step 3: Solve for x3
(2) + 2(0.5) + 3x3 = 6
x3 = [6 – 2 – 2(0.5)] / 3 = 1
Use to solve the system Lx = b where L is a lower triangular matrix.
Forward SubstitutionExample:
33231
2221
11
0
00
lll
ll
l
L
![Page 12: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/12.jpg)
12
LU DecompositionIdea: To solve Ax = b• Step 1: Decompose A into L and U such that
A = LU where
33
2322
131211
33231
2221
11
00
00
00
u
uu
uuu
U
lll
ll
l
L
We will discuss later how to compute L and U when given A. For now, let's assume we can!
• Step 2: Solve LUx = b using forward and back substitution.
![Page 13: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/13.jpg)
13
Solving LUx = b
)onsubstitutiback
usingCompute(:2
)onsubstitutiforward
usingCompute(:1
)(Let
xyUxStep
ybLyStep
Uxyby
UxL
![Page 14: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/14.jpg)
14
Decomposing A using Gauss Elimination
U is formed by applying the forward elimination of Gauss Elimination (without including the right hand side vector)
"33
'23
'22
131211
a
aa
aaa
U
22
3232
11
3131
11
2121
3231
21 '
',,where
1
01
001
a
af
a
af
a
af
ff
fL
L is formed by the factors used to eliminate each elements during the forward elimination process as
![Page 15: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/15.jpg)
15
Decomposing A into L and U -- Example
102.03.0
3.071.0
2.01.03
A
102.03.0
293333.000333.70
2.01.03
12,3
1.02121 rowfrowf
02.1019.00
293333.000333.70
2.01.03
13,3
3.03131 rowfrowf
1st iteration (1st column), eliminating a21
1st iteration (1st column), eliminating a22
![Page 16: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/16.jpg)
16
99996.92.03.0
3.070999999.0
2.01.03
LU
0120.1000
293333.000333.70
2.01.03
23,00333.7
19.03232 rowfrowf
2st iteration (2st column), eliminating a32
0120.1000
293333.000333.70
2.01.03
10271300.010000000.0
0103333333.0
001
UL
Thus we can express A as LU where
To verify
Decomposing A into L and U -- Example
Note: There are some round-off error.
![Page 17: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/17.jpg)
17
LU Decomposition• In LU decomposition, the l and u values are
obtained from
11
1
1
1
1
ijulaul
ijulau
j
kkjikijjjij
i
kkjikijij
Note: These are just mathematical expressions representing the steps involved in Gauss Elimination.
![Page 18: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/18.jpg)
18
Compact Representation
For U, the lower part are all 0's.
"333231
'23
'2221
131211
aff
aaf
aaa
1
01
001
3231
21
ff
fL
"33
'23
'22
131211
a
aa
aaa
U
For L, the diagonal elements are all 1's and the upper part are all 0's.
When implementing the algorithm, we can store both U and L back into A.
![Page 19: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/19.jpg)
19
Forward and Back Substitution
121for1 ,,nnia
xayx
a
yx
ii
n
ij jiji
inn
nn
To solve Ly = b, we use forward substitution
nibayyi
jjijii ...,,3,2for
1
1
To solve Ux = y, we use back substitution
"333231
'23
'2221
131211
aff
aaf
aaaNote: Here the aij represents the coefficients of the resulting matrix A produced by the LU Decomposition algorithm.
![Page 20: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/20.jpg)
20
Effect of Pivoting
333231
2322
13
0
00
aaa
aa
a
A
If we directly apply Gauss elimination algorithm (with pivoting), we will need to swap row 3 with row 1 and eventually yield L and U as
11
2322
333231
00
0
100
010
001
a
aa
aaa
UL
Let
Pivot row in 1st iteration
![Page 21: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/21.jpg)
21
Effect of Pivoting
If we are given a vector b, can we solve Ax = b as LUx = b?
11
2322
333231
333231
2322
13
00
0
100
010
001
0
00
a
aa
aaa
UL
aaa
aa
a
A
No! We can't because we have swapped the rows during the elimination step.
In implementing the LU decomposition, we have to remember how the rows were swapped so that we can reorder the elements in x and b accordingly.
![Page 22: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/22.jpg)
22
333231
232221
131211
aaa
aaa
aaa
333231
232221
131211
aaa
aaa
aaa
Addressing the issue of row swapping in implementation
i o[i]
1 1
2 2
3 3
To remember how the rows were swapped, we can introduce an array, o[] , to store the indexes to the rows. Initially, o[i] = i for i = 1, 2, …, N.
Whenever we need to swap rows, we only swap the content in o[]. For example, after swapping row 1 with row 3
i o[i]
1 3
2 2
3 1
![Page 23: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/23.jpg)
23
Addressing the issue of row swapping in implementation
If rows were swapped, A ≠A'. However, o[i] tells us that row i in A corresponds row o[i] in A'. To solve Ax = b, we can first reorder the elements in x and b to produce x' and b' as
x'i = xo[i] and b'i = bo[i]
and then solve A'x' = b' as L'U'x' = b'.
Note: In actual implementation, we do not need to explicitly create x' and b'. We can refer to their elements directly as xo[i] and bo[i].
AL'
U' A'
LU Decomposition with pivoting
=
![Page 24: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/24.jpg)
24
Illustration: Using o[]
i o[i]
1 3
2 2
3 1
25.0
5.0
75.05.1
5.03
124
31
21
31
21
f
f
f
f
221
142
124
124
142
221
1 2 2
2 4 1
4 2 1
Array A
i o[i]
1 1
2 2
3 3
1 2 2
2 4 1
4 2 1
Array A
After swapping row 3 with row 1
Initial states.
Pick row 3 as pivot row
i o[i]
1 3
2 2
3 1
0.25 1.5 0.75
0.5 3 0.5
4 2 1
Array A
f31 is calculated as A[o[3],1] / A[o[1],1] and the result is stored back to A[o[3],1].
After the 1st iteration.
Implementation
![Page 25: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/25.jpg)
25
Pseudocode for LU Decomposition
// Assume arrays start with index 1 instead of 0.// a: Coef. of matrix A; 2-D array. Upon successful// completion, it contains the coefficients of // both L and U.// b: Coef. of vector b; 1-D array// n: Dimension of the system of equations// x: Coef. of vector x (to store the solution)// tol: Tolerance; smallest possible scaled // pivot allowed.// er: Pass back -1 if matrix is singular. // (Reference var.) LUDecomp(a, b, n, x, tol, er) { Declare s[n] // An n-element array for storing scaling factors Declare o[n] // Use as indexes to pivot rows. // oi or o(i) stores row number of the ith pivot row. er = 0 Decompose(a, n, tol, o, s, er) if (er != -1) Substitute(a, o, n, b, x)}
![Page 26: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/26.jpg)
26
Pseudocode for decomposing A into L and UDecompose(a, n, tol, o, s, er) {
for i = 1 to n { // Find scaling factors
o[i] = i
s[i] = abs(a[i,1])
for j = 2 to n
if (abs(a[i,j]) > s[i])
s[i] = abs(a[i,j])
}
for k = 1 to n-1 { Pivot(a, o, s, n, k) // Locate the kth pivot row
// Check for singular or near-singular cases if (abs(a[o[k],k]) / s[o[k]]) < tol) { er = -1 return } // Continue next page
![Page 27: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/27.jpg)
27
for i = k+1 to n { factor = a[o[i],k] / a[o[k],k]
// Instead of storing the factors // in another matrix (2D array) L, // We reuse the space in A to store // the coefficients of L. a[o[i],k] = factor // Eliminate the coefficients at column j // in the subsequent rows for j = k+1 to n a[o[i],j] = a[o[i],j] – factor *
a[o[k],j] }
} // end of "for k" loop from previous page
// Check for singular or near-singular cases if (abs(a[o[n],n]) / s[o[n]]) < tol) er = -1}
![Page 28: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/28.jpg)
28
Psuedocode for finding the pivot rowPivot(a, o, s, n, k) { // Find the largest scaled coefficient in column k p = k // p is the index to the pivot row big = abs(a[o[k],k]) / s[o[k]]) for i = k+1 to n { dummy = abs(a[o[i],k] / s[o(i)]) if (dummy > big) { big = dummy p = i } }
// Swap row k with the pivot row by swapping the // indexes. The actual rows remain unchanged dummy = o[p] o[p] = o[k] o[k] = dummy}
![Page 29: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/29.jpg)
29
Psuedocode for solving LUx = bSubstitute(a, o, n, b, x) { Declare y[n] y[o[1]] = b[o[1]] for i = 2 to n { sum = b[o[i]] for j = 1 to i-1 sum = sum – a[o[i],j] * b[o[j]] y[o[i]] = sum }
x[n] = y[o[n]] / a[o[n],n] for i = n-1 downto 1 { sum = 0 for j = i+1 to n sum = sum + a[o[i],j] * x[j] x[i] = (y[o[i]] – sum) / a[o[i],i] }}
Forwardsubstitution
Backsubstitution
![Page 30: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/30.jpg)
30
About the LU Decomposition Pseudocode
• If the LU decomposition algorithm is to be used to solve Ax = b for various b's, then both array "a" and "o" have to be saved.
• Subroutine substitute() can be applied repeatedly for various b's.
• In subroutine substitute(), we can also reuse the space in "b" to store the elements of "y" instead of declaring the new array "y".
![Page 31: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/31.jpg)
31
Question
What is the cost to decompose A into L and U?
![Page 32: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/32.jpg)
32
LU Decomposition vs. Gauss Elimination
To solve Ax = bi, i = 1, 2, 3, …, K
LU Decomposition
Compute L and U once – O(n3)
Forward and back subsitution – O(n2)
Total cost = O(n3) + K * O(n2)
Gauss Elimination
Total cost: K * O(n3)
![Page 33: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/33.jpg)
33
Computing Matrix Inverse
333231
232221
1312111
333231
232221
131211
ˆˆˆ
ˆˆˆ
ˆˆˆ
aaa
aaa
aaa
A
aaa
aaa
aaa
A
1
0
0
ˆ
ˆ
ˆ
0
1
0
ˆ
ˆ
ˆ
0
0
1
ˆ
ˆ
ˆ
33
23
13
32
22
12
31
21
11
a
a
a
LU
a
a
a
LU
a
a
a
LU
Solve
Let
First, decompose A into L and U, then
![Page 34: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/34.jpg)
34
Pseudocode for finding matrix inverse using LU Decomposition// a: Given matrix A; 2-D array.// ai: Stores the coefficients of A-1
// n: Dimension of A// tol: Smallest possible scaled pivot allowed.// er: Pass back -1 if matrix is singular.LU_MatrixInverse(a, ai, n, tol, er) { Declare s[n], o[n], b[n], x[n]
Decompose(a, n, tol, o, s, er) if (er != -1) { for i = 1 to n { for j = 1 to n // Construct the unit vector b[j] = 0 b[i] = 1 Substitute(a, o, n, b, x)
for j = 1 to n ai[j,i] = x[j] } }}
![Page 35: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/35.jpg)
35
Two types of LU decomposition• Doolittle Decomposition or factorization
nn
n
n
nnnnnn
n
n
u
uu
uuu
ll
l
aaa
aaa
aaa
A
00
0
1
01
001
222
11211
21
21
21
22221
11211
• Crout Decomposition
100
10
1
0
00
2
112
21
2221
11
21
22221
11211
n
n
nnnnnnnn
n
n
u
uu
lll
ll
l
aaa
aaa
aaa
A
![Page 36: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/36.jpg)
36
Crout Decomposition
1
1
1
1
1
1
1
11
11
11
1for
for
1...,,3,2for
...,,2for
...,,2,1for
n
kknnknnnn
j
iikjijkjjjk
j
kkjikijij
jj
ii
ulal
jkulalu
jiulal
nj
njl
au
nial
![Page 37: Systems of Linear Equations](https://reader035.fdocuments.in/reader035/viewer/2022062221/56812a85550346895d8e208c/html5/thumbnails/37.jpg)
37
Summary
• LU Decomposition Algorithm
• LU Decomposition vs. Gauss Elimination– Similarity– Advantages (Disadvantages?)– Computing Cost