Solve two-step equations Solve equations with variables on both sides
Numerical methodsfyzikazeme.sk/mainpage/stud_mat/nm/lecture3.pdf · – row switching 4 ... we...
Transcript of Numerical methodsfyzikazeme.sk/mainpage/stud_mat/nm/lecture3.pdf · – row switching 4 ... we...
Lecture 3
Numerical methodsSystem of linear equations
Lecture 3
OUTLINE
1. System of linear equations (linear system)2. Gaussian elimination3. Pivoting4. LU decomposition5. Cholesky decomposition6. Effect of round-off errors7. Conditionality8. System of homogeneous linear equations
Basic definitions
System of linear equations (linear system)
1 11 1 1
1 21 2 2
1 1
a aa a
a a
n n
n n
m n mn m
x x bx x b
x x b
+ + =
+ + =
+ + =
We can assign two matrixes to the system (1):- coefficient matrix - augmented matrix
111 12 1n
221 22 2n
m1 m2 mn
a a aa a a
A
a a a m
bb
b
æ ö÷ç ÷ç ÷ç ÷ç ÷¢ ç= ÷ç ÷ç ÷÷ç ÷ç ÷çè ø
11 12 1n
21 22 2n
m1 m2 mn
a a aa a a
A
a a a
æ ö÷ç ÷ç ÷ç ÷ç ÷ç= ÷ç ÷ç ÷÷ç ÷ç ÷çè ø
(1)
Denote
1
2
n
xx
x
æ ö÷ç ÷ç ÷ç ÷ç ÷ç= ÷ç ÷ç ÷÷ç ÷ç ÷çè ø
x
1
2
m
bb
b
æ ö÷ç ÷ç ÷ç ÷ç ÷ç= ÷ç ÷ç ÷÷ç ÷ç ÷çè ø
b
Then the linear system (1) reads
Ax = b
System of linear equations (linear system)
1 111 1n
2 221 2n
m1 mn
a aa a
a a n m
x bx b
x b
æ ö æ öæ ö ÷ ÷÷ç çç ÷ ÷÷ç çç ÷ ÷÷ç çç ÷ ÷÷ç çç ÷ ÷÷ç çç =÷ ÷÷ç çç ÷ ÷÷ç çç ÷ ÷÷ ÷ ÷÷ç çç ÷ ÷÷ç çç ÷ ÷÷ç ççè øè ø è ø
If b = then the system (1) is called homogeneous
if b then it is called nonhomogeneous
0
00
0
00
System of linear equations (linear system)
Definition We say that the n-tuple (r1, …, rn) is solution of system (1), if Ar = b, where
r = .
n
2
1
r
rr
Note
System (1) is consistent, if it has at least one solution; otherwise it is inconsistent.
System of linear equations (linear system)
Direct methods
attempt to solve the problem by a finite sequence of operations.In the absence of rounding errors,
direct methods would deliver an exact solution.
System of linear equations (linear system)
Iterative methods
delivers only approximate solution.The number of steps depends
on required precision.
x-2y+4z+ t =-62x+3y- z+2t =13 | (2)-2*(1)2x+5y+ z+ t = 8 | (3)-2*(1)3x+ y+3z+ t = 1 | (4)-3*(1)
x-2y+4z+ t =-6 7y-9z =259y-7z- t =20 | (3)-9/7*(2)7y-9z-2t =19 | (4)-7/7*(2)
Example: Find the solution of linear system in real numbers.
x-2y+ 4z+ t = -6 7y -9z = 2532/7z- t =-85/7
-2t = -6
x = 1, y = 1, z = - 2, t = 3
Definition: Elementary row operations (ERO)on a matrix are:
– row switching ↔
– row multiplication →
– row addition →
Elementary row operations
Definition: We say that matrix A of type m x n isrow equivalent with the matrix B of type m x nif it is possible to transform A into Bby a sequence of elementary row operations.
Row equivalence of matrices
Definition: We say that squared matrix U is upper triangular (or right triangular)if all the entries below the main diagonal are zero.
Upper triangular matrix
Theorem: Each matrix is row equivalentwith some upper triangular matrix.
Row equivalence of matrices
Definition: The rank of matrix A is the number of nonzero rowsof triangular matrix equivalent to the matrix A, we will denote it h(A).
or
The rank of A is the maximal number of linearly independent rows of A.
Rank of a matrix
Consequence: Row equivalent matrices have the same rank.
x-2y+4z+ t =-62x+3y- z+2t =13 /(2)-2*(1)2x+5y+ z+ t = 8 /(3)-2*(1)3x+ y+3z+ t = 1 /(4)-3*(1)
x-2y+4z+ t = -6 7y-9z = 259y-7z- t = 20 /(3)-9/7*(2)7y-9z-2t = 19 /(4)-7/7*(2)
Example: Find the solution of linear system in real numbers.
18
136
1152
21321421
1313
1 2 4 10 7 9 00 9 7 10 7 9 2
æ ö- ÷ç ÷ç ÷ç - ÷ç ÷ç ÷ç ÷- -ç ÷÷ç ÷ç ÷ç - -è ø
-6252019
x-2y+ 4z+ t = -6 7y- 9z = 2532/7z- t =-85/7
-2t = -6
1 2 4 10 7 9 00 0 32 / 7 10 0 0 2
æ ö- ÷ç ÷ç ÷ç - ÷ç ÷ç ÷ç ÷-ç ÷÷ç ÷ç ÷ç -è ø
-625-85/7-6
x = 1, y = 1, z = - 2, t = 3
Gaussian elimination
We showed a principle ofGaussian elimination (GE):
Using the finite number of elementary row operationson the coefficient matrix we getthe upper triangular matrix
andusing back substitution we calculate the vector of unknowns.
Gaussian elimination
We showed a principle ofGaussian elimination (GE):
Using the finite number of elementary row operationson the coefficient matrix we getthe upper triangular matrix
andusing back substitution we calculate the vector of unknowns.
Forward elimination:
LetA(0) = A, b(0) = b
elements of matrix A(0) will be aij(0) = aij
elements of vector b(0) will be bi(0) = bi
Gaussian elimination
algorithm of GE
Gaussian elimination
multiplicator assuresthat
element (i,k) of matrix A(k)
will be zero
algorithm of GE
Gaussian elimination
leading coefficient(pivot)
has to be non-zero
algorithm of GE
Gaussian elimination
Example:Solve the linear system
on the hypothetical computer, which works in decimal system
with five digits mantissa.
The exact solution is
Gaussian elimination
Gaussian elimination
Gaussian elimination – partial pivoting
Partial pivotingis the modification of GE which assures
that the absolute values of multiplicatorsare less than or equal to 1.
The algorithm selects the entry with largest absolute value
from the column of the matrix that is currently being considered
as the pivot element.
Gaussian elimination – partial pivoting
Gaussian elimination – complete (maximal) pivoting
Pivoting
Partial or complete pivoting?usually partial pivoting is enough
Partial pivoting is generally sufficient to adequately reduce round-off error.
Complete pivoting is usually not necessary to ensure numerical stability and,
due to the additional cost of searching for the maximal element, the improvement in numerical stability
is typically outweighed by its reduced efficiency
for all but the smallest matrices.
Gaussian elimination
The number of arithmetic operations in forward elimination: 323
n»
The number of arithmetic operations in back substitution : 2n»
Computational efficiency
x-2y+4z+ t =-62x+3y- z+2t =13 /(2)-2*(1)2x+5y+ z+ t = 8 /(3)-2*(1)3x+ y+3z+ t = 1 /(4)-3*(1)
x-2y+4z+ t = -6 7y-9z = 259y-7z- t = 20 /(3)-9/7*(2)7y-9z-2t = 19 /(4)-7/7*(2)
18
136
1152
21321421
1313
1 2 4 10 7 9 00 9 7 10 7 9 2
æ ö- ÷ç ÷ç ÷ç - ÷ç ÷ç ÷ç ÷- -ç ÷÷ç ÷ç ÷ç - -è ø
-6252019
x-2y+ 4z+ t = -6 7y- 9z = 2532/7z- t =-85/7
-2t = -6
1 2 4 10 7 9 00 0 32 / 7 10 0 0 2
æ ö- ÷ç ÷ç ÷ç - ÷ç ÷ç ÷ç ÷-ç ÷÷ç ÷ç ÷ç -è ø
-625-85/7-6
x = 1, y = 1, z = - 2, t = 3
LU decomposition (factorization)
x-2y+4z+ t =-62x+3y- z+2t =13 /(2)-2*(1)2x+5y+ z+ t = 8 /(3)-2*(1)3x+ y+3z+ t = 1 /(4)-3*(1)
x-2y+4z+ t = -6 7y-9z = 259y-7z- t = 20 /(3)-9/7*(2)7y-9z-2t = 19 /(4)-7/7*(2)
18
136
1152
21321421
1313
1 2 4 17 9 09 7 17 9 2
223
æ ö- ÷ç ÷ç ÷ç - ÷ç ÷ç ÷ç ÷- -ç ÷÷ç ÷ç ÷ç - -è ø
-6252019
x-2y+ 4z+ t = -6 7y- 9z = 2532/7z- t =-85/7
-2t = -6
1 2 4 17 9 00 32 / 7 10 0
22
23
æ ö- ÷ç ÷ç ÷ç - ÷ç ÷ç ÷ç ÷-ç ÷÷ç ÷ç ÷ç -è ø
-625-85/7-6
x = 1, y = 1, z = - 2, t = 3
LU decomposition (factorization)
x-2y+4z+ t =-62x+3y- z+2t =13 /(2)-2*(1)2x+5y+ z+ t = 8 /(3)-2*(1)3x+ y+3z+ t = 1 /(4)-3*(1)
x-2y+4z+ t = -6 7y-9z = 259y-7z- t = 20 /(3)-9/7*(2)7y-9z-2t = 19 /(4)-7/7*(2)
18
136
1152
21321421
1313
1 2 4 17 9 09 7 17 9 2
223
æ ö- ÷ç ÷ç ÷ç - ÷ç ÷ç ÷ç ÷- -ç ÷÷ç ÷ç ÷ç - -è ø
-6252019
x-2y+ 4z+ t = -6 7y- 9z = 2532/7z- t =-85/7
-2t = -6
22 9 / 73 1
1 2 4 17 9 0
32 / 7 10 2
æ ö- ÷ç ÷ç ÷ç - ÷ç ÷ç ÷ç ÷-ç ÷÷ç ÷ç ÷ç -è ø
-625-85/7-6
x = 1, y = 1, z = - 2, t = 3
LU decomposition (factorization)
x-2y+4z+ t =-62x+3y- z+2t =13 /(2)-2*(1)2x+5y+ z+ t = 8 /(3)-2*(1)3x+ y+3z+ t = 1 /(4)-3*(1)
x-2y+4z+ t = -6 7y-9z = 259y-7z- t = 20 /(3)-9/7*(2)7y-9z-2t = 19 /(4)-7/7*(2)
18
136
1152
21321421
1313
1 2 4 17 9 09 7 17 9 2
223
æ ö- ÷ç ÷ç ÷ç - ÷ç ÷ç ÷ç ÷- -ç ÷÷ç ÷ç ÷ç - -è ø
-6252019
x-2y+ 4z+ t = -6 7y- 9z = 2532/7z- t =-85/7
-2t = -6
22 9 / 73 1
1 2 4 17 9 0
32 / 7 120
æ ö- ÷ç ÷ç ÷ç - ÷ç ÷ç ÷ç ÷-ç ÷÷ç ÷ç ÷ç -è ø
-625-85/7-6
x = 1, y = 1, z = - 2, t = 3
LU decomposition (factorization)
LU decomposition (factorization)
Multiplicators mij from the forward elimination are placedinto lower triangular matrix L)
LU decomposition (factorization)
1 2 4 1 1 0 0 0 1 2 4 12 3 1 2 2 1 0 0 0 7 9 02 5 1 1 2 9 / 7 1 0 0 0 32 / 7 13 1 3 1 3 1 0 1 0 0 0 2
æ ö æ ö æ ö- -÷ ÷ ÷ç ç ç÷ ÷ ÷ç ç ç÷ ÷ ÷ç ç ç- -÷ ÷ ÷ç ç ç÷ ÷ ÷ç ç ç= ⋅÷ ÷ ÷ç ç ç÷ ÷ ÷-ç ç ç÷ ÷ ÷÷ ÷ ÷ç ç ç÷ ÷ ÷ç ç ç÷ ÷ ÷ç ç ç -è ø è ø è øL UA
= ⋅A L U
We obtained LU decomposition of matrix A (so called Doolittle version – ones on the diagonal of matrix L)
because
Doolittle algorithm makes the LU decomposition
using the half number of arithmetic operations
313
n»
LU decomposition (factorization)
LU decomposition of Ais useful for solving of sequences of tasks
if the new right-hand-side bi could be estimated only afterwe solve the previous linear systems
for k < i.
To solve the system means to solve
LU decomposition with partial pivoting
LU decomposition with partial pivotingis standard routine of each library of codes.
Input parameter is matrix A.
Output are three matrices:L, U and permutation matrix P
whereLU = PA.
Permutation matrix comes form identity matrix after row exchange.
Solution of Ax=b is given by solution of
(show this)
LU decomposition with partial pivoting
Instead of matrix P we work withthe vector of row permutations p.
At the beginning we put
and we just exchange the elements of vector p.
Gaussian elimination
Definition: We say that square matrix Ais diagonally dominant if
Gaussian elimination
Definition: Symmetric matrix A is positive-definite if
for any non-zero vector x it holds
0Tx x⋅ ⋅ >A
If A is regular then
is positive-definite.
TA A⋅
Gaussian elimination
Sylvester’s criterion: Symmetric matrix Ais positive-definite, if and only if the leading principal minors are positive, i.e. if
Gaussian elimination
Gaussian elimination is numerically stable for diagonally dominant
or positive-definite matrices.
Cholesky decomposition
TA L L= ⋅
Cholesky decomposition theorem: There is only one way, how the
decompose the positive-definite matrix A into
where L is the lower triangular matrix.
Cholesky decomposition
TA L L= ⋅
Cholesky decomposition theorem: There is only one way, how the
decompose the positive-definite matrix A into
where L is the lower triangular matrix.
Cholesky algorithm:
The number of arithmetic operations: 313
n»
The effect of round-off errors
Example:On the hypothetical computer working decimal systemwith three digits mantissa solve the following system
The effect of round-off errors
GE with partial pivotingassures the small residua
However, the small residuum does not givean estimation of the error of solution
Conditionality
In order to evaluate the conditionality of problem„to find solution x of system Ax = b“
we need to assesthe effect of change of A a b
to the solution x.
Conditionality
Norm of the vectorclass of vector norms lp,
The most often we use p=1, p=2, or p
Manhattan Euclid Chebyshev
Conditionality
Properties
1. av = | a | v , (absolute homogeneity or absolute scalability)
2. u + v ≤ u v (triangle inequality or subadditivity)
3. If v = 0 then v is the zero vector (separates points)
Conditionality
Condition number of a matrix
Let
where maximum and minimum is considered for all nonzero x.Ratio M/m is called condition number of matrix A:
Conditionality
Lets consider linear systemAx = b
and other system, which could be obtained by changing the r.h.s.:, or
we can consider as an error of band
corresponding error of solution x.
Conditionality
Because thenfrom the definition of M and m
it follows that
so for
where is a residuum.
Condition number of matrix acts asan amplifier of relative error !
It is possible to show, that also changes in matrix Aleads to the similar effects on results.
Conditionality
Norm of matrixthe number M is also known as the norm of matrix
and also it holds
Equivalently we could write the condition number of matrix as
Conditionality
“Entrywise” norm of matrix
2
1 1
n n
ijFi j
aA= =
= åå Frobenius norm (consistent with l2)
( )
1
1 1
n n ppijp p
j jvec a
= =
æ ö÷ç ÷ç= = ÷ç ÷ç ÷è øååA A
,max iji j
a¥
=A max norm
x+2y+ z- t = 12x+3y- z+2t = 34x+7y+ z = 55x+7y-4z+7t =10
Example: Find the solution of linear system in real numbers.
x+2y+ z- t = 12x+3y- z+2t = 34x+7y+ z = 55x+7y-4z+7t =10
Example: Find the solution of linear system in real numbers.
10531
0174
21321121
74-75
5111
3 4310
43101121
1290
0211
0 0000
43101121
000
x+2y+ z- t = 1-y-3z+4t = 1
0z+0t = 20t = 0
system has no solution
x-2y+4z+ t =-62x+3y- z+2t =133x+ y+3z+3t = 7x+5y-5z+ t =19
Another example: Find the solution of linear system in rational numbers.
x-2y+4z+ t =-62x+3y- z+2t =133x+ y+3z+3t = 7x+5y-5z+ t =19
197
136-
3313
21321421
15-51
2525256-
7 0970
09701421
090
x-2y+4z+ t =-67y-9z+0t =25
0z+0t = 00t = 0
00
256-
00 0000
09701421
00
Another example: Find the solution of linear system in rational numbers.
x-2y+4z+ t =-67y-9z+0t =25
0z+0t = 00t = 0
System has infinite number of solutions
00
256-
00 0000
09701421
00
t = p, z = q, p,q Q
q79
725y pq
710
78x
Qqp, ,pq,q,
79
725p,q
710
78S
For example: if q = 1, p = 0, we get particular solution
0,1,
734,
72
Another example: Find the solution of linear system in rational numbers.
Frobenius theorem: System of nonhomogeneous equationshas solutionif and only if, the rank of coefficient matrixis equal to the rank of augmented matrix.
Consequence 1: If h(A) = h(A´) = n (n is the number of unknowns), then the system has just one solution.
Consequence 2: If h(A) = h(A´) < n (n is the number of unknowns), then the system has infinite number of solutionsandn-h unknowns could be freely chosen.
Consequence 3: If h(A) h(A´) then the system has no solution.
System of linear equations
System of homogeneous linear equations
0axax
0axax0axax
mnnm11
2nn211
1nn111
The system has always solution because h(A)=h(A´)
System of homogeneous linear equations
Theorem: System of homogeneous equations has nontrivial solutionif and only if h(A) n.(trivial solution is (0,0,...,0) )
x-2y+4z+ t =02x+3y- z+2t =03x+ y+3z+3t =0x+5y-5z+ t =0
0 0 3313
21321421
15-51
00
0 70 0970
09701421
090
00
0000
00 0000
09701421
00
x-2y+4z+ t =07y-9z+0t =0
0z+0t =00t =0
Example: Find the solution of linear system in rational numbers.
t = p, z = q, p,q Q
q79y
pq7
10x
Qqp, ,pq,q,
79p,q
710S
Example: Find the solution of linear system in rational numbers.
x-2y+4z+ t =07y-9z+0t =0
0z+0t =00t =0
Summary of notions
• homogeneous and nonhomogeneous system of linear equations
• coefficient matrix and augmented matrix
• elementary row operations
• row equivalent matrices
• rank of a matrix
• upper and lower triangular matrix
• Gaussian elimination (pivoting)
• LU decomposition (Doolittle method)
• Cholesky decomposition
• Frobenius theorem
• solution of homogeneous linear systems