Review of Linear Algebra (1)
Transcript of Review of Linear Algebra (1)
-
8/19/2019 Review of Linear Algebra (1)
1/45
Chapter 1: Linear Algebra
- Vector space
-Linear transformation
- Equivalances for invertibility matrices
-
8/19/2019 Review of Linear Algebra (1)
2/45
Scalar
• A quantity (variable), described by a singlereal number
Linear Algebra & Matrices, MfD 2010
e.g. Intensity of each
voxel in an MRI scan
-
8/19/2019 Review of Linear Algebra (1)
3/45
Vector • Not a physics vector (magnitude, direction)
Linear Algebra & Matrices, MfD 2010
xn
x x21
i.e. A column
of numbers
VECTOR=
EXAMPLE:
-
8/19/2019 Review of Linear Algebra (1)
4/45
Matrices• Rectangular display of vectors in rows and columns
• Can inform about the same vector intensity at different
times or different voxels at the same time
• Vector is just a n x 1 matrix
Linear Algebra & Matrices, MfD 2010
333231232221
131211
x x x x x x
x x x
-
8/19/2019 Review of Linear Algebra (1)
5/45
Matrices
Square (3 x 3)
333231
232221
131211
d d d
d d d
d d d
D
Linear Algebra & Matrices, MfD 2010
Matrix locations/size defined as rows x columns (R x C)
d i j : ith row, jth column
333231232221
131211
x x x x x x
x x x
Rectangular (3 x 2)
333231
232221
131211
x x x
x x x
x x x
333231
232221
131211
x x x
x x x
x x x
333231
232221
131211
x x x
x x x
x x x
333231
232221
131211
x x x
x x x
x x x
963
852
741
A
6
5
4
3
2
1
A
3 dimensional (3 x 3 x 5)
-
8/19/2019 Review of Linear Algebra (1)
6/45
Basic concepts
• Vector in Rn is an ordered
set of n real numbers.
– e.g. v = (1,6,3,4) is in R4
– “(1,6,3,4)” is a column
vector: – as opposed to a row
vector:
• m-by-n matrix is an object
with m rows and n columns,
each entry fill with a real
number:
4
3
6
1
4361
239
6784821
-
8/19/2019 Review of Linear Algebra (1)
7/45
Matrices in MATLAB
963
5
4
11
11
0
0
0
Linear Algebra & Matrices, MfD 2010
963
852
741
X X=[1 4 7;2 5 8;3 6 9]Matrix(X)
;=end of a row
Description Type into MATLAB Meaning
2nd Element of 3rd column X(2,3) 8
X(3, :)3rd row
(X(row,column))Reference matrix values
Note the : refers to all of row or column and , is the divider between rows and columns
Elements 2&3 of column 2 X( [1 2], 2)
Special types of matrix
All zeros size 3x1
All ones size 2x2
zeros(3,1)
ones(2,2)
-
8/19/2019 Review of Linear Algebra (1)
8/45
Transposition
column row row column
Linear Algebra & Matrices, MfD 2010
9
4
3T d
476
145
321
A
413
742
651
T
A
2
1
1
b 211T b 943d
-
8/19/2019 Review of Linear Algebra (1)
9/45
Basic concepts
• Transpose: reflect vector/matrix on line:
bab
a T
d b
ca
d c
ba T
– Note: (Ax)T=xT AT (We’ll define multiplication soon…)
• Vector norms:
– Lp norm of v = (v1,…,vk) is (Σi |vi|p)1/p
– Common norms: L1, L2
– Linfinity = maxi |vi|
• Length of a vector v is L2(v)
-
8/19/2019 Review of Linear Algebra (1)
10/45
Matrix Calculations
Addition – Commutative: A+B=B+A
– Associative: (A+B)+C=A+(B+C)
Subtraction
- By adding a negative matrix
6543
15320412
1301
5242
BA
Linear Algebra & Matrices, MfD 2010
-
8/19/2019 Review of Linear Algebra (1)
11/45
Scalar multiplication
Scalar * matrix = scalar multiplication
Linear Algebra & Matrices, MfD 2010
-
8/19/2019 Review of Linear Algebra (1)
12/45
Matrix Multiplication“When A is a m x n matrix & B is a k x l matrix, AB is only possible
if n =k . The result will be an m x l matrix ”
A 1 A 2 A 3
A 4 A 5 A 6
A 7 A 8 A 9
A 10 A 11 A 12
m
n
x
B13 B14
B15 B16
B17 B18
l
k
Simp ly put, can ONLY perform A *B IF:
Number of columns in A = Number of rows in B
= m x l matrix
Linear Algebra & Matrices, MfD 2010
Hint:
1) If you see this message in MATLAB:
??? Error using ==> mtimes
Inner matrix dimensions must agree -Then columns in A is not equal to rows in B
-
8/19/2019 Review of Linear Algebra (1)
13/45
Matrix multiplication
• Multiplication method:Sum over product of respective rows and columns
Matlab does all this for you!Simply type: C = A * B
Linear Algebra & Matrices, MfD 2010
Hints:
1) You can work out the size of the output (2x2). In MATLAB, if you pre-allocate a
matrix this size (e.g. C=zeros(2,2)) then the calculation is quicker
-
8/19/2019 Review of Linear Algebra (1)
14/45
Matrix multiplication
• Matrix multiplication is NOT commutativei.e the order matters!
– AB≠BA
• Matrix multiplication IS associative
– A(BC)=(AB)C
• Matrix multiplication IS distributive
– A(B+C)=AB+AC
– (A+B)C=AC+BCLinear Algebra & Matrices, MfD 2010
-
8/19/2019 Review of Linear Algebra (1)
15/45
Identity matrix A special matrix which plays a similar role as the number 1 in number
multiplication?
For any nxn matrix A , we have A I n = I n A = A
For any nxm matrix A , we have I n A = A , and A I
m= A (so 2 possible matrices)
Linear Algebra & Matrices, MfD 2010
If the answers always A, why use an identity matrix?
Can’t divide matrices, therefore to solve may problems have to use the inverse. The
identity is important in these types of calculations.
-
8/19/2019 Review of Linear Algebra (1)
16/45
Identity matrix
1 2 3 1 0 0 1+0+0 0+2+0 0+0+3
4 5 6 X 0 1 0 = 4+0+0 0+5+0 0+0+6
7 8 9 0 0 1 7+0+0 0+8+0 0+0+9
Worked
example A I 3 = A
for a 3x3 matrix:
• In Matlab: eye(r, c) produces an r x c identity matrix
Linear Algebra & Matrices, MfD 2010
-
8/19/2019 Review of Linear Algebra (1)
17/45
Basic concepts
• Vector products:
– Dot product:
– Outer product:
22112
1
21 vuvuv
vuuvuvu T
2212
2111
21
2
1
vuvu
vuvuvv
u
uuvT
-
8/19/2019 Review of Linear Algebra (1)
18/45
Matrices as linear transformations
5
5
1
1
50
05(stretching)
1
1
1
1
01
10(rotation)
-
8/19/2019 Review of Linear Algebra (1)
19/45
Matrices as linear transformations
10
01
0110 (reflection)
0
1
1
1
00
01(projection)
(shearing)
y
cy x
y
xc
10
1
-
8/19/2019 Review of Linear Algebra (1)
20/45
Matrices as sets of constraints
22
1
z y x
z y x
2
1
112
111
z
y
x
-
8/19/2019 Review of Linear Algebra (1)
21/45
Special matrices
f
ed
cba
00
0
c
ba
00
0000
ji
h g f
ed c
ba
00
0
0
00
f ed
cb
a
0
00
diagonal upper-triangular
tri-diagonal lower-triangular
100
010
001
I (identity matrix)
-
8/19/2019 Review of Linear Algebra (1)
22/45
Vector spaces
• Formally, a vector space is a set of vectors which isclosed under addition and multiplication by real numbers.
• A subspace is a subset of a vector space which is a
vector space itself, e.g. the plane z=0 is a subspace of
R3 (It is essentially R2.).
• We’ll be looking at Rn and subspaces of Rn
Our notion of planes in R3
may be extended to
hyperplanes in Rn
(ofdimension n-1)
Note: subspaces must
include the origin (zero
vector).
-
8/19/2019 Review of Linear Algebra (1)
23/45
Linear system & subspaces
• Linear systems define certainsubspaces
• Ax = b is solvable iff b may be
written as a linear combination
of the columns of A• The set of possible vectors b
forms a subspace called the
column space of A
3
2
1
31
32
01
b
b
b
v
u
3
2
1
3
3
0
1
2
1
b
b
b
vu(1,2,1)
(0,3,3)
-
8/19/2019 Review of Linear Algebra (1)
24/45
Linear system & subspaces
Null space: {(c,c,-c)}
0
0
0
31
32
01
vu
0
00
431
532101
z
y x
The set of solutions to Ax = 0 forms a subspace
called the null space of A.
Null space: {(0,0)}
-
8/19/2019 Review of Linear Algebra (1)
25/45
Linear independence and basis
0
0
0
|||
|||
3
2
1
321
c
c
c
vvv
Recall nullspace contained
only (u,v)=(0,0).
i.e. the columns are linearly
independent.
• Vectors v1,…,vk are linearly independent ifc1v1+…+ckvk = 0 implies c1=…=ck=0
(0,1)
(1,0)
(1,1)
(2,2)
0
0
0
31
32
01
v
u
i.e. the nullspace is the origin
-
8/19/2019 Review of Linear Algebra (1)
26/45
Linear independence and basis
1
0
0
2
0
1
0
2
0
0
1
2
2
2
2
• If all vectors in a vector space may be
expressed as linear combinations of v1,…,vk,
then v1,…,vk span the space.
(0,0,1)
(0,1,0)
(1,0,0)
(.1,.2,1)
(.3,1,0)
(.9,.2,0)
1
2.
1.
2
0
1
3.
29.1
0
2.
9.
57.1
2
2
2
-
8/19/2019 Review of Linear Algebra (1)
27/45
Linear independence and basis
• A basis is a set of linearly independent vectors which span thespace.
• The dimension of a space is the # of “degrees of freedom” of the
space; it is the number of vectors in any basis for the space.
• A basis is a maximal set of linearly independent vectors and a
minimal set of spanning vectors.
1
0
0
2
0
1
0
2
0
0
1
2
2
2
2
(0,0,1)
(0,1,0)
(1,0,0)
(.1,.2,1)
(.3,1,0)
(.9,.2,0)
1
2.
1.
2
0
1
3.
29.1
0
2.
9.
57.1
2
2
2
-
8/19/2019 Review of Linear Algebra (1)
28/45
Linear independence and basis
• Two vectors are orthogonal if their dot product is 0.• An orthogonal basis consists of orthogonal vectors.
• An orthonormal basis consists of orthogonal vectors
of unit length.
1
0
0
2
0
1
0
2
0
0
1
2
2
2
2
(0,0,1)
(0,1,0)
(1,0,0)
(.1,.2,1)
(.3,1,0)
(.9,.2,0)
1
2.
1.
2
0
1
3.
29.1
0
2.
9.
57.1
2
2
2
-
8/19/2019 Review of Linear Algebra (1)
29/45
About subspaces
• The rank of A is the dimension of the column space of A.
• It also equals the dimension of the row space of A (the
subspace of vectors which may be written as linear
combinations of the rows of A).
3132
01 (1,3) = (2,3) – (1,0)
Only 2 linearly independentrows, so rank = 2.
-
8/19/2019 Review of Linear Algebra (1)
30/45
About subspacesFundamental Theorem of Linear Algebra:
If A is m x n with rank r,
Column space(A) has dimension r
Nullspace(A) has dimension n-r (= nullity of A)
Row space(A) = Column space(AT) has dimension r Left nullspace(A) = Nullspace(AT) has dimension m - r
Rank-Nullity Theorem: rank + nullity = n => prove it!
(0,0,1)
(0,1,0)
(1,0,0)
00
10
01m = 3
n = 2
r = 2
-
8/19/2019 Review of Linear Algebra (1)
31/45
Non-square matrices
1
1
1
32
10
01
2
1
x
x
32
10
01m = 3
n = 2
r = 2
System Ax=b may
not have a solution
(x has 2 variablesbut 3 constraints).
310
201
m = 2
n = 3
r = 2System Ax=b is
underdetermined
(x has 3 variables
and 2 constraints).
1
1
310
201
3
2
1
x
x
x
-
8/19/2019 Review of Linear Algebra (1)
32/45
Basis transformations
• Before talking about basis transformations,
we need to recall matrix inversion and
projections.
-
8/19/2019 Review of Linear Algebra (1)
33/45
Matrix inversion
• To solve Ax=b, we can write a closed-form solution if
we can find a matrix A-1
s.t. AA-1 =A-1 A=I (identity matrix)
• Then Ax=b iff x=A-1
b:x = Ix = A-1 Ax = A-1b
• A is non-singular iff A-1 exists iff Ax=b has a unique
solution.
• Note: If A-1,B-1 exist, then (AB)-1 = B-1 A-1,
and (AT)-1 = (A-1)T
-
8/19/2019 Review of Linear Algebra (1)
34/45
Projections
2
2
2
000
010
001
0
2
2
(0,0,1)
(0,1,0)
(1,0,0)
(2,2,2)
a = (1,0)
b = (2,2)
0
2a
aa
bac
T
T
-
8/19/2019 Review of Linear Algebra (1)
35/45
Basis transformations
2
29.1
57.1
100
2.12.
1.3.9.
2
2
2
2
2
2
100
010
001
2
2
2
(0,0,1)
(0,1,0)
(1,0,0)
(.1,.2,1)
(.3,1,0)
(.9,.2,0)
We may write v=(2,2,2) in terms of an alternate basis:
Components of (1.57,1.29,2) are projections of v onto new basis vectors,
normalized so new v still has same length.
-
8/19/2019 Review of Linear Algebra (1)
36/45
Basis transformations
Given vector v written in standard basis, rewrite as vQin terms of basis Q.
If columns of Q are orthonormal, vQ = QTv
Otherwise, vQ = (QTQ)QTv
-
8/19/2019 Review of Linear Algebra (1)
37/45
Special matrices
• Matrix A is symmetric if A = AT
• A is positive definite if xT Ax>0 for all non-zero x ( positive semi-
definite if inequality is not strict)
222
100
010
001
cba
c
b
a
cba
222
100
010
001
cba
c
b
a
cba
-
8/19/2019 Review of Linear Algebra (1)
38/45
Special matrices
• Matrix A is symmetric if A = AT
• A is positive definite if xT Ax>0 for all non-zero x ( positive semi-
definite if inequality is not strict)
• Useful fact: Any matrix of form AT A is positive semi-definite.
To see this, xT(AT A)x = (xT AT)(Ax) = (Ax)T(Ax) ≥ 0 => prove
it!
-
8/19/2019 Review of Linear Algebra (1)
39/45
Determinants
• If det(A) = 0, then A is
singular.
• If det(A) ≠ 0, then A is
invertible.• To compute:
– Simple example:
– Matlab: det(A)
bcad d c
ba
det
-
8/19/2019 Review of Linear Algebra (1)
40/45
Determinants
• m-by-n matrix A is rank-deficient if it has
rank r < m (≤ n)
• Thm: rank(A) < r iff
det(A) = 0 for all t-by-t submatrices,
r ≤ t ≤ m
-
8/19/2019 Review of Linear Algebra (1)
41/45
Eigenvalues & eigenvectors
• How can we characterize matrices?
• The solutions to Ax = λx in the form of eigenpairs
(λ,x) = (eigenvalue,eigenvector) where x is non-zero
• To solve this, (A – λI)x = 0• λ is an eigenvalue iff det(A – λI) = 0
-
8/19/2019 Review of Linear Algebra (1)
42/45
Eigenvalues & eigenvectors
(A – λI)x = 0
λ is an eigenvalue iff det(A – λI) = 0
Example:
2/100
64/30541
A
)2/1)(4/3)(1(
2/100
64/30541
)det(
I A
2/1,4/3,1
-
8/19/2019 Review of Linear Algebra (1)
43/45
Eigenvalues & eigenvectors
10
02 A Eigenvalues λ = 2, 1 with
eigenvectors (1,0), (0,1)
(0,1)
(2,0)
Eigenvectors of a linear transformation A are not rotated (but will be
scaled by the corresponding eigenvalue) when A is applied.
v Av
(1,0)
-
8/19/2019 Review of Linear Algebra (1)
44/45
Solving Ax=b
x + 2y + z = 0y - z = 2
x +2z= 1
-------------
x + 2y + z = 0y - z = 2
-2y + z= 1
-------------
x + 2y + z = 0
y - z = 2
- z = 5
5100
2110
0121
1120
2110
0121
1201
2110
0121 Write system ofequations in matrixform.
Subtract first row fromlast row.
Add 2 copies of secondrow to last row.
Now solve by back-substitution: z = -5, y = 2-z = 7, x = -2y-z = -9
-
8/19/2019 Review of Linear Algebra (1)
45/45
Solving Ax=b & condition numbers
• Matlab: linsolve(A,b)
• How stable is the solution?
• If A or b are changed slightly, how much does it
effect x?• The condition number c of A measures this:
c = λmax/ λmin
• Values of c near 1 are good.