Linalg Tutorial - Stanford University
Transcript of Linalg Tutorial - Stanford University
![Page 1: Linalg Tutorial - Stanford University](https://reader030.fdocuments.in/reader030/viewer/2022012500/61796e04cdbcfc719e70aa47/html5/thumbnails/1.jpg)
Quick Tour of Linear Algebra and Graph Theory
Quick Tour of Linear Algebra and GraphTheory
CS224w: Social and Information Network AnalysisFall 2012 Yu ”Wayne” Wu
Based on Borja Pelato’s version in Fall 2011
![Page 2: Linalg Tutorial - Stanford University](https://reader030.fdocuments.in/reader030/viewer/2022012500/61796e04cdbcfc719e70aa47/html5/thumbnails/2.jpg)
Quick Tour of Linear Algebra and Graph Theory
Basic Linear Algebra
Matrices and Vectors
Matrix: A rectangular array of numbers, e.g., A ∈ Rm×n:
A =
a11 a12 . . . a1na21 a22 . . . a2n
......
...am1 am2 . . . amn
Vector: A matrix consisting of only one column (default) orone row, e.g., x ∈ Rn
x =
x1x2...
xn
![Page 3: Linalg Tutorial - Stanford University](https://reader030.fdocuments.in/reader030/viewer/2022012500/61796e04cdbcfc719e70aa47/html5/thumbnails/3.jpg)
Quick Tour of Linear Algebra and Graph Theory
Basic Linear Algebra
Matrix Multiplication
If A ∈ Rm×n, B ∈ Rn×p, C = AB, then C ∈ Rm×p:
Cij =n∑
k=1
AikBkj
Special cases: Matrix-vector product, inner product of twovectors. e.g., with x , y ∈ Rn:
xT y =n∑
i=1
xiyi ∈ R
![Page 4: Linalg Tutorial - Stanford University](https://reader030.fdocuments.in/reader030/viewer/2022012500/61796e04cdbcfc719e70aa47/html5/thumbnails/4.jpg)
Quick Tour of Linear Algebra and Graph Theory
Basic Linear Algebra
Matrix Multiplication
![Page 5: Linalg Tutorial - Stanford University](https://reader030.fdocuments.in/reader030/viewer/2022012500/61796e04cdbcfc719e70aa47/html5/thumbnails/5.jpg)
Quick Tour of Linear Algebra and Graph Theory
Basic Linear Algebra
Properties of Matrix Multiplication
Associative: (AB)C = A(BC)
Distributive: A(B + C) = AB + ACNon-commutative: AB 6= BA
Proof of associativity:Let L = (AB)C and R = A(BC), then we can showLij =
∑pk = 1
∑nl = 1 (ail ◦ blk ) ◦ ckj =∑n
l = 1∑p
k = 1 ail ◦(blk ◦ ckj
)= Rij .
![Page 6: Linalg Tutorial - Stanford University](https://reader030.fdocuments.in/reader030/viewer/2022012500/61796e04cdbcfc719e70aa47/html5/thumbnails/6.jpg)
Quick Tour of Linear Algebra and Graph Theory
Basic Linear Algebra
Operators and properties
Transpose: A ∈ Rm×n, then AT ∈ Rn×m: (AT )ij = Aji
Properties:(AT )T = A(AB)T = BT AT
(A + B)T = AT + BT
![Page 7: Linalg Tutorial - Stanford University](https://reader030.fdocuments.in/reader030/viewer/2022012500/61796e04cdbcfc719e70aa47/html5/thumbnails/7.jpg)
Quick Tour of Linear Algebra and Graph Theory
Basic Linear Algebra
Identity Matrix
Identity matrix: I = In ∈ Rn×n:
Iij =
{1 i=j,0 otherwise.
∀A ∈ Rm×n: AIn = ImA = A
![Page 8: Linalg Tutorial - Stanford University](https://reader030.fdocuments.in/reader030/viewer/2022012500/61796e04cdbcfc719e70aa47/html5/thumbnails/8.jpg)
Quick Tour of Linear Algebra and Graph Theory
Basic Linear Algebra
Diagonal Matrix
Diagonal matrix: D = diag(d1,d2, . . . ,dn):
Dij =
{di j=i,0 otherwise.
![Page 9: Linalg Tutorial - Stanford University](https://reader030.fdocuments.in/reader030/viewer/2022012500/61796e04cdbcfc719e70aa47/html5/thumbnails/9.jpg)
Quick Tour of Linear Algebra and Graph Theory
Basic Linear Algebra
Other Special Mtrices
Symmetric matrices: A ∈ Rn×n is symmetric if A = AT .Orthogonal matrices: U ∈ Rn×n is orthogonal ifUUT = I = UT U
![Page 10: Linalg Tutorial - Stanford University](https://reader030.fdocuments.in/reader030/viewer/2022012500/61796e04cdbcfc719e70aa47/html5/thumbnails/10.jpg)
Quick Tour of Linear Algebra and Graph Theory
Basic Linear Algebra
Linear Independence and Rank
A set of vectors {x1, . . . , xn} is linearly independent if@{α1, . . . , αn}:
∑ni=1 αixi = 0
Rank: A ∈ Rm×n, then rank(A) is the maximum number oflinearly independent columns (or equivalently, rows)Properties:
rank(A) ≤ min{m,n}rank(A) = rank(AT )rank(AB) ≤ min{rank(A), rank(B)}rank(A + B) ≤ rank(A) + rank(B)
![Page 11: Linalg Tutorial - Stanford University](https://reader030.fdocuments.in/reader030/viewer/2022012500/61796e04cdbcfc719e70aa47/html5/thumbnails/11.jpg)
Quick Tour of Linear Algebra and Graph Theory
Basic Linear Algebra
Rank from row-echelon forms
![Page 12: Linalg Tutorial - Stanford University](https://reader030.fdocuments.in/reader030/viewer/2022012500/61796e04cdbcfc719e70aa47/html5/thumbnails/12.jpg)
Quick Tour of Linear Algebra and Graph Theory
Basic Linear Algebra
Matrix Inversion
If A ∈ Rn×n, rank(A) = n, then the inverse of A, denotedA−1 is the matrix that: AA−1 = A−1A = IProperties:
(A−1)−1 = A(AB)−1 = B−1A−1
(A−1)T = (AT )−1
The inverse of an orthogonal matrix is its transpose
![Page 13: Linalg Tutorial - Stanford University](https://reader030.fdocuments.in/reader030/viewer/2022012500/61796e04cdbcfc719e70aa47/html5/thumbnails/13.jpg)
Quick Tour of Linear Algebra and Graph Theory
Basic Linear Algebra
Eigenvalues and Eigenvectors
A ∈ Rn×n, λ ∈ C is an eigenvalue of A with thecorresponding eigenvector x ∈ Cn (x 6= 0) if:
Ax = λx
eigenvalues: the n possibly complex roots of thepolynomial equation det(A− λI) = 0, and denoted asλ1, . . . , λn
![Page 14: Linalg Tutorial - Stanford University](https://reader030.fdocuments.in/reader030/viewer/2022012500/61796e04cdbcfc719e70aa47/html5/thumbnails/14.jpg)
Quick Tour of Linear Algebra and Graph Theory
Basic Linear Algebra
Eigenvalues and Eigenvectors Properties
Usually eigenvectors are normalized to unit length.If A is symmetric, then all the eigenvalues are real and theeigenvectors are orthogonal to each other.tr(A) =
∑ni=1 λi
det(A) =∏n
i=1 λi
rank(A) = |{1 ≤ i ≤ n|λi 6= 0}|
![Page 15: Linalg Tutorial - Stanford University](https://reader030.fdocuments.in/reader030/viewer/2022012500/61796e04cdbcfc719e70aa47/html5/thumbnails/15.jpg)
Quick Tour of Linear Algebra and Graph Theory
Basic Linear Algebra
Matrix Eigendecomposition
A ∈ Rn×n, λ1, . . . , λn the eigenvalues, and x1, . . . , xn theeigenvectors. P = [x1|x2| . . . |xn], D = diag(λ1, . . . , λn), then:
![Page 16: Linalg Tutorial - Stanford University](https://reader030.fdocuments.in/reader030/viewer/2022012500/61796e04cdbcfc719e70aa47/html5/thumbnails/16.jpg)
Quick Tour of Linear Algebra and Graph Theory
Basic Linear Algebra
Matrix Eigendecomposition
Therefore, A = PDP−1.In addition:
A2 = (PDP−1)(PDP−1) = PD(P−1P)DP−1 = PD2P−1
By induction, An = PDnP−1.A special case of Singular Value Decomposition
![Page 17: Linalg Tutorial - Stanford University](https://reader030.fdocuments.in/reader030/viewer/2022012500/61796e04cdbcfc719e70aa47/html5/thumbnails/17.jpg)
Quick Tour of Linear Algebra and Graph Theory
Basic Linear Algebra
Convex Optimization
A set of points S is convex if, for any x , y ∈ S and for any0 ≤ θ ≤ 1,
θx + (1− θ)y ∈ S
A function f : S → R is convex if its domain S is a convexset and
f (θx + (1− θ)y) ≤ θf (x) + (1− θ)f (y)
for all x , y ∈ S, 0 ≤ θ ≤ 1.
![Page 18: Linalg Tutorial - Stanford University](https://reader030.fdocuments.in/reader030/viewer/2022012500/61796e04cdbcfc719e70aa47/html5/thumbnails/18.jpg)
Quick Tour of Linear Algebra and Graph Theory
Basic Linear Algebra
Logistic Regression
![Page 19: Linalg Tutorial - Stanford University](https://reader030.fdocuments.in/reader030/viewer/2022012500/61796e04cdbcfc719e70aa47/html5/thumbnails/19.jpg)
Quick Tour of Linear Algebra and Graph Theory
Basic Linear Algebra
Gradient Descent
![Page 20: Linalg Tutorial - Stanford University](https://reader030.fdocuments.in/reader030/viewer/2022012500/61796e04cdbcfc719e70aa47/html5/thumbnails/20.jpg)
Quick Tour of Linear Algebra and Graph Theory
Basic Linear Algebra
Submodularity
A function f : S → R is submodular if for any subset A ⊆ B,
f (A ∪ {x})− f (A) ≥ f (B ∪ {x})− f (B)
Submodular functions allow approximate discreteoptimization.
![Page 21: Linalg Tutorial - Stanford University](https://reader030.fdocuments.in/reader030/viewer/2022012500/61796e04cdbcfc719e70aa47/html5/thumbnails/21.jpg)
Quick Tour of Linear Algebra and Graph Theory
Basic Linear Algebra
Greedy Set Cover
If the optimal solution contains m sets, greedy algorithm finds aset cover with at most m loge n sets.
![Page 22: Linalg Tutorial - Stanford University](https://reader030.fdocuments.in/reader030/viewer/2022012500/61796e04cdbcfc719e70aa47/html5/thumbnails/22.jpg)
Quick Tour of Linear Algebra and Graph Theory
Basic Linear Algebra
Proofs
Induction:1 Show result on base case, associated with n = k0
2 Assume result true for n ≤ i . Prove result for n = i + 13 Conclude result true for all n ≥ k0
Example:For all natural number n, 1 + 2 + 3 + ...+ n = n∗(n+1)
2Base case: when n = 1, 1 = 1.Assume statement holds for n = k , then1 + 2 + 3 + ...+ k = k∗(k+1)
2 .We see 1+2+3+ ...+(k +1) = k∗(k+1)
2 +(k +1) = (k+1)(k+2)2 .
![Page 23: Linalg Tutorial - Stanford University](https://reader030.fdocuments.in/reader030/viewer/2022012500/61796e04cdbcfc719e70aa47/html5/thumbnails/23.jpg)
Quick Tour of Linear Algebra and Graph Theory
Basic Linear Algebra
Proofs
Contradiction (reductio ad absurdum):1 Assume result is false2 Follow implications in a deductive manner, until a
contradiction is reached3 Conclude initial assumption was wrong, hence result true
Example:Let’s try to prove there is no greatest even integer.First suppose there is one, name N, and for any even integer n,we have N ≥ n.Now let define M as N + 2. Then we see M is even, but alsogreater than N. Thus by contradiction we prove the statement.
![Page 24: Linalg Tutorial - Stanford University](https://reader030.fdocuments.in/reader030/viewer/2022012500/61796e04cdbcfc719e70aa47/html5/thumbnails/24.jpg)
Quick Tour of Linear Algebra and Graph Theory
Basic Linear Algebra
Graph theory
Definitions: vertex/node, edge/link, loop/cycle, degree,path, neighbor, tree, clique,. . .Random graph (Erdos-Renyi): Each possible edge ispresent with some probability p(Strongly) connected component: subset of nodes that canall reach each otherDiameter: longest minimum distance between two nodesBridge: edge connecting two otherwise disjoint connectedcomponents
![Page 25: Linalg Tutorial - Stanford University](https://reader030.fdocuments.in/reader030/viewer/2022012500/61796e04cdbcfc719e70aa47/html5/thumbnails/25.jpg)
Quick Tour of Linear Algebra and Graph Theory
Basic Linear Algebra
Basic algorithms
BFS: explore by “layers”DFS: go as far as possible, then backtrackGreedy: maximize goal at each stepBinary search: on ordered set, discard half of the elementsat each step
![Page 26: Linalg Tutorial - Stanford University](https://reader030.fdocuments.in/reader030/viewer/2022012500/61796e04cdbcfc719e70aa47/html5/thumbnails/26.jpg)
Quick Tour of Linear Algebra and Graph Theory
Basic Linear Algebra
BFS
![Page 27: Linalg Tutorial - Stanford University](https://reader030.fdocuments.in/reader030/viewer/2022012500/61796e04cdbcfc719e70aa47/html5/thumbnails/27.jpg)
Quick Tour of Linear Algebra and Graph Theory
Basic Linear Algebra
DFS
![Page 28: Linalg Tutorial - Stanford University](https://reader030.fdocuments.in/reader030/viewer/2022012500/61796e04cdbcfc719e70aa47/html5/thumbnails/28.jpg)
Quick Tour of Linear Algebra and Graph Theory
Basic Linear Algebra
Complexity
Number of operations as a function of the problemparameters.Examples
1 Find shortest path between two nodes:DFS: very bad idea, could end up with the whole graph as asingle pathBFS from origin: good ideaBFS from origin and destination: even better!
2 Given a node, find its connected componentLoop over nodes: bad idea, needs N path searchesBFS or DFS: good idea