Spectral-Lagrangian solvers for non-linear non-conservative Boltzmann Transport Equations
Spectral Graph Theory, Linear Solvers, and Applications
Transcript of Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Spectral Graph Theory,Linear Solvers, and
Applications
Gary Miller
Carnegie Mellon Universityjoiny work with Yiannis Koutis and David Tolliver
Theory and Practice of Computational LearningJune 9, 2009
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Linear Systems
3x 2y −z = 32x −5y 4z = 7−x 1/2y 2z = 2
Fundamental ConstraintSystem
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Matrix Form
3 2 −12 −5 4
−1 1/2 2
xyz
=
372
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Solving the General Case
• Dense Case: O(n3) or O(n2.81) Strassen.
• Sparse Case: Still O(n2.81).
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Solving the General Case
• Dense Case: O(n3) or O(n2.81) Strassen.• Sparse Case: Still O(n2.81).
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
An Easy Case
• Upper and Lower Triangular Systems
• O(m) time where m = number of nonzerosentries.
• Goal: Find more easy cases.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
An Easy Case
• Upper and Lower Triangular Systems• O(m) time where m = number of nonzeros
entries.
• Goal: Find more easy cases.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
An Easy Case
• Upper and Lower Triangular Systems• O(m) time where m = number of nonzeros
entries.• Goal: Find more easy cases.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Symmetric Matrices
• Assume A is symmetric, A = AT .
• Assume A is positive definite, xTAx > 0 forx 6= 0
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Symmetric Matrices
• Assume A is symmetric, A = AT .• Assume A is positive definite, xTAx > 0 forx 6= 0
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Direct MethodsGaussian Elimination Matrices
• Goal: algorithms that minimize work andspace.
• Trick: View nonzero entries as an undirectedgraph and view pivoting as a graphoperation.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Direct MethodsGaussian Elimination Matrices
• Goal: algorithms that minimize work andspace.
• Trick: View nonzero entries as an undirectedgraph and view pivoting as a graphoperation.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Pivoting
Viewing pivoting as a graph operation.
• Let G = (V, E) be a graph and v a vertex.
• PIV OT (v) :• Make a clique out of neighbors of v.• Remove v.
• Fill: New edges formed.• Work: All edges “touched”.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Pivoting
Viewing pivoting as a graph operation.
• Let G = (V, E) be a graph and v a vertex.• PIV OT (v) :
• Make a clique out of neighbors of v.• Remove v.
• Fill: New edges formed.• Work: All edges “touched”.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Pivoting
Viewing pivoting as a graph operation.
• Let G = (V, E) be a graph and v a vertex.• PIV OT (v) :
• Make a clique out of neighbors of v.• Remove v.
• Fill: New edges formed.
• Work: All edges “touched”.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Pivoting
Viewing pivoting as a graph operation.
• Let G = (V, E) be a graph and v a vertex.• PIV OT (v) :
• Make a clique out of neighbors of v.• Remove v.
• Fill: New edges formed.• Work: All edges “touched”.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Good Pivot Strategies
1970s and 1980s• Planar systems: O(n3/2) work and O(n log n)
fill/space.
• 3D Systems: O(n2) work and O(n3/2)fill/space.
• O(n3/2) space is a problem for ML sizeproblems.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Good Pivot Strategies
1970s and 1980s• Planar systems: O(n3/2) work and O(n log n)
fill/space.• 3D Systems: O(n2) work and O(n3/2)
fill/space.
• O(n3/2) space is a problem for ML sizeproblems.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Good Pivot Strategies
1970s and 1980s• Planar systems: O(n3/2) work and O(n log n)
fill/space.• 3D Systems: O(n2) work and O(n3/2)
fill/space.• O(n3/2) space is a problem for ML size
problems.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Pure Iterative Methods
Solving Ax = b.• Basic method: x(i+1) = (I − A)x(i) + b
• Convergence/Rate is determined by ||I −A||.• Accelerated Methods: Chebyshev Iteration,
Conjugate Gradient.• CG is still too slow.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Pure Iterative Methods
Solving Ax = b.• Basic method: x(i+1) = (I − A)x(i) + b
• Convergence/Rate is determined by ||I −A||.
• Accelerated Methods: Chebyshev Iteration,Conjugate Gradient.
• CG is still too slow.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Pure Iterative Methods
Solving Ax = b.• Basic method: x(i+1) = (I − A)x(i) + b
• Convergence/Rate is determined by ||I −A||.• Accelerated Methods: Chebyshev Iteration,
Conjugate Gradient.
• CG is still too slow.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Pure Iterative Methods
Solving Ax = b.• Basic method: x(i+1) = (I − A)x(i) + b
• Convergence/Rate is determined by ||I −A||.• Accelerated Methods: Chebyshev Iteration,
Conjugate Gradient.• CG is still too slow.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Preconditioned Iterative Methods
Solving B−1Ax = B−1b = b′.• Basic method: x(i+1) = (I −B−1A)x(i) + b′
• Computing the term z = B−1Ax(i)
• y = Ax(i) Forward Multiply• Bz = y Solve the preconditioner system
• Goal: Minimize the number of iteration whileminimizing the cost of the solve.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Preconditioned Iterative Methods
Solving B−1Ax = B−1b = b′.• Basic method: x(i+1) = (I −B−1A)x(i) + b′
• Computing the term z = B−1Ax(i)
• y = Ax(i) Forward Multiply• Bz = y Solve the preconditioner system
• Goal: Minimize the number of iteration whileminimizing the cost of the solve.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Preconditioned Iterative Methods
Solving B−1Ax = B−1b = b′.• Basic method: x(i+1) = (I −B−1A)x(i) + b′
• Computing the term z = B−1Ax(i)
• y = Ax(i) Forward Multiply• Bz = y Solve the preconditioner system
• Goal: Minimize the number of iteration whileminimizing the cost of the solve.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Classic Preconditioners
• Jacobi: B = Diagonal(A).
• Gauss-Seidel: B = UpperTriangular(A).• SSOR: B = (L + 1
ωD) 1ωD(L + 1
ωD)
• Still too slow and unreliable.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Classic Preconditioners
• Jacobi: B = Diagonal(A).• Gauss-Seidel: B = UpperTriangular(A).
• SSOR: B = (L + 1ωD) 1
ωD(L + 1ωD)
• Still too slow and unreliable.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Classic Preconditioners
• Jacobi: B = Diagonal(A).• Gauss-Seidel: B = UpperTriangular(A).• SSOR: B = (L + 1
ωD) 1ωD(L + 1
ωD)
• Still too slow and unreliable.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Classic Preconditioners
• Jacobi: B = Diagonal(A).• Gauss-Seidel: B = UpperTriangular(A).• SSOR: B = (L + 1
ωD) 1ωD(L + 1
ωD)
• Still too slow and unreliable.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Graph LaplaciansApplications
Graph Laplacian• G = (V,E, w) weighted undirected graph, wij > 0.
• Weighted incidence matrix:
Aij ={
wij if eij ∈ E0 otherwise
• Degree of vi: di =∑
j wij
•
D =
d1 0. . .
0 dn
• Laplacian: L = D −A
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Graph LaplaciansApplications
Graph Laplacian• G = (V,E, w) weighted undirected graph, wij > 0.• Weighted incidence matrix:
Aij ={
wij if eij ∈ E0 otherwise
• Degree of vi: di =∑
j wij
•
D =
d1 0. . .
0 dn
• Laplacian: L = D −A
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Graph LaplaciansApplications
Graph Laplacian• G = (V,E, w) weighted undirected graph, wij > 0.• Weighted incidence matrix:
Aij ={
wij if eij ∈ E0 otherwise
• Degree of vi: di =∑
j wij
•
D =
d1 0. . .
0 dn
• Laplacian: L = D −A
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Graph LaplaciansApplications
Graph Laplacian• G = (V,E, w) weighted undirected graph, wij > 0.• Weighted incidence matrix:
Aij ={
wij if eij ∈ E0 otherwise
• Degree of vi: di =∑
j wij
•
D =
d1 0. . .
0 dn
• Laplacian: L = D −A
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Graph LaplaciansApplications
Graph Laplacian• G = (V,E, w) weighted undirected graph, wij > 0.• Weighted incidence matrix:
Aij ={
wij if eij ∈ E0 otherwise
• Degree of vi: di =∑
j wij
•
D =
d1 0. . .
0 dn
• Laplacian: L = D −A
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Graph LaplaciansApplications
Classic Applications of the Laplacian
• View each edge a conductor with conductance wij .
• Let V be a column vector of voltages• If LV = c the c residual current needed to maintain the
given voltages.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Graph LaplaciansApplications
Classic Applications of the Laplacian
• View each edge a conductor with conductance wij .• Let V be a column vector of voltages
• If LV = c the c residual current needed to maintain thegiven voltages.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Graph LaplaciansApplications
Classic Applications of the Laplacian
• View each edge a conductor with conductance wij .• Let V be a column vector of voltages• If LV = c the c residual current needed to maintain the
given voltages.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Graph LaplaciansApplications
Graph Laplacian’s and the HeatEquations
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Graph LaplaciansApplications
Graph Laplacian’s and RandomWalks
Transition Matrix: D−1LG
Fundamental Eigenvectors: O(n + m) (Spielman Teng)Trick: Inverse Powering only requires O(log n) iterations.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Graph LaplaciansApplications
Laplacian’s and Spring Mass Systems
• G = (V, E, w) weighted graph and wij isviewed a spring constant.
• M is a diagonal matrix of mass constants• Fact: Modes of vibration of Spring-Mass
system G, M are:Eigen-pairs of LGx = λMx.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Graph LaplaciansApplications
Laplacian’s and Spring Mass Systems
• G = (V, E, w) weighted graph and wij isviewed a spring constant.
• M is a diagonal matrix of mass constants
• Fact: Modes of vibration of Spring-Masssystem G, M are:Eigen-pairs of LGx = λMx.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Graph LaplaciansApplications
Laplacian’s and Spring Mass Systems
• G = (V, E, w) weighted graph and wij isviewed a spring constant.
• M is a diagonal matrix of mass constants• Fact: Modes of vibration of Spring-Mass
system G, M are:Eigen-pairs of LGx = λMx.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Graph LaplaciansApplications
Spring Mass System
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Graph LaplaciansApplications
Graph Laplacian’s and MaximumFlow
Graph Maximum Flow: O((m + n)3/2) (Daitch Spielman)
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Graph LaplaciansApplications
Graph Laplacian’s and ConvexProgramming
Nonuniform TV Denoising: O((m + n)3/2) (Koutis M SinopTolliver)
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
Spanning Tree Preconditioners• Vaidya ’93: Use Maximum Weight Spanning
Tree (MST) plus a few edges.
b
c
a
d
e f g
h
j
i
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
Spanning Tree Preconditioners• Advantages: Easy to find and Easy to solve
their systems.
b
c
a
d
e f g
h
j
i
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
Spanning Tree Preconditioners• Problem: Small edge weights differences
can make MST bad.
b
c
a
d
e f g
h
j
i
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
Low Stretch Spanning Trees• EEST ’05: Use low stretch spanning trees
plus a few edges.
b
c
a
d
e f g
h
j
i
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
Low Stretch Spanning Trees• Advantages: Better condition number.
b
c
a
d
e f g
h
j
i
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
Low Stretch Spanning Trees• Problem: Super linear time to find.
b
c
a
d
e f g
h
j
i
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
Low Stretch Spanning Trees• Richter-M ’04: There are no good spanning
trees even for square mesh.
b
c
a
d
e f g
h
j
i
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
Steiner Tree Preconditioners
• Gremban-M ’94: Use Steiner trees.
b
c
a
d
e f g
h
j
i
g
a b
c d
h i je
f
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
Steiner Tree Preconditioners
• Advantages: Better condition number forgraphs like square mesh.
b
c
a
d
e f g
h
j
i
g
a b
c d
h i je
f
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
Steiner Tree Preconditioners
• Advantages: Good experimental results.
b
c
a
d
e f g
h
j
i
g
a b
c d
h i je
f
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
Steiner Tree Preconditioners
• Problem: Hard to construct in general andanalyze.
b
c
a
d
e f g
h
j
i
g
a b
c d
h i je
f
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
Steiner Forest Preconditioners
• Koutis-M ’07: Use Steiner Forest.
b
c
a
d
e f g
h
j
i
g
a b
c d
h i je
f
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
Steiner Forest Preconditioners
• Advantages: Easy to find and works wellwith recursive solvers.
b
c
a
d
e f g
h
j
i
g
a b
c d
h i je
f
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
Steiner Forest Preconditioners
• Advantages: Good experimental results.
b
c
a
d
e f g
h
j
i
g
a b
c d
h i je
f
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
Steiner Forest Preconditioners
• Problem: Analysis only for planar systems.
b
c
a
d
e f g
h
j
i
g
a b
c d
h i je
f
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
Recursive Methods
• Vaidya ’93: Idea: Tree have a lot of degree1-2 degree nodes. Pivot on these nodes andthen find a preconditioner for this graph.
d h i jf ga b ec
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
Recursive Methods
• Vaidya ’93: Idea: Tree have a lot of degree1-2 degree nodes. Pivot on these nodes andthen find a preconditioner for this graph.
The reduced graph.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
The Error
• Recurrence: u(i+1) = Gu(i) + b, for usG = I − A.
• Error: e(i) = u(i) − u, where u = Gu + b
• Fact: e(i) = Gie(0)
• We need: limi→∞ Gi = 0
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
The Error
• Recurrence: u(i+1) = Gu(i) + b, for usG = I − A.
• Error: e(i) = u(i) − u, where u = Gu + b
• Fact: e(i) = Gie(0)
• We need: limi→∞ Gi = 0
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
The Error
• Recurrence: u(i+1) = Gu(i) + b, for usG = I − A.
• Error: e(i) = u(i) − u, where u = Gu + b
• Fact: e(i) = Gie(0)
• We need: limi→∞ Gi = 0
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
The Error
• Recurrence: u(i+1) = Gu(i) + b, for usG = I − A.
• Error: e(i) = u(i) − u, where u = Gu + b
• Fact: e(i) = Gie(0)
• We need: limi→∞ Gi = 0
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
Condition Number
• Ax = λx
• DEF: λ Eigenvalue and x Eigenvector.• Λ(A) = {0 ≤ λ1 ≤ · · ·λn}.• Condition Number: κ(A) = λn/λ1
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
Condition Number
• Ax = λx
• DEF: λ Eigenvalue and x Eigenvector.
• Λ(A) = {0 ≤ λ1 ≤ · · ·λn}.• Condition Number: κ(A) = λn/λ1
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
Condition Number
• Ax = λx
• DEF: λ Eigenvalue and x Eigenvector.• Λ(A) = {0 ≤ λ1 ≤ · · ·λn}.
• Condition Number: κ(A) = λn/λ1
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
Condition Number
• Ax = λx
• DEF: λ Eigenvalue and x Eigenvector.• Λ(A) = {0 ≤ λ1 ≤ · · ·λn}.• Condition Number: κ(A) = λn/λ1
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
Convergence Rates
• Basic Method: Convergence Rate= O(1/κ(A))
• Conjugate Gradient: O(1/√
κ(A))
• Conjugate Gradient: ≈ 1/diameter(A).
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
Convergence Rates
• Basic Method: Convergence Rate= O(1/κ(A))
• Conjugate Gradient: O(1/√
κ(A))
• Conjugate Gradient: ≈ 1/diameter(A).
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
Convergence Rates
• Basic Method: Convergence Rate= O(1/κ(A))
• Conjugate Gradient: O(1/√
κ(A))
• Conjugate Gradient: ≈ 1/diameter(A).
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
Generalized Condition Number
• Goal: Bound condition number of B−1A
• Note: B−1Ax = λx iff Ax = λBx
• DEF: λ is a generalized eigenvalue.• Condition Number: κ(B−1A) = λn/λ1
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
Generalized Condition Number
• Goal: Bound condition number of B−1A
• Note: B−1Ax = λx iff Ax = λBx
• DEF: λ is a generalized eigenvalue.• Condition Number: κ(B−1A) = λn/λ1
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
Generalized Condition Number
• Goal: Bound condition number of B−1A
• Note: B−1Ax = λx iff Ax = λBx
• DEF: λ is a generalized eigenvalue.
• Condition Number: κ(B−1A) = λn/λ1
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
Generalized Condition Number
• Goal: Bound condition number of B−1A
• Note: B−1Ax = λx iff Ax = λBx
• DEF: λ is a generalized eigenvalue.• Condition Number: κ(B−1A) = λn/λ1
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
The Support
• Positive Semi-Definite: ∀x xTAx ≥ 0
• Support of A by B:σ(A/B) = min{τ : τB − A is PSD}
• Fact: κ(A, B) = σ(A, B) · σ(B, A).
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
The Support
• Positive Semi-Definite: ∀x xTAx ≥ 0
• Support of A by B:σ(A/B) = min{τ : τB − A is PSD}
• Fact: κ(A, B) = σ(A, B) · σ(B, A).
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
The Support
• Positive Semi-Definite: ∀x xTAx ≥ 0
• Support of A by B:σ(A/B) = min{τ : τB − A is PSD}
• Fact: κ(A, B) = σ(A, B) · σ(B, A).
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
Estimating Support
• G and H are graphs and V (G) = V (H)
• Path Embedding: φ : E(G) then paths(H)s.t. φ(eij) = Vi · · ·Vj.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
Estimating Support ExampleEG: G ≡ K4 and H ≡ 4-cycle
Congestion ≡ 3 and Dilation ≡ 2Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
History of Planar Solvers
• 1950’s O(n2) (Conjugate Gradient)
• 1970’s O(n1.5) (Nested Dissection) (LRT)• 1990’s O(n1.2) (Combinatorial
Preconditioners) (Vaidya)• 2000’s O(n log2 n) (Low stretch spanning
trees) (ST)• 2006’s O(n) (separator based
preconditioners) (KM)
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
History of Planar Solvers
• 1950’s O(n2) (Conjugate Gradient)• 1970’s O(n1.5) (Nested Dissection) (LRT)
• 1990’s O(n1.2) (CombinatorialPreconditioners) (Vaidya)
• 2000’s O(n log2 n) (Low stretch spanningtrees) (ST)
• 2006’s O(n) (separator basedpreconditioners) (KM)
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
History of Planar Solvers
• 1950’s O(n2) (Conjugate Gradient)• 1970’s O(n1.5) (Nested Dissection) (LRT)• 1990’s O(n1.2) (Combinatorial
Preconditioners) (Vaidya)
• 2000’s O(n log2 n) (Low stretch spanningtrees) (ST)
• 2006’s O(n) (separator basedpreconditioners) (KM)
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
History of Planar Solvers
• 1950’s O(n2) (Conjugate Gradient)• 1970’s O(n1.5) (Nested Dissection) (LRT)• 1990’s O(n1.2) (Combinatorial
Preconditioners) (Vaidya)• 2000’s O(n log2 n) (Low stretch spanning
trees) (ST)
• 2006’s O(n) (separator basedpreconditioners) (KM)
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
History of Planar Solvers
• 1950’s O(n2) (Conjugate Gradient)• 1970’s O(n1.5) (Nested Dissection) (LRT)• 1990’s O(n1.2) (Combinatorial
Preconditioners) (Vaidya)• 2000’s O(n log2 n) (Low stretch spanning
trees) (ST)• 2006’s O(n) (separator based
preconditioners) (KM)
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
General Laplacian Solver
O(n + m) (Spielman Teng)
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.80
2
4
6
8
10
12
14
16
18
20
Number of Pixels (in millions)
Ru
nn
ing
Tim
e (s
ecs)
Two dimensional images
Our solverMATLAB’s direct solver
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Combinatorial PreconditionersRecursive Preconditioned MethodsAnalysisRun Times
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90
2
4
6
8
10
12
Number of Pixels (in millions)
Ru
nn
ing
Tim
e (s
eco
nd
s)Three dimensional images
Our solverdirect solver
Out of memory
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Symmetric Diagonally DominateSystems
• Goal: Show how to solve SDD using regularLaplacians.
• Let G = (V, E, w) such that ij ∈ E:wij 6= 0• Weighted incidence matrix: A.• Degree of vi: di =
∑j |wij|
• Generalized Laplacian: L = D − A• Note: Every SDD is a Generalized
Laplacian.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Symmetric Diagonally DominateSystems
• Goal: Show how to solve SDD using regularLaplacians.
• Let G = (V, E, w) such that ij ∈ E:wij 6= 0
• Weighted incidence matrix: A.• Degree of vi: di =
∑j |wij|
• Generalized Laplacian: L = D − A• Note: Every SDD is a Generalized
Laplacian.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Symmetric Diagonally DominateSystems
• Goal: Show how to solve SDD using regularLaplacians.
• Let G = (V, E, w) such that ij ∈ E:wij 6= 0• Weighted incidence matrix: A.
• Degree of vi: di =∑
j |wij|• Generalized Laplacian: L = D − A• Note: Every SDD is a Generalized
Laplacian.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Symmetric Diagonally DominateSystems
• Goal: Show how to solve SDD using regularLaplacians.
• Let G = (V, E, w) such that ij ∈ E:wij 6= 0• Weighted incidence matrix: A.• Degree of vi: di =
∑j |wij|
• Generalized Laplacian: L = D − A• Note: Every SDD is a Generalized
Laplacian.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Symmetric Diagonally DominateSystems
• Goal: Show how to solve SDD using regularLaplacians.
• Let G = (V, E, w) such that ij ∈ E:wij 6= 0• Weighted incidence matrix: A.• Degree of vi: di =
∑j |wij|
• Generalized Laplacian: L = D − A
• Note: Every SDD is a GeneralizedLaplacian.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Symmetric Diagonally DominateSystems
• Goal: Show how to solve SDD using regularLaplacians.
• Let G = (V, E, w) such that ij ∈ E:wij 6= 0• Weighted incidence matrix: A.• Degree of vi: di =
∑j |wij|
• Generalized Laplacian: L = D − A• Note: Every SDD is a Generalized
Laplacian.Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Basic Properties: GeneralizedLaplacian
• xTLx =∑wij>0 wij(xi − xj)
2 −∑
wij<0 wij(xi + xj)2
• Thus L is positive semidefinite.• Claim: Rank = n− 1 if G is connected.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Basic Properties: GeneralizedLaplacian
• xTLx =∑wij>0 wij(xi − xj)
2 −∑
wij<0 wij(xi + xj)2
• Thus L is positive semidefinite.
• Claim: Rank = n− 1 if G is connected.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Basic Properties: GeneralizedLaplacian
• xTLx =∑wij>0 wij(xi − xj)
2 −∑
wij<0 wij(xi + xj)2
• Thus L is positive semidefinite.• Claim: Rank = n− 1 if G is connected.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Solving Generalized Laplacian byChange of Variables
• First Idea: Find a change of variables it get aregular Laplacian.
• Note: multiplying the ith column and row by−1 preserves Laplacian.
• This is just Flipping xi and bi to −xi and −bi.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Solving Generalized Laplacian byChange of Variables
• First Idea: Find a change of variables it get aregular Laplacian.
• Note: multiplying the ith column and row by−1 preserves Laplacian.
• This is just Flipping xi and bi to −xi and −bi.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Solving Generalized Laplacian byChange of Variables
• First Idea: Find a change of variables it get aregular Laplacian.
• Note: multiplying the ith column and row by−1 preserves Laplacian.
• This is just Flipping xi and bi to −xi and −bi.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Generalized Laplacians Example
+1
V1
−1
V2−1V3 2 +1 −1
+1 2 +1−1 +1 2
x1
x2
x3
=
b1
b2
b3
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Change of Variables for Laplacians
+1
V1
−1
V2−1V3
+1−1
2 −1 +1−1 2 +1+1 +1 2
−x1
x2
x3
=
−b1
b2
b3
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Orientable Generalized Laplacians
• DEF: G = (V, E, w) is orientable is ∃sequence of flips s.t. w > 0.
• DEF: LG is orientable if G is.• Note: Orientability is linear testable, greedy.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Orientable Generalized Laplacians
• DEF: G = (V, E, w) is orientable is ∃sequence of flips s.t. w > 0.
• DEF: LG is orientable if G is.
• Note: Orientability is linear testable, greedy.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Orientable Generalized Laplacians
• DEF: G = (V, E, w) is orientable is ∃sequence of flips s.t. w > 0.
• DEF: LG is orientable if G is.• Note: Orientability is linear testable, greedy.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Orientable Generalized Laplacians
• Claim: If G is connected and not orientablethen L is SPD.
• Proof: Suppose xTLx = 0 and x 6= 0
• Pick a spanning tree T of G and orient it andflipping x.
• WLOG: x is the all-ones vector, a contra!,since G still has negative edges.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Orientable Generalized Laplacians
• Claim: If G is connected and not orientablethen L is SPD.
• Proof: Suppose xTLx = 0 and x 6= 0
• Pick a spanning tree T of G and orient it andflipping x.
• WLOG: x is the all-ones vector, a contra!,since G still has negative edges.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Orientable Generalized Laplacians
• Claim: If G is connected and not orientablethen L is SPD.
• Proof: Suppose xTLx = 0 and x 6= 0
• Pick a spanning tree T of G and orient it andflipping x.
• WLOG: x is the all-ones vector, a contra!,since G still has negative edges.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Orientable Generalized Laplacians
• Claim: If G is connected and not orientablethen L is SPD.
• Proof: Suppose xTLx = 0 and x 6= 0
• Pick a spanning tree T of G and orient it andflipping x.
• WLOG: x is the all-ones vector, a contra!,since G still has negative edges.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Two-Fold Covers• DEF: The 2-fold cover G = (V , W , w) ofG = (V, W, w) is:
• V = {V1, V1, . . . , Vn, Vn}• E: If wij > 0
add edges < Vi, Vj > and < Vi, Vj > withweight wij
• E: If wij < 0add edges < Vi, Vj > and < Vi, Vj > withweight −wij
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Two-Fold Covers• DEF: The 2-fold cover G = (V , W , w) ofG = (V, W, w) is:
• V = {V1, V1, . . . , Vn, Vn}
• E: If wij > 0add edges < Vi, Vj > and < Vi, Vj > withweight wij
• E: If wij < 0add edges < Vi, Vj > and < Vi, Vj > withweight −wij
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Two-Fold Covers• DEF: The 2-fold cover G = (V , W , w) ofG = (V, W, w) is:
• V = {V1, V1, . . . , Vn, Vn}• E: If wij > 0
add edges < Vi, Vj > and < Vi, Vj > withweight wij
• E: If wij < 0add edges < Vi, Vj > and < Vi, Vj > withweight −wij
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Two-Fold Covers• DEF: The 2-fold cover G = (V , W , w) ofG = (V, W, w) is:
• V = {V1, V1, . . . , Vn, Vn}• E: If wij > 0
add edges < Vi, Vj > and < Vi, Vj > withweight wij
• E: If wij < 0add edges < Vi, Vj > and < Vi, Vj > withweight −wij
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Example: Two-Fold Cover
+1
V1
−1
V2−1V3
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Two-Fold Covers and SolvingGeneralized Laplacians
• Let L = L(G) and L = G
• Note:
Lx = b then L
(x
−x
)=
(b
−b
)•
L
(xy
)=
(b
−b
)then L(x/2− y/2) = b
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Two-Fold Covers and SolvingGeneralized Laplacians
• Let L = L(G) and L = G• L is a regular Laplacian which we can solve
quickly.
• Note:
Lx = b then L
(x
−x
)=
(b
−b
)•
L
(xy
)=
(b
−b
)then L(x/2− y/2) = b
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Two-Fold Covers and SolvingGeneralized Laplacians
• Let L = L(G) and L = G
• Note:
Lx = b then L
(x
−x
)=
(b
−b
)
•
L
(xy
)=
(b
−b
)then L(x/2− y/2) = b
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Two-Fold Covers and SolvingGeneralized Laplacians
• Let L = L(G) and L = G
• Note:
Lx = b then L
(x
−x
)=
(b
−b
)•
L
(xy
)=
(b
−b
)then L(x/2− y/2) = b
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
SDD systems
• The 2-fold trick can be run without the factorof two in space and time.
• There should be uses of negative weights inrecommendation problems.
• Naive approach does not seem to work right.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
SDD systems
• The 2-fold trick can be run without the factorof two in space and time.
• There should be uses of negative weights inrecommendation problems.
• Naive approach does not seem to work right.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
SDD systems
• The 2-fold trick can be run without the factorof two in space and time.
• There should be uses of negative weights inrecommendation problems.
• Naive approach does not seem to work right.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Spectral Graph Partitioning• Idea: Pick a few low frequency eigenvectors.
• Use these vectors to embed the graph in Rd
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Spectral Graph Partitioning• Idea: Pick a few low frequency eigenvectors.• Use these vectors to embed the graph in Rd
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
The Blue Sky ProblemShi Malik applied to an image:
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
The Blue Sky ProblemShi Malik solution:
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
The Blue Sky ProblemSpectral Rounding applied to Image:
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Spectral RoundingEdge Reweighting
Algorithm:• Solve Lf = λ2Df .• Reweight graph edges getting L′ and D′.• Solve L′f = λ2D
′f
• Repeat while λ2 6= 0.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
SR: The reweighting scheme
• View D, f , and λ2 as a function of L
• Subject to Lf = λ2Df and fTDf = 1.• We get: ∂λ2
∂eij= (fi − fj)
2 − λ2(f2i + f 2
j )
• Take a “small” step in the direction of thegradient.
• If an edge goes negative set it to zero.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Medical Examples of SR
Breast Tumors
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
Medical Examples of SR
Breast Tumors
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications
IntroductionGraph Based Methods
Iterative Methods for LaplaciansSymmetric Diagonally Dominate Systems
Spectral RoundingOpen Questions
• Find fast methods for any SPD system.• Find spectral methods that find better cut by
using more than one eigenvector.• Find solvers that work in the L2 norm.• A implementable solver with near linear time
guarantees.
Gary L. Miller Spectral Graph Theory, Linear Solvers, and Applications