Math 671: Tensor Train decomposition methods IIcoronae/Talk_UM_TT_lecture2.pdf · 2018-01-05 ·...
Transcript of Math 671: Tensor Train decomposition methods IIcoronae/Talk_UM_TT_lecture2.pdf · 2018-01-05 ·...
TT methods
Math 671: Tensor Train decomposition methodsII
Eduardo Corona1
1University of Michigan at Ann Arbor
December 13, 2016
TT methods
Table of Contents
1 What we’ve talked about so far:
2 The Tensor Train decomposition
3 Fast algorithms for TT arithmetic
TT methods
What we’ve talked about so far:
What we’ve talked about so far:
We talked about tensors, and how we can vectorize a tensoror tensorize a vector or matrix. This creates relationshipsbetween tensor and vector / matrix algebra.
Our goal: to use the Tensor train decomposition to compressfunctions (vectors) and linear operators (matrices).
We discussed unfolding matrices for tensorized vectors andmatrices, when they come from function and kernelevaluations, respectively.
TT methods
What we’ve talked about so far:
Tensorized vector index (domain hierarchy)
i
Ωi
i
i
x
1
2
3
1
1 1
1 1 1 1
0
0 0
0 0 0 0
Domain
UnfoldingMatrix
FT ( i i , i ) = f(x )1 2 3 i
TT methods
What we’ve talked about so far:
Model problem: Integral Equations for PDEs
Many boundary value problems from classical physics, when cast asboundary or volume integral equations, take the form
A[σ](x) = a(x)σ(x) +
∫Γ
K (x , y)σ(y)dS(y) = f (x), ∀x ∈ Γ,
K (x , y) kernel function related to PDE fundamental solution.
Typically singular near diagonal (y = x) but otherwise smooth.
We prefer Fredholm of the 2nd kind (Identity + Compact).
TT methods
What we’ve talked about so far:
Tensorized matrix index (interaction / block hierarchy)
A =
A1 =
16× 16
M1
=
16× r1
U1 ⇒ G1
[ ]
r1 × 16
V1
4r1 × 4
M2
=
4r1 × r2
U2 ⇒ G2
[ ]
r2 × 4
V2
4r2 × 1
M3 = U3 ⇒ G3
=
Unfolding matrix A` → all interactions between source and targetnodes at a given level.
TT methods
The Tensor Train decomposition
What is the TT decomposition?
For a d dimensional tensor A sampled at N =∏d
i=1 ni pointsindexed by (i1, i2, . . . , id), this decomposition can be written as:
A(i1, i2, . . . , id) ≈∑
α1,...,αd−1
G1(i1, α1)G2(α1, i2, α2) . . .Gd(αd−1, id)
Each Gk is known as a tensor core. Auxiliary indices αk determinenumber of terms in the decomposition and run from 1 to rk . rk isknown as the kth TT rank.
TT methods
The Tensor Train decomposition
Intuition: extension of the SVD
For a matrix A of size N × N and rank k , using the SVD wefind:
A(i , j) =∑α
U(i , α)S(α, α)V (α, j)
Where U(i , α),V (α, j) are members of orthonormal bases forthe row and column spaces, respectively.
You need to store two matrices of size N × k ( 2kN numbersinstead of N2)
TT methods
The Tensor Train decomposition
Observations about the TT
Note each tensor core Gk(αk−1, ik , αk) is a 3-tensor, thatdepends on only one tensor index ik .
You can think of it as an extension of the SVD to tensors.
Quasi-optimal analogue to the optimal low rankapproximation.
We need to store d tensors of size rk−1 × nk × rk .
If all ranks rk are bounded by r and mode sizes are equal to n,that is: ∑
k
nk rk−1rk ≤ dnr2
If ranks are small, that can be much smaller than N = nd
TT methods
The Tensor Train decomposition
How to compute TT cores and ranks?
The k-th TT rank is the rank of the k-th unfolding matrix Ak .
A TT decomposition may be obtained by a series of low rankapproximations (using a truncated SVD, interpolativedecompositions, or others).
TT methods
The Tensor Train decomposition
TT decomposition algorithm (1)
If we compute our favorite low rank approximation of A1, weobtain:
A1(i1, i2..id) = U1(i1, α1)V1(α1, i2..id)
The first core G1(α1, i1) is a reshaping of U1.
TT methods
The Tensor Train decomposition
TT decomposition algorithm (2)
We can iterate this procedure for V1(α1i2, i3, .., id):
V1(α1i2, i3..id) = U2(α1i2, α2)V2(α2, i3..id)
The second core G2(α1, i2, α2) is a reshaping of U2.
TT methods
The Tensor Train decomposition
TT decomposition algorithm (k)
We continue this low rank approximation process until we runout of indices.
We end up with d tensor cores Gk(αk−1, ik , αk).
From this construction, a low rank approximation for anyunfolding matrix Ak can be obtained by contracting all indicesexcept αk
TT methods
The Tensor Train decomposition
TT for vectors (function samples)
Along with View function unfoldingmat.m, I’ve also addedget ttf.m to obtain the TT decomposition for the tenzorizedform of f (x) sampled in N = 2d points in [a, b].
This uses the tt tensor, full and round routines from the TTtoolbox.
TT methods
The Tensor Train decomposition
TT for matrices (Kernel samples)
Along with View matrix unfoldingmat.m, I’ve also addedget tta.m to obtain the TT decomposition for the tenzorizedform of K (x , y) sampled in N × N = 2d × 2d points in[a, b]× [c , d ].
This uses the tt tensor, tt matrix, full, and round routinesfrom the TT toolbox.
TT methods
The Tensor Train decomposition
TT for matrices
A =
A1 =
16× 16
M1
=
16× r1
U1 ⇒ G1
[ ]
r1 × 16
V1
4r1 × 4
M2
=
4r1 × r2
U2 ⇒ G2
[ ]
r2 × 4
V2
4r2 × 1
M3 = U3 ⇒ G3
=
TT methods
The Tensor Train decomposition
Why does it achieve better compression?
TT rank is low if there exists a small basis of interactions
For many examples from differential and integral equations,ranks are actually bounded or grow very slowly (like logN)
Other examples have higher growth (Toeplitz have ranks withgrowth N1/2)
Symmetries, and particularly translation or rotation invariancereduce TT ranks significantly.
TT methods
Fast algorithms for TT arithmetic
TT Toolbox
All the routines discussed in this lecture can be found in theMatlab TT Toolbox, by Ivan Oseledets and his researchgroup.
The two families of objects we’ll be working with are tttensor and tt matrix.
Besides basic routines to manipulate these, we’ll be looking atfast algorithms for TT matrices in the cross and solvesubdirectories.
TT methods
Fast algorithms for TT arithmetic
Fast algorithms for TT
Given function f (x), kernel K (x , y) or tensor we can sampleentries from,
amen cross or dmrg cross: produce TT decompositionsfaster (O
(r3d)) than the standard TT algorithm.
Given matrix A in the TT matrix format, we can compute
mtimes: Matrix-vector product y = Ab for a dense vector b
mtimes: If b is also compressed in the tt tensor format, ituses a different algorithm that computes y in tt tensor form.
amen solve2 or dmrg solve3: solve linear system Ax = b orproduce a TT decomposition for A−1.
TT methods
Fast algorithms for TT arithmetic
Fast compression example: amen cross
Inputs: a tensor entry evaluator routine.
How does it work: starting from an initial guess, it iteratesover all cores trying to find the best approximation for givenranks.
Each core is updated while the others stay fixed.
After each pass, it uses an approximation of the residual toincrease ranks / tensor evaluations until convergence.
Function call: amen cross(Size,Eval,acc,params);
TT methods
Fast algorithms for TT arithmetic
Fast matrix-vector apply
Inputs: matrix in tt matrix form, vector b in dense form.
How does it work: applying Gk , it contracts one column indexjk at a time, producing results for the corresponding row indexik .
Function call: y = mtimes(TTA,b);
TT methods
Fast algorithms for TT arithmetic
Fast solve and inversion example: amen solve
Inputs: matrix in tt matrix form, vector b in tt tensor form.
How does it work: starting from an initial guess, it iteratesover all cores solving local linear systems.
Each linear system holds all cores but one fixed.
After each pass, it uses an approximation of the residual toincrease ranks / tensor evaluations until convergence.
Function call: amen cross(Size,Eval,acc,params);