Tensor Decompositions, Matrix Completion andSingular Values

Harm Derksen

University of Michigan

ICERM, Computational Nonlinear AlgebraJune 3, 2014

Harm Derksen Tensors, LRMC and Singular Values

Tensor Product Spaces

F a fieldV (i) ∼= Fni F-vector space for i = 1, 2, . . . , dV = V (1) ⊗ V (2) ⊗ · · · ⊗ V (d) ∼= Fn1×···×nd tensor product space

Definition

A pure tensor is a tensor of the form v (1) ⊗ v (2) ⊗ · · · ⊗ v (d)

(v (i) ∈ V (i)).

Problem

Write a given tensor as a sum of the smallest number of puretensors.

(for F = R,C: Canonical Polyadic (CP) decompositions,PARAFAC, CANDECOMP)

Harm Derksen Tensors, LRMC and Singular Values

Tensor Product Spaces

F a fieldV (i) ∼= Fni F-vector space for i = 1, 2, . . . , dV = V (1) ⊗ V (2) ⊗ · · · ⊗ V (d) ∼= Fn1×···×nd tensor product space

Definition

A pure tensor is a tensor of the form v (1) ⊗ v (2) ⊗ · · · ⊗ v (d)

(v (i) ∈ V (i)).

Problem

Write a given tensor as a sum of the smallest number of puretensors.

(for F = R,C: Canonical Polyadic (CP) decompositions,PARAFAC, CANDECOMP)

Harm Derksen Tensors, LRMC and Singular Values

Applications

I psychometrics

I chemometrics

I algebraic complexity theory

I signal processing

I numerical linear algebra

I computer vision

I numerical analysis

I data mining

I graph analysis

I neuroscience

I economics/finance

Harm Derksen Tensors, LRMC and Singular Values

Application: Fluorescence Spectroscopy

We have p samples of mixtures of unknown chemical compounds.Every mixture is excited with light of m different wavelengths.Light of n wavelengths is emitted from the mixture . Measuringthe intensities, one obtains an m × n excitation-emisson matrix forevery sample. This yields a p ×m × n matrix, which is a 3-waytensor T .Every chemical compound corresponds to a rank 1 tensor. Bywriting T as the sum of r = rank(T ) pure tensors, we candistinguish r chemical compounds and find the excitation-emissionmatrix for each of them.

Harm Derksen Tensors, LRMC and Singular Values

Application:Fluorescence Spectroscopy(Image: Lei Li, Andrew Barron)

Harm Derksen Tensors, LRMC and Singular Values

Tensor rank

Definition

The rank of a tensor T ∈ V is the smallest nonnegative integer rsuch that we can write T as a sum of r pure tensors.

For example,

T = e1 ⊗ e1 ⊗ e1 + e1 ⊗ e2 ⊗ e2 + e2 ⊗ e1 ⊗ e2 + e2 ⊗ e2 ⊗ e1

has rank 2 because

T = 12(e1+e2)⊗(e1+e2)⊗(e1+e2)+ 1

2(e1−e2)⊗(e1−e2)⊗(e1−e2)

If d = 2 and V = V (1) ⊗ V (2) ∼= Rd1×d2 then the tensor rank isjust the rank of a matrix.

Harm Derksen Tensors, LRMC and Singular Values

Tensor rank

Definition

The rank of a tensor T ∈ V is the smallest nonnegative integer rsuch that we can write T as a sum of r pure tensors.

For example,

T = e1 ⊗ e1 ⊗ e1 + e1 ⊗ e2 ⊗ e2 + e2 ⊗ e1 ⊗ e2 + e2 ⊗ e2 ⊗ e1

has rank 2 because

T = 12(e1+e2)⊗(e1+e2)⊗(e1+e2)+ 1

2(e1−e2)⊗(e1−e2)⊗(e1−e2)

If d = 2 and V = V (1) ⊗ V (2) ∼= Rd1×d2 then the tensor rank isjust the rank of a matrix.

Harm Derksen Tensors, LRMC and Singular Values

Tensor rank

Definition

The rank of a tensor T ∈ V is the smallest nonnegative integer rsuch that we can write T as a sum of r pure tensors.

For example,

T = e1 ⊗ e1 ⊗ e1 + e1 ⊗ e2 ⊗ e2 + e2 ⊗ e1 ⊗ e2 + e2 ⊗ e2 ⊗ e1

has rank 2 because

T = 12(e1+e2)⊗(e1+e2)⊗(e1+e2)+ 1

2(e1−e2)⊗(e1−e2)⊗(e1−e2)

If d = 2 and V = V (1) ⊗ V (2) ∼= Rd1×d2 then the tensor rank isjust the rank of a matrix.

Harm Derksen Tensors, LRMC and Singular Values

Tensor rank

Definition

The rank of a tensor T ∈ V is the smallest nonnegative integer rsuch that we can write T as a sum of r pure tensors.

For example,

T = e1 ⊗ e1 ⊗ e1 + e1 ⊗ e2 ⊗ e2 + e2 ⊗ e1 ⊗ e2 + e2 ⊗ e2 ⊗ e1

has rank 2 because

T = 12(e1+e2)⊗(e1+e2)⊗(e1+e2)+ 1

2(e1−e2)⊗(e1−e2)⊗(e1−e2)

If d = 2 and V = V (1) ⊗ V (2) ∼= Rd1×d2 then the tensor rank isjust the rank of a matrix.

Harm Derksen Tensors, LRMC and Singular Values

Application: Algebraic Complexity Theory

V = Matn,n(F)⊗Matn,n(F)⊗Matn,n(F)

Tn =∑n

i ,j ,k=1 ei ,j ⊗ ej ,k ⊗ ek,i

rank(Tn) is the number of multiplications needed to multiply twon × n matrices. Clearly rank(Tn) ≤ n3.

Theorem

If rank(Tm) ≤ k , then rank(Tn) = O(nlogm(k)) and two n × nmatrices can be multiplied using O(nlogm(k)) arithmetic operations.

Harm Derksen Tensors, LRMC and Singular Values

Application: Algebraic Complexity Theory

V = Matn,n(F)⊗Matn,n(F)⊗Matn,n(F)

Tn =∑n

i ,j ,k=1 ei ,j ⊗ ej ,k ⊗ ek,i

rank(Tn) is the number of multiplications needed to multiply twon × n matrices. Clearly rank(Tn) ≤ n3.

Theorem

If rank(Tm) ≤ k , then rank(Tn) = O(nlogm(k)) and two n × nmatrices can be multiplied using O(nlogm(k)) arithmetic operations.

Harm Derksen Tensors, LRMC and Singular Values

Application: Algebraic Complexity Theory

Theorem (Strassen 1969)

rank(T2) ≤ 7, so rank(Tn) = O(nlog2(7)) = O(n2.8073...).

Theorem (Williams 2012)

rank(Tn) = O(n2.3727)

Theorem (Masseranti, Raviolo 2013)

rank(Tn) ≥ 3n2 − 2√

2n3/2 − 3n.

Harm Derksen Tensors, LRMC and Singular Values

Low Rank Matrix Completion

Problem

Given a partially filled matrix, complete the matrix such that theresulting matrix has the smallest possible rank.

For example −2 · ·2 3 ·· 6 2

can be completed to a rank 1 matrix−2 −3 −1

2 3 14 6 2

Harm Derksen Tensors, LRMC and Singular Values

Low Rank Matrix Completion

Problem

Given a partially filled matrix, complete the matrix such that theresulting matrix has the smallest possible rank.

For example −2 · ·2 3 ·· 6 2

can be completed to a rank 1 matrix−2 −3 −12 3 14 6 2

Harm Derksen Tensors, LRMC and Singular Values

Low Rank Matrix Completion

Problem

Given a partially filled matrix, complete the matrix such that theresulting matrix has the smallest possible rank.

For example −2 · ·2 3 ·· 6 2

can be completed to a rank 1 matrix−2 −3 −1

2 3 14 6 2

Harm Derksen Tensors, LRMC and Singular Values

Application: The Netflix Problem

The DVD rental company Netflix has 480,189 users, and 17,770movies. The user ratings for every user can be put in a480, 189× 17, 770 matrix A = (ai ,j) for which only few entries areknown (since most users have only seen a fraction of the 17,770movies).

Presumably, the rank of the matrix A is low. Using Low rankmatrix completion, Netflix can predict whether users like a moviethey have seen, and will recommend movies to the users.

Harm Derksen Tensors, LRMC and Singular Values

Reduction LRMC to Tensor Rank

A is n ×m matrixare allowed to change entries in positions (i1, j1), . . . , (is , js).mrank(A) minimal possible rank

Define

T =s∑

k=1

eik ⊗ ejk ⊗ ek + A⊗ es+1 ∈ Cn ⊗ Cm ⊗ Cs+1

Theorem (D.)

rank(T ) = mrank(A) + s

Harm Derksen Tensors, LRMC and Singular Values

Reduction LRMC to Tensor Rank

A is n ×m matrixare allowed to change entries in positions (i1, j1), . . . , (is , js).mrank(A) minimal possible rank

Define

T =s∑

k=1

eik ⊗ ejk ⊗ ek + A⊗ es+1 ∈ Cn ⊗ Cm ⊗ Cs+1

Theorem (D.)

rank(T ) = mrank(A) + s

Harm Derksen Tensors, LRMC and Singular Values

Example

A =

(1 t· 1

)For T = e2 ⊗ e1 ⊗ e1 + e1 ⊗ (e1 + te2)⊗ e2 + e2 ⊗ e2 ⊗ e2 we haverank(T ) = 1 + mrank(A).

mrank(A) = 1 if t 6= 0 and mrank(A) = 2 if t = 0.

If t = 0, then T = e2 ⊗ e1 ⊗ e1 + e1 ⊗ e1 ⊗ e2 + e2 ⊗ e2 ⊗ e2 hasrank 3.

If t 6= 0 then rank(T ) = 2 and

T = e2 ⊗ e1 ⊗ (e1 − t−1e2) + (e1 + t−1e2)⊗ (e1 + te2)⊗ e2

Harm Derksen Tensors, LRMC and Singular Values

Example

A =

(1 t· 1

)For T = e2 ⊗ e1 ⊗ e1 + e1 ⊗ (e1 + te2)⊗ e2 + e2 ⊗ e2 ⊗ e2 we haverank(T ) = 1 + mrank(A).

mrank(A) = 1 if t 6= 0 and mrank(A) = 2 if t = 0.

If t = 0, then T = e2 ⊗ e1 ⊗ e1 + e1 ⊗ e1 ⊗ e2 + e2 ⊗ e2 ⊗ e2 hasrank 3.

If t 6= 0 then rank(T ) = 2 and

T = e2 ⊗ e1 ⊗ (e1 − t−1e2) + (e1 + t−1e2)⊗ (e1 + te2)⊗ e2

Harm Derksen Tensors, LRMC and Singular Values

Example

A =

(1 t· 1

)For T = e2 ⊗ e1 ⊗ e1 + e1 ⊗ (e1 + te2)⊗ e2 + e2 ⊗ e2 ⊗ e2 we haverank(T ) = 1 + mrank(A).

mrank(A) = 1 if t 6= 0 and mrank(A) = 2 if t = 0.

If t = 0, then T = e2 ⊗ e1 ⊗ e1 + e1 ⊗ e1 ⊗ e2 + e2 ⊗ e2 ⊗ e2 hasrank 3.

If t 6= 0 then rank(T ) = 2 and

T = e2 ⊗ e1 ⊗ (e1 − t−1e2) + (e1 + t−1e2)⊗ (e1 + te2)⊗ e2

Harm Derksen Tensors, LRMC and Singular Values

Example

A =

(1 t· 1

)For T = e2 ⊗ e1 ⊗ e1 + e1 ⊗ (e1 + te2)⊗ e2 + e2 ⊗ e2 ⊗ e2 we haverank(T ) = 1 + mrank(A).

mrank(A) = 1 if t 6= 0 and mrank(A) = 2 if t = 0.

If t = 0, then T = e2 ⊗ e1 ⊗ e1 + e1 ⊗ e1 ⊗ e2 + e2 ⊗ e2 ⊗ e2 hasrank 3.

If t 6= 0 then rank(T ) = 2 and

T = e2 ⊗ e1 ⊗ (e1 − t−1e2) + (e1 + t−1e2)⊗ (e1 + te2)⊗ e2

Harm Derksen Tensors, LRMC and Singular Values

Rank Minimization Problem (RM)

Problem (Rank Minimization)

Given A,B1, . . . ,Bs ∈ Matp,q(F), minimizerank(A + x1B1 + · · ·+ xsBs).

LRMC is a special case of RM where Bk = eik ,jk .

Theorem (D.)

The Tensor Rank problem can be reduced to RM and RM can bereduced to LRMC.

For example, suppose that T = (ti ,j ,k)2i ,j ,k=1 is a 2× 2× 2 tensor.

Harm Derksen Tensors, LRMC and Singular Values

Rank Minimization Problem (RM)

Problem (Rank Minimization)

Given A,B1, . . . ,Bs ∈ Matp,q(F), minimizerank(A + x1B1 + · · ·+ xsBs).

LRMC is a special case of RM where Bk = eik ,jk .

Theorem (D.)

The Tensor Rank problem can be reduced to RM and RM can bereduced to LRMC.

For example, suppose that T = (ti ,j ,k)2i ,j ,k=1 is a 2× 2× 2 tensor.

Harm Derksen Tensors, LRMC and Singular Values

t1,1,1 t1,2,1 a1,1 a1,2 a1,3 0 0 0 0 0 0 0 0 0 0 0 0 0 0t2,1,1 t2,2,1 a2,1 a2,2 a2,3 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 1 0 0 λ1,1 0 0 0 0 0 0 0 0 0 0 0 0 00 0 0 1 0 0 λ1,2 0 0 0 0 0 0 0 0 0 0 0 00 0 0 0 1 0 0 λ1,3 0 0 0 0 0 0 0 0 0 0 0

b1,1 b2,1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0b1,2 b2,2 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0b1,3 b2,3 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 t1,1,2 t1,2,2 a1,1 a1,2 a1,3 0 0 0 0 0 00 0 0 0 0 0 0 0 t2,1,2 t2,2,2 a2,1 a2,2 a2,3 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 1 0 0 λ2,1 0 0 0 0 00 0 0 0 0 0 0 0 0 0 0 1 0 0 λ2,2 0 0 0 00 0 0 0 0 0 0 0 0 0 0 0 1 0 0 λ2,3 0 0 00 0 0 0 0 0 0 0 b1,1 b2,1 0 0 0 1 0 0 0 0 00 0 0 0 0 0 0 0 b1,2 b2,2 0 0 0 0 1 0 0 0 00 0 0 0 0 0 0 0 b1,3 b2,3 0 0 0 0 0 1 0 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 a1,1 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 a2,1 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 b1,1 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 b2,1 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 a1,2 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 a2,2 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 b1,2 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 b2,2 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 a1,30 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 a2,30 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 b1,30 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 b2,3

.

The smallest rank over all ai ,j , bi ,j , λi ,j (1 ≤ i ≤ 2, 1 ≤ j ≤ 3) is12 + rank(T ).

Harm Derksen Tensors, LRMC and Singular Values

Motivation: compressed sensing and convex relaxation

For x ∈ Rn, its sparsity is measured by ‖x‖0 = #{i | xi 6= 0}.

Problem

Given A ∈ Matm,n(R), b ∈ Rm, find a solution x ∈ Rn for Ax = bwith ‖x‖0 minimal (a sparsest solution).

But, ‖ · ‖0 is not convex and this optimization problem is difficult,

so instead we consider:

Problem (Basis Pursuit)

Given A ∈ Matm,n(R), b ∈ Rm, find a solution x ∈ Rn for Ax = bwith ‖x‖1 minimal.

Basis Pursuit can be solved by linear programming and is generallyfast. Under reasonable assumptions, Basis Pursuit also gives thesparsest solutions (Candes-Tao, Donoho).

Harm Derksen Tensors, LRMC and Singular Values

Motivation: compressed sensing and convex relaxation

For x ∈ Rn, its sparsity is measured by ‖x‖0 = #{i | xi 6= 0}.

Problem

Given A ∈ Matm,n(R), b ∈ Rm, find a solution x ∈ Rn for Ax = bwith ‖x‖0 minimal (a sparsest solution).

But, ‖ · ‖0 is not convex and this optimization problem is difficult,so instead we consider:

Problem (Basis Pursuit)

Given A ∈ Matm,n(R), b ∈ Rm, find a solution x ∈ Rn for Ax = bwith ‖x‖1 minimal.

Basis Pursuit can be solved by linear programming and is generallyfast. Under reasonable assumptions, Basis Pursuit also gives thesparsest solutions (Candes-Tao, Donoho).

Harm Derksen Tensors, LRMC and Singular Values

Nuclear Norm

V (i) Hilbert space. Instead of the CP model, we consider:

Problem (Convex Decomposition)

Given a tensor T , write T =∑r

i=1 vi for some r and some puretensors v1, . . . , vr such that

∑ri=1 ‖vi‖2 is minimal.

Finding a convex decomposition seems to be easier than finding aCP decomposition. Heuristically, we expect convex decompositionsto give a CP decomposition or at least a low rank decomposition.

Definition (Lim-Comon)

The nuclear norm ‖T‖? is the smallest value of∑r

i=1 ‖vi‖2 whereT =

∑ri=1 vi and v1, . . . , vr are pure tensors.

For some tensors we know the nuclear norm but not the rank.

Harm Derksen Tensors, LRMC and Singular Values

Nuclear Norm

V (i) Hilbert space. Instead of the CP model, we consider:

Problem (Convex Decomposition)

Given a tensor T , write T =∑r

i=1 vi for some r and some puretensors v1, . . . , vr such that

∑ri=1 ‖vi‖2 is minimal.

Finding a convex decomposition seems to be easier than finding aCP decomposition. Heuristically, we expect convex decompositionsto give a CP decomposition or at least a low rank decomposition.

Definition (Lim-Comon)

The nuclear norm ‖T‖? is the smallest value of∑r

i=1 ‖vi‖2 whereT =

∑ri=1 vi and v1, . . . , vr are pure tensors.

For some tensors we know the nuclear norm but not the rank.

Harm Derksen Tensors, LRMC and Singular Values

Spectral Norm

Definition

The spectral norm is defined by

‖T‖σ = max{|〈T , v〉| | v pure tensor with ‖v‖2 = 1}.

The spectral norm is dual to the nuclear norm, in particular

|〈T ,S〉| ≤ ‖T‖?‖S‖σ

for all tensors S ,T .

Harm Derksen Tensors, LRMC and Singular Values

Example: Determinant Tensor

Consider the tensor

Dn =∑σ∈Sn

sgn(σ)eσ(1) ⊗ eσ(2) ⊗ · · · ⊗ eσ(n) ∈ Cn ⊗ · · · ⊗ Cn.

Clearly rank(Dn) ≤ n!, actually( nbn/2c

)≤ rank(Dn) ≤ (56)bn/3cn!.

‖Dn‖σ = max{| det(v1v2 · · · vn)| | ‖v1‖2 = · · · = ‖vn‖2 = 1} = 1

by Hadamard’s inequality.

‖Dn‖? = ‖Dn‖?‖Dn‖σ ≥ 〈Dn,Dn〉 = n!, so

Theorem (D.)

‖Dn‖? = n!

Harm Derksen Tensors, LRMC and Singular Values

Example: Determinant Tensor

Consider the tensor

Dn =∑σ∈Sn

sgn(σ)eσ(1) ⊗ eσ(2) ⊗ · · · ⊗ eσ(n) ∈ Cn ⊗ · · · ⊗ Cn.

Clearly rank(Dn) ≤ n!, actually( nbn/2c

)≤ rank(Dn) ≤ (56)bn/3cn!.

‖Dn‖σ = max{| det(v1v2 · · · vn)| | ‖v1‖2 = · · · = ‖vn‖2 = 1} = 1

by Hadamard’s inequality.

‖Dn‖? = ‖Dn‖?‖Dn‖σ ≥ 〈Dn,Dn〉 = n!, so

Theorem (D.)

‖Dn‖? = n!

Harm Derksen Tensors, LRMC and Singular Values

Example: Permanent Tensor

Pn =∑

σ∈Sn eσ(1) ⊗ eσ(2) ⊗ · · · ⊗ eσ(n) ∈ Cn ⊗ · · · ⊗ Cn.

‖Pn‖σ = max{| perm(v1v2 · · · vn)| | ‖v1‖2 = · · · = ‖vn‖2 = 1}

Theorem (Carlen, Lieb and Moss, 2006)

max{perm(v1v2 · · · vn) | ‖v1‖2 = · · · = ‖vn‖2 = 1} = n!/nn/2

‖Pn‖? =nn/2

n!‖Pn‖?‖Pn‖σ ≥

nn/2

n!〈Pn,Pn〉 = nn/2.

Harm Derksen Tensors, LRMC and Singular Values

Example: Permanent Tensor

Pn =∑

σ∈Sn eσ(1) ⊗ eσ(2) ⊗ · · · ⊗ eσ(n) ∈ Cn ⊗ · · · ⊗ Cn.

‖Pn‖σ = max{| perm(v1v2 · · · vn)| | ‖v1‖2 = · · · = ‖vn‖2 = 1}

Theorem (Carlen, Lieb and Moss, 2006)

max{perm(v1v2 · · · vn) | ‖v1‖2 = · · · = ‖vn‖2 = 1} = n!/nn/2

‖Pn‖? =nn/2

n!‖Pn‖?‖Pn‖σ ≥

nn/2

n!〈Pn,Pn〉 = nn/2.

Harm Derksen Tensors, LRMC and Singular Values

Example: Permanent Tensor

Theorem (Glynn 2010)

Pn =1

2n−1

∑δ

(∏ni=1 δi

)(∑n

i=1 δiei )⊗ · · · ⊗ (∑n

i=1 δiei )

where δ runs over {1} × {−1, 1}n−1.

In particular,( nbn/2c

)≤ rank(Pn) ≤ 2n−1 and ‖Pn‖? ≤ nn/2, so

Theorem (D.)

‖Pn‖? = nn/2

Harm Derksen Tensors, LRMC and Singular Values

Example: Permanent Tensor

Theorem (Glynn 2010)

Pn =1

2n−1

∑δ

(∏ni=1 δi

)(∑n

i=1 δiei )⊗ · · · ⊗ (∑n

i=1 δiei )

where δ runs over {1} × {−1, 1}n−1.

In particular,( nbn/2c

)≤ rank(Pn) ≤ 2n−1 and ‖Pn‖? ≤ nn/2, so

Theorem (D.)

‖Pn‖? = nn/2

Harm Derksen Tensors, LRMC and Singular Values

t-Orthogonality

Definition

Pure tensors v1, v2, . . . , vr are t-orthogonal if

r∑i=1

|〈vi ,w〉|2/t ≤ 1

for every pure tensor w with ‖w‖2 = 1.

1-orthogonal ⇔ orthogonalIf t > s then t-orthogonal ⇒ s-orthogonal.

Theorem (D.)

If v1, . . . , vr ∈ V are t-orthogonal, then r ≤ dim(V )1/t .

Harm Derksen Tensors, LRMC and Singular Values

t-Orthogonality

Definition

Pure tensors v1, v2, . . . , vr are t-orthogonal if

r∑i=1

|〈vi ,w〉|2/t ≤ 1

for every pure tensor w with ‖w‖2 = 1.

1-orthogonal ⇔ orthogonalIf t > s then t-orthogonal ⇒ s-orthogonal.

Theorem (D.)

If v1, . . . , vr ∈ V are t-orthogonal, then r ≤ dim(V )1/t .

Harm Derksen Tensors, LRMC and Singular Values

t-Orthogonality

Definition

Pure tensors v1, v2, . . . , vr are t-orthogonal if

r∑i=1

|〈vi ,w〉|2/t ≤ 1

for every pure tensor w with ‖w‖2 = 1.

1-orthogonal ⇔ orthogonalIf t > s then t-orthogonal ⇒ s-orthogonal.

Theorem (D.)

If v1, . . . , vr ∈ V are t-orthogonal, then r ≤ dim(V )1/t .

Harm Derksen Tensors, LRMC and Singular Values

Horizontal and Vertical Tensor Product

Theorem (“horizontal tensor product”, D.)

If v1, . . . , vr are t-orthogonal, and w1, . . . ,wr are s-orthogonal,then v1 ⊗ w1, . . . , vr ⊗ wr are (s + t)-orthogonal.

If V = V (1) ⊗ · · · ⊗ V (d) and W = W (1) ⊗ · · · ⊗W (d), then

V �W := (V (1) ⊗W (1))⊗ · · · ⊗ (V (d) ⊗W (d)).

Theorem (“vertical tensor product”, D.)

If v1, v2, . . . , vr ∈ V and w1, . . . ,ws ∈W are t-orthogonal, then{vi � wj | 1 ≤ i ≤ r , 1 ≤ j ≤ s} are t-orthogonal.

Harm Derksen Tensors, LRMC and Singular Values

Horizontal and Vertical Tensor Product

Theorem (“horizontal tensor product”, D.)

If v1, . . . , vr are t-orthogonal, and w1, . . . ,wr are s-orthogonal,then v1 ⊗ w1, . . . , vr ⊗ wr are (s + t)-orthogonal.

If V = V (1) ⊗ · · · ⊗ V (d) and W = W (1) ⊗ · · · ⊗W (d), then

V �W := (V (1) ⊗W (1))⊗ · · · ⊗ (V (d) ⊗W (d)).

Theorem (“vertical tensor product”, D.)

If v1, v2, . . . , vr ∈ V and w1, . . . ,ws ∈W are t-orthogonal, then{vi � wj | 1 ≤ i ≤ r , 1 ≤ j ≤ s} are t-orthogonal.

Harm Derksen Tensors, LRMC and Singular Values

The Diagonal Singular Value Decomposition

Definition

Suppose that (?) : T =∑r

i=1 λivi such that λ1 ≥ · · · ≥ λr > 0and v1, . . . , vr are 2-orthogonal pure tensors of unit length, then(?) is called a diagonal singular value decomposition of T (DSVD).

If d = 2 (tensor product of 2 spaces) then the DSVD is the usualsingular value decomposition. For d > 2, the DSVD is differentfrom the Higher Order Singular Value Decomposition defined byDe Lathauer, De Moor, and Vandewalle. Not every tensor has aDSVD.

Harm Derksen Tensors, LRMC and Singular Values

The Diagonal Singular Value Decomposition

Theorem (D.)

If T has a DSVD then

‖T‖? =∑i

λi , ‖T‖2 =

√∑i

λ2i , ‖T‖σ = λ1

Theorem (D.)

If λ1 > λ2 > · · · > λr then the DSVD is unique.

Theorem (D.)

If v1, . . . , vr are t-orthogonal with t > 2, then the DSVD is unique.

Harm Derksen Tensors, LRMC and Singular Values

The Diagonal Singular Value Decomposition

Theorem (D.)

If T has a DSVD then

‖T‖? =∑i

λi , ‖T‖2 =

√∑i

λ2i , ‖T‖σ = λ1

Theorem (D.)

If λ1 > λ2 > · · · > λr then the DSVD is unique.

Theorem (D.)

If v1, . . . , vr are t-orthogonal with t > 2, then the DSVD is unique.

Harm Derksen Tensors, LRMC and Singular Values

Example: Matrix Multiplication Tensor

e1, . . . , en ∈ Cn are orthogonale1 ⊗ e1, . . . , en ⊗ en ∈ Cn ⊗ Cn are 2-orthogonal

e1 ⊗ e1 ⊗ 1, . . . , en ⊗ en ⊗ e1 ∈ Cn ⊗ Cn ⊗ C are 2-orthogonale1 ⊗ 1⊗ e1, . . . , en ⊗ 1⊗ en ∈ Cn ⊗ C⊗ Cn are 2-orthogonal1⊗ e1 ⊗ e1, . . . , 1⊗ en ⊗ en ∈ C⊗ Cn ⊗ Cn are 2-orthogonal

Using vertical tensor product, we get

{(ei ⊗ ej)⊗ (ej ⊗ ek)⊗ (ek ⊗ ei ) | 1 ≤ i , j , k ≤ n}

are 2-orthogonal.

Harm Derksen Tensors, LRMC and Singular Values

Example: Matrix Multiplication Tensor

e1, . . . , en ∈ Cn are orthogonale1 ⊗ e1, . . . , en ⊗ en ∈ Cn ⊗ Cn are 2-orthogonal

e1 ⊗ e1 ⊗ 1, . . . , en ⊗ en ⊗ e1 ∈ Cn ⊗ Cn ⊗ C are 2-orthogonale1 ⊗ 1⊗ e1, . . . , en ⊗ 1⊗ en ∈ Cn ⊗ C⊗ Cn are 2-orthogonal1⊗ e1 ⊗ e1, . . . , 1⊗ en ⊗ en ∈ C⊗ Cn ⊗ Cn are 2-orthogonal

Using vertical tensor product, we get

{(ei ⊗ ej)⊗ (ej ⊗ ek)⊗ (ek ⊗ ei ) | 1 ≤ i , j , k ≤ n}

are 2-orthogonal.

Harm Derksen Tensors, LRMC and Singular Values

Example: Matrix Multiplication Tensor

e1, . . . , en ∈ Cn are orthogonale1 ⊗ e1, . . . , en ⊗ en ∈ Cn ⊗ Cn are 2-orthogonal

e1 ⊗ e1 ⊗ 1, . . . , en ⊗ en ⊗ e1 ∈ Cn ⊗ Cn ⊗ C are 2-orthogonale1 ⊗ 1⊗ e1, . . . , en ⊗ 1⊗ en ∈ Cn ⊗ C⊗ Cn are 2-orthogonal1⊗ e1 ⊗ e1, . . . , 1⊗ en ⊗ en ∈ C⊗ Cn ⊗ Cn are 2-orthogonal

Using vertical tensor product, we get

{(ei ⊗ ej)⊗ (ej ⊗ ek)⊗ (ek ⊗ ei ) | 1 ≤ i , j , k ≤ n}

are 2-orthogonal.

Harm Derksen Tensors, LRMC and Singular Values

Example: Matrix Multiplication Tensor

Theorem (D.)

The matrix multiplication tensor

Tn =n∑

i ,j ,k=1

ei ,j ⊗ ej ,k ⊗ ek,i

is a DSVD.

The singular values of Tn are

1, 1, . . . , 1︸ ︷︷ ︸n3

In particular,

‖Tn‖? =n3∑i=1

1 = n3.

Harm Derksen Tensors, LRMC and Singular Values

Example: Discrete Fourier Transform

DefineFn =

∑1≤i,j,k≤n

i+j+k≡0 mod n

ei ⊗ ej ⊗ ek

This tensor is related to the multiplication of univariatepolynomials. Clearly rank(Fn) ≤ n2 and ‖Fn‖? ≤ n2.

Discrete Fourier Transform (DFT):

Fn =n∑

j=1

√n( 1√

n

∑ni=1 ζ

ijei )⊗ ( 1√n

∑ni=1 ζ

ijei )⊗ ( 1√n

∑ni=1 ζ

ijei ).

where ζ = eπi/n. This is the unique DSVD of Fn. So the singularvalues are

√n, . . . ,

√n (n times), rank(Fn) = n and ‖Fn‖? = n

√n.

Harm Derksen Tensors, LRMC and Singular Values

Example: Discrete Fourier Transform

DefineFn =

∑1≤i,j,k≤n

i+j+k≡0 mod n

ei ⊗ ej ⊗ ek

This tensor is related to the multiplication of univariatepolynomials. Clearly rank(Fn) ≤ n2 and ‖Fn‖? ≤ n2.

Discrete Fourier Transform (DFT):

Fn =n∑

j=1

√n( 1√

n

∑ni=1 ζ

ijei )⊗ ( 1√n

∑ni=1 ζ

ijei )⊗ ( 1√n

∑ni=1 ζ

ijei ).

where ζ = eπi/n. This is the unique DSVD of Fn. So the singularvalues are

√n, . . . ,

√n (n times), rank(Fn) = n and ‖Fn‖? = n

√n.

Harm Derksen Tensors, LRMC and Singular Values

Generalization: Group Algebra Multiplication Tensor

G is a group with n elements and CG ∼= Cn is the group algebra

TG =∑

g ,h∈Gg ⊗ h ⊗ h−1g−1.

DFT case corresponds to G = Z/nZ.

Theorem (D.)

TG has a DSVD and its singular values are√nd1, . . . ,

√nd1︸ ︷︷ ︸

d31

, . . . ,√

nds, . . . ,

√nds︸ ︷︷ ︸

d3s

where d1, d2, . . . , ds are the dimension of the irreduciblerepresentations of G .

Harm Derksen Tensors, LRMC and Singular Values

Generalization: Group Algebra Multiplication Tensor

G is a group with n elements and CG ∼= Cn is the group algebra

TG =∑

g ,h∈Gg ⊗ h ⊗ h−1g−1.

DFT case corresponds to G = Z/nZ.

Theorem (D.)

TG has a DSVD and its singular values are√nd1, . . . ,

√nd1︸ ︷︷ ︸

d31

, . . . ,√

nds, . . . ,

√nds︸ ︷︷ ︸

d3s

where d1, d2, . . . , ds are the dimension of the irreduciblerepresentations of G .

Harm Derksen Tensors, LRMC and Singular Values

Pn no DSVD

Suppose that Pn has singular values λ1, . . . , λr .

‖Pn‖? = nn/2 =∑r

i=1 λi

‖Pn‖σ = n!nn/2

= λ1

‖Pn‖22 = n! =∑r

i=1 λ2i

λ1

r∑i=1

λi = n! =r∑

i=1

λ2i

so λ1 = · · · = λr , and r = rλ1λ1

= ‖Dn‖?‖Dn‖σ = nn

n! .If n > 2 then r is not an integer!

Harm Derksen Tensors, LRMC and Singular Values

Dn no DSVD

Suppose that Dn has singular values λ1, . . . , λr . Thenλ1 = · · · = λr = 1 and r = n! (similar calculation).

v1, v2, . . . , vr

are 2-orthogonal, so r ≤ (dimV )1/2 = nn/2. So

n! = r ≤ nn/2

Not possible for n > 2.

Harm Derksen Tensors, LRMC and Singular Values

Slope decomposition

Definition

T = T1 + T2 + · · ·+ Ts is called a slope decomposition if〈Ti ,Tj〉 = ‖Ti‖?‖Tj‖σ for all i ≤ j and

‖T1‖?‖T1‖σ

<‖T2‖?‖T2‖σ

< · · · < ‖Tr‖?‖Tr‖σ

.

Slope decomposition is unique if it exists. If T has a DSVD, thenit also has a slope decomposition. But not every tensor has a slopedecomposition.

Harm Derksen Tensors, LRMC and Singular Values

Singular values for tensors with slope decomposition

Definition

Suppose that T = T1 + · · ·+ Ts is a slope decomposition,µi = ‖Ti‖σ, and λi = µi + µi+1 + · · ·+ µs for all i . Then T has

singular value λi with multiplicity ‖Ti‖?‖Ti‖σ −

‖Ti−1‖?‖Ti−1‖σ .

Dn has singular value 1 with multiplicity n!Pn has singular value n!

nn/2with multiplicity nn

n! .

Lemma

If T has slope decomposition and singular values λ1, . . . , λr withmultiplicities m1, . . . ,mr , then ‖T‖σ = λ1, ‖T‖? =

∑i miλi and

‖T‖22 =∑

i miλ2i .

Harm Derksen Tensors, LRMC and Singular Values

Singular values for tensors with slope decomposition

Definition

Suppose that T = T1 + · · ·+ Ts is a slope decomposition,µi = ‖Ti‖σ, and λi = µi + µi+1 + · · ·+ µs for all i . Then T has

singular value λi with multiplicity ‖Ti‖?‖Ti‖σ −

‖Ti−1‖?‖Ti−1‖σ .

Dn has singular value 1 with multiplicity n!Pn has singular value n!

nn/2with multiplicity nn

n! .

Lemma

If T has slope decomposition and singular values λ1, . . . , λr withmultiplicities m1, . . . ,mr , then ‖T‖σ = λ1, ‖T‖? =

∑i miλi and

‖T‖22 =∑

i miλ2i .

Harm Derksen Tensors, LRMC and Singular Values

Singular values for tensors with slope decomposition

Definition

Suppose that T = T1 + · · ·+ Ts is a slope decomposition,µi = ‖Ti‖σ, and λi = µi + µi+1 + · · ·+ µs for all i . Then T has

singular value λi with multiplicity ‖Ti‖?‖Ti‖σ −

‖Ti−1‖?‖Ti−1‖σ .

Dn has singular value 1 with multiplicity n!Pn has singular value n!

nn/2with multiplicity nn

n! .

Lemma

If T has slope decomposition and singular values λ1, . . . , λr withmultiplicities m1, . . . ,mr , then ‖T‖σ = λ1, ‖T‖? =

∑i miλi and

‖T‖22 =∑

i miλ2i .

Harm Derksen Tensors, LRMC and Singular Values

Generalizations

One can a singular spectrum for arbitrary tensors compatible withthe singular spectrum for tensors with slope decomposition, but:one may get continuous spectra or negative multiplicities.

One can define a slope decomposition and singular values wheneverwe have a vector space with two norms dual to each other.

Thank You!

Harm Derksen Tensors, LRMC and Singular Values

Generalizations

One can a singular spectrum for arbitrary tensors compatible withthe singular spectrum for tensors with slope decomposition, but:one may get continuous spectra or negative multiplicities.

One can define a slope decomposition and singular values wheneverwe have a vector space with two norms dual to each other.

Thank You!

Harm Derksen Tensors, LRMC and Singular Values

Generalizations

One can a singular spectrum for arbitrary tensors compatible withthe singular spectrum for tensors with slope decomposition, but:one may get continuous spectra or negative multiplicities.

One can define a slope decomposition and singular values wheneverwe have a vector space with two norms dual to each other.

Thank You!

Harm Derksen Tensors, LRMC and Singular Values