Lecture 3. Transpositions, Kronecker Products, and ... · Lecture 1. Introduction to Tensor...

From Matrix to Tensor:The Transition to Numerical Multilinear Algebra

Lecture 3. Transpositions, KroneckerProducts, and Contractions

Charles F. Van Loan

Cornell University

The Gene Golub SIAM Summer School 2010Selva di Fasano, Brindisi, Italy

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions

Where We Are

Lecture 1. Introduction to Tensor Computations

Lecture 2. Tensor Unfoldings

Lecture 3. Transpositions, Kronecker Products, Contractions

Lecture 4. Tensor-Related Singular Value Decompositions

Lecture 5. The CP Representation and Rank

Lecture 6. The Tucker Representation

Lecture 7. Other Decompositions and Nearness Problems

Lecture 8. Multilinear Rayleigh Quotients

Lecture 9. The Curse of Dimensionality

Lecture 10. Special Topics


What is this Lecture About?

Preview

Now we turn our attention to the engagement of thesemultidimensional data objects in two elementary operations.

Transposition. While there is only one way transpose a matrixthere are an exponential number of ways to transpose an order-dtensor. This will require (surprise surprise) more facility withmulti-indexing.

Contraction. The contraction of two tensors can be regarded as ageneralization of matrix-matrix multiplication. Analogous to howthe serious business of matrix decompositions builds on matrixmultiplication, we will find that tensor contractions have afoundational role to play in tensor computations.


What is this Lecture About?

More “Bridging the Gap”

The Kronecker product helps bridge the gap between matrixcomputations and tensor computations. For example, thecontraction between two tensors can sometimes be “reshaped” intoa matrix computation that involves Kronecker products.

So in advance of our introduction to tensor contractions, we willget familiar with this all-important matrix operation and some ofits nearby “cousins.”


Tensor Transposition

Example: A(1:4, 1:5, 1:6) → B(1:5, 1:6, 1:4)

A = randn([ 4 5 6]);B = zeros([5 6 4]);for i1=1:4

for i2 = 1:5for i3 = 1:6

B(i2,i3,i1) = A(i1,i2,i3);end

endend

How to say this: B = A< [ 2 3 1 ] >.

Tensor Transposition: The 3rd Order Case

Six possibilities...

If A ∈ IRn1×n2×n3 , then there are 6 = 3! possible transpositionsidentified by the notation A< [i j k] > where [i j k] is a permutationof [1 2 3]:

B =

A< [1 2 3] >

A< [1 3 2] >

A< [2 1 3] >

A< [2 3 1] >

A< [3 1 2] >

A< [3 2 1] >

=⇒

bijk

bikj

bjik

bjki

bkij

bkji

= aijk

for i = 1:n1, j = 1:n2, k = 1:n3.

Tensor Transposition: General Case

Definition

If A ∈ IRn1×···×nd and p = [p1, . . . , pd ] is a permutation of theindex vector 1:d , then A ∈ IRnp1×···×npd denotes thep-transpose of A and is defined by

A(jp1 , . . . , jpd) = A(j1, . . . , jd)

where 1 ≤ jk ≤ nk for k = 1:d .

Matlab Tensor Toolbox: Transposition

% If A is an order -d tensor and

% p is a permutation of 1:d, then

% the following assigns the p-transpose

% of A to B...

B = permute(A,p)

Problem 3.1. Any square matrix can be written as the sum of a symmetricmatrix and a skew-symmetric matrix:

A =

„A + AT

2

«+

„A− AT

2

«Formulate a generalization of this result for order-3 tensors and write aMatlab function that embodies your idea.


Tensor Notation: Subscript Vectors (Review)

Reference

If A ∈ IRn1×···×nd and i = (i1, . . . , id) with 1 ≤ ik ≤ nk for k = 1:d ,then

A(i) ≡ A(i1, . . . , ik)

We say that i is a subscript vector. Bold font will be useddesignate subscript vectors.

Bounds

If L and R are subscript vectors having the same dimension, thenL ≤ R means that Lk ≤ Rk for all k.

Special Cases

A subscript vector of all ones is denoted by 1. (Dimension clearfrom context.) If N is an integer, then N = N · 1.


Tensor Transposition: General Case

Definition

If A ∈ IRn1×···×nd and p = [p1, . . . , pd ] is a permutation of theindex vector 1:d , then A ∈ IRnp1×···×npd denotes thep-transpose of A and is defined by

A(j(p)) = A(j) 1 ≤ j ≤ n.

Problem 3.2. Verify that if p and q are both permutations of 1:d , then`A´<q> = A<p(q)>

where A ∈ IRn1×···×nd .

Problem 3.3. Suppose A ∈ IRn1×···×nd and that B = A[ p ] where p is apermutation of 1:d . Specify a permutation matrix P so thatB(k) = A(p(k))P.

What is a Kronecker Product?

Definition

B ⊗ C is a block matrix whose ij-th block is bijC .

An Example...[b11 b12

b21 b22

]⊗ C =

[b11C b12C

b21C b22C

]

Replicated Block Structure


What Is a Kronecker Product?

There are three ways to regard A = B ⊗ C ...

If

A =

b11 · · · b1n...

. . ....

bm1 · · · bmn

⊗

c11 · · · c1q...

. . ....

cp1 · · · cpq

then

(i). A is an m-by-n block matrix with p-by-q blocks.

(ii). A is an mp-by-nq matrix of scalars.

(iii). A is an unfolded 4-th order tensor A ∈ IRp×q×m×n:

A(i1, i2, i3, i4) = B(i3, i4)C (i1, i2)


All Possible Entry-Entry Products

4× 9 = 36

[b11 b12

b21 b22

]⊗

c11 c12 c13

c21 c22 c23

c31 c32 c33

=

b11c11 b11c12 b11c13 b12c11 b12c12 b12c13

b11c21 b11c22 b11c23 b12c21 b12c22 b12c23

b11c31 b11c32 b11c33 b12c31 b12c32 b12c33

b21c11 b21c12 b21c13 b22c11 b22c12 b22c13

b21c21 b21c22 b21c23 b22c21 b22c22 b22c23

b21c31 b21c32 b21c33 b22c31 b22c32 b22c33


Hierarchical

Kronecker Products of Kronecker Products

B ⊗ C ⊗ D=[

b11 b12

b21 b22

]⊗

c11 c12 c13 c14

c21 c22 c23 c24

c31 c32 c33 c34

c41 c42 c43 c44

⊗

d11 d12 d13

d21 d22 d23

d31 d32 d33

A 2-by-2 block matrix whose entries are 4-by-4 block matriceswhose entries are 3-by-3 matrices.


Kronecker Product Properties

Some Basic Facts...

(B ⊗ C )T = BT ⊗ CT

(B ⊗ C )−1 = B−1 ⊗ C−1

(B ⊗ C )(D ⊗ F ) = BD ⊗ CF

B ⊗ (C ⊗ D) = (B ⊗ C )⊗ D

Note that B ⊗ C 6= C ⊗ B. However, it can be shown that

C ⊗ B = PT (B ⊗ C )Q

where P and Q are perfect shuffle permutations. More later.


Kronecker Products and Structure

Structured Factors Produce a Structured Product...

B and C B ⊗ C

NonsingularLower(Upper) triangularBandedSymmetricPositive DefiniteStochasticToeplitzPermutationOrthogonal

NonsingularLower(Upper) TriangularBlock BandedSymmetricPositive DefiniteStochasticBlock ToeplitzPermutationOrthogonal


Kronecker Products and Data Sparsity

Example 1.

If B,C ∈ IRm×m andA ≈ B ⊗ C

then we are using O(m2) numbers ( the bij and cij) torepresent/approximate O(m4) data (the aij .)

Example 2.

If Bij ∈ IRm×m and

A ≈r∑

k=1

Bk,1 ⊗ Bk,2 ⊗ · · · ⊗ Bk,d

then we are using rdm2 numbers to represent/approximate amatrix with m2d entries.


Kronecker Products and Fast Matrix-Vector Multiplication

A Basic Reshaping Result...

Y = CXBT ⇔ vec(Y ) = (B ⊗ C )vec(X )

In Practice...

function y = KronTimesVector(B,C,y)% y = kron(B,C)*x[mb,nb] = size(B);[mc,nc] = size(C);Y = C*reshape(x,nc,nb)*B’;y = reshape(Y,mc*mb,1);

If both matrices are n-by-n, then work is O(n3). Ordinarily, an n2-by-n2

matrix-vector product would be O(n4).


Problem 3.4. How would you compute the matrix-vector producty = (Ip ⊗ A⊗ Ir )x where A ∈ IRq×q and x ∈ IRpqr? Assess the amount ofwork.

Problem 3.5. How would you compute the matrix-vector product

y = (A1 ⊗ A2 ⊗ A3)x where Ai ∈ IRn×n and x ∈ IRn3

?. Hint:

(A1 ⊗ A2 ⊗ A3) = (A1 ⊗ In2)(In ⊗ A2 ⊗ In)(In2 ⊗ A3)

Assess the amount of work.

Problem 3.6. How would you compute the matrix-vector product

y = (A1 ⊗ · · · ⊗ Ad)x where Ai ∈ IRn×n and x ∈ IRn3

?. Assess the amountof work.


Kronecker Products and Matrix Factorizations

Just take the Kronecker Product of the Factors...

LU: (PB ⊗ PC )(B ⊗ C ) = (LB ⊗ LC )(UB ⊗ UC )

Cholesky: B ⊗ C = (LB ⊗ LC )(LB ⊗ LC )T

Schur: (QB ⊗ QC )T (B ⊗ C )(QB ⊗ QC ) = TB ⊗ TC

QR: B ⊗ C = (QB ⊗ QC )(RB ⊗ RC )

SVD: B ⊗ C = (UB ⊗ UC )(ΣB ⊗ ΣC )(VB ⊗ VC )T

There are some annoying details if the matrices are rectangular, e.g.,

(ΣB ⊗ ΣC ) needs to be permuted to obtain a true diagonal form.


Problem 3.7. How would you minimize ‖ (A1 ⊗ · · · ⊗ Ad)x − b ‖2 giventhat each A1, . . . , Ad ∈ IRm×n has rank ri < n?

Perfect Shuffle Permutations

Transposition is a Permutation

This: [b11 b12 b13

b21 b22 b23

]=

a11 a12

a21 a22

a31 a32

T

Is the same as this:

b11

b21

b12

b22

b13

b23

=

1 0 0 0 0 00 0 0 1 0 00 1 0 0 0 00 0 0 0 1 00 0 1 0 0 00 0 0 0 0 1

a11

a21

a31

a12

a22

a32



Formal Definition

If n = pr then the mod-p perfect shuffle permutation Sp,r isdefined by

Sp,rx =

x(1:r :n)x(2:r :n)

...x(r :r :n)

where x ∈ IRn.

Implication for High Performance Computing: Matrix transpose

involves non-unit stride access to matrix elements.


Problem 3.8. (a) Think about the perfect shuffling of a 52-card deck andexplain why we use the term “perfect shuffle” when referring to Sp,r . (b)Show that Sr,p = ST

p,r . (c) Explain why Sp,r is aptly called the “mod-p sortpermutation.”



The Connection to Matrix Transpose...

If X ∈ IRp×r and Y = XT , then

vec(Y ) = Sp,r · vec(X )

The Connection to Kronecker Products...

If A1 ∈ IRm1×n1 and A2 ∈ IRm2×n2 then

A2 ⊗ A1 = STm1,m2

· (A1 ⊗ A2) · Sn1,n2


The Connection to Tensor Transpose

If A ∈ IRn1×n2×n3 , then

B =

A< [1 2 3] >

A< [1 3 2] >

A< [2 1 3] >

A< [2 3 1] >

A< [3 1 2] >

A< [3 2 1] >

=⇒ vec(B) =

In1n2n3

Sn2,n3⊗ In1

In3⊗ Sn1,n2

Sn1,n2n3

Sn1n2,n3

(Sn1,n2⊗ In3)Sn1n2,n3

·vec(A)

Implication for High Performance Computing: Tensor transpose

involves non-unit stride access to matrix elements.

Problem 3.9. Suppose A ∈ IRn1×···×nd , N = n1 · · · nd , and that p is apermutation of 1:d that involves swapping a single pair of indices, e.g.,[1 4 3 2 5]. Determine a permutation matrix P ∈ IRN×N so that ifB = A, then

vec(B) = P · vec(A).

Tensor Contractions

Definition

A contraction between two tensors produces a third tensor througha summation process.


Linear Algebra Operations in Tensor Contraction Terms

The Contraction of an Order-1 Tensor with an Order-1 Tensor

c =n∑

k=1

a(k)·b(k) (Dot Product)



The Contraction of an Order-2 Tensor and an Order-1 Tensor

c(i) =n∑

k=1

A(i , k)·b(k) (Matrix-Vector Product)



The Contraction of an Order-2 Tensor and an Order-2 Tensor

C (i , j) =n∑

k=1

A(i , k)·B(k, j) (Matrix-Matrix Product)


Tensor Contractions: Subscript Level

Example 1. (A Single-Index Contraction)

C(i , j , α3, α4, β3, β4, β5) =∑k

A(i , k, α3, α4) · B(k, j , β3, β4, β5)

⇑7th

Order

⇑4th

Order

⇑5th

Order

A = A(1:n1, 1:n2, 1:n3, 1:n4)

B = B(1:m1, 1:m2, 1:m3, 1:m4, 1:m5)

⇒ C = C(1:n1, 1:m2, 1:n3, 1:n4, 1:m3, 1:m4, 1:m5)

Order(C) = Order(A) + Order(B) - 2



Example 2. (A Double-Index Contraction)

C(i , j , p, q) =∑k

∑s

A(i , s, p, k) ∗ B(s, j , k, q)

⇑4th

Order

⇑4th

Order

⇑4th

Order

Order(C) = Order(A) + Order(B) - 2 - 2


Tensor Contractions: Summation Notation

Summation

If L = (L1, . . . , Lk) and R = (R1, . . . ,Rk) then

R∑i=L

≡R1∑

i1=L1

R2∑i2=L2

· · ·Rk∑

ik=Lk

Example: Frobenius Norm of A ∈ IRn1×···×nd

‖ A ‖F =

√√√√ n∑i=1

A(i)2



Example 3. (A 4-Index Contraction)

C(γ1, γ2, γ3, γ4)

=m∑

i=1

A(i1, i2, i3, i4)·W (i1, γ1)·X (i2, γ2)·Y (i3, γ3)·Z (i4, γ4)

⇑4th

Order

⇑2nd

Order

⇑2nd

Order

⇑2nd

Order

⇑2nd

Order

Order(C) = Order(A) + Order(W) + Order(X )+ Order(Y)+Order(Z) - 2 - 2 - 2 - 2


Towards a High-Performance Contraction Framework

In Reality...

Tensor computations are matrix computations. A contraction canbe regarded as a generalization of matrix-matrix multiplication.

The Challenge...

Expose the underlying matrix multiplications with good notation.

Two Reasons:

1. To facilitate tensor-level thinking.2. To facilitate level-3 BLA exploitation

Let us revisit the five examples rephrasing them to expose the underlyingmatrix products.



The Block Vec Unfolding

Blockvec(A) =

266666666666664

A( : , : , 1, 1, 1)

A( : , : , 2, 1, 1)

A( : , : , 1, 2, 1)

A( : , : , 2, 2, 1)

A( : , : , 1, 1, 2)

A( : , : , 2, 1, 2)

A( : , : , 1, 2, 2)

A( : , : , 2, 2, 2)

377777777777775A ∈ IRn1×n2×2×2×2

The Block-Vector Product

»F1

F2

–⊗

24 G1

G2

G3

35 =

26666664F1G1

F1G2

F1G3

F2G1

F2G2

F2G3

37777775⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions

Tensor Contractions: Matrix Level

Example 1.

C(i , j , α3, α4, β3, β4, β5) =∑k

A(i , k, α3, α4)B(k, j , β3, β4, β5)

As a collection of matrix-matrix products...

C( : , : , α3, α4, β3, β4, β5) = A( : , : , α3, α4)B( : , : , β3, β4, β5)



As a Block Vector Product...

C( : , : , 1, 1, 1, 1, 1)C( : , : , 1, 1, 2, 1, 1)C( : , : , 1, 1, 1, 2, 1)

...C( : , : , 2, 2, 1, 2, 2)C( : , : , 2, 2, 2, 2, 2)

=

A( : , : , 1, 1)A( : , : , 2, 1)A( : , : , 1, 2)A( : , : , 2, 2)

⊗

B( : , : , 1, 1, 1)B( : , : , 2, 1, 1)B( : , : , 1, 2, 1)B( : , : , 2, 2, 1)B( : , : , 1, 1, 2)B( : , : , 2, 1, 2)B( : , : , 1, 2, 2)B( : , : , 2, 2, 2)

32 matrix-matrix products


Problem 3.10. Suppose A ∈ IRn1×···×nd and B ∈ IRm1×···×me withn2 = m1. If

C(α1, β2, α3, . . . , αd , β3, . . . , βe) =

n2Xk=1

A(α1, k, α3, . . . , αd) · B(k, β2, . . . , βe)

then we refer to C as the canonical contraction of A and B. Write aMatlab function C = CanonContract(A,B) that computes this tensorgiven that A and B are tensors.



Block Matrix Unfolding for A = A(1:n1, 1:n2, 1:n3, 1:n4)

BlockMat(A) =

A( : , : , 1, 1) A( : , : , 1, 2) · · · A( : , : , 1, n4)

A( : , : , 2, 1) A( : , : , 2, 2) · · · A( : , : , 2, n4)...

.... . .

...

A( : , : , n3, 1) A( : , : , n3, 2) · · · A( : , : , n3, n4)

A(i , j , k, `) is the (i , j) entry of block (k, `)

Note that tenmat(A,[1 3],[2 4]) produces the BlockMatunfolding.



Example 2.

C(i , j , p, q) =∑k

∑s

A(i , s, p, k) · B(s, j , k, q)

As a collection of matrix products and sums...

C( : , : , p, q) =∑k

A( : , : , p, k) · B( : , : , k, q)

As a product of two block matrices...

BlockMat(C) = BlockMat(A) · BlockMat(B)


Problem 3.11. Write an efficient Matlab function C = Eg2(A,B) thattakes two order-4 tensors A and B and produces the order-4 tensor definedin Example 2.



Example 3.

C(m1,m2,m3,m4)

=m∑

i=1

A(i1, i2, i3, i4)·W (i1,m1)·X (i2,m2)·Y (i3,m3)·Z (i4,m4)

As a block-matrix product...

BlockMat(C) = (W ⊗ Y ) · BlockMat(A) · (X ⊗ Z )


Problem 3.12. Write an efficient Matlab function C = Eg5(A,W,X,Y,Z)

that takes an order-4 tensor A and four suitably dimensioned matrices W ,X , Y , and Z and produces the order-4 tensor C defined in Example 5.


High Performance Tensor Computations

There are Locality Issues...

If the B( : , : , p, r) and C( : , : , r , q) are stored continguously, then

A(i , j , p, q) =∑k

∑s

B(i , s, p, k) ∗ C(s, j , k, q)

is nice because the participating “slices” are laid out nicely:

A( : , : , p, q) =∑

r

B( : , : , p, r) ∗ C ( : , : , r , q)

Unfriendly:

A(i , j , p, q) =∑k

∑s

B(i , p, s, k) ∗ C(k, s, j , q)

LAPACK has it easy: BC , BCT , BTC , BTCT . Tensor BLAS?


High Performance Tensor Computations

There are Data Structure/Layout Issues...

SupposeA = A(1:n1, . . . , 1:n7).

Can regard A as a 4-th order tensor with 3rd order entries. Storethe 3rd order subtensors contiguously.

Can regard A as an aggregation of “little” 7th order tensors. (Likea block matrix.)

The what-fits-in-cache inequality:

m7 ≤ Ncache


Summary of Lecture 3.

Key Words

A transposition of an order-d tensor is defined by apermutation p of 1:d . Element i is mapped to element p(i).

The Kronecker product of two matrices is a highly structuredblock matrix.

Perfect shuffle permutations are a handy way to describetensor transposition.

A contraction between two tensors is a summation processthat produces a new tensor.


Lecture 3. Transpositions, Kronecker Products, and ... · Lecture 1. Introduction to Tensor...

Documents

Transcript of Lecture 3. Transpositions, Kronecker Products, and ... · Lecture 1. Introduction to Tensor...