Matrix Multiplication Based Computations of the ... · The most efﬁcient routine in linear...

IntroductionMatrix multiplication based linear algebraComputing the characteristic polynomial

Conclusion and perspectives

Matrix Multiplication Based Computations ofthe Characteristic Polynomial

Clément PERNET,joint work with Arne Storjohann

Symbolic Computation GroupUniversity of Waterloo

Joint Lab Meeting ORCCA-SCG,February 9, 2007

Clément Pernet Matrix multiplication based Characteristic Polynomial



Introduction

Dense Linear Algebra over a Field:one of the usual models for complexity in linear algebraapplied to

R: floating point linear algebraGF (q), Zp and Z ( using CRT)

Applications in exact computation:Cryptography : number field sievesRepresentation theory : null space basis

Topology : Smith normal formsGraph theory : characteristic polynomial

...




Approach

ProblemCompute the characteristic polynomial of a dense matrix over afield

Deterministic or Las Vegas randomized algorithmns

Asymptotic time complexity...... and practical algorithms

⇒Balance between asymptotic complexity and practicalefficiency considerations

space complexity




Approach


Deterministic or Las Vegas randomized algorithmnsAsymptotic time complexity...... and practical algorithms

⇒Balance between asymptotic complexity and practicalefficiency considerationsspace complexity




Approach


Deterministic or Las Vegas randomized algorithmnsAsymptotic time complexity...... and practical algorithms⇒Balance between asymptotic complexity and practical

efficiency considerations

space complexity




Approach


Deterministic or Las Vegas randomized algorithmnsAsymptotic time complexity...... and practical algorithms⇒Balance between asymptotic complexity and practical

efficiency considerationsspace complexity




Outline

1 Matrix multiplication based linear algebraMatrix multiplication: a building blockReductions to matrix multiplication

2 Computing the characteristic polynomialState of the artA new algorithmAlgorithm into practice




Matrix multiplication: a building blockReductions to matrix multiplication

Outline







Asymptotic complexity

Matrix multiplication:

Folklore: 2n3 − n2

Strassen 1969: 7n2.807 + o(n2.807)

Winograd 1971: 6n2.807 + o(n2.807)

...Coppersmith Winograd 1990: O

(n2.376)

⇒O (nω), where ω denotes an admissible exponent





Efficiency in practice

The most efficient routine in linear algebra.Several reasons:

dedicated processor instruction fused-mac: z ← xy + z

simple structure of the dot-product (pipelining is easy)

sub-cubic algorithmused to be considered as not practicablebeware of unstability with floating point numbersbut improves efficiency over finite fields







dedicated processor instruction fused-mac: z ← xy + zsimple structure of the dot-product (pipelining is easy)








dedicated processor instruction fused-mac: z ← xy + zsimple structure of the dot-product (pipelining is easy)enables better memory management








dedicated processor instruction fused-mac: z ← xy + zsimple structure of the dot-product (pipelining is easy)enables better memory managementsub-cubic algorithm

used to be considered as not practicablebeware of unstability with floating point numbersbut improves efficiency over finite fields





Memory management considerations

CPU-Memory communication: bandwidth gap⇒Hierarchy of several cache memory levels

Imposes a structure for algorithms: operationsmust be blocked to increase data locality and fitin the cache

CPU

L1

RAM

L2

L3

Reuse of the dataWork� Data to amortize memory transfer⇒reach the peak performance of the CPU

Matrix multiplication: n3 � n2

⇒well suited for block design





Memory management considerations

CPU-Memory communication: bandwidth gap⇒Hierarchy of several cache memory levels

Imposes a structure for algorithms: operationsmust be blocked to increase data locality and fitin the cache ...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

Reuse of the dataWork� Data to amortize memory transfer⇒reach the peak performance of the CPU

Matrix multiplication: n3 � n2

⇒well suited for block design





Practical implementation over finite fields

Matrix multiplication

0

1000

2000

3000

4000

5000

6000

0 1000 2000 3000 4000 5000 6000 7000

Mfo

ps

n

Multiplication classique dans Z/65521Z sur un P4, 3.4 GHz

Standard Bloc-33







0

1000

2000

3000

4000

5000

6000

0 1000 2000 3000 4000 5000 6000 7000

Mfo

ps

n


fgemm Standard Bloc-33







0

1000

2000

3000

4000

5000

6000

0 1000 2000 3000 4000 5000 6000 7000

Mfo

ps

n


dgemm fgemm Standard Bloc-33







600

800

1000

1200

1400

1600

1800

2000

0 500 1000 1500 2000 2500 3000

Mfo

ps

n

Multiplication rapide dans Z/65521Z sur un PIII-1.6Ghz (1Mo de cache L2)

ClassiqueWinograd 1 niveau

Winograd 2 niveauxWinograd 3 niveauxWinograd 4 niveauxWinograd 5 niveaux





Outline







Other linear algebra problems

Asymptotic complexity:

Used to be in O(n3)

Room for improvement: O (nω) for everyone ?

Practical efficiency: reuse the efficient matrix multiplicationkernel





Reductions to matrix multiplication

Matrix Inversion [Strassen 69]

[A BC D

]−1

=

[A−1

I

] [I −B

I

] [I

(D − CA−1B)−1

] [I

CA−1 I

]

1: Compute E = A−1 (Recursive call)2: Compute F = D − CEB (MM)3: Compute G = F−1 (Recursive call)4: Compute H = −EB (MM)5: Compute J = HG (MM)6: Compute K = CE (MM)7: Compute L = E + JK (MM)8: Compute M = GK (MM)

9: Return[

E JM G

]






Matrix Inversion [Strassen 69]

[A BC D

]−1

=

[A−1

I

] [I −B

I

] [I

(D − CA−1B)−1

] [I

CA−1 I

]1: Compute E = A−1 (Recursive call)2: Compute F = D − CEB (MM)3: Compute G = F−1 (Recursive call)4: Compute H = −EB (MM)5: Compute J = HG (MM)6: Compute K = CE (MM)7: Compute L = E + JK (MM)8: Compute M = GK (MM)

9: Return[

E JM G

]Clément Pernet Matrix multiplication based Characteristic Polynomial





TRSM: Multiple triangular system solving[A B

C

]−1 [DE

]=

[A−1

I

] [I −B

I

] [I

C−1

] [DE

]

1: Compute F = C−1E (Recursive call)2: Compute G = D − BF (MM)3: Compute H = A−1G (Recursive call)

4: Return[HF

]






LU decomposition[A BC D

]=

[LA

CU−1A LE

] [UA L−1

A BUE

]where E = D − CA−1B

1: Compute A = LAUA (Recursive call)2: Compute F = CU−1

A (TRSM)3: Compute G = L−1

A B (TRSM)4: Compute E = D − FG (MM)5: Compute E = LEUE (Recursive call)

6: Return([

LAF LE

],

[UA G

UE

])






Divide and conquer approach:⇒ involves computations with dimensions n

2i for i = 1 . . . log2 n

⇒ overall time complexity by geometric progression

log2 n∑i=1

2i( n

2i

)ω

= O (nω)

These reductions reduce to in O (nω) the following problemsdet, rank, rank profile,echelon form, inverse, system solving.

What about the characteristic polynomial ?




State of the artA new algorithmAlgorithm into practice

Outline







Pre-Strassen age

Leverrier 1840: trace of powers of A, and Newton’s formulaimproved/rediscovered by Souriau, Faddeev,Frame and CsankyO

(n4), based on Matrix multiplication

Suited for parallel computation model

Danilevskii 1937: elementary row/column operations⇒O

(n3)

Hessenberg 1942: transformation to quasi-upper triangularand determinant expansion formula.⇒O

(n3)

But no trivial translation into a block algorithm with O (nω)complexity.





Pre-Strassen age

Leverrier 1840: trace of powers of A, and Newton’s formulaimproved/rediscovered by Souriau, Faddeev,Frame and CsankyO

(n4), based on Matrix multiplication

Suited for parallel computation modelDanilevskii 1937: elementary row/column operations

⇒O(n3)

Hessenberg 1942: transformation to quasi-upper triangularand determinant expansion formula.⇒O

(n3)

But no trivial translation into a block algorithm with O (nω)complexity.





Post-Strassen age

Preparata & Sarwate 1978: Update Csanky with fast matrixmultiplication⇒O

(nω+1)

Keller-Gehrig 1985, alg.1: computes (A2i)i=1... log2 n to form a

Krylov basis.

O (nω log n)the best complexity up to now

Keller-Gehrig 1985, alg.2: inspired by Danilevskii, blockoperations

O (nω)Only valid with generic matrices





Outline







Statement

Theorem

If A is a n × n matrix over a field having more than 2n2

elements, the characteristic polynomial of A can be computedin O (nω) field operations by a Las Vegas randomized algorithm.





Definition (degree d Krylov matrix of one vector v )

K =[v Av . . . Ad−1v

]Property

A× K = K ×

0 ∗1 ∗

. . . ∗1 ∗

⇒if d = n, K−1AK = CPAcar

⇒[Keller-Gehrig, alg. 2] computes K in O (nω)






K =[v Av . . . Ad−1v

]Property

A× K = K ×

0 ∗1 ∗

. . . ∗1 ∗

⇒if d = n, K−1AK = CPA

car

⇒[Keller-Gehrig, alg. 2] computes K in O (nω)






K =[v Av . . . Ad−1v

]Property

A× K = K ×

0 ∗1 ∗

. . . ∗1 ∗

⇒if d = n, K−1AK = CPA

car⇒[Keller-Gehrig, alg. 2] computes K in O (nω)





Definition (degree k Krylov matrix of several vectors vi )

K =[

v1 . . . Ak−1v1 v2 . . . Ak−1v2 . . . vl . . . Ak−1vl]

Property

1

0

1

1

0

1

1

0

1

k k k

A× K = K×





FactIf (d1, . . . dl) is lexicographically maximal such that

K =[

v1 . . . Ad1−1v1 . . . vl . . . Adl−1vl]

is non-singular, then

1

0

1

0

1

1

1

0

1

d2 dld1

K−1AK =





Principle

k -shifted form:

1

0

1

1

0

1

1

0

1

k k ≤ k

try to inflate each slice by one ...... to obtain the k + 1-shifted form





Principle

k -shifted form:

1

0

1

1

0

1

1

0

1

k k ≤ k

try to inflate each slice by one ...

... to obtain the k + 1-shifted form





Principle

k + 1-shifted form:

1

0

1

0

1

0

1

1

1

≤ k + 1k + 1 d3

try to inflate each slice by one ...... to obtain the k + 1-shifted form





Principle

Compute iteratively from 1-shifted form to d1-shifted form

each diagonal block appears in the increasing degreeuntil the shifted Hessenberg form is obtained:

1

0

1

0

1

1

1

0

1

d2 dld1

How to transform from k to k + 1-shifted form ?





Principle

Compute iteratively from 1-shifted form to d1-shifted formeach diagonal block appears in the increasing degree

until the shifted Hessenberg form is obtained:

1

0

1

0

1

1

1

0

1

d2 dld1






Principle

Compute iteratively from 1-shifted form to d1-shifted formeach diagonal block appears in the increasing degreeuntil the shifted Hessenberg form is obtained:

1

0

1

0

1

1

1

0

1

d2 dld1






Krylov normal extension

for any k -shifted form

1

0

1

1

0

1

1

0

1

k k ≤ k

Ak = c1 c2 c3

compute the n × (n + k) matrix

1

11

11

1

k kk

K = c1 c2 c3

and form K by picking its first linearly independent columns.





Krylov normal extension

LemmaIf #F > 2n2, with high probability, the matrix K will have the form

1

11

1

1

1

k k ≤ k

K = c1 c2

and Ak+1 = K−1Ak K will be in k + 1 shifted form





The algorithm

Form K : just copy the columns of Ak

Compute K : rank profile of KApply the similarity transformation K−1AkK

How to use matrix multiplication knowing the structure ?





The algorithm


Compute K : rank profile of K

Apply the similarity transformation K−1AkK






The algorithm


Compute K : rank profile of KApply the similarity transformation K−1AkK






Permutations: compressing the dense columns

0

1

1

1

1

0

1

0

1

1

0

1

×PAk = = Q×c2c2c3c1 c3

c1

1

1

0

1

1

1

11

1

×P′K = = Q′× c2c1c1 c2





Reduction to Matrix multiplication

Rank profile: derived from LQUP⇒O

(k

(nk

)ω)

Similarity transformation: parenthesing

⇒O(k

(nk

)ω)Overall complexity: summing for each iteration:

n∑k=1

k(n

k

)ω

= nωn∑

k=1

(1k

)ω−1

= O (nω)







(k

(nk

)ω)Similarity transformation: parenthesing

K−1AK = Q′T[

I ∗0 ∗

]P ′T Q

[I ∗0 ∗

]PQ′

[I ∗0 ∗

]P ′

⇒O(k

(nk


n∑k=1

k(n

k

)ω

= nωn∑

k=1

(1k

)ω−1

= O (nω)







(k

(nk


K−1AK = Q′T([

I ∗0 ∗

](P ′T Q

([I ∗0 ∗

](PQ′

[I ∗0 ∗

]))))P ′

⇒O(k

(nk


n∑k=1

k(n

k

)ω

= nωn∑

k=1

(1k

)ω−1

= O (nω)







(k

(nk


K−1AK = Q′T([

I ∗0 ∗

](P ′T Q

([I ∗0 ∗

](PQ′

[I ∗0 ∗

]))))P ′

⇒O(k

(nk

)ω)

Overall complexity: summing for each iteration:

n∑k=1

k(n

k

)ω

= nωn∑

k=1

(1k

)ω−1

= O (nω)







(k

(nk


K−1AK = Q′T([

I ∗0 ∗

](P ′T Q

([I ∗0 ∗

](PQ′

[I ∗0 ∗

]))))P ′

⇒O(k

(nk


n∑k=1

k(n

k

)ω

= nωn∑

k=1

(1k

)ω−1

= O (nω)





Outline







Heuristic improvement

The randomization:

the iterate vectors at the first iteration must be random vectors.or equivalently

the matrix has to be preconditioned: M−1AM for a randommatrix M.

⇒as expensive as the rest of the algorithm





A block Krylov preconditoner

1: Pick n/c random vectors U =[u1 . . . un/c

].

2: M =[U AU . . . Ac−1

]

3: if M is non singular then4: M−1AM = Hc is in c-shifted form.5: else6: complete M into a non singular matrix M by adding some

columns at the end

7: then M−1

AM =

[Hc ∗

R

]8: end if







].

2: M =[U AU . . . Ac−1

]3: if M is non singular then4: M−1AM = Hc is in c-shifted form.

5: else6: complete M into a non singular matrix M by adding some

columns at the end

7: then M−1

AM =

[Hc ∗

R

]8: end if







].

2: M =[U AU . . . Ac−1

]3: if M is non singular then4: M−1AM = Hc is in c-shifted form.5: else6: complete M into a non singular matrix M by adding some

columns at the end

7: then M−1

AM =

[Hc ∗

R

]8: end if





Efficiency balancing parameter

c small: full square matrix multiplications, but more opsc large: tends to matrix-vector products, but less ops

⇒parameter c balances efficiency

120

130

140

150

160

170

180

190

200

210

220

230

0 50 100 150 200

Tim

e (s

)

Preconditionning parameter c

Finding the optimal preconditionning paramater, n=5000

1 block of order 50005 blocks of order 100010 blocks of order 500





Experiments

n LU-Krylov New algorithm200 0.024 0.032300 0.06s 0.088s500 0.248s 0.316s750 1.084s 1.288s

1000 2.42s 2.296s5000 267.6s 153.9s

10 000 1827s 991s20 000 14 652s 7097s30 000 48 887s 24 928s

Computation time for 1 Frobenius block matrices, Itanium2-64 1.3Ghz, 192Gb





Experiments

0.01

0.1

1

10

100

1000

10000

100000

100 1000 10000 100000

Tim

e (s

)

Matrix order

Comparison for 1 frobenius block matrices over Z/(547909)

New algorithmLU−Krylov

Timing comparison between the new algorithm and LU-Krylov, logarithmicscales, Itanium2-64 1.3Ghz, 192Gb





Comparison to Magma

n magma-2.11 LU-Krylov New algorithm100 0.010s 0.005s 0.006s300 0.830s 0.294s 0.105s500 3.810s 1.316s 0.387s800 15.64s 4.663s 1.387s1000 29.96s 10.21s 2.755s1500 102.1s 33.36s 7.696s2000 238.0s 79.13s 17.91s3000 802.0s 258.4s 61.09s5000 3793s 1177s 273.4s7500 MT 4209s 991.4s

10 000 MT 8847s 2080s

Computation time for 1 Frobenius block matrices, Athlon 2200, 1.8Ghz, 2Gb

MT: Memory thrashing





Results:

Las Vegas reduction to matrix multiplication,The Frobenius normal form is easily derivable in O (nω)......but no transformation matrixAdaptive combination with block Krylov in practice.

Still to be done:

Condition on the size of the field is a limitation. Eberly’salgorithm ?Ideally: derandomization ? (deterministic)


Matrix Multiplication Based Computations of the ... · The most efﬁcient routine in linear...

Documents

Transcript of Matrix Multiplication Based Computations of the ... · The most efﬁcient routine in linear...