Dynamic Programming( Matrix Multiplication)cs.indstate.edu/~arash/algo2lec6.pdfMatrix...

Dynamic Programming( MatrixMultiplication)

Arash Rafiey

21 January 2016

Arash Rafiey Dynamic Programming( Matrix Multiplication)

Matrix Multiplications

Given: a “chain” of matrices (A1,A2, . . . An), with Ai havingdimension di−1 × di .Goal: compute the product A1 · A2 · · ·An as fast as possible

Clearly, time to multiply two matrices depends on dimensionsDoes the order of multiplication (= parenthesization) matter?Example: n = 4. Possible orders:

(A1(A2(A3A4)))

(A1((A2A3)A4))

((A1A2)(A3A4))

((A1(A2A3))A4)

(((A1A2)A3)A4)


Matrix Multiplications

Given: a “chain” of matrices (A1,A2, . . . An), with Ai havingdimension di−1 × di .Goal: compute the product A1 · A2 · · ·An as fast as possibleClearly, time to multiply two matrices depends on dimensionsDoes the order of multiplication (= parenthesization) matter?Example: n = 4. Possible orders:

(A1(A2(A3A4)))

(A1((A2A3)A4))

((A1A2)(A3A4))

((A1(A2A3))A4)

(((A1A2)A3)A4)


Suppose A1 is 10× 100, A2 is 100× 5, A3 is 5× 50, and A4 is50× 10Assume that multiplication of a (p × q)-matrix and a(q × r)-matrix takes pqr steps (a straightforward algorithm)

Order 2: (A1((A2A3)A4))

100 · 5 · 50 + 100 · 50 · 10 + 10 · 100 · 10 = 85, 000

Order 5: (((A1A2)A3)A4)

10 · 100 · 5 + 10 · 5 · 50 + 10 · 50 · 10 = 12, 500

Seems it might be a good idea to find a “good” order


How many orders are there? Can we just check all of them?( we look only at fully parenthesized matrix products)

Let P(n) be the number of orders of a sequence of n matricesClear, P(1) = 1 (only one matrix)If n ≥ 2, a matrix product is the product of two matrixsub-products. Split may occur between k-th and (k + 1)-stposition, for any k = 1, 2, . . . , n − 1 (“top-level multiplication”)Thus

P(n) =

1 if n = 1∑n−1

k=1 P(k) · P(n − k) if n ≥ 2

P(n) = 1n

(2n−2n−1

)Unfortunately, P(n) = Ω(4n/n3/2), and thus (easier to see)P(n) = Ω(2n)Thus “brute-force approach” (check all parenthesization) is nogood



Let P(n) be the number of orders of a sequence of n matricesClear, P(1) = 1 (only one matrix)

If n ≥ 2, a matrix product is the product of two matrixsub-products. Split may occur between k-th and (k + 1)-stposition, for any k = 1, 2, . . . , n − 1 (“top-level multiplication”)Thus

P(n) =

1 if n = 1∑n−1

k=1 P(k) · P(n − k) if n ≥ 2

P(n) = 1n

(2n−2n−1




Let P(n) be the number of orders of a sequence of n matricesClear, P(1) = 1 (only one matrix)If n ≥ 2, a matrix product is the product of two matrixsub-products. Split may occur between k-th and (k + 1)-stposition, for any k = 1, 2, . . . , n − 1 (“top-level multiplication”)

Thus

P(n) =

1 if n = 1∑n−1

k=1 P(k) · P(n − k) if n ≥ 2

P(n) = 1n

(2n−2n−1





P(n) =

1 if n = 1∑n−1

k=1 P(k) · P(n − k) if n ≥ 2

P(n) = 1n

(2n−2n−1

)

Unfortunately, P(n) = Ω(4n/n3/2), and thus (easier to see)P(n) = Ω(2n)Thus “brute-force approach” (check all parenthesization) is nogood




P(n) =

1 if n = 1∑n−1

k=1 P(k) · P(n − k) if n ≥ 2

P(n) = 1n

(2n−2n−1



We will use the Dynamic programming approach to optimallysolve this problem.The four basic steps when designing Dynamic programmingalgorithm:

1 Characterize the structure of an optimal solution2 Recursively define the value of an optimal solution3 Compute the value of an optimal solution in a bottom-up

fashion4 Construct an optimal solution from computed information


1. Characterizing structure

Let Ai ,j = Ai · · ·Aj for i ≤ j . If i < j , then any parenthesization of

Ai ,j must split product at some k, i ≤ k < j , i.e., compute Ai ,k ,Ak+1,j , and then Ai ,k · Ak+1,j .

Hence, for some k, the cost of computing Ai ,j is

the cost of computing Ai ,k plus

the cost of computing Ak+1,j plus

the cost of multiplying Ai ,k and Ak+1,j .


Optimal substructure:

Suppose that optimal parenthesization of Ai ,j splits theproduct between Ak and Ak+1.Then, parenthesizations of Ai ,k and Ak+1,j within this optimalparenthesization must be also optimal(otherwise, substitute the opt. parenthesization of Ai ,k (resp.Ak+1,j) to current parenthesization of Ai ,j and obtain a bettersolution — contradiction)

Use optimal substructure to construct an optimal solution:

1 split into two subproblems (choosing an optimal split),2 find optimal solutions to subproblem,3 combine optimal subproblem solutions.


A recursive solution

Let m[i , j ] denote minimum number of multiplications needed tocompute Ai ,j = Ai · Ai+1 · · ·Aj (full problem: m[1, n]).Recursive definition of m[i , j ]:

if i = j , then m[i , j ] = m[i , i ] = 0 (no multiplication needed)

if i < j , assume optimal split at k, i ≤ k < j . Since eachmatrix Ai is di−1 × di , Ai ,k is di−1 × dk and Ak+1,j is dk × dj ,

m[i , j ] = m[i , k] + m[k + 1, j ] + di−1 · dk · dj

We do not know optimal value of k. There are j − ipossibilities, k = i , i + 1, . . . , j − 1, hence

m[i , j ] =

0 if i = jmini≤k<jm[i , k] + m[k + 1, j ] if i < j+di−1 · dk · dj

We also keep track of optimal splits:

s[i , j ] = k ⇔ m[i , j ] = m[i , k] + m[k + 1, j ] + di−1 · dk · dj

(s[i , j ] is a value of k at which we split the product Ai ,j to obtainan optimal parenthesization)


A recursive solution

Let m[i , j ] denote minimum number of multiplications needed tocompute Ai ,j = Ai · Ai+1 · · ·Aj (full problem: m[1, n]).Recursive definition of m[i , j ]:

if i = j , then m[i , j ] = m[i , i ] = 0 (no multiplication needed)if i < j , assume optimal split at k, i ≤ k < j . Since eachmatrix Ai is di−1 × di , Ai ,k is di−1 × dk and Ak+1,j is dk × dj ,

m[i , j ] = m[i , k] + m[k + 1, j ] + di−1 · dk · dj

We do not know optimal value of k. There are j − ipossibilities, k = i , i + 1, . . . , j − 1, hence

m[i , j ] =

0 if i = jmini≤k<jm[i , k] + m[k + 1, j ] if i < j+di−1 · dk · dj

We also keep track of optimal splits:

s[i , j ] = k ⇔ m[i , j ] = m[i , k] + m[k + 1, j ] + di−1 · dk · dj

(s[i , j ] is a value of k at which we split the product Ai ,j to obtainan optimal parenthesization)


Computing the optimal costs

Want to compute m[1, n], minimum cost for multiplyingA1 · A2 · · ·An.Recursively, it would take Ω(2n) steps: the same subproblems arecomputed over and over again.However, if we compute in a bottom-up fashion, we can reducerunning time to polynomial in n.

The recursive equation shows that cost m[i , j ] (product of j − i + 1matrices) depends only on smaller subproblems:for k = 1, . . . , j − 1,

Ai ,k is a product of k − i + 1 < j − i + 1 matrices,Ak+1,j is a product of j − k < j − i + 1 matrices.

Algorithm should fill table m in order of increasing lengths ofchains.


Matrix-Chain-Order(p)1. n := length[p]− 12. for i := 1 to n3. m[i , i ] := 04. for ` := 2 to n4. for i := 1 to n − ` + 15. j := i + `− 1 m[i , j ] := ∞6. for k := i to j − 17. q := m[i , k] + m[k + 1, j ] + di−1 · dk · dj

8. if q < m[i , j ]9. m[i , j ] := q s[i , j ] := k

10. return m and s


Palindrome Subsequence Problem

Definition. Z = z1z2 . . . zk is a subsequence of S = s1s2 . . . sn ifthere exists an increasing sequence of indexes:1 ≤ i1 < i2 < · · · < ik ≤ n such that zj = sijExample.

S= G G C A C T G T A C↓ ↓ ↓ ↓

Z= G C C A

Z = GCCA is a subsequence of S = GGCACTGTACDefinition : We say a sequence is palindrome if reading it fromleft to right is the same as reading it from right to left.

A C G T G C A


Solution

Problem : We are given a sequence S of length n.

Goal: Design an algorithm that finds a maximum lengthsubsequence of S which is palindrome.

S= G G C A C T G T A C↓ ↓ ↓ ↓ ↓

Z= G C A C G

The optimal solution is :

S= G G C A C T G T A C↓ ↓ ↓ ↓ ↓ ↓ ↓

Z= C A T G T A C


Suppose S = x1, x2, . . . , xn is the input sequence of length n.

Lemma

Let Z = z1, z2, . . . , zk be a longest palindrome subsequence ofxi , xi+1, . . . , xj .

1 If z1 = xi and zk = xj then z2, z3, . . . , zk−1 is a longestpalindrome subsequence of xi+1, xi+2, . . . , xj−1.

2 If z1 6= xi then z1, z2, . . . , zk is a longest palindromesubsequence of xi+1, xi+2, . . . , xj .

3 If zk 6= xj then z1, z2, . . . , zk is a longest palindromesubsequence of xi , xi+2, . . . , xj−1.

Proof by contradiction!Case 1 z1 = xi and zk = xj . Suppose Z is not a longestpalindrome subsequence of xi , xi+1, . . . , xj . Let Z ′ be a longestpalindrome subsequence of xi+1, xi+2, . . . , xj−1. Note that|Z ′| > k − 2. Now z1Z

′zk is a palindrome subsequence ofxi , xi+1, . . . , xj and longer than Z , a contradiction.


Case 2. z1 6= xi . We have z1 = x`, ` > i and hence Z is a longestpalindrome subsequence of x`, x`+1, . . . , xj . Since x`, x`+1, . . . , xj isa subsequence of xi+1, . . . , xj , Z is a longest palindromesubsequence of xi+1, xi+2, . . . , xj .

Case 3. zk 6= xj . Similar to (2).


Now we design an algorithm based on the Lemma. LetS = x1, x2, . . . , xn.

Let M[i , j ] be the length of a longest palindrome subsequence fromxi to xj . If i = j then M[i , i ] = 1 (Since xi by itself is apalindrome).

Our goal is to compute M[1, n].

Recursive formula :

If xi = xj then M[i , j ] = M[i + 1, j − 1] + 2 otherwise :

M[i , j ] = maxM[i + 1, j ],M[i , j − 1]


Longest-palindrome-subsequence(S)

1. int n = length(S);

// Initializing M.2. for (i = 1 to n) M[i , i ] = 1; // by definition

3. for (j = 1 to n − 1 )

4. for (i = 1 to n − j)

5. if (S [i ] = S [i + j ])

6. M[i , i + j ] = M[i + 1, i + j − 1] + 27. else

8. M[i , i + j ] = maxM[i + 1, i + j ],M[i , i + j − 1]


Dynamic Programming( Matrix Multiplication)cs.indstate.edu/~arash/algo2lec6.pdfMatrix...

Documents

Transcript of Dynamic Programming( Matrix Multiplication)cs.indstate.edu/~arash/algo2lec6.pdfMatrix...