Dynamic Programming( MatrixMultiplication)

Arash Rafiey

21 January 2016

Arash Rafiey Dynamic Programming( Matrix Multiplication)

Matrix Multiplications

Given: a “chain” of matrices (A1,A2, . . . An), with Ai havingdimension di−1 × di .Goal: compute the product A1 · A2 · · ·An as fast as possible

Clearly, time to multiply two matrices depends on dimensionsDoes the order of multiplication (= parenthesization) matter?Example: n = 4. Possible orders:

(A1(A2(A3A4)))

(A1((A2A3)A4))

((A1A2)(A3A4))

((A1(A2A3))A4)

(((A1A2)A3)A4)

Arash Rafiey Dynamic Programming( Matrix Multiplication)

Matrix Multiplications

Given: a “chain” of matrices (A1,A2, . . . An), with Ai havingdimension di−1 × di .Goal: compute the product A1 · A2 · · ·An as fast as possibleClearly, time to multiply two matrices depends on dimensionsDoes the order of multiplication (= parenthesization) matter?Example: n = 4. Possible orders:

(A1(A2(A3A4)))

(A1((A2A3)A4))

((A1A2)(A3A4))

((A1(A2A3))A4)

(((A1A2)A3)A4)

Arash Rafiey Dynamic Programming( Matrix Multiplication)

Suppose A1 is 10× 100, A2 is 100× 5, A3 is 5× 50, and A4 is50× 10Assume that multiplication of a (p × q)-matrix and a(q × r)-matrix takes pqr steps (a straightforward algorithm)

Order 2: (A1((A2A3)A4))

100 · 5 · 50 + 100 · 50 · 10 + 10 · 100 · 10 = 85, 000

Order 5: (((A1A2)A3)A4)

10 · 100 · 5 + 10 · 5 · 50 + 10 · 50 · 10 = 12, 500

Seems it might be a good idea to find a “good” order

Arash Rafiey Dynamic Programming( Matrix Multiplication)

Suppose A1 is 10× 100, A2 is 100× 5, A3 is 5× 50, and A4 is50× 10Assume that multiplication of a (p × q)-matrix and a(q × r)-matrix takes pqr steps (a straightforward algorithm)

Order 2: (A1((A2A3)A4))

100 · 5 · 50 + 100 · 50 · 10 + 10 · 100 · 10 = 85, 000

Order 5: (((A1A2)A3)A4)

10 · 100 · 5 + 10 · 5 · 50 + 10 · 50 · 10 = 12, 500

Seems it might be a good idea to find a “good” order

Arash Rafiey Dynamic Programming( Matrix Multiplication)

How many orders are there? Can we just check all of them?( we look only at fully parenthesized matrix products)

Let P(n) be the number of orders of a sequence of n matricesClear, P(1) = 1 (only one matrix)If n ≥ 2, a matrix product is the product of two matrixsub-products. Split may occur between k-th and (k + 1)-stposition, for any k = 1, 2, . . . , n − 1 (“top-level multiplication”)Thus

P(n) =

1 if n = 1∑n−1

k=1 P(k) · P(n − k) if n ≥ 2

P(n) = 1n

(2n−2n−1

)Unfortunately, P(n) = Ω(4n/n3/2), and thus (easier to see)P(n) = Ω(2n)Thus “brute-force approach” (check all parenthesization) is nogood

Arash Rafiey Dynamic Programming( Matrix Multiplication)

How many orders are there? Can we just check all of them?( we look only at fully parenthesized matrix products)

Let P(n) be the number of orders of a sequence of n matricesClear, P(1) = 1 (only one matrix)

If n ≥ 2, a matrix product is the product of two matrixsub-products. Split may occur between k-th and (k + 1)-stposition, for any k = 1, 2, . . . , n − 1 (“top-level multiplication”)Thus

P(n) =

1 if n = 1∑n−1

k=1 P(k) · P(n − k) if n ≥ 2

P(n) = 1n

(2n−2n−1

)Unfortunately, P(n) = Ω(4n/n3/2), and thus (easier to see)P(n) = Ω(2n)Thus “brute-force approach” (check all parenthesization) is nogood

Arash Rafiey Dynamic Programming( Matrix Multiplication)

How many orders are there? Can we just check all of them?( we look only at fully parenthesized matrix products)

Let P(n) be the number of orders of a sequence of n matricesClear, P(1) = 1 (only one matrix)If n ≥ 2, a matrix product is the product of two matrixsub-products. Split may occur between k-th and (k + 1)-stposition, for any k = 1, 2, . . . , n − 1 (“top-level multiplication”)

Thus

P(n) =

1 if n = 1∑n−1

k=1 P(k) · P(n − k) if n ≥ 2

P(n) = 1n

(2n−2n−1

)Unfortunately, P(n) = Ω(4n/n3/2), and thus (easier to see)P(n) = Ω(2n)Thus “brute-force approach” (check all parenthesization) is nogood

Arash Rafiey Dynamic Programming( Matrix Multiplication)

How many orders are there? Can we just check all of them?( we look only at fully parenthesized matrix products)

Let P(n) be the number of orders of a sequence of n matricesClear, P(1) = 1 (only one matrix)If n ≥ 2, a matrix product is the product of two matrixsub-products. Split may occur between k-th and (k + 1)-stposition, for any k = 1, 2, . . . , n − 1 (“top-level multiplication”)Thus

P(n) =

1 if n = 1∑n−1

k=1 P(k) · P(n − k) if n ≥ 2

P(n) = 1n

(2n−2n−1

)

Unfortunately, P(n) = Ω(4n/n3/2), and thus (easier to see)P(n) = Ω(2n)Thus “brute-force approach” (check all parenthesization) is nogood

Arash Rafiey Dynamic Programming( Matrix Multiplication)

How many orders are there? Can we just check all of them?( we look only at fully parenthesized matrix products)

Let P(n) be the number of orders of a sequence of n matricesClear, P(1) = 1 (only one matrix)If n ≥ 2, a matrix product is the product of two matrixsub-products. Split may occur between k-th and (k + 1)-stposition, for any k = 1, 2, . . . , n − 1 (“top-level multiplication”)Thus

P(n) =

1 if n = 1∑n−1

k=1 P(k) · P(n − k) if n ≥ 2

P(n) = 1n

(2n−2n−1

)Unfortunately, P(n) = Ω(4n/n3/2), and thus (easier to see)P(n) = Ω(2n)Thus “brute-force approach” (check all parenthesization) is nogood

Arash Rafiey Dynamic Programming( Matrix Multiplication)

We will use the Dynamic programming approach to optimallysolve this problem.The four basic steps when designing Dynamic programmingalgorithm:

1 Characterize the structure of an optimal solution2 Recursively define the value of an optimal solution3 Compute the value of an optimal solution in a bottom-up

fashion4 Construct an optimal solution from computed information

Arash Rafiey Dynamic Programming( Matrix Multiplication)

1. Characterizing structure

Let Ai ,j = Ai · · ·Aj for i ≤ j . If i < j , then any parenthesization of

Ai ,j must split product at some k, i ≤ k < j , i.e., compute Ai ,k ,Ak+1,j , and then Ai ,k · Ak+1,j .

Hence, for some k, the cost of computing Ai ,j is

the cost of computing Ai ,k plus

the cost of computing Ak+1,j plus

the cost of multiplying Ai ,k and Ak+1,j .

Arash Rafiey Dynamic Programming( Matrix Multiplication)

1. Characterizing structure

Let Ai ,j = Ai · · ·Aj for i ≤ j . If i < j , then any parenthesization of

Ai ,j must split product at some k, i ≤ k < j , i.e., compute Ai ,k ,Ak+1,j , and then Ai ,k · Ak+1,j .

Hence, for some k, the cost of computing Ai ,j is

the cost of computing Ai ,k plus

the cost of computing Ak+1,j plus

the cost of multiplying Ai ,k and Ak+1,j .

Arash Rafiey Dynamic Programming( Matrix Multiplication)

Optimal substructure:

Suppose that optimal parenthesization of Ai ,j splits theproduct between Ak and Ak+1.Then, parenthesizations of Ai ,k and Ak+1,j within this optimalparenthesization must be also optimal(otherwise, substitute the opt. parenthesization of Ai ,k (resp.Ak+1,j) to current parenthesization of Ai ,j and obtain a bettersolution — contradiction)

Use optimal substructure to construct an optimal solution:

1 split into two subproblems (choosing an optimal split),2 find optimal solutions to subproblem,3 combine optimal subproblem solutions.

Arash Rafiey Dynamic Programming( Matrix Multiplication)

A recursive solution

Let m[i , j ] denote minimum number of multiplications needed tocompute Ai ,j = Ai · Ai+1 · · ·Aj (full problem: m[1, n]).Recursive definition of m[i , j ]:

if i = j , then m[i , j ] = m[i , i ] = 0 (no multiplication needed)

if i < j , assume optimal split at k, i ≤ k < j . Since eachmatrix Ai is di−1 × di , Ai ,k is di−1 × dk and Ak+1,j is dk × dj ,

m[i , j ] = m[i , k] + m[k + 1, j ] + di−1 · dk · dj

We do not know optimal value of k. There are j − ipossibilities, k = i , i + 1, . . . , j − 1, hence

m[i , j ] =

0 if i = jmini≤k<jm[i , k] + m[k + 1, j ] if i < j+di−1 · dk · dj

We also keep track of optimal splits:

s[i , j ] = k ⇔ m[i , j ] = m[i , k] + m[k + 1, j ] + di−1 · dk · dj

(s[i , j ] is a value of k at which we split the product Ai ,j to obtainan optimal parenthesization)

Arash Rafiey Dynamic Programming( Matrix Multiplication)

A recursive solution

Let m[i , j ] denote minimum number of multiplications needed tocompute Ai ,j = Ai · Ai+1 · · ·Aj (full problem: m[1, n]).Recursive definition of m[i , j ]:

if i = j , then m[i , j ] = m[i , i ] = 0 (no multiplication needed)if i < j , assume optimal split at k, i ≤ k < j . Since eachmatrix Ai is di−1 × di , Ai ,k is di−1 × dk and Ak+1,j is dk × dj ,

m[i , j ] = m[i , k] + m[k + 1, j ] + di−1 · dk · dj

We do not know optimal value of k. There are j − ipossibilities, k = i , i + 1, . . . , j − 1, hence

m[i , j ] =

0 if i = jmini≤k<jm[i , k] + m[k + 1, j ] if i < j+di−1 · dk · dj

We also keep track of optimal splits:

s[i , j ] = k ⇔ m[i , j ] = m[i , k] + m[k + 1, j ] + di−1 · dk · dj

(s[i , j ] is a value of k at which we split the product Ai ,j to obtainan optimal parenthesization)

Arash Rafiey Dynamic Programming( Matrix Multiplication)

A recursive solution

Let m[i , j ] denote minimum number of multiplications needed tocompute Ai ,j = Ai · Ai+1 · · ·Aj (full problem: m[1, n]).Recursive definition of m[i , j ]:

if i = j , then m[i , j ] = m[i , i ] = 0 (no multiplication needed)if i < j , assume optimal split at k, i ≤ k < j . Since eachmatrix Ai is di−1 × di , Ai ,k is di−1 × dk and Ak+1,j is dk × dj ,

m[i , j ] = m[i , k] + m[k + 1, j ] + di−1 · dk · dj

We do not know optimal value of k. There are j − ipossibilities, k = i , i + 1, . . . , j − 1, hence

m[i , j ] =

0 if i = jmini≤k<jm[i , k] + m[k + 1, j ] if i < j+di−1 · dk · dj

We also keep track of optimal splits:

s[i , j ] = k ⇔ m[i , j ] = m[i , k] + m[k + 1, j ] + di−1 · dk · dj

(s[i , j ] is a value of k at which we split the product Ai ,j to obtainan optimal parenthesization)

Arash Rafiey Dynamic Programming( Matrix Multiplication)

Computing the optimal costs

Want to compute m[1, n], minimum cost for multiplyingA1 · A2 · · ·An.Recursively, it would take Ω(2n) steps: the same subproblems arecomputed over and over again.However, if we compute in a bottom-up fashion, we can reducerunning time to polynomial in n.

The recursive equation shows that cost m[i , j ] (product of j − i + 1matrices) depends only on smaller subproblems:for k = 1, . . . , j − 1,

Ai ,k is a product of k − i + 1 < j − i + 1 matrices,Ak+1,j is a product of j − k < j − i + 1 matrices.

Algorithm should fill table m in order of increasing lengths ofchains.

Arash Rafiey Dynamic Programming( Matrix Multiplication)

Computing the optimal costs

Want to compute m[1, n], minimum cost for multiplyingA1 · A2 · · ·An.Recursively, it would take Ω(2n) steps: the same subproblems arecomputed over and over again.However, if we compute in a bottom-up fashion, we can reducerunning time to polynomial in n.

The recursive equation shows that cost m[i , j ] (product of j − i + 1matrices) depends only on smaller subproblems:for k = 1, . . . , j − 1,

Ai ,k is a product of k − i + 1 < j − i + 1 matrices,Ak+1,j is a product of j − k < j − i + 1 matrices.

Algorithm should fill table m in order of increasing lengths ofchains.

Arash Rafiey Dynamic Programming( Matrix Multiplication)

Matrix-Chain-Order(p)1. n := length[p]− 12. for i := 1 to n3. m[i , i ] := 04. for ` := 2 to n4. for i := 1 to n − ` + 15. j := i + `− 1 m[i , j ] := ∞6. for k := i to j − 17. q := m[i , k] + m[k + 1, j ] + di−1 · dk · dj

8. if q < m[i , j ]9. m[i , j ] := q s[i , j ] := k

10. return m and s

Arash Rafiey Dynamic Programming( Matrix Multiplication)

Arash Rafiey Dynamic Programming( Matrix Multiplication)

Palindrome Subsequence Problem

Definition. Z = z1z2 . . . zk is a subsequence of S = s1s2 . . . sn ifthere exists an increasing sequence of indexes:1 ≤ i1 < i2 < · · · < ik ≤ n such that zj = sijExample.

S= G G C A C T G T A C↓ ↓ ↓ ↓

Z= G C C A

Z = GCCA is a subsequence of S = GGCACTGTACDefinition : We say a sequence is palindrome if reading it fromleft to right is the same as reading it from right to left.

A C G T G C A

Arash Rafiey Dynamic Programming( Matrix Multiplication)

Solution

Problem : We are given a sequence S of length n.

Goal: Design an algorithm that finds a maximum lengthsubsequence of S which is palindrome.

S= G G C A C T G T A C↓ ↓ ↓ ↓ ↓

Z= G C A C G

The optimal solution is :

S= G G C A C T G T A C↓ ↓ ↓ ↓ ↓ ↓ ↓

Z= C A T G T A C

Arash Rafiey Dynamic Programming( Matrix Multiplication)

Solution

Problem : We are given a sequence S of length n.

Goal: Design an algorithm that finds a maximum lengthsubsequence of S which is palindrome.

S= G G C A C T G T A C↓ ↓ ↓ ↓ ↓

Z= G C A C G

The optimal solution is :

S= G G C A C T G T A C↓ ↓ ↓ ↓ ↓ ↓ ↓

Z= C A T G T A C

Arash Rafiey Dynamic Programming( Matrix Multiplication)

Suppose S = x1, x2, . . . , xn is the input sequence of length n.

Lemma

Let Z = z1, z2, . . . , zk be a longest palindrome subsequence ofxi , xi+1, . . . , xj .

1 If z1 = xi and zk = xj then z2, z3, . . . , zk−1 is a longestpalindrome subsequence of xi+1, xi+2, . . . , xj−1.

2 If z1 6= xi then z1, z2, . . . , zk is a longest palindromesubsequence of xi+1, xi+2, . . . , xj .

3 If zk 6= xj then z1, z2, . . . , zk is a longest palindromesubsequence of xi , xi+2, . . . , xj−1.

Proof by contradiction!Case 1 z1 = xi and zk = xj . Suppose Z is not a longestpalindrome subsequence of xi , xi+1, . . . , xj . Let Z ′ be a longestpalindrome subsequence of xi+1, xi+2, . . . , xj−1. Note that|Z ′| > k − 2. Now z1Z

′zk is a palindrome subsequence ofxi , xi+1, . . . , xj and longer than Z , a contradiction.

Arash Rafiey Dynamic Programming( Matrix Multiplication)

Case 2. z1 6= xi . We have z1 = x`, ` > i and hence Z is a longestpalindrome subsequence of x`, x`+1, . . . , xj . Since x`, x`+1, . . . , xj isa subsequence of xi+1, . . . , xj , Z is a longest palindromesubsequence of xi+1, xi+2, . . . , xj .

Case 3. zk 6= xj . Similar to (2).

Arash Rafiey Dynamic Programming( Matrix Multiplication)

Now we design an algorithm based on the Lemma. LetS = x1, x2, . . . , xn.

Let M[i , j ] be the length of a longest palindrome subsequence fromxi to xj . If i = j then M[i , i ] = 1 (Since xi by itself is apalindrome).

Our goal is to compute M[1, n].

Recursive formula :

If xi = xj then M[i , j ] = M[i + 1, j − 1] + 2 otherwise :

M[i , j ] = maxM[i + 1, j ],M[i , j − 1]

Arash Rafiey Dynamic Programming( Matrix Multiplication)

Now we design an algorithm based on the Lemma. LetS = x1, x2, . . . , xn.

Let M[i , j ] be the length of a longest palindrome subsequence fromxi to xj . If i = j then M[i , i ] = 1 (Since xi by itself is apalindrome).

Our goal is to compute M[1, n].

Recursive formula :

If xi = xj then M[i , j ] = M[i + 1, j − 1] + 2 otherwise :

M[i , j ] = maxM[i + 1, j ],M[i , j − 1]

Arash Rafiey Dynamic Programming( Matrix Multiplication)

Longest-palindrome-subsequence(S)

1. int n = length(S);

// Initializing M.2. for (i = 1 to n) M[i , i ] = 1; // by definition

3. for (j = 1 to n − 1 )

4. for (i = 1 to n − j)

5. if (S [i ] = S [i + j ])

6. M[i , i + j ] = M[i + 1, i + j − 1] + 27. else

8. M[i , i + j ] = maxM[i + 1, i + j ],M[i , i + j − 1]

Arash Rafiey Dynamic Programming( Matrix Multiplication)