CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman...
Transcript of CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman...
![Page 1: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/1.jpg)
CS 4102: AlgorithmsLecture 18: Greedy Algorithms
David WuFall 2019
![Page 2: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/2.jpg)
Warm-Up
2
Why is an algorithm’s space complexity important?
Why might a memory-intensivealgorithm be undesirable?
![Page 3: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/3.jpg)
Disadvantages of Large Memory Complexity
Using too much memory forces you to use slow memoryMemory is expensiveMay have too little memory for the algorithm to even runLots of memory hinders parallelismContention for the memoryMemory ≤ time
3
![Page 4: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/4.jpg)
Today’s Keywords
Greedy AlgorithmsChoice FunctionCache ReplacementHardware & Algorithms
4
CLRS Readings: Chapter 16
![Page 5: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/5.jpg)
Homework
HW6 due Tuesday, November 5, 11pm• Dynamic programming and greedy algorithms• Written (use LaTeX!) – Submit both zip and pdf (two separate attachments)!
HW10A also due Tuesday, November 5, 11pm• No late submissions allowed
5
![Page 6: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/6.jpg)
Review: Huffman Encoding
Choose the least frequent pair, combine into a subtree
6
G:14 E:13 L:9 I:8 W:6 N:3 S:3 A:2 D:2 R:2 Y:2 K:1 P:1 Q:1 U:1
Letter frequencies
![Page 7: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/7.jpg)
Review: Huffman Encoding
7
G:14 E:13 L:9 I:8 W:6 N:3 S:3 A:2 D:2 R:2 Y:2 K:1 P:1
Q:1 U:1
20 1
Subproblem of size 𝑛 − 1!
Choose the least frequent pair, combine into a subtree
Letter frequencies
![Page 8: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/8.jpg)
Review: Huffman Encoding
8
G:14 E:13
270 1
L:9 I:8
170 1
Q:1 U:1
20 1
K:1 P:1
20 1
40 1
N:3 S:3
60 1
10
0 1
W:6
R:2 Y:2
40 1
A:2 D:2
40 1
80 1
14
0 1
240 1
410 1
680 1
![Page 9: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/9.jpg)
Review: Optimality of Huffman Encoding
Proof Idea:• Show that there is an optimal tree where the least frequent
characters are siblings• Exchange argument
• Show that making them siblings and solving the new smaller sub-problem results in an optimal solution• Proof by contradiction
9
Greedy choice property
Optimal substructure
![Page 10: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/10.jpg)
Huffman Analysis: Exchange ArgumentClaim: if 𝑐&, 𝑐( are the least-frequent characters, then there is an optimal prefix-free code where 𝑐&, 𝑐( are siblings
• Equivalently: encodings of 𝑐&, 𝑐( have the same length and differ only in their last bit
10𝑐&
𝑇*+,
𝑐(
Case 1: Suppose 𝑐&, 𝑐( are siblings in 𝑇*+,. Proof. Consider some optimal tree 𝑇*+,
Then claim holds
![Page 11: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/11.jpg)
Huffman Analysis: Exchange ArgumentClaim: if 𝑐&, 𝑐( are the least-frequent characters, then there is an optimal prefix-free code where 𝑐&, 𝑐( are siblings
• Equivalently: encodings of 𝑐&, 𝑐( have the same length and differ only in their last bit
11
Case 2: Suppose 𝑐&, 𝑐( are not siblings in 𝑇*+,
Proof. Consider some optimal tree 𝑇*+,
𝑐(
𝑎
𝑐&
𝑇*+,
𝑏
Let 𝑎, 𝑏 be sibling leaves of maximum depth
Optimal tree must be full (every non-leaf node has two children);
otherwise, can move a leaf node up and reduce the encoding size
Why must this exist?
Exchange argument: Since 𝑓01 ≤ 𝑓2 and 𝑓03 ≤ 𝑓4, swapping 𝑐& with 𝑎 (and 𝑐( with 𝑏) cannot increase the cost of the tree
![Page 12: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/12.jpg)
Huffman Analysis: Optimal Substructure
Claim: An optimal solution for 𝐹 involves finding an optimal solution for 𝐹′, then adding 𝑐&, 𝑐( as children to 𝜎
12
𝑐& 𝑐(
𝑐& 𝑐(
𝜎
𝐹′
𝐹
Proof by contradiction: If there is a better solution for 𝐹, then can use that to obtain a better solution for 𝐹8, which contradicts optimality of solution for 𝐹8
![Page 13: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/13.jpg)
Caching Problem
13
Why is an algorithm’s space complexity important?
Why might a memory-intensivealgorithm be undesirable?
![Page 14: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/14.jpg)
von Neumann Bottleneck
Reading from memory is slowBig memory = slow memorySolution: hierarchical memoryTakeaway for algorithms: More memory accesses means bigger runtime
14
CPU, registers L1 Cache
If not look hereHopefully your data in here
0.2 ns1 ns
L2 Cache
If not look here
4 ns
L3 Cache
If not look here
20 ns
Main Memory
If not look here
100 nsDisk seek: 10 ms
![Page 15: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/15.jpg)
Caching Problem
Cache misses are very expensiveWhen we load something new into cache, we must eliminate something already thereWe want the best cache “schedule” to minimize the number of misses
15
![Page 16: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/16.jpg)
Caching Problem Definition
Input: • 𝑘 = size of the cache• 𝑀 = 𝑚&,𝑚(,…𝑚> = memory access pattern
Output: • “Schedule” for the cache (list of items in the cache at each time) which
minimizes cache misses
16
![Page 17: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/17.jpg)
Caching Example
17
Sequence of cache accesses
Cachecontents
A
B
C
A
B
C
A
B
C
A B C D A D E A D B A E C E A
A
B
C
Must evict something to make room for D
![Page 18: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/18.jpg)
Caching Example
18
Sequence of cache accesses
Cachecontents
Suppose we evict A
Must evict something to make room for D
A B C D A D E A D B A E C E A
A
B
C
A
B
C
A
B
C
D
B
C
A
B
C
![Page 19: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/19.jpg)
Caching Example
19
Sequence of cache accesses
Cachecontents
Suppose we evict C
Must evict something to make room for D
A B C D A D E A D B A E C E A
A
B
C
A
B
C
A
B
C
A
B
D
A
B
C
Objective: Devise cache eviction strategy to minimize number of cache missesSimplifying assumption: We know the entire sequence of accesses ahead of time
(valid assumption for data-oblivious computations)
![Page 20: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/20.jpg)
Greedy Algorithms
Requires optimal substructure• Solution to larger problem contains the solution to a smaller one• Only a single subproblem to consider
General Blueprint:1. Identify a greedy choice property
• Show that this choice is guaranteed to be included in some optimal solution2. Repeatedly apply the choice property until no subproblems remain
20
![Page 21: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/21.jpg)
Greedy Strategy
21
Sequence of cache accesses
Cachecontents
A B C D A D E A D B A E C E A
A
B
C
A
B
C
A
B
C
A
B
C
Belady eviction policy: Evict the item accessed farthest in the future
![Page 22: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/22.jpg)
Greedy Strategy
22
Sequence of cache accesses
Cachecontents
A B C D A D E A D B A E C E A
A
B
C
A
B
C
A
B
C
A
B
C
Belady eviction policy: Evict the item accessed farthest in the future
Greedy choice: Evict C
![Page 23: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/23.jpg)
Greedy Strategy
23
Sequence of cache accesses
Cachecontents
A
B
C
A
B
C
A
B
C
A B C D A D E A D B A E C E A
A
B
D
A
B
D
A
B
C
A
B
D
Belady eviction policy: Evict the item accessed farthest in the future
![Page 24: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/24.jpg)
Greedy Strategy
24
Sequence of cache accesses
Cachecontents
A
B
C
A
B
C
A
B
C
A B C D A D E A D B A E C E A
A
B
D
A
B
D
A
B
C
A
B
D
Belady eviction policy: Evict the item accessed farthest in the future
Greedy choice: Evict B
![Page 25: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/25.jpg)
Greedy Strategy
25
Sequence of cache accesses
Cachecontents
A
B
C
A
B
C
A
B
C
A B C D A D E A D B A E C E A
A
B
D
A
B
D
A
E
D
A
E
D
A
E
D
A
B
C
A
B
D
Belady eviction policy: Evict the item accessed farthest in the future
![Page 26: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/26.jpg)
Greedy Strategy
26
Sequence of cache accesses
Cachecontents
A
B
C
A
B
C
A
B
C
A B C D A D E A D B A E C E A
A
B
D
A
B
D
A
E
D
A
E
D
A
E
D
A
B
C
A
B
D
Belady eviction policy: Evict the item accessed farthest in the future
Greedy choice: Evict D
![Page 27: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/27.jpg)
Greedy Strategy
27
Sequence of cache accesses
Cachecontents
Belady eviction policy: Evict the item accessed farthest in the future
A
B
C
A
B
C
A
B
C
A B C D A D E A D B A E C E A
A
B
D
A
B
D
A
E
D
A
E
D
A
E
B
A
E
B
A
E
D
A
B
C
A
B
D
A
E
B
![Page 28: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/28.jpg)
Greedy Strategy
28
Sequence of cache accesses
Cachecontents
Belady eviction policy: Evict the item accessed farthest in the future
A
B
C
A
B
C
A
B
C
A B C D A D E A D B A E C E A
A
B
D
A
B
D
A
E
D
A
E
D
A
E
B
A
E
B
A
E
D
A
B
C
A
B
D
A
E
B
Greedy choice: Evict B
![Page 29: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/29.jpg)
Greedy Strategy
29
Sequence of cache accesses
Cachecontents
Belady eviction policy: Evict the item accessed farthest in the future
A
B
C
A
B
C
A
B
C
A B C D A D E A D B A E C E A
A
B
D
A
B
D
A
E
D
A
E
D
A
E
B
A
E
B
A
E
D
A
B
C
A
B
D
A
E
C
A
E
C
A
E
B
Total number of cache misses: 4
![Page 30: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/30.jpg)
Greedy Algorithm
init cache = first k accesses
for each i = 1,...,n:
if m[i] in cache:print cache
else:
z = furthest-in-future from cacheevict z, add m[i] to cache
print cache
30
m[i]: element accessed on 𝑖@A step
![Page 31: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/31.jpg)
Running Time of Greedy Algorithm
init cache = first k accesses
for each i = 1,...,n:
if m[i] in cache:print cache
else:
z = furthest-in-future from cacheevict z, add m[i] to cache
print cache
31
𝑂(𝑘)𝑛 times𝑂(𝑘)𝑂(𝑘)
𝑂(𝑘𝑛)
𝑂(1)𝑂(𝑘)
Overall runtime: 𝑂(𝑘𝑛()
![Page 32: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/32.jpg)
Proof of Correctness: Exchange Argument
Common technique to show correctness of a greedy algorithm
General idea: argue that at every step, the greedy choice is part of some optimal solution
Approach: Start with an arbitrary optimal solution and show that exchanging an item from the optimal solution with your greedy choice makes the new solution no worse (i.e., the greedy choice is as good as the optimal choice)
32
![Page 33: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/33.jpg)
Exchange Argument for Belady Caching Algorithm
Let 𝑆 be the schedule chosen by the greedy algorithmLet 𝑆∗ be any optimal schedule (that minimizes the number of cache misses)Lemma: If 𝑆G and 𝑆 agree on the first 𝑖 accesses, then there is a schedule 𝑆GH& that agrees with 𝑆 on the first 𝑖 + 1 accesses such that
misses 𝑆GH& ≤ misses 𝑆GCorrectness then follows by induction:
33
𝑆∗Agrees with 𝑆
on first 0 accesses
𝑆& 𝑆(Agrees with 𝑆
on first access
Agrees with 𝑆on first 2 accesses
… 𝑆Lemma Lemma Lemma Lemma
Optimal Greedy
misses 𝑆 = misses 𝑆> ≤ misses 𝑆>N& ≤ ⋯ ≤ misses 𝑆P = misses(𝑆∗)
Optimal Optimal
![Page 34: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/34.jpg)
Exchange Argument for Belady Caching Algorithm
34
𝑆G
𝑆
𝑆GH& ? ? ? ?
first 𝑖 accesses
must agree with 𝑆
Goal: Need to fill in the rest of 𝑆GH& to have no
more cache misses than 𝑆G
![Page 35: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/35.jpg)
Exchange Argument for Belady Caching Algorithm
35
=
Consider access 𝑚GH& = 𝑑Case 1: 𝑑 is in the cache. Then, neither 𝑆G nor 𝑆 need to evict an element so we can use the same cache for 𝑆GH&
Cache after 𝑖 accesses with policy 𝑆G
𝑓𝑒
Cache after 𝑖 accesses with policy 𝑆
𝑓𝑒
Lemma: If 𝑆G and 𝑆 agree on the first 𝑖 accesses, then there is a schedule 𝑆GH& that agrees with 𝑆 on the first 𝑖 + 1 accesses such that misses 𝑆GH& ≤ misses 𝑆G
Since 𝑆G agrees with 𝑆 for the first 𝑖 accesses, the state of the cache at access 𝑖 + 1will be identical
Cache after 𝑖 accesses with policy 𝑆GH&
𝑓𝑒Remaining evictions will follow 𝑆G:
misses 𝑆GH& = misses 𝑆G
![Page 36: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/36.jpg)
Cache after 𝑖 accesses with policy 𝑆G
Exchange Argument for Belady Caching Algorithm
36
=
Consider access 𝑚GH& = 𝑑Case 2: 𝑑 is not in the cache and both 𝑆G and 𝑆 evict the same element (e.g., 𝑓) from the cache. In this case, we can use the same cache for 𝑆GH&
𝑓𝑒
Cache after 𝑖 accesses with policy 𝑆
𝑓𝑒
Lemma: If 𝑆G and 𝑆 agree on the first 𝑖 accesses, then there is a schedule 𝑆GH& that agrees with 𝑆 on the first 𝑖 + 1 accesses such that misses 𝑆GH& ≤ misses 𝑆G
Since 𝑆G agrees with 𝑆 for the first 𝑖 accesses, the state of the cache at access 𝑖 + 1will be identical
Cache after 𝑖 accesses with policy 𝑆GH&
𝑑𝑒Remaining evictions will follow 𝑆G:
misses 𝑆GH& = misses 𝑆G
![Page 37: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/37.jpg)
Exchange Argument for Belady Caching Algorithm
37
=
Consider access 𝑚GH& = 𝑑Case 3: 𝑑 is not in the cache and 𝑆G and 𝑆 evict different elements (e.g., 𝑆G evicts 𝑒and 𝑆 evicts 𝑓)
Cache after 𝑖 accesses with policy 𝑆G
𝑓𝑒
Cache after 𝑖 accesses with policy 𝑆
𝑓𝑒
Lemma: If 𝑆G and 𝑆 agree on the first 𝑖 accesses, then there is a schedule 𝑆GH& that agrees with 𝑆 on the first 𝑖 + 1 accesses such that misses 𝑆GH& ≤ misses 𝑆G
Since 𝑆G agrees with 𝑆 for the first 𝑖 accesses, the state of the cache at access 𝑖 + 1will be identical
≠Cache after 𝑖 + 1 accesses with policy 𝑆
𝑑𝑒
Cache after 𝑖 + 1 accesses with policy 𝑆G
𝑓𝑑
![Page 38: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/38.jpg)
Exchange Argument for Belady Caching Algorithm
38
=
Consider access 𝑚GH& = 𝑑Case 3: 𝑑 is not in the cache and 𝑆G and 𝑆 evict different elements (e.g., 𝑆G evicts 𝑒and 𝑆 evicts 𝑓)
Cache after 𝑖 accesses with policy 𝑆G
𝑓𝑒
Cache after 𝑖 accesses with policy 𝑆
𝑓𝑒
Lemma: If 𝑆G and 𝑆 agree on the first 𝑖 accesses, then there is a schedule 𝑆GH& that agrees with 𝑆 on the first 𝑖 + 1 accesses such that misses 𝑆GH& ≤ misses 𝑆G
Since 𝑆G agrees with 𝑆 for the first 𝑖 accesses, the state of the cache at access 𝑖 + 1will be identical
≠Cache after 𝑖 + 1 accesses with policy 𝑆
𝑑𝑒
Cache after 𝑖 + 1 accesses with policy 𝑆G
𝑓𝑑
Key observation: caches only differ in a single element
![Page 39: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/39.jpg)
Exchange Argument for Belady Caching Algorithm
39
𝑆G
𝑆
𝑆GH& ? ? ? ?
first 𝑖 accesses
must agree with 𝑆
Objective: Need to fill in the rest of 𝑆GH& to have no more cache misses than 𝑆G
evict 𝑒, add 𝑑
evict 𝑓, add 𝑑
![Page 40: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/40.jpg)
Exchange Argument for Belady Caching Algorithm
40
𝑆G
𝑆
𝑆GH& 𝑚,
first 𝑖 accesses
must agree with 𝑆
Objective: Need to fill in the rest of 𝑆GH& to have no more cache misses than 𝑆G
evict 𝑒, add 𝑑
evict 𝑓, add 𝑑 Strategy: Copy 𝑆G until first access in 𝑆G that involves adding or
removing 𝑒 or 𝑓 from the cache
first access that adds/removes 𝑒 or 𝑓 in 𝑆G
![Page 41: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/41.jpg)
Exchange Argument for Belady Caching Algorithm
41
𝑆G
𝑆
𝑆GH& 𝑚,
first 𝑖 accesses
must agree with 𝑆
Objective: Need to fill in the rest of 𝑆GH& to have no more cache misses than 𝑆G
evict 𝑒, add 𝑑
evict 𝑓, add 𝑑 Strategy: Copy 𝑆G until first access in 𝑆G that involves adding or
removing 𝑒 or 𝑓 from the cache
first access that adds/removes 𝑒 or 𝑓 in 𝑆G
Three cases: 𝑚, = 𝑒; 𝑚, = 𝑓; 𝑚, ≠ 𝑒, 𝑓
![Page 42: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/42.jpg)
Exchange Argument for Belady Caching Algorithm
42
𝑆G
𝑆GH&
𝑒
𝑚,
first 𝑖 accesses evict 𝑒, add 𝑑
evict 𝑓, add 𝑑
Case 1:𝑚, = 𝑒
evict some element 𝑥 and add 𝑒 to cache
Cache after 𝑡 − 1 accesses with policy 𝑆G
𝑓
Cache after 𝑡 − 1 accesses with policy 𝑆GH&
𝑒
Two possibilities:• 𝑥 = 𝑓. Then, cache after 𝑡 accesses with policy 𝑆G is identical to that with policy 𝑆GH&:
Cache after 𝑡 accesses with policy 𝑆G
𝑒Remaining evictions will follow 𝑆G:misses 𝑆GH& < misses 𝑆G
![Page 43: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/43.jpg)
Exchange Argument for Belady Caching Algorithm
43
𝑆G
𝑆GH&
𝑒
𝑚,
first 𝑖 accesses evict 𝑒, add 𝑑
evict 𝑓, add 𝑑
Case 1:𝑚, = 𝑒
evict some element 𝑥 and add 𝑒 to cache
Two possibilities:• 𝑥 ≠ 𝑓.
Cache after 𝑡 − 1 accesses with policy 𝑆G
𝑓𝑥
Cache after 𝑡 accesses with policy 𝑆G
𝑓𝑒
?
Cache after 𝑡 − 1 accesses with policy 𝑆G
𝑓
Cache after 𝑡 − 1 accesses with policy 𝑆GH&
𝑒
![Page 44: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/44.jpg)
Exchange Argument for Belady Caching Algorithm
44
𝑆G
𝑆GH&
𝑒
𝑚,
first 𝑖 accesses evict 𝑒, add 𝑑
evict 𝑓, add 𝑑
Case 1:𝑚, = 𝑒
evict some element 𝑥 and add 𝑒 to cache
Two possibilities:• 𝑥 ≠ 𝑓. 𝑆GH& will also evict 𝑥 from the cache and load 𝑓, so caches now match.
Cache after 𝑡 − 1 accesses with policy 𝑆GH&
𝑒𝑥
Cache after 𝑡 accesses with policy 𝑆GH&
𝑒𝑓
Cache after 𝑡 − 1 accesses with policy 𝑆G
𝑓
Cache after 𝑡 − 1 accesses with policy 𝑆GH&
𝑒
![Page 45: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/45.jpg)
Exchange Argument for Belady Caching Algorithm
45
𝑆G
𝑆GH&
𝑒
𝑚,
first 𝑖 accesses evict 𝑒, add 𝑑
evict 𝑓, add 𝑑
Case 1:𝑚, = 𝑒
evict some element 𝑥 and add 𝑒 to cache
Two possibilities:• 𝑥 ≠ 𝑓. 𝑆GH& will also evict 𝑥 from the cache and load 𝑓, so caches now match.
Remaining evictions will follow 𝑆G:misses 𝑆GH& = misses 𝑆G
Cache after 𝑡 accesses with policy 𝑆GH&
𝑒𝑓
Cache after 𝑡 − 1 accesses with policy 𝑆G
𝑓
Cache after 𝑡 − 1 accesses with policy 𝑆GH&
𝑒
![Page 46: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/46.jpg)
Exchange Argument for Belady Caching Algorithm
46
𝑆G
𝑆
𝑆GH& 𝑚,
first 𝑖 accesses
must agree with 𝑆
Objective: Need to fill in the rest of 𝑆GH& to have no more cache misses than 𝑆G
evict 𝑒, add 𝑑
evict 𝑓, add 𝑑 Strategy: Copy 𝑆G until first access in 𝑆G that involves adding or
removing 𝑒 or 𝑓 from the cache
first access that adds/removes 𝑒 or 𝑓 in 𝑆G
Three cases: 𝑚, = 𝑒; 𝑚, = 𝑓; 𝑚, ≠ 𝑒, 𝑓
![Page 47: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/47.jpg)
Exchange Argument for Belady Caching Algorithm
47
𝑆G
𝑆GH& 𝑚,
first 𝑖 accesses evict 𝑒, add 𝑑
evict 𝑓, add 𝑑
Case 2:𝑚, = 𝑓
element 𝑒 was not accessed here (by definition of 𝑚,)
Contradiction: greedy choice is to evict element that is furthest in future, but element 𝑓 is used before element 𝑒
Conclusion: this case cannot happen
![Page 48: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/48.jpg)
Exchange Argument for Belady Caching Algorithm
48
𝑆G
𝑆
𝑆GH& 𝑚,
first 𝑖 accesses
must agree with 𝑆
Objective: Need to fill in the rest of 𝑆GH& to have no more cache misses than 𝑆G
evict 𝑒, add 𝑑
evict 𝑓, add 𝑑 Strategy: Copy 𝑆G until first access in 𝑆G that involves adding or
removing 𝑒 or 𝑓 from the cache
first access that adds/removes 𝑒 or 𝑓 in 𝑆G
Three cases: 𝑚, = 𝑒; 𝑚, = 𝑓; 𝑚, ≠ 𝑒, 𝑓
![Page 49: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/49.jpg)
Exchange Argument for Belady Caching Algorithm
49
𝑆G
𝑆GH& 𝑚,
first 𝑖 accesses evict 𝑒, add 𝑑
evict 𝑓, add 𝑑
Case 3:𝑚, ≠ 𝑒, 𝑓
evict 𝑓 and add 𝑚, = 𝑥 to cache
Cache after 𝑡 − 1 accesses with policy 𝑆G
𝑓
Cache after 𝑡 accesses with policy 𝑆G
𝑥
Observation: not loading 𝑒, 𝑓 so must be evicting either 𝑒 or 𝑓In 𝑆G, 𝑒 was already evicted and has not been loaded (by definition of 𝑚,)Only option is for 𝑆G to evict 𝑓
Cache after 𝑡 − 1 accesses with policy 𝑆G
𝑓
Cache after 𝑡 − 1 accesses with policy 𝑆GH&
𝑒
![Page 50: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/50.jpg)
Exchange Argument for Belady Caching Algorithm
50
𝑆G
𝑆GH& 𝑚,
first 𝑖 accesses evict 𝑒, add 𝑑
evict 𝑓, add 𝑑
Case 3:𝑚, ≠ 𝑒, 𝑓
evict 𝑓 and add 𝑚, = 𝑥 to cache
Observation: not loading 𝑒, 𝑓 so must be evicting either 𝑒 or 𝑓In 𝑆G, 𝑒 was already evicted and has not been loaded (by definition of 𝑚,).Only option is for 𝑆G to evict 𝑓
Cache after 𝑡 − 1 accesses with policy 𝑆G
𝑓
Cache after 𝑡 accesses with policy 𝑆G
𝑥
Cache after 𝑡 − 1 accesses with policy 𝑆G
𝑓
Cache after 𝑡 − 1 accesses with policy 𝑆GH&
𝑒
Evict 𝑒 and add 𝑥 in 𝑆GH&will yield same cache state!
![Page 51: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/51.jpg)
Exchange Argument for Belady Caching Algorithm
51
𝑆G
𝑆GH& 𝑚,
first 𝑖 accesses evict 𝑒, add 𝑑
evict 𝑓, add 𝑑
Case 3:𝑚, ≠ 𝑒, 𝑓
evict 𝑓 and add 𝑚, = 𝑥 to cache
Cache after 𝑡 accesses with policy 𝑆GH&
𝑥Remaining evictions will follow 𝑆G:misses 𝑆GH& = misses 𝑆G
Cache after 𝑡 − 1 accesses with policy 𝑆G
𝑓
Cache after 𝑡 − 1 accesses with policy 𝑆GH&
𝑒
Evict 𝑒 and add 𝑥 in 𝑆GH&will yield same cache state!
![Page 52: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/52.jpg)
Exchange Argument for Belady Caching Algorithm
52
𝑆G
𝑆
𝑆GH& 𝑚,
first 𝑖 accesses
must agree with 𝑆
Conclusion: In all three cases, we can construct a strategy wheremisses 𝑆GH& ≤ misses(𝑆G)
evict 𝑒, add 𝑑
evict 𝑓, add 𝑑 Strategy: Copy 𝑆G until first access in 𝑆G that involves adding or
removing 𝑒 or 𝑓 from the cache
first access that adds/removes 𝑒 or 𝑓 in 𝑆G
Three cases: 𝑚, = 𝑒; 𝑚, = 𝑓; 𝑚, ≠ 𝑒, 𝑓
![Page 53: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/53.jpg)
Exchange Argument for Belady Caching Algorithm
Let 𝑆 be the schedule chosen by the greedy algorithmLet 𝑆∗ be any optimal schedule (that minimizes the number of cache misses)Lemma: If 𝑆G and 𝑆 agree on the first 𝑖 accesses, then there is a schedule 𝑆GH& that agrees with 𝑆 on the first 𝑖 + 1 accesses such that
misses 𝑆GH& ≤ misses 𝑆GCorrectness then follows by induction:
53
𝑆∗Agrees with 𝑆
on first 0 accesses
𝑆& 𝑆(Agrees with 𝑆
on first access
Agrees with 𝑆on first 2 accesses
… 𝑆Agrees with 𝑆
on all 𝑛accesses
Lemma Lemma Lemma LemmaOptimal Greedy
misses 𝑆 = misses 𝑆> ≤ misses 𝑆>N& ≤ ⋯ ≤ misses 𝑆P = misses(𝑆∗)
Optimal Optimal
![Page 54: CS 4102: Algorithmsjh2jf/courses/fall2019/cs4102/lectures/... · Review: Optimality of Huffman Encoding Proof Idea: •Show that there is an optimal tree where the least frequent](https://reader031.fdocuments.in/reader031/viewer/2022013023/6021eec59e87047b88365ca7/html5/thumbnails/54.jpg)
Belady Caching
54
Sequence of cache accesses
Cachecontents
Belady eviction policy: Evict the item accessed farthest in the future
A
B
C
A
B
C
A
B
C
A B C D A D E A D B A E C E A
A
B
D
A
B
D
A
E
D
A
E
D
A
E
B
A
E
B
A
E
D
A
B
C
A
B
D
A
E
C
A
E
C
A
E
B
In online settings, we do not know exact sequence of memory accesses, so cannot compute farthest future accessHeuristic: past access pattern is a good predictor for futureStrategy: evict the least-recently used item (LRU caching)