An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack...
Transcript of An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack...
![Page 1: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/1.jpg)
An Analysis of Facebook Photo Caching
Qi Huang, Ken Birman, Robbert van Renesse, Wyatt Lloyd, Sanjeev Kumar, Harry C. Li
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.Copyright is held by the Owner/Author(s).SOSP’13, Nov. 3–6, 2013, Farmington, Pennsylvania, USA.ACM 978-1-4503-2388-8/13/11.http://dx.doi.org/10.1145/2517349.2522722
![Page 2: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/2.jpg)
The problem● 250 billion photos on Facebook in 2013● 350 million more uploaded daily● High availability and low latency required for delivery infrastructure.
![Page 3: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/3.jpg)
Facebook's Photo-Serving Stack
Client
Browser cache
Akamai CDN
Facebook storage and caches
![Page 4: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/4.jpg)
Facebook's Photo-Serving Stack
Focus exclusively on Facebook stack
Client
Browser cache
Akamai CDN
Facebook storage and caches
![Page 5: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/5.jpg)
Edge Caches (FIFO)
● tens of independent edge caches across the US
Client
Browser cache
PoP
Edge cache
![Page 6: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/6.jpg)
Edge Caches (FIFO)
Purpose:
● reduce latency● reduce bandwidth utilization between clients and datacenters
Client
Browser cache
PoP
Edge cache
![Page 7: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/7.jpg)
Global Origin Cache (FIFO)
● a single global cache● requests routed to specific data centers based on hash of photo id
Client
Browser cache
PoP
Edge cache
Data center
Data center
Origin cache
hash(id)
![Page 8: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/8.jpg)
Global Origin Cache (FIFO)
Purpose:
● traffic sheltering for I/O bound backend
Client
Browser cache
PoP
Edge cache
Data center
Data center
Origin cache
hash(id)
![Page 9: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/9.jpg)
Haystack backend
● located in the same data centers as origin servers
Client
Browser cache
PoP
Edge cache
Data center
Data center
Origin cache
Backend (Haystack)
Backend (Haystack)
hash(id)
![Page 10: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/10.jpg)
Resizers
● photos are served in different sizes to different users● resized photos are treated as distinct objects by the cache layer
Client
Browser cache
PoP
Edge cache
Data center
Data center
Origin cache
Backend (Haystack)
Backend (Haystack)
hash(id)
R
R
Resizer
![Page 11: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/11.jpg)
Methodology
![Page 12: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/12.jpg)
Stack instrumentation
● desktop clients only● data for browser cache inferred by counting requests seen at browsers and
edge caches
Client
Browser cache
PoP
Edge cache
Data center
Origin cache
Backend (Haystack)R
Instrumentation scope
![Page 13: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/13.jpg)
Sampling method● request-based sampling: collect X% of requests● object-based sampling: collect X% of objects
![Page 14: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/14.jpg)
Sampling method● request-based sampling: collect X% of requests● object-based sampling: collect X% of objects
○ by deterministic test on photo id
![Page 15: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/15.jpg)
Object-based Sampling● better coverage of unpopular photos● simplified cross-stack analysis
![Page 16: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/16.jpg)
Analysis
![Page 17: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/17.jpg)
The collected data● One-month long trace● 77M requests from 13.2M user browsers ● 1.3M unique photos (2.6M photo objects)
![Page 18: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/18.jpg)
Traffic shelteringClient
Browser cache
PoP
Edge cache
Data center
Origin cache
Backend (Haystack)R
77.2M
65.5%
26.6M 11.2M
58% 31.8%
7.6M
![Page 19: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/19.jpg)
Popularity Distribution
● approximately Zipfian
![Page 20: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/20.jpg)
Traffic share by photo popularity
![Page 21: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/21.jpg)
Geographic Traffic Distribution
![Page 22: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/22.jpg)
Atlanta
Client To Edge Cache Traffic
![Page 23: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/23.jpg)
20% Atlanta
5%California
5%New York
35%Washington D.C.
5%Dallas
10%Miami
Client To Edge Cache Traffic
![Page 24: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/24.jpg)
Client To Edge Cache Traffic
● routing algorithm considers latency, edge capacity and ISP peering cost● substantial remote traffic is normal
![Page 25: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/25.jpg)
Edge Cache to Origin Cache Traffic
Traffic flow based on content, not locality.
![Page 26: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/26.jpg)
Cross-Region Traffic at Backend
● Caused by data migrations and hardware failures● Only 0.2% of Origin Cache to Backend traffic
![Page 27: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/27.jpg)
Potential Improvements
![Page 28: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/28.jpg)
Cache Simulation● replay the trace● use the first 25% to warm the cache● evaluate using the remaining 75% of the trace
![Page 29: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/29.jpg)
Object-hit Ratios for Edge Cache
● S4LRU improves 9% at x over FIFO● Still a lot of room for improvement
![Page 30: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/30.jpg)
S4LRU
● Four queues are maintained at levels 0 to 3
Cache space
L3
L2
L1
L0
![Page 31: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/31.jpg)
S4LRU
● Missed objects inserted into the head of level 0 queue
Cache space
L3
L2
L1
L0
![Page 32: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/32.jpg)
S4LRU
● On a cache hit, the item is moved to the head of the next higher queue
Cache space
L3
L2
L1
L0
![Page 33: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/33.jpg)
S4LRU
● items evicted from the tail of a queue to the head of the next lower queue
Cache space
L3
L2
L1
L0
![Page 34: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/34.jpg)
Byte-hit Ratios for Edge Cache
● LFU fails to reduce bandwidth utilization over the current solution
![Page 35: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/35.jpg)
Origin cache
S4LRU improves Origin slightly more than Edge (14%)
![Page 36: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/36.jpg)
Social-Network Analysis
![Page 37: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/37.jpg)
Content Age Effect
![Page 38: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/38.jpg)
Traffic share for non-profile photos by age
![Page 39: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/39.jpg)
Traffic share for social activity groups.
Number of followers
![Page 40: An Analysis of Facebook Photo Cachingiwanicki/courses/ds/2015/... · Facebook's Photo-Serving Stack Focus exclusively on Facebook stack Client Browser cache Akamai CDN Facebook storage](https://reader033.fdocuments.in/reader033/viewer/2022050311/5f730ace8fa5043d60246c45/html5/thumbnails/40.jpg)