Caching – Methodology & Strategies

By Vu Van Tiep

Manager, Head Of Web Platform - VNG

Skype: tiepvv

Scale Up vs Scale Out

Scale UP

Scale Out

Load Balancing

Agenda

• What

• Cache Layer

• Cache Application/Service• Methodology

• Strategies

• Performance?

• Demo

• Q&A

Cache – What?

• Is a hardware or softwarecomponent that stores data sofuture requests for that data canbe served faster;

• The data stored in a cache mightbe the result of an earliercomputation, or the duplicateof data stored elsewhere

https://en.wikipedia.org/wiki/Cache_(computing)

Application

Cache

Database

Hit

MissUpdateCache

Cache Layer

Client (Browser) Application Caching Database

- Browser- Proxy- CDN

- in-memory- Server Caching

- Distributed in memory key/value

- Service:• Memcache• Membase• Redis• …

- Database caching• Entity In memory

- DB:• MySQL• MongoDB• Casandra, LevelDB• …

1 2

34

56

Http Layer Caching

Browser & Proxy

Client(Browser)

Web Server1

2

Client (Browser)

Client(Browser)

Web ServerProxy

Client(Browser) ProxyCache

Cache

1

2

1 2

34

Response Header• Expires

• Last-Modified

• Etag

• Cache-Control

• Pragram

Browser Caching – Http Header

Browser Caching – Http Header

Browser Caching – Local Storage

Application Layer Caching

Types/Levels of Cache

Level 1 (L1) Cache

It is also called primary or internal cache. It is built directly into the processor chip. It has small capacity from 8 Km to 128 Kb.

Level 2 (L2) Cache

It is slower than L1 cache. Its storage capacity is more, i-e. From 64 Kb to 16 MB. The current processors contain advanced transfer cache on processor chip that is a type of L2 cache. The common size of this cache is from 512 kb to 8 Mb.

Level 3 (L3) Cache

This cache is separate from processor chip on the motherboard. It exists on the computer that uses L2 advanced transfer cache. It is slower than L1 and L2 cache. The personal computer often has up to 8 MB of L3 cache.

L1 cache reference 0.5 ns

Branch mispredict 5 ns

L2 cache reference 7 ns

Mutex lock/unlock 100 ns

Main memory reference 100 ns

Compress 1K bytes with Zippy 10,000 ns

Send 2K bytes over 1 Gbps network 20,000 ns (1)

Read 1 MB sequentially from memory 250,000 ns (2)

Round trip within same datacenter 500,000 ns

Disk seek 10,000,000 ns 1 + 2 < (3)

Read 1 MB sequentially from network 10,000,000 ns

Read 1 MB sequentially from disk 30,000,000 ns

Send packet CA->Netherlands->CA 150,000,000 ns

From Jeff Dean - google.com

Latency Numbers

Note: 1 ns = 10^-9 seconds

Cache: Methodology

Data/Object

Data/Object

Serialize

DeSerialize

Compress

DeCompress

MemcacheOr

Redis

Set

Get

1 2 3

123

Serialize

Disk / Network

Java Object PHP Object C# Object

Write ReadSerialize DeSerialize

Performance Comparison

Text Serialize

Json – size: 167 bytes, se: 6ms, de: 3ms

XML – 235 bytes

Binary Serialize

Thrift – size: 64 bytes, se: 53ms, de:1ms

ProtoBuf – size: 31 bytes, se: 34ms, de: 4ms

Binary serialize take less space

XML, JsonProtoBuf, ThriftMessage PackApache Arvo…

Framework

Compress/Decompress

Name Ratio C.speed D.speedMB/s MB/s

zstd 0.8.2 -1 2.877 330 940

zlib 1.2.8 -1 2.73 95 360

brotli 0.4 -0 2.708 320 375

QuickLZ 1.5 2.237 510 605

LZO 2.09 2.106 610 870

LZ4 r131 2.101 620 3100

Snappy 1.1.3 2.091 480 1600

LZF 3.6 2.077 375 790

As a reference, several fastcompression algorithms were testedand compared on a Core i7-3930KCPU @ 4.5GHz, using lzbench, anopen-source in-memory benchmarkby @inikep compiled with GCC 5.4.0,with the Silesia compression corpus.

https://github.com/facebook/zstd

Caching Strategies

Cache Strategies

1. Optimize Data Size

2. Hot Object

3. In-Process

4. Lazy load

5. Distributed

Optimize Data Size

Name Ratio C.speed D.speed

MB/s MB/s

zstd 0.8.2 -1 2.877 330 940

zlib 1.2.8 -1 2.73 95 360

brotli 0.4 -0 2.708 320 375

QuickLZ 1.5 2.237 510 605

LZO 2.09 2.106 610 870

LZ4 r131 2.101 620 3100

Snappy 1.1.3 2.091 480 1600

LZF 3.6 2.077 375 790

CompressSerialize

Method Size (smaller is better)

Thrift — TCompactProtocol 278 (not bad)

Thrift — TBinaryProtocol 460

Protocol Buffers 250 (winner!)

RMI 905

REST — JSON 559

REST — XML 836

High Level Goals:

• Transparent between multiple programming

languages (PHP/Java/Cpp/…)

• Maintain Right balance between:

• Efficiency (how much time/space?)• Availability of existing libraries

Hot Object => Eviction Algorithms

2G Ram

10GData

ServerData

How to cache?

Eviction Algorithms: Least Frequently Used (LFU)

A B C D E F

7 6 3 2 4 1

A B C D E F

7 6 3 3 4 1

Access

Most Frequently Used

Least Frequently Used

Eviction Algorithms: Least Recently Used (LRU)

1 2 3 4 5 6

4 1 2 3 5 6

Access

In-Process

Request

Application

Local cache Backend/Database

Application

Lazy Load

Request

Logic

Caching (Memory)

Backend/Database

Application

Warmup Worker

Distributed Cache

Server 1

Server 3

Server 2Server 4

Distributed CacheClient 1

Client 2

Client 3

Performance: Profiler is the key

Demo

Contact info:

• Name: Vu Van Tiep

• Email: tiepvv@vng.com.vn / vuvantiep@gmail.com

• Skype: tiepvv

Q&A

Thank You!

Reference

• https://en.wikipedia.org/wiki/Cache_(computing)

• http://www.slideshare.net/IgorAnishchenko/pb-vs-thrift-vs-avro

• https://github.com/facebook/zstd

• http://www.codeproject.com/Articles/56138/Consistent-hashing