Memcached + Innodb = Performance. The Waffle Grid Project.

33
Partners.org | © MySQL AB 2005 | www.mysql.com 1 Memcached + Innodb = Performance The Waffle Grid Project Matt Yonkovit and Yves Trudeau

description

Waffle Grid is a distributed level 2 cache for Innodb based on memcached.

Transcript of Memcached + Innodb = Performance. The Waffle Grid Project.

Page 1: Memcached + Innodb = Performance. The Waffle Grid Project.

Partners.org | © MySQL AB 2005 | www.mysql.com 1

Memcached + Innodb = Performance

The Waffle Grid Project

Matt Yonkovit and Yves Trudeau

Page 2: Memcached + Innodb = Performance. The Waffle Grid Project.

2Copyright 2009 MySQL AB The World’s Most Popular Open Source Database

Agenda

• Why?• Presenting WaffleGrid• Benchmarks• Status and roadmap• WaffleGrid quick start• Questions

Page 3: Memcached + Innodb = Performance. The Waffle Grid Project.

3Copyright 2009 MySQL AB The World’s Most Popular Open Source Database

Why?: Common Problems

• Clients Face Similar issues – Some clients are stuck with 32 bit architectures– Other clients actually exceed their current servers

resources– While other clients simply outgrow their disk before they

outgrow the rest of their server– Many clients don't have the financial resources to scrap

their “good enough” servers to buy servers with > 128GB of memory

Page 4: Memcached + Innodb = Performance. The Waffle Grid Project.

4Copyright 2009 MySQL AB The World’s Most Popular Open Source Database

Key Goal's for High Performance

• As much “hot” ( heavily used ) data should fit into memory as possible ( all is best )‏

– Disk IO is the one of if not the most expensive operation in your system

– Millisecond access -vs- microsecond– Current disks >3ms

Page 5: Memcached + Innodb = Performance. The Waffle Grid Project.

5Copyright 2009 MySQL AB The World’s Most Popular Open Source Database

A bigger buffer pool matters

Page 6: Memcached + Innodb = Performance. The Waffle Grid Project.

6Copyright 2009 MySQL AB The World’s Most Popular Open Source Database

Scalability

• A huge draw back to the standard MySQL InnoDB storage engine is its difficulty to scale unless you plan for it up front... – Many clients find out too late they need to rewrite

portions of their applications to get better performance: • Read/Write splitting • Sharding

– Tools over the years have popped up trying to address these issues

• MySQL-Proxy• Spock-Proxy• HiveDB

Page 7: Memcached + Innodb = Performance. The Waffle Grid Project.

7Copyright 2009 MySQL AB The World’s Most Popular Open Source Database

Real World Example

• Your database has grown up to 120GB, of which 50GB is “hot” active most of the time

• Your server is limited to 32GB, you are out of the InnoDB buffer pool so disk access is heavy

• Database performance suffers from the disk access• You have two other decommissioned servers with 16GB of

memory

Page 8: Memcached + Innodb = Performance. The Waffle Grid Project.

8Copyright 2009 MySQL AB The World’s Most Popular Open Source Database

What can you do?

• Buy a server with 64 GB or 128 GB of ram?– Simple– Costly

• Shard the database to the other 2 servers?– Complex – Application rewrite for sharding needed

• Find a way to use the memory of the old servers?– How?

Page 9: Memcached + Innodb = Performance. The Waffle Grid Project.

9Copyright 2009 MySQL AB The World’s Most Popular Open Source Database

Presenting Waffle Grid!

• What is Waffle Grid?– Waffle Grid is a distributed level 2 cache for Innodb

based on memcached

Page 10: Memcached + Innodb = Performance. The Waffle Grid Project.

10Copyright 2009 MySQL AB The World’s Most Popular Open Source Database

What does that mean?

It means you can dramatically increase the size of the Innodb buffer pool by using the memory in other remote servers!

Page 11: Memcached + Innodb = Performance. The Waffle Grid Project.

11Copyright 2009 MySQL AB The World’s Most Popular Open Source Database

How does it work?

• When the buffer pool is full, InnoDB needs to free the LRU page• The LRU page is valid, there is just no enough memory to hold it• Why just freeing it... send it to memcached• Before reading a page on disk, ask memcached first and if

lucky, save a disk IO operation

Page 12: Memcached + Innodb = Performance. The Waffle Grid Project.

12Copyright 2009 MySQL AB The World’s Most Popular Open Source Database

How does it work

Page 13: Memcached + Innodb = Performance. The Waffle Grid Project.

13Copyright 2009 MySQL AB The World’s Most Popular Open Source Database

How does it work (cont.)

Page 14: Memcached + Innodb = Performance. The Waffle Grid Project.

14Copyright 2009 MySQL AB The World’s Most Popular Open Source Database

Optimal Configuration

• Network speed! 100Mb/s is too slow• Network speed! 1000Mb/s is ok• Network speed! Faster then 1000Mb/s is best• Waffle is going to work best if you give it a dedicated private

connections between servers specifically for memcached access

Page 15: Memcached + Innodb = Performance. The Waffle Grid Project.

15Copyright 2009 MySQL AB The World’s Most Popular Open Source Database

Disclaimers

• I feel like the drug company putting out those disclaimers...• Waffle Grid will NOT! Directly boost write performance on your

disk, by reading less it can free up resources that may enable more writes...

• If your right constrained already, your mileage may very... • When testing out the dolphin interconnects we hit a write

constraint, the faster interconnects could not help because of it. • Waffle Grid is not going to help overcome CPU issues

bottlenecks

Page 16: Memcached + Innodb = Performance. The Waffle Grid Project.

16Copyright 2009 MySQL AB The World’s Most Popular Open Source Database

Special Modified Memcached

• MRU -vs- LRU– Memcached also has a LRU logic– The problem is, memcached is a level 2 cache– Retrieving a block from memcached invalidates the block,

the good version is now in the buffer pool– A memcached GET must put the block at the top of the LRU

list (MRU logic)‏

– Patch available with WaffleGrid

Page 17: Memcached + Innodb = Performance. The Waffle Grid Project.

17Copyright 2009 MySQL AB The World’s Most Popular Open Source Database

Benchmarks

• Tools– DBT2 – TPC like benchmark– Sysbench generic testing tool which includes OLTP tests

Page 18: Memcached + Innodb = Performance. The Waffle Grid Project.

18Copyright 2009 MySQL AB The World’s Most Popular Open Source Database

Benchmarks

Page 19: Memcached + Innodb = Performance. The Waffle Grid Project.

19Copyright 2009 MySQL AB The World’s Most Popular Open Source Database

Benchmarks

Page 20: Memcached + Innodb = Performance. The Waffle Grid Project.

20Copyright 2009 MySQL AB The World’s Most Popular Open Source Database

Benchmarks – Interconnect speed

Page 21: Memcached + Innodb = Performance. The Waffle Grid Project.

21Copyright 2009 MySQL AB The World’s Most Popular Open Source Database

Benchmarks – faster disk?

Page 22: Memcached + Innodb = Performance. The Waffle Grid Project.

22Copyright 2009 MySQL AB The World’s Most Popular Open Source Database

Dolphin Interconnect Latency

Page 23: Memcached + Innodb = Performance. The Waffle Grid Project.

23Copyright 2009 MySQL AB The World’s Most Popular Open Source Database

Dolphin Interconnect

500K trans 500K Trans 1M Trans0

200

400

600

800

1000

1200

1400

1600

503

14411498

502 1437 1497

504

14431502

464 898 783

463

899773

461 902 778

1gbe -vs- Dolphin Interconnects

Sysbench ReadOnly Test

Dolphin #1 Dolphin #2 Dolphin #3 1gbe #1 1gbe #2 1gbe #3

Page 24: Memcached + Innodb = Performance. The Waffle Grid Project.

24Copyright 2009 MySQL AB The World’s Most Popular Open Source Database

EC2 Unique Challenges

• Network is shared on EC2• Boxes are shared• Boxes tend to be underpowered• Shared environment means slowdowns are a fact of life• Performance is similar to a 100Mb/s connection

Page 25: Memcached + Innodb = Performance. The Waffle Grid Project.

25Copyright 2009 MySQL AB The World’s Most Popular Open Source Database

Waffle Grid on EC2

Page 26: Memcached + Innodb = Performance. The Waffle Grid Project.

26Copyright 2009 MySQL AB The World’s Most Popular Open Source Database

Waffle Grid Everywhere!

• Waffle Gird can be deployed in many situations. – Those looking for the fastest possible performance will

benefit from the ability to heavy lift huge datasets into memory

– Those who are on a budget can reuse hardware and cobble together a Waffle Grid with many nodes boosting performance without incurring additional costs

– Waffle Grid can be deployed in the cloud, enabling cloud computing users to boost performance on demand.

Page 27: Memcached + Innodb = Performance. The Waffle Grid Project.

27Copyright 2009 MySQL AB The World’s Most Popular Open Source Database

Status and roadmap

• Cache coherence between startup– Coded and partially tested, added a prefix to the memcached

key• Dynamic memcached server list

– Almost there– Allow to add more memory on the fly– Will invalidate current memcached cached pages

• Better error handling– Partly done

Page 28: Memcached + Innodb = Performance. The Waffle Grid Project.

28Copyright 2009 MySQL AB The World’s Most Popular Open Source Database

Status and roadmap (2)

• Move from patches to a full branch– First port, the InnoDB Plugin– On launchpad: lp:~y-trudeau/wafflegrid/waffleGrid-Innodb-

plugin• Drizzle/Innodb Plugin port

– Should be straightforward from the InnoDB Plugin• Other engines?

– PBXT, Falcon• BETTER PERFORMANCE!

– EC2, High Latency environments, better hits rates

Page 29: Memcached + Innodb = Performance. The Waffle Grid Project.

29Copyright 2009 MySQL AB The World’s Most Popular Open Source Database

Waffle Grid quick start

• Grab libmemcached-wafflegrid, compile and install – Branch lp:~yonkovim/wafflegrid/libmemcached

• Grab a recent MySQL source tree– http://dev.mysql.com/downloads/mysql/5.1.html#source

• Grab Waffle Grid for the InnoDB plugin– bzr branch lp:~y-trudeau/wafflegrid/waffleGrid-Innodb-plugin– Need bzr version 1.6 or higher

Page 30: Memcached + Innodb = Performance. The Waffle Grid Project.

30Copyright 2009 MySQL AB The World’s Most Popular Open Source Database

Waffle Grid quick start (2)

• Replace the directory storage/innobase by the Waffle Grid code• Grab the compile script

– http://bazaar.launchpad.net/%7Ey-trudeau/wafflegrid/main/annotate/head%3A/BUILD-scripts/compile-pentium-max-memc

• Compile and install MySQL• Grab memcached version for Waffle Grid

– branch lp:~yonkovim/wafflegrid/memcached

Page 31: Memcached + Innodb = Performance. The Waffle Grid Project.

31Copyright 2009 MySQL AB The World’s Most Popular Open Source Database

Waffle Grid quick start (3)

• Compile and install memcached• Start memcached

– From a non-root account: “memcached -d”• Start MySQL• From mysql client shell

– set global innodb_memcached_enable = 1;– show innodb status\G will keep give you stats about Waffle

Grid puts/hits/misses and timing

Page 32: Memcached + Innodb = Performance. The Waffle Grid Project.

32Copyright 2009 MySQL AB The World’s Most Popular Open Source Database

Get Involved?

• http://www.wafflegrid.com/• https://code.launchpad.net/wafflegrid• Contributions are welcome

Page 33: Memcached + Innodb = Performance. The Waffle Grid Project.

33Copyright 2009 MySQL AB The World’s Most Popular Open Source Database

Questions ?