Revolutionizingenterprise web

development

Memcache Optimization

Agenda

• What is memcache and what components do you need?

• Why use it and how do you know it’s working?

• Limitations and configuration options

• Basic troubleshooting

• Common configuration strategies

• Tips

What is memcache?

• “Memcached is an in-memory key-value store for small chunks of arbitrary data”

• A system to cache data in RAM rather than in a database.

• Replaces Drupal’s default database cache.

Who Uses It?

• Facebook

• YouTube

• Craigslist

• Twitter

• Flicr

• LiveJournal

• Wikipedia

• Bebo

• Wordpress.com

• Digg

• Typepad

• Yellowbot

• Many others…

Sources: http://memcached.org and http://lineofthought.com/tools/memcached

How does memcache.d fit into an overall performance architecture?

• Only 1 component of the architecture

• Compliments other tools such as Varnish and APC.

Why use memcache?

• Helps improve the scalability of your site.

• Reduces request execution time on the web server.

• Reduces database load.

• Allows more distribution of resources and processing.

• In some use cases, can improve page response time*

* This is very application specific, for most sites the improvement will be so small that most users will not be able to tell the difference. (e.g. 0.1 second improvement on stock Drupal install)

Stats

• Performance and Scalability

• Cache read performance: 2x faster

• Cache write performance: 40x faster

• Reduces number of database queries by up to 50%

• Actual statistics* (Drupal core vs. Memcache)

• Cache Read Operations: 3.5 sec vs. 1.5 sec

• Cache Write Operations: 100 sec vs 2.5 sec

• DB Queries: 70 vs. 33.* Cache Read/Write stats are for performing 10,000 operations

** Your results will vary, tests performed on a single dev VM with default D7 install.

Components of a memcache system

• Memcache.d Daemon (v 1.4.x)

• PHP Extension

• PECL Memcache (> v 2.2.1)

• Use version 3.0.5 or later• PECL Memcache.d (> 2.x)

• TIP: Use the setting “memcache_extension” to select which library to use if you have both installed.

• Memcache Module

• Drupal 7.9+, Pressflow 6, or Drupal 6 with core patch

PECL Memcache vs. Memcached

PECL Memcache PECL MemcachedStable, not as frequently updated

Newer, more features

Easy to install More complicated install with multiple dependenciesSlightly faster in some casesigbinary support

What is igbinary?

• Alternate to built-in php serialize/unserialize functions.

• Higher performance and lower memory utilization in many cases.

• Binary serialization rather than string.

• Requires special php.ini configuration to fully take advantage of. Requires special calls to use.

• Resources: • http://codepoets.co.uk/2011/php-serialization-igbinary/

• https://github.com/igbinary/igbinary/

• http://drupal.org/node/839444

Installing Memcache.d

• On Debian based systems:

• Sudo apt-get install memcached

• Sudo apt-get install php-pear

• For PECL Memcache

• Sudo pecl install memcache-beta• For PECL Memcached

• Sudo apt-get install libmemcached libmemcached-dev pcre3 pcre3-dev

• (optional) Sudo pecl install igbinary • Be sure to do the config steps!

• Sudo pecl install memcached

How do you know it’s working?

• Memcache Statistics (Memcache Admin Module)

• Enable “Show memcache statistics at bottom of each page”

• You should see at least 10 – 20 cache operations listed on several bins including: “cache”, “cache_menu”, and “cache_bootstrap”.

• Refresh the page a couple times, you should see a hit number greater than 0 on almost every line and almost no cache set operations.

How do you know it’s working?

• Devel Module

• Enable the option “Display query log” sorted by source.

• Should see no queries hitting the main Drupal cache tables (e.g. “cache”, “cache_bootstrap”, “cache_menu”, etc.)

Diagnostic/Troubleshooting Techniques

• Use Telnet to directly query cache instances

• Useful commands:

• Stats – displays general stats• Stats slabs – displays per slab stats• Stats items – displays per item stats• Flush_all – Deletes everything in the cache• Get / set – Gets or sets a cache item

• Use the memcache stats on each page and Memcache Stats Report to see what’s happening

• Most of the same info as via telnet.

• TIP: You may get an error from the report if you don’t have at least one memcache bin defined.

Important Memcache Stats

• Cache Hit/Miss Ratio > 99% hits ideal

• Connections < 1024 (less than configured limit)

• Evictions == 0

• Free/Used Memory > 20% free memory

Sizing Memcache.d

• Look at the cache_* tables in your database.

• Sum the total size of these tables (e.g. in phpmyadmin)

• Multiply by at least 2 and round up to the nearest multiple of 8MB.

• TIP: Try to keep instance size under 512MB.

• Example:

• Cache table size=30MB, x2 = 60MB.

• Round up to 64MB.

Settings.php Options:

• cache_backends

• cache_default_class

• cache_class_form (cache_class_<bin name>)

• Drupal 7 only• session_inc (D6 only for now)

• memcache_servers

• memcache_bins

• memcache_key_prefix

• memcache_options

Configuration options

Configuration options

Recommended Memcached Options (in settings.php):$conf['memcache_options'] = array(

Memcached::OPT_COMPRESSION => FALSE,

Memcached::OPT_DISTRIBUTION => Memcached::DISTRIBUTION_CONSISTENT,

Memcached::OPT_BINARY_PROTOCOL => TRUE,

);

For a single memcache.d instance, this option is slightly faster:

Memcached::OPT_DISTRIBUTION => Memcached::DISTRIBUTION_MODULA,

NOTE: These settings are only valid if using PECL Memcached

Limitations

• Max key size: 250 bytes

• Can use larger keys, but pay a hashing penalty

• Max data size: 1 MB

• Has no redundancy or locking

* Most of these limitations can be overcome by custom compiling the Memcache.d libraries or adding other extensions to Memcache.d.

Common configuration strategies

3 Common Strategies:

• Basic configuration

• Basic distributed configuration

• Multiple bins distributed configuration

Basic Configuration

Good for:

• Sites that run on 1-2 web servers.

• Mostly read-only content or infrequently changing content.

$conf['cache_backends'][] = 'sites/all/modules/memcache/memcache.inc';

$conf['cache_default_class'] = 'MemCacheDrupal';

$conf['cache_class_form'] = 'DrupalDatabaseCache';

$conf['memcache_servers'] = array(‘mysql.localdomain:11211' => 'default',);

$conf['memcache_key_prefix'] = ‘mysitename';

$conf['memcache_options'] = array(

Memcached::OPT_COMPRESSION => FALSE,

Memcached::OPT_DISTRIBUTION => Memcached::DISTRIBUTION_MODULA,

Memcached::OPT_BINARY_PROTOCOL => TRUE,

);

Basic Configuration:settings.php

Simple distributed configuration

Good for:

• Sites running on 2 or more web servers.

• Sites that are a mix of read-only and regularly changing content, but mostly read-only.

$conf['cache_backends'][] = 'sites/all/modules/memcache/memcache.inc';

$conf['cache_default_class'] = 'MemCacheDrupal';

$conf['cache_class_form'] = 'DrupalDatabaseCache';

$conf['memcache_servers'] = array(

‘web1.localdomain:11211' => 'default',

‘web2.localdomain:11211’ => ‘default’,

);

$conf['memcache_key_prefix'] = ‘mysitename';

$conf['memcache_options'] = array(

Memcached::OPT_COMPRESSION => FALSE,

Memcached::OPT_DISTRIBUTION => Memcached::DISTRIBUTION_CONSISTENT,

Memcached::OPT_BINARY_PROTOCOL => TRUE,

);

Simple distributed configuration:settings.php

Dedicated bins distributed configuration

Good for:

• Site running on 3 or more web servers.

• Frequently added or changing content.

• Large amounts of cached content.

// Same settings as previous configs

$conf['memcache_servers'] = array(

‘memcache1.localdomain:11211' => 'default',

‘memcache2.localdomain:11211’ => ‘default’,

‘memcache3.localdomain:11211' => ‘static',

);

$conf[‘memcache_bins’] = array(

‘cache’ => ‘default’,

‘cache_bootstrap’ => ‘static’,

‘cache_menu’ => ‘static’,

‘cache_field’ => ‘static’,

‘cache_filter’ => ‘static’,

‘cache_path’ => ‘static’,

);

// Same settings as previous configs

Dedicated bins distributed configuration:settings.php

Other Tips

• Do not set “Minimum cache lifetime”.

• Do not set expiration more than 30 days.

• You do not need to enable the Memcache module or enable Drupal caching to use Memcache.

• When using multiple bins, make sure you explicitly define at least one cache table to each bin to avoid errors on the stats report.

• When splitting out multiple servers/bins, move things out in this order:

• Mostly read-only content (menus, filters, paths)

• Page cache

• High traffic cache tables (session, cache)

References

• Memcache.d: http://memcached.org/

• PECL Memcache: http://pecl.php.net/package/memcache

• PECL Memcached:http://pecl.php.net/package/memcached

• Drupal Module: http://drupal.org/project/memcache

• Memcached.php: http://www.phpdeveloper.org/news/10247

References

• PECL Memcache vs. Memcached:

• http://code.google.com/p/memcached/wiki/PHPClientComparison

• http://brian.moonspot.net/php-memcached-issues

• Alternate Memcache.d Init script: http://www.bxtra.net/articles/2011-07-27/memcached-multiple-instances-startup-script-initd-on-centos-56-64-bits

Thank You

Shawn Smiley, Lead Engineerd.o: Shawn_Smileye: shawn.smiley@achieveinternet.com