Revolutionizing enterprise web development
description
Transcript of Revolutionizing enterprise web development
![Page 1: Revolutionizing enterprise web development](https://reader036.fdocuments.in/reader036/viewer/2022081505/56815e60550346895dccdb10/html5/thumbnails/1.jpg)
Revolutionizingenterprise web
development
Memcache Optimization
![Page 2: Revolutionizing enterprise web development](https://reader036.fdocuments.in/reader036/viewer/2022081505/56815e60550346895dccdb10/html5/thumbnails/2.jpg)
Agenda
• What is memcache and what components do you need?
• Why use it and how do you know it’s working?• Limitations and configuration options• Basic troubleshooting• Common configuration strategies• Tips
![Page 3: Revolutionizing enterprise web development](https://reader036.fdocuments.in/reader036/viewer/2022081505/56815e60550346895dccdb10/html5/thumbnails/3.jpg)
What is memcache?
• “Memcached is an in-memory key-value store for small chunks of arbitrary data”
• A system to cache data in RAM rather than in a database.
• Replaces Drupal’s default database cache.
![Page 4: Revolutionizing enterprise web development](https://reader036.fdocuments.in/reader036/viewer/2022081505/56815e60550346895dccdb10/html5/thumbnails/4.jpg)
Who Uses It?
• Facebook• YouTube• Craigslist• Twitter• Flicr• LiveJournal• Wikipedia
• Bebo• Wordpress.com• Digg• Typepad• Yellowbot• Many others…
Sources: http://memcached.org and http://lineofthought.com/tools/memcached
![Page 5: Revolutionizing enterprise web development](https://reader036.fdocuments.in/reader036/viewer/2022081505/56815e60550346895dccdb10/html5/thumbnails/5.jpg)
How does memcache.d fit into an overall performance architecture?
• Only 1 component of the architecture
• Compliments other tools such as Varnish and APC.
![Page 6: Revolutionizing enterprise web development](https://reader036.fdocuments.in/reader036/viewer/2022081505/56815e60550346895dccdb10/html5/thumbnails/6.jpg)
Why use memcache?
• Helps improve the scalability of your site.• Reduces request execution time on the web
server.• Reduces database load.• Allows more distribution of resources and
processing.• In some use cases, can improve page response
time*
* This is very application specific, for most sites the improvement will be so small that most users will not be able to tell the difference. (e.g. 0.1 second improvement on stock Drupal install)
![Page 7: Revolutionizing enterprise web development](https://reader036.fdocuments.in/reader036/viewer/2022081505/56815e60550346895dccdb10/html5/thumbnails/7.jpg)
Stats
• Performance and Scalability• Cache read performance: 2x faster• Cache write performance: 40x faster• Reduces number of database queries by up to
50%• Actual statistics* (Drupal core vs. Memcache)
• Cache Read Operations: 3.5 sec vs. 1.5 sec• Cache Write Operations: 100 sec vs 2.5 sec• DB Queries: 70 vs. 33.
* Cache Read/Write stats are for performing 10,000 operations
** Your results will vary, tests performed on a single dev VM with default D7 install.
![Page 8: Revolutionizing enterprise web development](https://reader036.fdocuments.in/reader036/viewer/2022081505/56815e60550346895dccdb10/html5/thumbnails/8.jpg)
Components of a memcache system
• Memcache.d Daemon (v 1.4.x)• PHP Extension
• PECL Memcache (> v 2.2.1) • Use version 3.0.5 or later
• PECL Memcache.d (> 2.x)• TIP: Use the setting “memcache_extension” to
select which library to use if you have both installed.
• Memcache Module• Drupal 7.9+, Pressflow 6, or Drupal 6 with core
patch
![Page 9: Revolutionizing enterprise web development](https://reader036.fdocuments.in/reader036/viewer/2022081505/56815e60550346895dccdb10/html5/thumbnails/9.jpg)
PECL Memcache vs. Memcached
PECL Memcache PECL MemcachedStable, not as frequently updated
Newer, more features
Easy to install More complicated install with multiple dependenciesSlightly faster in some casesigbinary support
![Page 10: Revolutionizing enterprise web development](https://reader036.fdocuments.in/reader036/viewer/2022081505/56815e60550346895dccdb10/html5/thumbnails/10.jpg)
What is igbinary?
• Alternate to built-in php serialize/unserialize functions.
• Higher performance and lower memory utilization in many cases.
• Binary serialization rather than string.• Requires special php.ini configuration to fully take
advantage of. Requires special calls to use.• Resources:
• http://codepoets.co.uk/2011/php-serialization-igbinary/• https://github.com/igbinary/igbinary/• http://drupal.org/node/839444
![Page 11: Revolutionizing enterprise web development](https://reader036.fdocuments.in/reader036/viewer/2022081505/56815e60550346895dccdb10/html5/thumbnails/11.jpg)
Installing Memcache.d
• On Debian based systems:• Sudo apt-get install memcached• Sudo apt-get install php-pear• For PECL Memcache
• Sudo pecl install memcache-beta• For PECL Memcached
• Sudo apt-get install libmemcached libmemcached-dev pcre3 pcre3-dev
• (optional) Sudo pecl install igbinary • Be sure to do the config steps!
• Sudo pecl install memcached
![Page 12: Revolutionizing enterprise web development](https://reader036.fdocuments.in/reader036/viewer/2022081505/56815e60550346895dccdb10/html5/thumbnails/12.jpg)
How do you know it’s working?
• Memcache Statistics (Memcache Admin Module)• Enable “Show memcache statistics at bottom of
each page”• You should see at least 10 – 20 cache operations
listed on several bins including: “cache”, “cache_menu”, and “cache_bootstrap”.
• Refresh the page a couple times, you should see a hit number greater than 0 on almost every line and almost no cache set operations.
![Page 13: Revolutionizing enterprise web development](https://reader036.fdocuments.in/reader036/viewer/2022081505/56815e60550346895dccdb10/html5/thumbnails/13.jpg)
How do you know it’s working?
• Devel Module• Enable the option “Display query log” sorted by
source.• Should see no queries hitting the main Drupal cache tables (e.g. “cache”, “cache_bootstrap”, “cache_menu”, etc.)
![Page 14: Revolutionizing enterprise web development](https://reader036.fdocuments.in/reader036/viewer/2022081505/56815e60550346895dccdb10/html5/thumbnails/14.jpg)
Diagnostic/Troubleshooting Techniques
• Use Telnet to directly query cache instances• Useful commands:
• Stats – displays general stats• Stats slabs – displays per slab stats• Stats items – displays per item stats• Flush_all – Deletes everything in the cache• Get / set – Gets or sets a cache item
• Use the memcache stats on each page and Memcache Stats Report to see what’s happening• Most of the same info as via telnet.• TIP: You may get an error from the report if you don’t
have at least one memcache bin defined.
![Page 15: Revolutionizing enterprise web development](https://reader036.fdocuments.in/reader036/viewer/2022081505/56815e60550346895dccdb10/html5/thumbnails/15.jpg)
Important Memcache Stats
• Cache Hit/Miss Ratio > 99% hits ideal• Connections < 1024 (less than configured limit)• Evictions == 0• Free/Used Memory > 20% free memory
![Page 16: Revolutionizing enterprise web development](https://reader036.fdocuments.in/reader036/viewer/2022081505/56815e60550346895dccdb10/html5/thumbnails/16.jpg)
Sizing Memcache.d
• Look at the cache_* tables in your database.• Sum the total size of these tables (e.g. in
phpmyadmin)• Multiply by at least 2 and round up to the nearest
multiple of 8MB.• TIP: Try to keep instance size under 512MB.• Example:
• Cache table size=30MB, x2 = 60MB.• Round up to 64MB.
![Page 17: Revolutionizing enterprise web development](https://reader036.fdocuments.in/reader036/viewer/2022081505/56815e60550346895dccdb10/html5/thumbnails/17.jpg)
Settings.php Options:• cache_backends• cache_default_class• cache_class_form (cache_class_<bin name>)
• Drupal 7 only• session_inc (D6 only for now)• memcache_servers• memcache_bins• memcache_key_prefix• memcache_options
Configuration options
![Page 18: Revolutionizing enterprise web development](https://reader036.fdocuments.in/reader036/viewer/2022081505/56815e60550346895dccdb10/html5/thumbnails/18.jpg)
Configuration options
Recommended Memcached Options (in settings.php):$conf['memcache_options'] = array(
Memcached::OPT_COMPRESSION => FALSE,
Memcached::OPT_DISTRIBUTION => Memcached::DISTRIBUTION_CONSISTENT,
Memcached::OPT_BINARY_PROTOCOL => TRUE,
);
For a single memcache.d instance, this option is slightly faster:Memcached::OPT_DISTRIBUTION => Memcached::DISTRIBUTION_MODULA,
NOTE: These settings are only valid if using PECL Memcached
![Page 19: Revolutionizing enterprise web development](https://reader036.fdocuments.in/reader036/viewer/2022081505/56815e60550346895dccdb10/html5/thumbnails/19.jpg)
Limitations
• Max key size: 250 bytes• Can use larger keys, but pay a hashing penalty
• Max data size: 1 MB• Has no redundancy or locking
* Most of these limitations can be overcome by custom compiling the Memcache.d libraries or adding other extensions to Memcache.d.
![Page 20: Revolutionizing enterprise web development](https://reader036.fdocuments.in/reader036/viewer/2022081505/56815e60550346895dccdb10/html5/thumbnails/20.jpg)
Common configuration strategies
3 Common Strategies:• Basic configuration• Basic distributed configuration• Multiple bins distributed configuration
![Page 21: Revolutionizing enterprise web development](https://reader036.fdocuments.in/reader036/viewer/2022081505/56815e60550346895dccdb10/html5/thumbnails/21.jpg)
Basic Configuration
Good for:• Sites that run on 1-2 web servers.• Mostly read-only content or
infrequently changing content.
![Page 22: Revolutionizing enterprise web development](https://reader036.fdocuments.in/reader036/viewer/2022081505/56815e60550346895dccdb10/html5/thumbnails/22.jpg)
$conf['cache_backends'][] = 'sites/all/modules/memcache/memcache.inc';
$conf['cache_default_class'] = 'MemCacheDrupal';
$conf['cache_class_form'] = 'DrupalDatabaseCache';
$conf['memcache_servers'] = array(‘mysql.localdomain:11211' => 'default',);$conf['memcache_key_prefix'] = ‘mysitename';
$conf['memcache_options'] = array(
Memcached::OPT_COMPRESSION => FALSE,
Memcached::OPT_DISTRIBUTION => Memcached::DISTRIBUTION_MODULA, Memcached::OPT_BINARY_PROTOCOL => TRUE,
);
Basic Configuration:settings.php
![Page 23: Revolutionizing enterprise web development](https://reader036.fdocuments.in/reader036/viewer/2022081505/56815e60550346895dccdb10/html5/thumbnails/23.jpg)
Simple distributed configuration
Good for:• Sites running on 2 or more web
servers.• Sites that are a mix of read-only
and regularly changing content, but mostly read-only.
![Page 24: Revolutionizing enterprise web development](https://reader036.fdocuments.in/reader036/viewer/2022081505/56815e60550346895dccdb10/html5/thumbnails/24.jpg)
$conf['cache_backends'][] = 'sites/all/modules/memcache/memcache.inc';
$conf['cache_default_class'] = 'MemCacheDrupal';
$conf['cache_class_form'] = 'DrupalDatabaseCache';
$conf['memcache_servers'] = array(
‘web1.localdomain:11211' => 'default',
‘web2.localdomain:11211’ => ‘default’,
);
$conf['memcache_key_prefix'] = ‘mysitename';
$conf['memcache_options'] = array(
Memcached::OPT_COMPRESSION => FALSE,
Memcached::OPT_DISTRIBUTION => Memcached::DISTRIBUTION_CONSISTENT,
Memcached::OPT_BINARY_PROTOCOL => TRUE,
);
Simple distributed configuration:settings.php
![Page 25: Revolutionizing enterprise web development](https://reader036.fdocuments.in/reader036/viewer/2022081505/56815e60550346895dccdb10/html5/thumbnails/25.jpg)
Dedicated bins distributed configuration
Good for:• Site running on 3 or
more web servers.• Frequently added or
changing content.• Large amounts of
cached content.
![Page 26: Revolutionizing enterprise web development](https://reader036.fdocuments.in/reader036/viewer/2022081505/56815e60550346895dccdb10/html5/thumbnails/26.jpg)
// Same settings as previous configs
$conf['memcache_servers'] = array(
‘memcache1.localdomain:11211' => 'default',
‘memcache2.localdomain:11211’ => ‘default’,
‘memcache3.localdomain:11211' => ‘static',
);
$conf[‘memcache_bins’] = array(
‘cache’ => ‘default’,
‘cache_bootstrap’ => ‘static’,
‘cache_menu’ => ‘static’,
‘cache_field’ => ‘static’,
‘cache_filter’ => ‘static’,
‘cache_path’ => ‘static’,
);
// Same settings as previous configs
Dedicated bins distributed configuration:settings.php
![Page 27: Revolutionizing enterprise web development](https://reader036.fdocuments.in/reader036/viewer/2022081505/56815e60550346895dccdb10/html5/thumbnails/27.jpg)
Other Tips• Do not set “Minimum cache lifetime”.• Do not set expiration more than 30 days.• You do not need to enable the Memcache module or
enable Drupal caching to use Memcache.• When using multiple bins, make sure you explicitly define
at least one cache table to each bin to avoid errors on the stats report.
• When splitting out multiple servers/bins, move things out in this order:• Mostly read-only content (menus, filters, paths)• Page cache• High traffic cache tables (session, cache)
![Page 28: Revolutionizing enterprise web development](https://reader036.fdocuments.in/reader036/viewer/2022081505/56815e60550346895dccdb10/html5/thumbnails/28.jpg)
References• Memcache.d: http://memcached.org/• PECL Memcache:
http://pecl.php.net/package/memcache• PECL Memcached:
http://pecl.php.net/package/memcached• Drupal Module:
http://drupal.org/project/memcache• Memcached.php:
http://www.phpdeveloper.org/news/10247
![Page 29: Revolutionizing enterprise web development](https://reader036.fdocuments.in/reader036/viewer/2022081505/56815e60550346895dccdb10/html5/thumbnails/29.jpg)
References
• PECL Memcache vs. Memcached:• http://code.google.com/p/memcached/wiki/
PHPClientComparison• http://brian.moonspot.net/php-memcached-
issues• Alternate Memcache.d Init script:
http://www.bxtra.net/articles/2011-07-27/memcached-multiple-instances-startup-script-initd-on-centos-56-64-bits
![Page 30: Revolutionizing enterprise web development](https://reader036.fdocuments.in/reader036/viewer/2022081505/56815e60550346895dccdb10/html5/thumbnails/30.jpg)
Thank YouShawn Smiley, Lead Engineerd.o: Shawn_Smileye: [email protected]