Magento Performance Testing for High Performance

MAGENTO HOSTINGBest Practices for Optimum Performance v1.6

Copyright 2000 2013 Nexcess.net LLC. All Rights Reserved.This work is licensed under aCreative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

April 2013

Brad Boegler [email protected]

Magento Hosting Best Practices for Optimum Performance


2

Table Of Contents

Introduction 3Testing Criteria 4

Terms and Definitions 4Benchmarking Tools 5Software Tools & Plug-ins 5Benchmarking Parameters 6

Benchmarking Hardware Architecture 7Software and Benchmarking Configurations 9

Software Versions 9Anatomy of a Siege Test Raw Stress Testing 9Anatomy of a Gatling Test User Modeling Stress Testing 9

The Test Cases 11Test 1: File Synchronization NFS vs. Local File System 11Test 1: Results 11Test 2: Apache vs. Nginx 12Test 2: Results 13Test 3: MySQL versus Percona 15Test 3: Results 15Optimized Magento Site versus Optimized Magento Site with Varnish & Turpentine 17Test 4: Best configuration versus Varnish & Turpentine 17

Result Observations and Optimal Configurations 20The Importance of the Magento Enterprise Full Page Cache 20The Magento DEFAULT_LIFETIME 21Response Times 23

Understanding Magento Caching 25Magento Backend Cache 25Additional improvements 26Magento Full Page Cache 26Magento Session Cache 26PHP opcode cache 26

Conclusion and Additional Performance Recommendations 27Appendix A: System Configurations 28Appendix B: Benchmarking Configurations 28

Gatling Tool 28Siege 28



3

IntroductionMagento is a very popular and powerful eCommerce platform with very demanding resource requirements. While it will run out of the box using default configuration settings, extensive post-installation configuration is recommended to achieve optimal site performance (especially for high volume sites).

This white paper is meant as a guide for you to ensure youre getting the most out of your Magento installation by using best practices developed through real-world experience at Nexcess, where weve been working with the platform since 2007. We outline the differences between default and/or popular available configurations, listing the benefits and drawbacks of these changes, and what weve found to be optimal for scaling Magento. These different configurations are extensively benchmarked showing the benefits between configuration op-tions and the maximum performance available from each chosen setting.

Using the configuration information contained within this white paper, a load-balanced web server cluster run-ning Magento Enterprise is able to process over 550 transactions per second (equating to more than 47 million transactions per day). Further performance is obtained by utilizing the Nexcess Turpentine plug-in (with the Varnish reverse proxy), which can provide up to 6,940 transactions per second (over 600 million transactions per day).

All system-level configuration files are provided at the end of this document and show the exact system settings and application configuration parameters used. All benchmark application test configurations are also provided to show exactly how each test was run and how, if necessary, each can be reproduced.

We welcome your input and feedback. Any questions or comments may be directed to [email protected].



4

Testing CriteriaTerms and Definitions

The following terminology can at times be confusing in relation to the actual process occurring. The terms used throughout this white paper are defined as follows:

BuyerA client who completes an order.

ClientA user and (more specifically) the browser and/or browser mechanism used by the user. It does not, for the pur-poses of this paper, always denote the customer of an eCommerce transaction.

OrderThe successful completion of the checkout process within Magento.

Orders per second (also denoted as O/s or Orders/sec)The number of orders processed during one second.

RequestAny HTTP communication sent by a client (browser, etc.) to a Magento server. This includes requests sent over HTTPS and is synonymous with an HTTP request.

Requests per second (also denoted as R/s or Requests/sec)The amount of requests being sent to a Magento server during one second. This metric does not take into ac-count how many requests result in a successful response from the server (that is, a complete transaction).

ResponseAny HTTP communication sent by a Magento server to a client. This includes those requests sent over HTTPS and is synonymous with an HTTP response. Bear in mind, responses are not synonymous with a web page. A single web page may have hundreds of individual responses associated with it, including images, HTML pages, style sheets, JavaScript and other static or dynamic content.

TransactionThe combination of a successful request and response.

Transactions per second (also denoted as T/s or Transactions/sec)The number of transactions completed during one second.

UserSynonymous with a client (for the purposes of this paper).

VisitorA client who simply browses a website and does not complete an order.



5

Benchmarking Tools

Two benchmarking tools, Siege and the Gatling Stress Tool, were used extensively in testing.

Siege is an open-source, HTTP-based benchmarking utility which excels in obtaining the raw load capabilities of a website. It can stress a web address (or a list of web addresses) with various concurrency levels. It is an ideal load tester for determining the peak transactions per second for an individual client and the specific response times experienced.

The Gatling Stress Tool is also an open-source tool and is used to benchmark interactive features of a site. It can be configured to perform complex multi-step actions such as adding items to a cart and performing a checkout in the same manner a human client would. Because of this, Gatling is excellent for testing the number of orders per second a site is capable of processing under various load situations.

Software Tools & Plug-ins

The following software and other tools are mentioned throughout this white paper and are important enough to the performance of Magento to merit a brief explanation:

Alternative PHP Cache (APC)

An opcode cache for PHP. When PHP processes a script it has to digest the PHP code into an intermediate format that can be run by the PHP interpreter. Much of the work done on this digestion can be re-used in future exe-cution of the script by saving some metadata about the script in the form of opcodes. This can speed up each PHP scripts processing time by saving some time associated with compilation. APC can also be used as a mem-ory-based caching system to cache key/value pairs in memory much like memcached (see description below), though it suffers from the same deficiencies.

memcached

A memory-based caching system. By using a simple key/value system to store data in a table in memory, access to the data in memcached is very fast. However, a disadvantage to its implementation is no targeting is used when flushing the cache. A cache flush is an all-or-nothing operation which can be non-optimal if the bulk of the cache is not dirty. memcached uses a threaded model, allowing it to process requests in parallel efficiently.

PHP FastCGI Process Manager (PHP-FPM)

This allows PHP processes to be pooled and reserved on a per-user basis. Traditional execution of the PHP inter-preter involved starting up an entirely new process to handle a given request via a mechanism like CGI. PHP-FPM allows for having many PHP processes running, which greatly enhances performance. Beyond this, PHP-FPM has the added security benefit of allowing the PHP processes themselves to be assigned to a given user (which allows user segmentation not available when using traditional Apache+mod_php).



6

Redis

A memory-based caching system. It uses a simple key/value system to store data in a table in memory (much like memcached). Redis adds the advantage of tags to the storage model, which allows an application using Redis to target any dirty sections of the cache during flush operations. Its current implementation is single-threaded, meaning it can only process requests serially (that is, one at a time) for each instance of Redis.

Varnish

A web application accelerator (also known as a caching HTTP reverse proxy). Varnish can be used to drastically speed up web applications with minimal performance degradation when configured correctly. Varnish stores the assembled web pages (or parts of web pages) in memory and relieves the application itself from having to perform any work when a cache hit is realized. The Nexcess Turpentine Magento extension makes use of Varnish to greatly enhance the performance of Magento Enterprise. By default, Varnish does not work well with Magen-to because it doesnt cache requests with cookies. Magento sends a frontend cookie with every request caus-ing a (near) zero hit-rate for Varnishs cache. Turpentine configures Varnish to work with Magento and modifies Magentos behavior to significantly improve the cache hit rate.

Benchmarking Parameters

Since every Magento store is different and the amount of data housed within one can make a big difference when it comes to performance, weve standardized on the following Magento configuration for this white paper: Magento version: Enterprise 1.12.0.2Stores: 1SKU count: 100,000 (thanks to our friends at Magento for providing this pre-built data set)Categories: 10Subcategories: 40Some standard configurations are set throughout the testing as they are best practices for Magento. These con-figurations are:

Two level Magento backend cache is enabled. First level is set for memcached. Second level is set to Redis with the CM_Cache plug-in.

The two-level backend cache patch is applied. Sessions are stored in a dedicated memcached instance. The Magento compiler is enabled. The Magento full page cache is enabled (with the storage set to a second Redis instance). The APC PHP opcode cache is used.



7

Benchmarking Hardware ArchitectureA Nexcess MCE-SIP-200 server cluster was used for benchmarking in all of our test runs. This cluster consists of a firewall, two web application servers, a single database server, a dedicated load balancer, and a dedicated file server (which also shares a role as a Magento application caching server).

Web ApplicationServer 1

Web ApplicationServer 2

Master DatabaseServer

Dedicated ClusterFile Server andCaching Server

Dedicated LoadBalancer

Dedicated ManagedFirewall

Incoming requestsOutgoing TracInternal File and Cache TracInternal Database Trac

Internet

Nexcess RedundantInternal 10GbE

Network

Diagram 1: Nexcess MCE-SIP-200 Cluster Architecture

The Nexcess MCE-SIP-200 cluster used for testing consisted of the following hardware:

Web Application Servers:Dell R4202x Intel Xeon 8 Core CPUs E5-2450 @ 2.10GHz24GB RAM2x 147GB 15K SAS RAID-1 (H710)

Database Server:Dell R4202x Intel Xeon 6 Core CPUs E5-2420 @ 1.90GHz24GB Ram4x 147GB 15K SAS RAID-10 (H710)



8

File Server / Caching Server:Dell R4202x Intel Xeon 6 Core CPUs E5-2420 @ 1.90GHz24GB Ram4x 147GB 15K SAS RAID-10 (H710)

Load Balancer:Dell R4202x Intel Xeon 6 Core CPUs E5-2420 @ 1.90GHz16GB Ram2x 73GB 15K SAS RAID-1 (H710)

Benchmarking Server (from which all tests were run):Dell R4202x Intel Xeon 6 Core CPUs E5-2450 @ 2.10GHz24GB Ram4x 73GB 15K SAS RAID-10 (H710)

The testing server was placed across a gigabit network which was connected from within the Nexcess data cen-ter in Dearborn, MI to the public interfaces of the testing cluster.



9

Software and Benchmarking ConfigurationsSoftware Versions

The following software versions were installed on all servers in the MCE-SIP-200 cluster used for testing:

CentOS 6.3 (x86_64) Percona 5.5.28PHP 5.3.17 MySQL 5.5.29-1APC 3.1.9-5 memcached 3.0.6-3Redis 2.4.10-1 Apache 2.2.23 (pre-fork MPM)Nginx 1.0.15-3 PHP53-fpm-5.3.17-2Varnish 3.0.2 Nexcess Magento-Turpentine 0.3.1

Furthermore, the benchmarking server ran the following software versions:

Siege 2.72 Gatling Tool 1.4.1

Anatomy of a Siege Test Raw Stress Testing

A Siege test was run on each specific configuration for 60 seconds and then repeated once per minute over 100 minutes (for a total of 100 tests). Data was collected after each 60 second run. In addition to the 100 minute tests, additional tests were run for a duration of 240 minutes each to observe any long term behavior of the Magento configured environment.

To ensure a broad spectrum of targets, we created a static list of pages on the test site using a method popular-ized by Ashley Schroder on his site, magespeedtest.com. Utilizing the sitemap.xml from the data sets provided by Magento on the active test site, we were provided with over 100,000 pages (including every product and category page) to test. Details of sitemap-based testing can be found at http://www.aschroder.com/2010/12/lay-ing-siege-to-magento-performance-testing-your-magento-webstores/.

Each test was run five times against a particular configuration. The test results were then averaged. Before each complete run, all relevant Magento caches were flushed and then pre-warmed by having Siege hit every URL in our test file exactly once. The test file consisted of a complete Magento sitemap.

Anatomy of a Gatling Test User Modeling Stress Testing

We used Gatling to simulate two basic scenarios:

1. A buyer in the following use case (in sequence):a. Navigate to the home page.b. Navigate to a category page.c. Navigate to a product page.d. Add product to the cart.e. View the cart.f. Proceed through checkout process fully using the check/money-order payment method.



10

2. A visitor in anonymous catalog navigation (in sequence):a. Navigate to the home page.b. Navigate to a random category page.c. Navigate to a random product page.

The stress tests were configured to simulate both visitors and buyers with a 100:1 ratio. Thus, given 2,000 visitors, there would be roughly 20 buyers. The intention behind the Gatling test was to place real world stress on a full Magento environment with actual orders being placed. Simulating buyers engages parts of the system, such as database bound requests, that are much harder to simulate with a simple Siege test. This allowed us to find the optimal performance parameters for more realistic traffic profiles.

Each test was run five times against a particular configuration. The test results were then averaged. Before each complete run, all relative Magento caches were flushed so each new configuration to be tested would begin with an unpopulated and empty, or cold, fast backend, slow backend, session, and full page cache.



11

The Test CasesTest 1: File Synchronization NFS vs. Local File System

File synchronization is a necessity when running multiple web application servers with Magento. Both the code and metadata within Magento should be synchronized across all web application servers in order for it to func-tion properly.

In this test we compared a pure NFS solution against a hybrid NFS/local store solution. NFS (or any shared stor-age engine) plays a key role in ensuring certain metadata is synchronized in real time. Thus, for the sake of this test, the media directory is always shared across all web application servers using NFS.

We tested two separate options for file synchronization between the web application servers. The first used NFS for all synchronization and the second used the local file systems on each web application server to serve con-tent. When using a local store method, we provided file synchronization via rsync.

There are both advantages and disadvantages to each configuration option. While NFS offers greater simplicity of site management (any code changes are immediately propagated to the web application servers) there is some performance overhead required. This overhead shows up in the benchmarking results. The local file sys-tem performs better as long as the local disk I/O subsystem on each server is properly designed using fast SAS or SSD drives and a good caching controller.

There is one additional comment regarding the shared media directory over NFS: When used with a CDN for distributing static content with images, most images will come from the CDN and not the NFS-mounted file sys-tems. This prevents further performance degradation from using NFS while maintaining Magento administrative functionality.

Test 1: Results

This test was implemented utilizing Siege with 250 concurrent users.



12

0

100

200

300

400

500

600

T=1 T=10 T=20 T=30 T=40 T=50 T=60 T=70 T=80 T=90 T=100

Local File System vs. NFS Transactions Per Second

NFS File System Local File System

Time in Minutes

Tran

sacti

ons

per

Seco

nd

Diagram 2: Local File System vs. NFS Transaction Per Second

Running Magento on a local file system resulted in an average of 554 transactions per second while an NFS-mounted file system resulted in an average of 500 transactions per second. This accounts for a roughly 11% performance increase and is consistent regardless of the web server software used.

The choice of whether to run Magento from either an NFS fileserver or locally on each web node comes down to convenience and ease of usability of the site configuration as opposed to gains to be made in performance. NFS offers the benefit of your site files existing in only one location. When making changes to the site, or moving a site configuration from development or staging to production, only one location must be modified. Using NFS, these changes will be instantly visible on each web application server. On the other hand, using local file systems requires each web node to contain an entire copy of the Magento site and configuration. The copy then must be kept in synchronization to the fileserver via rsync or other means. In this configuration, the Magento media directory will still be shared across NFS. The Magento admin functionality would otherwise be broken during im-age uploads as the uploaded image would not be immediately syncronized between all web application servers. This will result in broken images to visitors on servers other than the node the image was uploaded on which will persist until the media directory is resynchronized. However, as long as careful planning and execution of site updates are followed by the site development team during new code changes and releases, the additional 11% performance gain can be realized by using local files.

Test 2: Apache vs. Nginx

Apache and Nginx are the two most popular HTTP servers used with Magento. Despite the performance num-bers Nginx produces on a properly configured web server, Apache is very competitive when also properly configured.

The important step to be considered is how PHP is initiated when running Apache. The standard configuration is to use mod_php and doing so can provide reasonable performance. Apache with mod_php can be tuned to



13

perform well on moderately loaded systems but for heavily loaded systems Apache with PHP-FPM is preferred. More importantly, however, Apache with PHP-FPM provides for full user separation that isnt easily attainable using Apache with mod_php. For this reason we dont recommend its use in high performance environments and do not explicitly entertain it in this whitepaper. We find most comparisons of Nginx and Apache are unbal-anced because of this, as Nginx is benchmarked using PHP-FPM while Apache uses mod_php, suPHP or some other non PHP-FPM method of handing off PHP scripts to the PHP interpreter. We benchmarked both Apache and Nginx with PHP-FPM. This keeps the comparison balanced.

Test 2: Results

Tran

sacti

ons

per

Seco

nd

T=1 T=10 T=20 T=30 T=40 T=50 T=60 T=70 T=80 T=90 T=1000

100

200

300

400

500

600

Apache + PHP-FPM vs. Nginx + PHP-FPM

Apache + PHP-FPM Nginx + PHP-FPM

Time in Minutes

Diagram 3: Apache + PHP-FPM vs. Nginx + PHP-FPM

The first test run was a direct comparison of Apache with PHP-FPM and Nginx with PHP-FPM. For this Siege test we used 250 concurrent users.

The graph shown in Diagram 3 shows the differences between Apache and Nginx when both are utilizing PHP-FPM. The differences in performance between the two separate configurations was negligible. Nginx had less than a 1% advantage over Apache in performance over every test run. The initial ramp-up period of the graph shows a slower transactions per second count as the various Magento caches run through their warming period. Once the caches are fully warmed (at about 30 minutes into the test) site performance remains constant. The variance between runs also becomes very consistent once the caches are warmed (as shown in the graphs in Diagram 4 and Diagram 5).



14

Tran

sacti

ons

per

Seco

nd

1st minute 2nd thru 5th minute 6th thru 10th minute 11h thru 20th minute 21st thru 30th minute 31st thru 40th minute 41st thru 60th minuteAverage Trans/s 95.08 114.43 152.46 273.23 485.08 547.84 555.11+- Margin 0.00 14.24 20.45 46.33 33.34 1.94 1.61Lower Limit 95.00 100.19 132.01 226.91 451.73 545.90 553.50Upper Limit 100.00 128.67 172.91 319.56 518.42 549.79 556.72

0

100

200

300

400

500

600

Nginx + PHP-FPM Performance

Average Trans/s +- Margin Lower Limit Upper Limit

Digram 4: Nginx + PHP-FPM Performance

Tran

sacti

ons

per

Seco

nd

1st minute 2nd thru 5th minute 6th thru 10th minute 11h thru 20th minute 21st thru 30th minute 31st thru 40th minute 41st thru 60th minuteAverage Trans/s 96.34 115.12 154.53 274.87 482.03 542.34 546.87+- Margin 0.00 13.63 21.90 44.48 31.57 5.39 1.15Lower Limit 90.00 101.49 132.63 230.39 450.46 536.95 545.72Upper Limit 100.00 128.75 176.43 319.35 513.60 547.72 548.02

0

100

200

300

400

500

600

Apache + PHP-FPM Performance

Average Trans/s +- Margin Lower Limit Upper Limit

Diagram 5: Apache + PHP-FPM Performance

Choosing whether to run Magento on Apache or Nginx becomes a personal choice for those running the site rather than one based on pure performance. One advantage of Nginx is it has a built-in proxy cache similar to Varnish. That cache can be used to significantly increase performance beyond the numbers shown here, but re-quires extensive configuration and development to make it compatible with a Magento site. This advantage over Apache is eliminated given one can use Varnish with Apache to achieve the same results.



15

The biggest Apache advantage is the benefit of being fully compatible with Magento with no modifications to site rewrites or .htaccess files needed.

Test 3: MySQL versus Percona

MySQL is an extremely popular open source relational database management system (RDBMS) for use in many web-based applications. Magento fully supports MySQL for its database and it is the recommended database to use in both the community and enterprise editions.

Percona is also an RDBMS, but is a drop-in replacement for MySQL and designed to offer higher performance over MySQL. Percona specifically excels at InnoDB storage engine performance with its own InnoDB engine, XtraDB. Percona also offers built-in support for better handling of NUMA-based hardware. This is particularly important when using very large Magento databases

The purpose of this comparison was to see if a properly tuned installation of Percona can outperform a properly tuned instance of MySQL for a Magento site.

Test 3: Results

For the MySQL and Percona comparison, the Gatling tool was used for all testing. Running a simple Siege test showed no difference between the two systems. Siege in not useful in this case as most data is located within the various caches and, once warmed, the database subsystem is barely utilized. Utilizing Gatlings capabilities to simulate actual users on the site, we were able to significantly stress the database and exceed more than 7,000 queries per second. The Gatling tests simulate the actual ordering process, creating a significant amount of write-bound queries (which do not occur when a user simply browses the site).

The Gatling testing was performed in two parts. The first was using various concurrencies of buyers and visitors at a 100:1 ratio. This test was run against the MCE-SIP-200 cluster running first with MySQL and then again with Percona. Each test was run five times and then averaged to compare the number of total transactions per second against the cluster.



16

0

200

400

600

800

1000

1200

1400

1600

1800

2000

1 Buyer100 Browsers

10 Buyers1000 Browsers





MySQL vs. Percona

MySQL Percona

dnoceS reP snoitcasnarT

Diagram 6: MySQL vs. Percona

The results depicted in Diagram 6 show, as the number of buyers and visitors increase, MySQL and Percona have fairly equal results (with a slight margin to Percona). This trend continues until the workload increased to 30 buyers and 3,000 visitors. This is where Perconas advantage became apparent. It continued to hold true un-der an increased workload to 50 buyers and 5,000 visitors. Also, these results show all static assets returned by Gatling mixed with orders being processed and provide a more realistic real world scenario of buyers and visitors navigating the site.

The second part of this test was run to calculate the maximum number of orders per second an MCE-SIP-200 cluster running Percona is capable of processing. Using Gatling, a buyer test without any simulated visitors was run. Various amounts of buyers, from 750 to 3,000, were queued and run against the cluster to see how many orders by these buyers could be processed per second. A buyer in this test consists of the following sequence:

1. Navigate to the home page.2. Navigate to a category page.3. Navigate to a product page4. Add product to the cart.5. View the cart.6. Proceed through checkout process fully using the check/money-order payment method.

Each calculated order is one user performing all of the above tasks successfully.



17

Percona

0

2

4

6

8

10

12

14

16

18

20

750 1000 2000 3000

Ord

ers

Per S

econ

d

Number of Buyers

Orders per Second Using Percona

Diagram 7: Orders per Second Using Percona

At the peak of 3,000 buyers, Magento is capable of processing 18.9 orders per second which equates to roughly 1.6 million orders per day.

Optimized Magento Site versus Optimized Magento Site with Varnish & Turpentine

Varnish is a reverse proxy HTTP application accelerator. Out of the box, Magento is not directly compatible with Varnish due to the dynamic nature of a Magento session-based site. By default, Varnish doesnt cache requests with cookies and Magento sends a front-end cookie with every request causing a (near) zero hit-rate for Varnishs cache. Turpentine provides Varnish configuration files (VCLs) to work with Magento and modifies Magentos behavior to significantly improve the cache hit rate.

A test was performed utilizing Varnish along with the Turpentine plug-in and compared against a site not using Varnish. Because user actions (such as adding items to a cart and checking out) cannot be cached, very few per-formance differences were noticed during these tests. Where Varnish helps is during a typical user browsing the site.

Test 4: Best configuration versus Varnish & Turpentine

This test was performed to demonstrate the capability of a compatible and properly configured reverse proxy in front of Magento. For this test we utilized Varnish along with the Nexcess Turpentine plug-in on the MCE-SIP-200 cluster. Varnish was installed on each individual web application server in front of Apache and PHP-FPM.

Turpentine does not work when the full page cache is enabled due to the way sessions are handled. On a FPC cache hit, Magento runs very little PHP code and instead pulls the cached page from disk (or memcached, Redis, etc). With Turpentine this is a problem because it normally adds some headers to tell Varnish what is and is not OK to cache, and whether Varnish should do ESI processing on the response (for ESI included blocks). This results



18

in the FPC caching a page with ESI injected blocks, but without varnish running the ESI processing on the page which effectively means those blocks will be missing. Additionally, it can result in Varnish caching pages like the customer login page that should not be cached. Thus, the FPC was disabled during the following tests with Varnish & Turpentine. Note Varnish is so efficient at serving cached pages directly from memory, these tests were limited by the benchmarking utility, the available network bandwidth, and benchmarking server specifications. Two different Siege tests were run with concurrency levels of 250 and 500 on each configuration.

0

1000

2000

3000

4000

5000

6000

7000

8000

250 500

dnoceS reP snoitcasnarT

Concurrency

Varnish / Turpentine vs. FPC

Apache / PHP-FPM + FPC Apache / PHP-FPM + Varnish / Turpentine

Diagram 8: Varnish / Turpentine vs. FPC

As can be seen above, running Varnish with Turpentine in front of Apache + PHP-FPM can provide a significant performance increase. At 250 concurrent users, transactions per second increased from an average of 548 per second to 3,800 per second with Varnish & Turpentine. At 500 concurrent users, Varnish & Turpentine again out-performed the existing Apache + PHP-FPM setup with an average of 6,940 transactions per second over a one hour run. This is a benchmarked 1,300% increase in performance, which potentially provides over 600 million transactions per day.



19

1

10

100

1000

10000

T=1

T=2

T=3

T=4

T=5

T=6

T=7

T=8

T=9

T=10

T=11

T=12

T=13

T=14

T=15

T=16

T=17

T=18

T=19

T=20

T=21

T=22

T=23

T=24

T=25

T=26

T=27

T=28

T=29

T=30

T=31

T=32

T=33

T=34

T=35

T=36

T=37

T=38

T=39

T=40

T=41

T=42

T=43

T=44

T=45

T=46

T=47

T=48

T=49

T=50

T=51

T=52

T=53

T=54

T=55

T=56

T=57

T=58

T=59

T=60

Tran

sact

ions

Per

Sec

ond

Time (minutes)

Varnish / Turpentine vs. FPC

Apache / PHP-FPM + Varnish / TurpentineApache / PHP-FPM + FPC

Diagram 9: Varnish / Turpentine vs. FPC

Looking at the transaction rate graph above shown with a logarithmic scale, the non-Varnish & Turpentine configuration shows the ramp-up period of the full page cache over the first 30 minutes. With Varnish, the cache is immediately populated in the first 60 seconds of the run since nearly every single page has been hit during this time frame. This high rate of transactions per second is maintained over the 60 minute test with some observed variance. The reason behind this variance comes down the capabilities of the benchmarking server. At this rate our benchmarking server was seeing transfer rates of over 900Mb/s to the MCE-SIP-200 cluster which was ultimately limiting our results. Running the Gatling test with concurrencies higher than 3,000 provided no additional performance numbers as the gigabit connection between the cluster and the benchmark server were saturated.

Running the Gatling buyers-only test to calculate orders per second with Varnish & Turpentine resulted in only a moderate performance increase to 21.2 orders per second (up from 18.5). This is expected as most steps in the buying process are HTTP POST operations and therefore cannot be cached.



20

Result Observations and Optimal ConfigurationsThe Importance of the Magento Enterprise Full Page Cache

One of the biggest built-in performance improvements in Magento is the full page cache. When enabled, and once fully warmed, the number of transactions per second was over 500% higher compared to the same site with the FPC disabled (see Diagram 10).

Tran

sact

ions

Per

Sec

ond

0

100

200

300

400

500

600

T=1

T=3

T=5

T=7

T=9

T=11

T=13

T=15

T=17

T=19

T=21

T=23

T=25

T=27

T=29

T=31

T=33

T=35

T=37

T=39

T=41

T=43

T=45

T=47

T=49

T=51

T=53

T=55

T=57

T=59

T=61

T=63

T=65

T=67

T=69

T=71

T=73

T=75

T=77

T=79

T=81

T=83

T=85

T=87

T=89

T=91

T=93

T=95

T=97

T=99

Full Page Cache Enabled vs. Disabled

FPC Disabled FPC Enabled

Time In Minutes

Diagram 10: Full Page Cache Enabled vs. Disabled

The graph presented in Diagram 11 shows the relationship of the fast backend, slow backend, and full page cache utilization in megabytes along with the hit ratios for all three caches (in percentages). The number of benchmarked transactions per second is graphed alongside these cache variables to correlate the cache usage versus site performance.



21

Cach

e H

it Pe

rcen

tage

0

10

20

30

40

50

60

70

80

90

100

0

500

1000

1500

2000

2500

3000

3500

4000

4500T=

1T=

3T=

5T=

7T=

9T=

11T=

13T=

15T=

17T=

19T=

21T=

23T=

25T=

27T=

29T=

31T=

33T=

35T=

37T=

39T=

41T=

43T=

45T=

47T=

49T=

51T=

53T=

55T=

57T=

59T=

61T=

63T=

65T=

67T=

69T=

71T=

73T=

75T=

77T=

79T=

81T=

83T=

85T=

87T=

89T=

91T=

93T=

95T=

97T=

99

Cache Utilization vs. Transactions Per Second

Full Page Cache Usage (MB)Fast Cache Usage (MB)Slow Cache Usage (MB)Transfer Rate (Transactions Per Second)

Full Page Cache Hit Rate PercentageFast Cache Hit Rate PercentageSlow Cache Hit Rate Percentage

Time In Minutes

MB

used

/ Tr

ansa

ctio

ns p

er S

econ

d

Diagram 11: Cache Utilization vs. Transactions Per Second

The resulting data shows the sites transactions per second performance is heavily dependent upon the caches being fully populated with data. All three caches ramped-up in the first 30 minutes of benchmarking and then remained at a constant level over time, thereby providing the best performance. Because the hit rate is averaged over time, once the first minute of data was available, the hit rate was already over 75% for both the fast cache and full page cache. It then continued to climb upwards as the test progresses. Also note the data in the fast cache was fully populated and the fast cache was large enough to hold all data necessary, thus the slow cache was rarely ever used (even though it was also fully populated).

The Magento DEFAULT_LIFETIME

During some three hour-long and higher benchmark runs, an interesting artifact was observed. Roughly two hours into the benchmark, a slight dip would occur, deviating from the standard benchmark baseline estab-lished once all caches were built.



22

0

2

4

6

8

10

12

14

T=1

T=4

T=7

T=10

T=13

T=16

T=19

T=22

T=25

T=28

T=31

T=34

T=37

T=40

T=43

T=46

T=49

T=52

T=55

T=58

T=61

T=64

T=67

T=70

T=73

T=76

T=79

T=82

T=85

T=88

T=91

T=94

T=97

T=10

0T=

103

T=10

6T=

109

T=11

2T=

115

T=11

8T=

121

T=12

4T=

127

T=13

0T=

133

T=13

6T=

139

T=14

2T=

145

T=14

8T=

151

T=15

4T=

157

T=16

0T=

163

T=16

6T=

169

T=17

2T=

175

T=17

8

MBp

s

Website Throughput

Time in Minutes

Diagram 12: Website Throughput

The cause of this dip results from the fact there is a default lifetime variable in app/code/core/Mage/Core/Model/Cache.php set to 7,200 seconds (or two hours).

class Mage _ Core _ Model _ Cache{ const DEFAULT _ LIFETIME = 7200; const OPTIONS _ CACHE _ ID = core _ cache _ options; const INVALIDATED _ TYPES = core _ cache _ invalidate; const XML _ PATH _ TYPES = global/cache/types;

At the 7,200 second expiry time, data will begin to be automatically purged from the various caches. This data is immediately repopulated into each cache, keeping each caches hit rate high, but the purging is enough to cause a 16MB/s drop in the outgoing traffic along with an 80 transaction per second drop at peak flushing.



23

Time in Minutes

Tran

sact

ions

per

Sec

ond

0

100

200

300

400

500

600

T=1

T=4

T=7

T=10

T=13

T=16

T=19

T=22

T=25

T=28

T=31

T=34

T=37

T=40

T=43

T=46

T=49

T=52

T=55

T=58

T=61

T=64

T=67

T=70

T=73

T=76

T=79

T=82

T=85

T=88

T=91

T=94

T=97

T=10

0T=

103

T=10

6T=

109

T=11

2T=

115

T=11

8T=

121

T=12

4T=

127

T=13

0T=

133

T=13

6T=

139

T=14

2T=

145

T=14

8T=

151

T=15

4T=

157

T=16

0T=

163

T=16

6T=

169

T=17

2T=

175

T=17

8

Website Transactions

Diagram 13: Website Transactions

Since the cache was pre-warmed before the test runs all of the caches expired around the same time. On a lower traffic site, the expiration of cached pages will be more graceful and would not cause such a pronounced dip as seen here. Bear in mind, on a high traffic site, flushing the cache will cause a fast cache build and similar expi-ration behavior as seen in Diagram 13. Because of this, it is not recommended to flush any caches during peak traffic hours.

Response Times

An important part of any Magento store is fast page response time. Benchmarking the MCE-SIP-200 cluster with a 250 concurrent user Siege test and the full page cache enabled, page response times were sustained at under a half second once the cache was warmed roughly 30 minutes into the test. The exact same benchmark ran with the full page cache disabled in Magento Enterprise resulted in a best page response time of 2.04 seconds at 250 concurrent connections. This again reinforces the importance of the FPC within Magento Enterprise.



24

0

0.5

1

1.5

2

2.5

3

3.5

T=1

T=3

T=5

T=7

T=9

T=11

T=13

T=15

T=17

T=19

T=21

T=23

T=25

T=27

T=29

T=31

T=33

T=35

T=37

T=39

T=41

T=43

T=45

T=47

T=49

T=51

T=53

T=55

T=57

T=59

T=61

T=63

T=65

T=67

T=69

T=71

T=73

T=75

T=77

T=79

T=81

T=83

T=85

T=87

T=89

T=91

T=93

T=95

T=97

T=99

Response Time

Without FPC With FPC

Time In Minutes

Page

Loa

d Ti

me

In S

econ

ds

Diagram 14: Response Time



25

Understanding Magento CachingWhen Magento and cache are used in the same sentence, many more details must be provided to be sure any caching mechanism being talked about is understood. There are many different caches that can be used in (and alongside) Magento, including, but not limited to, a PHP opcode cache, session cache, full-page cache, fast backend cache, slow backend cache and Varnish cache. Properly implementing and configuring these can result in considerable performance increases compared to a site without caching or without a correctly configured caching system.

Magento Backend Cache

The Magento backend cache is the most common cache used as it is a native part of the Zend Framework (of which Magento takes advantage). This cache can be configured in two different implementations: a single backend or a two-level backend cache. There are also many methods of storage the Magento backend cache can be configured to use, including the local file system, memcached, Redis and APC. When enabled, the default method is to use the local file system to store cache data as individual files on disk. While this is an improvement over not using caching at all, there are better memory-based options available such as memcached or Redis that can greatly enhance performance.

Redis is preferred because of its tag support (something the other memory cache engines lack). When Magento was configured to use Redis as the sole caching backend in a single, non-two-level configuration, a slowdown in transactions per second was noticed while performing our benchmark tests. Upon further investigation, it was realized Redis, being a single-threaded application, does not take advantage of multi-core CPU systems. The lack of thread support, along with the high number of incoming serialized requests, had a slight effect on performance. Redis documentation acknowledges this limitation and suggests using parallel Redis instances in situations like this. Unfortunately, as of this writing, these separate instances are not truly distributed and fault tolerant. Redis is aware of this and is working on a product named Redis Sentinel which should solve this issue. We will revisit this testing with the latest stable version of Redis as a single-level cache solution when the tools permit.

A two-level cache configuration was also tested and resulted in the highest benchmarked transaction numbers and best site functionality. Redis was used as the tag-supporting slow, or second level, cache, and memcached as the first level or fast cache. The benefit of this setup was we could use the multi-threaded capabilities of mem-cached while retaining the tag-support of Redis (all without sacrificing memory-backed data stores for disk I/O). A second dedicated Redis instance was also set up for exclusive use by the full page cache module in Magento Enterprise (detailed below).

One last thought should be kept in mind regarding the backend cache and APC. APC is an allowed cache meth-od to be used throughout the various Magento cache configurations, but its use is not recommended. APC is tied to the running PHP-FPM parent process, and, in the event PHP-FPM is restarted, it will forcefully flush your cache (which is not an ideal behavior). If APC is also being used for a PHP opcode cache, using it for the backend cache will result in greater APC memory fragmentation.



26

Additional improvements

When using Magentos two-level cache built on the Zend Framework, there is a known bug preventing the cache from operating correctly. The result of this bug prevents the first level cache from refreshing correctly, which in turn makes the first level cache perform much worse than it should. There is a patch available that will fix this bug and it is recommended for use in all two-level cache implementations. Information on this bug and a patch can be found at http://www.fabrizio-branca.de/magento-zend-frameworks-twolevels-cache-back-end-mess.html. All benchmarks in this white paper were run with the Zend two-level cache bug fix implement-ed.

Magento Full Page Cache

The full page cache is a feature offered in Magento Enterprise. The basic idea of the Magento full page cache is to store full HTML output of a page request. It is another cache which can be set to use various backend mech-anisms such as files, or memory-based using memcached or Redis. When running Magento with the full page cache enabled, and having the cache be fully populated and hot, a significant performance increase can be observed.

Magento Session Cache

Sessions within Magento are stored within their own cache and the location of this cache is customizable. The file system is the default location for sessions to be stored, and while this is fine for most purposes, additional performance can be gained by removing the file system from the session-based operations and storing sessions in memory via memcached. Using the database for session storage is another option, although it is not normally recommended.

It is important to be sure the session storage location is shared between all web application servers in a multi-server setup. Load balancer persistence (if enabled) will keep connections on the same web server, but in the event that the client moves to an additional server, or switches HTTP to HTTPS, having a centralized session location will ensure the clients credentials and current cart remain consistent.

It is also a good practice to create an additional memcached instance on a different host or port to store session data. Using the same memcached instance for both Magento backend cached data and session data is not rec-ommended, as flushing the Magento backend cache will also flush the clients session data.

PHP opcode cache

There are many Magento-compatible open-source PHP opcode optimizers and caches. These caches (such as APC, eAccelerator, and Xcache) will cache the bytecode compiled from source by PHP. The advantage of this is frequently used pieces of code can be cached, thereby saving processing time as PHP does not have to recom-pile the same piece of code multiple times. While Magento will work with APC, eAccelerator, and Xcache, APC is the recommended opcode cache to be used with Magento. While all three caches offer similar performance, only APC is used in all benchmarking tests. For all tests in this white paper, APC was enabled on both web appli-cation servers.



27

Conclusion and Additional Performance Recommen-dationsAs demonstrated by the above testing and benchmarking procedures, a load-balanced, two web application server cluster with a single database server can process 47 million transactions and over 1.6 million orders per day. Thus, the following should be considered best practices for a Magento installation for maximum perfor-mance:

Run either Nginx or Apache with PHP-FPM. Both offer nearly identical performance utilizing PHP-FPM, but Apache benefits include better drop-in compatibility with Magento than Nginx. While Nginx re-quires more custom configuration to work with Magento, it does offer built-in reverse proxy capability. Bear in mind, one does not simply turn this reverse proxy on; it will require extensive configuration to work with a production ready Magento store.

Enable the full page cache when using Magento Enterprise with its own dedicated Redis backend. The FPC can offer a significant increase in performance over a Magento site not utilizing it. Avoid flushing the FPC during peak traffic hours as it can severely affect site performance while the cache rebuilds. Please note that sharing the FPC Redis cache instance with the Magento 2nd level cache (or any other caching module) is not recommended and can actually degrade performance. Make sure the FPC Redis instance is dedicated to the FPC.

Utilize a two-level cache with memcached as the fast cache and Redis as the slow cache. Be sure avail-able memory in each of these caches is high enough to prevent forced evictions. Evictions can be checked by observing the built-in stats for each cache instance. If using the two-level cache, be sure to apply the two-level cache bug fix.

Use a dedicated memcached instance for sessions. Be sure this instance is shared across all web applica-tion nodes so session data is available to a user on any server node and protocol.

Use local file systems over NFS. Local file systems will offer an 11% performance increase over NFS mounted file systems at the cost of additional complexity. If your actual traffic is well under the maxi-mum traffic capabilities of your cluster, you can use NFS for overall simplicity of site maintenance and operation.

Use Percona over MySQL if Magento will be used in high traffic installations. Percona provides better all around performance with Magento along with consistent write performance.

Use Varnish as a reverse proxy server in front of your HTTP server. Utilize the Nexcess Turpentine plug-in for better compatibility between Varnish and Magento. In this configuration, a transaction per second increase of 1,300% beyond the 47 million transactions per day cited, along with a 15% improvement in orders per second can be realized.

Bear in mind, all results here were tested on Magento Enterprise without any optional extensions or plug-ins in-stalled with exception of the Cm_RedisSession add-on for Redis functionality and the Nexcess Turpentine plugin where marked as used. The addition of Magento extensions and plug-ins, depending on their functionality, inte-



28

gration, and usage, can drastically affect overall site performance. It is always a good practice to benchmark your site before and after an extension installation so you are aware of any performance degradation from it.

Appendix A: System ConfigurationsAll system level configuration files used for benchmarking, including the Magento Enterprise local.xml configu-rations, are available on github at: https://github.com/nexcess/whitepaper-v1-configs.

Appendix B: Benchmarking ConfigurationsGatling Tool

All code used for the Gatling Tool tests is available on github at: https://github.com/nexcess/magento-gat-ling-tests

Siege

The following command line option was used for all Siege tests:

/usr/local/bin/siege -c $CONCURRENCY -i -b -t60s -f urls.txt --log=$SIEGELOG -A Siege

urls.txt is the Magento sitemap parsed into a text file. $CONCURRENCY is the concurrency passed to the test. $SIEGELOG is the path to log the output from Siege.

IntroductionTesting CriteriaTerms and DefinitionsBenchmarking ToolsSoftware Tools & Plug-insBenchmarking Parameters

Benchmarking Hardware ArchitectureSoftware and Benchmarking ConfigurationsSoftware VersionsAnatomy of a Siege Test Raw Stress TestingAnatomy of a Gatling Test User Modeling Stress Testing

The Test CasesTest 1: File Synchronization NFS vs. Local File SystemTest 1: ResultsTest 2: Apache vs. NginxTest 2: ResultsTest 3: MySQL versus PerconaTest 3: ResultsOptimized Magento Site versus Optimized Magento Site with Varnish & TurpentineTest 4: Best configuration versus Varnish & Turpentine

Result Observations and Optimal ConfigurationsThe Importance of the Magento Enterprise Full Page CacheThe Magento DEFAULT_LIFETIMEResponse Times

Understanding Magento CachingMagento Backend CacheAdditional improvementsMagento Full Page CacheMagento Session CachePHP opcode cache

Conclusion and Additional Performance RecommendationsAppendix A: System ConfigurationsAppendix B: Benchmarking ConfigurationsGatling ToolSiege

Magento Performance Testing for High Performance

Documents

Transcript of Magento Performance Testing for High Performance