Simple  Steps  to  Win  the  Site  Performance  Battle    (and  other  hopefully  interesting  stuff)    CHRIS  WELLS  –  CEO  –  NEXCESS.NET  LLC  SOUTHFIELD,  MI    USA  

Quick  Facts  About  Detroit,  MI  •  It’s  the  potato  chip  capital  of  the  world  •  based  on  consumption  L  

•  It  had  the  1st  mile  of  concrete  road  (1909)  •  Between  6  and  7  mile  roads  on  M-­‐1  (Woodward)  

•  Home  to  the  only  floating  Post  Office  in  the  U.S.  •  J.W.  Westcott  Co  (Zip  code:    48222)  

•  It  has  the  most  registered  bowlers  nationwide    

•  Biggest  municipal  bankruptcy  in  the  U.S.  •  Which  makes  us  National  Bankruptcy  Champions!!  

•  Eat  your  heart  out  Jefferson  County,  AL!!    We  did  4  x  better  than  you!!  

Notes  About  This  Talk  •  Lessons  learned  from  MagentoLive  Australia  •  If  masses  walk  out  mid-­‐talk  I’m  being:  

A)  WAY  too  technical  

B)  Ridiculously  boring      

•  Lessons  learned  from  MagentoLive  UK  •  If  I  sweat  profusely:  

A)  DO  NOT  BE  ALARMED  

B)  Assume  the  technical  folks  located  at  the  back  of  the  room  simply  swapped  out  the  real  talk  I  had  prepared  with  something  they  thought  was  more  fun  and  I’m  simply  dying  a  slow  35  minute  death  on  stage  before  your  very  eyes.  

The  Basics  NOT  On  Today’s  Agenda  •  OS  level  set  up  /  configuration  

•  Apache  or  Nginx  

• Magento  caching  

• Magento  EE’s  FPC  

•  Varnish  /  Turpentine  or  other  reverse  proxy  caching  techniques  

•  Other  performance  extensions    

•  At  this  point  we’re  all  well  aware…  

•  So  let’s  dig  a  little  deeper  into  some  fun  stuff!  

*ON*  Today’s  Agenda  •  PHP  –  92%  

• MySQL  –  3%  (if  time  permits)  

•  Drives  –  5%  (if  time  permits)  

•  If  you  leave  here  today  with  some  food  for  thought  then  I’m  happy!  

PHP  –  Personal  Home  Page  •  Yes,  I  know..  It’s  now  “PHP:  Hypertext  preprocessor”  

•  Magento  used  to  be  version  locked  to  PHP  <=  5.3  

•  As  of  January  Magento  officially  supports  PHP  5.4!    Woohoo!  •  You  may  need  to  patch  your  Magento  for  it  to  work  with  5.4  

•  So…?  

•  PHP  5.4  is  a  good  deal  faster  than  5.3  •  Empty  hash  table  optimizations  

•  Literal  tables  •  Interned  strings  •  Zend  Engine  VM  tuning  

•  But  what  does  this  mean  for  Magento?  

Notes  On  Magento  • Magento  is  very  CPU  heavy  •  The  bulk  of  this  heaviness  is  within  PHP  code    

•  MySQL  CPU  usage  is  typically  bound  by  PHP’s  CPU  usage  (speaking  in  single  server  terms)  

• Magento  is  not  very  I/O  heavy  (contrary  to  popular  belief)  •  Again,  I/O  is  bound  by  PHP’s  CPU  usage  

•  More  on  I/O  later  

•  Quantitative  notes  to  follow  J  

Let’s  Test  Us  Some  PHP  •  1-­‐hour  Siege  runs  (meant  to  tax  PHP,  not  MySQL)  

•  EE  tests  •  Siege  of  20,000  SKUs  hitting  ~20,000  random  URLs  

•  FPC  enabled  (Redis)  

•  CE  tests  •  Siege  of  120  SKUs  hitting  ~120  URLs.  

•  Bare  metal  configuration  of:  •  Dell  R420  –  Dual  E5-­‐2420  6-­‐core  •  H710  512MB  NV  +  4  x  15K  300GB  RAID  10  

•  24G  RAM  

•  Ok  ok,  let’s  see  what  we’ve  got!  

PHP  5.3.24  (CE  Front-­‐end  Baseline)  

0"

20"

40"

60"

80"

100"

120"

140"

1" 2" 3" 4" 5" 6" 7" 8" 9" 10" 11" 12" 13" 14" 15" 20" 25" 30" 35" 40" 45" 50" 55" 60"

Siege"tran

sac6on

s"/"se

cond

"

Dura6on"(in"minutes)"

PHP"5.3.24"

~42  t/sec  

PHP  5.3.24  vs.  PHP  5.4.28  (CE  Front-­‐end)  

0"

20"

40"

60"

80"

100"

120"

140"

1" 2" 3" 4" 5" 6" 7" 8" 9" 10" 11" 12" 13" 14" 15" 20" 25" 30" 35" 40" 45" 50" 55" 60"

Siege"tran

sac6on

s"/"se

cond

"

Dura6on"(in"minutes)"

PHP"5.3.24"

0"

20"

40"

60"

80"

100"

120"

140"

1" 2" 3" 4" 5" 6" 7" 8" 9" 10" 11" 12" 13" 14" 15" 20" 25" 30" 35" 40" 45" 50" 55" 60"

Siege"tran

sac6on

s"/"se

cond

"

Dura6on"(in"minutes)"

PHP"5.3.24"

PHP"5.4.28"

~52  t/sec  (~23%  increase!)  

Magento  cache    population  

PHP  5.3.24  (EE  Front-­‐end  Baseline)  

0"

100"

200"

300"

400"

500"

600"

700"

800"

1" 2" 3" 4" 5" 6" 7" 8" 9" 10" 11" 12" 13" 14" 15" 20" 25" 30" 35" 40" 45" 50" 55" 60"

Siege"tran

sac6on

s"/"se

cond

"

Dura6on"(in"minutes)"

PHP"5.3.24"

~550  t/sec  

GO  GO  FPC!  

PHP  5.3.24  vs.  PHP  5.4.28  (EE  Front-­‐end)  

0"

100"

200"

300"

400"

500"

600"

700"

800"

1" 2" 3" 4" 5" 6" 7" 8" 9" 10" 11" 12" 13" 14" 15" 20" 25" 30" 35" 40" 45" 50" 55" 60"

Siege"tran

sac6on

s"/"se

cond

"

Dura6on"(in"minutes)"

PHP"5.3.24"

0"

100"

200"

300"

400"

500"

600"

700"

800"

1" 2" 3" 4" 5" 6" 7" 8" 9" 10" 11" 12" 13" 14" 15" 20" 25" 30" 35" 40" 45" 50" 55" 60"

Siege"tran

sac6on

s"/"se

cond

"

Dura6on"(in"minutes)"

PHP"5.3.24"

PHP"5.4.28"

 ~615  t/sec  

(~12%  increase!!)    

FPC  /  Magento  cache  population  amplifies  base  PHP  version  improvements  

PHP  5.3.24  vs.  PHP  5.4.28/.14  (EE  Front-­‐end)  

0"

100"

200"

300"

400"

500"

600"

700"

800"

1" 2" 3" 4" 5" 6" 7" 8" 9" 10" 11" 12" 13" 14" 15" 20" 25" 30" 35" 40" 45" 50" 55" 60"

Siege"tran

sac6on

s"/"se

cond

"

Dura6on"(in"minutes)"

PHP"5.3.24"

PHP"5.4.14"

PHP"5.4.28"

Even  upgrading  from  5.4.14  shows  gains  of  ~4%    

PHP  5.3.24  vs.  PHP  5.4.28  (Re-­‐index  from  CLI)  

0" 10" 20" 30" 40" 50" 60" 70" 80" 90"100"

PHP"5.3.24"

PHP"5.3.24"

Time  In  Seconds  

86#88#

90#92#

94#96#

98#100#

PHP#5.3.24#

PHP#5.4.28#

~8%  improvement  

Let’s  Push  Things  a  Little  Further...  

PHP  5.4.28  vs.  PHP  5.5.12  (EE/Front-­‐end)  

0"

100"

200"

300"

400"

500"

600"

700"

800"

1" 2" 3" 4" 5" 6" 7" 8" 9" 10" 11" 12" 13" 14" 15" 20" 25" 30" 35" 40" 45" 50" 55" 60"

Siege"tran

sac6on

s"/"se

cond

"

Dura6on"(in"minutes)"

PHP"5.4.28"

PHP"5.5.12"

 ~781  t/sec  

~25%  better  than  5.4  ~41%  better  than  5.3  

WOW  J    

FPC  /  Magento  cache  population  again  

amplifies  base  PHP  version  improvements  

Notes  On  PHP  5.5.x+  •  !!  Warning  –  Magento  does  not  yet  officially  support  5.5+  !!    

•  APC  goes  out  •  Magento  configs  may  need  to  be  tweaked  if  you’re  using  APC’s  key/val  caching  

•  Some  extensions  will  need  to  be  updated  /  re-­‐coded  for  sure  

•  OPcache  is  introduced  •  OPcache  is  the  name  of  the  bundled  ZendOptimizer+  opcode  caching  system  

•  Seems  to  work  out  of  the  box  without  too  much  fuss  

•  More  research  is  needed  here  –  was  very  surprised  with  the  performance  results  

•  !!  Warning  –  Magento  does  not  yet  officially  support  5.5+  !!    

PHP  5.4.28  vs.  PHP  5.5.12  (Re-­‐index  from  CLI)  

85#86#

87#88#

89#90#

91#

PHP#5.4.28#

PHP#5.5.12#

Time  In  Seconds  

~4%  improvement  

Let’s  Push  Things  a  Little  Further…  

PHP  5.5.12  vs.  PHP  5.6.0-­‐BETA2  (EE/Front-­‐end)  

0"

100"

200"

300"

400"

500"

600"

700"

800"

1" 2" 3" 4" 5" 6" 7" 8" 9" 10" 11" 12" 13" 14" 15" 20" 25" 30" 35" 40" 45" 50" 55" 60"

Siege"tran

sac6on

s"/"se

cond

"

Dura6on"(in"minutes)"

PHP"5.5.12"

PHP"5.6.0"(BETA"2)"

 

L  Essentially  equal  within  

margin  of  error    

PHP  5.5.12  vs.  PHP  5.6.0-­‐BETA2  (Re-­‐index  from  CLI)  

72# 74#76#

78#80#

82#84#

86#88#

PHP#5.5.12#

PHP#5.6.0#BETA2#

Time  In  Seconds  

BUT,  we  got  ~11%  improvement  on  the  re-­‐index!  

 ~21.5%  better  than  5.3.24!  

Let’s…  

PHP  5.5.12  vs.  PHP-­‐NG  (EE/Front-­‐end)  

Notes  on  PHP-­‐NG  •  Removes  numerous  heap  allocations  (and  de-­‐allocations)  

•  Stores  more  native  data  directly  on  the  stack  

•  Removes  the  need  to  garbage  collect  basic  primitives  (bool,  long,  etc)  

•  PHP’s  reported  Wordpress  benchmarks  show  very  good  results  •  26.75  sec  -­‐>  18.95  sec  (36%  improvement)  

•  9.5M  instructions  -­‐>  5M  internal  instructions  executed  (90%  reduction)  

•  Take  some  comfort  in  knowing  that  more  gains  are  on  the  way  from  PHP  folks  directly  

Simple  Step  #1  –  Use  PHP  5.4.28  (for  now)    **But  watch  out  for  future  PHP  releases!  

MySQL  –  “My  SQL”  • My  is  the  Michael  Widinius’s  daughter’s  name  (this  was  news  to  me)  

•  Lots  of  “brand”  options  on  the  MySQL  front  unlike  PHP  •  MariaDB  

•  Percona  Server  /  Percona  Cluster  •  Oracle’s  MySQL  

•  Equally  many  options  on  the  storage  engine  front  •  MyISAM  

•  InnoDB  •  XtraDB  

Let’s  Test  Us  Some  MySQL  •  ~1  hour  re-­‐index  runs  (ideally  meant  to  put  some  load  on  MySQL)  

•  Full  CLI-­‐based  re-­‐index  

•  Tested  latest  EE  with  various  versions/brands  of  MySQL  

•  DB  contained  800,000  SKUs  (thanks  Magento  performance  team)  

•  Bare  metal  configuration  of:  •  Dell  R420  –  Dual  E5-­‐2420  6-­‐core  •  H710  512MB  NV  +  4  x  15K  300GB  RAID  10  

•  24G  RAM  

•  Ok  ok,  let’s  see  what  we’ve  got!  

Magento  CLI  Re-­‐index  (EE)  

Time  In  Seconds  

3100$ 3150$ 3200$ 3250$ 3300$3350$

3400$3450$

3500$3550$

MySQL$5.1.71$

Percona$Cluster$5.5.34$

MariaDB$10.0.10$

MySQL$5.6.17$

Percona$Server$5.6.17$

Magento  CLI  Re-­‐index  (EE)  

Time  In  Seconds  

0"5"

10"15"

20"25"

Percona"Server"5.6.17"

MariaDB"10.0.10"

MySQL"5.1.71"

Percona"Cluster"5.5.34"

MySQL"5.6.17"

Time"Spent"in""sys""

Thoughts  On  The  Tests  •  Re-­‐index  runs,  while  DB  intensive,  are  still  PHP  heavy  

•  Oracle’s  MySQL  did  surprisingly  well  •  If  concurrency  were  higher  I’d  expect  Percona/MariaDB  to  excel  

•  DB  bound  writes  (beyond  the  re-­‐index)  were  missing  •  This  would  tax  the  DB  servers  considerably  

•  TCP  (and  other)  overhead  for  Percona  Cluster  doesn’t  seem  to  kill  performance  

Simple  Step  #2  –    Use  a  modern  version  of  MySQL  (ideally  XtraDB  based)  

Hard  and  Solid  State  Disk  Drives  •  Hard  drive  vs.  disk  drive  vs.  solid  state  drive  vs.  SAS,  etc.  •  IOPs  (I/O  operations  /  sec)  •  More  is  better,  but  don’t  throw  money  away  

•  All  SSDs  not  created  equal  •  Don’t  believe  quoted  specs!  •  Don’t  buy  consumer  grade  SSDs!  

•  Quality  SSDs  are  ~2x  the  cost  of  quality  HDD  of  same  size  

•  Know  your  workload  profile!  

Live  Client  Case  Study  • Magento  EE  1.12.0.1    

•  ~700,000  SKUs  

•  350+  orders  /  hour  

•  15  x  web  servers  (bare  metal)  •  Dell  R420,  Dual  E5-­‐2450  8-­‐core,  24GB  RAM  

•  2  x  DB  servers  (replicated)  –  peak  @  ~20,000  qps  

•  1  x  file  server  (shared  storage  for  media)  

•  FPC  /  Magento  cache  in  Redis  

•  ~1  week  sample  size  

Database  Server  IO  Requirements  

Read   Write  

Average  IOPs:   3.66   471  

Min  IOPs:   0.0   6.87  

Max  IOPs:   830   2250  

Standard  deviation:   34.5   197.0  

95%  below:   60.4   795  

99.999%  below:   151   1310  

•  Percona  Server  5.5.35  •  128GB  RAM  /  60GB  MySQL  DB  (innodb_buffer_pool  =  75G)  

•  8  x  300GB  15k  HDD/SAS  RAID  10    -­‐  H700/1G  RAID  card  •  Replicating  to  a  hot-­‐standby  DB  server    (unused  by  Magento)  

???  

File  Server  IO  Requirements  

Read   Write  

Average  IOPs:   13.7   7.65  

Min  IOPs:   5.58   2.97  

Max  IOPs:   85.8   186  

Standard  deviation:   6.72   11.0  

95%  below:   24.7   25.8  

99.999%  below:   42.3   54.6  

•  Serving  /media  and  /var  only  over  NFS  

•  24GB  RAM  -­‐  600GB  data  on  disk  

•  4  x  600GB  15k  HDD/SAS  (RAID  10)  -­‐  H700/512M  RAID  card  

Notes  On  Drives  •  Get  quality  drives  regardless  of  technology  •  No  consumer  SSDs!  

•  Caches  are  very  good  at  relieving  disk-­‐bound  I/O  •  MySQL  query  cache  

•  Redis/memcaches  

•  RAID  card  caches  •  On-­‐disk  controller  caches  

•  Invest  in  CPU  over  IOPs  unless  you  can  afford  both  

• Magento  is  typically  bound  by  CPU,  not  I/O  –  Know  your  workload  profile  

Simple  Step  #3  –    Use  HDD  &  SSD  drives  as  needs/costs  dictate  

 Thank  you!    **  Thanks  for  not  walking  out  while  I  sweat.    Come  collect  your  free  bag  of  chips!!  

CHRIS  WELLS  –  CLWELLS  -­‐AT-­‐  NEXCESS.NET