Walmart pagespeed-slide

Real User Monitoring

@Walmarthttp://www.destructoid.com

Presentation Schedule

• Cliff - 10-20m on RUM

• Aaron – 10-20m on Monitoring in real-time

• Balaji – 10-20m on Correlating RUM and Business Analytics

• Final Q&A

Cliff Crocker- Performance- Reliability- Platform & Site Analytics

Twitter: @cliffcrocker

shhhh…..

• We are not the fastest retail site on the internet today.

IE 8 – Web Page Test Comp Index – Item Page Performance

11s – Fully Loaded time

4

What gets measured, gets done…

• Before we start to optimize – see where our pain is and prepare to measure for success.

• You never go on a diet without first stepping on the scale…

5

Source: Flickr

Synthetic Monitoring

Pros– Technology is great– Real Browsers (IE, Chrome,

FF)– Built in Alerting– Charting/Trending– Waterfall analysis– Screen shots & headers– Object level detail

Cons– False Positives– Impossible/impractical to

measure everything– Fixed number of browser/OS

combinations– Simulated bandwidth

constraints at best– Too few data points for

statistical relevance

6

~2.2% of Global Internet usage - Thanksgiving this year

7

• If only those users could tell us about their site experience…

8

Source: Flickr

Enter boomerang.js - https://github.com/yahoo/boomerang

boomerang.js with NavTiming (thanks Buddy & Phil)

• Doc Complete, Page Response, Page Processing, TTFB, DNS, Connect & more• Cookie data (for parent domain)• Location (geolookup on IP)• Referrer• User Agent• Anything else you want to stuff into the beacon

+9

Source: Flickr

Start with a url group (i.e. ‘Item Page’)

Find offenders that have the most impact

Identify usage patterns for browsers

(Chrome and Safari users more active on weekends)

12

What we found scared us…

Home Page Performance – Jan 201214

Set some goals and SLAs

• Focus on ‘Page Processing Time’ First– see Golden Rule – 80% of time spent here (more like 90% for Walmart)

• Look at backend & network response time SLAs later• Use 95th Percentile• Set Achievable SLAs• Revisit Monthly• Celebrate Wins!

15

Case Study: Item Page

• Problem: – Page takes ~24s for slowest 5% of users

• Too many elements• Slow third party modules• Several other pagespeed ‘no-nos’

• Goal: Meet SLA for February– 20s (95th percentile)

• Approach:– Scrum team dedicated to perf optimization for 1 sprint– Team pools resources and ideas - focuses on biggest bang

16

Success!

8s off the 95th %tile

17

Success!

2.7s under February SLA

18

RUM in Near Real Time

BEACON SERVERBEACON SERVERCLIENTCLIENT INTERNETINTERNET

MAGIC SMOKE

How boomerang.js Works

1. client downloads JavaScript payload2. client sends RUM payload to beacon3. beacon server responds with HTTP 204

WHO CARES!WHO CARES!

20

Aaron Kulick- Performance- Founder: SF & SV Web Performance Group

http://www.sfwebperf.org

Twitter: @GoFastWeb

WHO AM I?

Configuring boomerang.js is EASY

Getting Actionable Data Is HARD

22

Where’s the data?

The initial incarnation of RUM @ WMT leveraged Akamai at the edge.

• data reliability issues• data availability issues• data “freshness” issues

It works… for certain values of work

Source: Flickr

Source: Flickr

23

HUBBLEHUBBLE

Enter Hubble

JETTYJETTY

DURABLE QUEUEDURABLE QUEUE

APACHE FLUME NGAPACHE FLUME NG

HDFSHDFS UDPUDP

HADOOPHADOOP

24

REAL TIMEREAL TIME

The Bridge Between Then and Now

• Etsy/StatsD - https://github.com/etsy/statsd– Timers– Counters– Simple Aggregation

• Min, Max, Mean, Median, 95th Percentile

• Whisper, Carbon & Graphite - https://launchpad.net/graphite– Scalable Realtime Graphing– RRD-like Storage Requirements + Federated Option– Supports Irregular Updates

The Now Bits

HUBBLEHUBBLEUDPUDP StatsDStatsD

AGGREGATION

Graphite+

Whisper

Graphite+

WhisperUDP/TCPUDP/TCP

STORAGE+

DISPLAY

• Fixed some calculation problems in Etsy/StatsD• Added median calculation for timers• Incorporated multiple flush interval patches

Pretty Pictures

RAW DATA FORMATuswmt.all.t_page.upper_95(9289.0),1329259510,1329260710,10|15904.0,9184.0,9125.0,12736.0,11735.0,16776.0,8484.0,10839.0,14620.0,7579.0,8871.0,8240.0,12390.0,5211.0,10301.0,24784.0,9410.0,16554.0,9609.0,11871.0,12751.0,9797.0,11003.0,15962.0,7953.0,7707.0,4181.0,11616.0,11746.0,12814.0,10566.0,24782.0,18303.0,20904.0,7718.0,8531.0,7312.0,9614.0,8749.0,11671.0,5989.0,9832.0,10592.0,11611.0,16946.0,18858.0,14360.0,15927.0,10470.0,10140.0,11307.0,9739.0,9772.0,9875.0,13641.0,11626.0,14758.0,6529.0,11727.0,10194.0,8003.0,10639.0,7297.0,9891.0,10312.0,12497.0,11557.0,11406.0,12456.0,12939.0,11029.0,10813.0,11737.0,10618.0,14128.0,16879.0,15865.0,6255.0,14605.0,8861.0,27425.0,10948.0,19666.0,7185.0,13266.0,13156.0,15111.0,13110.0,15151.0,8666.0,16775.0,10110.0,10387.0,17274.0,22183.0,8937.0,13168.0,12267.0,11891.0,9635.0,10446.0,8129.0,9550.0,9229.0,8375.0,8657.0,11119.0,6799.0,9094.0,21952.0,14989.0,16828.0,9001.0,13444.0,10332.0,13609.0,9266.0,13349.0,11546.0,9289.0uswmt.all.t_page.median(1844.0),1329259510,1329260710,10|4165.0,2333.5,2073.5,2584.0,2547.0,2627.5,2401.0,1575.0,2170.0,1169.0,1970.0,1838.0,2083.0,5211.0,2496.0,3242.5,1541.0,1437.5,1928.0,1971.0,1776.0,3108.0,2010.5,2044.0,2325.5,2640.0,1733.0,3924.0,2629.0,1867.5,1782.0,2370.5,2921.0,4783.0,2260.0,1340.0,3256.0,2297.0,2565.0,1874.0,2000.0,2483.5,2705.5,2432.0,1809.0,2826.0,2204.0,2695.0,1045.0,1615.5,2250.0,2387.0,1562.5,1998.0,2512.0,2139.0,1482.0,2138.5,2100.0,2583.0,2652.0,3277.0,2549.0,1755.0,2196.5,2766.0,2989.5,3638.0,3034.0,3615.5,2650.5,5207.5,3023.0,1941.0,1918.5,1768.0,3048.0,1522.5,2710.0,1392.0,2402.0,2005.0,3246.0,1383.0,1880.0,2398.0,1833.0,2579.0,2052.0,2622.0,2089.0,1102.0,1296.0,3339.0,2132.5,2831.0,3466.0,2131.0,2026.0,2754.0,3228.5,1000.0,2075.0,2011.5,2428.0,4019.5,2788.0,1665.0,1968.0,2695.5,2873.0,1752.0,2314.5,1766.0,2971.0,3091.5,2205.5,3033.0,2476.0,1844.0

27

The Work Tomorrow…

The Good•Metric Throughput•Commodity Storage•Commodity Infrastructure

The Bad•More Metric Throughput•Calculation Complexity•Web Sockets (pretty)•Metric Fan-out

28

WATCH THIS SPACE

29

Is Page Performance a Factor of Site Conversion? And how big is it?February, 2012

vs

31

Walmart.com - Fun Facts

• Reach–Millions of Shoppers/week. –Billions of page requests/year - Spikes up to 1500%

–Billions of internal product search volumes/year

• Scale–Millions of active product SKUs + Market Place–Millions of pages indexed in search engines

• Complexity–1/4th of page contents served by partners, affiliates and Marketplace –Multiple departments, 10+ checkout paths

Page Performance & Site Conversion – Feb 2012

32

So, how do you monitor?...


33

Few Industry Benchmarks…

• Factoid 1: Large eCommerce site extensively A/B tested page performance and published a study showing 100 millisecond delay = 1% drop in revenue

• Factoid 2: Search Engines A/B tested performance and found that a 500 millisecond delay caused a 20% drop in traffic.

• Factoid 3: In an experiment across multiple retailers, a 1 second delay caused a 7% decline in conversion


34

So, how big is it for Walmart.com?


35

Agenda

• Phase 1 – Baseline Measurement - Impact of Site Performance on Conversion, Bounce rates & Revenue

• Phase 2 - Targets for Page Performance

• Phase 3 – Optimization Results

• Key Highlights & Takeaways


36

Agenda


• Phase 2 – Targets for Page Performance




37

Impact of site performance on overall site conversion rate….

Baseline – 1 in 2 site visits had response time > 4 seconds* Sharp decline in conversion rate as average site load time increases from 1 to 4 seconds

* Overall average site load time is lower for the converted population (3.22 Seconds) than the non-converted population (6.03 Seconds)

Note: Load Time here is the time taken from head of the page to page ready (T_Page)


38

@ Page level….

Page load time is lower for Buyers compared to Non-Buyers

* The Page load time is highest for certain pages - 6.38 secs when there was a conversion and 8.06 where there was no conversion.



39

@ Department level….

Department load time is lower for Buyers compared to Non-Buyers

* Key Categories has 2-3 seconds difference b/w buyer Vs non-buyer



40

What about bounce?

Page Bounce Rate Vs Response Time

* Key pages have high bounce rates which correlates with high T_Page as well

* Significant difference ( up to 9secs) in T_Page between bounced and non-bounced for landing pages.



41

Bounce rates @ department level….

Department Bounce Rate Vs Response Time

* High T_Page for key pages (up to 19.82s) and key department making Bounce rate significantly higher



42

Agenda






43

Phase 2 – Targets for Page Performance……

Conversion & Bounce Rate Impacts Drives Prioritization



44

Agenda






45

Phase 3 – Success Story….

First Win….and yes conversion had positive improvements…


46

• Page speed matters for site conversion!

• Monitor real user performance in a “Big Data” way!!

• Every 1s improvement = Up to 2% increase in CVR

• 100ms improvement = Up to 1% incremental revenue

• SEO benefits for entry pages and reduce bounces

• Test & Learn - Target segments and run A/B Tests focused on improving page performance

Key Highlights


We’re Hiring for Everything!!!!

47

Source: Flickr

http://www.walmartlabs.com/open-positions/https://walmartstores.com/careers/apply/?ba=eCom @cliffcrocker or @GoFastWeb

Walmart pagespeed-slide

Education

Transcript of Walmart pagespeed-slide