Who I Am

Cliff Crocker

VP Product

SOASTA, Inc.

@cliffcrocker

http://www.flickr.com/photos/luc/

Real Users

http://www.flickr.com/photos/spullara/

Happy Users

http://www.flickr.com/photos/sybrenstuvel/

Not SoHappy

RUM 101Real User Measurement (RUM) is a technology for collecting performance metrics directly from the browser of an end user.

• Involves instrumentation of the page with JavaScript

• Measurements are fired across the network to a collection point through a small request object (beacon)

<JS>MySite.com

<beacon>

Origins of RUM2005-6: Steve S. builds

JS tool to measure page load from the browser during a Yahoo! hackday

2008: Steve open sources ‘Episodes’. White pages open

sources ‘Jiffy’

2010: Yahoo! Open sources

‘boomerang.js’ the same day Navigation Timing is introduced

in IE9 beta

• http://stevesouders.com/episodes/• https://code.google.com/p/jiffy-web/• http://lognormal.github.com/boomerang/doc/• https://dvcs.w3.org/hg/webperf/raw-

file/tip/specs/NavigationTiming/Overview.html

This helped…

Navigation Timing API

• Exposes network level timing details to JavaScript

• DNS lookup, TCP connect, SSL handshake, First Byte time, and more

• For a given web request, you can tell exactly where the time was spent

• Supported by IE >= 9, Chrome >=6, Firefox >=7, Android Browser .=4.0, others

• No exposure into rendering times

C’mon Safari, be cool…

http://caniuse.com/nav-timing

~70% Browser Support

Why RUM is Awesome

• Reduces the level of convincing we have to do within our organization that ‘faster is better’

• The only way to measure perceived performance from your end user

• Easy to maintain

• Only getting better…– Check out:http://blog.patrickmeenan.com/2013/07/measuring-

performance-of-user-experience.html• User Timing for RUM and Synthetic

– Fun with resource timing: https://github.com/andydavies/waterfall

What gives??

No RUM

RUM • According to a study done in 2012* only 1 in 5 of the top ecommerce sites are using some form of RUM

*http://www.webperformancetoday.com/2012/11/13/real-user-monitoring-rum-ecommerce-sites/

Challenges with RUM

1. It’s a “data” problem

2. People don’t like change

3. Hard to “sell” internally

For a modest sized site, in a given day

1-3 Million Page Views

Perf Timing

User Agent

OS/Device

Geography

Page Segmentation

Bandwidth Classification

A/B Bucket

‘Other’

Many have advocated aggressive sampling of data…

~1% sample, looking at a period of about 3 hours for a given page segment (Checkout)

1% > checkout

Filtered to the United States…

1% > > checkout

Looking specifically at California…

1% > > > checkout

IE9 Browsers only.

1% > > > checkout

Looking at all traffic.

100% > checkout

All filters applied, identified bimodal distribution driven by backend time.

100% > > > checkout

> backend time

Sampled at 1%

Not sampled

To be fair…

Read: http://www.webperformancetoday.com/2013/03/06/big-

data-vs-big-enough-data/

Many use a batch solution…

• Batch processing large amounts of data has been a traditional approach for RUM

– Great for looking at historical data, trending, etc.

– Big Data is not necessarily Fast Data

• Not ‘Operational’

In order for RUM to be operationally relevant, you need the data in real-time

Synthetic vs. RUM

http://www.flickr.com/photos/myvector/

It is Not Either Or

• Synthetic and RUM coexist

• Important to understand:

– Geographic performance

– Object level detail (waterfall)

– Browser performance

– Competitive positioning

– Site coverage

Performance by Geography

RUM

????

Performance by Geography

Synthetic

Understanding Performance by Geography

• RUM provides better visibility into geographic variability (simply more data points from more locations)

RUM 1

Synthetic 0

Object Level Detail

RUM 1

Synthetic 1

Waterfall available for most major browsers available for synthetic monitoring (IE, FF (Commercial), Chrome, Safari)

Limited support for waterfall using resource timing (IE10 & Chrome stable) – using waterfall.js

Browser Coverage

RUM 2

Synthetic 1

SyntheticSupport for major browsers

Commercial (mainstream synthetic)

WebPagetestsupport

RUMSupport for any browser that executes JS

Competitive Positioning

RUM 2

Synthetic 2

Synthetic

Ability to measure competitors easily

RUM

More complex, no standard benchmarks (yet!)

Commercial offerings

WebPagetest.org

Site Coverage

RUM 3

Synthetic 2

Synthetic

Choose a ‘happy path’ or spot check

RUM

No blind spots!

http://scoutmastercg.com/wp-content/uploads/typimg/green_laser.png

http://www.flickr.com/photos/areese/with/4002609608/

Synthetic Monitoring

• Pros

– Technology is great

– Real Browsers (IE, Chrome, FF)

– Built in Alerting

– Waterfall analysis

– Screen shots & headers

– Object level detail

• Cons

– False Positives

– Impossible/impractical to measure everything

– Fixed number of browser/OS combinations

– Simulated bandwidth constraints at best

– Too few data points for statistical relevance

RUM

Cast a wide net• Identify key areas of concern• Understand real user impact• Tie this back to $$ (next)

Synthetic

Diagnostic tool• Identify issues in a ‘lab’• Reproduce a problem found with

RUM

http://www.flickr.com/photos/84338444@N00/with/3780079044/

http://www.flickr.com/photos/ezioman/

RUM in the Workplace

Studies show…

However, you have to look at your own data

Understand your KPIs and How to Track Them

• Retail: Order Conversion, Add to Cart Conversion, Revenue

• Media: In Line Display Ad Views, Comments,

• Other: Facebook likes/Social Media sharing

Bottom Line, if you want your business to care – then make it relevant for

them (speak their language)

Strategy for Success – RUM First

RUM First

• All the data (or statistically relevant sample)

• Real-time and historical

Supplement with

Synthetic

• Reduce measurements based on coverage with RUM

• Use for adhocdiagnostic & competitive benchmarking

Map to business

KPIs

• YOUR goals

• YOUR data

• Single source of truth

mPulse Litehttp://www.soasta.com/free/