NCLB Consolidated Monitoring Integrated Approach to Title III Monitoring.
A Modern Approach to Performance Monitoring
-
Upload
cliff-crocker -
Category
Internet
-
view
629 -
download
0
description
Transcript of A Modern Approach to Performance Monitoring
A Modern Approach to Performance Monitoring
Cliff Crocker@cliffcrockerSOASTAVP Product, mPulse
1. How fast am I?
2. How fast should I be?
3. How do I get there?
Synthetic vs. RUM
What is RUM?Real User Measurement (traditional definition)Synthetic 101Synthetic monitoring (for purposes of this discussion) refers to the use of automated agents (bots) to measure your website from different physical locations.
• A set ‘path’ or URL is defined• Tests are run either adhoc or scheduled and data is collected
What is RUM?Real User Measurement (traditional definition)RUM 101Real User Measurement (RUM) is a technology for collecting performance metrics directly from the browser of an end user.
• Involves instrumentation of your website via JavaScript
• Measurements are fired across the network to a collection point through a small request object (beacon)
<JS>
<beacon>
RUM
Cast a wide net• Identify key areas of concern• Understand real user impact• Map performance to human behavior
& $$
Synthetic
Diagnostic tool• Identify issues in a ‘lab’/remove
variables• Reproduce a problem found with RUM http://www.flickr.com/photos/84338444@N00/with/3780079044/
http://www.flickr.com/photos/ezioman/
The Early Days of RUM
• Round-trip time• Start/stop timers via JavaScript• Early contributors:• Steve Souders/Episodes• Philip Tellis/Boomerang.js• Both widely in use today
Navigation TimingWhat gives RUM a kick
Browser Support for Navigation Timing
!
11© 2014 SOASTA CONFIDENTIAL - All rights reserved.
Navigation TimingExamples
DNS: Domain Lookup Time
function getPerfStats() { var timing = window.performance.timing; return { dns: timing.domainLookupEnd - timing.domainLookupStart};}
12© 2014 SOASTA CONFIDENTIAL - All rights reserved.
Navigation TimingExamples
TCP: Connection Time to Server
function getPerfStats() { var timing = window.performance.timing; return { connect: timing.connectEnd - timing.connectStart};}
13© 2014 SOASTA CONFIDENTIAL - All rights reserved.
Navigation TimingExamples
TTFB: Time to First Byte
function getPerfStats() { var timing = window.performance.timing; return { ttfb: timing.responseStart - timing.connectEnd};}
14© 2014 SOASTA CONFIDENTIAL - All rights reserved.
Navigation TimingExamples
Base Page
function getPerfStats() { var timing = window.performance.timing; return { basePage: timing.responseEnd - timing.responseStart};}
15© 2014 SOASTA CONFIDENTIAL - All rights reserved.
Navigation TimingExamples
Front-end Time
function getPerfStats() { var timing = window.performance.timing; return { frontEnd: timing.loadEventStart - timing.responseEnd};}
16© 2014 SOASTA CONFIDENTIAL - All rights reserved.
Navigation TimingExamples
Page Load Time
function getPerfStats() { var timing = window.performance.timing; return { load: timing.loadEventStart - timing.navigationStart};}
Measuring AssetsYour page is composed of many different parts
• Strength of synthetic• Full visibility into asset performance• Images• JavaScript• CSS• HTML
• A lot of which is served by third-parties• CDN!
Object Level RUMGoing beyond page level metrics
Browser Support for Resource Timing
CORS: Cross-Origin Resource Sharing
Timing-Allow-Origin = "Timing-Allow-Origin" ":" origin-list-or-null | "*"
Start/End time only unless Timing-Allow-Origin HTTP Response Header defined
Resource Timing
var rUrl = ‘http://www.akamai.com/images/img/cliff-crocker-dualtone-150x150.png’;var me = performance.getEntriesByName(url)[0];var timings = { loadTime: me.duration, dns: me.domainLookupEnd - me.domainLookupStart, tcp: me.connectEnd - me.connectStart, waiting: me.responseStart - me.requestStart, fetch: me.responseEnd - me.responseStart}
Measuring a single resource
Resource TimingOther uses
• Slowest resources• Time to first image• Response time by domain• Time a group of assets• Response time by initiator type (element type)• Cache-hit ratio for resources
For examples see: http://www.slideshare.net/bluesmoon/beyond-page-level-metrics
Resource TimingCool projects
• PerfMap - https://github.com/zeman/perfmap• Mark Zeman
• Waterfall.js - https://github.com/andydavies/waterfall• Andy Davies
1. How fast am I?
2. How fast should I be?
3. How do I get there?
Picking a NumberHow do you set front-end performance goals?
• Industry benchmarks?
• Apdex?• Analyst report?• Case studies?
“Synthetic monitoring shows you how you relate to your competitors, RUM shows you how you relate to your customers.”
– Buddy Brewer
BenchmarkingHow you relate to the competition
BenchmarkingSOASTA mPulseUX
• http://soasta.com/mpulseUX
BenchmarkingImportant to remember that it’s not always about Page Load
• Page construction• Requests• Images• Size
• Other important metrics• Speedindex• Start Render• PageSpeed/Yslow scoring
Performance is a Business Problem
Yahoo! - 2008Slower pages cause increase in abandonment
Increase of 400ms causes 5-9% increase in user abandonment
http://www.slideshare.net/stubbornella/designing-fast-websites-presentation
Shopzilla - 2009Faster pages increase site conversion rate
A reduction in Page Load time of 5s increased site conversion 7-12%!
http://assets.en.oreilly.com/1/event/29/Shopzilla%27s%20Site%20Redo%20-%20You%20Get%20What%20You%20Measure%20Presentation.ppt
Walmart - 2012Impact of slow pages on conversion rate
http://minus.com/msM8y8nyh#1e
SF WebPerf – 2012Up to 2% conversion drop for every additional second
So What?
SIMULATION
How Fast Should You Be?Key takeaways
• Use synthetic measurement for benchmarking your competitors
• Understand how fast your site needs to be to reach business goals/objectives with RUM
• You must look at your own data
1. How fast am I?
2. How fast should I be?
3. How do I get there?
Real users are not normal
Page Load TimesAll Countries, User Agents, Carriers, Operating Systems
2.76s – Median
10.45s – p95
17.26s – p98
Platform Median 95th Percentile
98th Percentile
Mobile 3.62s 12.53s 20.04s
Desktop 2.44s 9.31s 15.86s
Page Load TimesBy Platform
Page Load TimesBy Operating System
OS Median 95th Percentile
98th Percentile
Windows 7 2.41s 9.29s 15.89s
Mac OS X/10 2.30s 8.11s 13.45s
iOS7 3.27s 10.64s 15.79s
Android 4 4.06s 14.30s 27.93s
iOS8 3.53s 11.54s 19.72s
Windows 8 2.67s 10.75s 18,74s
Other FactorsAll Countries, User Agents, Carriers, Operating Systems
• Geography• User Agent• Connection Type• Carrier/ISP• Device Type
Not All Pages are Created EqualTop of funnel impact (Browse Pages)
For a typical eCommerce site, conversion rate drops by up to
50% when “browse” pages increase from 1
to 6 seconds
Not All Pages are Created EqualBottom of funnel impact (Checkout pages)
However, there is much less
impact to conversion
when “checkout”
pages degrade
How Do I Get There?Key takeaways
• Focus on the highest value opportunities/demographics
• Identify key pages that have the most impact on your KPIs
• Prioritize based on reducing friction within the funnel or critical path
Thank You!
AttributionsImages
1. Modern (Title Slide): https://www.flickr.com/photos/looking4poetry/3477854720/in/photolist-6ijV3q-9t3sjb-cj8VWE-MNjPA-4yBqug-2rk5he-4nZVzJ-4yE45J-9t3Fvj-2rptr5-69ymdr-6XVTH-6QZ4dg-9t3HN5-8geUkB-6oEXs5-eH9zS-H2XTt-cKdJvb-7GNKWx-cKdKZd-5Rw3qt-jfT5Dx-Jfiuy-bTop6R-2m8kAB-Jfiuu-7E8eMf-9jqD6-9hYAvd-Jfius-gsqr7U-7KiNAZ-8kzG9V-euXdb-44qJNN-47jBuY-nsxZrZ-7cGjQL-4cfHKq-cGzbSC-aQtWPR-8y3sR3-6okjAW-5A7BSC-6aRsvh-eSMHkX-kxvdyK-9t3udu-c8EkYu
2. Man vs. Machine: https://www.flickr.com/photos/eogez/3289851965/in/photolist-61HmnB-9nkQu3-dfinF1-4GjK24-4GjL6Z-4GoV6C-dibVCU-4GoTXo-4GjF3k-4GjH7Z-4GoR6h-ek6eT9-ek6eUU-4fADUx-4fEDV9-65JA6W-65Jsxu-65JDnj-4fAE12-65JDTh-65JBZu-65Ek9M-65JtDu-65JuiN-65JzFS-65EfiH-65Jv63-65JxX7-65Enyc-65Eh4c-65Jxjm-65JwmU-65EhUP-65JCML-65EjxP-65JBQf-65Ee4Z-65JD4b-65JBF1-65Ec1r-65EkPB-65Jy4o-65Eg96-65JAjb-65Emfe-65Efoz-65JBam-65Eniv-65JCrU-65Eed2
3. https://www.flickr.com/photos/bradybd/2818154005/
4. https://www.flickr.com/photos/perspective/149321089/
5. Dartboard: http://en.wikipedia.org/wiki/Darts#mediaviewer/File:Darts_in_a_dartboard.jpg
References:
6. http://calendar.perfplanet.com/2011/a-practical-guide-to-the-navigation-timing-api/
7. http://www.slideshare.net/bluesmoon/beyond-page-level-metrics