Edge 2016 can webpagetest be trusted
-
Upload
akamaidevrel -
Category
Technology
-
view
28 -
download
4
Transcript of Edge 2016 can webpagetest be trusted
![Page 1: Edge 2016 can webpagetest be trusted](https://reader034.fdocuments.in/reader034/viewer/2022042723/587165b61a28ab58758b556b/html5/thumbnails/1.jpg)
© AKAMAI - EDGE 2016
Can Webpagetest be Trusted?Pierre Lermant
![Page 2: Edge 2016 can webpagetest be trusted](https://reader034.fdocuments.in/reader034/viewer/2022042723/587165b61a28ab58758b556b/html5/thumbnails/2.jpg)
©2016 AKAMAI | FASTER FORWARDTM
WPT
It is THE synthetic performance testing tool
● Free to use public instanceThrough UI or APIs
● Feature rich
● Maintained to latest standards
● Can be hosted privately
![Page 3: Edge 2016 can webpagetest be trusted](https://reader034.fdocuments.in/reader034/viewer/2022042723/587165b61a28ab58758b556b/html5/thumbnails/3.jpg)
©2016 AKAMAI | FASTER FORWARDTM
Goal:Explore WPT public instances repeatability and consistency
![Page 4: Edge 2016 can webpagetest be trusted](https://reader034.fdocuments.in/reader034/viewer/2022042723/587165b61a28ab58758b556b/html5/thumbnails/4.jpg)
©2016 AKAMAI | FASTER FORWARDTM
● Thirty (30) URLs tested: Alexa top 10 from Shopping, Business, News
● Primarily focused on two metrics: onload time and speedIndex
● Chrome browser, 9 runs, medians, first view only, video on
● Tests ran between Sunday October 9 and Sunday October 16o All using APIs from python scriptso All URLs run in parallel from different agent locations
Experimental Setup
![Page 5: Edge 2016 can webpagetest be trusted](https://reader034.fdocuments.in/reader034/viewer/2022042723/587165b61a28ab58758b556b/html5/thumbnails/5.jpg)
©2016 AKAMAI | FASTER FORWARDTM
Reliability
![Page 6: Edge 2016 can webpagetest be trusted](https://reader034.fdocuments.in/reader034/viewer/2022042723/587165b61a28ab58758b556b/html5/thumbnails/6.jpg)
©2016 AKAMAI | FASTER FORWARDTM
Impact of location and retries on successful URLs (9 runs, 30 URLs)
Reliability
![Page 7: Edge 2016 can webpagetest be trusted](https://reader034.fdocuments.in/reader034/viewer/2022042723/587165b61a28ab58758b556b/html5/thumbnails/7.jpg)
©2016 AKAMAI | FASTER FORWARDTM
Lesson # 1:● Check number of successful runs
Especially from non EC2 locationsIf 9 runs needed, plan for a few retries
![Page 8: Edge 2016 can webpagetest be trusted](https://reader034.fdocuments.in/reader034/viewer/2022042723/587165b61a28ab58758b556b/html5/thumbnails/8.jpg)
©2016 AKAMAI | FASTER FORWARDTM
How many runs?
![Page 9: Edge 2016 can webpagetest be trusted](https://reader034.fdocuments.in/reader034/viewer/2022042723/587165b61a28ab58758b556b/html5/thumbnails/9.jpg)
©2016 AKAMAI | FASTER FORWARDTM
Impact of run number on main metrics, 30 URLs - Oct 14 run
How many runs?
![Page 10: Edge 2016 can webpagetest be trusted](https://reader034.fdocuments.in/reader034/viewer/2022042723/587165b61a28ab58758b556b/html5/thumbnails/10.jpg)
©2016 AKAMAI | FASTER FORWARDTM
Impact of run numbers on Coefficient of Variation (%) - Oct 14 run
How many runs?
![Page 11: Edge 2016 can webpagetest be trusted](https://reader034.fdocuments.in/reader034/viewer/2022042723/587165b61a28ab58758b556b/html5/thumbnails/11.jpg)
©2016 AKAMAI | FASTER FORWARDTM
Lesson # 2:● First run yields slightly higher
onload and speedIndex values● Variability seems to settle
around 10-15%, except for TTFB
![Page 12: Edge 2016 can webpagetest be trusted](https://reader034.fdocuments.in/reader034/viewer/2022042723/587165b61a28ab58758b556b/html5/thumbnails/12.jpg)
©2016 AKAMAI | FASTER FORWARDTM
Intra-Location consistency
![Page 13: Edge 2016 can webpagetest be trusted](https://reader034.fdocuments.in/reader034/viewer/2022042723/587165b61a28ab58758b556b/html5/thumbnails/13.jpg)
©2016 AKAMAI | FASTER FORWARDTM
Impact of concurrent runs in same locations on median onload time
Intra location consistency
Oct 13 Night Oct 14 Day Oct 16 Night
![Page 14: Edge 2016 can webpagetest be trusted](https://reader034.fdocuments.in/reader034/viewer/2022042723/587165b61a28ab58758b556b/html5/thumbnails/14.jpg)
©2016 AKAMAI | FASTER FORWARDTM
Impact of concurrent runs in same locations on median speedIndexIntra location consistency
Oct 13 Night Oct 14 Day Oct 16 Night
![Page 15: Edge 2016 can webpagetest be trusted](https://reader034.fdocuments.in/reader034/viewer/2022042723/587165b61a28ab58758b556b/html5/thumbnails/15.jpg)
©2016 AKAMAI | FASTER FORWARDTM
Lesson # 3:● Intra-location results within ~10 %
![Page 16: Edge 2016 can webpagetest be trusted](https://reader034.fdocuments.in/reader034/viewer/2022042723/587165b61a28ab58758b556b/html5/thumbnails/16.jpg)
©2016 AKAMAI | FASTER FORWARDTM
Inter-Location consistency
![Page 17: Edge 2016 can webpagetest be trusted](https://reader034.fdocuments.in/reader034/viewer/2022042723/587165b61a28ab58758b556b/html5/thumbnails/17.jpg)
©2016 AKAMAI | FASTER FORWARDTM
Impact of location (desktop fios) on median onload times
Inter location consistency
Oct 10 Night Oct 13 Night Oct 14 Day Oct 16 Night
![Page 18: Edge 2016 can webpagetest be trusted](https://reader034.fdocuments.in/reader034/viewer/2022042723/587165b61a28ab58758b556b/html5/thumbnails/18.jpg)
©2016 AKAMAI | FASTER FORWARDTM
Impact of location (desktop fios) on median speedIndex times
Inter location consistency
Oct 10 Night Oct 13 Night Oct 14 Day Oct 16 Night
![Page 19: Edge 2016 can webpagetest be trusted](https://reader034.fdocuments.in/reader034/viewer/2022042723/587165b61a28ab58758b556b/html5/thumbnails/19.jpg)
©2016 AKAMAI | FASTER FORWARDTM
Lesson # 4:● Results vary greatly with location
EC2 instances slower than private ?
![Page 20: Edge 2016 can webpagetest be trusted](https://reader034.fdocuments.in/reader034/viewer/2022042723/587165b61a28ab58758b556b/html5/thumbnails/20.jpg)
©2016 AKAMAI | FASTER FORWARDTM
RelativeInter-location consistency
![Page 21: Edge 2016 can webpagetest be trusted](https://reader034.fdocuments.in/reader034/viewer/2022042723/587165b61a28ab58758b556b/html5/thumbnails/21.jpg)
©2016 AKAMAI | FASTER FORWARDTM
Relative site speed ranking per location (%)
Relative consistency
● Dulles:Fios v.s. ec2-us-east-1:FiosOnload: 14 %speedIndex: 24 %
● ec2-us-west-1:Fios v.s. SanJose:FiosOnload: 15 %speedIndex: 15 %
![Page 22: Edge 2016 can webpagetest be trusted](https://reader034.fdocuments.in/reader034/viewer/2022042723/587165b61a28ab58758b556b/html5/thumbnails/22.jpg)
©2016 AKAMAI | FASTER FORWARDTM
Lesson # 5:● Relative speed ranking
(roughly) preserved
![Page 23: Edge 2016 can webpagetest be trusted](https://reader034.fdocuments.in/reader034/viewer/2022042723/587165b61a28ab58758b556b/html5/thumbnails/23.jpg)
©2016 AKAMAI | FASTER FORWARDTM
Accuracy
![Page 24: Edge 2016 can webpagetest be trusted](https://reader034.fdocuments.in/reader034/viewer/2022042723/587165b61a28ab58758b556b/html5/thumbnails/24.jpg)
©2016 AKAMAI | FASTER FORWARDTM
WPT results vs actual browser experience: onload times
Accuracy
Average Median Oct 16th run
![Page 25: Edge 2016 can webpagetest be trusted](https://reader034.fdocuments.in/reader034/viewer/2022042723/587165b61a28ab58758b556b/html5/thumbnails/25.jpg)
©2016 AKAMAI | FASTER FORWARDTM
Lesson # 6:● Location Dulles:Cable
(=default) closest to Home!
![Page 26: Edge 2016 can webpagetest be trusted](https://reader034.fdocuments.in/reader034/viewer/2022042723/587165b61a28ab58758b556b/html5/thumbnails/26.jpg)
©2016 AKAMAI | FASTER FORWARDTM
Emulation v.s. Real Device
![Page 27: Edge 2016 can webpagetest be trusted](https://reader034.fdocuments.in/reader034/viewer/2022042723/587165b61a28ab58758b556b/html5/thumbnails/27.jpg)
©2016 AKAMAI | FASTER FORWARDTM
Emulation vs real device (Moto. G) for various agents: onload timesEmulation v.s. Real Device
Oct 10 Night Oct 13 Night Oct 14 Day Oct 14 Night Oct 16 Night
Emulation
Emulation
![Page 28: Edge 2016 can webpagetest be trusted](https://reader034.fdocuments.in/reader034/viewer/2022042723/587165b61a28ab58758b556b/html5/thumbnails/28.jpg)
©2016 AKAMAI | FASTER FORWARDTM
Emulation vs real for various locations: speedIndex timesEmulation v.s. Real Device
Oct 10 Night Oct 13 Night Oct 14 Day Oct 14 Night Oct 16 Night
Emulation
Emulation
![Page 29: Edge 2016 can webpagetest be trusted](https://reader034.fdocuments.in/reader034/viewer/2022042723/587165b61a28ab58758b556b/html5/thumbnails/29.jpg)
©2016 AKAMAI | FASTER FORWARDTM
Lesson # 7:● Large (> 20%) discrepancies
between real device and emulation (Dulles + Motorola G)
![Page 30: Edge 2016 can webpagetest be trusted](https://reader034.fdocuments.in/reader034/viewer/2022042723/587165b61a28ab58758b556b/html5/thumbnails/30.jpg)
©2016 AKAMAI | FASTER FORWARDTM
More ...
![Page 31: Edge 2016 can webpagetest be trusted](https://reader034.fdocuments.in/reader034/viewer/2022042723/587165b61a28ab58758b556b/html5/thumbnails/31.jpg)
©2016 AKAMAI | FASTER FORWARDTM
Compare WPT onload with Rum onload
Compare behavior of public instances with private ones
Find out what’s causing discrepancies between locations
Throttle emulation CPUs to better approximate real devices
More ...
![Page 32: Edge 2016 can webpagetest be trusted](https://reader034.fdocuments.in/reader034/viewer/2022042723/587165b61a28ab58758b556b/html5/thumbnails/32.jpg)
©2016 AKAMAI | FASTER FORWARDTM
Summary
● Location has a huge impact on main performance metrics
● For a given location, allow for about 10-15% “repeatability margin”
● More testing is needed (surprise!)