WebPagetestPrivate InstancesPatrick [email protected]@patmeenan
The Basics
● Presentation is available on slideshare○ http://www.slideshare.net/patrickmeenan○ Mostly contains links to everything I’ll be showing
● Video will be uploaded to YouTube○ https://www.youtube.com/user/patmeenan
● I’ll send links and announcements of both to Twitter and Google+○ https://twitter.com/patmeenan - @patmeenan○ https://plus.google.com/u/0/+PatrickMeenan/posts
Sites being tested
WPT Web Server
Desktop Agent
System Architecture
Node.js Mobile AgentMobitest Agent
Sites being tested
WPT Web Server
Agent
Data Flow
User
Submit test (UI or API)http://<server>/runtest.php
Server:
- Creates test IDYYMMDD_<hash>_<num>
- Creates directory for testresults/YY/MM/DD/<hash>/<num>
- Writes test job to work directory for locationwork/jobs/<location>/<ID>.<priority>
- Adds test to queue file (lossy, can be re-built from the work directory)tmp/<location hash>.queue
- Redirects to result page
Sites being tested
WPT Web Server
Agent
User
Poll for test statushttp://<server>/result/<ID>/
Sites being tested
WPT Web Server
Agent
User
Poll for work for locationhttp://<server>/work/getwork.php
Sites being tested
WPT Web Server
Agent
User
- Launch Browser- Navigate to page- Measure/Collect Data
Sites being tested
WPT Web Server
Agent
User
Upload large files individually(images, tcpdump, timeline, etc)http://<server>/work/resultimage.php
Sites being tested
WPT Web Server
Agent
User
Complete run(or full test if all runs are done)http://<server>/work/workdone.php
Sites being tested
WPT Web Server
Agent
User
View completed testhttp://<server>/result/<ID>/
Server Config
● Any server that can run php○ Linux tends to work best○ Can run on the same PC as test agents if needed
● Copy www/ directory from distribution to site docroot● Configure rewrites for friendly URLs (optional)
○ .htaccess included for apache○ nginx.conf can be included into nginx server config
● Fix anything that http://<server>/install/ complains about○ PHP ini settings: allow_url_fopen, memory and post file size limits○ PHP modules: GD, zip, zlib, curl, apc, sqlite○ Command-line utilities: ffmpeg, imagemagick, jpegtran, exiftool○ Filesystem permissions (and directories)
Nginx configserver {
listen 80;server_name webpagetest.example.com;
root /var/www;
location ~ \.php$ {fastcgi_pass 127.0.0.1:9000;fastcgi_param SCRIPT_FILENAME /var/www$fastcgi_script_name;fastcgi_param HTTP_MOD_REWRITE On;include fastcgi_params;
}
include /var/www/nginx.conf;}
Server WPT Config
● Copy/modify settings/*.sample○ settings/settings.ini
■ Contact information■ Maximum number of runs■ Screen shot quality■ Show Slow integration■ Test archiving (covered later)
○ settings/locations.ini■ Definition for all of the test locations
Locations.ini
Grouping
Browsers
[locations]1=Group_12=Group_2default=Group_1
[Group_1]1=NYC2=NYC_Mobilelabel=”New York (Chrome, Firefox, IE, Android)”
[NYC]browser=Chrome,Firefox,IE 11label=”New York”
[NYC_Mobile]browser=Motorola G - Chrome,Motorola G - Chrome Betalabel=”New York”
[Group_2]...
[locations]1=Group_12=Group_2default=Group_1
[Group_1]1=NYC2=NYC_Mobilelabel=”New York (Chrome, Firefox, IE, Android)”
[NYC]browser=Chrome,Firefox,IE 11label=”New York”
[NYC_Mobile]browser=Motorola G - Chrome,Motorola G - Chrome Betalabel=”New York”
[Group_2]...
List of the group IDs in order to be displayed
[locations]1=Group_12=Group_2default=Group_1
[Group_1]1=NYC2=NYC_Mobilelabel=”New York (Chrome, Firefox, IE, Android)”
[NYC]browser=Chrome,Firefox,IE 11label=”New York”
[NYC_Mobile]browser=Motorola G - Chrome,Motorola G - Chrome Betalabel=”New York”
[Group_2]...
“Location ID”s that are part of the group in the order the browsers are to be displayed.
String displayed for the “Test Location”
[locations]1=Group_12=Group_2default=Group_1
[Group_1]1=NYC2=NYC_Mobilelabel=”New York (Chrome, Firefox, IE, Android)”
[NYC]browser=Chrome,Firefox,IE 11label=”New York”
[NYC_Mobile]browser=Motorola G - Chrome,Motorola G - Chrome Betalabel=”New York”
[Group_2]...
These are the location ID’s the agents connect toand that are used when submitting tests throughthe API
[locations]1=Group_12=Group_2default=Group_1
[Group_1]1=NYC2=NYC_Mobilelabel=”New York (Chrome, Firefox, IE, Android)”
[NYC]browser=Chrome,Firefox,IE 11label=”New York”
[NYC_Mobile]browser=Motorola G - Chrome,Motorola G - Chrome Betalabel=”New York”
[Group_2]...
All of the browsers from all of the locations in a group are collected and displayed in the Browser select box.
[locations]1=Group_12=Group_2default=Group_1
[Group_1]1=NYC2=NYC_Mobilelabel=”New York (Chrome, Firefox, IE, Android)”
[NYC]browser=Chrome,Firefox,IE 11label=”New York”
[NYC_Mobile]browser=Motorola G - Chrome,Motorola G - Chrome Betalabel=”New York”
[Group_2]...
All of the agents for a given location ID must support all of the browsers listed
[NYC]browser=Chrome,Firefoxlabel=”New York”
[NYC_IE10]browser=IE 10label=”New York”
[NYC_IE11]browser=IE 11label=”New York”
[NYC_Mobile]browser=Motorola G - Chrome,Motorola G - Chrome Betalabel=”New York”
A single agent can pull from multiple locations (configure it as a comma-separated list).
location=NYC,NYC_IE10
[locations]1=Group_12=Group_2default=Group_1
[Group_1]1=NYC2=NYC_Mobilelabel=”New York (Chrome, Firefox, IE, Android)”lat=40.712918lng=-74.005469group=North America
[NYC]browser=Chrome,Firefox,IE 11label=”New York”
[NYC_Mobile]browser=Motorola G - Chrome,Motorola G - Chrome Betalabel=”New York”
[Group_2]...
Add map=1 to settings.ini
[locations]1=Group_12=Group_2default=Group_1
[Group_1]1=NYC2=NYC_Mobilelabel=”New York (Chrome, Firefox, IE, Android)”lat=40.712918lng=-74.005469group=North America
[NYC]browser=Chrome,Firefox,IE 11label=”New York”
[NYC_Mobile]browser=Motorola G - Chrome,Motorola G - Chrome Betalabel=”New York”
[Group_2]...
[locations]1=Group_12=Group_2default=Group_1
[Group_1]1=NYC2=NYC_Mobilelabel=”New York (Chrome, Firefox, IE, Android)”
[NYC]browser=Chrome,Firefox,IE 11label=”New York”connectivity="Google Fiber (1Gbps)"[email protected],[email protected]=1
When traffic shaping is not supported. Shows up in the UI as the connectivity information.
[locations]1=Group_12=Group_2default=Group_1
[Group_1]1=NYC2=NYC_Mobilelabel=”New York (Chrome, Firefox, IE, Android)”
[NYC]browser=Chrome,Firefox,IE 11label=”New York”connectivity="Google Fiber (1Gbps)"[email protected],[email protected]=1
Shared key for securing agent communications (optional)
[locations]1=Group_12=Group_2default=Group_1
[Group_1]1=NYC2=NYC_Mobilelabel=”New York (Chrome, Firefox, IE, Android)”
[NYC]browser=Chrome,Firefox,IE 11label=”New York”connectivity="Google Fiber (1Gbps)"[email protected],[email protected]=1
Expected number of agents (for alerting)
[locations]1=Group_12=Group_2default=Group_1
[Group_1]1=NYC2=NYC_Mobilelabel=”New York (Chrome, Firefox, IE, Android)”
[NYC]browser=Chrome,Firefox,IE 11label=”New York”connectivity="Google Fiber (1Gbps)"[email protected],[email protected]=1
Email addresses for alerting when agents stop connecting.
[locations]1=Group_12=Group_2default=Group_1
[Group_1]1=NYC2=NYC_Mobilelabel=”New York (Chrome, Firefox, IE, Android)”
[NYC]browser=Chrome,Firefox,IE 11label=”New York”connectivity="Google Fiber (1Gbps)"[email protected],[email protected]=1
Hide the location from the UI (still works through the API and from http://<server>/?hidden=1 )
Desktop Agent Config
● Prepare Windows○ Auto logon○ Disable screen savers and screen power management (never sleep)○ Disable UAC○ Copy agent/* to local disk (c:/webpagetest/ in the examples)○ Install dummynet (optional for traffic shaping)○ Create startup shortcut to wptdriver.exe (and urlblast.exe if used)
● Install browsers○ Firefox can self-install (necessary for updates)
● Configure wptdriver.ini○ Server URL○ Location ID○ Browsers
wptdriver.ini[WebPagetest]url=http://127.0.0.1/location=Testbrowser=chromeTime Limit=120;key=TestKey123;Automatically install and update support software (Flash, Silverlight, etc)software=http://www.webpagetest.org/installers/software.dat
[chrome]exe="%PROGRAM_FILES%\Google\Chrome\Application\chrome.exe"options='--load-extension="%WPTDIR%\extension" --user-data-dir="%PROFILE%" --no-proxy-server'installer=http://www.webpagetest.org/installers/browsers/chrome.dat
[Firefox]exe="%PROGRAM_FILES%\Mozilla Firefox\firefox.exe"options='-profile "%PROFILE%" -no-remote'installer=http://www.webpagetest.org/installers/browsers/firefox.dattemplate=firefox
[Safari]exe="%PROGRAM_FILES%\Safari\Safari.exe"
[IE]exe="%PROGRAM_FILES%\Internet Explorer\iexplore.exe"
EC2 Test Agents
● AMI’s available in all regions○ IE8-11, Chrome, Firefox, Safari 5 (Windows)
● Pass configuration through user data when launching○ wpt_server=www.webpagetest.org wpt_loc=EC2_East
● No need to ever touch/login to instance● Kill/Start as needed● c1.medium or better recommended
EC2 Test Agents - User Data
EC2 Test Agents - Locations
Sites being tested
WPT Web Server
Mobile Agent (Node)
Node.js Mobile Agent
Mobile Agents - Android
● Android 4.4+ recommended● Root (Nexus and Moto phones are easiest)● Install Chrome and Chrome Beta from Play store● Configure Play to auto-update● Disable screen lock (security settings)● Configure developer mode
○ Enable USB debugging○ Disable screen sleep
Mobile Agents - iOS (alpha)
● Jailbreak● install/configure iOS webkit debug proxy● build openurl ipa
○ Needs to be signed with your dev key
Mobile Agents - Tethered host config
● Linux or Windows (Mac may work)○ Multiple phones per tethered agent
● Node.js● imagemagick● Launch wptdriver.bat (or wptdriver.sh)● Config is passed by command-line
○ Device ID○ Server URL○ Location ID
● Windows - consider using adbwatch○ Kills/restarts adb is it gets hung○ Launches/restarts agents if they crash
Sites being tested
WPT Web Server
Mobile Agent (Node) - Cell Network
Node.js Mobile Agent
Sites being tested
WPT Web Server
Mobile Agent (Node) - RNDIS
Node.js Mobile Agent
IPFW(Optional)
Sites being tested
WPT Web Server
Mobile Agent (Node) - WiFi
Node.js Mobile Agent
IPFW Bridge(Optional)WiFi Access
Point
Priority Queues
● Tests are pulled from queues based on test priority
● 10 Priority levels (0-9)○ 0 = highest, 9 = lowest
● Defaults○ 0 - Tests submitted through the UI○ 4 - Bulk tests submitted through the UI○ 5 - Tests submitted through the API
● Queue lengths reported on location check UI○ http://<server>/getLocations.php
https://www.flickr.com/photos/lunchtimemama/110765169/
API Keys
settings/keys.ini (disabled if file is missing)
[server]secret=7a5c742c021c414ca0dd1bec42e8f64bkey=acdd95754c654042a526847acb96f692
[acdd95754c654042a526847acb96f692]description=Web [email protected]=0
[6d3a661b7401486b8a5d4f78962f8b80]description=API [email protected]=5000priority=8
[acd5ffd840094cdda054a0b7e57d4546]description=API [email protected]=100
API Keys[server]secret=7a5c742c021c414ca0dd1bec42e8f64bkey=acdd95754c654042a526847acb96f692
[acdd95754c654042a526847acb96f692]description=Web [email protected]=0
[6d3a661b7401486b8a5d4f78962f8b80]description=API [email protected]=5000priority=8
[acd5ffd840094cdda054a0b7e57d4546][email protected]=100
Arbitrary stringused for hashing keys sent to browser(I usually use GUIDs)
API Keys[server]secret=7a5c742c021c414ca0dd1bec42e8f64bkey=acdd95754c654042a526847acb96f692
[acdd95754c654042a526847acb96f692]description=Web [email protected]=0
[6d3a661b7401486b8a5d4f78962f8b80]description=API [email protected]=5000priority=8
[acd5ffd840094cdda054a0b7e57d4546][email protected]=100
Server API keyTo be used by the Web UI
API Keys[server]secret=7a5c742c021c414ca0dd1bec42e8f64bkey=acdd95754c654042a526847acb96f692
[acdd95754c654042a526847acb96f692]description=Web [email protected]=0
[6d3a661b7401486b8a5d4f78962f8b80]description=API [email protected]=5000priority=8
[acd5ffd840094cdda054a0b7e57d4546][email protected]=100
Each API key defined as an ini file section
API Keys[server]secret=7a5c742c021c414ca0dd1bec42e8f64bkey=acdd95754c654042a526847acb96f692
[acdd95754c654042a526847acb96f692]description=Web [email protected]=0
[6d3a661b7401486b8a5d4f78962f8b80]description=API [email protected]=5000priority=8
[acd5ffd840094cdda054a0b7e57d4546][email protected]=100
Contact info shows up in usage stats
API Keys[server]secret=7a5c742c021c414ca0dd1bec42e8f64bkey=acdd95754c654042a526847acb96f692
[acdd95754c654042a526847acb96f692]description=Web [email protected]=0
[6d3a661b7401486b8a5d4f78962f8b80]description=API [email protected]=5000priority=8
[acd5ffd840094cdda054a0b7e57d4546][email protected]=100
Allowed “Page Loads” per day(tests x runs)x 2 if also recording repeat view
API Keys[server]secret=7a5c742c021c414ca0dd1bec42e8f64bkey=acdd95754c654042a526847acb96f692
[acdd95754c654042a526847acb96f692]description=Web [email protected]=0
[6d3a661b7401486b8a5d4f78962f8b80]description=API [email protected]=5000priority=8
[acd5ffd840094cdda054a0b7e57d4546][email protected]=100
Forced test priorityDefaults to 5 for API(can be specified in request)
API Keys[server]secret=7a5c742c021c414ca0dd1bec42e8f64bkey=acdd95754c654042a526847acb96f692
[acdd95754c654042a526847acb96f692]description=Web [email protected]=0
[6d3a661b7401486b8a5d4f78962f8b80]description=API [email protected]=5000priority=8
[acd5ffd840094cdda054a0b7e57d4546][email protected]=100
Minimal key configuration
Private Sites
WPT Web ServerIntranet Desktop Agent
Firewalls...
Public Sites
Public Desktop Agent
Server Lock-down
● Use keys for test locations● Open access to /work/* for test agents● Auth access (or IP rules) for rest of the UI
Private Sites
WPT Web Server
Intranet Desktop Agent
Firewalls...
Public Sites
Public Desktop Agent
WPT RelayServer
Relay Server
● Configure locations for public agents● Configure API keys for internal instance to use for relay● add headless=1 to settings.ini
○ Disbles UI for submitting tests, requires API key for all tests
● Results are pulled back to internal instance○ never visible on external
Relay Server - locations.ini
● Configure locations.ini like normal on the relay server● Local server configures 1:1 mapping of locations
○ forwarded to the relay server
[Public_EC2_East]browser=IE 11,Chrome,Firefoxlabel="EC2 East (Public)"relayServer="http://<relay server>/"relayKey=<your API key>relayLocation=EC2_East
API Key and Location IDon the Relay Server
Config like normal location
Results Storage
● Each test is self-contained in a directory under results/
● Grouped by year, month and day○ results/YY/MM/DD/…
● Prune old tests? Delete directories.
Results Archiving
● WPT can seamlessly restore missing tests from “archive”● Supports archiving to local path or S3-like storage● Configure archive settings in settings.ini● Test copied to archive after it completes (as 1 zip file per test)● To delete old archived tests need to run/schedule script
○ php cli/archive.php● Local Path:
○ zip files copied to specified directory○ YY/MM/DD/<hash>/<id>.zip
● S3-like○ zip files uploaded to bucket with <id>.zip filename○ Can restore from URL or S3 API
Results Archiving - Local Path
● Settings.ini○ archive_dir=/archive/○ archive_days=2
● Local Path:○ zip files copied to specified directory○ YY/MM/DD/<hash>/<id>.zip
● 2nd Level archiving○ settings.ini: archive2_dir=/mnt/archive/○ archive zip files merged into daily zip files○ cli/archive2.php archive script
Results Archiving - S3
● Settings.ini○ archive_s3_server=s3.amazonaws.com○ archive_s3_key=<access key>○ archive_s3_secret=<secret>○ archive_s3_bucket=<bucket>
● zip files uploaded to bucket with <id>.zip filename
● Can use URL access to restore○ Defaults to using API○ archive_s3_url=http://s3.amazonaws.com/
Bulk Testing in the UI (Private Instances)
Bulk Testing in the UI (Private Instances)
● Can be disabled through settings.ini○ noBulk=1
● Submit list of URLs to test
● UI options apply to each test
● Scripts can be run against each URL○ %URL% in script gets replaced with test URL○ %HOST% in script gets replaced with host from test URL○ %HOSTR% in script gets replaced with host from URL after redirects
● Aggregate results
API - Submitting a test
● Anything you can do with the UI you can do with the API○ If a field isn’t documented, just inspect the HTML on the UI to get the
field name :-)
● Make sure to encode parameters○ urlencode if GET○ form encode if POST
API Response
http://www.webpagetest.org/runtest.php?url=www.aol.com&f=json
{"statusCode":200,"statusText":"Ok","data":{
"testId":"140610_FY_N2F","ownerKey":"04121a1e5b17a59ba9ddd666ae020cf52cee4abe","jsonUrl":"http:\/\/www.webpagetest.org\/jsonResult.php?test=140610_FY_N2F","xmlUrl":"http:\/\/www.webpagetest.org\/xmlResult\/140610_FY_N2F\/","userUrl":"http:\/\/www.webpagetest.org\/result\/140610_FY_N2F\/","summaryCSV":"http:\/\/www.webpagetest.org\/result\/140610_FY_N2F\/page_data.csv","detailCSV":"http:\/\/www.webpagetest.org\/result\/140610_FY_N2F\/requests.csv"
}}
Node API Wrapper
Created by Marcel Duran (@marcelduran)
https://github.com/marcelduran/webpagetest-api
$ npm install webpagetest -g
webpagetest test -k <key> http://twitter.com/marcelduran
webpagetest status 140610_F0_T5K
webpagetest results 140610_F0_T5K
webpagetest test http://twitter.com/marcelduran --poll 5 --timeout 60
CI Integration
https://github.com/marcelduran/webpagetest-api/wiki/Test-Specs
Match result to provided specs{ "median": { "firstView": { "requests": 20, "render": 400, "loadTime": 3000, "score_gzip": { "min": 90 } } }}
CI Integration cont.
Uses Mocha with reporters suitable for integration with:● Jenkins● Travis-CI● Drone.io● Just about anything else you can imagine
grunt-perfbudget
Created by Tim Kadlec (@tkadlec)
https://github.com/tkadlec/grunt-perfbudget
Results logging
● As each test comes in, full result data is written to log files○ Page Data (load time, Speed index, requests, custom metrics, etc)○ Request Data (details for every individual request)
● Each test run or request is on it’s own log file line● JSON-formatted● Config information is included
○ Test URL○ Run #○ First/Repeat View○ Test label○ Location○ Browser○ Connectivity○ Test ID○ URL to test result
Results logging - integration
● Splunk● Logster
○ Logster -> StatsD = Arbitrary metrics trending automatically● Track individual requests across all pages
○ ads JS performance○ CDN response times○ Effectiveness of image compression○ so much more
Benchmarks (Private Instances)
● Still pretty experimental● Recurring tests run automatically
○ Configuration is a bit ugly but flexible● Tests are scheduled, run and aggregated automatically
○ Top-level trended aggregate view○ All-metrics trended aggregate view○ Per-page trended view○ Scatter Plot for a given run○ Filmstrip comparisons
Monitoring the System
● Hourly Agent Monitoring● getLocations.php● getTesters.php● checkTesters.php● Custom cron jobs/scripts
Feedback Please!
Thank You!
Top Related