WebPagetestPrivate InstancesPatrick Meenanpmeenan@webpagetest.org@patmeenan

The Basics

● Presentation is available on slideshare○ http://www.slideshare.net/patrickmeenan○ Mostly contains links to everything I’ll be showing

● Video will be uploaded to YouTube○ https://www.youtube.com/user/patmeenan

● I’ll send links and announcements of both to Twitter and Google+○ https://twitter.com/patmeenan - @patmeenan○ https://plus.google.com/u/0/+PatrickMeenan/posts

Sites being tested

WPT Web Server

Desktop Agent

System Architecture

Node.js Mobile AgentMobitest Agent

Sites being tested

WPT Web Server

Agent

Data Flow

User

Submit test (UI or API)http://<server>/runtest.php

Server:

- Creates test IDYYMMDD_<hash>_<num>

- Creates directory for testresults/YY/MM/DD/<hash>/<num>

- Writes test job to work directory for locationwork/jobs/<location>/<ID>.<priority>

- Adds test to queue file (lossy, can be re-built from the work directory)tmp/<location hash>.queue

- Redirects to result page

Sites being tested

WPT Web Server

Agent

User

Poll for test statushttp://<server>/result/<ID>/

Sites being tested

WPT Web Server

Agent

User

Poll for work for locationhttp://<server>/work/getwork.php

Sites being tested

WPT Web Server

Agent

User

- Launch Browser- Navigate to page- Measure/Collect Data

Sites being tested

WPT Web Server

Agent

User

Upload large files individually(images, tcpdump, timeline, etc)http://<server>/work/resultimage.php

Sites being tested

WPT Web Server

Agent

User

Complete run(or full test if all runs are done)http://<server>/work/workdone.php

Sites being tested

WPT Web Server

Agent

User

View completed testhttp://<server>/result/<ID>/

Server Config

● Any server that can run php○ Linux tends to work best○ Can run on the same PC as test agents if needed

● Copy www/ directory from distribution to site docroot● Configure rewrites for friendly URLs (optional)

○ .htaccess included for apache○ nginx.conf can be included into nginx server config

● Fix anything that http://<server>/install/ complains about○ PHP ini settings: allow_url_fopen, memory and post file size limits○ PHP modules: GD, zip, zlib, curl, apc, sqlite○ Command-line utilities: ffmpeg, imagemagick, jpegtran, exiftool○ Filesystem permissions (and directories)

Nginx configserver {

listen 80;server_name webpagetest.example.com;

root /var/www;

location ~ \.php$ {fastcgi_pass 127.0.0.1:9000;fastcgi_param SCRIPT_FILENAME /var/www$fastcgi_script_name;fastcgi_param HTTP_MOD_REWRITE On;include fastcgi_params;

}

include /var/www/nginx.conf;}

/install/

Server WPT Config

● Copy/modify settings/*.sample○ settings/settings.ini

■ Contact information■ Maximum number of runs■ Screen shot quality■ Show Slow integration■ Test archiving (covered later)

○ settings/locations.ini■ Definition for all of the test locations

Locations.ini

Grouping

Browsers

[locations]1=Group_12=Group_2default=Group_1

[Group_1]1=NYC2=NYC_Mobilelabel=”New York (Chrome, Firefox, IE, Android)”

[NYC]browser=Chrome,Firefox,IE 11label=”New York”

[NYC_Mobile]browser=Motorola G - Chrome,Motorola G - Chrome Betalabel=”New York”

[Group_2]...

[locations]1=Group_12=Group_2default=Group_1

[Group_1]1=NYC2=NYC_Mobilelabel=”New York (Chrome, Firefox, IE, Android)”

[NYC]browser=Chrome,Firefox,IE 11label=”New York”

[NYC_Mobile]browser=Motorola G - Chrome,Motorola G - Chrome Betalabel=”New York”

[Group_2]...

List of the group IDs in order to be displayed

[locations]1=Group_12=Group_2default=Group_1

[Group_1]1=NYC2=NYC_Mobilelabel=”New York (Chrome, Firefox, IE, Android)”

[NYC]browser=Chrome,Firefox,IE 11label=”New York”

[NYC_Mobile]browser=Motorola G - Chrome,Motorola G - Chrome Betalabel=”New York”

[Group_2]...

“Location ID”s that are part of the group in the order the browsers are to be displayed.

String displayed for the “Test Location”

[locations]1=Group_12=Group_2default=Group_1

[Group_1]1=NYC2=NYC_Mobilelabel=”New York (Chrome, Firefox, IE, Android)”

[NYC]browser=Chrome,Firefox,IE 11label=”New York”

[NYC_Mobile]browser=Motorola G - Chrome,Motorola G - Chrome Betalabel=”New York”

[Group_2]...

These are the location ID’s the agents connect toand that are used when submitting tests throughthe API

[locations]1=Group_12=Group_2default=Group_1

[Group_1]1=NYC2=NYC_Mobilelabel=”New York (Chrome, Firefox, IE, Android)”

[NYC]browser=Chrome,Firefox,IE 11label=”New York”

[NYC_Mobile]browser=Motorola G - Chrome,Motorola G - Chrome Betalabel=”New York”

[Group_2]...

All of the browsers from all of the locations in a group are collected and displayed in the Browser select box.

[locations]1=Group_12=Group_2default=Group_1

[Group_1]1=NYC2=NYC_Mobilelabel=”New York (Chrome, Firefox, IE, Android)”

[NYC]browser=Chrome,Firefox,IE 11label=”New York”

[NYC_Mobile]browser=Motorola G - Chrome,Motorola G - Chrome Betalabel=”New York”

[Group_2]...

All of the agents for a given location ID must support all of the browsers listed

[NYC]browser=Chrome,Firefoxlabel=”New York”

[NYC_IE10]browser=IE 10label=”New York”

[NYC_IE11]browser=IE 11label=”New York”

[NYC_Mobile]browser=Motorola G - Chrome,Motorola G - Chrome Betalabel=”New York”

A single agent can pull from multiple locations (configure it as a comma-separated list).

location=NYC,NYC_IE10

[locations]1=Group_12=Group_2default=Group_1

[Group_1]1=NYC2=NYC_Mobilelabel=”New York (Chrome, Firefox, IE, Android)”lat=40.712918lng=-74.005469group=North America

[NYC]browser=Chrome,Firefox,IE 11label=”New York”

[NYC_Mobile]browser=Motorola G - Chrome,Motorola G - Chrome Betalabel=”New York”

[Group_2]...

Add map=1 to settings.ini

[locations]1=Group_12=Group_2default=Group_1

[Group_1]1=NYC2=NYC_Mobilelabel=”New York (Chrome, Firefox, IE, Android)”lat=40.712918lng=-74.005469group=North America

[NYC]browser=Chrome,Firefox,IE 11label=”New York”

[NYC_Mobile]browser=Motorola G - Chrome,Motorola G - Chrome Betalabel=”New York”

[Group_2]...

[locations]1=Group_12=Group_2default=Group_1

[Group_1]1=NYC2=NYC_Mobilelabel=”New York (Chrome, Firefox, IE, Android)”

[NYC]browser=Chrome,Firefox,IE 11label=”New York”connectivity="Google Fiber (1Gbps)"key=123456agents=6notify=pmeenan@webpagetest.org,admin@webpagetest.orghidden=1

When traffic shaping is not supported. Shows up in the UI as the connectivity information.

[locations]1=Group_12=Group_2default=Group_1

[Group_1]1=NYC2=NYC_Mobilelabel=”New York (Chrome, Firefox, IE, Android)”

[NYC]browser=Chrome,Firefox,IE 11label=”New York”connectivity="Google Fiber (1Gbps)"key=123456agents=6notify=pmeenan@webpagetest.org,admin@webpagetest.orghidden=1

Shared key for securing agent communications (optional)

[locations]1=Group_12=Group_2default=Group_1

[Group_1]1=NYC2=NYC_Mobilelabel=”New York (Chrome, Firefox, IE, Android)”

[NYC]browser=Chrome,Firefox,IE 11label=”New York”connectivity="Google Fiber (1Gbps)"key=123456agents=6notify=pmeenan@webpagetest.org,admin@webpagetest.orghidden=1

Expected number of agents (for alerting)

[locations]1=Group_12=Group_2default=Group_1

[Group_1]1=NYC2=NYC_Mobilelabel=”New York (Chrome, Firefox, IE, Android)”

[NYC]browser=Chrome,Firefox,IE 11label=”New York”connectivity="Google Fiber (1Gbps)"key=123456agents=6notify=pmeenan@webpagetest.org,admin@webpagetest.orghidden=1

Email addresses for alerting when agents stop connecting.

[locations]1=Group_12=Group_2default=Group_1

[Group_1]1=NYC2=NYC_Mobilelabel=”New York (Chrome, Firefox, IE, Android)”

[NYC]browser=Chrome,Firefox,IE 11label=”New York”connectivity="Google Fiber (1Gbps)"key=123456agents=6notify=pmeenan@webpagetest.org,admin@webpagetest.orghidden=1

Hide the location from the UI (still works through the API and from http://<server>/?hidden=1 )

Desktop Agent Config

● Prepare Windows○ Auto logon○ Disable screen savers and screen power management (never sleep)○ Disable UAC○ Copy agent/* to local disk (c:/webpagetest/ in the examples)○ Install dummynet (optional for traffic shaping)○ Create startup shortcut to wptdriver.exe (and urlblast.exe if used)

● Install browsers○ Firefox can self-install (necessary for updates)

● Configure wptdriver.ini○ Server URL○ Location ID○ Browsers

wptdriver.ini[WebPagetest]url=http://127.0.0.1/location=Testbrowser=chromeTime Limit=120;key=TestKey123;Automatically install and update support software (Flash, Silverlight, etc)software=http://www.webpagetest.org/installers/software.dat

[chrome]exe="%PROGRAM_FILES%\Google\Chrome\Application\chrome.exe"options='--load-extension="%WPTDIR%\extension" --user-data-dir="%PROFILE%" --no-proxy-server'installer=http://www.webpagetest.org/installers/browsers/chrome.dat

[Firefox]exe="%PROGRAM_FILES%\Mozilla Firefox\firefox.exe"options='-profile "%PROFILE%" -no-remote'installer=http://www.webpagetest.org/installers/browsers/firefox.dattemplate=firefox

[Safari]exe="%PROGRAM_FILES%\Safari\Safari.exe"

[IE]exe="%PROGRAM_FILES%\Internet Explorer\iexplore.exe"

EC2 Test Agents

● AMI’s available in all regions○ IE8-11, Chrome, Firefox, Safari 5 (Windows)

● Pass configuration through user data when launching○ wpt_server=www.webpagetest.org wpt_loc=EC2_East

● No need to ever touch/login to instance● Kill/Start as needed● c1.medium or better recommended

EC2 Test Agents - User Data

EC2 Test Agents - Locations

Sites being tested

WPT Web Server

Mobile Agent (Node)

Node.js Mobile Agent

Mobile Agents - Android

● Android 4.4+ recommended● Root (Nexus and Moto phones are easiest)● Install Chrome and Chrome Beta from Play store● Configure Play to auto-update● Disable screen lock (security settings)● Configure developer mode

○ Enable USB debugging○ Disable screen sleep

Mobile Agents - iOS (alpha)

● Jailbreak● install/configure iOS webkit debug proxy● build openurl ipa

○ Needs to be signed with your dev key

Mobile Agents - Tethered host config

● Linux or Windows (Mac may work)○ Multiple phones per tethered agent

● Node.js● imagemagick● Launch wptdriver.bat (or wptdriver.sh)● Config is passed by command-line

○ Device ID○ Server URL○ Location ID

● Windows - consider using adbwatch○ Kills/restarts adb is it gets hung○ Launches/restarts agents if they crash

Sites being tested

WPT Web Server

Mobile Agent (Node) - Cell Network

Node.js Mobile Agent

Sites being tested

WPT Web Server

Mobile Agent (Node) - RNDIS

Node.js Mobile Agent

IPFW(Optional)

Sites being tested

WPT Web Server

Mobile Agent (Node) - WiFi

Node.js Mobile Agent

IPFW Bridge(Optional)WiFi Access

Point

Priority Queues

● Tests are pulled from queues based on test priority

● 10 Priority levels (0-9)○ 0 = highest, 9 = lowest

● Defaults○ 0 - Tests submitted through the UI○ 4 - Bulk tests submitted through the UI○ 5 - Tests submitted through the API

● Queue lengths reported on location check UI○ http://<server>/getLocations.php

https://www.flickr.com/photos/lunchtimemama/110765169/

API Keys

settings/keys.ini (disabled if file is missing)

[server]secret=7a5c742c021c414ca0dd1bec42e8f64bkey=acdd95754c654042a526847acb96f692

[acdd95754c654042a526847acb96f692]description=Web UIcontact=me@example.comlimit=0

[6d3a661b7401486b8a5d4f78962f8b80]description=API Keycontact=batch@example.comlimit=5000priority=8

[acd5ffd840094cdda054a0b7e57d4546]description=API Keycontact=notme@example.comlimit=100

API Keys[server]secret=7a5c742c021c414ca0dd1bec42e8f64bkey=acdd95754c654042a526847acb96f692

[acdd95754c654042a526847acb96f692]description=Web UIcontact=me@example.comlimit=0

[6d3a661b7401486b8a5d4f78962f8b80]description=API Keycontact=batch@example.comlimit=5000priority=8

[acd5ffd840094cdda054a0b7e57d4546]contact=notme@example.comlimit=100

Arbitrary stringused for hashing keys sent to browser(I usually use GUIDs)

API Keys[server]secret=7a5c742c021c414ca0dd1bec42e8f64bkey=acdd95754c654042a526847acb96f692

[acdd95754c654042a526847acb96f692]description=Web UIcontact=me@example.comlimit=0

[6d3a661b7401486b8a5d4f78962f8b80]description=API Keycontact=batch@example.comlimit=5000priority=8

[acd5ffd840094cdda054a0b7e57d4546]contact=notme@example.comlimit=100

Server API keyTo be used by the Web UI

API Keys[server]secret=7a5c742c021c414ca0dd1bec42e8f64bkey=acdd95754c654042a526847acb96f692

[acdd95754c654042a526847acb96f692]description=Web UIcontact=me@example.comlimit=0

[6d3a661b7401486b8a5d4f78962f8b80]description=API Keycontact=batch@example.comlimit=5000priority=8

[acd5ffd840094cdda054a0b7e57d4546]contact=notme@example.comlimit=100

Each API key defined as an ini file section

API Keys[server]secret=7a5c742c021c414ca0dd1bec42e8f64bkey=acdd95754c654042a526847acb96f692

[acdd95754c654042a526847acb96f692]description=Web UIcontact=me@example.comlimit=0

[6d3a661b7401486b8a5d4f78962f8b80]description=API Keycontact=batch@example.comlimit=5000priority=8

[acd5ffd840094cdda054a0b7e57d4546]contact=notme@example.comlimit=100

Contact info shows up in usage stats

API Keys[server]secret=7a5c742c021c414ca0dd1bec42e8f64bkey=acdd95754c654042a526847acb96f692

[acdd95754c654042a526847acb96f692]description=Web UIcontact=me@example.comlimit=0

[6d3a661b7401486b8a5d4f78962f8b80]description=API Keycontact=batch@example.comlimit=5000priority=8

[acd5ffd840094cdda054a0b7e57d4546]contact=notme@example.comlimit=100

Allowed “Page Loads” per day(tests x runs)x 2 if also recording repeat view

API Keys[server]secret=7a5c742c021c414ca0dd1bec42e8f64bkey=acdd95754c654042a526847acb96f692

[acdd95754c654042a526847acb96f692]description=Web UIcontact=me@example.comlimit=0

[6d3a661b7401486b8a5d4f78962f8b80]description=API Keycontact=batch@example.comlimit=5000priority=8

[acd5ffd840094cdda054a0b7e57d4546]contact=notme@example.comlimit=100

Forced test priorityDefaults to 5 for API(can be specified in request)

API Keys[server]secret=7a5c742c021c414ca0dd1bec42e8f64bkey=acdd95754c654042a526847acb96f692

[acdd95754c654042a526847acb96f692]description=Web UIcontact=me@example.comlimit=0

[6d3a661b7401486b8a5d4f78962f8b80]description=API Keycontact=batch@example.comlimit=5000priority=8

[acd5ffd840094cdda054a0b7e57d4546]contact=notme@example.comlimit=100

Minimal key configuration

Private Sites

WPT Web ServerIntranet Desktop Agent

Firewalls...

Public Sites

Public Desktop Agent

Server Lock-down

● Use keys for test locations● Open access to /work/* for test agents● Auth access (or IP rules) for rest of the UI

Private Sites

WPT Web Server

Intranet Desktop Agent

Firewalls...

Public Sites

Public Desktop Agent

WPT RelayServer

Relay Server

● Configure locations for public agents● Configure API keys for internal instance to use for relay● add headless=1 to settings.ini

○ Disbles UI for submitting tests, requires API key for all tests

● Results are pulled back to internal instance○ never visible on external

Relay Server - locations.ini

● Configure locations.ini like normal on the relay server● Local server configures 1:1 mapping of locations

○ forwarded to the relay server

[Public_EC2_East]browser=IE 11,Chrome,Firefoxlabel="EC2 East (Public)"relayServer="http://<relay server>/"relayKey=<your API key>relayLocation=EC2_East

API Key and Location IDon the Relay Server

Config like normal location

Results Storage

● Each test is self-contained in a directory under results/

● Grouped by year, month and day○ results/YY/MM/DD/…

● Prune old tests? Delete directories.

Results Archiving

● WPT can seamlessly restore missing tests from “archive”● Supports archiving to local path or S3-like storage● Configure archive settings in settings.ini● Test copied to archive after it completes (as 1 zip file per test)● To delete old archived tests need to run/schedule script

○ php cli/archive.php● Local Path:

○ zip files copied to specified directory○ YY/MM/DD/<hash>/<id>.zip

● S3-like○ zip files uploaded to bucket with <id>.zip filename○ Can restore from URL or S3 API

Results Archiving - Local Path

● Settings.ini○ archive_dir=/archive/○ archive_days=2

● Local Path:○ zip files copied to specified directory○ YY/MM/DD/<hash>/<id>.zip

● 2nd Level archiving○ settings.ini: archive2_dir=/mnt/archive/○ archive zip files merged into daily zip files○ cli/archive2.php archive script

Results Archiving - S3

● Settings.ini○ archive_s3_server=s3.amazonaws.com○ archive_s3_key=<access key>○ archive_s3_secret=<secret>○ archive_s3_bucket=<bucket>

● zip files uploaded to bucket with <id>.zip filename

● Can use URL access to restore○ Defaults to using API○ archive_s3_url=http://s3.amazonaws.com/

Bulk Testing in the UI (Private Instances)

Bulk Testing in the UI (Private Instances)

● Can be disabled through settings.ini○ noBulk=1

● Submit list of URLs to test

● UI options apply to each test

● Scripts can be run against each URL○ %URL% in script gets replaced with test URL○ %HOST% in script gets replaced with host from test URL○ %HOSTR% in script gets replaced with host from URL after redirects

● Aggregate results

API - Submitting a test

● Anything you can do with the UI you can do with the API○ If a field isn’t documented, just inspect the HTML on the UI to get the

field name :-)

● Make sure to encode parameters○ urlencode if GET○ form encode if POST

API Response

http://www.webpagetest.org/runtest.php?url=www.aol.com&f=json

{"statusCode":200,"statusText":"Ok","data":{

"testId":"140610_FY_N2F","ownerKey":"04121a1e5b17a59ba9ddd666ae020cf52cee4abe","jsonUrl":"http:\/\/www.webpagetest.org\/jsonResult.php?test=140610_FY_N2F","xmlUrl":"http:\/\/www.webpagetest.org\/xmlResult\/140610_FY_N2F\/","userUrl":"http:\/\/www.webpagetest.org\/result\/140610_FY_N2F\/","summaryCSV":"http:\/\/www.webpagetest.org\/result\/140610_FY_N2F\/page_data.csv","detailCSV":"http:\/\/www.webpagetest.org\/result\/140610_FY_N2F\/requests.csv"

}}

Node API Wrapper

Created by Marcel Duran (@marcelduran)

https://github.com/marcelduran/webpagetest-api

$ npm install webpagetest -g

webpagetest test -k <key> http://twitter.com/marcelduran

webpagetest status 140610_F0_T5K

webpagetest results 140610_F0_T5K

webpagetest test http://twitter.com/marcelduran --poll 5 --timeout 60

CI Integration

https://github.com/marcelduran/webpagetest-api/wiki/Test-Specs

Match result to provided specs{ "median": { "firstView": { "requests": 20, "render": 400, "loadTime": 3000, "score_gzip": { "min": 90 } } }}

CI Integration cont.

Uses Mocha with reporters suitable for integration with:● Jenkins● Travis-CI● Drone.io● Just about anything else you can imagine

grunt-perfbudget

Created by Tim Kadlec (@tkadlec)

https://github.com/tkadlec/grunt-perfbudget

Results logging

● As each test comes in, full result data is written to log files○ Page Data (load time, Speed index, requests, custom metrics, etc)○ Request Data (details for every individual request)

● Each test run or request is on it’s own log file line● JSON-formatted● Config information is included

○ Test URL○ Run #○ First/Repeat View○ Test label○ Location○ Browser○ Connectivity○ Test ID○ URL to test result

Results logging - integration

● Splunk● Logster

○ Logster -> StatsD = Arbitrary metrics trending automatically● Track individual requests across all pages

○ ads JS performance○ CDN response times○ Effectiveness of image compression○ so much more

Benchmarks (Private Instances)

● Still pretty experimental● Recurring tests run automatically

○ Configuration is a bit ugly but flexible● Tests are scheduled, run and aggregated automatically

○ Top-level trended aggregate view○ All-metrics trended aggregate view○ Per-page trended view○ Scatter Plot for a given run○ Filmstrip comparisons

Monitoring the System

● Hourly Agent Monitoring● getLocations.php● getTesters.php● checkTesters.php● Custom cron jobs/scripts

bit.ly/usingwebpagetest

Feedback Please!

Thank You!