Automating Visual Regression Testing
with BackstopJS@angelariggs_ PNSQC | October 2018
How It’s Useful and
How We Used It
@angelariggs_ PNSQC | October 2018
@angelariggs_ PNSQC | 2018
Front-end testing is hard
@angelariggs_ PNSQC | 2018
Mobile Desktop
Tablet
@angelariggs_ PNSQC | 2018
Fonts Images Layout
@angelariggs_ PNSQC | 2018
Fragile CSS ecosystems
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
Front-end testing is hard
@angelariggs_ PNSQC | 2018
The Problem: Disappearing content
on deploys
@angelariggs_ PNSQC | 2018
Manual testing was tedious & ineffective
@angelariggs_ PNSQC | 2018
BackstopJS
@angelariggs_ PNSQC | 2018
- Open-source package
@angelariggs_ PNSQC | 2018
- Open-source package - Before & after screenshots
@angelariggs_ PNSQC | 2018
- Open-source package - Before & after screenshots - Chrome Headless, PhantomJS, or Slimer
@angelariggs_ PNSQC | 2018
- Open-source package - Before & after screenshots - Chrome Headless, PhantomJS, or Slimer - UI simulations
@angelariggs_ PNSQC | 2018
- Open-source package - Before & after screenshots - Chrome Headless, PhantomJS, or Slimer - UI simulations - HTML report for comparison
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
Success: Automated visual regression testing
helped us find and fix regressions before it became the client’s problem
@angelariggs_ PNSQC | 2018
But…
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
70 URLs
@angelariggs_ PNSQC | 2018
70 URLs x 3 CSS selectors
@angelariggs_ PNSQC | 2018
70 URLs x 3 CSS selectors x 2 viewports
@angelariggs_ PNSQC | 2018
70 URLs x 3 CSS selectors x 2 viewports 420 screenshots
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
1. Run screenshot references
@angelariggs_ PNSQC | 2018
1. Run screenshot references 2. Go play Towerfall for 35 minutes
@angelariggs_ PNSQC | 2018
1. Run screenshot references 2. Go play Towerfall for 35 minutes 3. Developer deploys
@angelariggs_ PNSQC | 2018
1. Run screenshot references 2. Go play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests
@angelariggs_ PNSQC | 2018
1. Run screenshot references 2. Go play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests 5. Go have lunch
@angelariggs_ PNSQC | 2018
1. Run screenshot references 2. Go play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests 5. Go have lunch 6. Review report
@angelariggs_ PNSQC | 2018
1. Run screenshot references 2. Go play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests 5. Go have lunch 6. Review report 7. Feedback to developer
@angelariggs_ PNSQC | 2018
1. Run screenshot references* 2. Go play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests* 5. Go have lunch 6. Review report 7. Feedback to developer
@angelariggs_ PNSQC | 2018
* Remember to include “no-timeout” parameter
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
1. Run screenshot references 2. Go play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests 5. Go have lunch 6. Review report* 7. Feedback to developer
@angelariggs_ PNSQC | 2018
“Automated”
@angelariggs_ PNSQC | 2018
Narrowing the Scope
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
- Increase viewport height to grab entire page - One CSS selector per URL - Remove URLs that use the same template
@angelariggs_ PNSQC | 2018
420 screenshots
@angelariggs_ PNSQC | 2018
420 screenshots ➡ 40 screenshots
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
1. Run screenshot references* 2. Go play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests* 5. Go have lunch 6. Review report* 7. Feedback to developer
@angelariggs_ PNSQC | 2018
1. Tester runs screenshot references 2. Developer deploys 3. Tester runs screenshot test 4. Review report 5. Feedback to developer
@angelariggs_ PNSQC | 2018
Smoke test for all staging and prod deploys
@angelariggs_ PNSQC | 2018
But…
@angelariggs_ PNSQC | 2018
- Four separate configuration files
@angelariggs_ PNSQC | 2018
- Four separate configuration files - Lots of back & forth between developer and tester
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
More automated Still tedious
@angelariggs_ PNSQC | 2018
- Developers oversee deploys
@angelariggs_ PNSQC | 2018
- Developers oversee deploys - Developers should oversee deploy tests
@angelariggs_ PNSQC | 2018
Make it Easy
@angelariggs_ PNSQC | 2018
Custom JavaScript configuration
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
--env=prod
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
4 config files
@angelariggs_ PNSQC | 2018
4 config files ➡ 1 config file
@angelariggs_ PNSQC | 2018
- detailed README for setup and use - template repo for quick setup on new projects
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
But…
@angelariggs_ PNSQC | 2018
backstop reference--configPath=backstop.js--pathFile=paths --env=prod
@angelariggs_ PNSQC | 2018
backstop reference--configPath=backstop.js--pathFile=paths --env=prod
@angelariggs_ PNSQC | 2018
backstop reference--configPath=backstop.js--pathFile=paths --env=prod
@angelariggs_ PNSQC | 2018
backstop reference--configPath=backstop.js--pathFile=paths --env=prod
@angelariggs_ PNSQC | 2018
backstop reference--configPath=backstop.js--pathFile=paths --env=prod--refHost=http://prod-site.com
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
make prod-reference
@angelariggs_ PNSQC | 2018
make prod-test
@angelariggs_ PNSQC | 2018
@angelariggs_ PNSQC | 2018
- Smoke tests for all deploys on all projects
@angelariggs_ PNSQC | 2018
1. Dev runs references 2. Dev deploys 3. Dev runs tests 4. Dev reviews the report
@angelariggs_ PNSQC | 2018
- Smoke tests for all deploys on all projects - Regression test during feature testing
@angelariggs_ PNSQC | 2018
1. Run references on master branch 2. Check out feature branch 3. Run tests on feature branch 4. Review the report
@angelariggs_ PNSQC | 2018
- Smoke tests for all deploys on all projects - Regression test during feature testing - Local use during feature development
@angelariggs_ PNSQC | 2018
1. Create and check out feature branch 2. Run references on fresh branch 3. Do local development 4. Run tests on branch periodically 5. Review report
@angelariggs_ PNSQC | 2018
Front-end testing is hard easy
@angelariggs_ PNSQC | 2018
Problem ➡ Solution
@angelariggs_ PNSQC | 2018
Problem ➡ Idea
@angelariggs_ PNSQC | 2018
Problem ➡ Idea ➡ Iterate
@angelariggs_ PNSQC | 2018
Problem ➡ Idea ➡ Iterate ➡ Iterate
@angelariggs_ PNSQC | 2018
Problem ➡ Idea ➡ Iterate ➡ Iterate ➡ Solution
@angelariggs_ PNSQC | 2018
BackstopJS
@angelariggs_ PNSQC | 2018
BackstopJS ➡ Refined configuration
@angelariggs_ PNSQC | 2018
Refined config ➡ Customized config
@angelariggs_ PNSQC | 2018
Automation is useful*
@angelariggs_ PNSQC | 2018
Find the right automation solution for your problem
@angelariggs_ PNSQC | 2018
Automated testing requires manual work
@angelariggs_ PNSQC | 2018
Front-end testing is easy easier with the right automation
https://github.com/angelariggs/visual-regression
@angelariggs_ PNSQC | October 2018
Thank you!
@angelariggs_ PNSQC | October 2018
Top Related