Adam CarmiCo-Founder and VP R&D at Applitools

AUTOMATED VISUAL TESTING IN THE CLOUD

Enhance Your Cross-browser Coverage

Chris BroesamleSales Engineer at Sauce Labs

AGENDA

Why automated visual testing? Challenges in visual test automation Demo #1: Visual test automation with Applitools Eyes Demo #2: Cross-browser testing in the cloud with SauceLabs Demo #3: Running a cross-browser visual test in the cloud When to perform automated visual testing? Q&A

WHAT IS VISUAL TESTING?

A quality assurance activity aimed to verify that a Graphical User Interface appears correctly to users

A VISUAL BUG

AND ANOTHER…

WHY SHOULD IT BE AUTOMATED?

THE TEST MATRIX IS TOO BIG TO APPROXIMATE MANUALLY Web browsers Devices Operating systems Screen resolutions Responsive designs L10n

WHY SHOULD IT BE AUTOMATED?

WHY SHOULD IT BE AUTOMATED?

NATIVE / HYBRID MOBILE APPS Harder to roll back changes

Can’t push daily

Updates take battery and data

Higher quality bar

WHY SHOULD IT BE AUTOMATED?

Many are already doing it…

PhantomCSS Fighting Layout Bugs

CSS Critc

Wraith

Needle

Grunt PhotoBox

dpdxt

WebdriverCSS

EyesHuxley

FBSnapshotTestCase

GeminiSelenium Visual Diff

VisualCeption

Specter

Snap And Compare

AGENDA

Why automated visual testing? Challenges in visual test automation Demo #1: Visual test automation with Applitools Eyes Demo #2: Cross-browser testing in the cloud with SauceLabs Demo #3: Running a cross-browser visual test in the cloud When to perform visual testing? Q&A

THE WORKFLOW

Drive the AUT and take screenshots

Compare screenshots with baseline images

Report differences

Update the baseline

THE CHALLENGES

Drive the AUT and take screenshots

Compare screenshots with baseline images

Report differences

Update the baseline

SCALABILITY CHALLENGES

Image comparison False positives

Dynamic data

Baseline maintenance Multiple baselines

Non-locality of changes

WHY NOT?

FALSE POSITIVES

ANTI ALIASING 1/2

ANTI ALIASING 2/2

BRIGHTNESS 1/2

BRIGHTNESS 2/2

SUB PIXEL SHIFTS 1/2

SUB PIXEL SHIFTS 2/2

IMAGE SCALING 2/2

IMAGE SCALING 2/2

ANDMORE…

1 pixel offsets in element positioning

Dynamic content

Moving elements

Images of different size

Performance

AGENDA

Why automated visual testing?

Challenges in visual test automation

Demo #1: Visual test automation with Applitools Eyes

Demo #2: Cross-browser testing in the cloud with SauceLabs

Demo #3: Running a cross-browser visual test in the cloud

When to perform visual testing?

Q&A

Chris Broesamle

Agenda

• Sauce Labs Overview

• Demo of Selenium on Sauce OnDemand

What is Sauce Labs?

We’ve built the worlds largest grid for

executing Selenium & Appium tests. We’re

open-source at heart and contribute to both

projects.

Selenium Grid

• A local test script is created using Remote

WebDriver, which sends test commands to a grid

• A grid consists of a hub server that directs requests

and node servers that host OS/browser

configurations for execution

• Your local hub might be “gridhub.qa.mypage.com”

• All you need to do is change that to

“ondemand.saucelabs.com”

Local Selenium Grid

HubReceives test

requests and

relays them

to Desired

Node

NodesExecute

commands

on browsers

as directed

by the hub

Selenium on Sauce

450+configurations

Sauce Labs Infrastructure

• Environments are kept up to date by our engineering

team

• Early access to browsers depends on browser

developers

• Browsers are retired as OS no longer supports it or

WebDriver versions are deprecated

• Selenium versions are chosen at run time based on

compatibility with desired browser

Integrations

• Plugins for Jenkins, Bamboo, TeamCity, and

Travis CI

• Manual testers can open defects in Jira from

Sauce

• Plugins handle authentication with Sauce,

launching Sauce Connect, OS/browser selection

in a build, and integration of results

Sauce Labs REST API

• Authentication

• Provisioning

• Jobs (including test results)

• Information

• Usage

• Tunnels

• Storage

Selenium on Sauce

• Demo

AGENDA

Why automated visual testing?

Challenges in visual test automation

Demo #1: Visual test automation with Applitools Eyes

Demo #2: Cross-browser testing in the cloud with SauceLabs

Demo #3: Visual test automation in the cloud

When to perform visual testing?

Q&A

WHERE DOES IT FIT?

• Component

s

• Code

review

• Developers

• Designers

• QA

WHERE DOES IT FIT?

• Pages

• Page

sections

• Developers

• Designers

• QA

• Others

WHERE DOES IT FIT?

• Staging vs.

Production

• Ops

• QA

WHERE DOES IT FIT?

• Monitoring

• Ops

• QA

QUESTIONS?