Crowd-sourced Automated Firefox UI Testing

Crowd-sourced AutomatedFirefox UI Testing

Henrik Skupin <[email protected]>Lead of QA Test Automation

quality.mozilla.org qa automation

Thursday, November 4, 2010

mailto:[email protected]

mailto:[email protected]

Who is Mozilla QA?


WHO IS MOZILLA QA

Group of about 30 employees / contractors

Are you serious?

No!


WHO IS MOZILLA QA

1000s of community members across the globe

who are partners to the Mozilla QA core team


Mozilla Scale

400+Developers

10,000+Nightly Testers

2,500,000Beta Testers

400,000,000Users

400,000,000

350,000,000

300,000,000

250,000,000

200,000,000

150,000,000

100,000,000

50,000,0000Thursday, November 4, 2010

What are our Areas of Work?


• Testing new features in beta versions and nightly builds

• i.e. Switch to Tab, Add-ons Manager, ...

• Testing release builds before they get shipped

• Smoketests across all platforms

• ~ 300 basic functional tests (BFT)

• Software update tests

WHAT ARE OUR AREAS OF WORK?


A Brief History of Automation in Mozilla QA


A BRIEF HISTORY OF AUTOMATION IN MOZILLA QA

•No automated functional tests existent by early 2009

• 100% manual testing of new Firefox features

• 100% manual testing of release builds (limited to a single platform)

Huge amount of manual resources necessary!



So what has been changed so far?



• Integration of MozMill (based on WindMill) for functional tests

• Automation of basic tests (30% done - ~100 tests)

• Automation of update tests (nearly done - Only 1single person needed to run 192 different update paths across 5 platforms and 4 P1 locales in under 2 hours)

Perfect time to open up testing for everyone


Crowd Sourced Testing


CROWD SOURCED TESTING

Jeff Howe: "It’s not outsourcing; it’s crowdsourcing."



Crowdsourcing relies on people, and thus works when participants are treated as people



What can be achieved?

• Support testers, localizers, and add-on authors to allow them to focus on their main tasks

• Constantly testing builds across platforms and locales

• Assembling test results from a large number of different environments (settings and performance)

How can this be done?


MozMill Crowd Extension


MOZMILL CROWD EXTENSION

Pre-requisites:

• Trivial setup (“One click install”)

• Uses MozMill to run tests in any version of Firefox

• Support for different test-runs (basic tests, add-ons, ...)

• Reporting test results to a central database

Under Construction


MOZMILL CROWD EXTENSION

What remains:

• How reliable are our tests in the crowd

• Fix focus issues to allow background tests (webdriver)

• Analyzing results based on the environment

• ...


DEMO

What will be shown?

• Setup of the test environment by using MozMill-Crowd

• Test-run for a subset of general Firefox tests

• Analyzing test results

• Test-run for the Google Toolbar extension


REFERENCES

QA Test Automation team page:http://quality.mozilla.org/teams/automation/

Mozmill Tests Repository:https://developer.mozilla.org/en/Mozmill_Tests

Mozmill Code:https://github.com/mozautomation/mozmill

Mozmill Crowd Extension Project:https://wiki.mozilla.org/QA/Mozmill_Test_Automation/Crowd_Testing


http://quality.mozilla.org/teams/automation/

http://quality.mozilla.org/teams/automation/

https://developer.mozilla.org/en/Mozmill_Tests

https://developer.mozilla.org/en/Mozmill_Tests

https://github.com/mozautomation/mozmill

https://github.com/mozautomation/mozmill

QUESTIONS & ANSWERS

Questions?


Thanks


Crowd-sourced Automated Firefox UI Testing

Technology

Transcript of Crowd-sourced Automated Firefox UI Testing