Mozmill CIContinuous Integration with Mozmill

Henrik Skupin <hskupin@mozilla.com>Co-Lead of QA Automation Services

05.02.2012 - FOSDEM

quality.mozilla.org qa automation services

Overview

Benefits of Mozmill

Current situation in testing Firefox releases / builds

Goals with Continuous Integration

Current Status

Roadmap

Questions

BENEFITS OF MOZMILL

Test harness for functional UI driven tests

Support for any Gecko / XULRunner based application

Test Execution independent of the used locale

No need to create custom builds with tests enabled

Easy to setup due to our preconfigured test environment

Centralized reporting mechanism for test results

CURRENT SITUATION IN TESTING

Firefox Releases

Test cycles for candidate builds and application updates

Test execution across platforms and locales based on configuration file

Manual triggered download and execution via Pulse

Pulse triggers test machines to start wanted tests

CURRENT SITUATION IN TESTING

Daily Firefox builds

Currently run for Nightly, Aurora, and Namoroka

Tests triggered by crontab or Windows tasks

Tests run against pre-installed builds

Executed tests for en-US locale only:Update, Functional, L10n, Endurance, Remote, Add-ons

GOALS WITH CONTINUOUS INTEGRATION

Goals

Free-up human resources by killing the last manual tasks

Better hardware utilization with distributed test-runs

Flexible configuration of builds to test and machines

Immediate test execution when builds have been made available

Increase the number of builds and locales to test

CURRENT STATUS

Status for Firefox Releases

CURRENT STATUS

Status for Daily Firefox Builds

Note: Not everything has been implemented yet!

ROADMAP

Increase # of executors to run multiple tests concurrently on the same node

Extend hardware to execute tests for all localized daily builds like Nightly and Aurora

Switch the on-demand test framework for releases to use a Pulse consumer instead of configuration files

Tight integration with Case Conductor to combine automated and manual test results

ROADMAP

Setup a CI system for localized builds

Test tinderbox builds triggered by checkins from localizers

Execute L10n tests to prove quality of the localization (e.g. cut-off elements, access keys, ...)

Upload created reports and screenshots to a public dashboard for further processing

We need your ideas and your enthusiasm to create a good suite of L10n tests!

REFERENCES

QA Automation Services:https://wiki.mozilla.org/QA/Automation_Services

Mozmill CI for Release Builds:https://wiki.mozilla.org/QA/Automation_Services/Projects/Mozmill_Automation/On_Demand_Test_Framework

Mozmill CI for Daily Builds:https://wiki.mozilla.org/QA/Automation_Services/Projects/Mozmill_Automation/Triggered_Daily_Test-runs

Mozmill CI for L10n Builds:https://wiki.mozilla.org/QA/Automation_Services/Projects/Mozmill_Automation/Triggered_l10n_Test-runs

QUESTIONS & ANSWERS

Any Questions?

Thanks