Experience Report: Test Automation in an Agile Environment

Len VazOct 13, 2010

Software Development Process

• 1 week sprints• No UATs• Use Jira for Bug Tracking• Use Grass Hopper for user stories• Test cases tied to user stories• Corresponding blank test cases entered into

the automation framework• Automated builds using Hudson

Review the product

• Documentation• Engineering• Product organization

Breaking down system to test

installer

browser

console

Malware database

Malware analyzer

Test Environment

• Architecture• Hardware• Software• Tools

Architecture

Controller – Chameleon3

SUT – Chameleon4 SUT – Chameleon2 SUT – Chameleon1

SUT – chameleon5-8

Hudson – build machine1

2

3

4

5

6

78

9

10

1. Controller polls Hudson2. a: c:\rundir\doWeHaveANewBuid.py – builds a test package2. b: chameleonTest.py – reimages SUT (xp, vb, vu or w7), distributes test package and runs it3. Test package is distributed and run4. Results of tests are written back to a mysql database on the controller

An IIS server on the controller hosts a PHP server that has reports that display the test results

SUT – chameleon5-8 – future expansionxp - XP Professional SP3vb – Vista Business SP2vu – Vista Ultimate SP1w7 – Windows 7 SP1

Database schematests

PK,FK7 run_idPK id

FK2 test_idFK1 os_idFK5 testname_idFK6 testmachine_id reason start_time end_timeFK8 build_idFK3 status_idFK4 sp_idFK9 user_id

test

PK test_id

test_type

os

PK os_id

os_type

servicepacks

PK sp_id

sp_name

status

PK status_id

status_type

testname

PK testname_id

name description

testmachine

PK testmachine_id

name

run

PK run_id

testmachine_id start_time end_time

builds

PK build_id

name

user

PK user_id

name

performance

PK performance_test_run_id

FK1 testname_idFK2 testmachine_idFK3 os_idFK4 build_id duration

Resources

• Hardware• Software• People (Planning)• Standards

Automation

• IDE– Eclipse Downloads– Pydev - Python IDE for Eclipse

• Frameworks– PyUnit - Python Unit Testing Framework

• Setting up common code• Standards

Reporting

• Using Visualizations• Interactive Charts• Google Visualization API Reference

Report

Report Header

Report Details

Report Details

Report Performance

Lessons learned

• Continuous improvement• Continuous integration testing• No TDD – lot more work for QA• Lack of Infrastructure integrated testing• Estimation is not an easy process• Without reporting – no extra resources