1
Making Test Automation Work in
Agile Projects
ØREDEV 2012 Lisa Crispin
With Material from Janet Gregory
2
Introductions: Experience, Goals
§ Form groups: § New to automation § One to two years experience w/ automation § More than two years experience
§ Talk with at least two other people § Tell each other your learning goals for today § Note the most interesting one you hear
Copyright 2012: Lisa Crispin 5
Vive la Rebellion!
Successful test automation is an innovation! • Let go of traditional roles, reach across them • Visualize how tests should look • Be smart about when to automate • Experiment to overcome obstacles • Find/build the right tools
6
Exercise: Your Learning Goals
§ Write one interesting learning goal you heard from another participant at the start of the class on a sticky note.
§ Write your number one learning goal for today on a sticky note.
§ Put the sticky notes on your “learning goals” sheet
§ Group similar ones together
Copyright 2012: Lisa Crispin 7
Let’s start by defining “agile”
Agile teams: q Deliver business value
frequently q at a sustainable pace q while adapting to the changing
needs of the business Source: Elisabeth Hendrickson
Copyright 2012: Lisa Crispin 11
Exercise: Your barriers Individually and silently – write one barrier
hindering your team per sticky note Put these on the “impediments” wall chart
for your table group Talk to your fellow group members – do you
see any patterns?
Copyright 2012: Lisa Crispin 13
The Whole-Team Approach § Automated tests are code § Respecting the tests § Collaborating § Commitment to quality § Return on investment
Copyright 2012: Lisa Crispin 14
Automated tests are code public class CalculatorFixture extends
ColumnFixture { public String startDate; public String endDate; public double startBalance; public double endBalance; public String irrTarget; private Calculator calculator = new
Calculator(); private Double irr;
Copyright 2012: Lisa Crispin 17
Testers are especially good at… § Eliciting examples § Turning them into tests § Ensuring the right testing gets done § Exploratory testing
Copyright 2012: Lisa Crispin 20
Experiment: Iteration 1 § Pair up: one will be tester, one programmer § Sit back to back so you face away from each
other § Tester gets a drawing which needs to be
replicated § Tester tells the programmer what to draw, all at
one time. § Programmer draws the shapes based on what
the tester explained. § No talking during ‘coding’!
§ Tester “tests” the drawing, reports “bugs” on index cards
§ Programmer fixes the “bugs” § How long did it take? Will the customer be
happy?
Copyright 2012: Lisa Crispin 21
Experiment: Iteration 2 § Collaborate! § Tester tells programmer what to draw, watches
the programmer draw, answers questions, points out ‘defects’ for programmer to fix immediately § (Don’t show the programmer the drawing, that
makes it too easy, we’re trying to simulate real coding)
§ How long did it take? Will the customer be happy?
Thanks to the members of the agile-games group and Kane Mar for ideas & pictures for this game
Copyright 2012: Lisa Crispin 22
Ways to collaborate
Pair: Tester-Coder Tester-Tester Any role – any role
Copyright 2012: Lisa Crispin
Under-commit
Plan less work than you think you can do Including all test automation
Copyright 2012: Lisa Crispin 27
Learn to write maintainable tests § Get over the “hump of pain”
From Gerard Meszaros’ XUnit Test Patterns
Whole-team software development
28
Create a user story
Write Customer (Q2) Tests
Start thinking how to code
Expand tests – Story Tests
Automate Q2 Tests
Exploratory testing
TDD
Customer User Acceptance
Product owner Product owner/ Tester Tester Tester/Programmer Programmer
Pair, “Show Me”
Copyright 2012: Lisa Crispin
Exercise
• Think of an experiment to get your whole team engaged in automating tests • Share with your group
Copyright 2012: Lisa Crispin 30
Getting Over the Hump § The test automation pyramid § The agile testing quadrants § What should be automated § What shouldn't § Difficult areas
Copyright 2012: Lisa Crispin 35
What Should We Automate? § Quadrant 1 tests
§ Unit, component, TDD § Quadrant 2 tests
§ API, service-level § Quadrant 4 tests
§ Load, performance, stress § Quadrant 3 tests?
§ Leverage automation where useful
Copyright 2012: Lisa Crispin 36
What Shouldn’t We Automate?
§ Quadrant 2 tests § Wizard of Oz, prototyping
§ Quadrant 3 tests § Usability, UAT, ET
§ Tests that will never fail? § Assess risk
§ ROI not enough § One-off tests
Copyright 2012: Lisa Crispin 37
Where Should We Be Careful?
§ GUI tests § Watch ROI
§ End-to-End tests § Push testing down to lowest level
§ Remember the Pyramid
Copyright 2012: Lisa Crispin 38
Hard to Automate?
§ Legacy code § Hard to automate, or just lack of skill? § “Working Effectively with Legacy Code” –
Feathers § “Strangling” – Fowler, Thomas
Copyright 2012: Lisa Crispin 40
Agile Automation Strategy
§ What hurts the most § Layered approach § Applying agile principles
§ Small chunks/thin slices § Smart test design
§ Choosing the right tools
Copyright 2012: Lisa Crispin 41
What Hurts the Most § Use retrospectives § Keep an impediment backlog
Copyright 2012: Lisa Crispin 42
Multi-Layered Approach Example:
§ Learn TDD at unit level § Automate GUI smoke tests
Copyright 2012: Lisa Crispin 43
Simplicity
§ Make the biggest problem 20% better § Understand the problem first § Try simplest approach first § Work in small chunks, thin slices § Incremental & iterative
Copyright 2012: Lisa Crispin 44
Automate a Slice at a Time
Example: 4-step UI to validate, upload profit sharing contribution data
• Thread 1: All four pages with navigation • Thread 2: Select year, enter description on page
1, display on page 2, browse and upload file on page 2
• Thread 3: Validate data in file, display on page 3 • Thread 4: Persist data, display ‘success’
message on page 4
Copyright 2012: Lisa Crispin 46
Exercise: Thin Slices
Here’s our user story: As an internet shopper, I want to create an account so that I do not have to enter my address and billing information each time I make a purchase Draw a mind map for this feature on a big sheet of paper Identify a basic end-to-end slice of functionality that can be coded, tested, and automated. If you have time, identify additional slices.
47
Test Design Patterns/Principles § Patterns, eg, Page
Object § One clear purpose § Don’t Repeat Yourself
Copyright 2012: Lisa Crispin 48
Page Object Pattern A “page object” is a test object that holds the details of all the elements on a web page that might be involved in an automated test.
Copyright 2012: Lisa Crispin 49
Page Object Pattern
§ Test object holds all details of one page § Hide details and eliminate duplication
Go To ePS Admin Page Click Link eps admin Wait Until Page Contains plan administration Location Should Be ${EPSADMIN URL}
Copyright 2012: Lisa Crispin 50
Facilitating Exploratory Testing
§ Templates to set up tests § Consoles to facilitate visual inspection
Copyright 2012: Lisa Crispin 52
Iterative Feedback
§ Spike two different approaches § Pick one to try for N # of iterations § Use retrospectives to evaluate
Copyright 2012: Lisa Crispin 53
Learn by Doing § Courage – don’t be afraid to fail § Production code practices for test code § Incremental, thin slices § Experiment
Copyright 2012: Lisa Crispin 55
Choosing Tools
§ Team effort § Time § Requirements § Focus on goals, problems, not tools. § Experiment
Copyright 2012: Lisa Crispin 56
Understand the Purpose
§ Who’s using the tests? What for? § What’s being automated? § Existing tools, environment § Who’s doing what for automating?
Copyright 2012: Lisa Crispin 57
What Fits Your Situation • Existing skills • Language of application under test • Collaboration needs • What’s being automated • Life span, future use of tests
Copyright 2012: Lisa Crispin 58
Test Drivers/Frameworks § Divide into sub-teams and prototype
solutions § Have a “bake-off” showing each prototype
Copyright 2012: Lisa Crispin 59
Where To Find Tools
§ http://bit.ly/AgileTestTools § www.softwareqatest.com/qattls1.html § www.testingfaqs.org § www.opensourcetesting.org § groups.yahoo.com/group/agile-testing § Grow your own!
Copyright 2012: Lisa Crispin 60
Example: One Team’s Tool Choices
• IntelliJ Idea • Jenkins, ant, Maven • JUnit • FitNesse • Canoo WebTest • Watir • Robot Framework • JMeter • Selenium 2.0 / WebDriver with Geb framework
Copyright 2012: Lisa Crispin 61
Exercise: Test Frameworks
• What format might work for your team’s tests? • Write examples on sticky notes
• What patterns, frameworks and drivers do you want to learn more about?
• Place sticky notes next to impediment they might help overcome
• Discuss how this would help solve problem
Copyright 2012: Lisa Crispin 62
Making Test Automation Work
§ Time to do it right § Experiments, bake-offs § Testable architecture § Test data § Managing tests
Copyright 2012: Lisa Crispin 63
Time To Do It Right
§ Limit scope, don’t over-commit § Write automation task cards § Quality must be team goal § Long-term, will let you go faster
Copyright 2012: Lisa Crispin 65
Testable Architecture • Layered architecture
• eg. UI, business logic, data access • Ports and Adapters pattern
• App can work without UI or database • Ports accept outside events • Adapters convert for human or automated
users
Copyright 2012: Lisa Crispin 66
Test Data
§ Avoid database access when possible § Setup/Teardown
§ Independent, rerunnable tests § Canonical data
§ Refresh before each test run § Customizable data for ET § Production-like data
§ Get customers to provide example data
Copyright 2012: Lisa Crispin 67
Managing Automated Tests
§ Tests as Living Documentation § Continuous Integration § Reporting results § Metrics
Copyright 2012: Lisa Crispin 68
Tests as Living Documentation § Understandable § Who will really use them? § Once passing, must always pass
Copyright 2012: Lisa Crispin 70
Given/Then/When Example
Scenario: Valid name search returns results GIVEN that Kant is a supervisor with employees AND Kant has an employee named Smith WHEN Kant navigates to the employee name
search page AND enters the value “S” THEN Kant will see a search result that includes
Smith
Copyright 2012: Lisa Crispin 73
Test Management Tools § Manage tests, code together § Some tools have own management § What problem are you trying to solve?
Copyright 2012: Lisa Crispin 74
Exercise: Tests as Documentation
• Pair up • Walk around the room and talk about
what approach may work for your team • BDD? • Tabular style? • Examples? • Something else? • How will people in different roles
use them?
75
Key Success Factors
Whole Team Approach
Thin Slices
Short Feedback Loops
Experiments
Specification by Example
Copyright 2012: Lisa Crispin 77
Remember § It’s a team problem! § Tackle automation problems with diversity § Try small experiments § Start the test automation rebellion with baby
steps
Copyright 2012: Lisa Crispin 79
Agile Testing: A Practical Guide for Testers and Agile Teams
By Lisa Crispin and Janet Gregory www.agiletester.ca
Copyright 2012: Lisa Crispin 81
Beautiful Testing: Leading Professionals Reveal How They Improve Software
Edited by Tim Riley, Adam Goucher
Includes chapter by yours truly
Copyright 2012: Lisa Crispin 82
Test Patterns
Xunit Test Patterns: Refactoring Test Code
By Gerard Meszaros
Copyright 2012: Lisa Crispin Copyright 2008 Janet Gregory, DragonFire
83
Specification by Example How successful teams deliver the right software
Gojko Adzic
Case studies from > 50 teams
Copyright 2012: Lisa Crispin Copyright 2008 Janet Gregory, DragonFire
84
ATDD by Example A practical guide to Acceptance Test-Driven development
Markus Gaertner
Copyright 2012: Lisa Crispin 85
Agile Test Automation Resources
§ dhemery.com/pdf/writing_maintainable_automated_acceptance_tests.pdf § lisacrispin.com § janetgregory.ca § gokjo.net § exampler.com § [email protected] § testobsessed.com § testingreflections.com § www.beer30.org
Top Related