Making Test Automation Work in Agile Projects - Agile Testing...
Transcript of Making Test Automation Work in Agile Projects - Agile Testing...
1
Making Test Automation Work in Agile Projects
StarWest 2009
Lisa Crispin With Material from Janet Gregory
Copyright 2009 Lisa Crispin and Janet Gregory
2
Topics – Making Automation Work
Learn ways to: Overcome common obstacles Use automation to “keep up” Succeed with the “whole team” approach Use automation to enhance ET Evaluate and implement tools Use ROI to help succeed
Copyright 2009 Lisa Crispin and Janet Gregory
3
Why Automate?
Free up time for most important work Repeatable Safety net Quick feedback Help drive coding Manual tests are error prone Tests provide documentation
You know what your system does
Copyright 2009 Lisa Crispin and Janet Gregory
4
Barriers to Test Automation
What’s holding you back?
Fear Changing code
Skills
?????
Not designed For testability
Copyright 2009 Lisa Crispin and Janet Gregory
5
Pain and Fear Programmers don’t feel manual test pain Testers treated as safety net Fear
Programmers lack testing skills Testers lack programming skills
Copyright 2009 Lisa Crispin and Janet Gregory
6
It’s Worth It ROI – explain to management “Present value” of automated tests Acknowledge hump of pain
Copyright 2009 Lisa Crispin and Janet Gregory
Initial Investment
Hump of pain Legacy code, changing code Tools, infrastructure, time
Time
Effo
rt
7
Copyright 2009 Lisa Crispin and Janet Gregory Copyright 2008: Janet Gregory, DragonFire Inc.
8
What Affects ROI
Good test practices increase ROI Simple, well-designed, refactored tests Test resources improve over time
Poor test practices reduce ROI Tests are hard to understand Tests are hard to maintain
Copyright 2009 Lisa Crispin and Janet Gregory
9
Questions on Why Automate?
Copyright 2009 Lisa Crispin and Janet Gregory
10
Small Group Exercise
Write obstacles hindering your team on small Post-Its..
Stick them on the wall or flip chart. Group similar ones. What patterns emerge? Are there any unexpected barriers? Share patterns, surprising outcomes with the
class.
Copyright 2009 Lisa Crispin and Janet Gregory
11
Getting Over the Hump
The test automation pyramid The agile testing quadrants What should be automated What shouldn't Difficult areas
Copyright 2009 Lisa Crispin and Janet Gregory
12
Test Automation Pyramid
Copyright 2009 Lisa Crispin and Janet Gregory
13
Agile Testing Quadrants
Copyright 2009 Lisa Crispin and Janet Gregory
What Should We Automate Quadrant 1 Tests
Unit, Component, TDD (all) Quadrant 2 Tests
Behind GUI, API, web services, GUI Quadrant 3 Tests
Leverage automation where useful Setting up tests, tools, data generation
Quadrant 4 Tests Load, performance, stress
14
Copyright 2009 Lisa Crispin and Janet Gregory
15
What Else Can We Automate?
Automate tedious or repetitive tasks
Testing-related or otherwise Continuous integration, builds Deployments Checking for updates Parsing, comparing output Automating tasks for business
Copyright 2009 Lisa Crispin and Janet Gregory
What We Shouldn’t Automate
Wizard of Oz tests Paper prototyping Usability tests / look and feel Customer adhoc testing Exploratory testing Tests that never fail ??? One-off tests Tests covered elsewhere
16
Copyright 2009 Lisa Crispin and Janet Gregory
17
Where Should We Be Careful GUI tests
Need to test the GUI Can do ATDD - Watch ROI
Automating end to end tests Don't go overboard, automating every path Aim for minimum feature coverage
Remember the pyramid Push testing down as low as you can Robust lower-level tests = better ROI
Copyright 2009 Lisa Crispin and Janet Gregory
18
Hard to Automate? Legacy code Hard to automate, or just lack of skill? “Working Effectively with Legacy Code” –
Feathers “Strangling” – Fowler, Thomas
http://martinfowler.com/bliki/StranglerApplication Mike Thomas article: http://samoht.com/wiki/wiki.pl?
Software_Development
Copyright 2009 Lisa Crispin and Janet Gregory
19
Discussion
Where should you start? Or do next? Is there low-hanging fruit? What’s the best ROI?
Copyright 2009 Lisa Crispin and Janet Gregory
20
Agile Automation Strategy
What hurts the most Layered approach Applying agile principles Choosing the right tools
Copyright 2009 Lisa Crispin and Janet Gregory
21
What Hurts the Most
Keep an impediment backlog What’s biggest obstacle?
Time Tools Code design Testability
Copyright 2009 Lisa Crispin and Janet Gregory
22
Multi-Layered Approach Example: TDD delivers unit, component tests ATDD delivers API level, GUI tests GUI smoke tests as regression safety net For new or legacy code
Copyright 2009 Lisa Crispin and Janet Gregory
23
Specialists Needed?
• May bring in shared or outside resources Performance, load Security User experience design Toolsmiths
Dev team still responsible for completing all tests
Multiple viewpoints provide synergy
Copyright 2009 Lisa Crispin and Janet Gregory
24
Discussion
What skills would help your team overcome automation barriers?
What skills are missing from your team? Where could you get them?
Copyright 2009 Lisa Crispin and Janet Gregory
Applying Agile Principles
Whole team approach Learn by doing Simplicity Iterative feedback Small chunks
Copyright 2009 Lisa Crispin and Janet Gregory
26
Whole-Team Approach
Team responsible for testing, quality Team responsible for testing activities
Includes automation Whole team has all the skills needed Team designs for ease of test automation
Copyright 2009 Lisa Crispin and Janet Gregory
27
Learn by Doing
Courage – don’t be afraid to fail Use agile coding practices for automation
Object-oriented, libraries, modules Simple design Pairing Refactoring Test-first if scripts have logic
Copyright 2009 Lisa Crispin and Janet Gregory
28
Simplicity
Address one or two needs at a time Understand the problem first Try simplest approach first Work in small chunks, thin slices Incremental & iterative
Copyright 2009 Lisa Crispin and Janet Gregory
29
Iterative Feedback
Commit – Try a new tool/framework for “n” iterations
Plan automation tasks for each iteration Use retrospective to evaluate
Copyright 2009 Lisa Crispin and Janet Gregory
30
Automate a Steel Thread at a Time: Example
4-step UI to upload item information from csv file into inventory system
Thread 1: All four pages, user can navigate Thread 2: Select age category, enter info on page 1,
display on page 2, browse and upload file on page 2 Thread 3: Validate data in file, display on page 3 Thread 4: Persist data, display ‘success’ message on
page 4
Copyright 2009 Lisa Crispin and Janet Gregory
Steel Thread Example A steel thread diagram, with “passes” numbered 1, 2. 3 & 4.
31
Copyright 2009 Lisa Crispin and Janet Gregory
32
Small Group Exercise You’re a development team at Agile Testing
Toys. The story on the next slide is your next priority.
Break the story down into thin slices or threads whose tests can be automated. Prioritize them. How could the functionality be delivered incrementally?
Copyright 2009 Lisa Crispin and Janet Gregory
33
Story …
Assumptions/ Conditions of Satisfaction User has already entered shipping address. User can choose different options for different items. The options are Ground, 2 day and Overnight. Items > 20 lbs are Ground only API to cost calculator is available, takes shipping type,
postal code and weight, returns shipping cost.
As an Agile Testing Toys shopper, I want to select a shipping type for my items during checkout and see the shipping cost.
Copyright 2009 Lisa Crispin and Janet Gregory
34
Questions on
Creating an Automation Strategy?
Copyright 2009 Lisa Crispin and Janet Gregory
Choosing Your Tools
One size does not fit all !
Copyright 2009 Lisa Crispin and Janet Gregory
36
Find Time for Evaluating 1. Get commitment for time to research 2. Budget time
Individual or group 3. Determine your requirements 4. Do some basic research 5. Compile a list of tools to consider 6. Try the ones which meet your criteria
Copyright 2009 Lisa Crispin and Janet Gregory
37
Understand the Purpose
• What’s being automated? e.g. Ajax, SSL support; load; embedded s/w Speed up exploratory testing, build test data
• Existing tools, environment e.g.. integration with build process Reporting needs
• Who’s writing, maintaining the tests? • Who’s using the tests? What for? • What are your constraints?
Copyright 2009 Lisa Crispin and Janet Gregory
38
What Fits Your Situation
Existing skills on team Language of application under test Collaboration needs Utilities (automating tedious tasks) vs.
testing Life span, future use of tests
Copyright 2009 Lisa Crispin and Janet Gregory
39
Vendor Tools - Pros
• Existing expertise • Some built on open-source libraries • Fast ramp-up for non-programmers • Perceived as safe choice • Training, support • Part of existing tool set • May have robust features
Copyright 2009 Lisa Crispin and Janet Gregory
40
Vendor Tools - Cons
• Tend to be heavyweight • Tend to be programmer-unfriendly • Scripts may be brittle, high-maintenance
• Capture-playback problematic • Not designed for long-term maintainability
• Can be pricey
Copyright 2009 Lisa Crispin and Janet Gregory
41
Open-Source Tools - Pros
• Designed by test-infected programmers • Designed for agile environments • Designed for maintainability • Programmer-friendly • May have excellent support, tutorials, doc • Easily customized • Low up-front cost
Copyright 2009 Lisa Crispin and Janet Gregory
42
Open-Source Tools - Cons
• May be difficult for non-programmers • Depends on the tool/framework
• Future enhancements may be uncertain • Training, support can be an issue • Be sure to look for active development
community
Copyright 2009 Lisa Crispin and Janet Gregory
43
Home-Brewed - Pros
• Programmer-friendly • Integration with app, IDEs
• Development framework may support • Rails, Ruby, Groovy
• Can build on top of existing framework • Fit, Slim, Watir, RSpec
• Specifically tailored to needs • Someone’s there to address problems
Copyright 2009 Lisa Crispin and Janet Gregory
44
Home-Brewed - Cons
• Team needs enough bandwidth, expertise • Reporting framework • Allow test specification by non-
programmers • Could be harder sell to management
Copyright 2009 Lisa Crispin and Janet Gregory
45
Example: A Java Team’s Choices
IntelliJ Idea, Eclipse for IDEs CruiseControl, Hudson for CI JUnit for TDD, unit, component FitNesse for functional, behind GUI Canoo WebTest for GUI regression smoke tests Watir to aid exploratory testing JMeter for load, performance testing Perl scripts for comparing files, making files
human-readable Java programs for concatenating forms, cover
letters
Copyright 2009 Lisa Crispin and Janet Gregory
46
Example: UPM Team’s Choices
Visual Studio, Resharper MBUnit Hudson for CI FitNesse/SWAT, DBFit for GUI, functional tests ncover for code coverage Confluence wiki for documentation
Copyright 2009 Lisa Crispin and Janet Gregory
47
Discussion
Any tool selection success stories to share?
Copyright 2009 Lisa Crispin and Janet Gregory
48
Questions about choosing tools?
Copyright 2009 Lisa Crispin and Janet Gregory
49
Making Test Automation Work
Time to learn, implement, design Testable architecture Test data Managing tests
Copyright 2009 Lisa Crispin and Janet Gregory
50
Time To Do It Right
Limit scope Write automation task cards Quality must be goal Long-term, will let you go faster
Copyright 2009 Lisa Crispin and Janet Gregory
51
Testable Architecture Layered architecture
e.g.. UI, business logic, data access Ports and Adapters pattern
App can work without UI or database Ports accept outside events Adapters convert for human or automated
users
Copyright 2009 Lisa Crispin and Janet Gregory
52
Test Data
Avoid database access when possible Setup/Teardown
Independent, rerunnable tests Canonical data
Refresh before each test run Customizable data for ET Production-like data
Get customers to provide example data
Copyright 2009 Lisa Crispin and Janet Gregory
53
Managing Automated Tests
Documenting tests Tests as documentation Finding what you need
Reporting results Analyze failures Let CI do the work
Test coverage
Copyright 2009 Lisa Crispin and Janet Gregory
54
Example: My Team’s Tool Choices
Copyright 2009 Lisa Crispin and Janet Gregory
55
Test Management Tools Tool may include management framework
FitNesse Wiki – documentation + executable tests Hierarchy
Rasta – spreadsheets to organize tests Twist – uses Eclipse IDE
What problem are you trying to solve? Simplest approach Use SCCS for tests
Copyright 2009 Lisa Crispin and Janet Gregory
56
Small Group Exercise Look at the barriers from the first exercise. What’s one thing you can do when you get
back to the office to overcome one of your team’s barriers? Write it on a post-it and put it next to the barrier on the flip chart paper.
Share your first action item with the class
Copyright 2009 Lisa Crispin and Janet Gregory
57
Make Automation Work for You Don’t try to find one perfect solution Start simple Budget time to evaluate, implement Use feedback, retrospectives Apply agile principles, practices Budget time to refactor, try new alternatives
Copyright 2009 Lisa Crispin and Janet Gregory
58
Remember
It’s a team problem! Build foundation of core agile practices Experiment Baby steps
Copyright 2009 Lisa Crispin and Janet Gregory
59
Questions? “Aha” Moments?
Copyright 2009 Lisa Crispin and Janet Gregory
60
Some Agile Testing Tool Resources • awta.wikispaces.com/2009ToolsList • softwareqatest.com/qattls1.html • opensourcetesting.org • nunit.org/index.php • webtest.canoo.com • testingfaqs.org • junit.org • wtr.rubyforge.org • fit.c2.com • gojko.net/fitnesse/dbfit • fitnesse.org • selenium.openqa.org • jameleon.sourceforge.net • artofunittesting.com/Chapters/Tools_and_frameworks
Copyright 2009 Lisa Crispin and Janet Gregory 61
Now Available
Agile Testing: A Practical Guide for Testers and Agile Teams
By Lisa Crispin and Janet Gregory www.agiletester.ca
Copyright 2009 Lisa Crispin and Janet Gregory 62
Test Patterns
Xunit Test Patterns: Refactoring Test Code
By Gerard Meszaros
Copyright 2009 Lisa Crispin and Janet Gregory Copyright 2008 Janet Gregory, DragonFire
63
Bridging the Communication Gap Specification By Example and Acceptance Testing
Gojko Adzic