Automated Web Service Testing in Agile
Edmund Barton
Contents
1. TEC & Industry Training – the “puzzle”
2. ITR as a solution to “the puzzle”
3. Our Agile Team 3. Our Agile Team
4. Technology & Tools
5. A Test Approach in Agile
6. What went well
7. What didn’t
8. Lessons Learned
Take-homes
• Where modern software development is
• Where functional testing is heading
• How to reduce the testing “bottleneck”
• Insight into a project that used test automation well
1. TEC & Industry Training – the “Puzzle”
The Client
Tertiary Education Commission
Multi-$bn budget across the Tertiary Education Sector
1. TEC & Industry Training – the “Puzzle”
Governmental body
Industry Training
• Funded by TEC
• Workplace training programmes
1. TEC & Industry Training – the “Puzzle”
• Modern Apprenticeships
Industry Training Organisations – ITOs
Question?
• How do we know how well ITOs are performing?
1. TEC & Industry Training – the “Puzzle”2. ITR as a solution to “the Puzzle”
The Answeris to collect data
... like a vac
I.T.R. or Industry Training Register
Collects data about all Industry Training
TEC gets a real-time view of Industry Training
2. ITR as a solution to “the Puzzle”
TEC gets a real-time view of Industry Training sector
• Accurate measures of ITOs performance
• Checks data against TECs policy
Data source for further analysis of Tertiary Sector
I.T.R. as an IT solution
ITR is a Web Service, from scratch
No User Interface
2. ITR as a solution to “the Puzzle”
Application
e.g.
Student
Management
System
Web
Service
XML SOAP Messages
Building ITR
2. ITR as a solution to “the Puzzle3. Our Agile Team
A candidate for Agile?
Requires Flexibility
3. Our Agile Team
ITR’s Agile Team
Requirements
partially
Therefore a
candidate for Agile
3. Our Agile Team
partially
flexible
Resources
3-6 developers
2-3 testers
Timescale
Fixed April ‘10
to Feb ‘11
candidate for Agile
Agile Team
“The Business”
Development / Project Management
3. Our Agile Team
Development / Project Management
Test / QA - Qual IT
ITR Project goes Agile
Adopted SCRUM
Testing at the same time as Development
3. Our Agile Team
Multiple User Stories per iteration
Software “Demo” after each iteration
Question:
• But there’s no UI, how did we demo progress?
What was the role of the tester in this project?
3. Our Agile Team
Role of the Tester in Agile
Test Automation Engineer
Distributor of knowledge
3. Our Agile Team
Project “glue”
Story quality gate
Usual defect wrangling / management
Facilitator of business acceptance during iteration
Example Agile User Story
As a generic example:
As a:Consumer of ITR
3. Our Agile Team
I want to:receive an error message if I submit an invalid message
So that:I know I’ve made a mistake
List of Acceptance Criteria
conditions where this story applies
Agile Project Approach
Fixes
This high-level process applies to each user story
3. Our Agile Team
Development Test
Bugs
Fixes
Application Feature
AutomatedTest
Process Outputs
Plan / Design
Agile Project Approach
Test all the way through the iteration
More efficient use of test resources
3. Our Agile Team
Use “Test Driven Development”
Reliance on Automation technology
What technology was used?
... here comes the science bit
3. Our Agile Team 4. Technology & Tools
Agile Development Technologies
4 . Technology & Tools
We assume that:
• An application will function the same in each environment (if we control data and configuration)
Agile Development Technologies
Each team member (inc Test) has:
• own VM (a Build workstation)
4 . Technology & Tools
• own VM (a Build workstation)
• Visual Studio 2008
• Team Foundation Server
• Common test data
• Build Server status
• Team City
s
Build Server (Team City)
Provides feedback on the current state of the “solution” i.e. code doesn’t compile
4 . Technology & Tools
Team
Foundation
Server
Source code(under source control)
Team City
Build Server
Build Server (Team City)
Nightly Builds (Team City)
• Run ALL automated functional tests (Regression suite)
• Nightly feedback on state of “solution” i.e. some new
4 . Technology & Tools
• Nightly feedback on state of “solution” i.e. some new code regressed existing functionality
Broken builds
• Buy / Make the team coffee
Successful Nightly Builds
Essential for Agile software development
• Tell us we have working software
• Could be deployed right now!
4 . Technology & Tools
• Could be deployed right now!
New code hasn’t regressed other parts of the application
Rely heavily on the automated functional tests
Really important to have good test coverage
4. Technology & Tools5. A Test Approach in Agile
What was the testing approach?
Test Automation approach
Test Engineer has 3 key tasks:
• Create and support the automation framework
5. A Test Approach in Agile
• Create automated tests – validate features
• Add / remove from regression suite as project changes
• Can be a lot of work (ITR has over 2000 test cases)
Example of Tools
Open Source
5. A Test Approach in Agile
Front end for Web Services – the missing UI
Large community of users – easy to find help
Data Driven test cases
Example of framework
ITR written in .NET
Use NUnit to execute SOAP UI test cases
5. A Test Approach in Agile
ITR
Web
Service
SOAP Messages
Test suite project
.NET test
method
ITR
database
Example of testing
Really grey-box testing, not fully open white box testing but more in-depth than black box
Achieved 100% requirements coverage
5. A Test Approach in Agile
Achieved 100% requirements coverage
SOAP UI projects self documenting
Traceable via “Spreadsheet of Doom”
Agile – Approach to change
Agile projects change
Complete manual testing unfeasible
5. A Test Approach in Agile
Maintenance overhead?
Yes and No
• Yes – but don’t worry about throwing tests away
• Yes – but don’t maintain, rebuild
• No – just reflects churn within the project
So what worked well?
5. A Test Approach in Agile 6. What went well
… what’s so flash?What worked well?
1. Houston, we have a problem ...
6. What went well
How to make it faster?
Other options failed to make a difference:
• Database tweaking
• Disabling logging
6. What went well
• Disabling logging
• Better hardware
Key architectural components were too slow
ITR needed more than just refactoring
Improving performance
Developers:
• Rip out the poorly performing frameworks and re-engineered some of the architecture
6. What went well
engineered some of the architecture
• Re-factor poorly performing code
Use the automated functional test suite as the only validator of behaviour
Test First Development
A real improvement
Result
• Massive performance
6. What went well
performance improvement
• 13x increase in message processing speed
2. Better Deployments
Deploying wrong version of code = bad
Build server set up to automatically:
6. What went well
• run the regression suite
• check every test passed
• build a deployment package with correct versions
2. Better Deployments
Greatly simplified the deployment
Over 50 deployments
6. What went well
Guaranteed the functionality of the deployment package:
• All deployment problems were configuration related
• much easier to fix
• Zero deployment functionality problems
3. Using SOAP UI Tests for Business Acceptance
Web service has no UI
• Difficult to demo to the business
6. What went well
Demo the features created during the Agile iteration
Improved Agile acceptance “gates”
So what worked well?
6. What went well7. What could have been improved
• … what’s so flash?What could have been done better?
1. Agile troubles – Web Service
Fixed Time / Fixed Requirements – not enough resources. Requirements
partially
7. What could have been improved
Project struggled to be Agile
partially
flexible
Resources
3-6 developers
2-3 testers
Timescale
Fixed April ‘10
to Feb ‘11
1. Agile troubles – A Web Service
Web Services have set list of operations
SOAP Messages
7. What could have been improved
SOAP Messages
ITR
Web
Service
ITR Client
Cannot just remove them – they are mandatory
operation
2. Agile troubles – No Refactoring
Refactoring was discouraged in favour of more User Stories – “Velocity”
Iteration after iteration of on-the-fly decisions
7. What could have been improved
Iteration after iteration of on-the-fly decisions produces a terrible codebase
Often difficult to accommodate new requirements
Agile progress or “velocity” is not more important than code quality.
Address code quality as you go
So what worked well?
7. What could have been improved8. Lessons Learned
• … what’s so flash?Lessons Learned
1. Lessons about Agile Projects
• If the project isn’t a candidate, don’t bend it to fit
• Needs a strong PM with Agile experience
8. Lessons Learned
• Testing does fit – Testers become part of the development team
• Test Automation pretty much essential on projects of this scale
2. Lessons about Agile Development
• Agile projects are not just about “velocity”, quality must also be a driver
• Consequence of design decisions being made
8. Lessons Learned
• Consequence of design decisions being made on-the-fly is poor code quality
• There’s no getting away from a need to plan / design and architect
3. Lessons about Test Automation
Steep Learning Curve
Highly valuable
8. Lessons Learned
• Time Saved on Project Timelines
• Confidence in development for Business
• Test Driven Development and Regression
A final word from the Developers
From the post-implementation review
• “The testing was awesome”
8. Lessons Learned
Questions / Comments
Top Related