Agile Testing Strategies

8/8/2019 Agile Testing Strategies

1/5

Agile Testing Strategies

As Scott shows here, the quality of your system is only as good as the quality of yourtesting efforts.

By Scott AmblerDecember 12, 2006URL:http://www.ddj.com/tools/196603549

Scott is a DDJ Senior Contributing Editor and author of numerous IT books. He can be contacted atwww.ambysoft.com/ scottAmbler.html.

Over the past few months, I've had more and more people ask me about how we go about testing onagile projects. Agile developers are definitely "test infected," and this month, I explore several

strategies for testing on agile software development projects. A word of warning: Although weclearly don't follow our father's serial testing methodology of yesteryear, I suspect that we can stilllearn a few tricks from dear old dad.

Let's start by setting the philosophical groundwork:

First, you want to test as early as you possibly can because the potential impact of a defectrises exponentially over time (this isn't always true, but it's something to be concerned about).In fact, many agile developers prefer a test-first approach.

Second, you want to test as often as possible, and more importantly, as effectively as possible,to increase the chance that you'll find defects. Although this increases your costs in the short

term, studies have shown that greater investment in testing reduces the total cost of ownershipof a system due to improved quality.

Third, you want to do just enough testing for your situation: Commercial banking softwarerequires a greater investment in testing than membership administration software for yourlocal Girl Scouts group.

Fourth, pair testing, just like pair programming and modeling with others, is an exceptionallygood idea. My general philosophy is that software development is a lot like swimmingit'svery dangerous to do it alone.

Testing Throughout the Lifecycle

Figure 1 presents a high-level view of the agile lifecycle for the purpose of testing (see "Initiating anAgile Project" at www.ddj.com/dept/architect/188700850 for details). Agile projects go through anoften short Initiation phase (Iteration 0) where we set the foundation for the project; a Constructionphase where we develop the system in an evolutionary (iterative and incremental) manner; an End

Page 1 of 5Dr. Dobb's | Agile Testing Strategies | December 12, 2006

05/04/2010http://www.drdobbs.com/article/printableArticle.jhtml;jsessionid=2CQCY3NFREXJ...


2/5

Game phase where we transition our system into production; and a Production phase where weoperate the system and support users. Don't fear the serial boogeyman: The Initiation phase is not arequirements phase, nor is the End Game a testing phase.

[Click image to view at full size]

Figure 1: Test activities during the agile lifecycle.

Testing activities vary throughout the lifecycle. During Iteration 0, you perform initial setup tasks.This includes identifying the people who will be on the external "investigative" testing team,identifying and potentially installing your testing tools, and starting to schedule scarce resources

such as a usability-testing lab if required. If your project has a deadline, you likely want to identifythe date into which your project must enter the End Game. The good news is that you'll discover thatincreased testing during construction iterations enables you to do less testing during the End Game.

A significant amount of testing occurs during construction iterationsremember, agilists test often,test early, and usually test first. This is confirmatory testing against the stakeholder's current intentand is typically milestone-based at the unit level. This is a great start, but it's not the entire testingpicture (which is why we also need investigative testing that is risk-based at more of an integrationlevel). Regardless of the style, your true goal should be to test, not to plan to test, and certainly not towrite comprehensive documentation about how you intend to hopefully test at some point. Agilistsstill do planning, and we still write documentation, but our focus is on high-value activities such as

actual testing.

During the End Game, you may be required to perform final testing efforts for the release, includingfull system and acceptance testing. This is true if you are legislated to do so (common in life-criticalsituations such as medical software development) or if your organization has defined service-levelagreements with customers who require it. Luckily, if you've tested effectively during theconstruction iterations, your final testing efforts will prove to be straightforward and quick. If you'recounting on doing any form of "serious testing" during the End Game, then you're likely in troublebecause your team won't have sufficient time to act on any defects that you do find.

Testing During a Construction Iteration

The majority of testing occurs during construction iterations on agile projects. Your testing effort,ust like your system, evolves throughout construction. Figure 2 depicts two construction iterations,




3/5

indicating that there is confirmatory testing performed by the team, and in parallel, investigativetesting efforts ideally performed by a independent test team (I've adopted the terms "confirmatory"and "investigative" testing from Michael Bolton, a thought leader within the testing community).Although it isn't always possible to have an independent test team, particularly for small projects, itis highly desirable. Confirmatory testing focuses on verifying that the system fulfills the intent of thestakeholders as described to the team to date, whereas investigative testing strives to discoverproblems that the development team didn't consider.

There are two aspects to confirmatory testing: agile acceptance testing and developer testing, both of

which are automated to enable continuous regression testing throughout the lifecycle. Confirmatorytesting is the agile equivalent of testing to the specification, and in fact, we consider acceptance teststo be the primary part of the requirements specification and our developer tests to be the primary partof the design specification. Both of these concepts are applications of the agile practice of singlesourcing information whenever possible.

Agile acceptance testing is a mix of traditional functional testing and traditional acceptance testingbecause the development team and their stakeholders are doing it collaboratively. Developer testingis a mix of traditional unit testing and traditional class/component/service integration testing.Developer testing strives to verify both the application code and the database schema (for moreinformation, see my article "Ensuring Database Quality"; www.ddj.com/architect). Your goal is to

look for coding errors, perform at least coverage if not full path testing, and to ensure that the systemmeets the current intent of its stakeholders. Developer testing is often done in a test-first manner,where a single test is written and then sufficient production code is written to fulfill that test (seewww.agiledata.org/essays/ tdd.html for details). Interestingly, this test-first approach is considered adetailed design activity first and a testing activity second.

Automation is an important aspect of construction testing due to the increased need for regressiontesting on evolutionary projects. The Fitnesse testing framework (www.fitnesse.org), arguably arequirements documentation tool, is often used to automate agile acceptance tests. It is also possibleto generate acceptance test cases from use cases and scenario definitions or from process diagramssuch as UML activity diagrams or flow charts, and tools are beginning to emerge to do exactly this.

The XUnit frameworkJUnit (www.junit.org) for Java and VBUnit (www.vbunit.org) for VisualBasicis used to automate developer tests. Commercial testing tools such as HP Mercury'sTestDirector (www.mercury.com/us/products/quality-center/testdirector) or IBM Rational'sTestManager (www-128.ibm.com/developerworks/rational/ products/testmanager) are also good

Figure 2: Incremental testing throughout the agile development lifecycle.




4/5

options to consider, as they often prove more sophisticated than their open source alternatives. Staticcode analysis tools, such as FindBugs (findbugs.sourceforge.net), are often included in automatedtesting runs to help identify potential quality problems in the source code.

Investigative Testing

As I was writing this column, I was lucky enough to attend a presentation given by Dr. Cem Kanerto the Toronto Association of Systems and Software Quality (TASSQ). Kaner, coauthor ofLessonsLearned in Software Testing (Wiley, 2001), described his thoughts and experiences in softwaretesting, and as a result, helped me to conceptualize some of my own ideas. In particular, I had beenstruggling to properly describe the activities of investigative testing efforts during constructioniterations, and Kaner's presentation helped to coalesce my experiences.

A separate test team? Preposterous you say! Actually, there is significant value to be gained bysubmitting your system to an independent test team at intervals throughout the lifecycle so that theycan verify the quality of your work. Agile teams produce working software at the end of eachconstruction iteration; therefore, you have something new to test at that point. A common practice is

to provide a new version of the system at least once a week, regardless of your iteration length, aparticularly good strategy the closer you get to the End Game.

The investigative test team's goal should be to ask, "What could go wrong," and to explore potentialscenarios that neither the development team nor business stakeholders may have considered. They'reattempting to address the question, "Is this system any good?" and not, "Does this system fulfill thewritten specification?" The confirmatory testing efforts verify whether the system fulfills the intent,so simply repeating that work isn't going to add much value. Kaner promotes the idea that goodtesters look for defects that programmers missed, exploring the unique blind spots of the individualdevelopers.

Investigative testers describe potential problems in the form of defect storiesthe agile equivalent ofa defect report. A defect story is treated as a form of requirementit is estimated and prioritized andput on your requirements stack. The need to fix a defect is a type of requirement, so it makes perfectsense to address it just like any other requirement. As you would expect, during the End Game, theonly requirement type that you're working on is defect stories.

Your investigative testing will address common issues such as load/stress testing, integration testing,and security testing. Scenario testing, against both the system itself and the supportingdocumentation, is also common. You may also do some form of usability testingthe user interfaceis the system to most end users; therefore, usability is critical to success. The UI includes both thescreens that people interact with and the documentation that they read, implying that you need to testboth.

Good investigative testing efforts reveal any problems that developers missed long before theybecome too expensive to address. It also provides feedback to management that the team issuccessfully delivering high-quality working software on a regular basis. Kaner pointed out thatthere's no one right way to go about investigative testing, nor is there one correct list of techniques toemploy. Your efforts must reflect the goals of the project team that you're supporting. For example,is the goal to determine whether the system is ready to be shipped? Is it to ensure that the systeminteroperates with other existing systems? Is it to help the developers to identify problems in theirown testing efforts by pointing out the causes of defects that they missed? Is it to minimize thechance of a lawsuit against your organization or its managers? The context in which you are testingwill determine what and how you testnot only will the context be different for each project; thecontext also changes over the life of the project.

The type of confirmatory testing performed by agile teams is only one part of the testing pictureit




5/5

is the agile equivalent of traditional smoke testing. This is a great start, and having automatedregression testing provides the safety net required by evolutionary development techniques.Investigative testing enables you to explore the critical "big picture" issues, as well as the "littlepicture" issues that nobody thought of until now, which confirmatory testing typically does not.

Quality Is Job #1

The testing approach I've described here is different from the traditional, documentation-heavyapproach where you throw the system and its specifications over the wall to testers and hope for thebest. My experience is that the quality of your system is only as good as the quality of your testingefforts.

Thanks to Michael Bolton, Cem Kaner, Renu L. Rajani, and Steve Robinson for their insightfulfeedback.

Copyright 2010 United Business Media LLC

Agile Testing Resources

You should find the following resources full of provocative ideas about how to improve yourtesting efforts:

Agile Testing Mailing List (tech.groups.yahoo.com/group/ agile-testing).

Context-Driven Software Testing Mailing List (tech.groups.yahoo .com/group/software-testing).

"High-Volume Test Automation," by C. Kaner, W.P. Bond, and P. McGee(www.kaner.com/pdfs/HVAT_STAR.pdf).

James Bach's blog (www.satisfice.com/blog).

"Lessons Learned in Software Testing," by C. Kaner, J. Bach, and B. Pettichord(www.testinglessons.com).

Michael Bolton's testing articles (www.developsense.com/articles).

"Roadmap for Agile Testing" and "The Testing Team's Motto," by Brian Marick(www.testing.com/writings).

"What Is Exploratory Testing?" by James Bach (www.satisfice.com/articles/what_is_et.shtml).


05/04/2010h // d d bb / i l / i bl A i l jh l j i id 2CQCY3NFREXJ

Agile Testing Strategies

Documents

Transcript of Agile Testing Strategies