Anders Claesson Copyright © 2010, Enea AB 1(29)

Test Strategies in Agile Projects

2(29)

Contents

Agile development and testing

Test methods, tools and planning

Definition of Done

User Stories

Test ideas and guidelines

Test charters and exploratory

testing

Logging and reporting

When to stop testing

3(29)

Agile Development

Focus on rapid delivery of business value

Reducing risks

Continuous planning and feedback

Value demonstrated

Self-organizing teams

4(29)

Agile Testing

An iterative process from a customer's perspective

Testing is performed early and often

Testers are part of the development team

“User Stories” are tested

Close cooperation with developers and customers

Continuous integration and regression tests

All test results are logged

Defects are reported

5(29)

Scrum Overview

Product Backlog

Sprint Backlog

Daily

Sprint

Meeting

Sprint2 – 4 WeeksSprint

Plan

Risk

Analysis

Retrospective

Release

Plan Definition

of Done

Task Board

Burndown Chart

Release

Criteria

6(29)

Agile Testing Quadrants

Functional Tests

Examples

Story Tests

Prototypes

Simulations

Exploratory Testing

Scenarios

Usability Testing

UAT

Alpha/BetaQ2 Q3

Q1 Q4Unit Tests

Component

Tests

Performance

& Load Testing

Security Testing

“ility” Testing

Automated

& Manual

Automated Tools

Business-Facing

Technology-Facing

Supp

ort

ing the T

eam E

valu

ate

Pro

duct

Manual

Bri

an

Mari

ck's

agile

te

sting

matr

ix

7(29)

Find out what the Customer Values the Most

Importance

Weight %

Customer

SatisfactionQuality Attribute

User/Customer Value Analysis

Priority

IW / CS

4. Performance 20 1,1 18

2. Availability 15 0,8 18

1. Reliability 30 0,9 33

5. Service Level 10 1,0 10

6. Maintainability 5 0,8 6

3. Usability 20 1,1 18

Summary: 100 % X=0,96

Also analyze the complexity of the system including all valid and prioritized combinations of features and platforms

Risk/Cost

of failure

Supplier

costs

Customer

costsX < 1: Less satisfied

X > 1: More satisfied

High Critical

High Critical

Medium High

High Critical

High High

Medium Low

8(29)

Test Methods and Techniques

Requirements based testing

Design based testing

Risk based testing

Exploratory testing

Error guessing

Taxonomy based testing

Attack based testing

Model based testing

Scenario based testing

Combinatorial testing

Value based testing

Prototyping

9(29)

Three Different Views

User What do the user want to do with the system?

System What should the system be capable of doing?

Risks What problems may occur?

10(29)

How to Explore and Learn

11(29)

Test Planning

What

Why

Who

Where

When

How

Dependencies

Risks

Prio

Time

12(29)

Useful Tools for Agile Testinghttp://www.opensourcetesting.org/ http://www.satisfice.com/tools.shtml

Allpairs Test Case Generation Tool for combinatory testing

PICT Generation of combinatorial tests using orthogonal

arrays http://www.pairwise.org/tools.asp

Perlclip Testing of text fields or documents with different kinds

of stressful inputs

SpectorPro Logging/recording of all activities on a PC

http://www.spectorsoft.com/

TestExplorer Session based ET http://www.testexplorer.com

Session Tester An exploratory testing tool for managing and recording

session-based testing http://sessiontester.openqa.org/

Resource Viewer Is intended for viewing of resources in executable files

http://www.glocksoft.com/resource_viewer.htm

Rasta Keyword Driven Test Automation

http://rasta.rubyforge.org/index.html

List of Testing Tools: http://www.aptest.com/resources.html

13(29)

Definition of Done

Feature Story(-ies) or product backlog item(s)

Sprint A collection of features developed within a Sprint

Release Potentially shippable parts

Example, feature overview:

User management

Add new

Delete

ListSort

Paging

Search

Tags

Smart search

Regular expressions

Clone

Edit

14(29)

Definition of Done – Unit Level

Test Aspect Criteria for Done

Structure based testing

100% Software Design and Module Specifications covered

100% Statement Coverage of the source code

Boundary Value Analysis Testing

Equivalence Classes and Input Partitioning Testing

All Test Cases passed and no remaining faults to be corrected

All code reviewed

Known weaknesses described

Component testing reported (including obtained test coverage)

Integration testing

Internal and external interfaces in the sub-system covered by verifying protocols and syntax in all messages

More than 40% of all tests are negative test cases

15(29)

Definition of Done – Functional Level

Test Aspect Criteria for Done

Functional testing

100% requirements coverage.

100% coverage of the main flows in the operational scenarios.

100% of the highest risks covered.

100% of externally observable system states covered.

100% of externally observable failure modes covered.

Operational manuals tested.

All failures found are reported.

Boundary Values, Equivalence Classes and Input partitioning testing made for all input data.

All combinations of input and output parameters and values covered (pair-wise coverage).

16(29)

User Stories 1(2)

Card:

As a registered user, I want to log in, so I can access subscriber content

Conversation: User Login

Username:

Password:

LoginRemember me

[Message] Forgot password?

Store cookie if

ticked and login

successful

User’s email address,

Validate format

Authenticate against SRS

using the new web service

Go to forgotten password page

Display message here if not successful.

(See confirmation scenarios on the next page)

17(29)

User Stories 2(2)

Confirmation:

Success Valid user logged in and referred to the home page

a) Valid user name and password

b) “Remember me” ticked – Store cookie/automatic login next time

c) “Remember me” not ticked – Manual login next time

d) Password forgotten and a correct one is sent via email

Failure Display message:

a) “Email address in wrong format”

b) “Unrecognized user name, please try again”

c) “Incorrect password, please try again”

d) “Service unavailable, please try again”

e) Account has expired – refer to account renewal sales page

18(29)

Test Questions

- Which user/usage goals should be met?

- What user problems should be solved?

- Which user benefits should be achieved?

- Why does the orderer/customer want the system?

- Who are the customer(s) and the target user group?

- Which functions and characteristics are included?

- What are the most common and critical parts of the functionality from the users point of view?

- Are there any performance requirements included?

- What is an acceptable response time for the users?

- How tolerant should the system be to faulty input or user actions?

19(29)

Test Ideas

What do we need to find out about the system?------------------------------

1 What happens if …………………………………..?

2 What should happen when ………………………..?

3 Will the system be able to fulfil all its requirements?

4 What are the expectations and needs from the customer?

5 In what way may the system fail?

6 What problems were found in the previous release?

7 Are the requirements and the input specifications possible to

understand and test (sufficient testability)?

8 Will the system be reliable and resist failure in all situations?

9 Will the system be safe in all configurations and situations

it will be used?

10 How easy is it for real users to use the system?

11 How fast and responsive is the system?

12 Is it easy to install (and configure) onto its target platform?

13 How well does it work with external components and configurations?

14 How effectively can the system be tested (e.g. can log files be read)?

20(29)

Guidelines for Agile Testing 1(2)

1. Test in pairs

2. Prepare test charters in advance

3. Use exploratory testing

4. Build tests incrementally

5. Use test design patterns

6. Perform keyword/data driven tests

21(29)

Guidelines for Agile Testing 2(2)

7. End-to-end testing

8. Scenario based testing

9. Use automation for test data generation and execution

10.Frequent regression testing

11.Documentation testing

12.Log everything you do

22(29)

Test Charter 1(3)

Actor < Type of user >

Purpose < Describe the function, web page , test idea, … to be tested >

Setup < Preconditions i.e. concerning HW, content of data base(s),.. >

Priority < Importance of risk, function, web page, …. >

Reference(s) < Requirement, risk, test ideas, … >

Data < Whatever is needed for the activities, files >

Activities < A list of actions and test ideas >

Oracle notes < How to evaluate the product for correct results >

Variations < Alternative actions and evaluations >

Test Charter no # and <title>

23(29)

Test Charter 2(3)Test Charter 1: Analyze the copy/paste function of pictures

Actor: Normal user.

Purpose: To evaluate if the copy/paste function of pictures works in our word processor together with the most commonly used word processors on the market (Word, Power Point., etc.) and other programs where pictures can be inserted in the copy buffer for copy/paste operations in the PC. The purpose is also to see that no information is lost or corrupted.

Setup: A Dell PC with 2 Gb memory, our word processor, the Microsoft Office package professional and the home edition 2003 patch level xx, PDF reader version 7.1, Notepad version 4, Internet Explorer version 7, Opera version 5, Mozilla……etc. (the setup might be common for several charters and can therefore be described and referred to instead of repeating the same information in every charter).

Priority: High, because this function is used very frequently both within our own word processor, but also between other word processors and programs, the user may want to copy/paste pictures with ours.

Reference: Requirement abc_1 in document Def-07:034 revision R1.1. Risk number 3 from the risk assessment held on April 4 2007 regarding the copy function, which is documented and followed up in the document Rsk-07:012.

Data: A variety of pictures with different resolutions, both vector graphics as well as bit mapped pictures. The pictures could be photos or figures in web browsers for example. Complex pictures are also included and pictures which might be copy protected in some way.

24(29)

Test Charter 1: Analyze the copy/paste function of pictures, continued….

Activities: 1. Copy/paste a picture in our word processor from one location to another in the same document.

2. Copy/paste a picture to and from our word processor into/from Word, PowerPoint and Excel.

3. Copy a picture into our word processor from a variety of the mostly used Web browsers.

4. Copy/paste a picture to/from the most commonly used web site building tools such as Dreamweaver from Adobe.

5. Copy a picture from a PDF document using Acrobat Reader into our word processor.

6. Try to copy a write protected picture from the web or other source into our word processor.

7. Try to copy/paste a variety of non readable and some readable ASCII-characters or corrupted pictures.

Oracle Look if the size of the pasted picture changed on the screen (it should not).

notes: Check if there is any loss in the resolution of the picture especially when you copy and paste from other programs to or from ours.

Check which is the highest resolution picture that can be copied and how that affect the system.

Check for memory leaks and how much memory the copy/paste operation takes from the system and how that affects the use of other programs. Other programs should not slow down or be affected in any way.

Print pasted pictures to see if there are any differences in color, resolution or any other anomaly.

Variations:Try out (and find) the boundaries of how large pictures are possible to copy/paste.

Perform copy/paste with a large number of items in the copy/paste buffer in your PC.

Try to fill the copy/paste buffer to its limit and then copy/paste a picture and see what happens.

Try the longest file name of the picture you can type before making the copy/paste. You may use the tool Perlclip (download from http://www.satisfice.com/tools/perlclip.zip) to generate file names with a million characters or more if you want.

Test Charter 3(3)

25(29)

Exploratory Test Execution

1. Observe

2. State questions

3. Form hypothesis

4. Design the experiment

5. Test the hypothesis

6. Draw conclusions

7. State additional questions

The scientific approach

26(29)

A Program can Fail in many Ways

System

under

test

Program state

System state

Intended inputs Monitored outputs

Configuration and

system resources

Input from other cooperating

processes, clients or servers

Program state, including

uninspected outputs

System state

Impacts on connected

devices/system resources

Output to other cooperating

processes, clients or servers

27(29)

Exploratory Test Execution Logging

Use a logging tool:

Log everything you do

Information is traceable

Complement with an API logger/sniffer

Your notes show your reasoning while testing

Exploratory testing requires good logging

28(29)

Medium

Medium

High

Low

Low

Current

risk

level

Q

ass.

3

2

1+

2+

0

W 6

High

Pause

Blocked

High

None

Current

test

effort

Medium

High

High

Medium

Low

Initial

Risk

Level

Medium

High

High

Medium

Low

Needed test

effort

IR1212 under investigation.1+1+1+11Area 4

Crashes, IR123451+11Area 3

On track, no faults.221+11Area 2

Feature(s) not yet delivered

from design and integration.

Definition of Done not

fulfilled for functional testing.

No testing possible.

00Area 1

3

W 5

2+

W 4

2+

W 3

2

W 2

1

W 1

Configuration problems.Area 5

CommentsTest

area

Test effort and perceived Quality Level – Including risks and test coverage

Test Reporting

29(29)

All planned/required Test Charters/sessions and characteristics tests have been run and passed according to the current risk areas/levels and where faults have been found. The coverage objectives have been reached that were stated in the test goals (e.g. System Requirements, User Stories).

CoverageTesting should stop when:

• The probability of remaining faults has been reduced to a level that can be accepted by the customer.

• No open priority A Incident Reports.

• The systems’ risk level is within acceptable limits (i.e. no critical risks remain unsolved).

• The Definition of Done for all testing activities have been fulfilled.

• The product values have been demonstrated and accepted (i.e. implicit and explicit quality attributes are satisfied).

Quality

When the agreed ship date has been reached.

Time

When to Stop Testing