Steven K Allott - Effective Testing - SoftTest Ireland

Presented by Stephen Allott

CaseMaker InternationalA Diaz & Hilterscheid group company

Effective Testing

Using industry standard techniques

SoftTest Ireland 16th February 2006

SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 2

Agenda

• Why do we test software?

• How do we test effectively?

• Risk based testing• Black Box techniques• Pairwise testing

Photo: Courtesy of NASA


Why test software?


Read all about it!

US site – down for 3 days,

forced to return $30 fee to nearly

260,000 online “fantasy

baseball” players.

US site down for 2

days, $280,000 in

lost sales.

Shopping Site: 6-hour downtime:

$400,000 in direct costs plus additional

in discounts to inconvenienced

customers.

UK - Concerns over bugs

and delays in new air

traffic control software

UK – Payment clearing system failure at month end

delayed wages to many customers

UK - Televisions,

Cameras sold at

very low prices


Ineffective Testing

• Creates too many of the wrong tests• Duplicates effort unnecessarily• Costs too much and takes too long• Has an adverse impact on the business as a

whole


Effective Testing

• Is aligned with the business need• Uses a risk-based approach and formal

techniques to create fewer tests with increased coverage

• Facilitates communication with the business• Provides pre-production confidence• Saves time and money


Risk Based Testing

• Identify business risks• Assess the impact

– Severity, if it were to fail– Probability of failure

• Focus more testing on high risk areas

• Continually measure and monitor the risks


Test Case Design Techniques – 1

• Black box and white box techniques have been proven to be effective

• Techniques are described in BS7925• Analysis/design separated from execution• Improved and demonstrable test coverage



• Analysis of the test conditions• A formal design process gives you

– Objectivity– Repeatability– Consistency– Documentation– Auditability– Traceability– Confidence



• The design activity alone can find faults• Helps drive the need for good requirements• Use of formal techniques helps with estimates

of overall test effort required• Supports test automation


Black Box Techniques

• BS7925 techniques– Equivalence class

partitioning– Boundary value

analysis– Decision tables– State transition

testing– Syntax testing– Random testing

• Other techniques– Pairwise– Domain analysis– Use case testing


White Box Techniques

B

S

MCDC

BC

L

All Paths

DU

DPU

DCU

where: B - Branch coverage/Decision coverage BC - Branch Condition coverage BCC - Branch Condition Combination coverage DCU - All C-uses coverage DPU - All P-uses coverage DU - All du-paths coverage L - LCSAJ coverage MCDC - Modified Condition Decision coverage S - Statement coverage

BCC


Test Data

• Copy of live data• Realistic data volumes• Range of values and

inputs


Equivalence Class Partitioning

• Can you ever see a need to: – try the view from every seat of a cinema or theatre?– test drive every car in the showroom?– withdraw cash from all the ATMs in your country?

• If you answered “No, of course not!”– what would you do instead, and why?



• Equivalence class or partition• Ranges of data• Based on an assumption about the software’s

behaviour• Reduces the number of test cases whilst

maintaining reasonable test coverage



• An online car insurance quotation system allows a 30% discount for good drivers aged between 25 and 50 but charges full price if you are under 25. For older careful drivers a 20% discount is allowed and if you are retired we will be happy to quote you a special price

Under 17 17 to 24 25 to 49 50 to 64 65 and over

£1,000 £700 £800


Boundary Value Analysis

• Builds on Equivalence Class technique• Assumes errors are made on the boundaries• Applicable to value and structure of data

– where there is a continuous range of values– length and character type

• Generates three test cases per boundary– one on the boundary– one either side of the boundary– the smallest distance away


Boundary Value Analysis – Example

• An online car insurance quotation system – Under 17 – unable to quote– 17 to 24 (premium price)– 25 to 49 (30% discount)– 50 to 64 (20% discount)– 65 and over (price on application)

Under 17 17 to 24 25 to 49 50 to 64 65 and over

16,17,18 49,50,5124,25,26 64,65,66



• Online Banking (Account balance)• Lower boundaries

– EUR 0.01 (DR)– EUR 0.00 (CR)– EUR 0.01 (CR)

• Upper boundaries– EUR 999,999.99 (CR)– EUR 1,000,000.00 (CR)– EUR 1,000,000.01 (exceeds max for this account

type)



• Online Banking (Set up standing order)• Boundaries for from date

– Yesterday (invalid)– Today (should be allowed)– Tomorrow (should be allowed)

• Boundaries for until date– 12 months from today (ok)– 12 months less 1 day (ok)– 12 months and one day from today (invalid)


Decision Tables

• Another intuitive black box technique• Express complex business rules in a table• Cause-Effect graphing helps with the analysis • Define the combination of conditions (causes)• Define the resulting actions (effects)• Easy to create test cases


Example – Train Tickets

Conditions

Rule 1

Rule 2

Rule 3

Rule 4

Holds a railcard Y Y N N

Travels after 9 am

Y N Y N

ActionsPay £10

Pay £15

Pay £20

Pay £30


Example – Train Tickets Test Cases

Inputs

Test 1

Test 2

Test 3

Test 4

Holds a railcard Y Y N N

Travels after 9 am

Y N Y N

Expected results

Pay £10

Pay £15

Pay £20

Pay £30


Example – ATM

Conditions

Rule 1

Rule 2

Rule 3

Rule 4

Enter valid card Y Y N N

Enter valid PIN Y N Y N

Actions Auth Reject Reject Reject


Example – ATM “don’t care”

Conditions

Rule 1

Rule 2

Rule 3

Enter valid card Y Y N

Enter valid PIN Y N ---

Actions Auth Reject Reject


Example – File Backup

ConditionsRule

1Rule

2Rule

3Rule

4

Stick inserted N Y Y Y

Stick formatted --- N Y Y

Sufficient space --- --- N Y

Actions Fails Fails Fails Works

Pairwise Testing

Acknowledgement

This session on pairwise testing techniques is based on the book by Lee Copeland: A practitioner’s guide to software test

design, published by Artech House, 2004, ISBN 1-58053-791-X


Pairwise Testing

• Pairwise is based on testing pairs of combinations of variables rather than all combinations

• It reduces the number of test cases• Many faults are either

– Single mode– Double mode

• There are two recognised methods:– Orthogonal arrays– Allpairs algorithm


Pairwise Example

• Consider a simple database enquiry screen:

– It has 4 fields

– Each field has three possible choices

– Giving 3 X 3 X 3 X 3 = 81 possible combinations to test

• With Pairwise Testing

– You can reduce this to 9 tests!


Pairwise Example

• Holiday destination– UK, USA, Europe

• Price range (for family of 5)– Budget: Under £2000– Medium: £2000 to £5000– Expensive: Over £5000

• Standard of hotel– Basic, 4 star, luxury

• Availability– Now: Within 7 days from today– Soon: After a week but up to one month from today– Later: Later than one month from today


Orthogonal Array L9 (34)

A B C D

Row 1 1 1 1 1

Row 2 1 2 2 2

Row 3 1 3 3 3

Row 4 2 1 2 3

Row 5 2 2 3 1

Row 6 2 3 1 2

Row 7 3 1 3 2

Row 8 3 2 1 3

Row 9 3 3 2 1

This orthogonal array has 9 rows, 4 columns and three ‘levels’ per column



Where Price Hotel When

Row 1 1 1 1 1

Row 2 1 2 2 2

Row 3 1 3 3 3

Row 4 2 1 2 3

Row 5 2 2 3 1

Row 6 2 3 1 2

Row 7 3 1 3 2

Row 8 3 2 1 3

Row 9 3 3 2 1

Each variable becomes a column in the orthogonal array




Row 1 UK 1 1 1

Row 2 UK 2 2 2

Row 3 UK 3 3 3

Row 4 US 1 2 3

Row 5 US 2 3 1

Row 6 US 3 1 2

Row 7 Europe 1 3 2

Row 8 Europe 2 1 3

Row 9 Europe 3 2 1

For each column we fill out the possible choices




Row 1 UK Budget Basic Now

Row 2 UK Medium 4 star Soon

Row 3 UK High Luxury Later

Row 4 US Budget 4 star Later

Row 5 US Medium Luxury Now

Row 6 US High Basic Soon

Row 7 Europe Budget Luxury Soon

Row 8 Europe Medium Basic Later

Row 9 Europe High 4 star Now

Now the array is complete




Test 1 UK Budget Basic Now

Test 2 UK Medium 4 star Soon

Test 3 UK High Luxury Later

Test 4 US Budget 4 star Later

Test 5 US Medium Luxury Now

Test 6 US High Basic Soon

Test 7 Europe Budget Luxury Soon

Test 8 Europe Medium Basic Later

Test 9 Europe High 4 star Now

Each row becomes a test case – you can add more if you wish


How to use Orthogonal Arrays

• Identify the variables• Determine the number of choices for each• Locate an orthogonal array of the right size• Map the testing problem onto the array• Construct the test cases


What size array do we need?

Variable Choices

Browsers IE5, IE6, …. 8 in total

Plug-ins None, Real player, Media Player

Operating systemsW95, W98, W2000, NT, XP, XP-PRO

Server IIS, Apache, WebLogic

Server O/S NT, XP, Linux

How many possible combinations are there?


Using Orthogonal Arrays

• We require an array with 5 columns– Column 1 needs 8 levels (browsers)– Column 2 would need 3 levels (plug ins)– Column 3 would need 6 levels (O/S)– Columns 4 & 5 would need 3 levels (Server, Server O/S)

• Ideally we need 81 61 33 but it doesn’t exist!

• The next largest is L64 (82 43) – 2 columns of 1 – 8– 3 columns of 1 – 4

• So rather than 1,296 tests we need only 64 tests!• Remember - one test case per row


Allpairs Algorithm

• Devised by James Bach• Generates fewer tests• In our example, it generates just 48 tests

compared to 64 from using orthogonal arrays• Consider 100 variables, 2 choices

– Using orthogonal arrays you need 101 tests– Using all pairs you need only 10 tests


Pairwise Testing – Summary

• Test all pairs rather than all combinations– With 4 variables and 3 choices for each you would

require 81 test cases whereas you can cover all pairwise input combinations with only 9 tests

– For 13 variables and 3 choices for each you would require1,594,323 test cases whereas you can cover all pairwise input combinations with only 15 tests

• Pairwise guides us to help find– Single mode defects– Double mode defects

• You should always add any known risky combinations


Effective Testing – Summary

• Effective Testing uses– A risk-based approach– Formal test case design techniques

• The benefits are– Reduced number of test cases– Repeatable well documented tests– Improved planning, estimation and communication– Facilitates earlier involvement with the business– Find faults earlier– Give you a consistent approach to testing– Save time and money– Enables test automation


After all, what is quality?

Presented by Stephen Allott

CaseMaker InternationalA Diaz & Hilterscheid group company

Effective Testing

Using industry standard techniques

SoftTest Ireland 16th February 2006

Steven K Allott - Effective Testing - SoftTest Ireland

Documents

Transcript of Steven K Allott - Effective Testing - SoftTest Ireland