Steven K Allott - Effective Testing - SoftTest Ireland

Presented by Stephen Allott

CaseMaker InternationalA Diaz & Hilterscheid group company

Effective Testing

Using industry standard techniques

SoftTest Ireland 16th February 2006

Agenda

• Why do we test software?

• How do we test effectively?

• Risk based testing• Black Box techniques• Pairwise testing

Photo: Courtesy of NASA

Why test software?

Read all about it!

US site – down for 3 days,

forced to return $30 fee to nearly

260,000 online “fantasy

baseball” players.

US site down for 2

days, $280,000 in

lost sales.

Shopping Site: 6-hour downtime:

$400,000 in direct costs plus additional

in discounts to inconvenienced

customers.

UK - Concerns over bugs

and delays in new air

traffic control software

UK – Payment clearing system failure at month end

delayed wages to many customers

UK - Televisions,

Cameras sold at

very low prices

Ineffective Testing

• Creates too many of the wrong tests• Duplicates effort unnecessarily• Costs too much and takes too long• Has an adverse impact on the business as a

Effective Testing

• Is aligned with the business need• Uses a risk-based approach and formal

techniques to create fewer tests with increased coverage

• Facilitates communication with the business• Provides pre-production confidence• Saves time and money

Risk Based Testing

• Identify business risks• Assess the impact

– Severity, if it were to fail– Probability of failure

• Focus more testing on high risk areas

• Continually measure and monitor the risks

Test Case Design Techniques – 1

• Black box and white box techniques have been proven to be effective

• Techniques are described in BS7925• Analysis/design separated from execution• Improved and demonstrable test coverage

• Analysis of the test conditions• A formal design process gives you

– Objectivity– Repeatability– Consistency– Documentation– Auditability– Traceability– Confidence

• The design activity alone can find faults• Helps drive the need for good requirements• Use of formal techniques helps with estimates

of overall test effort required• Supports test automation

Black Box Techniques

• BS7925 techniques– Equivalence class

partitioning– Boundary value

analysis– Decision tables– State transition

testing– Syntax testing– Random testing

• Other techniques– Pairwise– Domain analysis– Use case testing

White Box Techniques

All Paths

where: B - Branch coverage/Decision coverage BC - Branch Condition coverage BCC - Branch Condition Combination coverage DCU - All C-uses coverage DPU - All P-uses coverage DU - All du-paths coverage L - LCSAJ coverage MCDC - Modified Condition Decision coverage S - Statement coverage

Test Data

• Copy of live data• Realistic data volumes• Range of values and

inputs

Equivalence Class Partitioning

• Can you ever see a need to: – try the view from every seat of a cinema or theatre?– test drive every car in the showroom?– withdraw cash from all the ATMs in your country?

• If you answered “No, of course not!”– what would you do instead, and why?

• Equivalence class or partition• Ranges of data• Based on an assumption about the software’s

behaviour• Reduces the number of test cases whilst

maintaining reasonable test coverage

• An online car insurance quotation system allows a 30% discount for good drivers aged between 25 and 50 but charges full price if you are under 25. For older careful drivers a 20% discount is allowed and if you are retired we will be happy to quote you a special price

Under 17 17 to 24 25 to 49 50 to 64 65 and over

£1,000 £700 £800

Boundary Value Analysis

• Builds on Equivalence Class technique• Assumes errors are made on the boundaries• Applicable to value and structure of data

– where there is a continuous range of values– length and character type

• Generates three test cases per boundary– one on the boundary– one either side of the boundary– the smallest distance away

Boundary Value Analysis – Example

• An online car insurance quotation system – Under 17 – unable to quote– 17 to 24 (premium price)– 25 to 49 (30% discount)– 50 to 64 (20% discount)– 65 and over (price on application)

Under 17 17 to 24 25 to 49 50 to 64 65 and over

16,17,18 49,50,5124,25,26 64,65,66

• Online Banking (Account balance)• Lower boundaries

– EUR 0.01 (DR)– EUR 0.00 (CR)– EUR 0.01 (CR)

• Upper boundaries– EUR 999,999.99 (CR)– EUR 1,000,000.00 (CR)– EUR 1,000,000.01 (exceeds max for this account

• Online Banking (Set up standing order)• Boundaries for from date

– Yesterday (invalid)– Today (should be allowed)– Tomorrow (should be allowed)

• Boundaries for until date– 12 months from today (ok)– 12 months less 1 day (ok)– 12 months and one day from today (invalid)

Decision Tables

• Another intuitive black box technique• Express complex business rules in a table• Cause-Effect graphing helps with the analysis • Define the combination of conditions (causes)• Define the resulting actions (effects)• Easy to create test cases

Example – Train Tickets

Conditions

Rule 1

Rule 2

Rule 3

Rule 4

Holds a railcard Y Y N N

Travels after 9 am

Y N Y N

ActionsPay £10

Pay £15

Pay £20

Pay £30

Example – Train Tickets Test Cases

Inputs

Test 1

Test 2

Test 3

Test 4

Holds a railcard Y Y N N

Travels after 9 am

Y N Y N

Expected results

Pay £10

Pay £15

Pay £20

Pay £30

Example – ATM

Conditions

Rule 1

Rule 2

Rule 3

Rule 4

Enter valid card Y Y N N

Enter valid PIN Y N Y N

Actions Auth Reject Reject Reject

Example – ATM “don’t care”

Conditions

Rule 1

Rule 2

Rule 3

Enter valid card Y Y N

Enter valid PIN Y N ---

Actions Auth Reject Reject

Example – File Backup

ConditionsRule

Stick inserted N Y Y Y

Stick formatted --- N Y Y

Sufficient space --- --- N Y

Actions Fails Fails Fails Works

Pairwise Testing

Acknowledgement

This session on pairwise testing techniques is based on the book by Lee Copeland: A practitioner’s guide to software test

design, published by Artech House, 2004, ISBN 1-58053-791-X

Pairwise Testing

• Pairwise is based on testing pairs of combinations of variables rather than all combinations

• It reduces the number of test cases• Many faults are either

– Single mode– Double mode

• There are two recognised methods:– Orthogonal arrays– Allpairs algorithm

Pairwise Example

• Consider a simple database enquiry screen:

– It has 4 fields

– Each field has three possible choices

– Giving 3 X 3 X 3 X 3 = 81 possible combinations to test

• With Pairwise Testing

– You can reduce this to 9 tests!

Pairwise Example

• Holiday destination– UK, USA, Europe

• Price range (for family of 5)– Budget: Under £2000– Medium: £2000 to £5000– Expensive: Over £5000

• Standard of hotel– Basic, 4 star, luxury

• Availability– Now: Within 7 days from today– Soon: After a week but up to one month from today– Later: Later than one month from today

Orthogonal Array L9 (34)

A B C D

Row 1 1 1 1 1

Row 2 1 2 2 2

Row 3 1 3 3 3

Row 4 2 1 2 3

Row 5 2 2 3 1

Row 6 2 3 1 2

Row 7 3 1 3 2

Row 8 3 2 1 3

Row 9 3 3 2 1

This orthogonal array has 9 rows, 4 columns and three ‘levels’ per column

Where Price Hotel When

Row 1 1 1 1 1

Row 2 1 2 2 2

Row 3 1 3 3 3

Row 4 2 1 2 3

Row 5 2 2 3 1

Row 6 2 3 1 2

Row 7 3 1 3 2

Row 8 3 2 1 3

Row 9 3 3 2 1

Each variable becomes a column in the orthogonal array

Row 1 UK 1 1 1

Row 2 UK 2 2 2

Row 3 UK 3 3 3

Row 4 US 1 2 3

Row 5 US 2 3 1

Row 6 US 3 1 2

Row 7 Europe 1 3 2

Row 8 Europe 2 1 3

Row 9 Europe 3 2 1

For each column we fill out the possible choices

Row 1 UK Budget Basic Now

Row 2 UK Medium 4 star Soon

Row 3 UK High Luxury Later

Row 4 US Budget 4 star Later

Row 5 US Medium Luxury Now

Row 6 US High Basic Soon

Row 7 Europe Budget Luxury Soon

Row 8 Europe Medium Basic Later

Row 9 Europe High 4 star Now

Now the array is complete

Test 1 UK Budget Basic Now

Test 2 UK Medium 4 star Soon

Test 3 UK High Luxury Later

Test 4 US Budget 4 star Later

Test 5 US Medium Luxury Now

Test 6 US High Basic Soon

Test 7 Europe Budget Luxury Soon

Test 8 Europe Medium Basic Later

Test 9 Europe High 4 star Now

Each row becomes a test case – you can add more if you wish

How to use Orthogonal Arrays

• Identify the variables• Determine the number of choices for each• Locate an orthogonal array of the right size• Map the testing problem onto the array• Construct the test cases

What size array do we need?

Variable Choices

Browsers IE5, IE6, …. 8 in total

Plug-ins None, Real player, Media Player

Operating systemsW95, W98, W2000, NT, XP, XP-PRO

Server IIS, Apache, WebLogic

Server O/S NT, XP, Linux

How many possible combinations are there?

Using Orthogonal Arrays

• We require an array with 5 columns– Column 1 needs 8 levels (browsers)– Column 2 would need 3 levels (plug ins)– Column 3 would need 6 levels (O/S)– Columns 4 & 5 would need 3 levels (Server, Server O/S)

• Ideally we need 81 61 33 but it doesn’t exist!

• The next largest is L64 (82 43) – 2 columns of 1 – 8– 3 columns of 1 – 4

• So rather than 1,296 tests we need only 64 tests!• Remember - one test case per row

Allpairs Algorithm

• Devised by James Bach• Generates fewer tests• In our example, it generates just 48 tests

compared to 64 from using orthogonal arrays• Consider 100 variables, 2 choices

– Using orthogonal arrays you need 101 tests– Using all pairs you need only 10 tests

Pairwise Testing – Summary

• Test all pairs rather than all combinations– With 4 variables and 3 choices for each you would

require 81 test cases whereas you can cover all pairwise input combinations with only 9 tests

– For 13 variables and 3 choices for each you would require1,594,323 test cases whereas you can cover all pairwise input combinations with only 15 tests

• Pairwise guides us to help find– Single mode defects– Double mode defects

• You should always add any known risky combinations

Effective Testing – Summary

• Effective Testing uses– A risk-based approach– Formal test case design techniques

• The benefits are– Reduced number of test cases– Repeatable well documented tests– Improved planning, estimation and communication– Facilitates earlier involvement with the business– Find faults earlier– Give you a consistent approach to testing– Save time and money– Enables test automation

After all, what is quality?

Presented by Stephen Allott

CaseMaker InternationalA Diaz & Hilterscheid group company

Effective Testing

Using industry standard techniques

SoftTest Ireland 16th February 2006

Steven K Allott - Effective Testing - SoftTest Ireland

Documents

Transcript of Steven K Allott - Effective Testing - SoftTest Ireland

AUTHOR: STEVEN DOWNES - Miniature Construction World · 12 model tractor AUTHOR: STEVEN DOWNES | PHOTOS: STEVEN DOWNES & MANUFACTURERS NEW RELEASES: Steven Downes looks over the latest

Agile Maturity and the - softtest.iesofttest.ie/Wp-content/Uploads/2016/10/SoftTest.pdfAgile Maturity and the Quality custody-battle SoftTest 2016 - “Adapting & Innovating for Change”

Building Software for a Continuous Delivery Modelsofttest.ie/wp-content/uploads/2017/03/SoftTest... · HPE Helion OpenStack Releases Accelerating with Automation 6 2015 2016 2017

LANGUAGE ORIGINS SOCIETY UNIVERSITY OF CAMBRIDGE Selwyn College September 1992 Robin Allott AUTISM AND THE MOTOR THEORY OF LANGUAGE.

Crisis, Catharsis & Customers Life Insurer Responses to the Great Recession Presented by Adrian Allott Senior Consultant 25 th May 2010.

Key Business Factors Steven Thorne Steven Thorne.

Allott Neurobiology & Behavior Text

SoftTest Ireland - Digital Engineering Evolution: DevOps

SoftTest Ireland Survey 2007

LANGUAGE ORIGINS SOCIETY UNIVERSITY OF NIJMEGEN 4-5 JULY 2003 Robin Allott LANGUAGE AND SPEECH AS MOTOR ACTIVITIES.

SoftTest Ireland: Model Based Testing - January 27th 2011

EUROPEAN SOCIETY FOR THE STUDY OF COGNITIVE SYSTEMS Robin Allott Presentations ------------------------------------------------- 1. The Evolution/Machine:

Neil Tompson - SoftTest Ireland

David Parnas - Documentation Based Software Testing - SoftTest Ireland

Stephen K. Allott - People Challenges for Test Teams - SoftTest Ireland

Stephen Allott Business of Software Conference Europe 2015 Scaling Software Companies

Allott - State Responsibility and the Unmaking of International Law

Whittaker How To Break Software Security - SoftTest Ireland

NEW COLLEGE 26-28 September 2011 EMBODIED LANGUAGE AND THE ASCENT OF INTELLIGENCE Gesture and Language : Mind and Body Robin Allott.

Test Process Improvement Asking the BIG questions? Graham Thomas SOFTTEST Belfast, Dublin, Galway 30 th Nov – 2 nd Dec 2011.