Online Covers for Presentations.PDF...BIO PRESENTATION International Conference On Software Testing...

BIOPRESENTATION

International Conference OnSoftware Testing Analysis & Review

May 12-16, 2003Orlando, FL USA

T11

May 15, 20031:30 PM

REQUIREMENTS-BASED

TESTING: AN OVERVIEW

Gary MogyorodiBIT, Inc.

Gary Mogyorodi Gary Mogyorodi has over 30 years of experience in the computing industry. Currently as a Lead Consultant with Bloodworth Integrated Technology (BIT), Inc., Mr. Mogyorodi consults, trains and mentors in software testing, specializing in Requirements-Based Testing. Mr. Mogyorodi has used Requirements-Based Testing since he began working for Bender & Associates in 1998. Prior to working for Bender & Associates, the majority of Mr. Mogyorodi’s career was with Dofasco Inc. From 1993, Mr. Mogyorodi was a Quality Assurance and Software Testing specialist, managing testing efforts, developing testing methodologies, and creating standards and procedures for quality assurance and testing. Prior to that, he worked at Dofasco as a Programmer, Systems Analyst and Manager of Software Development.. Gary Mogyorodi obtained a B. Math degree from the University of Waterloo, and an M.B.A. from McMaster University. A prolific speaker, Gary has delivered presentations at events including the (STC) Software Technology Conference 2001, the SQA User's Conference, CIPS (Canadian Information Processing Society, Toronto and Hamilton Chapters), TassQ (Toronto Association for System and Software Quality), CQAA (Chicago Quality Assurance Association), the STAR WEST Conference, the SQF (Software Quality Forum) Conference, the Toronto SPIN (Software Process Improvement Network), the PSQT/PSTT (Practical Software Quality Techniques/Practical Software Testing Techniques) North Conference (twice) and the PSQT/PSTT South Conference. Gary E. Mogyorodi, B.Math., M.B.A. Lead Consultant Bloodworth Integrated Technology (BIT), Inc.. 36A Mendota Road, Unit 8 Toronto, Ontario CANADA M8Y 1E8 Telephone: +1 (416) 521-7200 Email: [email protected] Corporate Headquarters: Bloodworth Integrated Technology (BIT), Inc. 12007 Sunrise Valley Drive, Suite 105 Reston, VA 20191 Telephone: + 1 (703) 295-0700 Web Site: www.bitspi.com

Process Engineering for Systems, Software and PeopleCopyright 2003

1

Requirements-Based Testing:An Overview

Gary E. Mogyorodi, B.Math., M.B.A.Lead Consultant

Bloodworth Integrated Technology (BIT), Inc.12007 Sunrise Valley Drive, Suite 105

Reston, VA 20191(703) 295-0700


2

Requirements-Based Testing: An Overview

© 2003 Bloodworth Integrated Technology (BIT), Inc. All rightsreserved.

No part of this material (including interior design, cover design,and illustrations) may be reproduced or transmitted in any form,by any means, (electronic, photocopying, recording, or otherwise)without the prior written permission of the publisher.

For authorization to photocopy items for internal corporate use,personal use, or for educational and/or classroom use, pleasecontact:

Bloodworth Integrated Technology (BIT), Inc.12007 Sunrise Valley Drive, Suite 105Reston, VA 20191 (703) 295-0700


3

Agenda

Ø The Business Case for Better Quality

Ø The Requirements-Based Testing Process

ØManagement Considerations

Ø Summary


4

Are You Suffering from Any of the Following…

Ø Poorly written requirements?Ø Your projects take longer than expected?Ø Too much scrap and rework in your software

development?Ø Defects aren’t discovered until System

Testing?Ø Test effectiveness varies from tester to tester?


5

The Goals

Ø Deliver MORE function,

Ø In LESS time,

ØWith FEWER resources,

AND

ØWith HIGHER QUALITY.


6

Cost of Software Errors

American Airlines

Outages in SABRE cost $20,000 per minute!1989 - 12 hour outage1994 - 5 hour outage

System error incorrectly showed flights full -$50,000,000 loss!

Denver Airport

$1,100,000 per day lost due to defects in baggagehandling system


7

(IBM, et. al.)

Relative Cost To Fix An Error

Phase In Which Found Cost Ratio Requirements 1 Design 3-6 Coding 10 System/Integration Testing 15-40 User Acceptance Testing 30-70 Operation 40-1000


8

Distribution of Bugs Distribution of EffortTo Fix Bugs

(James Martin)

Requirements82%

Design13%

Other4%Code

1%

Requirements56% Design

27%

Other10%Code

7%


9

Why Good Requirements are Critical

Standish Group Statistics for 1997

Ø In 1997, American companiesspent $100 BILLION for canceledsoftware projects.

Ø $45 BILLION spent for projectsthat significantly exceeded timeand budget estimates.

Standish Group Statistics for 2000

Ø In 2000, American companiesspent $84 BILLION on failedsoftware projects.

Ø $192 BILLION spent on projectsthat significantly exceeded time andbudget estimates, or had reducedfunctionality.


10

Why Good Requirements Are Critical

Top reasons for failure:

Ø Incomplete requirements and specifications

Ø Changing requirements and specifications

Ø Lack of user input

(Standish Group and other studies)


11

The Test Process - Make It:

ØTimely:Ø Integrated throughout the lifecycle

ØEffective:ØRigor in test definition

ØEfficient:ØHeavily automatedØMinimum number of test cases

ØManageable:ØMeasurableØPredictable


12

The “Standard” Development LifecycleRequirements

Design

Code

Test

Write UserManuals

Write TrainingMaterials

InternationalTranslations

TIM

E


13

Lifecycle With Testable Requirements andIntegrated Testing

TestRequirements

Requirements

TestDesign

Design

TestCode

Code

InternationalTranslations

Write UserManuals

Write TrainingManuals

TIM

E


14

The Requirements-Based Testing Methodology

1. Validate requirements (WHAT) against objectives (WHY)2. Apply use cases against requirements3. Perform initial Ambiguity Reviews4. Perform domain expert reviews5. Create Cause-Effect Graphs6. Logical consistency check and test cases designed by CaliberRBT7. Review test cases with Requirements Authors8. Validate test cases with Users/Domain experts9. Review test cases with Developers10. Walk test cases through Design11. Walk test cases through Code12. Verify code with test cases derived from requirements


15

Characteristics of a Testable Requirement

1. Deterministic

2. Unambiguous

3. Correct

4. Complete

5. Non-redundant

6. Lends itself to change control

7. Traceable

8. Readable by all projectmembers

9. Written in a consistent style

10. Processing rules reflectconsistent standards

11. Explicit

12. Logically consistent

13. Lends itself to re-usability

14. Terse

15. Annotated for criticality

16. Feasible


16

The Ambiguity Review Checklist

Ø Dangling elseØ Ambiguity of referenceØ Scope of actionØ OmissionsØ Causes without effectsØ Missing effectsØ Effects without causesØ Complete omissionsØ Missing causes

Ø Ambiguous logical operatorsØ Or, And, Nor, NandØ Implicit connectorsØ Compound operators

Ø NegationØ Scope of negationØ Unnecessary negationØ Double negation

Ø Ambiguous statementsØ Verbs, adverbs, adjectivesØ Variables, unnecessary aliases

Ø Random organizationØ Mixed causes and effectsØ Random case sequence

Ø Built-in assumptionsØ Functional/environmental

knowledgeØ Ambiguous precedence

relationshipsØ Implicit casesØ Etc.Ø I.E. versus E.G.Ø Temporal ambiguityØ Boundary ambiguity


17

Ambiguity Review Process

1. Make copy of original requirements.

2. Have non-domain-expert review for ambiguities.

3. Revise requirements as needed for ambiguity.

4. Have one or more domain experts reviewrequirements individually for content.

5. Domain experts meet to compare notes on content.

6. Revise requirements as needed for content.


18

Savings Via Early Testing

Ambiguity Reviews of requirements(Bender & Associates)Ø Costs per defects foundØ .85 hour/defectØ $75 hour fully burdened rate ($150K year)Ø $63.75 per defect

Ø Costs if found in integration test/system testØ $750 to $3,000 per defect (SEI)

Ø Cost if found in productionØ $10,000 per defect (HP)Ø $140,000 per defect (IBM)


19

Test Case Design Approaches

Goal:Design a necessary and sufficient set of testcases to ensure system integrity.Ø Production filesØGut feelØExhaustive “Combinatorics” of inputsØRigorous algorithms


20

Testing With Production Files

Ø Covers less than 30% of the code

Ø Exception cases not covered since data is alreadyscrubbed

Ø Time-dependent functions not covered

Ø Expected results not determined for every outputfield

Ø Might find some missing cases

Ø Have value in performance testing

Ø Have value in helping build test cases


21

Testing By Gut Feel

Totally dependent on who is doing the testing:

Ø How experienced they are at testingØ How experienced they are in the applicationØ How experienced they are in the technology that

the application runs onØ How they are feeling today

Even if all the tests run successfully, all you know isthat those tests run -- not that the system runssuccessfully


22

Testing By Brute Force Combinatorics

Amount of time to arrange 15 books in everypossible way:

2,487,996 YEARS at one change per minute

Length of paper required to write all the possible sequencesof the 26 letters of the alphabet:

160 million light years

QUESTION: how many variables are in your application?


23

Testing with a Rigorous Algorithm -- Example

This function has sixty-four possible combinations ofinput from which to select test cases:If the customer is a business client or a preferred personalclient and they have a checking account, $100,000 or more indeposits, no overdraft protection and fewer than 5 overdraftsin the last 12 months, then set up free overdraft protection.Otherwise, no overdraft protection.

How many test cases are required to confirm thatthe function works?

Answer: Six But can you name them?


24


25

Test Cases Designed By CaliberRBT

TEST#1 -- Automatic Check For Overdraft Protection

Cause states: The customer is a business client The customer has a checking account The customer has $100,000 or more in deposits The customer does not have overdraft protection Overdrawn less than five times in last 12 months

Effect states: Set up free overdraft protection

CaliberRBT takes the cause-effect information entered inVisio and designs a completeset of TEST CASES.

CaliberRBT takes the cause-effect information entered inVisio and designs a completeset of TEST CASES.


26

Test StatisticsAutomatic Check For Overdraft Protection

Run: Synthesis of New TestsNumber of input statements: 16

Number of Functional Variations: 9Number of infeasible variations: 0Number of untestable variations: 0

Number of new test cases defined: 6Number of tested variations: 9Number of Feasible Variations: 9Percentage of functional coverage of feasible variations: 9/9*100 = 100%

Number of tested variations: 9Percentage of functional coverage of testable variations: 9/9*100 = 100%

Number of Primary Causes: 6The THEORETICAL maximum number of test cases is: 2^6 = 64

The number of test cases generated by CaliberRBT is: 6

Summary statistics areproduced to aid in projectestimating and tracking.

Summary statistics areproduced to aid in projectestimating and tracking.

Test Statistics


27

Test Statistics For a Large ProblemTest StatisticsCHP_PG5_26/TOBACCO USE STATISTICS

Run: Synthesis of New TestsNumber of input statements: 112

Number of Functional Variations: 141Number of infeasible variations: 0Number of untestable Variations: 1

Number of new test cases defined: 22Number of tested variations: 140Number of Feasible Variations: 141Percentage of functional coverage of feasible variations: 140/141*100 = 99%

Number of tested variations: 140Percentage of functional coverage of testable variations: 140/140*100 = 100%

Number of Primary Causes: 37The THEORETICAL maximum number of test cases is: 2^37 = 137,438,953,472

The number of test cases generated by CaliberRBT is: 22


28

Ø Thought experimentØ Put 137,438,953,450 RED balls in this roomØ Add 22 GREEN balls to the room and mix wellØ Turn out the lights

Ø Pull out 22 ballsØWhat is the probability that you have selected the 22 green

ones?Ø Pull out 1,000 ballsØWhat is the probability that you have the 22 green ones now?

Ø Pull out 1,000,000 ballsØWhat is the probability that you have the 22 green ones now?

Justification for Rigorous Testing

** This is what “GUT FEEL” testing really is.**


29

Would You Rather Validate RequirementsThat Looks Like This:

James Joycian Novel Style

Dental Insurance Claims Payment SpecificationDentists with membership codes of 2, 3, or 9 are memberdentists. For claims referencing a non-member dentist orfor procedures not within the referenced dentist’s record, asystem table is used to calculate the amount paid.Otherwise, the amount submitted is paid. However, anoverride code of 1 or 9 allows the amount submitted to bepaid for non-member dentists or for procedures not withinthe referenced dentist’s record. When an override code isused an entry is made on the paid claims report.

Dental Insurance Claims Payment SpecificationDentists with membership codes of 2, 3, or 9 are memberdentists. For claims referencing a non-member dentist orfor procedures not within the referenced dentist’s record, asystem table is used to calculate the amount paid.Otherwise, the amount submitted is paid. However, anoverride code of 1 or 9 allows the amount submitted to bepaid for non-member dentists or for procedures not withinthe referenced dentist’s record. When an override code isused an entry is made on the paid claims report.


30

Or Validate Test Cases That Look Like This:

Cause States:The Dentist is a Member DentistThe procedure was not preauthorizedAn override code was entered

Effect States:This is a potential partial payment situationOverride the partial paymentPay the full amount of the claimMake an entry on the paid claims report

Cause States:The Dentist is a Member DentistThe procedure was preauthorized

Effect States:It is a valid procedure for the memberdentistPay the full amount of the claimDo not make an entry on the paid claimsreport

TEST CASE #1 TEST CASE #2


31

Management Considerations

Ø Staffing curve peaks earlierØRequirements written in more detailØDesign concurrent with requirementsØ Implementation preparation concurrent with

designØTesters involved from the beginningØTechnical writers involved earlier

Ø Total resources reducedØMinimize scrap and reworkØPlans have better focus on scope and priorities


32

Test Case Design/Test Management Status Report1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

Function Priority

Function Name

Req Drafted

Ambiguity Review

Complete

Req Corrected

for Ambiguity

Req Reviewed

for Content

Req Complete

Cause-Effect

Graphs Drawn

Test Cases

Designed

Test Cases

Reviewed

# of Functional Variations

# of Test

CasesModules

Coded

Test Cases Built

Test Cases

Executed

# of Defects

Identified

# of Defects Open

% Code

Coverage

1 A2 B3 C4 D5 E6 F7 G8 H9 I10 J11 K12 L13 M14 N15 O

TOTAL


33

Summary - What RBT Delivers:

ØVisual Test Case DesignØMaximum coverage with minimum testsØ100% functional coverageØ70-90% code coverage

ØQuantitative test progress metrics

ØTesting is no longer a bottleneck

ØHighly portable test scripts

ØTest cases for any application written in anylanguage running on any computer

Online Covers for Presentations.PDF...BIO PRESENTATION International Conference On Software Testing...

Documents

Transcript of Online Covers for Presentations.PDF...BIO PRESENTATION International Conference On Software Testing...