TDDD04Defect taxonomies and test plans Ola Leifler, ola.leifler@liu.se

Agenda• ISO/IEC/IEEE 29119 - Test planning • ISO/IEC/IEEE 1044 - Software anomalies

2

3

Organizationaltestprocess

ISO/IEC/IEEE, “ISO/IEC/IEEE international standard for software and systems engineering – software testing part 2: Test processes,” ISO/IEC/IEEE 29119-2:2013(E), pp. 1–68, Sept 2013.

Testmanagementprocesses

Testplanningprocess

Testmonitoringandcontrolprocess

Testcompletionprocess

Dynamictestprocesses

Testdesign&implementationprocess

Testenvironmentsetup&maintenanceprocess

Testexecutionprocess

Testincidentreportingprocess

4

Test planningUnderstandcontext

Organisetestplandevelopment

Iden5fyandanalyserisks

Iden5fyriskmi5ga5onapproaches

Designteststrategy

Determinestaffingandscheduling

Recordtestplan

Gainconsensusontestplan

Communicatetestplanandmake

available

Start

Testplan

scope

Testplandevelopmentschedule

Analyzedrisks

Mi5ga5onapproaches

Teststrategy

Scheduling,staffingprofile

DraHtestplan

Approvedtestplan

5

Organizationaltestprocess

ISO/IEC/IEEE, “ISO/IEC/IEEE international standard for software and systems engineering – software testing part 2: Test processes,” ISO/IEC/IEEE 29119-2:2013(E), pp. 1–68, Sept 2013.

Testmanagementprocesses

Testplanningprocess

Testmonitoringandcontrolprocess

Testcompletionprocess

Dynamictestprocesses

Testdesign&implementationprocess

Testenvironmentsetup&maintenanceprocess

Testexecutionprocess

Testincidentreportingprocess

Test design and implementation process

6

Iden%fyfeaturesets

Derivetestcondi%ons

Derivetestcoverageitems

Derivetestcases

Assembletestsets

Derivetestprocedures

Start

Featuresets

Testcondi%ons

Testcoverageitems

Testcases

Testsets

Testdesignspecifica%on

Testcasespecifica%on

Testcasespecifica%on

Testprocedures

andtestscripts

Example“The system shall accept insurance applicants over the age of 18 and under the age of 80 years on the day of application based on their input age in whole years; all others shall be rejected. Accepted applicants of 70 and over shall receive a warning that in the event of a claim they shall pay an excess of $1000.”

7

Derive test conditions• Completion criterion? • Valid input? • Invalid input? • What if we note the following: 40 <= Age <= 55 results in a discount

message (unspecified in the description). How do we handle that?

8

Derive test coverage items• Equivalence class items to cover

9

Derive test cases• Select representatives from each class to achieve 100% equivalence class

coverage

10

Assemble test sets• What can be automated? • What must be manually tested?

11

Derive test procedures• Ordering of test cases based on exposure/dependencies • Traceability

12

Software anomaliesHow to classify anomalies according to ISO/IEC/IEEE 1044

ISO/IEC/IEEE,“IEEEstandardclassificationforsoftwareanomalies,”IEEEStd1044-2009(RevisionofIEEEStd1044-1993)

Error(Mistake)

Fault(Defect,Bug)

Failure

maycause

Incident(Symptom)

maybeobservedas

Test

Testcaseexercises

mayinduce

Remember this?

Defect - FaultFaults are executed defects. Some defects are detected before they become faults

Attribute Definition

DefectID

Description Whatismissing,wrong,orunnecessary?

Status Currentstatewithindefectreportlifecycle.

Asset Thesoftwareasset(product,component,module,etc.)containingthedefect.

Artifact Thespecificsoftwareworkproductcontainingthedefect(specification,design,code).

Versiondetected

Versioncorrected

Priority Rankingforprocessingassignedbytheorganization

Severity Thehighestfailureimpactthatthedefectcould(ordid)cause,asdeterminedby(fromtheperspectiveof)theorganizationresponsibleforsoftwareengineering.

Probability

Effect Theclassofrequirementthatisimpactedbyafailurecausedbyadefect.

Type Isthedefectduetocode,data,configuration?

Mode Omissionoffeature?Misrepresentationofspec?Unnecessaryfeature?

Defect attributes

…

Example 1 Attribute Value Definition

Effect Functionality Actualorpotentialcauseoffailuretocorrectlyperformarequiredfunction(orimplementationofafunctionthatisnotrequired),includinganydefectaffectingdataintegrity.

Effect Usability Actualorpotentialcauseoffailuretomeetusability(easeofuse)requirements.

Effect Security Actualorpotentialcauseoffailuretomeetsecurityrequirements,suchasthoseforauthentication,authorization,privacy/confidentiality,accountability(e.g.,audittrailoreventlogging),andsoon.

Effect Serviceability Actualorpotentialcauseoffailuretomeetrequirementsforreliability,maintainability,orsupportability(e.g.,complexdesign,undocumentedcode,ambiguousorincompleteerrorlogging,etc.).

Example 1

Attribute Value Definition

Type Data Defectindatadefinition,initialization,mapping,access,oruse,asfoundinamodel,specification,orimplementation.Examples:Variablenotassigned,initialvalueorflagnotset,incorrectdatatype…

Type Interface Defectinspecificationorimplementationofaninterface(e.g.,betweenuserandmachine,betweentwointernalsoftwaremodules,betweensoftwaremoduleanddatabase,betweeninternalandexternalsoftwarecomponents,betweensoftwareandhardware,etc.).Examples:Incorrectmoduleinterfacedesignorimplementation,incorrectreportlayout(designorimplementation),incorrectorinsufficientparameterspassed

Attribute DefinitionFailureID Uniqueidentifierforthefailure.Status Currentstatewithinfailurereportlifecycle.Title Briefdescriptionofthefailureforsummaryreportingpurposes.Description Fulldescriptionoftheanomalousbehaviorandtheconditionsunderwhichitoccurred,

includingthesequenceofeventsand/oruseractionsthatprecededthefailure.

Analysis Finalresultsofcausalanalysisonconclusionoffailureinvestigation.Environment Identificationoftheoperatingenvironmentinwhichthefailurewasobserved.

Configuration Configurationdetailsincludingrelevantproductandversionidentifiers.

Testreference Identificationofthespecifictestbeingconducted(ifany)whenthefailureoccurred.

Incidentreference

Identificationoftheassociatedincidentifthefailurereportwasprecipitatedbyaservicedeskorhelpdeskcall/contact.

Defectreference

Identificationofthedefectassertedtobethecauseofthefailure.

Failure attributes

…

Case 1

“Sue calls service desk and reports she cannot log in to timesheet system because the password field is missing from the login screen.” — In this example, Sue has a problem in that she cannot log in, caused by a failure wherein the password field did not appear on the login screen, which was in turn caused by a defect inserted during coding of the Login.asp artifact.

Failure/Defect Attribute ValueFailure FailureID 1Failure Status OpenFailure Title MissingpasswordfieldFailure Description

Failure AnalysisFailure Environment

Defect Asset

Defect Artifact

Defect Severity

Defect Probability

Defect Effect

Defect Type

Defect Mode

Case 1

Case 2

“Joe calls service desk and reports he cannot log in to timesheet system because the password field is missing from the login screen.”

Duplicate: several failures can indicate the same fault

Case 3

“During a peer review for software requirements for a new financial management system, Alice discovers that values are in the requirements as thousands of dollars instead of as millions of dollars.”

Failure/Defect Attribute ValueDefect DefectID 2Defect Status ClosedDefect Description

Defect Asset

Defect Artifact

Defect Severity

Defect Probability

Defect Effect

Defect Type

Defect Mode

Case 3

Case 4

“Company A’s battery ran out of power because there was no low-power warning. The design of a security system monitoring system did not include a warning for low battery power, despite the fact that this feature was specified in the requirements.”

Failure/Defect Attribute ValueFailure FailureID 3Failure Status OpenFailure Title MissinglowpowerbatteryalertFailure Description

Failure AnalysisFailure Environment

Defect Asset

Defect Artifact

Defect Severity

Defect Probability

Defect Effect

Defect Type

Defect Mode

Case 4

SEI Risk Identification TaxonomyClass Element Attribute

ProductEngineering

Requirements

Stability

Completeness

Clarity

Validity

Feasibility

Precedent

Scale

DesignFunctionality

Difficulty

So, how does that relate to me as a tester?

IfIamconcernedabout… ThenIwanttoemphasize

Requirementstability Traceability(testplanning)

Incompleterequirements Exploratorytesting

Unclearrequirements Decisiontables/statetransitiontesting

Designdifficulty Control-flowtesting

… …