Test Case Effectiveness

8/10/2019 Test Case Effectiveness

1/16

Validating and Improving Test

Case Effectiveness

Yuri hernak

Presented by Michelle Straughan


2/16

Overview

Introduction

Test-Case Effectiveness Metric (TCE)

Improving Test-Case Effectiveness 5 steps to improving TCE

Summary


3/16

Introduction

Software testing is crucial to the successof product

Evaluation certifies product usageVerification of test specifications

Validation of test cases

New metric for evaluating test case

Framework for test-case effectiveness


4/16

Improving Test-Case Effectiveness Test-Case Effectiveness Metric

Formulation of metric to measure theeffectiveness of testing

Metric Definition: TCE = Ntc/Ntot*100%

TCE value compared to a baseline to

determine effectiveness (eg 75%baseline)


5/16

Improving Test-Case

EffectivenessAnalyze causes and areas improvements

are needed

Proposed improvement framework basedon test-escapes (defects test case suite didnot find but were side effects found in the

same test cycle)

Approach relies on analysis of misseddefects by test cases


6/16

5 steps to Improvement

1. Thoroughly understanding andproperly documenting the test process

2. Make Assumptions3. Gathering defect data and performing

causal analysis

4. Identifying main factors

5. Implementing corrective actions


7/16

Step 1: Thoroughlyunderstanding and properly

documenting the test processStages in Test

ProcessDescription Deliverables

Test Planning defines scope, objective andtest approach

Test Plan

Test Design designs test cases Test-CaseSpecifications

Test preparation andexecution-

preparation and setup of testenvironment, execution of test

cases, finding and reportingdefects

Defect Report

Test evaluation andimprovement

analyzing testing results Summary Report


8/16

Step 2: Make Assumptions

After completing step 1, project teammakes assumptions about factorsaffecting TCE

Test Planning Test objectives defined in test plan should

be traceable to functional specifications

Incomplete specifications lead to anincomplete test plan and leads to reducedTCE


9/16

Step 2: Make Assumptions Test Design

Necessary test cases to test logic of functionalfeatures need to be defined, if not adequatecoverage will be lacking

Test case specifications should correspond tobusiness rules

Deficiencies could be caused by: Incomplete test suite (all possible test cases are not

defined) Incomplete test cases proper logic is not tested

negative test cases Incorrect test-specifications

Test Preparation and Execution Test cases not executed Test cases improperly executed


10/16

Factors affecting test-case

effectiveness

Figure 1: Factors affecting test-case effectiveness


11/16

Step 3: Gathering defect data and

performing causal analysis

Figure 2: Factors based on causal analysis logic


12/16

Step 3: Gathering defect data and

performing causal analysis Testers need to track defects and

categorize defects. Test-escapes are

further categorized according to (fig. 2) Understand reasons for missed defects

Verify completeness of test specifications

Verify coverage of test-cases Check for test deficiencies

Verify the correctness of test cases


13/16

Step 4: Identifying main factors

Identifying factors that are causingmajor defects

Analyze these defects and focus on theseas most needing corrective measures


14/16

Step 4: Identifying main factors

Figure 3: Pareto chart


15/16

Step 5: Implementing corrective

actions Depending on project at hand, corrective

measures could vary. These may include: Revisiting and revising functional specifications,

which results in revising test case specifications Use of traceability matrix to meet coverage needs of

the project ensures that test cases meet businessrule

Training for testers on proper design techniques,

using checklists to help better design test casespecifications (involves revisiting and revising testcase specifications)

Training for testers on proper way to execute testsand verifying test results


16/16

Summary New framework is to aid in better

effectiveness of test-cases, however its notapplicable to some projects as it requires largenumber of defects

Framework would theoretically help to reducedefects present in a product when released toproduction, since it first checks the testprocess and if its not effective then provides

steps for achieving effectiveness Formalized way of achieving effectiveness,

other ways for improving effectiveness couldinclude the use of other testing techniques

Test Case Effectiveness

Documents

Transcript of Test Case Effectiveness