Comparison of available Methods to Estimate Effort, Performance and Cost with the Proposed Method
Estimate Testing Size and Effort Using Test Case Point Analysis
description
Transcript of Estimate Testing Size and Effort Using Test Case Point Analysis
![Page 1: Estimate Testing Size and Effort Using Test Case Point Analysis](https://reader035.fdocuments.in/reader035/viewer/2022062222/56816498550346895dd66f27/html5/thumbnails/1.jpg)
Estimate Testing Size and Effort Using Test Case Point
AnalysisVu [email protected] and QASymphony
Vistacon 2011, December 7th, 2011
![Page 2: Estimate Testing Size and Effort Using Test Case Point Analysis](https://reader035.fdocuments.in/reader035/viewer/2022062222/56816498550346895dd66f27/html5/thumbnails/2.jpg)
2
Agenda
• Objectives• Background and Motivation• Existing Estimation Methods• Test Case Point Analysis• Effort Estimation Methods using TCP• Conclusion and Future Work
![Page 3: Estimate Testing Size and Effort Using Test Case Point Analysis](https://reader035.fdocuments.in/reader035/viewer/2022062222/56816498550346895dd66f27/html5/thumbnails/3.jpg)
3
Objectives
• Discuss importance of software estimation, especially for testing projects
• Discuss existing methods for estimating testing activities
• Introduce an approach to estimating testing projects– Test Case Point Analysis– Effort Estimation using TCP
![Page 4: Estimate Testing Size and Effort Using Test Case Point Analysis](https://reader035.fdocuments.in/reader035/viewer/2022062222/56816498550346895dd66f27/html5/thumbnails/4.jpg)
4
Agenda
• Objectives• Background and Motivation• Existing Estimation Methods• Test Case Point Analysis• Effort Estimation Methods using TCP• Conclusion and Future Work
![Page 5: Estimate Testing Size and Effort Using Test Case Point Analysis](https://reader035.fdocuments.in/reader035/viewer/2022062222/56816498550346895dd66f27/html5/thumbnails/5.jpg)
Software Estimation and Its Importance
• Software estimation– process of determining the cost, time, staff, and other related
attributes of software projects, often before work is performed• Estimation is important for the success or failure of software
projects• Provides inputs for
– Making investment decisions– Budget and staff allocation– Tradeoff and risk analysis – Project planning– Stakeholder negotiation– etc.
5
![Page 6: Estimate Testing Size and Effort Using Test Case Point Analysis](https://reader035.fdocuments.in/reader035/viewer/2022062222/56816498550346895dd66f27/html5/thumbnails/6.jpg)
Popular Software Estimation Methods
• Sizing Methods– Source Lines of Code (SLOC)– Function Points Analysis– Use Case Points
• Effort Estimation Methods– Expert judgment/experience– Productivity index– COCOMO, SEER-SIM, SLIM models
• Software testing estimation methods– Using a test distribution percentage– Test Case Points Analysis (by Cognizant Technology Solutions) [2]
• Mainly based on the number of steps
6
![Page 7: Estimate Testing Size and Effort Using Test Case Point Analysis](https://reader035.fdocuments.in/reader035/viewer/2022062222/56816498550346895dd66f27/html5/thumbnails/7.jpg)
7
Motivation
• Testing accounts for up to 50% of project effort [1]• Current problems
– estimates are done for the whole project rather than testing specific
– lack of reliable methods designed for estimating size and effort of software testing
– vague definitions of testing productivity • due to the lack of a size measure for software testing
• There are needs of– accurately estimating effort of testing activities– measuring size and productivity of testing activities– measuring effectiveness and efficiency of software testing
![Page 8: Estimate Testing Size and Effort Using Test Case Point Analysis](https://reader035.fdocuments.in/reader035/viewer/2022062222/56816498550346895dd66f27/html5/thumbnails/8.jpg)
8
Motivation (cont’d)
• Our aim to– Defining a method for estimating the size of testing
activities– Defining methods to estimate testing effort, schedule,
and staff accurately using this size measure– Using this size measure as a basis for measuring
• productivity • effectiveness• other testing metrics
![Page 9: Estimate Testing Size and Effort Using Test Case Point Analysis](https://reader035.fdocuments.in/reader035/viewer/2022062222/56816498550346895dd66f27/html5/thumbnails/9.jpg)
9
Agenda
• Objectives• Background and Motivation• Existing Estimation Methods• Test Case Point Analysis• Effort Estimation Methods using TCP• Conclusion and Future Work
![Page 10: Estimate Testing Size and Effort Using Test Case Point Analysis](https://reader035.fdocuments.in/reader035/viewer/2022062222/56816498550346895dd66f27/html5/thumbnails/10.jpg)
10
Test Case Point Analysis
• Principles– Size must reflect the mass and complexity of the testing project– Size should correlate with testing effort– Test case point is measured using test cases as main input– Test case complexity is based on
• Checkpoints• Precondition• Test Data• Type of test case
![Page 11: Estimate Testing Size and Effort Using Test Case Point Analysis](https://reader035.fdocuments.in/reader035/viewer/2022062222/56816498550346895dd66f27/html5/thumbnails/11.jpg)
11
Test Case Point Analysis (cont’d)
• Process overview
Test Case Test Case Point Analysis
Test Case Points
Estimate Testing Effort
Estimated Effort
Project Attributes
![Page 12: Estimate Testing Size and Effort Using Test Case Point Analysis](https://reader035.fdocuments.in/reader035/viewer/2022062222/56816498550346895dd66f27/html5/thumbnails/12.jpg)
12
Test Case Point Analysis (cont’d)
• TCPA estimates the size of testing projects using test cases as input
• Steps1. Measure test case complexity
• Count checkpoints• Measure the complexity of precondition • Measure the complexity of data
2. Adjust Test Case Point by type of test• Consider 11 types of test case
o Each type has a weighto Adjust TCP using the weight
• Use functional test cases as baseline with the weight of 1.0
![Page 13: Estimate Testing Size and Effort Using Test Case Point Analysis](https://reader035.fdocuments.in/reader035/viewer/2022062222/56816498550346895dd66f27/html5/thumbnails/13.jpg)
13
Checkpoint
• Checkpoint – Is the condition in which the tester verifies whether the
result produced by the target function matches the expected criterion
– One test case consists of one or many checkpoints• Counting rule 1
One checkpoint is counted as one Test Case Point
![Page 14: Estimate Testing Size and Effort Using Test Case Point Analysis](https://reader035.fdocuments.in/reader035/viewer/2022062222/56816498550346895dd66f27/html5/thumbnails/14.jpg)
14
Test Case Precondition
• Precondition – specifies the condition to execute the test case – mainly affects the cost to execute the test case– may be related to data prepared for the test case
• Four levels of complexity– None– Low– Medium– High
![Page 15: Estimate Testing Size and Effort Using Test Case Point Analysis](https://reader035.fdocuments.in/reader035/viewer/2022062222/56816498550346895dd66f27/html5/thumbnails/15.jpg)
15
Test Case Precondition (cont’d)
• None– The precondition is not applicable or important to execute the test case – Or, the precondition is just reused from the previous test case to continue
the current test case• Low
– The condition for executing the test case is available with some simple modifications required
– Or, some simple set-up steps are needed • Medium
– Some explicit preparation is needed to execute the test case – The condition for executing is available with some additional modifications
required – Or, some additional set-up steps are needed
• High– Heavy hardware and/or software configurations are needed to execute the
test case
![Page 16: Estimate Testing Size and Effort Using Test Case Point Analysis](https://reader035.fdocuments.in/reader035/viewer/2022062222/56816498550346895dd66f27/html5/thumbnails/16.jpg)
16
Test Case Precondition (cont’d)
• Counting rule 2A: Counting Unadjusted Test Case Points for Precondition:
Each complexity level of precondition is assigned a number of Test Case Points
Based on our survey of 18 senior QA engineers
![Page 17: Estimate Testing Size and Effort Using Test Case Point Analysis](https://reader035.fdocuments.in/reader035/viewer/2022062222/56816498550346895dd66f27/html5/thumbnails/17.jpg)
17
Test Data
• Test Data – used to execute the test case – can be generated at the test case execution time, sourced from
previous tests, or generated by test scripts– Test data is test case specific, or general to a group of test cases, or
for the whole system
• Four levels of complexity– None– Low– Medium– High
![Page 18: Estimate Testing Size and Effort Using Test Case Point Analysis](https://reader035.fdocuments.in/reader035/viewer/2022062222/56816498550346895dd66f27/html5/thumbnails/18.jpg)
18
Test Data (cont’d)
• None– No test data preparation is needed
• Low– Simple test data is needed and can be created during the test case execution time– Or, the test case uses a slightly modified version of existing test data and requires little
or no effort to modify the test data
• Medium– Test data is deliberately prepared in advance with extra effort to ensure its
completeness, comprehensiveness, and consistency
• High– Test data is prepared in advance with considerable effort to ensure its completeness,
comprehensiveness, and consistency – This could include using support tools to generate data and a database to store and
manage test data– Scripts may be required to generate test data
![Page 19: Estimate Testing Size and Effort Using Test Case Point Analysis](https://reader035.fdocuments.in/reader035/viewer/2022062222/56816498550346895dd66f27/html5/thumbnails/19.jpg)
19
Test Data (cont’d)
• Counting rule 2B: Counting Unadjusted Test Case Points for Test Data:
Based on our survey of 18 senior QA engineers
Each complexity level of Test Data is assigned a number of Test Case Points
![Page 20: Estimate Testing Size and Effort Using Test Case Point Analysis](https://reader035.fdocuments.in/reader035/viewer/2022062222/56816498550346895dd66f27/html5/thumbnails/20.jpg)
20
Adjust Test Case Point
• Test Case Point counted till this point is considered Unadjusted Test Case Point (UTCP)
• UTCP is adjusted by considering types of test case– Each type of test case is assigned a weight– Adjusted Test Case Point (ATCP): n
ATCP = ∑UTCPi * Wi
i=1
• UTCPi - the number of UTCP counted for the test case ith.• Wi - the weight of the test case ith, taking into account its test type
![Page 21: Estimate Testing Size and Effort Using Test Case Point Analysis](https://reader035.fdocuments.in/reader035/viewer/2022062222/56816498550346895dd66f27/html5/thumbnails/21.jpg)
21
Weight by Type of Test
Based on our survey of 18 senior QA engineers
![Page 22: Estimate Testing Size and Effort Using Test Case Point Analysis](https://reader035.fdocuments.in/reader035/viewer/2022062222/56816498550346895dd66f27/html5/thumbnails/22.jpg)
22
Weight by Type of Test (cont’d)
![Page 23: Estimate Testing Size and Effort Using Test Case Point Analysis](https://reader035.fdocuments.in/reader035/viewer/2022062222/56816498550346895dd66f27/html5/thumbnails/23.jpg)
Test Case Point Analysis - Summary
• Process overview
Test Case
Count Checkpoints
Determine Precondition Complexity
Determine Test Data
Complexity
UTCP Adjust with Test Type
TCP
23
![Page 24: Estimate Testing Size and Effort Using Test Case Point Analysis](https://reader035.fdocuments.in/reader035/viewer/2022062222/56816498550346895dd66f27/html5/thumbnails/24.jpg)
24
Agenda
• Objectives• Background and Motivation• Existing Estimation Methods• Test Case Point Analysis• Effort Estimation Methods using TCP• Conclusion and Future Work
![Page 25: Estimate Testing Size and Effort Using Test Case Point Analysis](https://reader035.fdocuments.in/reader035/viewer/2022062222/56816498550346895dd66f27/html5/thumbnails/25.jpg)
25
Estimate Testing Effort
• Estimate testing effort using TCP• Test effort distribution, four phases
– Test Planning– Test Analysis and Design– Test Execution– Test Tracking and Reporting
• Effort estimation methods– Productivity index– Regression
Each of these phases may be performed multiple times
![Page 26: Estimate Testing Size and Effort Using Test Case Point Analysis](https://reader035.fdocuments.in/reader035/viewer/2022062222/56816498550346895dd66f27/html5/thumbnails/26.jpg)
26
Estimate Testing Effort (cont’d)
• When estimating TCP, assume that each phase is performed ONCE
• Effort is estimated dependent on how many times a phase is performed
![Page 27: Estimate Testing Size and Effort Using Test Case Point Analysis](https://reader035.fdocuments.in/reader035/viewer/2022062222/56816498550346895dd66f27/html5/thumbnails/27.jpg)
Productivity Index
• Effort is computed using productivity index of completed project
• Productivity index is measured as person-month per TCP
Effort = TCP * Productivity Index
Simple method
27
![Page 28: Estimate Testing Size and Effort Using Test Case Point Analysis](https://reader035.fdocuments.in/reader035/viewer/2022062222/56816498550346895dd66f27/html5/thumbnails/28.jpg)
Regression
• Estimate effort of new projects using size and effort of completed projects
0 100 200 300 400 500 600 700 800 900 10000
10
20
30
40
50
60
70
80
90
100
f(x) = 0.0728501072514328 x + 1.64076780728634
Adjusted TCP
Effo
rt (P
M)
28
Regression formula
![Page 29: Estimate Testing Size and Effort Using Test Case Point Analysis](https://reader035.fdocuments.in/reader035/viewer/2022062222/56816498550346895dd66f27/html5/thumbnails/29.jpg)
Tools Support by QASymphony [3]
• qTest – Quality Management Suite– Manage requirements and test cases – Plan testing activities and resources– Monitor and control testing activities– Perform measurement, analysis, and reporting
• Test case point analysis• Effort estimation
• qTrace – defect and scenario recording tool– Record defect – Record user scenarios
29
![Page 30: Estimate Testing Size and Effort Using Test Case Point Analysis](https://reader035.fdocuments.in/reader035/viewer/2022062222/56816498550346895dd66f27/html5/thumbnails/30.jpg)
Conclusion and Future Work
• It is important to estimate size and effort of testing projects
• Test Case Point Analysis (TCPA) was proposed to estimate size. Elements include– Checkpoints– Precondition– Test Data– Test Type
• Simple two methods for estimating effort were introduced– Productivity index– Regression
30
![Page 31: Estimate Testing Size and Effort Using Test Case Point Analysis](https://reader035.fdocuments.in/reader035/viewer/2022062222/56816498550346895dd66f27/html5/thumbnails/31.jpg)
Conclusion and Future Work
• Advantages of TCPA– Easy to implement– Reflecting real complexity of test cases– Independent with the number of steps
• Limitations and future work– Need empirical validations– Recalibrate constants used in the method– Support only manual testing– Lack of support for performance testing
31
![Page 32: Estimate Testing Size and Effort Using Test Case Point Analysis](https://reader035.fdocuments.in/reader035/viewer/2022062222/56816498550346895dd66f27/html5/thumbnails/32.jpg)
Thank You
![Page 33: Estimate Testing Size and Effort Using Test Case Point Analysis](https://reader035.fdocuments.in/reader035/viewer/2022062222/56816498550346895dd66f27/html5/thumbnails/33.jpg)
References
• [1] Y. Yang, Q. Li, M. Li, Q. Wang, An empirical analysis on distribution patterns of software maintenance effort, International Conference on Software Maintenance, 2008, pp. 456-459
• [2] N. Patel, M. Govindrajan, S. Maharana, S. Ramdas, “Test Case Point Analysis”, Cognizant Technology Solutions, White Paper, 2001
• [3] QASymphony: www.qasymphony.com