A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the...

105
School of Innovation, Design and Engineering A Model for Estimating the Execution Cost of Test Cases Author: Morteza Hosseini May-2015 Västerås-Sweden Supervisor: Dr. Mehrdad Saadatmand Examiner: Prof. Kristina Lundqvist

Transcript of A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the...

Page 1: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

School of Innovation, Design and Engineering

A Model for Estimating the Execution Cost of Test Cases

Author: Morteza Hosseini

May-2015

Västerås-Sweden

Supervisor: Dr. Mehrdad Saadatmand

Examiner: Prof. Kristina Lundqvist

Page 2: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 I

Abstract

Indubitably, testing plays a pivotal role in the software-developing life cycle through validating and verifying the quality of the software. In the testing process, however, estimating the test cost is an important aspect that needs to be taking into consideration. Due to a large volume of test cases and resource restrictions (budget, people, time-to-market, etc.), care must be taken in the process of selecting the most appropriate test case. Then, the costs for each individual test case must be determined based upon the standards and limitations of the organization to which it belongs. There are two main phases in estimating the test effort; namely, test case creation effort and test execution effort. The focus of the present study is, however, on estimating the latter one, i.e., test execution effort. It is commonly agreed that having a good command of test execution effort in early stages of the software development life cycle can help the practitioners of the field execute the most essential test cases. Accordingly, we attempted to estimate the test execution effort by developing a new approach in which we assigned a cost value to each test case based on some criteria that can be varying regarding target domain. The target domain can be any software system. Having done that, we evaluated our approach by applying it to both desktop and web applications.

Keywords: Testing, test effort, test execution, test effort estimation, test execution effort,

software testing.

Page 3: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 II

Acknowledgment

This thesis work has been conducted with the great help and support of thesis supervisor Mehrdad Saadatmand who provided with his useful guidelines and suggestions throughout this thesis work. Special thanks also go to the examiner, Professor Kristina Lundqvist for her support and contribution in this thesis work. I also wish to offer my sincere gratitude to Mr. Majid Dehghan executive manager and Mr. Kian Nasr project manager at Fara Andish Company, in addition Mr. Saeed Salehi executive manager and Mr. Tavakolnia developer at Partak Company for their faithfully collaboration for doing case studies at their companies.

Finally yet importantly, I would like to give thanks to my parents for their inmost support.

Seyed Morteza Hosseini

Västerås, May 2015

Page 4: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 III

Table of Contents

Abstract .....................................................................................................................................I

Acknowledgment ..................................................................................................................... II

1. Introduction ........................................................................................................................ 1

1.1 Problem Statement .............................................................................................. 2

1.2 Research Goals ................................................................................................... 3

1.3 Contribution ....................................................................................................... 3

2. Research Methodology ........................................................................................................ 4

3. Systematic Review Protocol ................................................................................................ 6

3.1 Question Formulation ......................................................................................... 6

3.2 Question Quality ................................................................................................. 6

3.3 Sources Selection ................................................................................................ 8

3.4 Execution Evaluation .......................................................................................... 8

3.5 Information Extraction ........................................................................................ 9

3.6 Results Summarization ..................................................................................... 10

4. Related Work .................................................................................................................... 11

4.1 Software Cost Estimation .................................................................................. 11

4.2 Prioritizing Test Case Methods ......................................................................... 12

4.2.1 Historical-Based Methods ........................................................................... 12

4.2.1.1 Test Case Prioritization for Black Box Testing ............................... 12

4.2.2 Statement-Based Techniques ....................................................................... 14

4.2.2.1 Total Branch Coverage ................................................................ 14

4.2.2.2 Additional Branch Coverage ........................................................ 14

4.2.2.3 Total Statement Coverage Prioritization ....................................... 14

4.2.2.4 Additional Total Statement Coverage Prioritization ...................... 15

4.2.2.5 Total Fault Exposing Potential (FEP) Prioritization ..................... 15

4.2.2.6 Additional Total Fault Exposing Potential (FEP) Prioritization....... 15

4.2.3 Function Level Techniques .......................................................................... 16

4.2.3.1 Total Function Coverage Prioritization .......................................... 16

4.2.3.2 Additional Function Coverage Prioritization ................................... 16

Page 5: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 IV

4.2.3.3 Total FEP Prioritization ................................................................. 16

4.2.3.4 Additional Total FEP Prioritization ............................................... 16

4.2.3.5 Total Fault Index (FI) Prioritization ............................................... 16

4.2.3.6 Additional Fault-Index (FI) Prioritization ...................................... 17

4.2.3.7 Total FI with FEP Coverage Prioritization ..................................... 19

4.2.3.8 Additional FI with FEP Coverage Prioritization ............................. 19

4.2.3.9 Total DIFF Prioritization ................................................................ 20

4.2.3.10 Additional DIFF Prioritization ...................................................... 20

4.2.3.11 Total DIFF with FEP Prioritization .............................................. 20

4.2.3.12 Additional DIFF with FEP Prioritization ...................................... 20

4.2.4 System Level Techniques ............................................................................ 21

4.2.4.1 Prioritization Using Relevant Slice .................................................. 21

4.2.5 Coverage-Based Techniques ........................................................................ 22

4.2.5.1 Fault Localization Technique .......................................................... 22

4.2.5.2 Fault Aware Test Case Prioritization (FATCP) ............................... 22

4.2.6 Model-Based Techniques ............................................................................ 25

4.2.6.1 Test Prioritization Using System Model ............................................. 25

4.2.6.2 Selective Test Prioritization ............................................................. 26

4.2.6.3 Model Dependence-Based Test Prioritization .................................. 26

4.2.7 Random Ordering ....................................................................................... 27

4.2.8 Optimal Ordering ....................................................................................... 27

4.2.9 Average Percentage of Fault Detected (APFD) ........................................... 27

4.2.10 Average Percentage of Fault Detected Cost Cognizant/Aware (APFDc) .... 29

4.2.11 Case-Based Ranking Test Case Prioritization .............................................. 30

4.2.12 Bayesian Network-based Prioritization ....................................................... 31

4.2.13 Test Case Prioritization Techniques Summary ............................................ 32

4.3 Test Execution Effort Estimation Methods .......................................................... 36

4.3.1 Use Case Points (UCP) ............................................................................. 38

4.3.2 Conventional Methods................................................................................. 39

4.3.2.1 Ad-hoc Method ................................................................................ 39

4.3.2.2 Percentage of Development Time .................................................... 39

4.3.2.3 Function Points Estimation .............................................................. 39

4.3.3 Estimation Model for Test Execution Effort based on Test Specifications ... 40

4.3.4 Early Test Effort Estimation by Using Use Case .......................................... 41

Page 6: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 V

4.3.5 Cognitive Information Complexity Measure ................................................ 42

4.3.6 New, Search, Modify/ Management Information System Functionality ........ 43

4.3.7 Simplified Method based on the NSM/MIS method ..................................... 43

4.3.8 Software Requirement Specification (SRS) .................................................. 43

4.3.9 Accumulated Efficiency Method ................................................................. 43

4.3.10 Case-Based Reasoning (CBR) Method ...................................................... 44

4.3.11 Cuckoo Search Method ............................................................................. 44

5. Our Effort Estimation Model ........................................................................................... 45

5.1 Metrics Descriptions ......................................................................................... 48

6. Case Study ......................................................................................................................... 51

6.1 Case Study 1 .......................................................................................................... 51

6.2 Case Study 2 .......................................................................................................... 74

7. Discussion .......................................................................................................................... 87

8. Summary and Conclusion ................................................................................................. 89

9. References ......................................................................................................................... 90

10. Appendix A........................................................................................................................ 94

Page 7: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 VI

List of Figures

Figure 1. Rela on Matrix R ............................................................................................................. 13 Figure 2. Addi onal Fault-Index Process .......................................................................................... 19 Figure 3. Relevant Slice Technique Example .................................................................................... 21 Figure 4. FATCP Algorithm ............................................................................................................. 23 Figure 5. EFSM Sample Model ......................................................................................................... 25 Figure 6. CBR High Level View ......................................................................................................... 30 Figure 7. BN Model Process ............................................................................................................ 31 Figure 8. The Es ma ng Process of Execution Effort ....................................................................... 40 Figure 9. Early Test Effort Estimation Approach .............................................................................. 42 Figure 10. Our Approach View ......................................................................................................... 46 Figure 11. Conversion Factor Calcula on ........................................................................................ 50 Figure 12. High Level View of Fara Andish ERP System ..................................................................... 52 Figure 13. Sale Module Communication of Fara Andish ERP System ................................................ 55 Figure 14. Modules Involved in TC1 ................................................................................................. 55 Figure 15. In-house Purchasing Module Communication of Fara Andish ERP System ....................... 59 Figure 16. Modules Involved in Test Case 2...................................................................................... 59 Figure 17. Payroll Module Communication of Fara Andish ERP System ............................................ 63 Figure 18. Modules Involved in Test Case 3...................................................................................... 63 Figure 19. Accounting Module Communication of Fara Andish ERP System ..................................... 67 Figure 20. Modules Involved in Test Case 4...................................................................................... 67 Figure 21. Personnel Module Communication of Fara Andish ERP System ....................................... 70 Figure 22. Modules Involved in Test Case 5...................................................................................... 70 Figure 23. High Level View of Partak CRM ........................................................................................ 74 Figure 24. High Level View of Web.Controllers Layer ....................................................................... 75 Figure 25. TC1 Layers Communica ons ............................................................................................ 77 Figure 26. TC2 Layers Communica ons ............................................................................................ 80 Figure 27. TC3 Layers Communica ons ............................................................................................ 83

Page 8: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 VII

List of Tables

Table 1. Keywords and Synonyms ...................................................................................................... 7 Table 2. Escalate Order ................................................................................................................... 13 Table 3. Deescalate Order ............................................................................................................... 13 Table 4. Total FI with FEP ................................................................................................................ 19 Table 5. APFD Rate of Fault ............................................................................................................. 28 Table 6. Test Case Priori za on Techniques .................................................................................... 34 Table 7. Statement Level & Func on Level Priori za on Techniques ............................................... 35 Table 8.Outset and Result Techniques ............................................................................................ 35 Table 9. UUCW ............................................................................................................................... 38 Table 10. Technical and Environmental Factors ............................................................................... 38 Table 11. Importance Degree Guideline ........................................................................................... 47 Table 12. Modules Ranking .............................................................................................................. 53 Table 13. Registra on Sale Dra Test Case ...................................................................................... 54 Table 14. Registra on Sale Dra Test Record................................................................................... 54 Table 15. Test Case 1 Sale Module Dependencies ............................................................................ 56 Table 16. Test Case 1 Storehouse Module Dependencies ................................................................. 56 Table 17. Test Case 1 Importance Degree ........................................................................................ 57 Table 18. Purchase Invoice Issuance Test Case ................................................................................. 58 Table 19. Purchase Invoice Issuance Test Record ............................................................................. 58 Table 20. Test Case 2 In-house Purchasing Module Dependencies ................................................... 60 Table 21. Test Case 2 Storehouse Module Dependencies ................................................................. 60 Table 22. Test Case 2 Importance Degree ........................................................................................ 61 Table 23. Salary Calculation Test Case ............................................................................................. 62 Table 24. Salary Calculation Test Record .......................................................................................... 62 Table 25. Test Case 3 Payroll Module Dependencies ........................................................................ 64 Table 26. Test Case 3 Accoun ng Module Dependencies ................................................................. 64 Table 27. Test Case 3 Importance Degree ........................................................................................ 65 Table 28. Accounting Code Registration Test Case ........................................................................... 66

Page 9: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 VIII

Table 29. Accoun ng Code Registra on Test Record ....................................................................... 66 Table 30. Accounting Module Dependencies ................................................................................... 68 Table 31. Test Case 4 Importance Degree ........................................................................................ 68 Table 32. Personnel Registration Test Case ...................................................................................... 69 Table 33. Registration of Personnel Test Record .............................................................................. 69 Table 34. Personnel Module Dependencies ..................................................................................... 71 Table 35. Test Case 5 Importance Degree ........................................................................................ 71 Table 36. Case Study Results............................................................................................................ 72 Table 37. Layers Ranking ................................................................................................................. 75 Table 38. Inserting New News Test Case .......................................................................................... 76 Table 39. Inserting New News Test Record ...................................................................................... 76 Table 40. Test Case 1 Layers Dependencies ..................................................................................... 78 Table 41. Test Case 1 Importance Degree ........................................................................................ 78 Table 42. Inserting New Image in Image Gallery Test Case ............................................................... 79 Table 43. Inserting New Image in Image Gallery Test Record ........................................................... 79 Table 44. Test Case 2 Layers Dependencies ..................................................................................... 80 Table 45. Test Case 2 Importance Degree ........................................................................................ 81 Table 46. Creating New Training Class Test Case .............................................................................. 82 Table 47. Creating New Training Class Test Record .......................................................................... 82 Table 48. Test Case 3 Layers Dependencies ..................................................................................... 83 Table 49. Test Case 3 Importance Degree ........................................................................................ 84 Table 50. Case Study 2 Results ......................................................................................................... 85 Table 51. Actual & Estimated Effort Difference ................................................................................ 87 Table 52. Acronyms ......................................................................................................................... 93

Page 10: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 IX

List of Charts

Chart 1. Number of Papers in each Source ......................................................................................... 4 Chart 2. Number of Papers in each Group .......................................................................................... 5 Chart 3. Papers Selection Procedure .................................................................................................. 6 Chart 4. Comparison of Total Actual Execu on Effort & Total Es mated Execu on Effort ................ 73 Chart 5. Actual & Estimated Execution Effort ................................................................................... 73 Chart 6. Comparison of Total Actual Execution Effort & Total Estimated Execution Effort ................ 85 Chart 7. Actual & Estimated Execution Effort ................................................................................... 86

Page 11: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 X

List of Equations

Equa on 1. FEP Award Value .......................................................................................................... 15 Equa on 2. Ordering Score ............................................................................................................. 23 Equa on 3. APFD Rate of Fault ....................................................................................................... 27 Equa on 4. APFD Formula ............................................................................................................... 28 Equa on 5. APFDC Formula ............................................................................................................. 29 Equa on 6. UUCP ........................................................................................................................... 38 Equa on 7. TEF Formula .................................................................................................................. 39 Equa on 8. AUCP ............................................................................................................................ 39 Equation 9. Es ma on Rela on ...................................................................................................... 41 Equation 10. Test Execu on Complexity ......................................................................................... 42 Equation 11. Accumulated Efficiency .............................................................................................. 44 Equa on 12. Conversion Factor ...................................................................................................... 72 Equa on 13. Es ma on Execu on Effort Formula ........................................................................... 72 Equa on 14. Case Study 2 Conversion Factor .................................................................................. 84

Page 12: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 1

1. Introduction

In modern software industry, having a high quality and trustable software is considered a key factor in developing a given company. The company needs its customers to have full confidence in the software products so, building this confidence is a big deal, but even bigger is maintaining it. One of the procedures on which the companies can rely to sort out this problem and produce products that are more reliable, is testing. In fact, the more accurate and earlier testing is done, the higher the quality of the software would be. In order to achieve such a high level of confidence and quality and to reduce the costs of the software, testing should be administered in early stage of the software development life cycle. As testing is a costly method in assuring the quality of software [34] and considering that testing can take 35% of total effort of software development [35], we should select the most appropriate ones among the huge number of possible test cases.

Selecting an appropriate test case is, however, a challenging task as executing all test cases is impossible, time consuming, and costly. There are some methods by which test cases can be prioritized. Some of these methods have utilized average percentage of fault detection metric in order to prioritize test cases [1] [6] [5]. Some others can be classified in the system level category meaning that they use a high level architecture of the system to prioritize test cases [8] [3], but prior to prioritization of test cases we need to know the cost of each single test case including cost of test case creation and cost of test case execution. There with test execution effort, the effort on developing tests, creating test cases, getting prepared for test execution, and evaluating results should also be taken into consideration for better estimation [31]. However, the focus of this study would be on the estimation of test execution effort.

The estimation of the test execution effort is a difficult task [30]. Several categories have been developed to estimate the cost of test execution effort, such as system level category [12] [19] [14] that belongs to this group, the second category is model-based that [28] can be considered in this category, and the third category is the group which utilizes requirements to estimate the execution effort, [14] [18] belongs to this group. In this study, we do a survey on these methods and also provide a new approach to estimate test effort execution by assigning a cost value to each test case based on some criteria that my vary regarding the target domain. The approach is evaluated through applying it to a desktop application and a web application. The rest of the study is organized as follow: Section 1 presents problem statement, our contribution in the mentioned problem, and research question. Section 2 describes research methodology, a new method that we employed in this research study. In Section 3, the systematic review protocol is described. Section 4 presents related works including the software cost estimation methods, the methods for prioritizing test case, and the methods for estimating test execution effort. Section 5 describes our new model designed for estimating test case execution effort. In Section 6, two case studies are presented to validate our model. Section 7 presents the discussion about the achieved results for case study 1 and case study 2. Finally, in Section 8 summaries and conclusions are presented.

Page 13: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 2

1.1 Problem Statement

Time and budget predictions are the most difficult tasks in software development process. Among the development phases, testing is the time consuming and costly process [15] due to the fact that it should be applied to all stages of the software development process. Static testing and dynamic testing, two different types of testing [15], can be considered as the main solutions in testing the system in order to build a high level of confidence for the system under test (SUT). In this regard, dynamic testing is believed to be more expensive compared to static testing [15] in that it requires SUT to be run. As such, one needs to predict the requirements of testing such as test preparation and test execution prior to the commencement of the testing phase in order to decrease the cost of testing. Test preparation incorporates such stages as test plan, test case creation, and so forth, which are not considered in this thesis work. Test execution, on the other hand, is a costly activity. The advantage of being informed about test execution effort is that one would be able to prioritize test cases with respect to the project limitation. To classify and order the test cases based on resources (budget, time, tools, and people), one needs to be familiar with the test cases effort. Actually, the problem is that we want to be able to estimate the execution costs of test cases so that we can prioritize and select them based on the amount of resources (time and/or money) that we have for testing. The root of the problem in another word is that we cannot execute all possible test cases. Estimating test execution effort is important in order to know the execution cost for reducing the cost of test process. There is not much investigation conducted on the estimation of test execution effort. In most existing methods that have been developed in code level, the estimation procedure could not have been accomplished in the early stages of the development process because it required the code. In this regard, few methods have been developed which have made use of architecture of the SUT such as utilizing use case and functional and non-functional requirements.

Page 14: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 3

1.2 Research Goals

Considering the aforementioned facts, the present study sets out to investigate the following research goals:

1. A survey on methods for prioritizing test cases. 2. A new approach for test case execution effort estimation.

Test case prioritization area and test case execution effort estimation more specifically, are the main areas on which the research questions evolve. However, the second question is considered as the main research question to be addressed in this thesis work. It intends to seek methods for assigning cost/effort value for execution of test cases. As such, it prioritizes test cases based on the efforts required to execute them.

1.3 Contribution

In this study we have done a survey of literature on software cost execution, methods for prioritizing test cases, and methods for estimating the execution cost of test cases. The test case prioritization techniques that we have used in present survey determine the order of test case execution and do not estimate the execution effort of test cases. As these methods work in line of code level, does not have any estimation of test execution effort in early stage of software development life cycle. Hence, from the existing methods we concluded that estimating test execution effort later on software development or even after execution of test cases could not be useful rather than estimating the execution cost of test cases prior to test case execution. The main consequence of our survey bodes that knowing the execution cost of test cases in early stages of software development life cycle assists us in order to decide which test case should execute first with consideration of our limitation and priorities. We also utilized directly some criteria such as finish-to-finish and finish-to-start parameters from previous works or used indirectly some other criteria as a pattern in order to achieve new parameters. We also provide a new approach to estimate test effort execution by assigning a cost value to each test case based on some criteria that may vary regarding the target domain. In order to estimate the test effort in a way that none of the drawbacks of existing methods (the drawbacks of existing methods will be mentioned in the Related Work Section) is experienced, we have developed a new approach through which the effort in early stage of the SUT would be estimated. To this end, our approach incorporates the entire system rather than Line of Code (LOC) or use cases. Moreover, the model is based on the modules/components subsumed within each test case. As such, it uses the architecture of the system and operates in a high level. In fact, our model tactfully estimates the effort in the early stage of the system life cycle. Ultimately, the purpose of this study and the test case execution estimation is to prioritize test cases based on their costs. The operationalization of this model is arranged in the following steps:

1. Extracting the modules, those are involved in each test case. 2. Assigning values to each module based on the criteria that are listed in Table 11 (these

criteria are flexible and can be updated based on the target domain). 3. Calculating a degree for executed test case based on the values assigned to the involved

modules. In the worst case, it would be needed to execute a test case actually, while the projects are not similar or there is no historical data available for the test case.

4. Computing the test case effort.

Page 15: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 4

2. Research Methodology

The first phase in conducting the study was to consult some systematic review guide papers [36] [37] in order to find the proper methodology and protocol for our research. Focusing on the guidelines suggested in these papers [38] [39] [40], we designed our protocol, which is completely elaborated in the Systematic Review Protocol Section. The main research question (the second one) is formulated based on the designed protocol. The papers were selected from the following sources based on the reviewing protocol with search terms and combination of them with AND, OR logical operation:

ACM digital library IEEE Springer Link Google Scholar

The test execution effort estimation and test case prioritization were found in a total of 176 papers. The contribution of each source is shown in Chart 1.

Chart 1. Number of Papers in each Source

In addition, four more thesis projects were added to the selected sources. Two groups of studies,

i.e., test case prioritization methods group and test case execution effort estimation methods group, were considered in a way so as to manage the final sources.

113 ; 64%

38; 22%

19; 11%

6; 3%

Number of papers found in each source

IEEE

Springer Link

Google Scholar

ACM Digital Library

Page 16: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 5

The first step was to read the titles of all appropriate papers. In cases in which the titles were clear

enough to recognize, they could be transferred to the relevant group of study. Alternatively, in the case where the title was not applicable, the study area was excluded from the sources. Reading the abstract of the selected papers was considered as the next step. To put it another way, the clarity, and applicability of the abstract was determined as the criterion for its transferring to the relevant group of study. Similarly, this selection process was employed for the Introduction and Conclusion Sections of the papers. Finally, 148 papers were opted as the target sources, out of which 36 papers were ultimately chosen. The number of papers related to the test case prioritization methods and test execution effort estimation methods is illustrated in Chart 2 and the selection procedure for papers is shown in Chart 3.

Chart 2. Number of Papers in each Group

24

124

28

0

20

40

60

80

100

120

140

Estimation group Prioritization group Excluded

Number of Papers in each Group

Page 17: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 6

Chart 3. Papers Selection Procedure

3. Systematic Review Protocol

In this section, we will describe the systematic review protocol, which deals with performing a plan to achieve review goal. Actually, the research question, research keywords and their synonyms, as well as their abbreviation and definition, are included in the review protocol. We would also describe the source selection procedure, search strings, and source selection criteria.

3.1 Question Formulation

Question Focus: To identify test case prioritization techniques and assign cost/effort value to test case execution.

3.2 Question Quality

Problem: Software testing needs test case execution. It is necessary to verify, prior to the test execution, to see if the selected set of test cases for execution is suitable. The purpose of such verification is, therefore, to prevent improper selection of the test cases for execution, which can affect the quality of the final product.

Question: What cost/effort value would be determined for the test cases?

176

36 3 30

20

40

60

80

100

120

140

160

180

200

Reading Title Reading Abstract Reading Introduction Reading Conclusion

Num

ber o

f pap

ers

Papers Selection Procedure

Page 18: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 7

Keywords and Synonyms

Search Term Synonym Abbreviation Definition

Test Case TC A case should try something in order to find out about it.

Selection Choose, Pick The act of choosing or selecting.

Prioritization Rate , Rank, Range, Order, Grade, Place,

Estimate, Set

Ordering test cases to catch the best execution arrange.

Software SW A computer written program including documentation of a computer system.

Cost Effort, Attempt

Value must be assign to test case by which will be decided to run test case in which order.

Estimation

Approximation, Measure, Quantify,

Determination, Evaluation

An approximate calculation of quantity or degree or worth.

Assign Allocate, Portion, Give, Specify, Put, Apply, Set

Methods Technique, Strategy,

Plan, Planning, Scenario, Schema

Execution Running Time Schedule

Static analysis Study, Examine Model base analysis

Non-functional analysis Testing Evaluate, Examine

Test script Table 1. Keywords and Synonyms

Page 19: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 8

3.3 Sources Selection

The Criteria Considered in Selecting Sources: Availability of articles on the web. Presence of search mechanisms using keywords. Availability of papers in the electronic form.

Studies Language: English.

Sources Search Method: The applied method is searching through web search engines.

3.4 Execution Evaluation

In order to execute search strings in web search engines, due to different criteria employed in each individual web engine, we had to adopt our search strings with the limitations existing in each web search engine because they did not operate equally with logical operators. Search Strings

(test case OR test script) AND (selection OR choose OR pick)

(test case OR test script) AND (selection OR choose OR pick) AND (methods OR techniques OR strategy OR plan OR planning OR scenario OR schema)

(test case OR test script) AND (prioritize OR prioritization OR rating OR ranking OR ranging OR order OR grading OR place OR setting)

(test case OR test script) AND (effort OR cost OR attempt)

(prioritize OR prioritization OR rating OR ranking OR ranging OR order OR grading OR place OR setting) AND (methods OR techniques OR strategy OR plan OR planning OR scenario OR schema)

(effort OR cost OR attempt) AND (assign OR allocate OR portion OR give OR specify OR put OR apply OR set)

(effort OR cost OR attempt) AND (estimation OR approximation OR measure OR quantify OR determination OR evaluation)

(effort OR cost OR attempt) AND (estimation OR approximation OR measure OR quantify OR determination OR evaluation) AND (methods OR techniques OR strategy OR plan OR planning OR scenario OR schema)

(effort OR cost OR attempt) AND (assign OR allocate OR portion OR give OR specify OR put OR apply OR set) AND (methods OR techniques OR strategy OR plan OR planning OR scenario OR schema)

(test case execution) AND (prioritize OR prioritization OR rating OR ranking OR ranging OR order OR grading OR place OR setting)

Page 20: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 9

Sources List

The following sources have been considered based on their credibility. ACM Digital Library Scholar.google.com IEEE Springer Link

Sources Selection after Evaluation

As the listed sources are among the most popular sources, all of which can be claimed to enjoy high quality criteria. References Checking: All the references selected for this study are from reliable sources.

Studies Definition Studies Inclusion and Exclusion Criteria Definition

The sources must be in English. Present techniques of test case prioritization and test execution effort estimation processes

prior to their execution.

Studies Type: All kinds of studies selected for the purpose of investigation are in line with the

research topic.

Procedures for Studies Selection

First, the search strings were administered in the selected sources. To select an initial set of studies, the titles of all selected studies were read and if a title was in line with the scope of the study, the study could be opted for investigation. In the next stage, the abstracts of all selected studies were read. In cases where an abstract was related to the research field, the Introduction and Conclusion Sections were then read. To refine this initial set of studies, their full texts were also read.

3.5 Information Extraction

Information Inclusion and Exclusion Criteria Definition: The extracted information from the

corpus data contains techniques, methods, and strategies for describing and evaluating the test case

prioritization and test execution effort estimation processes.

Page 21: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 10

3.6 Results Summarization

The results summarization Section will describe the total overview of the study results including

the procedure of presenting the results, comments about number of studies, search selection bias,

result application and finally recommendations (if any).

Results Presentation in Tables and Charts: Relevant tables and charts have been used in order to

reveal the results.

Final Comments

Number of Studies: initially, a total of 176 studies were taken into consideration, out of which 36 were selected. Search, Selection, and Extraction Bias: None of them has been defined. Publication Bias: has not been defined. Inter-Reviewers Variation: There was no variation. Results Application: The results of this study suggest that assigning cost/effort values to test cases and verifying test case execution processes must be determined prior to their execution. Recommendations: No one.

Page 22: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 11

4. Related Work Many methods and researches have been conducted on the software testing area during past

several years, some of which have been done on the test case prioritization, e.g., these methods have utilized average percentage of fault detection metric in order to prioritize test cases [1] [6] [5] and other ones can be classified in system level category[8] [3]. The others have been carried out on the test execution effort estimation among them [12] [19] [14] are in system level category (the system level methods are the ones that can be used in early stages of SDLC to provide high-level estimates of the execution cost of test cases [19]). System level and function level categories have been defined in [4], [28] is in model-based category, and [14] [18] utilize requirements to estimate the execution effort. In continuance, we have described a concise explanation of estimation methods that have generally been applied in the software settings. Following that, test case prioritization techniques and methods of estimation test case execution effort have been presented.

4.1 Software Cost Estimation

Oziegbe [29] believes that the methods utilized in the software cost estimation process are classified into two groups, i.e., algorithm and non-algorithm methods. Oziegbe in [29] further explains that algorithm methods are the ones, which use mathematic while non-algorithm methods are those, which make use of human judgments. According to Oziegbe [29], the following methods are considered as non-algorithm methods:

1. Analogy costing 2. Expert judgment 3. Parkinson’s law 4. Price-to-win 5. Bottom-up (Engineering-build) 6. Top-down

In addition, Jayakumar and Abran have classified the techniques [41] based on the:

1. Judgment and rules of thumb 2. Analogy and work breakdown 3. Factors and weights 4. Size 5. Fuzzy and other models

The process of comparing current projects to the previous ones proceeds in analogy costing

method. One of the drawbacks for analogy costing is that a similar project should be available and the previous project and the new one should be in the same domain in order for them to be compared and to estimate their efforts. However, in such cases, the estimation process is easy to be accomplished as we can use historical data of the old project. Oziegbe [29] does not recommend the Parkison’s law as its estimation is not realistic.

Page 23: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 12

4.2 Prioritizing Test Case Methods

In this Section, we explain 31 methods employed on the test case prioritization process including the code base (code coverage) methods, fault detection methods, system-level methods, and function-level methods. The code-base methods are the ones, which use the code or require the code in order to prioritize test cases, the fault detection methods are the methods that use average percentage of faults that are covered by each single test case, and function level methods use functions, which are covered by the test cases. The important point that should be considered before going forward is that two different concepts of test case reduction and test case prioritization should not confuse us. Evidently, test case reduction is a method through which we can reduce the number of test cases, while in test case prioritization we do not omit any test cases; but rather, we just pick them up in the desired order.

4.2.1 Historical-Based Methods

In historical-based methods group we present the methods, which use the information of the similar or previous projects in order to prioritize test cases.

4.2.1.1 Test Case Prioritization for Black Box Testing

Test case prioritization for black box testing [10] uses the historical information and executed information. Actually, it classifies the test cases into different categories based on the kind of fault they reveal and, consequently, adjust their priority upon their given categories. Grouping the test cases is based on the relation matrix R, which predicts the fault detection relation between the test cases. The algorithm for producing matrix R is illustrated in [10] to which you can refer for more details. When a test case reveals a fault, the test cases, which are in the same group (i.e., related test cases), will be given higher priority. The prioritization process is illustrated in the following sample (example is given from [10]):

Adjusting range α=2

Original order=t2, t3, t4, t5, t6

Page 24: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 13

t1, t2, t3 are in one group and t2, t4 in the other group

Figure 1. Rela on Matrix R [10]

Escalate algorithm: (detail about escalate algorithm can be found in [10])

When test case tm is tested, the next test case tm+1 will be given the highest priority; hence no need to escalate its priority. In our example, since the test case t1 is executed, the next test case, which is t2 does not need to escalate (t1 and t2 ignored). As t1, t3, and t6 are in the same group (the solid lines in relation matrix), t3 and t6 would be adjusted. t3 adjusted to highest priority, t4 and t5 are not in this group, so they do not need to be adjusted. The t6 escalates to higher priority (as adjusting range value is 2).

Step Order 1 t3 t2 t4 t5 t6 2 t3 t2 t6 t4 t5

Table 2. Escalate Order [10]

Deescalate algorithm: (detail about deescalate algorithm can be found in [10])

Deescalate algorithm involves all test cases in the reverse order. As the test case t6 has the lower priority, it is ignored. t5, t4, t2 do not need to be deescalated as they are not in t1’s group. t3 is adjusted to lower priority (two positions lower as the adjusting rang (α) is 2).

Step Order 1 t2 t3 t4 t5 t6 2 t2 t4 t5 t3 t6

Table 3. Deescalate Order [10]

Page 25: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 14

4.2.2 Statement-Based Techniques

In statement-based methods group, on the other hand, we present the methods, which use number of statements covered by each single test case that will be considered in order to prioritize test cases. Six methods will be presented in this group.

4.2.2.1 Total Branch Coverage

Total branch coverage [2] [7] uses the information of unmodified version of program (i.e., the version of program before execution) and does not need prior knowledge of faults, which is considered an advantage of this technique. It also uses the information of outset. Total branch coverage technique classifies the test cases in decreasing order in terms of the number of decisions (branches) each test case covers. A disadvantage of this technique, however, is that some branches may not be dealt with by any test case after executing the test cases. The amount of time this technique needs to run n branches is O (n log n) [2].

4.2.2.2 Additional Branch Coverage

Additional branch coverage [2] [7] recalculates coverage information for each test case that is not prioritized. This technique classifies test cases in the decreasing order in terms of the number of additional decisions (branches) each test case covers, which might have not been covered already. It deals with the process of choosing the test case that covers the greatest branch and set the coverage information. This process is reiterated several times until all branches are covered by at least one test case. This latter merit of the technique, i.e., covering all branches by at least one test case, is actually, what makes the difference between additional branch coverage and total branch coverage. The cost for running this technique for n branches is O (푛 ). [2]

4.2.2.3 Total Statement Coverage Prioritization

Total statement coverage [2] [4] [7] does not consider fault probability to cause a failure in relevant branch/statement for the related test case. In the same vein as total branch coverage technique, this technique also uses the information of unmodified version of program as well as the information of outset. That is why it does not need prior knowledge of faults, which is considered as one of its advantages. As mentioned above, total statement coverage technique is somehow similar to total branch coverage, but the only difference between two techniques is that the former covers statements rather than branches/decisions. The time duration required to execute this technique is O (mn+m log m) where m is the number of test cases and n is the number of statements. mn signifies the time needed in counting the statements covered by each test case, and m log m signifies the time required for classifying the test cases [7]. Assumption is that n is greater than m, so the time is O (mn).

Page 26: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 15

4.2.2.4 Additional Total Statement Coverage Prioritization

Additional total statement coverage [2] [4] [7] uses the feedback in order to focus on the statements not covered yet. Additional statement coverage is the same as additional branch coverage in that it covers statements rather than branches/decisions. A disadvantage of both branch and statement coverage techniques is that there is no guaranty to expose a fault by executing a test case and cover faulty statement because these techniques only take into account the coverage of a branch or statement by a test case. Consideration of fault ability/probability to cause a failure in relevant branch/statement for related test case is an important issue, which is ignored. The cost of additional statement coverage is O (푚 n) where m stands for the number of test cases and n for the number of statements.

4.2.2.5 Total Fault Exposing Potential (FEP) Prioritization

The fact that a fault can be exposed by a test case depends not only on the test case to be able to execute the relevant component but also on the ability of the fault to cause a failure in relevant component regarding that test case. This fault’s ability to expose a failure is named fault exposing potential (FEP), which should be determined in approximation. Total fault exposing potential [2] [4] [7] is an expensive method to be administered. The cost of measuring ms(s;t), which is mutation score, for each statement is high. It uses the information of the program before execution (unmodified version of program). It also makes use of the outset information. Actually, it is more expensive compared to the code coverage techniques. Total fault exposing potential is concerned about exposing a fault by a test case, which is based upon both executing test case of the relevant statement and the probability of fault to cause a failure in that statement. Total fault exposing potential technique orders the test cases based on the following calculations [2]: for given program P, statement s € P, test suite T, test case t € T, and ms(s; t) which is mutation score for each statement, s as the ratio of mutants of s exposed by test case t [2].

푎푤푎푟푑푣푎푙푢푒 = (푚푠(푠; 푡))

Equation 1. FEP Award Value [7]

A disadvantage of FEP is that the cost of calculating ms(s;t) is high. Total fault exposing potential prioritization classifies the test cases in order of these award values. The cost of FEP for m test cases and n statements is O(mn+mlogm) in time. In cases where n is greater than m, the cost is O(mn).

4.2.2.6 Additional Total Fault Exposing Potential (FEP) Prioritization

One of the prerequisites of this technique is the prior knowledge of the faults. It uses the feedback. Expanding total fault exposing potential technique leads to additional total fault exposing potential

Page 27: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 16

[2] [4] [7] technique. The main difference between these two techniques is the assigning of the lower award values for all test cases t that cover statements.

4.2.3 Function Level Techniques

In function-level methods group, we present the methods that use the number of functions, which are covered by each single test case considered in order to prioritize test cases. We will present 12 methods in this group.

4.2.3.1 Total Function Coverage Prioritization

Total function coverage [4] is less expensive compared to the total statement coverage technique. Total function coverage prioritization orders the test cases based on the number of functions executed by each test case. The cost of this technique is O(mn+m log m) where m is number of test cases and n is number of functions. Typically n is smaller than m.

4.2.3.2 Additional Function Coverage Prioritization

Additional function coverage prioritization [4], classifies the test cases with respect to the total number of additional functions each test case covers. When covering all the statements, reset the coverage vectors and applies additional function coverage on remaining test cases once more. The cost of this technique for m test cases and n functions is푂(푚 푛).

4.2.3.3 Total FEP Prioritization

Total FEP Prioritization [2] [4] [7] technique is similar to FEP in the statement level with replacing statements by functions.

4.2.3.4 Additional Total FEP Prioritization

This technique acts as additional total FEP [2] [4] [7] statement level but with an operation in the function level.

4.2.3.5 Total Fault Index (FI) Prioritization

Fault index is one of the techniques, which Elbaum et al [4] have mentioned in the function level techniques. Total fault index [4] uses the information of p (version of the program before execution/unmodified version of program), and the information of p′ (version of the program after execution/modified version of program). In total fault index prioritization technique, it has been supposed that each function has a potential talent to have fault in the modified version of that function

Page 28: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 17

called fault proneness. Bearing in mind that P is considered as a program and P′ as a modified version of P, the following steps are required to generate fault indexes for P′:

1- Production of fault index for each function in P. 2- Production of fault index for each function in P′. 3- Comparison function by function indexes for P′ against P.

The above steps are consecutively performed to generate the fault index for each function in P′ based on changes brought about by that function. Total fault indexes for each test case would be calculated based on the aggregate number of the fault indexes for that test case across the functions, which have been covered by the target test case. Test cases are classified in descending manner with respect to their total fault index. There is no need to execute test cases, so the cost is lower than that of FEP. The time for executing m test cases and n functions is O(mn).

4.2.3.6 Additional Fault-Index (FI) Prioritization

Additional fault index prioritization [4] technique succeeds the following steps:

1- Put all the functions, which have been covered already by test cases in a set of functions. This set of functions is called SOF (if all the functions contain in the SOF, the SOF reset to 0).

2- In order to find the next best test case, we calculate sum of fault indexes for each test case for all functions, which the related test case has covered, except the ones in SOF.

3- The winner test case, which has the next higher priority to execute, is the one with greater fault index value at step 2.

4- This process continues until all test cases are prioritized.

Page 29: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 18

The above process has been depicted in the Figure 1:

SOF

Set of functions that have been covered by previously executed test cases

Do all functions

exist in SOF?

YES

SOF=0

NO

Calculation the sum of fault indexes for each TC for all functions, which that TC has covered, except the ones in SOF.

Print the TC with greater sum FI

Do all TC have been prioritized?

YES

Finish NO

Page 30: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 19

Figure 2. Additional Fault-Index Process

4.2.3.7 Total FI with FEP Coverage Prioritization

This technique is analogous to total FEP coverage prioritization technique, with the only difference that is on the estimation of FI for each function. Actually, in total FI with FEP coverage technique [4] we have to estimate FI in addition to estimation of FEP for each function, which is executed by a test case. The next step is the calculation of sum of these values (FI+FEP) for each test case toward functions executed by that test case. The test case with greater summation is the next one in descending order of test case prioritization.

TC1 TC2 TC3

F1 FI 1 3 5 FEP 2 4 2

F2 FI 2 1 8 FEP 4 1 3

Total 5 9 18 Table 4. Total FI with FEP [4]

As illustrated in Table 3, the total estimation of FI and FEP to TC3 is greater than other ones, so TC3 is the next one in prioritization.

4.2.3.8 Additional FI with FEP Coverage Prioritization

The process using in additional FI with FEP coverage prioritization [4] technique is similar to that of additional FI coverage prioritization. The following steps are administered to classify the test case in ascending order:

1. Estimate the FI and FEP values for each function. 2. Compute sum of FI and FEP for each test case that is not prioritized yet across the functions

executed by that test case. 3. Select the test case with the greatest total value. 4. Set the value of FI and FEP to zero for the functions, which are covered by the selected test

case. (This step is equal to step 1 in additional FI coverage prioritization that put all the functions which have been covered already by test cases in a set of functions)

5. Repeat step 2 to 4 until all the functions values are zero. 6. If any test case remains, reiterate the step 1 to 5.

Page 31: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 20

4.2.3.9 Total DIFF Prioritization

Total DIFF technique [4] uses the information of unmodified version of program that is the version of program before execution (p) and p′ (modified version of program). Given that p′ is the modified version of p, total DIFF prioritization technique relies on the difference between p and p′. The change rate obtained is based on the number of lines which are listed by UNIX diff command as inserted, deleted, and changed lines both in p and p′.

4.2.3.10 Additional DIFF Prioritization

Additional DIFF [4] technique is similar to additional FI prioritization but is applied to diff output.

4.2.3.11 Total DIFF with FEP Prioritization

Total diff [4] is similar to total FI with FEP prioritization, but is applied to diff command output.

4.2.3.12 Additional DIFF with FEP Prioritization

Additional DIFF with FEP prioritization [4] is quite like additional FI with FEP prioritization, except for its application to diff command output.

Page 32: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 21

4.2.4 System Level Techniques

4.2.4.1 Prioritization Using Relevant Slice

Prioritization using relevant slice [3] is a system level technique that uses the information of p and p′, unmodified and modified version of the program, respectively. Modified version of the program is the version of program before execution and unmodified version of program is the version of program after execution. It needs prior knowledge of faults. This technique takes into account the number of statements/ branches that affect or have potential to affect the output of the relevant test case as well as total statement/branch coverage. We will illustrate this technique by using the following example. The darker-shaded lines indicate the relevant slice. This example is given from [3].

1: read(a, b, c); 2: int x=0, y=0, z=0; 3: x := a + 1; 4: y := b + 1; 5: z := c + 1; 6: int w := 0; B1: if x > 3 then B2: if z > 4 then 7: w := w + 1; endif endif B3: if y > 5 then 8: w := w + 1; endif 9: write(w);

Figure 3. Relevant Slice Technique Example [3]

When the execution of the above code with given input (a,b,c)=(1,5,4) reaches B1, the value of x becomes equal to 2, so B1 evaluates to false and the value of y becomes 6, so B3 evaluate to true later on. Accordingly, the output value of the program would be one. In this case, the line 3 statement changes to x=b+1 by mistake. By this mistake, the value of x becomes 6 when reaching to B1, so it evaluates to true, subsequently B2 evaluates to true and the value of w in line 7 is equal to 1. Continuing executing B3 evaluates to true (the value of y is 6 at this step) there upon the value of w is 2 as input. This example plays up that line 3 included in the relevant slice has the potential to affect

Page 33: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 22

the output. The order of test case in decreasing order for this technique is based on the test case weight, which determines [3], as would follow:

Test case weight = number of requirements in the relevant slice + total number of requirements exercised by the test case.

4.2.5 Coverage-Based Techniques

In coverage-based methods group, we present the methods that use the information covered by each single test case and will be considered in order to prioritize test cases. This group contains two methods.

4.2.5.1 Fault Localization Technique

Fault localization [5] is an automatic debug technique, which software developers use in order to decrease the debugging time. This measure is performed through finding the position of the faults in code. By doing this, the information about the position of fault or faults in code is revealed. The main objective of utilizing the fault localization is that parts of the program being covered by a failed test case are more capable of having the fault/faults than the other parts, which have been executed by a passed test case [5].

4.2.5.2 Fault Aware Test Case Prioritization (FATCP)

Previous test case prioritization techniques such as statement-level and function-level techniques only used coverage information of the test cases in order to classify test cases. Fault aware test case prioritization technique [5] is a technique, which uses both coverage information and historical information of test cases so as for them to be prioritized. To obtain this information, FATCP employs an automatic tool named APT (APFD Parsing Tool), which estimates APFD values and the order of test cases for execution. Historical information of test case, on the other hand, is calculated through fault localization technique (FL). The reason behind using fault localization technique, as mentioned in [5], is to improve rate of fault detection. To do so, FATCP uses a fault localization technique called TARANTULA. (For more information about TARANULA, refers to [5] Section 2.3 and for complete information refer to [7] in [5]). TARANTULA gives a suspiciousness score to each coverage entity, which indicates the probability of containing faults for each coverage entity. The more suspiciousness score, the more probability of existing fault. Subsequently, higher suspiciousness score means higher rate of fault detection for that test case. The FATCP algorithm has been depicted in Figure 3.

Page 34: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 23

Figure 4. FATCP Algorithm [5]

The process of FATCP is arranged in the following orders:

1. Assigning an order score(OS) to each test case, calculating based on equation 5: Given n test cases in a test suit, i position of related test case in prioritized queue.

푂푟푑푒푟푖푛푔푆푐표푟푒(푂푆) =푛 + 1 − 푖

Equation 2. Ordering Score [5]

2. Assigning suspiciousness score (SS) to each test case: Prior to executing test suit, test cases with higher suspiciousness score have higher rate of fault detection. However, after executing the test suit, the faults covered by test cases would be removed and, as a result, the rate of

Page 35: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 24

fault detection of the target test case decreases. It means that the higher the suspiciousness priority is the lower the rate of fault detection would be. As such, the new order of the test case should be adapted. To do so, we must replace the suspiciousness score of test cases with the opposite suspiciousness score to test cases. Sejun Kim et al [5] believe that in comparison with coverage-based techniques, FATCP has higher rate of fault detection. The good point about FATCP is that it can be applied to every coverage-based technique as it uses coverage information.

Page 36: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 25

4.2.6 Model-Based Techniques

In model-based methods group, we present the methods that use information of the SUT model in order to prioritize test cases. We will present three methods in this group.

4.2.6.1 Test Prioritization Using System Model

Model-based prioritization [8] utilizes the state-based model of system. Actually, model-based prioritization makes use of the information of executing system model due to the fact that administering the system model is faster and less expensive than the system itself. As this technique uses state-based model, executing system model means to execute an Extended Finite State Machine (EFSM). It assumes that the model of the system is executable. By doing so, it takes advantage of executing the model of system as well as the modified model of system in order to obtain the information about changes that are made in the system. The state-based model employed by this technique is Extended Finite State Machine (EFSM).

EFSM, an abbreviation used for Extended Finite State Machine, is used to model the state-based systems. The following components are subsumed within the EFSM: state, transition, action, condition, and event. At the time of occurring a condition is validated, if the condition validation is true, then a transition is made and transferred from one state to the other. As a consequence of this process, an action takes place. A sample model is shown in the Figure 4.

Figure 5. EFSM Sample Model [8]

The change, which model-based prioritization considers in modified model of the system, is to gather the information to comprise added and deleted transitions. Any other change that occurs in the system is considered in line with these two kinds of basic changes (adding transition and deleting transition). The process employed in this technique includes the following steps:

1. Identifying differences between two models (primary model and modified model) by recognizing the changes in the modified model of system.

2. Executing modified model for test cases in order to collect information.

Page 37: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 26

3. Prioritizing test cases: different methods used to prioritize test cases based on the information that have been gathered in the previous step. Two kinds of methods that are used in [8] includes:

Selective test prioritization Model dependence-base test prioritization

4.2.6.2 Selective Test Prioritization

The main purpose of Selective [8] method is to assign high priority to the test cases that have covered the modified transition, and to assign low priority to those ones that have not covered any modified transition. Selective method is comprised of two versions: The first version considers only added transitions and, as a result, assigns high priority to test cases, which have given an account of the modified transitions and low priority to the test cases that have not taken into coverage any modified transition. The second version, on the other hand, looks upon both added transitions, deleted transitions and then follows the same scenario for assigning priority to test cases as first version. The precise details of selective method can be found in [8].

4.2.6.3 Model Dependence-Based Test Prioritization

The objective of the model dependence-base [8] method is to use model-dependence analysis for achieving information about ways through which added transition and deleted transition interact with other parts of the system and using this information in the test case prioritization process. Two kinds of model dependence analysis have been utilized in [8].

Page 38: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 27

4.2.7 Random Ordering

The first and simplest method in prioritizing test cases is to order them randomly. Random ordering [2] [4] [7] technique can be administered in the experimental manner.

4.2.8 Optimal Ordering

Optimal ordering technique [2] [4] [7] uses the information of program before execution. As a result, it needs prior knowledge of the faults and test cases. Accordingly, it is not a proper technique in case of limitation to prioritize test case before execution. In order to achieve objective of this method that is an optimal ordering of test cases in a test suite, we need to know which fault is exposed by each test case. Considering these facts, the optimal ordering technique cannot be respected as a practical method.

4.2.9 Average Percentage of Fault Detected (APFD)

APFD [1] technique uses the information of the program before and after execution (unmodified and modified version of program). As such, one of the disadvantages of APFD technique is that running all test cases is a requirement and one needs to know execution time of test cases. This technique also uses the test case information pertaining to execution phase, so it needs prior knowledge about faults and test cases. APFD technique puts test cases in an order using average number of faults, which are detected by each test case during its execution time called rate of fault. Rate of fault for each test case is estimated based on the following formula:

Rate of fault (RF) =F /T

Equation 3. APFD Rate of Fault [1]

Where F is the number of faults detected by each test case, and T is test case execution time in minute. The higher fault detection ratio means the higher test case priority. In Table 2 you can observe ten sample test cases, TC1 to TC10 and faults have been detected by each test case.

Page 39: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 28

TC1 TC2 TC3 TC4 TC5 TC6 TC7 TC8 TC9 TC10 F1 * * F2 * * * * F3 * * * * F4 * F5 * * * F6 * F7 * * F8 * * F9 *

F10 * * Total Fault 2 3 1 3 2 3 2 2 2 3

Time 5 7 11 4 10 12 6 15 8 9 Table 5. APFD Rate of Fault [1]

Test case 1 rate of fault detection=2/5=0.4 Test case 2 rate of fault detection =3/7=0.42 Test case 3 rate of fault detection =1/11=0.09 Test case 4 rate of fault detection =3/4=0.75 Test case 5 rate of fault detection =2/10=0.2 Test case 6 rate of fault detection =3/12=0.25 Test case 7 rate of fault detection =2/6=0.33 Test case 8 rate of fault detection =2/15=0.133 Test case 9 rate of fault detection =2/8=0.25 Test case 10 rate of fault detection =2/9=0.22

According to the rate of fault detection mentioned above, TC4 has the highest priority and TC3 has the lowest priority. Accordingly, the order of test cases based on their priority is as follows: (This example is taken from [1])

T4, T2, T1, T7, T6, T9, T10, T5, T8, T3

The advantage of APFD technique is that it would be able to detect higher average number of the faults. This technique would be applicable for evaluating test case orders. We can evaluate the above order by using the following formula suggested by [1]:

퐴푃퐹퐷 = 1 −푇퐹1+ 푇퐹2+. . . . . . . . +푇퐹푚

푛푚 +12푛

Equation 4. APFD Formula

Where m is the number of faults that is 10, n is the number of test cases which is 10, and TFi is the position of the first test case in the above order that is an indication of the fault i. As an example, consider the above order of test case execution. In this example, the first test case TC4 would not be able to find F1; this is the case for the next test case, viz, TC2; finally, TC1, as the third case in this order, would find F1. As seen in the order mentioned above, the position of TC1 in the order

Page 40: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 29

execution queue is 3, so TF1 is 3. The same case occurs for TF2, based on the above test case order, the first test case, which exposes F1 is TC4; and finally, as TC4 is situated in the first position, TF2 in this case would be 1.

By putting the values in APFD equation, the following weight of APFD would be revealed:

퐴푃퐹퐷 = 1 −3 + 1 + 2 + 4 + 2 + 1 + 1 + 2 + 5 + 3

10 ∗ 10 +120 = 0.81

If we calculate the same values for non-prioritized test case, the APFD value would result as follow:

퐴푃퐹퐷 = 1 −1 + 4 + 2 + 7 + 2 + 4 + 5 + 3 + 6 + 1

10 ∗ 10 +120 = 0.70

The APFD values for prioritized and non-prioritized test cases indicate a higher value for prioritized test cases, compared to non-prioritized ones. As can be seen, based on this evaluation suggested by [1], this technique needs prior knowledge of the faults and execution time of the test cases and, as a consequence, it cannot be a proper technique to use in prioritizing test cases before their execution. Another disadvantage of this technique is that in case of having many test cases with long execution time, and also having time constraint, achieving the result takes a long time as running all test cases is required.

4.2.10 Average Percentage of Fault Detected Cost Cognizant/Aware (APFDc)

In order to prioritize test cases, the previous technique has some assumptions to be taken into consideration, which would follow: all faults have equal cost; all test cases have equal cost. The cost can be interpreted in various ways (time, hardware, engineer’s salary, human time, and etcetera). If these requirements are met, then APFD performs as is expected. However, in practice, not all faults have identical cost and there are some test cases with different costs. In such situations, APFD does not perform as is expected before [6]. APFDc [6] unlike APFD considers cost of the faults and test cases. Cost of a test case can include time of execution, setup, validation, and human time [6]. Similarly, cost of a fault, which is called fault severity, can be comprised of the time to alter it or other costs such as losing time to market, damage to assets, and so on [6]. The average percentage of fault detection with cost consideration is calculated using the equation 3. [6]

Equation 5. APFDC Formula [6]

Where T stands for the test suit, n stands for the number of test cases in test suit T. t1, t2, … , tn are test costs, F is fault suit, m is number of faults in fault suit F. f1, f2, …, fm are fault severity(cost). TFi is the first test case in T′ that has revealed the fault i, T′ is prioritized version of T. If all test costs are equal and all faults costs are similarly equal, then the equation is the same as APFD [6].

Page 41: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 30

4.2.11 Case-Based Ranking Test Case Prioritization

Case-Based Ranking (CBR) [9] is one of the numerable techniques, which need user knowledge in prioritization process. CBR utilizes user knowledge as well as some other prioritization criteria such as complexity, level of coverage, and historical information with association of machine learning algorithm. Due to space limitation, for more information about learning algorithm refer to [9]. A high-level view of CBR is shown in Figure 5.

Figure 6. CBR High Level View [9]

Where the TS is test suit, {t1, …, tn} are unordered test cases, f1, …, fm are prioritization indexes, and (ti, tj) are test case pairs. In case of disability for deciding prioritization merely based on prioritization indexes, CBR takes into consideration user knowledge iteratively throughout learning algorithm. In order to produce prioritized test cases, the learning algorithm can discontinue at any time. Some advantages of CBR are listed below:

Multiple indexes: Unlike previous techniques, CBR uses multiple indexes to prioritize test cases.

User centered: unlike the other methods, CBR uses user knowledge and make user involved in prioritization process.

Partial information: In case of incompetency of information, CBR can perform as powerful as expected before.

Incoherent data: CBR is robust to tackle with adversative information. Wide applicability: CBR can apply in any stage of testing, that is, before execution and

during regression testing.

Page 42: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 31

4.2.12 Bayesian Network-based Prioritization

The process of Bayesian Network (BN) [11] includes the following three steps:

1) Extracting evidences: Gathering information from different sources such as quality metrics, code changes, and test case coverage.

2) Building BN: to be used in order to estimate the probability of the test cases. (The precise explanation of building BN can be found in [11]).

3) Ordering test cases: The ordering of test cases is arranged in the following three stages:

a) Probabilities inference: investigate the succeeding probability of a test case bases on the model.

b) Test case selection: selecting some test cases and adding them to the ordered list. c) Updating BN: the test cases, which added to the ordered list in step (b) utilize to modifying

the model as the probability of test cases which overlap the same areas decrease. The produced model will transfer to the step (a), and this process will continue as far as all the test cases prioritize.

The high-level view of BN-based prioritization is demonstrated in Figure 7.

Figure 7. BN Model Process [11]

Page 43: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 32

4.2.13 Test Case Prioritization Techniques Summary

Throughout this thesis work, we have investigated test case prioritization techniques and have grouped them:

1. Statement level techniques 2. Function level techniques 3. Diff techniques 4. Fault index techniques 5. FEP techniques

Some of these techniques do not need prior knowledge of the faults and test cases before

execution. Some others use the information related to the modified version of the program (the version of program after execution) and get feedback. The gathered feedback will be utilized to improve the rate of fault detection. The other kinds of prioritization techniques such as total techniques need knowledge of faults and test cases, prior to their execution. The other important aspect of prioritization techniques is consideration of practical aspect of such techniques. The techniques mentioned at the start part of this summary are practical ones, whereas FEP techniques are not feasible and need more investigation.

Sejun Kim, et al [5] believe that both statement-level and function-level techniques are subsumed within the coverage-base techniques group using the rate of fault detection to prioritize test cases. Disadvantages of coverage-based techniques can be enlisted as follow:

Because the faults, which were detected by a test case, will be removed after detection, so the test case, which has executed them already, does not have performance as expected. Actually, the rate of fault detection related to relevant test case will decrease.

Coverage information consideration exclusively. Test case rate of fault detection will not change even when the removing faults were

detected by the relevant test case; subsequently the priority of that test case will not change and should be adjusted.

We have listed the test case prioritization techniques in Table 6 including the following information about each technique such as technique name, description, time, etc.

Technique name: technique name presents name of the technique.

Mnemonic: mnemonic illustrates the mnemonic name of the technique that would be easier to be memorized.

Description: The description presents a short description of how the technique performs.

Page 44: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 33

Time: time illustrates how long the technique takes to be executed. Using the information of P/P′: This column in Table 6 describes that the technique uses

either the information of the version of program before execution (p) or the information of the version of program after execution (p′).

Need Prior knowledge of Execution: this column determines if the technique needs the information about test cases prior to execution or not.

Level/Group: level/group presents to which category technique belongs. The “statement”, “function”, and “system” terms mean that the technique belongs to statement level category, function level or system level, respectively.

Use outset info/result info (static or dynamic testing): the outset information means using the information of the beginning of the testing and the result information means using the information of testing result.

Page 45: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 34

Technique Name Mnemonic Description

Time

Using the inform

ation of P/P′

Need Prior know

ledge of Execution

Level/Group

Use outset info/result

info(static or dynamic

testing)

Random Prioritization Random Prioritize test

cases randomly - - - - -

Optimal optimal Order test cases to optimize fault

detection rate - - - -

APFD - - - - - -

Total Branch Coverage branch-total

Prioritize based on branch coverage

O(n log n) - - statement -

Additional branch

coverage branch-addtl - O(푛 ) - - statement -

Total Statement Coverage st-total

Prioritize based on total number

of statement coverage

O(mn+mlogm) - - statement -

Additional Statement Coverage

st-addtl - O(푚 n) - - statement -

Total Fault Exposing Potential

FEP-total - O(mn+mlogm) - - statement -

Additional Total Fault Exposing

Potential FEP-addtl - - - - statement -

Prioritization Using Relevant

Slice - - - - - - -

Total Function Coverage

Prioritization

fn-total - O(mn+mlogm) - - - -

Additional Function Coverage

Prioritization

fn-addtl

M=number of test cases

N=number of functions

O(푚 n) - - - -

Total FEP Prioritization

fep-total - O(mn+mlogm

) - - function -

Additional Total FEP

Prioritization fep-addtl - - - × function -

Fault Aware Test Case

Prioritization (FATCP)

-

Prioritize based on coverage info

and historical info

- p System both

Table 6. Test Case Prioritization Techniques

Page 46: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 35

In Table 7, you can find the group to which a given technique belongs. We refer to techniques with their mnemonic names in Table 7.

Statement Level Function Level System Level Model Based

St-total Fn-total Fault-exposing-potential-addtl System model

St-addtl Fn-addtl Fault-exposing-potential-total

St-fep-total Fn-fep-total Relevant-slice St-fep-addtl Fn-fep-addtl Branch-total fi-total Branch-addtl fi-addtl

Fep fi-fep-total Fep-add fi-fep-addtl FATCP diff-total

diff-addtl diff-fep-total diff-fep-addttl Fun-add Fep-total Fep-add Fi-total

Table 7. Statement Level & Function Level Prioritization Techniques

In Table 8, it is observed that each technique uses either the outset (the time at which a test case is supposed to begin) information of the program or the result information (the information of executed test case).

Mnemonic Using Outset Information

Using Result Information

St-total × Fi-total × Br-total × St-fep-total × Fn-total × Fn-fep-total × Fep-total × Fep × St-addtl × Fep-add × Optimal × APFD × St-fep-addtl × Br-addtl × Br-add × Fn-addtl × Fn-fep-addtl × Fun-add ×

Table 8.Outset and Result Techniques

Page 47: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 36

4.3 Test Execution Effort Estimation Methods

Test effort estimation allows managers, developers, testers, and test managers to assign resources, schedule the project and test activities, performs risk management, and monitor test and project activity. It also enables visibility into the most important parts of the SUT while allowing prioritization of additional test effort in order to achieve high quality software [19][21]. Test effort estimation is increasingly important for the following reasons:

Effort estimation helps to estimate the budget of the project It enables the manager to assign test team members as well as test tools [16]. Without estimating the test effort, managers are not able to do the mentioned activity by

Badri et al and Kushawaha et al in [19] [16] [18] such as allocating test resources, planning and monitoring the test activity, planning to write test cases, drawing out the test result, and determining the critical parts of the software which require more testing effort.

As mentioned before, the methods for estimating the effort can be considered for two parts: test case creation and test case execution. In this thesis work, the focus is on the second part. Test creation effort includes the effort needed for test strategy creation and testing plan [29]. In webpage http://santoqa.blogspot.com, the estimation methods are considered in three groups as the most common methods including percentage, analogy, and expert [35].

Different techniques have been used by different test execution methods. The common techniques are as follow:

Use-case based techniques [13] [14] [17] estimate in the early stage and nullity depending on the code. Using software requirements and different technical and environmental factors are the main features of these kinds of techniques [18].

Code-based techniques [22] [13] [20] that have the following drawbacks: o Code dependency o Inability to predict in early stage of development o Inability to estimate the effort in early stages as we have developed the system

[26], the estimation process should be done before creating test cases [18]. Complexity value-based techniques [18]. Requirement-based techniques Model-based techniques [23] that are based on data-flow, control flow, or UML model

[19].

Badri et al [19] have utilized use cases in order to estimate test effort in suite size aspect. The authors have considered four metrics to describe the complexity of use cases and three metrics to illustrate different aspects of test suites. They have used K-means clustering method as a clustering technique to categorize use cases in three groups of simple, medium, and complex. Badri et al have done ranking of test cases, which are involved in each use case to simple, medium, and complex. Some Badri et al metrics [19] are in code-level such as the number of lines in test code (TLOC). A

Page 48: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 37

tool to estimate test execution effort is developed by E. Aranha et al in [22] based on the size and execution complexity. Actually, they [22] automate the approach that has been illustrated by Aranha et al [13]. One of the other code-based methods that have been utilized by Singh et al [24] uses artificial neural network (ANN) in order to estimate the test effort. The metric that has been used is the number of lines of codes that were added or changed in the class level. E Aranha et al in [27] have proposed another approach that is the same as the one in [13] using the following risk factors:

Team experience Test environment Tested product

The first approach that has been proposed by Aranha et al in [13] is applicable for static test environment. The process of test effort estimation is based on the execution point (“a unit of measure defined for describing the size and execution complexity of test cases” [13]) and conversion factor (“The conversion factor represents the relation between test execution effort and execution points”. “The conversion factor is given in seconds per execution point, indicating the number of seconds required to execute each execution point of a test case” [13].). The second approach they have proposed in [27] is applicable in dynamic test environment and uses a risk factor, which can be team experience, test environment conditions, tested product, etc.

Another type of methods test effort estimation that have been proposed in Certified Tester Foundation Level Syllabus [31] is in two following types:

1. Metric-based which estimates the effort by using metrics of former or similar projects. 1. Expert-based that uses the owner or expert’s estimation. 2. The third method divides the system to the smaller parts and estimates the effort of

divided parts. Then calculates the total effort of the project by summing the effort of smaller parts. This method is called bottom-up-estimate [30].

Dharmender and Kushwaha [18] have mentioned the commonly used measures and their evolution from 1980 to 2010. Kushawa has presented various methods. Based on their research, the measure, which has been employed in 1980, was test date based estimation. In 1990 model based estimation was the commonly used method. The roadmap indicates that the method’s trend goes to the high-level architecture of the SUT, as the common methods in 2009 and 2010 were use case based estimation and requirements based estimation methods, respectively.

Test case effort estimation usually comprises the following factors estimation [12]:

Size of the product. Effort in person-hours/months. Schedule in calendar month. Cost in currency.

Some metrics for test case effort estimation that have been mentioned by Saadatmand [15] include time, person month, and financial cost. The disadvantages of the methods, which are based on the LOC, can be the lack of any standard on how to count the number of lines as well as no possibility to count LOC prior to the development.

Page 49: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 38

4.3.1 Use Case Points (UCP) Use case point method [12] makes use of the use cases in order to estimate test case effort. The

estimation process is performed in the following steps:

1. Obtaining Unadjusted Actor Weights (UAW) based on the number of actors in the system. Actors are grouped as simple type such as end-user, which interacts with the system through GUI, average type like data stores that interacts via protocol, and complex type like other programs, which interacts with the system through API. Actor weight of simple actor is 1, average actor is 2, and complex actor is 3 [12].

2. Obtaining Unadjusted Use Case Weights (UUCW) based on the number of use cases in the system. Similar to actors, use cases are also in three types of simple, average, and complex. UUCW assigns to use cases either based on number of transactions or scenarios as have been shown in Table 9:

Use Case Type Number of Transaction Factor Simple <=3 1

Average 4-7 2 Complex >7 3

Table 9. UUCW [12]

3. Calculating Unadjusted UCP (UUCP) based on the following equation:

UUCP=UAW+UUCW

Equation 6. UUCP [12]

4. Calculating technical and environmental factors: the technical and environmental factors are listed in Table 10:

Factor Description Assigned Value T1 Test tools 5 T2 Documented inputs 5 T3 Development environment 2 T4 Test environment 3 T5 Test-ware reuse 3 T6 Distributed system 4 T7 Performance objectives 2 T8 Security features 4 T9 Complex interfacing 5

Table 10. Technical and Environmental Factors [12]

To calculate Total Environment Factors (TEF), we need for each factor to multiply assigned

weigh to assigned value.

Page 50: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 39

푇퐸퐹 = (푎푠푠푖푛푒푑푣푎푙푢푒 ∗ 푎푠푠푖푔푛푒푑푤푒푖푔ℎ푡)

Equation 7. TEF Formula

5. Adjusted UCP (AUCP) computation:

AUCP=UUCP*[0.65+ (0.01*TEF)]

Equation 8. AUCP [12]

Two values 0.65 and 0.01 are constant in the above formula.

6. Calculating the final effort that is obtained by multiplying AUCP to conversion factor. Conversion factor is person-hours effort.

Estimation in high level is a disadvantage of this method as UCP-based method because it works in testing activities level including test planning, test design, and test reporting as a whole. It might not be able to estimate each single test unit level such as test case or test suit [14].

Some other methods that according to [12] are subsumed within the category of conventional methods, which would follow in the next sections.

4.3.2 Conventional Methods

4.3.2.1 Ad-hoc Method

In this method test effort estimation is not based on an accurate timeframe. The estimation of test cases is accomplished based on either pre-decided timeline by managers or budget constraint. The ad-hoc method [12] is usually utilized by inexperienced organizations.

4.3.2.2 Percentage of Development Time

As the name suggests, percentage of development time [12] method estimates the test efforts based on the development time, which are obtained through line of code (LOC) or function points (FP) techniques. Although this method does not meet the scientific or technical principles, it serves in most cases.

4.3.2.3 Function Points Estimation

The efforts in Function Points Method [12] are basically measured on the basis of person-hours and transaction factors that are mostly extracted from previous projects. This method needs the required details and is not practical in that it does not document the use cases. As Xiaochun, et al [14]

Page 51: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 40

put it this method suffers from both constructive and adaptive problems. Another demerit of the method is that it requires a high cost to be implemented for the projects that are conducted in a big scale.

4.3.3 Estimation Model for Test Execution Effort based on the Test Specifications Estimation model for test execution effort based on the test specifications method [13] uses test

specifications such as size and execution complexity, which can be achieved through using control natural language (CNL). Actually, CNL is in a way the same as natural language but the only difference stems from some restriction in terms of the grammar and lexicon (these constraint can vary depending on the target domain) as the sentences written in CNL follow a given standard format. As a result, there is no possibility to describe an event and/or an object in different ways. The rule that should be obeyed in this technique is that each sentence contains one verb and zero or more argument in a sense in which verb indicates the action and argument implies the additional information about action. As an example, consider the following sentence: start the message center. In this example, the verb start is an action of starting an application and the argument the message center demonstrates the application that should be carried out. A wide view of how this method works is shown in Figure 8.

Figure 8. The Es ma ng Process of Execu on Effort [13]

The entire process would be described as follow:

1. Analyzing and assigning an execution point (size and execution complexity) to each test case. 1. Summing up all the execution points obtained through conducting step 1. 2. Estimating effort for executing all test cases in person-hours.

Generally, the size of a test case intimates the steps needed to execute it and execution complexity, on the other hand, implies the relation between tester and the product that is to be tested. In this regard, the TC1>TC2 means TC1 has bigger size and execution complexity than TC2. In addition, the similar relation (≈) means TC1 has smaller size and execution complexity than TC2.

Page 52: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 41

Assumption: TC1>TC2 only if TC1 is not similar to TC2

The equation 9 demonstrates the relationships between a and b that are execution points of test case t1 and t2; the first relation reveals the condition in which a is bigger than b, and the second relation illustrates the condition in which a is similar to b.

Equation 9. Estimation Relation [13]

The following is the whole process by which the size and execution complexity of a test case is computed:

The first step is to analyze each test step based on the system characteristics (C1, C2, …,Cn), which are functional and non-functional requirements. In the next step, with respect to the impact each characteristic has on the size and execution complexity of a test, we will assign an impact rate to them based on three scales of low, average, and high. Then, this impact rate would be considered as the criterion by which we can assign an execution point to each characteristic. Following this, the execution point of a test statement would be obtained by summing up the execution points of characteristics. Finally, summation of execution points of test statements is done to obtain test case execution point.

Some disadvantages of this method are listed below [14]:

1. Test cases should be ready prior to estimation. 2. As this method works on test step level, in case of having many test cases, the cost of

estimation is too high. 3. The relation between execution points and execution effort needs to be proven. 4. In order to use historical data, we need to execute the test cases prior to estimation.

4.3.4 Early Test Effort Estimation by Using Use Case

The main objective of this technique [14] is to use Use Cases in order to obtain the number of test cases, and then estimating effort based on three parameters; namely, test case number, execution complexity, and tester rank. Tester ranking will be assigned based on testing experience and knowledge in product to be tested. Test execution complexity is measured through equation 10:

푇푒푠푡퐸푥푒푐푢푡푖표푛퐶표푚푝푙푒푥푖푡푦 = [푠푐푎푙푒(푓푖) ∗ 푤푒푖푔ℎ푡(푓푖)]

Page 53: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 42

Equation 10. Test Execution Complexity [14]

Where scale(fi) and weight(fi) mean the ordinal scale value and the weight value of factor fi respectively. A high-level view of the whole process is shown in Figure 9.

Figure 9. Early Test Effort Es ma on Approach [14]

4.3.5 Cognitive Information Complexity Measure

Cognitive Information Complexity Measure (CICM) Method [16] uses the cognitive aspects of the practitioners. This approach is classified in the code-based category as it uses LOC and the number of operands and operators in each line.

Page 54: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 43

4.3.6 New, Search, Modify/ Management Information System Functionality

NSM/MIS method [17] uses the information of use cases in order to estimate test case effort. Actually, the basis of this method is the Use Case Point Method (UCPM), which is used in [12]. The use case information that is used in this method are actors, number of classes/transactions, technical factors, and environmental factors. This information delineates the values and if multiplied by a conversion factor, the estimated effort in time would be achieved. The difference between NSM/MIS and UCPM is that the two types of use cases, i.e. simple and complex, are considered in this method. As this method utilizes use case, we can put it in the group of model-based approaches. Disadvantage of this method is lack of the use case information.

4.3.7 Simplified Method based on the NSM/MIS method

Actually, this method is a simplified version of NSM/MIS [17] approach which considers just two types of actors that are human, and interface actors.

4.3.8 Software Requirement Specification (SRS)

SRS method [23] makes use of the software requirement specification. As this method uses required documents, it can estimates effort in the early stage of development phase. It uses improved requirement based complexity (IRBC), the input complexity (IC), output complexity (OC), and storage complexity (SC). It has grouped data storage, input, and output to different types and has assigned weight to them [23]. Then the calculation of IC, OC, and SC has been done. The input output complexity is obtained by combination of IC, OC, and SC.

4.3.9 Accumulated Efficiency Method

The main purpose of the Accumulated Efficiency Method [20] is to estimate test case execution effort by focusing on team efficiency. It puts into use the LOC and use case. Accumulated Efficiency Method is applied to manual execution and does not consider automatic execution since small test teams usually utilize manual testing instead of automatic test execution due to the cost restriction. Some advantages of this method would follow:

Objective judgment has no effect on it. It does not need any kind of controlled language. There is no need for data information.

The following variables have been used by this method to obtain accumulated efficiency metric: test case execution time and test case steps. The accumulated efficiency is calculated based on equation 11:

Page 55: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 44

Equation 11. Accumulated Efficiency [20]

Where di is the ith day, which is the sum of test steps, executed by day i, divided by total execution time. A disadvantage of this method is that it cannot estimate the effort prior to execution, whereas this is possible in our method (will be presented in Section 5) in the early stage. The other disadvantage is that it estimates the team effort, not each test case individually.

4.3.10 Case-Based Reasoning (CBR) Method

The CBR [21] approach uses data mining method to reduce the cost of testing and clustering technique to classify the data. It uses case base reasoning (CBR) which is a technique used for figuring out new problems by using previous situation and reusing that information. As this method uses the information from previous similar situations, one of its disadvantages is that, in case of no existence of these situations, we do not have any previous information to reuse in new problem. In addition, even in case of having similar situation it is not clear that it can adapt the information for new problem. In comparison with our approach (will be presented in Section 5) that is applicable for any size of project, this method is appropriate for small projects.

4.3.11 Cuckoo Search Method

Cuckoo search method [28] is similar to Nageswaran approach in [12]. The total number of parameters used by Nageswaran in [12] is 16, as well as 2 other parameters; namely, expertise of development team and expertise of test team. The main difference between Nageswaran approach [12] and cuckoo search method is that Nageswaran approach employs a fixed value weight to assign to the parameters while this method makes use of a range for each parameter. This range can be fixed or dynamic based on the target domain. With respect to the two new parameters (expertise of development team and expertise of test team), it divides them to three experienced groups including mixture of experienced and non-experienced as well as non-experienced, then assigns them a weight value but unlike its claim, it has used a fixed value for these two new parameters rather than a range. Another drawback of this method is that it needs at least a project as a historical data for which actual effort is known. Accordingly, in comparison to Nageswaran method [12], its advantage is that the weights assigned to parameters are not fixed and can be dynamic in a given range.

Page 56: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 45

5. Our Effort Estimation Model

In this Section, we present our new approach for estimating test case execution effort. Most of the existing methods that have been studied in the Related Work Section either are not in the system level or have not utilized architecture of the SUT to estimate in the early stage of the SDLC. Aranha and Borba have employed system characteristics exercised by the test step in each test case, which can estimate the execution effort in the early stages [13], but in case of having many test cases their model would be time consuming. Verifying system characteristics in each single test step for many numbers of test cases can be a rough task. We adopted our model for using system modules exercised by each test case instead of using test step. Our model works based on the test case. It means that the input of our model is test case. We have considered a significant degree/factor based on the importance of subsystems (modules, components, parts) in the system that are exercised by each test case. Consequently, as it is presented in Figure 10, test case serves as an input to the model. In the first phase, we determine the modules (components, parts) of the system that each test case would cover, demonstrated by module1, module2… modulen in step In the next step we analyze each module to find out the metrics that have been covered by that module. Then, an importance level (low, average, or high) will be assigned to those metrics in step Following this, in step the importance degree’s qualitative value will be mapped to a quantitative value. Finally, the importance degree of the test case that is the summation of all importance degrees related to all modules covered by the test case would be calculated. The guideline for importance degree of each part is illustrated in Table 11. The metrics value mapping process for qualitative values of low, average, and high to quantitative values of 3, 5, and 8 is based on Aranha and Borba method [13]. “In general, values 3, 5 and 8 are used to weight the levels Low, Average, and High, respectively” [13].

Page 57: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 46

Figure 10. Our Approach View

Importance degree would be determined by the factors that have been illustrated in the metric description section (the guideline for the importance level of modules can be different depending on the target domain). The way in which you can use Table 11 is that for each metric there are three qualitative values (low, average, and high). For data dependency metric, this means that if the number of data dependency is greater than 0 and less or equal to three, then its qualitative value will be low. If the number of data dependency is greater than 3 and less or equal to 5, then its qualitative value will be average. And finally, if the number of data dependency is greater than 5, its qualitative value will be high. Then for mapping the qualitative values to quantitative values, you can use the values in the second row of each metric that will map the qualitative values of low, average, and high to quantitative values 3, 5, and 8, respectively. The same scenario will go for security dependency metric, number of commuting, finish-to-start dependency, finish-to-finish dependency, external dependency, logical dependency, and the number of involved modules. For customer delivery priority, development time, and complexity metrics, the lower degree means the lower value of relevant metric. For instance, the complexity value of 5 has more complexity than complexity degree of 4. The degrees are ranked on a range of 1 to 5 for lower to higher degrees, respectively. That is, 1 means the lower degree and 5 means the higher degree.

Module1 . . . Module n

Metric1 . . . Metric n . . . Metric1 . . . Metric n

L A H . . . L A H . . . L A H . . . L A H

System Modules Exercised by the Test Case

3 . . . 5 . . . 5 . . . 8

50 Importance Degree of Test Case

Page 58: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 47

ID Metric Importance Level/Guideline/Rates Low Average High

1 Data Dependency

0<x<=3 3<x<=5 x>5

3 5 8

2 Security Dependency 0<x<=3 3<x<=5 x>5

3 5 8

3 Software Consumption Local Software Network Software Special Software

3 5 8

4 Hardware Consumption

Local Hardware(HW on

pc that TC is running)

Network Hardware Special HW e.g. Printer, Scanner

3 5 8

5 Customer Priority for Delivering 1,2 3,4 5

3 5 8

6 Development Time 1,2 3,4 5 3 5 8

7 Complexity 1,2 3,4 5

3 5 8

8 Number of Commuting 0<x<=5 5<x<=10 x>10

3 5 8

9 Input/ Output Type

text, character, integer, float etc.

Image, graphic, picture, animation,

etc. Audio, video, etc.

3 5 8

10 Data Storage Local Network

Remote data storage e.g via

Internet

3 5 8

11 Finish-to-Start Dependency 0<X<=3 3<x<=5 X>5

3 5 8

12 Finish-To-Finish Dependency 0<X<=3 3<x<=5 X>5

3 5 8

13 External-Dependency 0<X<=3 3<x<=5 X>5

3 5 8

14 Logical Dependency 0<X<=3 3<x<=5 X>5

3 5 8

15 Number of Involved System/Module/Layer 0<X<=3 3<x<=5 X>5 3 5 8

16 External I/O Source File/H.D.D USB/Network Internet 3 5 8

Table 11. Importance Degree Guideline

Page 59: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 48

We have supposed that testers have sufficient experience about the SUT, so we do not have considered tester skill level as a metric, which can be considered for precise estimation. The testers can be classified in two groups of non-experienced and experienced testers. As for executing a test case and in order to have an effective approach, all the involved systems should be considered, for instance, if in order to execute TC1 the systems a, b, and c are involved, then all the three systems should be considered.

5.1 Metrics Descriptions

Actually, among the following metrics, the researchers have defined some and the others have been taken from other references. Finish-to-Start Dependency and Finish-To-Finish Dependency metrics are in the development field and have been taken from [30] that we applied them in the test field. Data Dependency idea has been taken from [3] but we have changed it to be applicable for our approach. The other metrics including security dependency, software consumption, hardware consumption, customer delivery priority, development time, complexity, input/output type, data storage, external dependency, logical dependency, number of involved systems/modules, and external I/O source have been created by the researchers of the current study.

Data Dependency: we name starter the module where the test case starts from. As the starter requires receiving data from other involved modules, then it has data dependency to those modules.

Security Dependency: Security dependency metric occurs in case of existing dependency between modules either to receive username, password, and permission from other modules or to send to them.

Software Consumption: Software consumption concerns the software required for running the SUT and subsequently execute the test such as relevant operating system, network software, and so on. Generally, the software required in order to execute the test case (e.g. special test tools or test framework) would be consider a software consumption metric.

Hardware Consumption: This metric is the same as software consumption, but applicatory in the hardware.

Customer Delivery Priority: customer delivery priority considers the part of the system that is more or less important for the customer to deliver that part first. Indeed, customer assigns a priority to each part of the system so that this priority denotes the order of actual delivery. In order to have a mean value for total customer delivery priority of involved modules, we have calculated the mean of all customer delivery values.

Development Time: This metric is related to the time required to develop the relevant module. In order to have a mean value for total development time of the involved modules in the target test case, a mean value for all development times has been calculated.

Complexity: complexity metric is related to the complexity of relevant module. The complexity means how easy or difficult the module is to develop or test it. In order to have the total complexity

Page 60: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 49

value of all involved modules in the relevant test case, we have calculated the mean value for all complexity values to achieve an average complexity value for the target test case.

Number of Commuting: It presents the number of commuting between modules e.g. in case of having a web application the number of commuting between pages can be considered for this metric. This metric can be applicable depending on the target domain.

Input/ Output Type: Input/output type metric delineates the kind of data such as plain text, image, audio, and video, which are transferred among modules in the relevant system.

Data Storage: When a system uses local hard drive, a hard drive on a server, a remote hard drive in another place or via internet and so on, the data storage metric is the concerned parameter.

Finish-to-Start Dependency: it posits that we are working on two modules in an assumptive system. These two modules are interdependent and we aim to test them. The dependency between these modules is that in order to test the second module the first one should be finished. Accordingly, this kind of dependency is called finish-to-start dependency. “For example, you might want a completed, approved test plan before you start test development. This is called a finish-to-start dependency.” [30] “Finish-to-start dependency can be applied to both testing and developing aspects.” [30]. If we suppose that testing part A depends on testing part B; it means that we cannot test part A until the testing part B is finished. As such, we can claim that part A has finish-to-start dependency on part B [30]. The dependency in [30] is considered as the general aspect but we applied it to the testing process.

Finish-To-Finish Dependency: “As another example, suppose you decide to continue the system testing three weeks beyond both the completion of new feature development and the delivery of the last major change for testing. In this case, you have a finish-to-finish dependency.” [30]. “finish-to-finish dependency also can be applied to both testing and developing aspect” [30]. Actually, in finish-to-finish dependency the first part of relation should be suspended and waited for the second one to be finished.

External-Dependency: In case of having dependency to a system or module outside the SUT.

Logical Dependency: In case of having, dependency between one of the involved modules in the relevant test case to other module/modules, which is/are not, involved directly to the desired test case.

Number of Involved System/Module/Layer: It presents the number of modules that are involved in the relevant test case.

External I/O Source: This metric occurs in case of having input/output to the external source such as network or internet.

Page 61: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 50

CF (Conversion Factor) =푻풐풕풂풍푬풙풆풄풖풕풊풐풏푬풇풇풐풓풕푻풐풕풂풍푰풎풑풐풓풕풂풏풄풆푫풆품풓풆풆

= ퟒퟏ+ퟒퟕ+ퟔퟎ+ퟐퟎ+ퟖퟎ

ퟓퟖ+ퟓퟎ+ퟕퟏ+ퟒퟗ+ퟓퟒ=

ퟐퟒퟖ

ퟐퟖퟐ= ퟎ. ퟖퟕ

Estimated Effort of each Test Case= ID*CF

After computing importance degree of each test case, the conversion factor (CF) can be obtained either from historical data or by executing test cases. Conversion factor calculates in second per importance degree (CF also has been used in other methods [13]), and then multiply importance degree by CF for estimating the effort of test cases.

Effort=CF*Importance Degree

Test Case Actual Execution Effort Importance Degree TC1 41 58 TC2 47 50 . . . . . . . . . TCn 50 60

Figure 11. Conversion Factor Calculation

Some other metrics including tester rank (inexperienced, experienced) and TEF (Technical & Environmental Factor) that are used in other methods can be added for precise estimation. This method already has been applied to test steps, but due to a large volume of test cases in practice, considering test steps and assigning values at this refinement and granularity level might not be practical. As the number of test steps can grow greatly, the cost of doing the process itself will be expensive. Considering the above-mentioned reasons, we improved the method to be applicable to a higher level at test case and module/subsystem level, instead of test step and characteristic.

Page 62: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 51

6. Case Study

6.1 Case Study 1

In this Section, the result of applying our test execution effort estimation model to a desktop application domain at Fara Andish Company would be presented to validate the approach. First, we concisely illustrate the target system and introduce the Fara Andish Company. Fara Andish is a software company that works on different types of desktop applications. The main system, which they work on, and the technologies they use are described in continuance. It is an ERP (Enterprise Resource Planning) system, which is developed by using SQL Server, C# .NET, Microsoft Visual Studio, RUP methodology, Service Oriented, and Object Oriented methods. The system is comprised of the following modules:

Accounting Storehouse Purchasing Secretariat Payroll Treasure Sale Personnel Dynamic Report Builder Electronic Dashboard

A high-level view of the system containing the relationships among the modules is shown in Figure 12.

Page 63: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 52

Figure 12. High Level View of Fara Andish ERP System1

1 The figure is provided courtesy of Fara Andish System.

Page 64: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 53

In this case study, we have selected five test cases. The main principle in the process of test case selection was to select a variety of test cases in order for the following conditions to be covered:

Proceeding through external systems. Exchanging different types of data (e.g. integer, float, image, and file). Containing different number of modules to get more precise result. Covering different modules.

First of all, as shown in Table 12, we have taken into account the degrees of three metrics to be applied to our approach later on. The complexity, development time, and customer delivery priority (which means the system is more important for customer to deliver first), are ranked based on the project manager and developers experience at Fara Company. The lower degree means the lower value of relevant metric. For instance the payroll module with complexity 5 has more complexity than accounting module which has complexity degree 4. The degrees are ranked based on a range of 1 to 5 for lower and higher degrees, respectively. Hence, 1 means the lower degree and 5 means the higher degree.

Module Complexity Development Time Customer Delivery Priority

Accounting 4 4 5 After Sale Service 3 3 2 Dynamic Reporter Builder 1 1 5 E-Dashboard & Roll Document 3 3 4

Fixed Assets 2 2 3 Foreign Purchase 3 2 3 In-house Purchasing 2 1 3 Packaging 2 1 3 Payroll 5 4 4 Permisions 3 2 5 Personnel 2 2 4 Sale 4 4 3 Secretariat 2 2 4 Storehouse 4 5 4 Treasure 3 4 4

Table 12. Modules Ranking

Firstly, a full description of each test case is demonstrated in a table. Then a high-level view of exercised module and the modules that are involved in relevant test case is presented in next two subsequent pictures, respectively. Modules dependency and results have been illustrated in the relevant tables.

Page 65: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 54

TC1: Registering Sale Draft

The first test case that we have selected for case study 1 is registering the sale draft in the sale module. More detail of TC1 and its record have been illustrated in Table 13 and Table 14, respectively.

Test case ID:TC1 Date: <2013-12-25> Version: <1> Author: Seyed Morteza Hosseini

Reviewed by: <Kian Nasr(Project Manager)>

Used in: Fara ERP System

System version: <92.09.26.137>

Test environment: <win 7>

Test Suite: 1 Automated: <no>

Time for TC creation>N/A

TDT used: <>

Assumptions:

Starting point: Sale Draft Window. Test case: Registering new sale draft Pre-condition: There is no sale draft with the same number. Input Data (valid): 0-9, a-z, A-Z Output/visible result for passed: (post-condition) Sale draft should be registered and registration message should be showed. Side-effect (clean-up) no side effect for this test case. Clean up: Delete registered sale draft. Comment:

Table 13. Registration Sale Draft Test Case

Test Record

TR for Test case ID: <TC1>

Date: <2013-12-25> Executed Version: <1>

Tester: <Kian Nasr>

System version: <92.09.26.137>

Test environment: <win 7>

Test Suite: <1> Automated <no>

Result: <passed > Side-effect: Time for execution: <41 sec>

Failure ID nr: 1 Failure report date: <> Reported to:< > Reported by: <>

Failure description: Other: Comments:

Table 14. Registration Sale Draft Test Record

Page 66: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 55

Figure 13. Sale Module Communication of Fara Andish ERP System2

All communications concerned with the sale module and other modules have been demonstrated in Figure 13. Figure 14 has displayed the modules, which are involved in TC1 and their relations.

Figure 14. Modules Involved in TC1

2 The figure is provided courtesy of Fara Andish System.

Page 67: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 56

The dependencies of the sale module, storehouse module, as well as TC1 importance degree have been presented by Tables 15, 16, and 17, respectively. Based on the information in Table 15, sale module has security dependency with permission module as it receives security information from permission module. Sale module also has data dependency and finish-to-finish dependency with accounting, storehouse, e-dashboard, and permission modules.

Security Dependency

Data Dependency

Logical Dependency

Finish-To-Start

Dependency

Finish-To-Finish

Dependency

Sale

Accounting

Storehouse

E-Dashboard & Roll Document

Permission Table 15. Test Case 1 Sale Module Dependencies

The same explanation goes for storehouse module dependencies, which are security dependency and data dependency with permission module.

Security Dependency

Data Dependency

Logical Dependency

Finish-To-Start

Dependency

Finish-To-Finish

Dependency

Stor

ehou

se

Sale

Accounting

E-Dashboard & Roll Document

Permisions

Table 16. Test Case 1 Storehouse Module Dependencies

Page 68: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 57

The metrics and relevant values for modules involved in TC1 have been presented in Table 17. For instance, for the first metric that is data dependency metric there are 4 data dependencies in the sale module and 1 data dependency in the storehouse module. The total number of data dependencies is written in sum/average column, which is equal to 5 for this test case. The equivalent quantitative and qualitative values are average and 5, respectively. For the complexity, development time, and customer delivery priority metrics, the average value and the rounded value are written in the sum/average column. These values are written based on the importance degree guidelines in Table 11 as well as quantitative and qualitative values.

No

Module Sale

Accounting

Storehouse

Permission

e-Dashboard

Sum/A

verage

Qualitative value

Quantitative

value

Metric

1 Data Dependency 4 0 1 0 0 5 Average 5 2 Security Dependency 1 0 1 0 0 2 Low 3 3 Logic Dependency 0 0 0 0 0 0 - - 4 External Dependency 0 0 0 0 0 0 - - 5 Data Storage Network - Average 5 6 I/O Type data, image - Average 5 7 Number of Commuting 6 6 Average 5 8 Complexity 4 4 4 3 3 3.6≈4 Average 5 9 Development Time 4 4 5 2 3 3.6≈4 Average 5

10 Customer Delivery Priority 3 5 4 5 4 4.2≈4 Average 5

11 Hardware Consumption Barcode reader Network tools - High 8

12 Software Consumption Windows Installer, SQL Server, Report Viewer, .NET Framework, Windows Server - High 8

13 FTS Dependency 0 0 0 0 0 0 - - 14 FTF Dependency 4 0 0 0 0 4 Average 5

15 Number of Involved System 5 5 Average 5

Total (Importance Degree) 64 Table 17. Test Case 1 Importance Degree

Page 69: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 58

TC2: Purchase Invoice Issuance

Test case ID:TC2 Date: <2013-12-25> Version: <1> Author: Seyed Morteza Hosseini

Reviewed by: Kian Nasr

Used in: Fara ERP System

System version: <92.09.26.137>

Test environment: <win 7>

Test Suite: 1 Automated:<no>

Time for TC creation>N/A

TDT used: <>

Assumptions:

Starting point: Purchasing Invoice Window. Test case: Purchase invoice issuance Pre-condition: There is no purchase invoice with the same number. Selected invoice from storehouse has been approved and has no price already. Input Data (valid): 0-9, a-z, A-Z Output/visible result for passed: (post-condition) purchase invoice should be issued and relevant message should be showed. Side-effect (clean-up) no side effect for this test case. Clean-up: Delete issued purchase invoice. Comment:

Table 18. Purchase Invoice Issuance Test Case

Test Record

TR for Test case ID: <TC2>

Date: <2013-12-25> Executed Version: <1>

Tester: <Kian Nasr>

System version: <92.09.26.137>

Test environment: <win 7>

Test Suite: <1> Automated: <no>

Result: <passed > Side-effect: Time for execution: <47 sec>

Failure ID nr: <> Failure report date: <> Reported to:< > Reported by: <>

Failure description: Other: Comments:

Table 19. Purchase Invoice Issuance Test Record

Page 70: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 59

Figure 15. In-house Purchasing Module Communication of Fara Andish ERP System3

Figure 16. Modules Involved in Test Case 2

3 The figure is provided courtesy of Fara Andish System.

Page 71: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 60

Security Dependency

Data Dependency

Logical Dependency

Finish-To-Start

Dependency

Finish-To-Finish

Dependency

In-h

ouse

Pu

rcha

sing

Accounting

Storehouse

Permisions

Table 20. Test Case 2 In-house Purchasing Module Dependencies

Security Dependency

Data Dependency

Logical Dependency

Finish-To-Start

Dependency

Finish-To-Finish

Dependency

Stor

ehou

se

In-house Purchasing

N/A

Accounting

N/A

Permisions

N/A

Table 21. Test Case 2 Storehouse Module Dependencies

Page 72: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 61

NO

Module In-house Purchasing

Accounting

Storehouse

Permission

Sum/A

verage

Qualitative V

alue

Quantitative

Value

Metric

1 Data Dependency 3 0 1 0 4 Average 5

2 Security Dependency 1 0 1 0 2 Low 3

3 Logic Dependency 0 0 0 0 0 - -

4 External Dependency 0 0 0 0 0 - -

5 Data Storage Network - Average 5 6 I/O Type data, image -

7 Number of Commuting 4 4 Low 3

8 Complexity 2 4 4 3 3.25 ≈3 Average 5 9 Development Time 1 4 5 2 3 Average 5

10 Customer Delivery Priority 3 5 4 5 4.25 ≈4 Average 5

11 Hardware Consumption Network - Average 5

12 Software Consumption

Windows Installer, SQL Server, Report Viewer, .NET Framework, Windows Server

- High 8

13 FTS Dependency 0 0 0 0 0 - - 14 FTF Dependency 3 0 0 0 3 Low 3

15 Number of Involved System 4 4 Average 5

Total (Importance Degree) 52 Table 22. Test Case 2 Importance Degree

Page 73: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 62

TC3: Salary Calculation

Test case ID:TC3 Date: <2013-12-25> Version: <1> Author: Seyed Morteza Hosseini

Reviewed by: <Kian Nasr>

Used in: Fara ERP System

System version: <92.09.26.137>

Test environment: <win 7>

Test Suite: 1 Automated: <no>

Time for TC creation>N/A

TDT used: <>

Assumptions:

Starting point: Salary Operation Window. Test case: Salary Calculation Pre-condition: File from Exit & Entry, Contract, and Accounting Coding System should be ready. Input Data (valid):0-9, a-z, A-Z Output/visible result for passed: (post-condition) Salary should calculate. Side-effect (clean-up) no side effect for this test case. Clean-up: salary calculation can be done once a month. Comment:

Table 23. Salary Calculation Test Case

Test Record

TR for Test case ID: <TC3>

Date: <2013-12-25> Executed Version: <1>

Tester: <Kian Nasr>

System version: <92.09.26.137>

Test environment: <win 7>

Test Suite: <1> Automated: <no>

Result: <passed > Side-effect: Time for execution: <60 sec>

Failure ID nr: <> Failure report date: <> Reported to:< > Reported by: <>

Failure description: Other: Comments:

Table 24. Salary Calculation Test Record

Page 74: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 63

Figure 17. Payroll Module Communication of Fara Andish ERP System4

Figure 18. Modules Involved in Test Case 3

4 The figure is provided courtesy of Fara Andish System.

Page 75: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 64

Security Dependency

Data Dependency

Logical Dependency

Finish-To-Start

Dependency

Finish-To-Finish

Dependency

Payr

oll

Accounting

Permisions

Personnel

Report Builder

Treasure

Enter & Exit Sysytem

Table 25. Test Case 3 Payroll Module Dependencies

Security Dependency

Data Dependency

Logical Dependency

Finish-To-Start

Dependency

Finish-To-Finish

Dependency

Accounting Treasure

N/A

Table 26. Test Case 3 Accoun ng Module Dependencies

Page 76: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 65

NO

Module Payroll

Accounting

Personnel

Permission

Report

Builder

Treasure

Sum/A

verage

Qualitative value

Quantitative

value

Metric

1 Data Dependency 6 1 0 0 0 0 7 high 8 2 Security Dependency 1 0 0 0 0 0 1 Low 3 3 Logic Dependency 0 0 0 0 0 0 0 - - 4 External Dependency 1 0 0 0 0 0 1 Low 3 5 Data Storage Network - Average 5 6 I/O Type data, file - Average 5 7 Number of Commuting 6 6 Average 5 8 Complexity 5 4 2 3 1 3 3 Average 5 9 Development Time 4 4 2 2 1 3 2.6≈3 Average 5

10 Customer Delivery Priority 4 5 4 5 5 4 4.5≈5 High 8

11 Hardware Consumption Network tools - Average 5

12 Software Consumption Windows Installer, SQL Server, Report Viewer, .NET Framework, Windows Server - High 8

13 FTS Dependency 0 0 0 0 0 0 - - 14 FTF Dependency 6 0 0 0 0 6 High 8

15 Number of Involved System 7 7 High 8

Total (Importance Degree) 76 Table 27. Test Case 3 Importance Degree

Page 77: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 66

TC4: Accounting Code Registration

Test case ID:TC4 Date: <2013-12-25>

Version: <1> Author: Seyed Morteza Hosseini

Reviewed by: <Kian Nasr>

Used in: Fara ERP System

System version: <92.09.26.137>

Test environment: <win 7>

Test Suite: 1 Automated: <no>

Time for TC creation>N/A

TDT used: <>

Assumptions:

Starting point: Accounting Coding Window. Test case: Registering Accounting Code Pre-condition: The accounting code should not exist. Input Data (valid): 0-9, a-z, A-Z Output/visible result for passed: (post-condition) Accounting code should register. Side-effect (clean-up) no side effect for this test case. Clean-up: Created accounting code should delete. Comment:

Table 28. Accounting Code Registration Test Case

Test Record

TR for Test case ID: <TC4>

Date: <2013-12-25> Executed Version: <1>

Tester: <Kian Nasr>

System version: <92.09.26.137>

Test environment: <win 7>

Test Suite: <1> Automated <no>

Result: <passed > Side-effect: Time for execution: <20 sec>

Failure ID nr: <> Failure report date: <> Reported to:< > Reported by: <>

Failure description: Other: Comments:

Table 29. Accounting Code Registration Test Record

Page 78: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 67

Figure 19. Accounting Module Communication of Fara Andish ERP System5

Figure 20. Modules Involved in Test Case 4

5 The figure is provided courtesy of Fara Andish System.

Page 79: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 68

Security Dependency

Data Dependency

Logical Dependency

Finish-To-Start

Dependency

Finish-To-Finish

Dependency

Accounting Permisions

Table 30. Accounting Module Dependencies

NO

Module

Accounting

Permission

Sum/A

verage

Qualitative value

Quantitative

value

Metric

1 Data Dependency 1 0 1 Low 3 2 Security Dependency 1 0 1 Low 3 3 Logic Dependency 0 0 0 - - 4 External Dependency 0 0 0 - - 5 Data Storage Network - Average 5 6 I/O Type Data - low 3 7 Number of Commuting 1 1 Low 3 8 Complexity 4 3 3.5≈4 Average 5 9 Development Time 4 2 3 Average 5

10 Customer Delivery Priority 5 5 5 High 8

11 Hardware Consumption Network tools - Average 5

12 Software Consumption Windows Installer, SQL Server, Report

Viewer, .NET Framework, Windows Server

- High 8

13 FTS Dependency 0 0 0 - - 14 FTF Dependency 1 0 1 Low 3

15 Number of Involved System 2 2 Low 3

Total (Importance Degree) 54 Table 31. Test Case 4 Importance Degree

Page 80: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 69

TC5: Personnel Registration

Test case ID:TC5 Date: <2013-12-25>

Version: <1> Author: Seyed Morteza Hosseini

Reviewed by: <Kian Nasr>

Used in: Fara ERP System

System version: <92.09.26.137>

Test environment: <win 7>

Test Suite: 1 Automated <no>

Time for TC creation>N/A

TDT used: <>

Assumptions:

Starting point: Personnel Information Window. Test case: Personnel Registration Pre-condition: A personnel with the same information should not exist. Input Data (valid): 0-9, a-z, A-Z, image Output/visible result for passed: (post-condition) personnel should be registered. Side-effect (clean-up) no side effect for this test case. Clean-up: Created personnel should delete. Comment:

Table 32. Personnel Registration Test Case

Test Record

TR for Test case ID: <TC5>

Date: <2013-12-25> Executed Version: <1>

Tester: <Kian Nasr>

System version: <92.09.26.137>

Test environment: <win 7>

Test Suite: <1> Automated: <no>

Result: <passed > Side-effect: Time for execution: <80 sec>

Failure ID nr: <> Failure report date: <> Reported to:< > Reported by: <>

Failure description: Other: Comments:

Table 33. Registration of Personnel Test Record

Page 81: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 70

Figure 21. Personnel Module Communication of Fara Andish ERP System6

Figure 22. Modules Involved in Test Case 5

6 The figure is provided courtesy of Fara Andish System.

Page 82: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 71

Security Dependency

Data Dependency

Logical Dependency

Finish-To-Start

Dependency

Finish-To-Finish

Dependency

Personnel Permisions

Table 34. Personnel Module Dependencies

NO

Module

Personnel

Permission

Sum/A

verage

Qualitative value

Quantitative

value

Metric

1 Data Dependency 1 0 1 Low 3 2 Security Dependency 1 0 1 Low 3 3 Logic Dependency 0 0 0 - - 4 External Dependency 1 0 1 Low 3 5 Data Storage Network - Average 5 6 I/O Type Data, Image - Average 5 7 Number of Commuting 1 1 Low 3 8 Complexity 2 3 2.5≈3 Average 5 9 Development Time 2 2 2 Low 3

10 Customer Delivery Priority 4 5 4.5≈5 High 8

11 Hardware Consumption Network tools, Scanner Network tools - High 8

12 Software Consumption Windows Installer, SQL Server, Report

Viewer, .NET Framework, Windows Server

- High 8

13 FTS Dependency 0 0 0 - - 14 FTF Dependency 1 0 1 Low 3

15 Number of Involved System 2 2 Low 3

16 External I/O Source Network, USB - Average 5 Total (Importance Degree) 65

Table 35. Test Case 5 Importance Degree

Page 83: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 72

Conversion Factor =푻풐풕풂풍푨풄풕풖풂풍푬풙풆풄풖풕풊풐풏푬풇풇풐풓풕푻풐풕풂풍푰풎풑풐풓풕풂풏풄풆푫풆품풓풆풆

=ퟒퟏ ퟒퟕ ퟔퟎ ퟐퟎ ퟖퟎퟔퟒ ퟓퟐ ퟕퟔ ퟓퟒ ퟔퟓ

= ퟐퟒퟖퟑퟏퟏ

=0.79

Equation 12. Conversion Factor

Estimated Execution Effort=Conversion Factor*Importance Degree

Equation 13. Estimation Execution Effort Formula

In Table 36 the “+” and “-” signs indicate that the estimated effort is greater and smaller than the actual effort respectively.

Test Case Actual Execution Effort (in second)

Importance Degree

Estimated Execution

Effort(in second)

Actual & Estimated

Effort Difference

TC1 41 64 50.56 +9.56 TC2 47 52 41.08 -5.92 TC3 60 76 60.04 +0.04 TC4 20 54 42.66 +22.66 TC5 80 65 51.35 -28.65

Table 36. Case Study Results

Page 84: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 73

Chart 4. Comparison of Total Actual Execution Effort & Total Estimated Execution Effort

Chart 5. Actual & Estimated Execution Effort

0

10

20

30

40

50

60

70

80

90

TC 1 TC2 TC3 TC4 TC5

Actual Effort

Estimated Effort

41

47

60

20

80

50.56

41.08

60.04

42.66

51.35

0

10

20

30

40

50

60

70

80

90

TC1 TC2 TC3 TC4 TC5

Actual Effort

Estimated Effort

Page 85: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 74

6.2 Case Study 2

Having applied our approach on a desktop application, we tried to employ our estimation model on a web application domain at Partak Company’s CMS (Content Management System) in order to compare the results. Partak Company is a software company that works on both web and mobile applications. They develop applications for iOS in mobile domain. They use MVC technology to develop web applications that we selected one of their web applications for case study 2. A high-level view of Partak’s CMS is depicted in Figure 23.

Layer

Figure 23. High Level View of Partak CRM7

Due to the difficulty of showing the high level view of Partak CRM, which is really a big picture, a high level view of Web.Controllers Layer has been showed in Figure 24 as a sample.

7 The figure is provided courtesy of Partak CRM System.

Page 86: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 75

Figure 24. High Level View of Web.Controllers Layer

In this case study, we have selected three test cases. In the same vein as in the case study 1, in Table 37, we have demonstrated degrees of three metrics (complexity, development time, and customer delivery priority) that would be applied to our approach later on. The ranking process has been conducted based on the experience of project manager and developers at Partak Company.

No Layer Complexity Development Time

Customer Delivery Priority

1 Partak.Web.Areas.Admin.Controllers. 1 5 1 2 Partak.Model.Models 1 2 1 3 Partak.Model.ViewModels 1 1 1 4 Partak.Web.Data 4 5 4 5 Partak.Web.Classes 1 4 3 6 Partak.Web 1 4 5 7 Partak.Web.Areas.Admin.Classes 1 4 2

Table 37. Layers Ranking

Class Layer

Page 87: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 76

TC1: Inserting New News

Test case ID:<TC1>

Date: <2014-01-16> Version: <1> Author:< Seyed Morteza Hosseini>

Reviewed by: <Saeed Salehi(Project Manager)>

Used in: <Partak CMS System>

System version: <1.0 MVC Based>

Test environment: <Google Chrom>

Test Suite:< 1> Automated <no>

Time for TC creation>N/A

TDT used: <>

Assumptions: <>

Starting point: New Post Window. Test case: Inserting new news Pre-condition: Input Data (valid): 0-9, a-z, A-Z, image, any character Output/visible result for passed: (post-condition) New news should be showed in the news page. Side effect (clean up) no side effect for this test case. Clean-up: Comment:

Table 38. Inserting New News Test Case

Test Record

TR for Test case ID: <TC1>

Date: <2014-01-16> Executed Version:<1> Tester: <Saeed Salehi>

System version:<1.0 MVC Based>

Test environment: <Google Chrom>

Test Suite: <1> Automated:<no>

Result: <passed > Side-effect: Time for execution: <70 sec>

Failure ID nr: Failure report date: <> Reported to:< > Reported by: <>

Failure description: Other: Comments:

Table 39. Inserting New News Test Record

Layers Involved in TC1:

1. Partak.Web.Areas.Admin.Controllers 2. Partak.Model.Models 3. Partak.Model.ViewModels 4. Partak.Web.Data 5. Partak.Web.Classes 6. Partak.Web

Due to the space limitation in Figure 25, the names of the layers have been written concisely. For instance, the Partak.Web.Areas.Admin.Controllers Layer has been written Controllers briefly. The layers communications for test case 1 is drawn in Figure 25. The Controllers Layer communication

Page 88: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 77

with Models, ViewModels, Data, and Classes Layers and its communication with Web Layer are mutual and one-way connection, respectively.

Figure 25. TC1 Layers Communica ons

Due to the space limitation in the following tables, layer’s occurrence order has been written instead of their names for the relevant test case. For instance, for test case 1, the first layer that has been mentioned is “Partak.Web.Areas.Admin.Controllers” Layer, with number 1 in the list so that it has been written 1 instead of its name in the Table 40. Similarly, for Partak.Web Layer, which its order in list is 6, the value 6 has been written in the relevant table.

1.Controllers

2.Models

3.ViewModels

4.Data5.Classes

6.Web

Page 89: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 78

Layer

Security Dependency Data Dependency Logical Dependency

Finish-To-Start

Dependency

Finish-To-Finish

Dependency

1 6 3,2,5,4

2,3,4,5,6

2 1

1,3,4,5,6

3 1

1,2,4,5,6

4 1

1,2,3,5,6

5 1

1,2,3,4,6

6

1,2,3,4,5

Table 40. Test Case 1 Layers Dependencies

No

Layer

1 2 3 4 5 6

Sum/A

verage

Quantitative

value

Qualitative value

Metric

1 Data Dependency 4 1 1 1 1 0 8 High 8 2 Security Dependency 1 0 0 0 0 0 1 Low 3 3 Logic Dependency 0 0 0 0 0 0 0 - - 4 External Dependency 0 0 0 0 0 0 0 - - 5 Data Storage Remote - High 8 6 I/O Type text, character, integer, float etc. - Low 3

7 Number of commuting between pages 1 1 Low 3

8 Complexity 1 1 1 4 1 1 1.5≈2 Low 3 9 Development Time 5 2 1 5 4 4 3.5≈4 Average 5

10 Customer Delivery Priority 1 1 1 4 3 5 2.5≈3 Average 5

11 Hardware Consumption Network Tools - Average 5

12 Software Consumption MS Visual Studio, Third Party dlls, Glimpse, IE, MS SQL, IIS - High 8

13 FTS Dependency 0 0 0 0 0 0 0 - - 14 FTF Dependency 5 5 5 5 5 5 30 High 8

15 Number of Involved Layers 6 6 High 8

Total (Importance Degree) 67 Table 41. Test Case 1 Importance Degree

Page 90: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 79

TC2: Inserting New Image in Image Gallery

Test case ID:<TC2>

Date: <2014-01-16> Version: <1> Author:<Seyed Morteza Hosseini>

Reviewed by: <Saeed Salehi(Project Manager)>

Used in: <Partak CMS System>

System version: <1.0 MVC Based>

Test environment: <Google Chrom>

Test Suite: <1> Automated:<no>

Time for TC creation>N/A

TDT used:<> Assumptions: <>

Starting point: New Image Window. Test case: Inserting new image in image gallery Pre-condition: Input Data (valid): 0-9, a-z, A-Z, image, any character Output/visible result for passed: (post-condition) New image should be added to the image galley. Side-effect (clean-up) no side effect for this test case. Clean-up: Comment:

Table 42. Inserting New Image in Image Gallery Test Case

Test Record

TR for Test case ID: <TC2>

Date: <2014-01-16> Executed Version:<1> Tester: <Saeed Salehi>

System version:<1.0 MVC Based>

Test environment: <Google Chrom>

Test Suite: <1> Automated:<no>

Result: <passed > Side-effect: Time for execution: <50 sec>

Failure ID nr: Failure report date: <> Reported to:< > Reported by: <>

Failure description: Other: Comments:

Table 43. Inserting New Image in Image Gallery Test Record

Layers Involved in TC2:

1. Partak.Web.Areas.Admin.Controllers 2. Partak.Model.Models 3. Partak.Model.ViewModels 4. Partak.Web.Data 5. Partak.Web.Classes 6. Partak.Web 7. Partak.Web.Areas.Admin.Classes

Page 91: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 80

Figure 26. TC2 Layers Communica ons

Layer

Security Dependency Data Dependency Logical Dependency

Finish-To-Start

Dependency

Finish-To-Finish

Dependency

1 6 3,2,5,4,7

2,3,4,5,6,7

2 1

1,3,4,5,6,7

3 1

1,2,4,5,6,7

4 1

1,2,3,5,6,7

5 1

1,2,3,4,6,7

6

1,2,3,4,5,7

7 1

1,2,3,4,5,6

Table 44. Test Case 2 Layers Dependencies

1.Controllers

2.Models

3.ViewModels

4.Data

5.Classes

6.Web

7.Admin.Classes

Page 92: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 81

No

Layer

1 2 3 4 5 6 7

Sum/A

verage

Quantitative

value

Qualitative value

Metric

1 Data Dependency 5 1 1 1 1 0 1 10 High 8 2 Security Dependency 1 0 0 0 0 0 0 1 Low 3 3 Logic Dependency 0 0 0 0 0 0 0 0 - - 4 External Dependency 0 0 0 0 0 0 0 0 - - 5 Data Storage Remote - High 8 6 I/O Type text, character, integer, float etc. - Low 3

7 Number of commuting between pages 2 2 Low 3

8 Complexity 1 1 1 4 1 1 1 1.4≈1 Low 3 9 Development Time 5 2 1 5 4 4 4 3.5≈4 Average 5

10 Customer Delivery Priority 1 1 1 4 3 5 2 2.4≈2 Low 3

11 Hardware Consumption Network Tools - Average 5

12 Software Consumption MS Visual Studio, Third Party dlls, Glimpse, IE, MS SQL, IIS - High 8

13 FTS Dependency 0 0 0 0 0 0 0 0 - - 14 FTF Dependency 6 6 6 6 6 6 6 42 High 8

15 Number of Involved Layers 7 7 High 8

Total (Importance Degree) 65 Table 45. Test Case 2 Importance Degree

Page 93: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 82

TC3: Creating New Training Class

Test case ID:TC3 Date: <2014-01-16> Version: <1> Author:< Seyed Morteza Hosseini>

Reviewed by: <Saeed Salehi(Project Manager)>

Used in: <Partak CMS System>

System version: <1.0 MVC Based>

Test environment: <Google Chrom>

Test Suite: <1> Automated <no>

Time for TC creation>N/A

TDT used: <>

Assumptions:

Starting point: New Class Window. Test case: Creating new training class Pre-condition: Input Data (valid): 0-9, a-z, A-Z, any character Output/visible result for passed: (post-condition) New training class should be added to the class list. Side-effect (clean-up) no side effect for this test case. Clean-up: Comment:

Table 46. Creating New Training Class Test Case

Test Record

TR for Test case ID: <TC3>

Date: <2014-01-16> Executed Version:<1> Tester: <Saeed Salehi>

System version:<1.0 MVC Based>

Test environment: <Google Chrom>

Test Suite: <1> Automated: <no>

Result: <passed > Side-effect: Time for execution: <73 sec>

Failure ID nr: Failure report date: <> Reported to:< > Reported by: <>

Failure description: Other: Comments:

Table 47. Creating New Training Class Test Record

Layers Involved in TC3:

1. Partak.Web.Areas.Admin.Controllers 2. Partak.Model.Models 3. Partak.Model.ViewModels 4. Partak.Web.Data 5. Partak.Web.Classes 6. Partak.Web

Page 94: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 83

Figure 27. TC3 Layers Communica ons

Layer

Security Dependency Data Dependency Logical Dependency

Finish-To-Start

Dependency

Finish-To-Finish

Dependency

1 6 3,2,5,4

2,3,4,5,6

2 1

1,3,4,5,6

3 1

1,2,4,5,6

4 1

1,2,3,5,6

5 1

1,2,3,4,6

6

1,2,3,4,5

Table 48. Test Case 3 Layers Dependencies

1.Controllers

2.Models

3.ViewModels

4.Data5.Classes

6.Web

Page 95: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 84

No

Layer

1 2 3 4 5 6

Sum/A

verage

Quantitative

value

Qualitative value

Metric

1 Data Dependency 4 1 1 1 1 0 8 High 8 2 Security Dependency 1 0 0 0 0 0 1 Low 3 3 Logic Dependency 0 0 0 0 0 0 0 - - 4 External Dependency 0 0 0 0 0 0 0 - - 5 Data Storage Remote - High 8

6 I/O Type text, character, integer, float etc. - Low 3

7 Number of Commuting between Pages 1 1 Low 3

8 Complexity 1 1 1 4 1 1 1.5≈2 Low 3 9 Development Time 5 2 1 5 4 4 3.5≈4 Average 5

10 Customer Delivery Priority 1 1 1 4 3 5 2.5≈3 Average 5

11 Hardware Consumption Network Tools - Average 5

12 Software Consumption MS Visual Studio, Third Party dlls, Glimpse, IE, MS SQL, IIS

- High 8

13 FTS Dependency 0 0 0 0 0 0 0 - - 14 FTF Dependency 5 5 5 5 5 5 30 High 8

15 Number of Involved Layers 6 6 High 8

Total (Importance Degree) 67 Table 49. Test Case 3 Importance Degree

Estimated Execution Effort=Conversion Factor*Importance Degree

In Table 50 the “+” and “-” signs indicate that the estimated effort is greater and smaller than the actual effort respectively.

Conversion Factor =푻풐풕풂풍푨풄풕풖풂풍푬풙풆풄풖풕풊풐풏푬풇풇풐풓풕푻풐풕풂풍푰풎풑풐풓풕풂풏풄풆푫풆품풓풆풆

=ퟕퟎ ퟓퟎ ퟕퟑퟔퟕ ퟔퟓ ퟔퟕ

= ퟏퟗퟑퟏퟗퟗ

=0.96

Equation 14. Case Study 2 Conversion Factor

Page 96: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 85

Test Case Actual Execution Effort (in second)

Importance Degree

Estimated Execution

Effort(in second)

Actual & Estimated

Effort Difference

TC1 70 67 64.32 -5.68 TC2 50 65 62.4 +12.4 TC3 73 67 64.32 -8.68

Table 50. Case Study 2 Results

Comparison of total actual execution effort and total estimated execution effort has been presented in Chart 6. The actual effort is drawn in dark gray color and estimated effort in light gray color. The actual and estimated execution effort values for each single test case are shown in Chart 7.

Chart 6. Comparison of Total Actual Execution Effort & Total Estimated Execution Effort

0

10

20

30

40

50

60

70

80

TC 1 TC2 TC3

Actual Effort

Estimated Effort

Page 97: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 86

Chart 7. Actual & Estimated Execution Effort

70

50

73

64.32 62.4 64.32

0

10

20

30

40

50

60

70

80

TC1 TC2 TC3

Actual Effort

Estimated Effort

Page 98: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 87

7. Discussion

Case Study 1

Before going to case study discussion, for description of any metrics refer to Section 5.1 metrics description and related result tables of each test case in Section 6.1 case study 1. After evaluating the range of TC1 to TC5, based on our results, the estimated execution effort of TC1, TC2, and TC3 were adequately close to the actual values but the estimated effort of TC4 and TC5 were away from the real values. The reason that the effort of TC5 was away from the real value is that the importance degree of TC5 was not close to the desired value. The importance degree (ID) of TC5 is 65, which is away from its desired value that is 92. The same scenario goes for TC4. The obtained ID was 54, whereas desired value was 23. In the following these comparison is shown that shows the closer ID to the desired value the more accurate estimated effort.

Current ID Value Desired ID Value TC5 65 < 92 TC4 54 > 23

According to these findings, it was concluded that less difference will exist between ID and actual effort, the more accurate will be the estimated effort; hence, we shall try to have factors that are more realistic in ID calculation.

Test Case Actual Effort Estimated Effort ID & Actual Effort Difference

Actual & Estimated Effort

Difference TC1 41 50.56 23 +9.56 TC2 47 41.08 5 -5.92 TC3 60 60.04 16 +0.04 TC4 20 42.66 34 +22.66 TC5 80 51.35 -15 -28.65

Table 51. Actual & Estimated Effort Difference

According to Table 51 the actual effort value for TC1 is 41 and the estimated effort value of TC1 is 50.56, so the difference value of these values is +9.56 which demonstrates that the estimated effort based on our method is around 10 units more than the needed actual effort. The results imply that the effort estimated with our model is comparatively close to the actual effort. The same explanation goes for TC2, so the actual and estimated effort difference column proves that the difference value of actual and estimated effort is -5.92 which means the estimated effort value by our model is lower than the actual effort that is needed, but it is relatively close to the actual value. For TC3 the difference between actual effort and estimated effort values is +0.04 that demonstrates our model estimated value is almost correct. For TC4 and TC5 the actual effort and estimated effort difference values that are +22.66 and -28.65 respectively, show that the values which estimated by our model are not close the actual needed values. After reviewing the metrics, we discovered that we did not consider the data

Page 99: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 88

type between Personnel Module and external system for TC5, which in this case is H.D.D or server. The data type is an image that can be signature and picture of the personnel, so as a new metric the “external O/I source” were added to the metrics of TC5 (16th row in test case 5 importance degree Table, Table 35) we had new importance degree for TC5 that is 62. Another missing factor for TC5 was in HW Consumption metric so that we did not take into consideration the scanner needed for scanning the signature and picture of the personnel. Accordingly, HW Consumption value increased to 8 (high), therefore the new ID value were 65 (3 points added).

Case Study 2

The same as case study 1 before going to case study discussion, for description of any metrics refer to Section 5.1 metrics description and related result table of each test case in Section 6.2 case study 2. The Comparison of case studies’ results based on Table 50 reveals that for case study 2 the results are more accurate than case study 1. Having more experience in criteria selection (as we learnt from case study 1) or fewer test cases in case study 2 can be the reasons of having more accurate results in case study 2. After evaluating the range of TC1 to TC3, based on our results, the estimated execution effort of TC1, TC2, and TC3 were adequately close to the actual values. The results in Table 50 show the closer ID to the desired value the more accurate estimated effort. The same as case study 1 and according to these determinations, it was concluded that less difference between ID and actual effort, the more accurate will be the estimated effort; hence, we shall try to have more realistic factors in ID calculation.

According to Table 50 the actual effort value for TC1 is 70 and the estimated effort value of TC1 is 64.32, so the difference value of these values is -5.68 which demonstrates that the estimated effort based on our method is about 6 units less than the needed actual effort. The results imply that the effort estimated with our model is comparatively close to the actual effort. The same explanation goes for TC2, so the actual and estimated effort difference column proves that the difference value of actual and estimated effort is +12.4 which means the estimated effort value by our model is greater than the actual effort that is needed, but it is almost close to the actual value. For TC3 the difference between actual effort and estimated effort values is -8.68, which demonstrates our model estimated value is nearly close to the actual one.

Page 100: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 89

8. Summary and Conclusion

In the present thesis work, we attempted to present a new model for estimating the test execution effort. In our new model, we considered the number of modules subsumed within each test case. Then, based on the specified criteria and parameters, an importance value was assigned to each module (the criteria can vary depending on the target domain). Following that, the total importance value/degree of executed test cases was calculated. Finally, we tried to estimate an effort value by using the importance degree of relevant test case in our formula. Most of existing methods have used the code in order to estimate the required effort and, as such, they are classified into the code-level techniques category and do not estimate the effort in the early stage of software life cycle. Unlike the aforementioned methods, because our model uses the high level architecture of SUT, the effort would be estimated in the early stages. We improved the method to be applicable to a higher level at test case and module/subsystem level, instead of test step and characteristic. Compared to the existing methods, the main advantage of our model is that it does not need the historical data and in case of employing an experienced tester on the target domain, or having conducted previous similar projects, the executing phase of test cases would not be required. Finally, we put our approach into practice by evaluating it on a desktop as well as a web application domain. In the case of desktop application, we selected five test cases, while for web application only three test cases were considered. In both desktop and web applications, after tuning the formula, the obtained results were considerably close to the reality. As a recommendation for the future studies, more investigation on the criteria by which researchers can have more precise results. Evidently, improving the formula might lead to a more accurate estimation. Further to the mentioned recommendations, more research can be done in executing testing in early stage of software development life cycle. The more precise criteria in addition to adding new criteria lead to have a thorough paced towards the more accurate results. Aside recommendations, we encountered some restriction such as accessing to different target domains that was one of our main limitations during this thesis work, in addition to satisfy the companies' owner for giving permission in order to test our model on their products. Having variety of test cases which cover different aspects was also another restriction that we encountered which can be considered as more investigation, although we tried our best to test the model on different target domains and test cases. Finding testers with having experience in different domains was another problem in this thesis work, which can also consider as another item for investigation.

Page 101: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 90

9. References

[1] Praveen Ranjan Srivastava, “Test Case Prioritization”, Journal of Theoretical and Applied Information Technology, 2008.

[2] Gregg Rothermel, Roland H. Untch, Chengyun Chu, Mary Jean Harrold, “Test Case Prioritization: An Empirical Study”, Department of Computer Science Oregon State U. Corvallis, OR Department of Computer Science Middle Tenn. State U.Murfreesboro, TN. Department of Computer and Information Science Ohio State University Columbus, OH, Last Access Sep 2013.

[3] Dennis Jeffrey, Neelam Gupta, “Test Case Prioritization Using Relevant Slices”, Proceedings of the 30th Annual International Computer Software and Applications Conference (COMPSAC'06), 2006.

[4] Sebastian Elbaum, Alexey G. Malishevsky, Gregg Rothermel, “Test Case Prioritization:A Family of Empirical Studies”, IEEE Transactions On Software Engineering, VOL. 28, NO. 2, February 2002. [5] Sejun Kim, Jongmoon Baik, Department of Computer Science KAIST Daejeon, Korea, “An Effective Fault Aware Test Case Prioritization by Incorporating a Fault Localization Technique”, ESEM’10, Bolzano-Bozen, Italy, 2010. [6] Alexey G. Malishevsky, Joseph R. Ruthru, Gregg Rothermel, Sebastian Elbaum, ” Cost-cognizant Test Case Prioritization”, Technical Report TR-UNL-CSE-2006-0004, Department of Computer Science and Engineering, University of Nebraska{Lincoln, Lincoln, Nebraska, U.S.A., 12 March 2006. [7] Gregg Rothermel, Roland H. Untch, Chengyun Chu, Mary Jean Harrold, “Prioritizing Test Cases For Regression Testing”, IEEE Transactions On Software Engineering, VOL. 27, NO. 10, October 2001. [8] Bogdan Korel, Luay H. Tahat, Mark Harman, ” Test Prioritization Using System Models”, Proceedings of the 21st IEEE International Conference on Software Maintenance (ICSM’05), 2005. [9] Paolo Tonella, Paolo Avesani, Angelo Susi, ” Using the Case-Based Ranking Methodology for Test Case Prioritization”, 22nd IEEE International Conference on Software Maintenance (ICSM'06), 2006. [10] Bo Qu, Changhai Nie, Baowen Xu, Xiaofang Zhang, “Test Case Prioritization for Black Box Testing”, 31st Annual International Computer Software and Applications Conference (COMPSAC 2007), 2007.

Page 102: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 91

[11] Siavash Mirarab, Ladan Tahvildari, “An Empirical Study on Bayesian Network-based Approach for Test Case Prioritization”, International Conference on Software Testing, Verification and Validation, 2008. [12] Suresh Nageswaran, “Test Effort Estimation Using Use Case Points”, Quality Week 2001, San Francisco, California, USA, June 2001. [13] Eduardo Aranha, Paulo Borba, “An Estimation Model for Test Execution Effort”, In Proceedings of First International Symposium on Empirical Software Engineering and Measurement, 2007. [14] ZHU Xiaochun, ZHOU Bo, WANG Fan, QU Yi, CHEN Lu, “Estimate Test Execution Effort at an Early Stage: An Empirical Study”, International Conference on Cyberworlds, 2008. [15] Mehrdad Saadatmand, Mikael Sjödin,” On Combining Model-Based Analysis and Testing”, 10th International Conference on Information Technology: New Generations (ITNG 2013), USA, April 2013. [16] Dharmender Singh Kushwaha , A.K.Misra, ”Software Test Effort Estimation”, ACM SIGSOFT Software Engineering Notes, 2008. [17] Érika R. C. de Almeida, Bruno T. de Abreu, Regina Moraes, ” An Alternative Approach to Test Effort Estimation Based on Use Cases”, International Conference on Software Testing Verification and Validation, 2009. [18] Ashish Sharma, Dharmender Singh Kushwaha, ” A Metric Suite for Early Estimation of Software Testing Effort using Requirement Engineering Document and its validation”, International Conference on Computer & Communication Technology (ICCCT), 2011. [19] Mourad Badri, Linda Badri, William Flageol,” Predicting the Size of Test Suites from Use Cases: An Empirical Exploration”, IFIP International Federation for Information Processing, 2013. [20] Daniel Guerreiro e Silva, Bruno T. de Abreu, Mario Jino, ” A Simple Approach for Estimation of Execution Effort of Functional Test Cases”, International Conference on Software Testing Verification and Validation, 2009. [21] Ali Ilkhani, Golnoosh Abaee, “Extracting Test Cases by Using Data Mining ; Reducing the Cost of Testing”, International Conference on Computer Information Systems and Industrial Management Applications (CISIM), 2010. [22] Eduardo Aranha, Filipe de Almeida, Thiago Diniz, Vitor Fontes, Paulo Borba, “Automated Test Execution Effort Estimation Based on Functional Test Specifications”, Informatics Center – Federal University of Pernambuco (UFPE) Recife – PE – Brazil, last access Oct 2013. [23] Ashish Sharma, Dharmender Singh Kushwaha, ” A Empirical Approach for Early Estimation of Software Testing Effort using SRS document”,CIST, 2013. [24] Yogesh Singh, Arvinder Kaur, Ruchika Malhotra, ” Predicting Testing Effort using Artificial Neural Network”, Proceedings of the World Congress on Engineering and Computer Science, San Francisco, USA, 2008. [25] Benoit Baudry, Yves Le Traon, Gerson Sunyé, Jean-Marc Jézéquel, ” Measuring and Improving Design Patterns Testability”, In Proceedings of the Ninth International Software Metrics Symposium (METRICS’03), 2003.

Page 103: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 92

[26] Priya Chaudhary, C.S. Yadav, “An Approach for Calculating the Effort Needed on Testing Projects”, International Journal of Advanced Research in Computer Engineering & Technology, 2012. [27] Eduardo Aranha, Paulo Borba, “Empirical Studies of Test Execution Effort Estimation Based on Test Characteristics and Risk Factors”, last access Oct 2013. [28] Praveen Ranjan Srivastava, Abhishek Varshney, Priyanka Nama, ” Software Test Effort Estimation: a model based on cuckoo search”, Int. J. Bio-Inspired Computation, Vol. 4, No. 5, 2012. [29] Godwin Oziegbe, “A Framework for the Evaluation of Test Effort in Industrial Software Development”, Master thesis in Mälardalen University, Department of Innovation Design & Engineering, September 2011. [30] Rex Black, ”Test Estimation: Tools and Techniques for Realistic Predictions of Your Test Effort”, http://www.rbcs-us.com/documents/TestEstimation%28article%29.pdf, last access Nov 2013. [31] “Certified Tester Foundation Level Syllabus”, http://istqb.org/downloads/finish/16/133.html , last access Nov, 2013. [32] IEEE Computer Society Sponsored by the Software & Systems Engineering Standards Committee, “IEEE Standard for Software and System Test Documentation”, IEEE Std 829™-2008 (Revision of IEEE Std 829-1998). [33] Fara Andish Company website, www.faraandish.net, last access Dec, 2013. [34] Sigrid Eldh, ”On Test Design”, PhD thesis in Mälardalen University, Department Of Innovation Design & Engineering, September 2011. [35]http://santoqa.blogspot.com.ar/2013/08/test-estimations-issues-and-fermis.html?goback=.gde_99444_member_269802851#, last access Jan, 2014. [36] Paula Mian, Tayana Conte, Ana Natali, Jorge Biolchini and Guilherme Travassos, ” A Systematic Review Process for Software Engineering”, Brazil. [37] Dr. Rick Rabiser, ”Systematic Literature Reviews Introduction and Personal Experiences”, Euromicro SEAA, Austria, 2011. [38] Barbara Kitchenham, ”Procedures for Performing Systematic Reviews”, Software Engineering Group, Department of Computer Science, Keele University, Keele, Staffs, ST5 5BG, UK and Empirical Software Engineering National ICT Australia Ltd.Bay 15 Locomotive Workshop, Australian Technology Park, Garden Street, Eversleigh, NSW 1430, Australia, July, 2004. [39] Jorge Biolchini, Paula Gomes Mian, Ana Candida Cruz Natali, Guilherme Horta Travassos, ”Systematic Review in Software Engineering”, Systems Engineering and Computer Science Department COPPE / UFRJ, Rio de Janeiro, May 2005. [40] Barbara Kitchenham, ” Guidelines for Performing Systematic Literature Reviews in Software Engineering Version 2.3”, Software Engineering Group, School of Computer Science and Mathematics, Keele University, Keele, Staffs, ST5 5BG, UK and Department of Computer Science, University of Durham, Durham, UK, 9 July, 2007. [41] Kamala Ramasubramani Jayakumar, Alain Abran, “A Survey of Software Test stimation Techniques”, Journal of Software Engineering and Applications, 2013.

Page 104: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 93

Acronym

Meaning

APFD Average Percentage of Faults Detected AUCP Adjusted UCP BN Bayesian Network CBR Case-Based Ranking CF Conversion Factor CICM Cognitive Information Complexity Measure CMS Content Management System CNL Control Natural Language EFSM Extended Finite State Machine ERP Enterprise Resource Planning FATCP Fault Aware Test Case Prioritization FEP Total Fault Exposing Potential FP Function Points ID Importance Degree LOC Line Of Code NFR Non-Functional Requirement OS Order Score RF Rate of Fault detection SS Suspiciousness Score SUT System Under Test TC Test Case TEF Total Environment Factors UAW Unadjusted Actor Weights UCP Use Case Points UUCP Unadjusted UCP UUCW Unadjusted Use Case Weights

Table 52. Acronyms

Page 105: A Model for Estimating the Execution Cost of Test · PDF fileA Model for Estimating the Execution Cost of Test Cases ... 4.3.8 Software Requirement Specification (SRS) ... High Level

Seyed Morteza Hosseini – Master Thesis – 2015 94

10. Appendix A

Test Case Template

Test case ID:< > Date: <> Version: <> Author: <> Reviewed by:<> Used in: <X System>

System version: <>

Test environment: < > Test Suite:<> Automated <no>

Time for TC creation:<>

TDT used: <>

Assumptions: <>

Starting point: Test case: Pre-condition: Input Data (valid): Output/visible result for passed: (post-condition) Side-effect (clean-up) no side effect for this test case. Clean-up: Comment:

Test Record Template

TR for Test case ID: <>

Date: <> Executed Version: <> Tester: < >

System version:<> Test environment: < > Test Suite: <> Automated <no>

Result: <passed > Side-effect: Time for execution: <>

Failure ID nr: Failure report date: <> Reported to:< > Reported by: <>

Failure description: Other: Comments: