Using Process Enactment Data Analysis to Support Orthogonal Defect Classification for Software Process Improvement

Mehmet Söylemez, MSc.TUBITAK - BILGEM / Software Technologies

Research Institute (YTE), Ankara/Turkey mehmet.soylemez@tubitak.gov.tr

Ayça Tarhan, PhD. Computer Engineering Department

Hacettepe University, Ankara/Turkey atarhan@cs.hacettepe.edu.tr

Abstract—Defects occur in many software development projects. It is important to extract semantic information from defects to investigate their root causes and improve the process. In this study, enactment data of a software development process in which defects originated was used to support Orthogonal Defect Classification (ODC). In a sample project of a CMMI ML3 organization ODC was applied to thedefects and the utilization of the process enactment data was found to be effective and efficient in providing information about the root causes of the defects and deriving improvement actions. The defect attributes were analyzed and compared before and after applying suggested improvement actions. The comparison between the initial and the improved defect trigger and origin distributions showed that there was a positive change in the software development process of the project.

Keywords- orthogonal defect classification, software defect, root cause analysis, software process enactment, process data

I. INTRODUCTION

Software organizations use various tools to maintain records of software defects that emerge during the software development process in order to inform the developer about a specific defect, so they can make the necessary changes in the software. These records are of great value in terms of tracking defects, taking preventive actions and improving the process [1]. However, over time, the records accumulate and the volume of data makes it difficult to conduct analysis.

Semantic information obtained from defects is very important in the analysis of root causes. The briefer and more detailed this information is, the more effective and efficient the root cause analysis and the software process improvement. However, it is usually the definition, summary, priority and level of importance of defects that are recorded and not the detailed information that can contribute to the improvement of the software process [2]. Since the semantics extracted from defect records is not sufficient, problems may occur in the root cause analysis which makes it difficult to take actions that will prevent the occurrence of defects. The ideal situation is to perform quality assurance activities to identify defects and maintain records that contain the semantic information during the software development process as well as conducting regular analysis on these records and the process, and provide the developer with timely feedback. Thus, defects can be identified in the early stages of the software development process and the related cost can be reduced.

Orthogonal Defect Classification (ODC) is a defect analysis technique that provides developers with feedback during the development process and the opportunity to take preventive actions [3]. ODC is a bridge between the statistical defect model and defect root cause analysis. The statistical defect model supports a quantitative analysis using measures such as defect density and defect detection rates but, since this model does not include a causal analysis, it falls short in terms of taking actions to prevent the reoccurrence of defects [1]. Root cause analysis requires the evaluation of the process using the data collected during development for the purpose of improvement; however, this kind of analysis is not cost-effective in relation to the time spent and effort expended. The ODC technique eliminates these disadvantages by categorizing the defects into classes and reducing the cost of analysis [3, 4, 5, 6, 7, 8, 9].

This paper describes the analysis of defects of critical importance in product quality to improve the software development process of the TUBITAK - BILGEM / Software Technologies Research Institute (YTE). Qualitative data from the development process in which the defects originated was collected and evaluated to support the use of the ODC technique in root cause analysis. In the method followed for the implementation, ODC was applied to the selected defect sets as in other studies [3,4,5,6,7,8,9,10,11], and the results were evaluated. Unlike in other studies, the enactment data of the process from which the defects originated was collected through predefined forms and used in the root cause analysis. The process enactment data provides qualitative information concerning the course of the process from which the defects originated.

This article is organized as follows; the second section explains the underlying concepts of the study and summarizes the related studies; the third section describes the method followed for defect causal analysis; in the fourth and fifth sections, implementation of the method in a CMMI Maturity Level (ML) 3 software organization is explained, and the measured benefits and threats to validity of the implementation are discussed; finally, the conclusions are given in the sixth section.

II. BACKGROUND

Root cause analysis of defects aims to identify and conduct a detailed investigation on the basic points relating to the origin of defects using the semantic information extracted from the records. This analysis draws attention of the development team to the existing problems in the process

2013 Joint Conference of the 23nd International Workshop on Software Measurement (IWSM) and the Eighth International

Conference on Software Process and Product Measurement (Mensura)

978-0-7695-5078-7/13 $26.00 © 2013 IEEE

DOI 10.1109/IWSM-Mensura.2013.27

120

and develops awareness within the team to prevent the re-occurrence of a defect in a product. For an effective root cause analysis, the records should be well-kept and the defect classification system should have features that assist in accurate problem identification.

The best practices for software processes are defined by process reference models such as the Capability Maturity Model Integration Model (CMMI) [12], and the Causal Analysis and Resolution (CAR) process area in this model defines the necessary practices for causal analysis. Since root cause analysis is not cost-effective, it is not commonly used by software organizations. This has led to the development of techniques such as ODC [1] that reduces the cost of root cause analysis. The requirements of the CMMI CAR process area, the ODC technique and the reported implementation of this technique are discussed in the following sub-sections.

A. The CMMI Causal Analysis and Resolution (CAR) Process Area CMMI integrates many process reference models (e.g.

for development, acquisition, and services) that present the best practices to improve organizational processes. CAR is one of the process areas of CMMI at ML5 [12]. The purpose of CAR is to identify the causes of process outcomes and take actions to improve the process performance. Process outcomes are not only obtained from defects, but also from all positive or negative situations. CAR defines the causes of positive or negative outcomes in a process performance to prevent the re-occurrence of defects and increase quality and productivity [12].

Table I shows the specific goals and practices of CAR process area in which the selected positive or negative outcomes are analyzed to determine the causes, and the causes are addressed by implementing action proposals whose effects are then evaluated. In the case where the selected outcomes are defects, CAR recommends analyzing the root causes of these defects and proposing actions to prevent or reduce their re-occurrence.

TABLE I. SPECIFIC PRACTICES (SP) IN CMMI’S CAUSAL ANALYSIS AND RESOLUTION PROCESS AREA

Specific Goal 1 - Determine Causes of Selected OutcomesSP1.1 Select Outcomes for AnalysisSP1.2 Analyze Causes

Specific Goal 2 - Address Causes of Selected OutcomesSP2.1 Implement Action ProposalsSP2.2 Evaluate the Effect of Implemented ActionsSP2.3 Record Causal Analysis Data

B. Orthogonal Defect Classification (ODC) The ODC technique that was developed by Chillarege et

al. [1] in 1992 has attracted the interest of both industry and the academic world. The main purpose of ODC is to extract semantic information from defects in order to take actions against the re-occurrence of defects and improve the process. Thus, ODC can be considered as a method that can be used to implement the goals specified in the CAR process area.

The defect type and defect trigger are important attributes of the ODC technique. The defect types that can be used independently from a specific product are [1]: Interface, Function, Build / Package / Merge, Assignment, Document-

ation, Checking, Algorithm and Timing/Serialization. Defect types can be used for all defects that emerge in the various phases of the software development life cycle. The defect type is selected at the stage of correction by the person that repairs the defect. The root cause of defect types lies in the change made by the developer to repair the defect. For an effective analysis, it is crucial to select the correct defect type.

A defect trigger is an attribute that represents the condition that allows the identification of a software defect and is selected by the person recording the defect. While the identification of the defect type is crucial throughout the development process, the selection of the defect trigger provides information about the integrity of the validation process. Triggers are directly related to the activities that are the origin of the defects and they can be categorized into elements such as; requirement and design review, code inspection, unit test, functional test, and system test.

Defect attributes should be selected by the person who records and fixes the defect. To ensure that defect attributes are regularly gathered in a project, all people involved in the project should have the same understanding of the attributes thus, the defect attributes should be well-defined.

Following the classification of the defects using ODC, the distribution of defects by defect attributes (e.g. trigger, type, and origin) is obtained providing quantitative results from the semantic data. These quantitative results are evaluated by the development team, who may then take rapid preventive action. This eliminates the problems presented by root cause analysis, such as high cost and slow feedback to the developers.

C. Related Studies Bridge and Miller [3] applied ODC to the defects

identified in the software development process of the Motorola GSM Products Division's Base Station Systems group. They reported that the group analyzed defects using Fagan inspection data and the selected defect types were easily mapped into ODC defect types with only a few minor modifications. In a DVD player project developed by PHILIPS, Shenvi [4] used the ODC technique and suggested that guidelines for the prevention of defects can easily be determined with the help of defect type, activity and qualifier. Jalote and Agrawal [5] applied ODC in a project using the iterative software development model and proposed that the defect injection rate was considerably decreased. Identifying the defects in five different projects, Kumaresh and Baskaran [10] found the causes and presented measures to prevent their re-occurrence. Buglione and Abran [6] suggested that ODC overcomes the time disadvantage in the root cause analysis of defects and the CAR process area should be placed at ML2 in the CMMI. An experiment was conducted by Falessi and Cantone [11] to classify defects into defined categories and they found that orthogonal and effective classifications can be achieved by experience, and training is a precondition. Sudhan and Mishra [7] applied the ODC technique to the defects in Noida projects and observed that code rework and defect detection in the later phases of the software lifecycle were reduced, furthermore the quality

121

of deliverables improved. Focusing on the root causes of defects that are time consuming to fix, Kristiansen [18] applied ODC to the defects in a project and ascertained the following root causes; difficulty in locating the defect; time taken to clarify the defect; fixing the defect can cause new defects; and the developed function being new. Bassin and Santhanam [19] stated that ODC can be used efficiently to manage the maintenance of ported and outsourced software, and ODC offers an advantage in terms of providing rapid feedback in complex systems.

III. IMPROVEMENT METHOD

The improvement method that was used to analyze the root causes of the defects and drive improvement actions in a CMMI ML 3 organization is given in Figure 1 in Event-Driven Process Chain (EPC) notation. Different from existing studies, the improvement method makes use of process enactment data collection and analysis to support the ODC implementation in the root cause analysis. The process enactment data is collected by developers via the Process Attributes Forms (Figure 2) during the development.

The first stage is to choose the defect data set for the root cause analysis. This set may include; customer defects, system test defects, development defects, or the combination of these elements for one or more modules or projects. It is important that enactment data of the process in which these defects originated is collected and available. This data is then entered into a Process Verification Matrix [13] (see Appendix A) in order to clarify the consistency of process enactments against process attributes such as inputs, outputs, activities, infrastructure, communication channel, and staff.The inconsistencies detected in the attributes of process enactments are the candidate to originate software defects.

The flows upper side and lower side that succeed the activity of choosing defect data set for the root cause analysis

in Figure 1 indicate process enactment analysis and initial ODC implementation, respectively. The results obtained by both the flows are used to identify the root causes of the defects and represent them as a fishbone diagram. The causes on this diagram are then used to determine the opportunities for software process improvement.

IV. IMPLEMENTATION

In the projects implemented by TUBITAK – BILGEM / Software Technologies Research Institute (YTE), software testing is regularly undertaken in parallel to coding and the defects are systematically recorded; thus there are thousands of defect records. The YTE Project X has many versions and over time 5,200 defects have been recorded for five modules of the project (A, B, C, D and E). The defect records were kept in different CASE tools and no analysis was conducted on the root causes of the defects. Therefore, the results of this study are a valuable contribution to the improvement of the processes of the organization.

Project X uses an incremental and iterative software development lifecycle thus allowing the developers to add new attributes to the software at each increment and constantly improve the product through iterative cycles. The incremental approach also makes it possible to obtain rapid feedback from the early stages onwards.

The purpose of the analysis was to experience the root cause analysis of the defects to improve the development process in this CMMI ML3 certified organization. The primary research questions (RQs) were:

RQ-1. What is the distribution percent of software defects prior to performing the root cause analysis?

RQ-2. What is the distribution percent of software defects after performing the root cause analysis, identifying improvement actions, and implementing them?

Choose defect data set for root

cause analysis

The need to analyse the root

of the defects has arisen

Implement ODC and derive its

results

Evaluate ODC implementation

results

Gather and analyze enactment data of

the process that originated defects

Verify and evaluate process

enactment consistency

V

Determine root causes of defects

Determine process

improvementsuggestions

Root causes of defects identified

Defect data set for root cause

analysis

Defects grouped by type, trigger,

and origin

Fishbone diagram showing

defect root causes

Process Attributes Form

Process Verification

MatrixImprovement suggestion for

process context implemetation

input Activity output

EPC (Event-driven Process Chain) Notation

ODC implementation

evaluation results

Process enactment evaluation

results

V

activity flow

Figure 1. Improvement Method

122

Process Name Requirements Initial Analysis Phase Process Start Date: Process End Date:

Module / Component Name: Recorder: Record Date

PROCESS INPUTS No Input Name Y/N Detail 1 Requirement Determination Document 2 Customer Needs 3 Risk Management Process, Customer Requirements 4 Customer Requirements

PROCESS OUTPUTS No Output Name Y/N? Detail 1 Customer Needs 2 Customer Requirements 3 Risks 4 Operational Concepts and Scenario

PROCESS ACTIVITIES No Activity Name Y/N? Detail 1 Customer Needs Gathered 2 Customer Requirements Specified 3 Risks Specified 4 Operational Concepts and Scenario Specified

INFRASTRUCTURE No Question Y/N? Detail 1 Is there a problem with the office working area? 2 Is there a problem with the infrastructure used for development? 3 Is there a decrease in the resources and time reserved for process?

COMMUNICATION CHANNEL No Question Y/N? Detail 4 Is there a problem with the project manager? 5 Is there a problem with the customer? 6 Is there a problem with the other team members? 7 Is there a problem with the domain experts?

STAFF No Question Y/N? Detail 8 Have the practitioners received training for their roles in the process? 9 Have the practitioners had (sufficient) prior experience in order to carry out their roles in the process? 10 Did process practitioners need to work overtime?

Figure 2. An Example Process Attributes Form (for a related Process Verification Matrix see Appendix A)

We structured the analyses as a multiple-case and with a holistic design [14]. We consider that the holistic view will allow us to understand the performance of the development process as a whole by analyzing the distribution of the defects. Since we aimed to compare the performances of the development process before and after improving the process,two units of analyses within the study were constructed for these two states. While structuring the design, we also considered the existence of defect data and process enactment data, and the accessibility of process performers.

In order to respond to RQ-1, the high priority defects that were recorded in the development and maintenance stages of Module B and C in versions 1 to 6 of the project were addressed. In each version, new components were developed in addition to the maintenance of existing components. For each version, 50 defects were selected randomly and a total of 300 defects were individually reviewed by the first author who was also a developer of the modules B and C. The determination and recording of the defect attributes took approximately five minutes per defect. The assignment of these attributes was then reviewed by the three experienced developers of the project team in a total of six hours. In this initial implementation, the ODC technique was applied to the selected defects and the results of the implementation and process enactment data where the defects occurred were evaluated together to identify the root causes of the defects

[15]. In order to eliminate the root causes of the defects, suggestions were made and adopted to develop versions 7 to 12 of the software product.

In order to answer RQ-2, the ODC technique was applied to Module D which was developed in the versions 7 to 12.Following a briefing on the attributes of the defects by the first author, the developers entered the attributes of the type, trigger and origin in the defect lifecycle into the Defect Tracking Tool. From Module D that was developed through the improved process, 182 high priority defects were selected and analyzed using the ODC technique. The results of the analysis were compared with those of the previous implementation. The initial and improved results in the defect origin distribution and defect trigger distribution are shown in Figures 3 and 4, respectively.

As can be seen in Figure 3, the Requirement and Design defects were reduced the main factor being the review of the Requirement and Design (Figure 4). Failing to detect the Requirement and Design defects may cause defects that are costly to fix at a later date. Furthermore, detection in the early phases of the process is necessary to avoid a possible increase in the vulnerability of the software. The percentage of coding defects is higher than other defects. This type of defect is usually due to an oversight on the developer’s part and can be easily fixed by improving the unit tests and code reviews.

123

Figure 3. Defect Origin Distribution (Initial & Improved)

Figure 4. Defect Trigger Distribution (Initial & Improved)

As shown in Figure 4, the Requirement and Design reviews assisted in the detection of defects originated from the requirement and design phases. This type of defect could be prevented by carrying out particular activities prior to the system test phase. By detecting defects in the earlier phases, there are fewer defects in the system test phase. The reduced percent of defects in customer requests confirms the benefit of using defect triggers throughout the process.

V. THREATS TO VALIDITY

Improvement suggestions made in accordance with the results of the study were evaluated by five people consisting of project team leaders and members of the quality team. The following ordinal scale was used for the evaluation; 1: Strongly disagree, 2: Disagree, 3: Neutral, 4: Agree, 5: Strongly agree. Since the number of people participating in the evaluation was low, the average of the responses not the median was calculated. Table-II shows the improvement suggestions and the average values obtained from the evaluation of the suggestions. These suggestions, except the ones written in italics in the table, were applied to improve the project’s development process.

TABLE II. IMPROVEMENT SUGGESTIONS AND THEIR EVALUATION

The internal validity of the study is supported by the following factors in the initial and the improved implementations: the selected modules were from the same project; the team members and the development infrastructure were the same; and, the project’s process had been defined. However, the complexity of the software versions and level of the team’s experience in the developments, and the identification of defect attributes in the initial ODC implementation by the first author and by the developers in the second implementation are the threats to the internal validity of the study.

The threat to the external validity is that the study was carried out on a defect set selected from a single project of a single organization for a specific purpose. Further studies are needed to investigate the effectiveness of the improvement method in different contexts and to confirm the external validity of the results found in this research.

VI. CONCLUSIONS

In this study, the ODC technique was used to improve the software development process, and the root causes of the defects were identified from the evaluation of the results and the process enactment data that showed the origin of the defects. Improvement suggestions were made to eliminate the root causes and applied to a new development stage. The results showed that the distance between the injection and detection of defects was reduced; i.e., defects could be detected before the last phase of the lifecycle thus, reducing the development costs.

The use of the ODC technique for root cause analysis was verified and the effort requirement for identifying the root causes of the defects and deriving process improvement suggestions was observed to reduce by the use of the process enactment data.

In the initial defect analysis, the first author spent five minutes per defect to identify the attributes for the selected 300 defects, resulting in a total effort of 25 person-hours. The assigned attributes were then reviewed by three

Improvement Suggestion Avg.

Determine risks in the requirement engineering process, 4.4

Frequently communicate with the customer while determining interface requirements,

4.4

Reqularly validate requirements with the customer, 4.8

Reqularly review software requirements, 4.4

Plan handling of urgent requirements changes in the main releases rather than in the sub-releases,

4.0

Allocate sufficient time to decide on the best design based on many alternatives,

4.4

Model the selected design in detail, 4.4

Regularly review the detailed design, 4.0

Hold code reviews regularly, 4.8

Increase the efficiency and coverage of unit tests, 4.8

Improve the integration of SQL scripts into the development environment, and increase the efficiency of unit tests for database inquiries.

4.4

124

experienced developers in six hours, leading to an additional effort of 18 person-hours. The developers completed 17 Process Attribute Forms to collect process enactment data during the development, and they claimed that the completion of a form took about three minutes. Thus the total effort spent for the collection of the process enactment data was 51 person-minutes which is about a person-hour. Also, it took four person-hours to derive the root causes of the defects from the collected data and determine the process improvement suggestions. Therefore, the overall effort spent for the initial analysis was 48 person-hours.

In the defect analysis performed after applying the improvement suggestions, it took one person-hour to derive the distributions since the defect attributes were already entered by the developers.

In this study, we experienced that the utilization of the process enactment data in the defect causal analysis was supportive and cost-effective in the ODC implementation.However, although the collection of process enactment data required only little effort, convincing the developers to do so was not easy at the start of the study. It was only after the results of the study were shared with the developers that they could see more clearly the importance of recording the process enactment data. We find the management’s role and the feedback from the process improvement crucial here. In addition, a software application or add-on that enables the automatic collection of the enactment data for each phase of the process (such as; analysis, design and coding) can make the data collection easier.

REFERENCES

[1] R. Chillarege, I.S. Bhandari, J.K. Chaar, M.J. Halliday, D.S. Moebus, B.K. Ray, M.Y. Wong, “Orthogonal defect classification–a concept for in-process measurements”, IEEE Trans. Softw. Eng., vol. 18, 1992, pp. 943–956.

[2] M.R. Lyu, “Handbook of software reliability engineering”, Hight-stown, NJ, USA: McGraw-Hill, Inc., 1996.

[3] N. Bridge, C. Miller, “Orthogonal defect classification using defect data to improve software development”, Software Quality, vol.3, 1998, pp.1997–8.

[4] A.A. Shenvi, “Defect prevention with orthogonal defect classification”, In proceedings of the 2nd India software engineering conference, ser. ISEC’09. NewYork, NY, USA: ACM, 2009, pp. 83–88.

[5] P. Jalote, N. Agrawal, “Using defect analysis feedback for improving quality and productivity in iterative software development”, In proceedings of the 3rd ITI Conference, 2005, pp. 703 – 713.

[6] L. Buglione, A. Abran, “Introducing root-cause analysis and orthogonal defect classification at lower CMMI maturity levels”, 2008.

[7] K.H. Sudhan, U.K. Mishra, “Orthogonal Defects Classification of observed Defects in C-DAC Noida Projects”, Proceedings of ASCNT-2011, CDAC, Noida, India.

[8] J.M.W. Kristiansen, “Software Defect Analysis, Master Thesis”,NTNU Computer Science Department. Norway, 2009.

[9] K. Bassin, P. Santhanam, “Managing the Maintenance of Ported, Outsourced, and Legacy Software via Orthogonal Defect Classification”. In Proceedings of the IEEE International Conference

on Software Maintenance (ICSM'01) (ICSM '01). IEEE Computer Society, Washington, DC, USA, 726-.DOI=10.1109/ICSM.2001.972791 http://dx.doi.org/10.1109/ICSM.2001.972791, 2001.

[10] S. Kumaresh, R. Baskaran, “Defect analysis and prevention for software process quality improvement”, International Journal of Computer Applications, vol.8, no.7, 2010, pp.42–47, published By Foundation of Computer Science.

[11] D. Falessi, G. Cantone, “Exploring feasibility of software defects orthogonal classification”, International Conference on Software and Data Tecnologies (ICSOFT2006), M. H. Joaquim Filipe, Boris Shishkov, Ed. INSTICC Press, 2006.

[12] CMU/SEI, 2010, CMMI for Development V1.3. [13] A. Tarhan, O. Demirörs, “Apply Quantitative Management Now”,

IEEE Software, Vol.29, Issue: 3, May/June 2012, pp.77-85,. [14] R. K. Yin, “Case Study Research, Design and Methods”, 4th ed.

Newbury Park, Sage Publications, 2002. [15] M. Söylemez, A. Tarhan, A. Dikici, “An Investigation of Root

Causes of Software Defects by using Orthagonal Defect Classification”, in Proceedings of the 6th National Software Engineering Symposium, May 2012, pp. 77-84, , Ankara (in Turkish).

APPENDIX-A. AN EXAMPLE PROCESS VERIFICATION MATRIX

Process Enactments Process Attributes for Req.s Initial Analysis Phase PE1 PE2 PE3 ...

1 Inputs 1.1 Requirement Determination Document o 1.2 Customer Needs o o o o

1.3 Risk Management Process, Customer Requirements

1.4 Customer Requirements o o o 2 Outputs 2.1 Customer Needs o o o o 2.2 Customer Requirements o o o o 2.3 Risks 2.4 Operational Concepts and Scenario o o o o 3 Activities 3.1 Customer Needs Gathered o o o o 3.2 Customer Requirements Specified o o o o 3.3 Risks Specified o o 3.4 Operational Concepts and Scenario Specified o o o o 4 Infrastructure

4.1 Is there a problem with the office working area?

4.2 Is there a problem with the infrastructure used for development?

4.3 Is there a decrease in resources and time reserved for process?

5 Communicat-ion Channel

5.1 Is there a problem with the project manager? 5.2 Is there a problem with the customer? o

5.3 Is there a problem with the other team members?

5.4 Is there a problem with the domain experts? 6 Staff

6.1 Have the practitioners received training for their roles in the process? o o o

6.2 Have the practitioners had (sufficient) prior experience in order to carry out their roles in the process?

o o

6.3 Did process practitioners need to work overtime? o o

125