Defect prediction using social network analysis on issue repositories

Reporter: Dandan WangDate: 04/18/2011

Basic information

• Conference: ICSSP 2011• Authors – Serdar Bicer• Gerger consulting, Istanbul, Turkey

– Ayse Bsar Bener• Ryerson university, Ted Rogers School of information

Technology Management, Toronto, Canada

– Bora Caglayan• Bogazici university Department of Computer

Engineering, Istanbul, Turkey

Outline

1

•Introduction

2

•Methodology

3

•Results

4

•Conclusion

Introduction

• Objective – Overcome ceiling effects of defect predictors.

• Research question– What is the benefit of social network metrics on issue

repositories to predict defects?• Metrics – Social network metrics – Churn metrics

• Method– Naive Bayes (Learning based prediction model)

Outline

1

•Introduction

2

•Methodology

3

•Results

4

•Conclusion

Methodology

• Dataset • Communication structure in projects• Metrics used • Defect prediction model• Performance measures

Dataset • RTC

– Year: 2007 and 2008.– Team: Large distributed team and used the Jazz platform– Version control system, issue repository

• Drupal– Year: 2009-2010– Team : Large distributed team – Public CVS repository, issue repository(bug reports, feature requests, and

other tasks)

Data extraction process for datasets

• Nodes in graphs represents developers who commented on each file.

• Files were labeled as defective if they were modified after snapshot date.

Communication structure in projects

• RTC and Drupal projects are similar to each other in communication structure.

• Commenting on issues is the main task-related communication used by contributors in both projects. If a commit in version control system is related with an issue, issue number is written to commit message.

• Jazz framework automatically creates a connection from issue to change set, which is not available in Drupal.

• The issues are assigned to and owned by contributors. • Other project members express their opinions by commenting

on issues.

Metrics used

• While first 6 metrics were used in previous studies [22, 33, 44, 42],• Diameter, Clustering Coefficient, Bridge Rate, and Characteristic Path

Length are new metrics

Defect prediction model

• Metrics – Social network metrics on issue repositories

• Algorithm– Naive Bayes data mining algorithm

• Validation – 10*10-fold cross validation to eliminate sampling

bias– Cost-benefit analysis (Weka software)

Performance measures

• Widely used performance measures– Probability of detection(pd)– Probability of false alarms(pf)

Higher balances are better because their points (pd, pf) are closer to the ideal point (1, 0)

Cost-benefit analysis

Cost curve

• Cost curve is proposed by Drummond and Holte to supply the deficiencies of ROC curves. It is a visualization technique that shows classifier’s performance based on the cost of misclassification.– X: PC(+). Probability of positive class, combination of the two

misclassification costs and the class distribution into a single value.– Y: NEC. Normalized expected cost which denotes error rate.

Outline

1

•Objective

2

•Methodology

3

•Results

4

•Conclusion

Results

• Prediction performance analysis

• T-test analysis: statistically significantly• Cost-benefit analysis

Cost curves for datasets

Beneficial outcomes • Our proposed model either considerably decreases high false alarm rates without

compromising the detection rates or considerably increases low prediction rates without compromising low false alarm rates compared to churn metrics. In both cases this results in increase of overall prediction performance. Consequently, this leads to decrease in verification costs compared to churn metrics. Thus we recommend practitioners to collect social network metrics on issue repositories.

• We can interpret this result as structure of information flow in a developer communication network has significant effect on code quality. Since our metrics are directly related with network’s topology, this model can help managers to build developer networks more efficiently.

• We used only a recent part of developer communication history to construct our model. Communication between project members begins at the start and continues until the end of the project. But in this study, we did not collect full communication history. This is important for software teams which have begun to keep record of developer communication after the beginning of the project because our proposed model can also be used for these kind of projects.

Outline

1

•Objective

2

•Methodology

3

•Results

4

•Conclusion

Conclusion

• Reason: communication and coordination between developers is important but patterns of interaction between developers have not been investigated for defect prediction.

• Main contribution of this study is using new data source and metrics in the area of defect prediction.

• Performance analysis– Churn metrics, social network metrics– Pd,pf, balance

• Cost-benefit analysis. – Social network metrics on issue repositories reduced costs required for

verification of prediction results and made results closer to cost-adverse region of ROC curve.

Thank you!

Q&A