Promise 2011: "Customization Support for CBR-Based Defect Prediction"

CUSTOMIZATION SUPPORT FOR CBR-BASED DEFECT PREDICTION

Elham Paikari

Department of Electrical and Computer EngineeringUniversity of Calgary2500 University Drive, NWCalgary, AB, [email protected]

Bo Sun

Department of Computer ScienceUniversity of Calgary2500 University Drive, NWCalgary, AB, [email protected]

Guenther Ruhe

Department of Computer Science & Department of Electrical and Computer EngineeringUniversity of Calgary2500 University Drive, NWCalgary, AB, [email protected]

Emadoddin Livani

Department of Electrical and Computer EngineeringUniversity of Calgary2500 University Drive, NWCalgary, AB, [email protected]

2

Agenda

• Parameters of a CBR Model

• Parameters Instantiation

• Weighting Method SANN

• Frequency Analysis

• Dependency Network and the Customization Support

• Rules Transferability

• Conclusions and Future Work

3

CBR and the Parameters

Prediction Performance of

CBR model

Similarity Function

Number of Nearest

Neighbor Case

Weighting Technique used for

Attributes

Solution Algorithm

Each combination of the parameters :Instantiation of the general CBR-based prediction method

4

Instantiation Parameters of the CBR

Similarity (distance) function

Attribute weighting technique

Number of similar cases used for the prediction

Solution algorithm (how the new case is predicted from the known old ones)

Accuracy measurement of prediction technique

MLR

SANN

5

Sensitivity Analysis Based On NeuralNetwork (SANN)

A1 A2 … An

Mean(A1) Mean(A2) …. Mean(An)

Min(A1) Mean(A2) …. Mean(An)

Max(A1) Mean(A2) …. Mean(An)

Xmin(A1) NN OUTPUTmin(

A1)

Xmax (A1) NN OUTPUTmax

(A1)

∆1= |OUTPUTmin(A1) - OUTPUTmax(A1)|

Dataset

LOC CC ……………

6

AAE ARE AAE ARE AAE ARE AAE ARE AAE AREK=1 K=2 K=3 K=4 K=5

-0.05

0

0.05

0.1

0.15

0.2

Comparing SANN with MLR attribute selection (AAE and ARE)

CM1 KC3 PC1 PC3 PC4

AAE ARE AAE ARE AAE ARE AAE ARE AAE AREK=1 K=2 K=3 K=4 K=5

-0.02

0

0.02

0.04

0.06

0.08

Comparing SANN with un-weighted attribute weighting (AAE and ARE)

CM1 KC3 PC1 PC3 PC4

How different numbers of the nearest neighbors can affect the results?

What is the evaluation result in comparison with existing methods (un-weighted)

What is the evaluation result in comparison with existing methods (MLR)

7

Instantiation Parameters of the CBR

Similarity (distance) function

Attribute weighting technique

Number of similar cases used for the prediction

Solution algorithm (how the new case is predicted from the known old ones)

Accuracy measurement of prediction technique

MLR

SANN

8

Data Repository• PROMISE Repository

• 120 different CBR instantiations were created

and applied to 11 data sets from PROMISE repository

• Characterization of data setsData Set # Modules %Defective Code LOC

CM1 505 9.5 C 17KKC3 458 9.39 Java 8KMC1 9466 0.71 C++ 66KMW1 403 7.69 C 8KPC1 1107 6.87 C 26KPC2 5589 0.41 C 27KMC2 161 32.3 C++ 6KPC3 1563 10.24 C 36KPC4 1458 12.20 C 30KAR3 63 12.25 C 6KAR5 36 22.22 C 2K

9

Is One Instantiation Always the Best?

0 0.2 0.4 0.6 0.8 1 1.20

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

MMRE

Pre

d0.1 0.120.140.160.18 0.2 0.220.240.26

0.65

0.7

0.75

0.8

0.85

0.9

MMRE

Pre

d

0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.180.78

0.8

0.82

0.84

0.86

0.88

0.9

0.92

0.94

MMRE

Pre

d

0 0.05 0.1 0.15 0.20.75

0.8

0.85

0.9

0.95

1

MMRE

Pre

d

MW1

CM1

PC1 MC2

KC3 AR5

0.1 0.2 0.3 0.4 0.5 0.6 0.70

0.10.20.30.40.50.60.70.80.9

MMREP

red

0.06

0.08 0.

10.

120.

140.

160.

18 0.2

0.22

0.24

0.7

0.75

0.8

0.85

0.9

MMRE

Pre

d

10

Experimental Design for Frequency Analysis

Dataset

Min(MMRE)

120 differentinstantiation

MMRE Pred(0.25)

1

2

3

120

11differentDatasets

Max(Pred(0.25))

MMRE

1

2

131

Pred(0.25)

1

2

159

Dataset

Min(MMRE)


MMRE Pred(0.25)

1

2

v

120

Max(Pred(0.25))

Dataset

Min(MMRE)


MMRE Pred(0.25)

1

2

v

120

Max(Pred(0.25))

11

Frequency Analysis• Frequency of the best performance in single attribute

analysis • Neural network based sensitivity analysis (as the weighting

technique)• Un-weighted average (as the solution algorithm)• Maximum number of nearest neighbors (as the number of

nearest neighbors)

12

Similarity Function

FunctionFrequency

MMRE Pred(0.25)Manhattan 68 79Euclidean 63 80

Total 131 159

Weighting Technique

TechniqueFrequency

MMRE Pred(0.25)Uniform 41 47

MLR 35 52SANN 55 60Total 131 159

Solution Algorithm

AlgorithmFrequency

MMRE Pred(0.25)Un-weighted Average 72 88

Rank-weighted Average 59 71Total 131 159

Number of Nearest Neighbors

FrequencyMMRE Pred(0.25)

1 5 162 3 63 6 84 12 125 13 15

10 18 1915 16 1720 19 1625 19 2230 20 28

Total 131 159

Rule Induction Customization Support

Customization Support Using DNA• Dataset

• Eight attributes defined as condition attributes• Four data set-related attributes:

(NumOfModule),(DefectRatio),(Language),(LOC)• Four CBR-related attributes:

(SimFunc),(WeightingTech),(NumOfNN),(SolutionAlgorithm)

• The decision attributes: Pred(0.25) and MMRE

13

Data set

DNA

Rule Set

CBR model instantiated by

(p1,p2,p3,p4)

New data(a1,a2,a3,a4)

Recommendation f (a1,a2,a3,a4)

(a1,a2,a3,a4)

(p1,p2,p3,p4)

14

Given:NumOfModule = HighDefectRatio = HighLOC = MediumLanguage = JAVA

Question: How to customize a CBR defect prediction model towards achieving high prediction accuracy measured in MMRE?

Justification: Based on the data set characteristics, assumptions of rules 3, 4, 5, 11 and 12 are fulfilled. By comparing the probability distributions of MMRE rule No. 11 is the best in terms of having the highest probability (69.2%) to achieve “Low” MMRE.

Recommendation: Customize CBR model by means of:WeightingTech = SANNNumOfNN ≥ 10SolutionAlgorithm = Rank-weighted Average

Application of DNA Results Generation of the Decision Trees

15

Transferability of Rules across Sets of DataData Set # Modules %Defective Code LOC

CM1 505 9.5 C 17KKC3 458 9.39 Java 8KMC1 9466 0.71 C++ 66KMW1 403 7.69 C 8KPC1 1107 6.87 C 26KPC2 5589 0.41 C 27KMC2 161 32.3 C++ 6KPC3 1563 10.24 C 36KPC4 1458 12.20 C 30KAR3 63 12.25 C 6KAR5 36 22.22 C 2K

9 differentDatasets

MMRE

1

2

131

Pred(0.25)

1

2

159

Dataset

Min(MMRE)


MMRE Pred(0.25)

1

2

v

120

Max(Pred(0.25))

Dataset

Min(MMRE)


MMRE Pred(0.25)

1

2

v

120

Max(Pred(0.25))

Rule Induction

Data set

DNA

Rule Set

Customization Support

CBR model instantiated by

(p1,p2,p3,p4)

New data

(a1,a2,a3,a4)Recommendation

f (a1,a2,a3,a4)

•Prediction on Pred(0.25) = 48.33%• Prediction on MMRE = 62.92% •The per-class (High, Medium and Low) are calculated based on the confusion matrix…

16

Validation and Limitations

• Tools used for attribute selection, and modeling tasks• Neural network, regression analysis, CBR, and dependency network

analysis

• Only four parameters of the CBR instantiation• The composition of the training and testing data sets• Another aspect of the analysis undertaken is the definition

of classification intervals for dependency networks, • Two discretization algorithms• Sensitivity analysis

17

Conclusions and Future Work• Starting with 11 data sets from the PROMISE repository

• Calculating the prediction performance of 120 instantiations of the CBR-based

defect prediction model based on the value of the MMRE and Pred(0.25)

• The frequency analysis on the top performances

• Generating the DNA to provide a customization support for a new data set

• The compatibility of rule sets extracted from different contexts

• Enhancement of the validity with inclusion of further data sets

• Comparing the performance against other measures

• Other methods for rule induction

18

References

• Brady, A. and Menzies, T. 2010. Case-based reasoning vs parametric models for software quality optimization. In Proceedings

of the 6thInternational Conference on Predictive Models in Software Engineering, pp. 3:1-3:10.

• Catal, C. and Diri, B. 2009. A systematic review of software fault prediction studies. Expert Systems with Applications, vol. 36

(4), pp. 7346-7354.

• El Emam, K., Benlarbi, S., Goel, N., and Rai, S. N. 2001. Comparing case-based reasoning classifiers for predicting high risk

software components. The Journal of Systems and Software, vol. 55, pp. 301-320.

• Foss, T., Stensrud, E., Kitchenham, B., and Myrtveit , I. 2003. A simulation study of the model evaluation criterion MMRE.

IEEE Transactions on Software Engineering, vol. 29 (11), pp. 985- 995.

• Ganesan, K., Khoshgoftaar, T. M., and Allen, E. B. 2000. Case-based software quality prediction. International Journal of

Software Engineering and Knowledge Engineering, vol. 10(2), pp. 139–152.

• Paikari, E., Richter, M. M., and Ruhe, G. 2010. A comparative study of attribute weighting techniques for software defect

prediction using case-based reasoning. In Proceeding of the 22nd International Conference on Software Engineering and

Knowledge Engineering, pp. 380-386.

19

Thanks

Elham [email protected]

Promise 2011: "Customization Support for CBR-Based Defect Prediction"

Technology

Transcript of Promise 2011: "Customization Support for CBR-Based Defect Prediction"