1

Are Change Metrics Good Predictors for an Evolving Software Product Line?

Sandeep Krishnan, ISUChris Strasburg, ISU & Ames Laboratory

Robyn R. Lutz, ISU & JPL, California Institute of Technology

Katerina Goseva-Popstojanova, WVU

This research is supported by NSF grants 0916275 and 0916284

Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

2

BackgroundProduct line – “A family of products designed to take advantage of

their common aspects and predicted variabilities” [Weiss and Lai 1999]e.g., Nokia cellphones, HP printers, etc.

Products -

Commonalities – Shared by all products. e.g., Platform

Variabilities – Differentiate the products High-reuse variation

JDT, PDE, Mylyn, Webtools, etc. Reused in more than three products and for more than six years.

Low-reuse variation CDT, Datatools, Java EE tools. Reused in three or fewer products and for more than four years.

Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

3

Related WorkEclipse as a product line. [Chastek, McGregor, and Northrop,

2007], [Linden, 2009], [Krishnan et al., 2011].

Summary of previous work

Authors Eclipse releases

Metrics Results

Zimmerman, Premraj, Zeller

Eclipse 2.0(2002), 2.1(2003), 3.0(2004)

Code complexity metrics

Classification at package level is more accurate than at file level

Moser, Pedrycz, Succi

Eclipse 2.0(2002), 2.1(2003), 3.0(2004)

Code change metrics

J48 learner with change metrics gives better classification at file level than code metricsFailure-prone file - A file with one or more non-trivial

post-release

bugs recorded in the Eclipse Bugzilla database.

Important/Good predictor – Predictor providing high information gain for classification of failure-prone filesDept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

4

Product Line EvolutionProduct line evolution in two dimensions

P1

R1

P1

R2

P1

R3

P1

Rn

P2

R1

P2

R2

P2

R3

P2

Rn

Pn

R1

Pn

R2

Pn

R3

Pn

Rn

New Releases

New Product

s

Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

5

MotivationCan we leverage the reducing amount of change in product lines to better predict failure-prone files?

Time

Change

Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

6

Eclipse case study

Eclipse ClassicEclipse Java

Eclipse JavaEE

Eclipse C/C++

Eclipse products

Change data collected for 6 months before and after each release.

Failure categories

Blocker

Critical

Major

Normal

Minor

7

Research Questions

As a product evolves, do any change metrics serve as good predictors of failure-prone files?

Is there a subset of change metrics which are good predictors across all product line members?

Does our ability to predict failure-prone files improve as product line evolves?

Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

8

FindingsThe change metrics provide good classification of the

failure-prone files in the Eclipse product line.

As each product evolves, there is a stable set of change metrics that are prominent predictors of failure-prone files across its releases.

There is a subset of change metrics that is among the prominent predictors of all the products across most of the releases.

As the product line matures, prediction performance improves for each of the four Eclipse products.

Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

9

Data Source

Source of failure reports-

Source of change reports – CVS repository of Eclipse.

Data Timeline

Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

Replication ExtensionData Timeline

10

Approach

Weka J48

decision tree

learner

Get prediction results +

best predictors

Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

11

Replication Results

Release Moser et al. study This study

PC TPR FPR PC TPR FPR

2.0 82 69 11 88 55 5

2.1 83 60 10 85 63 9

3.0 80 65 13 84 62 9

Learner performance compared to previous results

Classification performance comparison for Eclipse Classic 2.0, 2.1, and 3.0

PC- Percentage of correctly classified instances

TPR- True positive rate

FPR- False positive rate

Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

12

Top predictors from this study Revisions, Weighted_age

Top predictors from previous study Max_changeset, Bugfixes, Revisions

Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

Replication Results

Release Top five predictors from Moser study

Top five predictors from this study

2.0 Max_Changeset, Revisions, Age, Bugfixes, Refactorings

Revisions, Weighted_Age, Ave_Changeset, Bugfixes, Max_Loc_Added

2.1 Bugfixes, Max_Changeset, Revisions, Max_Added, Max_Loc_Deleted

Revisions, CodeChurn, Age, Weighted_Age,Loc_Deleted

3.0 Revisions, Max_Changeset, Bugfixes, Age, Ave_Loc_Added

Revisions, Authors, Weighted_Age, CodeChurn, Age

Top five predictors for earlier releases of Eclipse Classic

13

Learner performance improves as single product evolves

Extension Results

Release PC TPR FPR

2.0 88 55 5

2.1 85 63 9

3.0 84 62 9

3.3 (Europa) 93 79 4

3.4 (Ganymede)

94 81 3

3.5 (Galileo) 97 86 2

3.6 (Helios) 97 85 2

Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

14

Revisions is good predictor for later releases also.Max_changeset is a good predictor also.

Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

Extension Results

Top five predictors for later releases of Eclipse Classic

Release Top five predictors

3.3 (Europa) Ave_Loc_Added, Revisions, Authors, Max_Changeset, Weighted_Age

3.4 (Ganymede)

Revisions, Age, Ave_Changeset, Max_Loc_Added, Max_Changeset

3.5 (Galileo) Loc_Added, Authors, Max_Changeset, Revisions, Weighted_Age

3.6 (Helios) Authors, Revisions, Max_Changeset, Bugfixes, CodeChurn

15

Learner performance improves as product line evolves

Percentage of correctly classified instances increases across releases for each product

Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

Extension Results

16

Extension Results

Percentage of true positives shows improvement across releases for each product

Learner performance improves as product line evolves

Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

17

Percentage of false positives shows reduces across releases for each product

Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

Extension ResultsLearner performance improves as product line

evolves

18

No common set of predictors across each product and each release.

Max_changeset, Revisions and Authors are prominent predictors for all products.

Some predictors are prominent for only one product.

Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

Extension Results

Product

Java JavaEE C/C++ Classic

Max_Changeset: 4, Revisions : 3, Authors: 3, Code Churn: 3

Authors: 4,Max_Changeset: 4, Revisions: 3 Bugfixes: 3

Revisions: 4,Authors: 3,Max_Changeset: 3Age: 3

Revisions: 4,Max_Changeset: 4,Authors: 3

Top five predictors for four products of Eclipse Product Line

19

Research Questions Revisited

• There is a small set of change metrics which serve as good

predictors

RQ1: As a product evolves, do any change metrics serve as good predictors of failure-

prone files?

• There is no subset of metrics common to all products and all

releases.• There are some differences in the set

of good predictors between products

RQ2: Is there a subset of change metrics

which are good predictors across all

product line members?

• As the Eclipse product line evolves, the predictions of failure-prone files

improve

RQ3:Does our ability to predict failure-prone

files improve as product line evolves?

Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

20

Thank You!

Our data is available at http://www.cs.iastate.edu/~lss/PROMISE11Data.tar.gz

Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011