Estimating Development Effort in FOSS Projects by Mining Software Repositories

- Openstack Survey Results

Gregorio Robles, Universidad Rey Juan CarlosCarlos Cervigón-Ávila, Universidad Rey Juan Carlos

Andrea Capiluppi, Brunel University

Executive Summary

● We propose a model to quantify the effort invested in a FOSS project by obtaining data from the versioning system.

● Our model is based on identifying full-time developers. To every full-time developer, we assign 1 person-month of effort each month.

● A survey with OpenStack developers has been used to validate our results. Over 100 OpenStack developers participated in the survey.

● Our analysis shows that the minimum estimation error is when we consider those OpenStack developers with over 12 commits every 6 months as full-time.

● A total of 9,000 person-months (750 person-years) for the development of the OpenStack project since its beginnings has been estimated.

● ~2,650 person-months have been invested during the last six months of 2013. This implies that we estimate that the actual effort on OpenStack has been around 440 person-months during the last 6 months.

2

About The Model

● Activity of a developer → effort measured in person-month

● Full-time developers → 40 hours/week → 1 person-month of effort each month

● We can identify with high precision those developers working full-time in a FOSS project

● Assumptions:● The number of contributions by full-time developers is a large

share of the total number of contributions

● We have used two possible measures for each developer:

● The model is based on finding a threshold value “t” for the number of commits (or active days) for which we indentify full-time developers with a minimum error

● The estimation error made for full-time developers is much lower than the error made for the rest of developers

● Commits/month

● Active days/month

3

About OpenStack

First release October 2010

# Authors (distinct) 1410

# Commits 89,871

# Commits (without bots) 68,587

SLoC (approx.) 1,650,000

➔ Young project with over 200 companies involved.

➔ 80% of the commits have been done by less than 8% of the overall authors.

➔ 90% of the commits have been done by about 17% of the overall authors.

➔ The OpenStack Fundation estimates that around 250 developers work professionally in the project.

4

Survey Results

● 1407 personalized emails were sent:

➔ About 300 emails bounced

➔ 131 developers replied the survey

➔ 5 answers were removed because they were empty or we could see from the answer in the free text box that the respondent had misunderstood/misinterpreted the questions

➔ We also changed 7 surveys to normalize the set, mainly because some respondents devoted 40 or more hours a week to the project mostly in programming, but stated they were occasional/part-time developers

➔ To analyze if the survey is representative of the project we have used the Wilcoxon signed-rank test. The result is that the survey in fact is respresentative of the project, therefore we can use these values.

5

Analysis

➢ To analyze the model we compare the classification of the authors (full-time or not full-time) from analyzing the project with the result obtained from the survey.

➢ Indicators used:● True positive (tp) → When we identify a full-time developer correctly● False negative (fn) → When we identify a full-time developer as non-full-time● True negative (tn) → When we identify a non-full-time developer correctly● False positive (fp) → When we identify a non-full-time developer as full-time

➢ From these indicators we calculate following five measures:● Precision● Recall● F-measure● Accuracy● Goodness (because false negatives compensate false positives!)

➢ We analyze 50 different values of the threshold “t” between 1 to 50.

6

Analysis Results

In these figures we can observe that for OpenStack the best results are obtained when the threshold "t" is between 9 and 12 commits every six months.

In this period the accuracy and goodness values achieve their maximum. From this value onwards, results get worse because the number of false positives increases.

7

Analysis Results

For values of "t" between 9 and 12, the model not only gets the best values for accuracy and goodness, the number of false positives and false negatives is compensated as well, mitigating the error that we obtain for the overall effort.

8

Effort Results

Assuming a threshold value t of 12:

● The estimation of effort invested in OpenStack lies around 9,000 person-months in total (750 person-years).

● 2,634 person-months have been invested during the last six months. This implies that we estimate that the actual effort on OpenStack has been around 440 person-months during the last 6 months.

The OpenStack foundation estimates that currently over 250 professional developers work on the project hired by companies, which backs these results.

9

Conclusions

➢ We faced two challenges, which have been achieved to a certain extent:

● To design a simple method to measure the effort provided in a sparse, distributed and uneven development scenario, like the FOSS one.

● To design a model so that it offers a reasonable prediction. This affects two aspects of the model:

➔ The activity threshold to consider a developer as full-time or not.➔ The credibility of the model.

➢ We have seen that the design of the model allows to compensate erroneous classification of developers.

➢ Since the majority of the effort is performed by those above the threshold, we could limit the overall margin of error by specifically focusing on those developers’ activities.

➢ The model is concerned exclusively with effort performed by developers. There are many other tasks in a FOSS project that are not considered in it.

10

Future Work

➢ We envisage to expand this study by...

● studying other FOSS projects (WebKit, Linux, WikiMedia, among others).

● performing a scientific experiment to obtain margins of error.

● investigating which of the two ways of measuring activity (commits or active days) is more suited for the model.

● comparing our results with the ones provided by traditional software estimation models used in industry, such as COCOMO.

● quantifying the effort required in a FOSS project to see if is it more profitable for a prospective adopting company to redo (“make”) their own system or to invest (“buy”) in the existing FOSS system.

● comparing our approach with previous effort estimation techniques for FOSS projects.

11

Contact

➢ More information can be found at the survey website:

http://ccervigon.libresoft.es➢ For the academic publication that contains the details of our research, refer to:

Gregorio Robles, Carlos Cervigón-Ávila, Jesús M. González-Barahona, Andrea Capiluppi, Daniel Izquierdo-Cortázar: “Estimating Development Effort in Free/Open Source Software Projects by Mining Software Repositories – A Case Study on OpenStack”. Proceedings of the 11th Working Conference on Mining Software Repositories, Hyderabad, India, June 2014.

➢ If you want to get in contact with the authors of this study / survey, please contact:

Gregorio Robles – grex (at) gsyc.urjc.es

12