The Polarsys Maturity Model:Assessing and Improving Software Project Quality

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo

https://polarsys.org/wiki/Maturity_Assessment_WG

Polarsys MeetingLudwigsburg (Germany), October 27st 2014

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo (PolarSys Maturity Assessment WG)Polarsys Maturity Model 1 / 33

c©2014Some rights reserved. This presentation is distributed under the

“Attribution-ShareAlike 3.0” license, by Creative Commons, available athttp://creativecommons.org/licenses/by-sa/3.0/

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo (PolarSys Maturity Assessment WG)Polarsys Maturity Model 2 / 33

Structure of the presentation

1 The Big Picture

2 The Quality Model

3 Mining Software Repositories

4 The PolarSys Dashboard

5 Next Steps

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo (PolarSys Maturity Assessment WG)Polarsys Maturity Model 3 / 33

The Big Picture

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo (PolarSys Maturity Assessment WG)Polarsys Maturity Model 4 / 33

The PolarSys ecosystem

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo (PolarSys Maturity Assessment WG)Polarsys Maturity Model 5 / 33

PolarSys in a nutshell

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo (PolarSys Maturity Assessment WG)Polarsys Maturity Model 6 / 33

Objectives...

What we want..

Assess maturity of components to ensure that the full stack is safe.Help people better understand, know and manage their project.Propose guidance for good practices and Eclipse processes.Involve people to make it a community-driven project.

What we do not want..

Judge or blame projects or people.Address individual characteristics.

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo (PolarSys Maturity Assessment WG)Polarsys Maturity Model 7 / 33

The Quality Model

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo (PolarSys Maturity Assessment WG)Polarsys Maturity Model 8 / 33

Defining Quality Requirements

There is no single definition of quality..Usual definitions of quality, existing standards and norms are notenough..OSS has an impact on the different quality concerns.We have to define our own quality requirements for Eclipse andPolarSys.

I Process: predictable outputs, well-defined structure of projects.I Ecosystem: nurture communities, good citizenship behaviour.I Product: maintainability, reliability, good practices.

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo (PolarSys Maturity Assessment WG)Polarsys Maturity Model 9 / 33

Composition of the Quality ModelQuality Attributes:

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo (PolarSys Maturity Assessment WG)Polarsys Maturity Model 10 / 33

Composition of the Quality Model

We further define:Quality attributes: define the concerns of quality.Measurement goals: generic measurement goals that can be appliedto all projects.Base metrics: raw numbers adapted to each project’s characteristics.

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo (PolarSys Maturity Assessment WG)Polarsys Maturity Model 11 / 33

The full quality model

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo (PolarSys Maturity Assessment WG)Polarsys Maturity Model 12 / 33

Mining Software Repositories

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo (PolarSys Maturity Assessment WG)Polarsys Maturity Model 13 / 33

Eclipse Software Repositories

The metrics used in the process are retrieved from several sources:Source Code: SonarQube, rule-checking tools.SCM, ITS, Mailing lists: Eclipse Dashboard (Grimoire).PMI JSON files from the Project Management Infrastructure project.Analysis of Forums, Web sites, or even manual surveys.

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo (PolarSys Maturity Assessment WG)Polarsys Maturity Model 14 / 33

Source code metrics

Source code metrics are retrieved from SonarQube, but other inputsmay be easily added.Metrics are computed as a ratio to limit the effect of size of software:

I Comment rate: number of comments per KLOC.I Complexity density: average number of paths per function.I Duplicated lines density: amount of duplicated lines per KLOC.

Some metrics are kept absolute because they are self-explicite:I Source Lines of Code: representative of the size of the software.I Depth of Inheritance Tree for OO languages.

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo (PolarSys Maturity Assessment WG)Polarsys Maturity Model 15 / 33

Rule-checking metrics

Rule-checking tools define good practices elaborated by theOpen-Source Community.PMD 5.2.1 and FindBugs 3.0.0 are used for now, but other tools canbe easily included in the framework: e.g. CheckStyle, FxCop, Lint,CppCheck..Examples of metrics include:

I NCC (NCC ANA, NCC CHA, etc.): number of violations (related toanalysability, changeability, etc.)

I ROKR (ROKR ANA, ROKR CHA, etc.): percentage of acquiredpractices (related to analysability, changeability, etc.)

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo (PolarSys Maturity Assessment WG)Polarsys Maturity Model 16 / 33

Configuration Management metrics

Software Configuration Management metrics are extracted from toolsmetadata.Supported tools are Git, Subversion, CVS, Bazaar, Mercurial.Examples of metrics include:

I Number of commits during last month.I Number of committers during last month.I Number of files committed during last month

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo (PolarSys Maturity Assessment WG)Polarsys Maturity Model 17 / 33

ITS metrics

Issue Tracking System metrics are extracted from tools metadata.Bugzilla, Jira, Redmine, Launchpad, GitHub and SourceForge ITS aresupported.Examples of metrics include:

I Number of bugs per thousands Lines of Code during last month.I Number of users active on the ITS during last month.I Number of updates to the ITS during last month.I Median time to fix issues during last month.

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo (PolarSys Maturity Assessment WG)Polarsys Maturity Model 18 / 33

MLS metrics

Mailing list metrics are extracted from the project’s archives.MBox and GMane are supported for now (forums are planned).Examples of metrics include:

I Number of mails exchanged during last month.I Number of threads exchanged during last month.I Number of active authors during last month.I Median ratio of answers on questions during last month.

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo (PolarSys Maturity Assessment WG)Polarsys Maturity Model 19 / 33

PMI metrics

The Project Management Infrastructure maintains a list of characteristicsfor all projects, with:

Information about repositories: SCM, ITS, MLS.Links for published content: web site, wiki, getting started, buildprocess, official documentation.Description of releases, date, scope, status of review...

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo (PolarSys Maturity Assessment WG)Polarsys Maturity Model 20 / 33

The PolarSys Dashboard

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo (PolarSys Maturity Assessment WG)Polarsys Maturity Model 21 / 33

Architecture of the platform

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo (PolarSys Maturity Assessment WG)Polarsys Maturity Model 22 / 33

The Dashboard: Home

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo (PolarSys Maturity Assessment WG)Polarsys Maturity Model 23 / 33

The Dashboard: project page for CDT

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo (PolarSys Maturity Assessment WG)Polarsys Maturity Model 24 / 33

The Dashboard: Quality Model for CDT

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo (PolarSys Maturity Assessment WG)Polarsys Maturity Model 25 / 33

The Dashboard: Measures for CDT

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo (PolarSys Maturity Assessment WG)Polarsys Maturity Model 26 / 33

The Dashboard: attributes for CDT

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo (PolarSys Maturity Assessment WG)Polarsys Maturity Model 27 / 33

The Dashboard: Practices for CDT

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo (PolarSys Maturity Assessment WG)Polarsys Maturity Model 28 / 33

The Dashboard: Actions for CDT

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo (PolarSys Maturity Assessment WG)Polarsys Maturity Model 29 / 33

Next Steps

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo (PolarSys Maturity Assessment WG)Polarsys Maturity Model 30 / 33

Next steps

Scales: 1-5 scales to instantly get an idea of what numbers mean.Improve presentation of results: visualisation, ergonomy.Add more content: metrics, rules, actions.Improve automation, ease the adoption setup for new-coming projects.Add more projects. What about yours?

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo (PolarSys Maturity Assessment WG)Polarsys Maturity Model 31 / 33

Join us!

The PolarSys wiki: polarsys.org/wikiMaturity Assessment on the wiki:polarsys.org/wiki/Maturity Assessment WGPolarSys Mailing list: dev.eclipse.org/mailman/listinfo/polarsys-iwgLook at the live dashboard prototype: castalia.camp/dl/dashboard/

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo (PolarSys Maturity Assessment WG)Polarsys Maturity Model 32 / 33

References

MACMIng2003 M. A. C. M. Ing, E. Georgiadou & W. Suryn (2003). Software QualityModel Requirements for Software Quality Engineering. QualityEngineering. doi:10.1.1.90.677

Kearney1986 Kearney, J. P., Sedlmeyer, R. L., Thompson, W. B., Gray, M. A., &Adler, M. A. (1986). Software complexity measurement.Communications of the ACM, 29(11), 1044–1050.

Kemerer1995 Kemerer, C. F. (1995). Software complexity and softwaremaintenance: A survey of empirical research. Annals of SoftwareEngineering, 1(1), 1–22.

B. Baldassari, G. Blondelle, J. M. Gonzalez-Barahona, D. Izquierdo (PolarSys Maturity Assessment WG)Polarsys Maturity Model 33 / 33