Continuous integration in a social-coding world

Bogdan Vasilescu @b_vasilescu

/ Department of Mathematics and Computer Science

Bogdan Vasilescu, Stef van Schuylenburg, Jules Wulms, !Alexander Serebrenik, Mark G. J. van den Brand!

Alexander Serebrenik @aserebrenik

How are GitHub developers using CI?

30th International Conference on Software Maintenance and Evolution, Victoria, BC, Canada, September 28 – October 3, 2014

#Developers #Commits #Pull Requests

Characteristics of the GitHub projects in our sample

1

10

100

1000

10000

100000

Sample of large and active GH repos!* not forks * ≥ 1 year * both commits

and pull requests

* Java, Python, or Ruby

* ≥ 10 changes last month

* ≥ 10 contributorsrails

elasticsearch ruby gradle

223 projects

2. Commits vs. Pull Requests: !Which builds succeed more often?!

Distribution of p−values from the Chi^2 tests

0.01

0.05 0.1

0.5 1

(45 / 84) statistically significant at 95%

Pull request builds are more successful!

0.2 0.5 1.0 2.0 5.0 10.0 20.0

Odds Ratios

(39 / 45) > 1

3. Not all projects are the same…!

Initiate CI process

Source Control

Build

TestReport

Commit

Tests

Developers

1. Usage of the Travis-CI service!

0

20

40

60

80

Java Python RubyProgramming.language

count

configured and using (113 / 223)

configured but not using !(93 / 223)

not configured (17 / 223)

• Python and Ruby

• Older than two years

• Not too many contributors

Pull request builds are more successful for:

• ≥5 successful pull requests

• ≥5 failed pull requests • ≥5 successful commits • ≥5 failed requests

(84 / 113)²

Prog. language Age (years) Contributors

Java Python Ruby < 2 2–4 > 4 < 17 17—33 > 33!

# projects 10 34 40 24 42 18 29 27 28

… s.t. χ2 p < 0.05 3 19 23 9 25 11 18 15 12

reject Stouffer H0? ✗ ✓ ✓ ✗ ✓ ✓ ✓ ✓ ✗

% odds ratio > 1 n/a 89 87 n/a 92 82 89 80 n/a