Standard Grimoire Report OPNFV Project 2016-Q4
Transcript of Standard Grimoire Report OPNFV Project 2016-Q4
Standard Grimoire Report
OPNFV Project
2016-Q4
January 11, 2017
Daniel Izquierdo CortazarChief Data [email protected]
This report would not exist without the effort of the people involved inthe development of the Grimoire toolset.
(cc) 2016 Bitergia. Some rights reserved.This work licensed under Creative Commons Attribution-ShareAlike 4.0
Unported License.To view a copy of full license, see
http://creativecommons.org/licenses/by-sa/4.0,or write to Creative Commons, 559 Nathan Abbott Way, Stanford,
California 94305, USA.
1
Executive Summary
This report provides a quantitative analysis of the current and past situ-ation of the OPNFV project. All the data presented in it is based on infor-mation retrieved from the software development repositories of the project.The analysis includes a summary of the general situation of the project,and specific analysis of some of its development processes (issue tracking,code review) and communication channels (mailing lists, IRC, AskBot). Forcomparison with the past, most of the data is shown on a quarterly basis.
2
Contents
1 Project overview 4
2 Communication and support-related activities 72.1 Mailing Lists . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.2 Questions and Answers . . . . . . . . . . . . . . . . . . . . . . 92.3 IRC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3 Details on OPNFV development community 133.1 Details of the project . . . . . . . . . . . . . . . . . . . . . . . 143.2 Activity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143.3 Community . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153.4 Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
A Metrics Definitions 19
B Source code and data sources 22
3
Chapter 1
Project overview
The report looks at activities across the OPNFV community during the lastquarter of 2016, comparing it to previous quarters.1.
Data source Activity 90 days Change (wrt to prev. 90 days)Gits 1590 commits -48%
Tickets 365 closed tickets -68%Mailing Lists 1754 sent emails -13%
Gerrit 1808 submitted reviews -36%Askbot 25 posted questions 13%
IRC 44332 messages -28%
Table 1.1: Activity during the last 90 days and its evolution
The overall development activity has decreased. Git and Gerrit activityhas decreased between a 36% and a 48%. This is also in line with the Jiraactivity with a decrease of 68%. Besides the communication channels keepdecreasing a 14% of mailing lists, 28% in the number of messages in the IRCchannels. On the other hand, Askbot shows an 13% of activity increase.If compared to previous periods, this quarter shows a similar activity levelthan the second quarter, while the first and third quarters of 2016 showhigher levels of activity in all areas.
1The analyzed data sources are available in appendixB
4
15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-Q2 16-Q3 16-Q40
10
20
30
40
50
60
70
80
10.500
28.500
41.500
58.00062.000 62.500
75.500
61.500
Developers per monthPeriod Authors per month15-Q1 10.515-Q2 28.515-Q3 41.515-Q4 58.016-Q1 62.016-Q2 62.516-Q3 75.516-Q4 61.5
In this quarter of 2016 the mean number of developers active per monthhas reached a total of 61.5. It is a decrease when compared to the previousquarter, but in line with the second quarter of 2016.
The total number of contributors divided into three sets (core, regularand casual2) follow a similar pattern.
15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-Q2 16-Q3 16-Q40
20
40
60
80
7
21
32
51 5255
63
46
8
16
25
3335
3337
30
5
16
24
29
34 33
46
41
Community structure
casualregularcore
Figure 1.1: Evolution during the last quarters of core, regular and casualdevelopers (based on git activity)
2Contributing developers are characterized as core, regular and casual depending ontheir activity in the git repositories. The classification is built by sorting contributors bytheir total number of commits; we sum the total commits per each individual contributors:the individuals whose commits sum up to 80% of the total number of commits in thequarter are the core contributors in that quarter. The regular contributors are thosewhose commits sum up to 95% of the total. The others are the casual contributors.
5
Period Core Regular Occasional15-Q1 5 8 715-Q2 16 16 2115-Q3 24 25 3215-Q4 29 33 5116-Q1 34 35 5216-Q2 33 33 5516-Q3 46 37 6316-Q4 41 30 46
Table 1.2: Characterization of developers by their total contribution to theproject
This report aims to provide some insight into the software developmentprocess of the OPNFV community measuring efficiency and process of thecommunity based on three metrics: the Review Efficiency Index (REI), theTime to Merge (TTM), and the Backlog Management Index (BMI). REI ismeasured as the number of closed (merged or abandoned) changesets out ofthe submitted changesets in a given period. TTM is measured as the timesince a review is submitted until this is closed. The BMI is measured as thenumber of closed tickets out of open tickets in a given period.
REI BMI TTM0.9 0.36 0.93 days
Table 1.3: Closed changesets out of opened changesets (REI), closed ticketout of opened tickets (BMI) and median time to merge in Gerrit (TTM)
6
Chapter 2
Communication andsupport-related activities
Analysis of the communication channels used for communication and support-related activities.
2.1 Mailing Lists
The following charts show activity in terms of emails sent, people sendingemails and people initiating threads per quarter. In addition, a table ispresented with the hot topics in the several analyzed mailing lists. Thisshows hot topics ordered by number of total posts in such thread.
15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-Q2 16-Q3 16-Q40
500
1000
1500
2000
2500
1518
1735 17081803
23942269
2105
1759
Emails sentPeriod Emails15-Q1 151815-Q2 173515-Q3 170815-Q4 180316-Q1 239416-Q2 226916-Q3 210516-Q4 1759
7
2.1. MAILING LISTS
15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-Q2 16-Q3 16-Q40
50
100
150
200
250
300
162
185
207 204
229
267251 252
People sending emailsPeriod People15-Q1 16215-Q2 18515-Q3 20715-Q4 20416-Q1 22916-Q2 26716-Q3 25116-Q4 252
15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-Q2 16-Q3 16-Q40
50
100
150
84
114 117124
133
157 159169
People initiating threadsPeriod People15-Q1 8415-Q2 11415-Q3 11715-Q4 12416-Q1 13316-Q2 15716-Q3 15916-Q4 169
8
2.2. QUESTIONS AND ANSWERS
Initial Author Numberand Date Subject Messagessw35882016-10-03 [opnfv-tsc] SDN World Congress 2016 presentation on
OPNFV EUAG19
myu2016-10-06 [opnfv-tech-discuss] OPNFV SPC Polestar WG Call
10/6/201617
Raymond Paik2016-12-15 [opnfv-tech-discuss] Starting to clean-up meetings
page on the wiki15
joehuang2016-12-14 [opnfv-tech-discuss] committer list maintainance 15Raymond Paik2016-10-09 [opnfv-tech-discuss] Announcing the 2016 TSC Chair
election result13
lukehinds2016-12-19 [Opnfv-security] Security checks at Gate 12Ryota MIBU2016-12-04 [opnfv-tech-discuss] [doctor] doctor CI now available 12Morgan Ri-chomme2016-10-27 [opnfv-tsc] sync before board/TSC meeting 11Yujun Zhang2016-12-15 [opnfv-tech-discuss] [all] TestPerf EcoSystem diagram
now editable11
Sindhur Malleni2016-11-22 [opnfv-users] OPNFV Colorado 2.0 with Triple0
RPMs11
2.2 Questions and Answers
The following charts show activity in the Ask site. Total number of ques-tions, number of answers, number of comments and people sending questionsare depicted. In addition two tables represent the hot topics activity in theAsk OPNFV site. These show information about the top visited questionsand questions with the highest number of people participating.
9
2.2. QUESTIONS AND ANSWERS
15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-Q2 16-Q3 16-Q40
10
20
30
40
0
3
23
36
30
45
24 25
QuestionsPeriod Questions15-Q1 015-Q2 315-Q3 2315-Q4 3616-Q1 3016-Q2 4516-Q3 2416-Q4 25
15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-Q2 16-Q3 16-Q40
10
20
30
40
50
60
02
30
44
33
57
2118
AnswersPeriod Answers15-Q1 015-Q2 215-Q3 3015-Q4 4416-Q1 3316-Q2 5716-Q3 2116-Q4 18
15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-Q2 16-Q3 16-Q40
10
20
30
40
1 1
18
22 21
41
6
1
CommentsPeriod Comments15-Q1 115-Q2 115-Q3 1815-Q4 2216-Q1 2116-Q2 4116-Q3 616-Q4 1
10
2.2. QUESTIONS AND ANSWERS
15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-Q2 16-Q3 16-Q40
5
10
15
20
25
0
3
14 14
20
23
19
14
People asking QuestionsPeriod People asking15-Q1 015-Q2 315-Q3 1415-Q4 1416-Q1 2016-Q2 2316-Q3 1916-Q4 14
• Top visited questions.
Question subject Visitscolorado release 10 arm iso not+ 18trying to install colorado on virtualbox+ 15opnfv deployment problem+ 13error installing openstack using fuel+ 9opnfv deployment in virtual environment+ 8linux kernel version used in colorado+ 7building colorado iso for armband+ 4deploy scenario os nosdn ovs ha+ 3build fail in single node build+ 2where exactly ie at what location+ 2
• Top questions with the highest number of people participating.
Question subject Peoplepartici-pating
opnfv deployment in virtual environment+ 2where exactly ie at what location+ 2error installing openstack using fuel+ 2colorado release 10 arm iso not+ 2trying to install colorado on virtualbox+ 2opnfv deployment problem+ 2error in opnfvfuel deployment error for+ 1error while installing colorado fuel from+ 1fuel not discovering nodes after pxe+ 1build fail in single node build+ 1
11
2.3. IRC
2.3 IRC
The community uses several IRC channels for asynchronous communication.This section shows information about the total number of messages sent inthe community during the last 7 quarters together with the number of peopleparticipating in such discussions.
15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-Q2 16-Q3 16-Q40
10000
20000
30000
40000
50000
60000
70000
4584
1783520636
18963
49851
54879
62522
44415
Messages in IRC channelsPeriod Messages15-Q1 458415-Q2 1783515-Q3 2063615-Q4 1896316-Q1 4985116-Q2 5487916-Q3 6252216-Q4 44415
15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-Q2 16-Q3 16-Q40
100
200
300
400
500
600
113
201
263293
461
544 534
486
People in IRC channelsPeriod People15-Q1 11315-Q2 20115-Q3 26315-Q4 29316-Q1 46116-Q2 54416-Q3 53416-Q4 486
12
Chapter 3
Details on OPNFVdevelopment community
Each breakdown is divided into three sections with information from thelast 6 quarters:
• activity: centered on the following metrics: commits from git activity,submitted, merge and abandoned reviews from the review system andopened and closed tickets from the issue tracking system.
• community: active core reviewers in gerrit, active authors in git andtop ten developers and top ten organizations contributing to the de-velopment in the last quarter.
• process: efficiency closing tickets, efficiency closing changesets, Timeto Merge (mean and median), number of patchsets (iterations) perchangeset and a study on the time waiting for a reviewer or submitteraction in the patchset review process.
13
3.1. DETAILS OF THE PROJECT
3.1 Details of the project
3.2 Activity
Commits in Git, submitted, merged and abandoned reviews in Gerrit andopened and closed issues in Jira.
15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-02 16-Q3 16-Q40
500
1000
1500
2000
2500
3000
3500
4000
224
692905
1571
3098
2023
2902
1812
157
612791
1373
2869
1793
3114
1599
Commits and reviews: general
reviewscommits
Period Commits Reviews15-Q1 157 22415-Q2 612 69215-Q3 791 90515-Q4 1373 157116-Q1 2869 309816-02 1793 202316-Q3 3114 290216-Q4 1599 1812
15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-02 16-Q3 16-Q40
200
400
600
800
1000
1200
1400
102
315
1103
634603
867 850
638
25
109
337
415481
515
1075
231
Opened and closed tickets: general
openedclosed
Period Closed Opened15-Q1 25 10215-Q2 109 31515-Q3 337 110315-Q4 415 63416-Q1 481 60316-02 515 86716-Q3 1075 85016-Q4 231 638
14
3.3. COMMUNITY
15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-02 16-Q3 16-Q40
500
1000
1500
2000
2500
3000
3500
105
549711
1220
2673
1599
2583
1447
47 84 103188
330 314183 183
Merged and abandoned reviews: general
mergedabandoned
Period Merged Abandoned15-Q1 105 4715-Q2 549 8415-Q3 711 10315-Q4 1220 18816-Q1 2673 33016-02 1599 31416-Q3 2583 18316-Q4 1447 183
3.3 Community
Active core reviewers in Gerrit, active authors in Git, top authors and or-ganizations in the last quarter
15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-02 16-Q3 16-Q40
20
40
60
80
14
34
59
6873
85 86
80
Active Core Reviewers: generalPeriod Active Core15-Q1 1415-Q2 3415-Q3 5915-Q4 6816-Q1 7316-02 8516-Q3 8616-Q4 80
15
3.4. PROCESS
15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-02 16-Q3 16-Q40
20
40
60
80
100
120
140
160
21
57
83
116124 125
151
123
Authors generalPeriod Authors15-Q1 2115-Q2 5715-Q3 8315-Q4 11616-Q1 12416-02 12516-Q3 15116-Q4 123
Commit (s) Author
143 Yujun Zhang
128 Jose Lausuch
115 Alexandru Avadanii
111 Morgan Richomme
78 Tim Rozet
71 matthew.lijun
65 Dan Radez
65 Fatih Degirmenci
65 zhihui wu
62 Serena Feng
Commit (s) Organizations
579 Huawei
326 ZTE Corporation
301 Ericsson
249 Red Hat
142 ENEA AB
141 Orange
87 Intel
75 Mirantis
42 Linux Foundation
25 NEC
23 ATT
16 SUSE
14 Dell
14 EMC
13 MontaVista
11 Juniper
7 Cisco
7 Nokia
7 Tata Consultancy
3.4 Process
Efficiency closing changesets and tickets, time to review (mean and median),number of patchests (iterations) per changeset and study on the time waitingfor a reviewer or submitter action in the patchset review process.
16
3.4. PROCESS
15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-02 16-Q3 16-Q40.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
0.250
0.3500.310
0.650
0.800
0.590
1.260
0.360
Efficiency closing tickets: generalPeriod Closed/Opened15-Q1 0.2515-Q2 0.3515-Q3 0.3115-Q4 0.6516-Q1 0.816-02 0.5916-Q3 1.2616-Q4 0.36
15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-02 16-Q3 16-Q40.0
0.2
0.4
0.6
0.8
1.0
0.680
0.910 0.900 0.900
0.970 0.950 0.9500.900
Changesets efficiency: generalPeriod (Aband. and Merg.)/Subm.15-Q1 0.6815-Q2 0.9115-Q3 0.915-Q4 0.916-Q1 0.9716-02 0.9516-Q3 0.9516-Q4 0.9
15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-02 16-Q3 16-Q40
1
2
3
4
5
6
7
8
4.250 4.330
6.840
4.050
3.550
3.990
3.430
3.870
0.760
0.330 0.2700.590
0.180 0.3400.530
0.930
Time to merge (days): general
meanmedian
Period Median Mean15-Q1 0.76 4.2515-Q2 0.33 4.3315-Q3 0.27 6.8415-Q4 0.59 4.0516-Q1 0.18 3.5516-02 0.34 3.9916-Q3 0.53 3.4316-Q4 0.93 3.87
17
3.4. PROCESS
15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-02 16-Q3 16-Q40.0
0.5
1.0
1.5
2.0
2.5
3.0
1.840
2.100 2.110
2.310
1.930 1.920
2.100
2.460
1.000 1.000 1.000 1.000 1.000 1.000 1.000
2.000
Patchsets per Changeset: general
meanmedian
Period Median Mean15-Q1 1.0 1.8415-Q2 1.0 2.115-Q3 1.0 2.1115-Q4 1.0 2.3116-Q1 1.0 1.9316-02 1.0 1.9216-Q3 1.0 2.116-Q4 2.0 2.46
15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-02 16-Q3 16-Q40
5
10
15
20
13.420
11.94013.110 13.430
12.070
18.790
9.790
12.680
4.990
0.3700.950 1.090
0.0500.670 0.300
1.360
Time waiting for the reviewer (days): general
avgmedian
Period Median Mean15-Q1 4.99 13.4215-Q2 0.37 11.9415-Q3 0.95 13.1115-Q4 1.09 13.4316-Q1 0.05 12.0716-02 0.67 18.7916-Q3 0.3 9.7916-Q4 1.36 12.68
15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-02 16-Q3 16-Q40
5
10
15
20
25
30
35
40
24.200
29.000
15.690
29.020
31.810
23.300
18.890
16.470
32.550
3.490
0.920
7.030
1.420 0.970 0.730 1.500
Time waiting for the submitter (days): general
avgmedian
Period Median Mean15-Q1 32.55 24.215-Q2 3.49 29.015-Q3 0.92 15.6915-Q4 7.03 29.0216-Q1 1.42 31.8116-02 0.97 23.316-Q3 0.73 18.8916-Q4 1.5 16.47
18
Appendix A
Metrics Definitions
• Commit: this is defined as the action(s) that performes a change inthe source code. Bots, merges and other type of automatic activity isremoved from the records. In addition, when aggregating several gitrepositories, this metric only counts unique revisions (unique hashesfound in the git repositories). In addition, all branches are aggregatedto the analysis.
• Submitted changeset: a changeset is the process of peer reviewingsource code changes. A submitted code is not merged to the mastercode of a given project till this is approved for at least one core reviewerof such project. A submitted changeset is defined as any changesetsubmitted to the Gerrit system.
• Merged and abandoned changsets: a merge is defined as the patchsetthat was finally submitted to the source code. An abandoned change-set is a potential merge that was finally dismissed by developers asbeing part of the source code. This status is found in the status of thefinal patchset. However, although a patchset can be merged or aban-doned, this action can be reverted. If a patchset presents several ofthese changes in the same period of time, only one of them is counted(the very last one). On the other hand, if those changes take place indifferent periods of analysis, both status would be counted.
• Open and closed ticket: a ticket in Jira is counted as closed if thestatus of such ticket is defined as ’Closed’. The rest of the tickets arecounted as opened tickets.
• Active Core Reviewer: a core reviewer has the possibility to use +2 or
19
-2 actions when reviewing the code. However, if there are developersthat for some period do not use those actions, those can not be mea-sured as core reviewer. Thus, this metric provides information about’active’ core reviewers. This can be also defined as those developersthat actively have used the +2 or -2 review action. This metric is alsofiltered by branch of activity, only using ’master’. This helps to detectactual core reviewers in each of the projects.
• Authors: a developer is defined as author if she is the owner of thepatchset sent for reviewing and this is merged into the source code.As previously indicated, automatic commits such bot’s are removedfrom this analysis.
• Efficiency closing issues: this metric is a derivation of the BacklogManagement Index (BMI) that measures the number of closed ticketsout of the opened tickets in a period of time. Values under 1.0 indicatesthat the number of closing issues is lower than the number of openedissues arriving. On the contrary, higher charts would indicate bettermaintenance effort by the community.
• Efficiency closing changesets: this metric is a derivation of the BacklogManagement Index as it is named as Review efficiency index (REI).As similarly used in the BMI index, this metrics measures the numberof closed changesets (merged or abandoned) out of the total numberof new changesets.
• Time to Merge: this time consists of the time between the first uploadof the first patchset (as defined as a submitted changeset) till the lastpatchset of the changeset is merged into the code and this is indicatedin the comments side of the Gerrit tool. This metric is provided innumber of days.
• Patchsets per changeset: this metric calculates the total number ofiterations in a changeset till this is abandoned or merged.
• Time waiting for the reviewer or the submitter: a changeset is waitingfor a reviewer action if a new patchset upload or a new changset arrivesto the system. On the other hand, a submitter action is required whena specific negative verification or reviewing action takes place (Verified-1/-2 or Code-Review -1/-2). In addition, when a Code-Review +2action takes place, it is assumed that the changset is closing and no
20
more times are registered either for the reviewer or the submitter. Forthis analysis, those patchsets flagged as work in progress are ignored.
Metrics measured in the general overview:
• Community structure, core, regular and casual developers: developersare ordered in descendant order by the number of commits authoredfor a given period. Core developers are defined as the list of developersthat reach 80% of the total commits. Regular is the set of developersthat are between that 80% and 95% of the commits. Casual develop-ers are found in the rest of the 5%. Bots are ignored in this list ofdevelopers.
• Developer per month: average of developers per month ignoring bots.
• Emails sent: number of emails sent by people to the several mailinglists. Bots are not registered.
• People sending emails: number of people sending those emails ignoringbots.
• People initiating threads: a thread is defined as a list of emails thathas the same root. There may exist threads of one email.
• Top threads: this list provides the longest threads in terms of numberof emails that have a common root email.
• Questions, answers and comments in Askbot.
• People asking questions in Askbot: number of people sending a newquestion.
• Top visited questions.
• Messages and people in IRC: this analysis ignores as a message thoseentries in the IRC channels that provide information about peopleentering or leaving the system.
21
Appendix B
Source code and data sources
The source code of the scripts and templates used to produce this reportare available from the GrimoireReports repository1.
The databases used for the analysis can be obtained from the “Datasources” panel2 of the Grimoire Dashboard for the project3.
1https://github.com/VizGrimoire/GrimoireReports2http://projects.bitergia.com/opnfv/browser/data_sources.html3http://projects.bitergia.com/opnfv/
22