Post on 15-Jul-2015
Who Should Review My Code?A File Location-Based Code-Reviewer Recommendation
Approach for Modern Code Review
Patanamon (Pick) Thongtanunam
Chakkrit Tantithamtavorn Raula G. Kula Norihiro
Yoshida Hajimu Iida Ken-ichi Matsumoto
patanamon-t@is.naist.jp @pamon
22nd IEEE International Conference on Software Analysis, Evolution, and Reengineering
Montréal, Quebec, Canada March 3 - 6, 2015
1
Modern Code Review is a tool-based, and used regularly in practice nowadays at industrial and OSS projects
2
Gerrit Code Review
Rietveld
Modern Code Review is a tool-based, and used regularly in practice nowadays at industrial and OSS projects
2
Gerrit Code Review
Rietveld
Modern Code Review is a tool-based, and used regularly in practice nowadays at industrial and OSS projects
2
Modern Code Review
Developer
3
Modern Code Review
DeveloperCode change
Gerrit Code Review
3
Modern Code Review
DeveloperCode change
Gerrit Code Review
3
Modern Code Review
Reviewers
Developer
Can you review my code?
Code change
Gerrit Code Review
Invite Reviewers
3
Modern Code Review
Coding style passed :) There’s a
bug!
Reviewers
Developer
Can you review my code?
Code change
Gerrit Code Review
Invite Reviewers
3
Code-Reviewer Assignment Problem
Code change
Gerrit Code Review
Invite Reviewers
Developer
4
Code-Reviewer Assignment ProblemWho should review
my code?
Code change
Gerrit Code Review
Invite Reviewers
New Developer
4
Code-Reviewer Assignment ProblemWho should review
my code?
Code change
Gerrit Code Review
Invite Reviewers
New Developer
4
RQ1: How do reviews with code-reviewer assignment problem impact reviewing time?
5
RQ1: How do reviews with code-reviewer assignment problem impact reviewing time?
Studied Projects
5
RQ1: How do reviews with code-reviewer assignment problem impact reviewing time?
Studied Projects
Sample Reviews 357 363 378 363
5
RQ1: How do reviews with code-reviewer assignment problem impact reviewing time?
Studied Projects
Sample Reviews 357 363 378 363#Reviews w/
Assignment Problem 10% 5% 30% 4%
5
RQ1: How do reviews with code-reviewer assignment problem impact reviewing time?
Studied Projects
Sample Reviews 357 363 378 363#Reviews w/
Assignment Problem 10% 5% 30% 4%Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systems
Rev
iew
ing
time
(Day
s)
Android OpenStack Qt LibreOffice
010
2030
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systems
Rev
iew
ing
time
(Day
s)
Android OpenStack Qt LibreOffice
010
2030
Reviewing Time (Days)
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systems
Rev
iew
ing
time
(Day
s)
Android OpenStack Qt LibreOffice
010
2030
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systems
Rev
iew
ing
time
(Day
s)
Android OpenStack Qt LibreOffice
010
2030
Review w/ Assignment Problem Review w/o Assignment Problem
18 Days Longer 13 Days Longer 9 Days Longer 6 Days Longer
5
RQ1: How do reviews with code-reviewer assignment problem impact reviewing time?
Studied Projects
Sample Reviews 357 363 378 363#Reviews w/
Assignment Problem 10% 5% 30% 4%
Reviews with code-reviewer assignment problem significantly take 12 days longer to approve a code change.
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systems
Rev
iew
ing
time
(Day
s)
Android OpenStack Qt LibreOffice
010
2030
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systems
Rev
iew
ing
time
(Day
s)
Android OpenStack Qt LibreOffice
010
2030
Reviewing Time (Days)
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systems
Rev
iew
ing
time
(Day
s)
Android OpenStack Qt LibreOffice
010
2030
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systems
Rev
iew
ing
time
(Day
s)
Android OpenStack Qt LibreOffice
010
2030
Review w/ Assignment Problem Review w/o Assignment Problem
18 Days Longer 13 Days Longer 9 Days Longer 6 Days Longer
5
6
RQ1: How do reviews with code-reviewer assignment problem impact reviewing time?
6
RQ1: How do reviews with code-reviewer assignment problem impact reviewing time?
6
RQ1: How do reviews with code-reviewer assignment problem impact reviewing time?
6
RQ1: How do reviews with code-reviewer assignment problem impact reviewing time?
A code-reviewer recommendation tool will be useful in distributed software development to speed up the code review process.
6
RQ1: How do reviews with code-reviewer assignment problem impact reviewing time?
RevFinder: A File Location-based Code-Reviewer Recommendation Approach
7
RevFinder: Overview
R1 R2
Changed Files- src/video/camera.java - src/video/player.javaCode-Reviewers
Reviewed by John
Changed Files- src/devices/screen.java
Code-Reviewers
Reviewed by Pat
Review History
8
RevFinder: Overview
R1 R2
Changed Files- src/video/camera.java - src/video/player.javaCode-Reviewers
Reviewed by John
Changed Files- src/devices/screen.java
Code-Reviewers
Reviewed by Pat
R3 (New)
Changed Files
- src/video/recorder.java
Code-Reviewers
?
Review History
8
RevFinder: Overview
R1 R2
Changed Files- src/video/camera.java - src/video/player.javaCode-Reviewers
Reviewed by John
Changed Files- src/devices/screen.java
Code-Reviewers
Reviewed by Pat
R3 (New)
Changed Files
- src/video/recorder.java
Code-Reviewers
?
RevFinder
Review History
Find reviewers who have reviewed the most similar file paths
8
RevFinder: Overview
R1 R2
Changed Files- src/video/camera.java - src/video/player.javaCode-Reviewers
Reviewed by John
Changed Files- src/devices/screen.java
Code-Reviewers
Reviewed by Pat
R3 (New)
Changed Files
- src/video/recorder.java
Code-Reviewers
?
RevFinder
Review History
8
RevFinder: Overview
R1 R2
Changed Files- src/video/camera.java - src/video/player.javaCode-Reviewers
Reviewed by John
Changed Files- src/devices/screen.java
Code-Reviewers
Reviewed by Pat
R3 (New)
Changed Files
- src/video/recorder.java
Code-Reviewers
?
RevFinder
Compare
Compare(FilePath_R2,FilePath_R3) = 0.3
Review History
8
RevFinder: Overview
R1 R2
Changed Files- src/video/camera.java - src/video/player.javaCode-Reviewers
Reviewed by John
Changed Files- src/devices/screen.java
Code-Reviewers
Reviewed by Pat
R3 (New)
Changed Files
- src/video/recorder.java
Code-Reviewers
?
RevFinder
Compare(FilePath_R2,FilePath_R3) = 0.3
Compare(FilePath_R1,FilePath_R3) = 0.6
Review History
8
Compare
RevFinder: Overview
R1 R2
Changed Files- src/video/camera.java - src/video/player.javaCode-Reviewers
Reviewed by John
Changed Files- src/devices/screen.java
Code-Reviewers
Reviewed by Pat
R3 (New)
Changed Files
- src/video/recorder.java
Code-Reviewers
?
RevFinder
Review History
Compare(FilePath_R2,FilePath_R3) = 0.3
Compare(FilePath_R1,FilePath_R3) = 0.69
RevFinder: Overview
R1 R2
Changed Files- src/video/camera.java - src/video/player.javaCode-Reviewers
Reviewed by John
Changed Files- src/devices/screen.java
Code-Reviewers
Reviewed by Pat
R3 (New)
Changed Files
- src/video/recorder.java
Code-Reviewers
?
RevFinder
Review History
Compare(FilePath_R1,FilePath_R3) = 0.6
“Pat should review R3”9
RevFinder: Overview
R1 R2
Changed Files- src/video/camera.java - src/video/player.javaCode-Reviewers
Reviewed by John
Changed Files- src/devices/screen.java
Code-Reviewers
Reviewed by Pat
R3 (New)
Changed Files
- src/video/recorder.java
Code-Reviewers
?
RevFinder
Review History
Compare(FilePath_R1,FilePath_R3) = 0.6
“Pat should review R3”9
RevFinder: FilePath Comparison Techniques
Longest Common Prefix
Longest Common Suffix
Longest Common Substring
Longest Common Subsequence
4 String Comparison Techniques
10
RevFinder: FilePath Comparison Techniques
Longest Common Prefix
Longest Common Suffix
Longest Common Substring
Longest Common Subsequence
4 String Comparison Techniques
10
RevFinder: FilePath Comparison Techniques
Longest Common Prefix
Longest Common Suffix
Longest Common Substring
Longest Common Subsequence Recommendation List
Combine (Borda Count)
4 String Comparison Techniques
10
RevFinder: FilePath Comparison Techniques
Longest Common Prefix
Longest Common Suffix
Longest Common Substring
Longest Common Subsequence Recommendation List
Combine (Borda Count)
4 String Comparison Techniques
Voting Ranks
10
RevFinder: FilePath Comparison Techniques
Longest Common Prefix
Longest Common Suffix
Longest Common Substring
Longest Common Subsequence Recommendation List
Combine (Borda Count)
4 String Comparison Techniques
Voting Ranks
10
Bubble-up correct reviewers
Empirical Evaluation
Studied Projects
Period ~4 years ~1year ~1 year ~2 year
#Reviews 5,126 6,586 23,810 6,523#Reviewers 94 82 202 63
11
Empirical Evaluation
Studied Projects
Period ~4 years ~1year ~1 year ~2 year
#Reviews 5,126 6,586 23,810 6,523#Reviewers 94 82 202 63
Baseline Approach: ReviewBot (Balachandran ICSE’13)
R1 R2R3 (New)
JohnPat
“Recommend reviewers who have reviewed the same line of code changes”
11
Empirical Evaluation
Studied Projects
Period ~4 years ~1year ~1 year ~2 year
#Reviews 5,126 6,586 23,810 6,523#Reviewers 94 82 202 63
Baseline Approach: ReviewBot (Balachandran ICSE’13)
R1 R2R3 (New)
JohnPat
“Recommend reviewers who have reviewed the same line of code changes”
John should review
R3
11
Empirical Evaluation
12
Empirical Evaluation
Accuracy
“RQ2: Does RevFinder accurately recommend reviewers?”
12
Empirical Evaluation
Accuracy
“RQ2: Does RevFinder accurately recommend reviewers?”
Ranking Performance
“RQ3: Does RevFinder provide a better ranking of correct reviewers?”
12
Empirical Evaluation
Accuracy
“RQ2: Does RevFinder accurately recommend reviewers?”
Ranking Performance
“RQ3: Does RevFinder provide a better ranking of correct reviewers?”
12
Empirical Evaluation
Accuracy
“RQ2: Does RevFinder accurately recommend reviewers?”
Measure: Top-k Accuracy
Ranking Performance
“RQ3: Does RevFinder provide a better ranking of correct reviewers?”
12
Empirical Evaluation: Accuracy
Top-k Accuracy measures how many reviews does an approach can correctly recommend with a Top-k recommendation.
13
Empirical Evaluation: Accuracy
Top-k Accuracy measures how many reviews does an approach can correctly recommend with a Top-k recommendation.
13
Top-5 Recommendation
Empirical Evaluation: Accuracy
Top-k Accuracy measures how many reviews does an approach can correctly recommend with a Top-k recommendation.
13
Top-5 Recommendation
Actual Reviewer
Empirical Evaluation: Accuracy
Top-k Accuracy measures how many reviews does an approach can correctly recommend with a Top-k recommendation.
13
Top-5 Recommendation
Actual Reviewer
Correct!
Empirical Evaluation: Accuracy
Top-k Accuracy measures how many reviews does an approach can correctly recommend with a Top-k recommendation.
Top-
10 A
ccur
acy
0%
22.5%
45%
67.5%
90%
10%
28%41%
29%
74%69%
87%86%
RevFinder ReviewBot
RevFinder can correctly recommended 69%-87% of reviews with a top-10 recommendation. RevFinder is 4 times more accurate than ReviewBot.
13
Top-5 Recommendation
Actual Reviewer
Correct!
Empirical Evaluation
Accuracy
“Does RevFinder accurately recommend reviewers?”
Measure: Top-k Accuracy
Ranking Performance
“Does RevFinder provide a better ranking of correct reviewers?”
14
Empirical Evaluation
Accuracy
“Does RevFinder accurately recommend reviewers?”
Measure: Top-k Accuracy
Ranking Performance
“Does RevFinder provide a better ranking of correct reviewers?”
14
On average, RevFinder can correctly recommend reviewers for 79% of
reviews with a top-10 recommendation.
Empirical Evaluation
Accuracy
“Does RevFinder accurately recommend reviewers?”
Measure: Top-k Accuracy
Ranking Performance
“Does RevFinder provide a better ranking of correct reviewers?”
14
On average, RevFinder can correctly recommend reviewers for 79% of
reviews with a top-10 recommendation.
Empirical Evaluation
Accuracy
“Does RevFinder accurately recommend reviewers?”
Measure: Top-k Accuracy
Ranking Performance
“Does RevFinder provide a better ranking of correct reviewers?”
14
On average, RevFinder can correctly recommend reviewers for 79% of
reviews with a top-10 recommendation.
Measure: Mean Reciprocal Rank (MRR)
Empirical Evaluation: Ranking Performance
Mean Reciprocal Rank measures an average rank of correct reviewer that an approach can recommend.
15
Empirical Evaluation: Ranking Performance
Mean Reciprocal Rank measures an average rank of correct reviewer that an approach can recommend.
15
Recommendation Lists
vs
Actual Reviewer
Empirical Evaluation: Ranking Performance
Mean Reciprocal Rank measures an average rank of correct reviewer that an approach can recommend.
15
Recommendation Lists
vs
Actual Reviewer
MRR = 1/3
Empirical Evaluation: Ranking Performance
Mean Reciprocal Rank measures an average rank of correct reviewer that an approach can recommend.
15
Recommendation Lists
vs
Actual Reviewer
MRR = 1/3
MRR = 1/1
Empirical Evaluation: Ranking Performance
Mean Reciprocal Rank measures an average rank of correct reviewer that an approach can recommend.
RevFinder recommended the correct code-reviewers with a median rank of 2 - 8. The code-reviewers ranking of RevFinder
is 3 times better than that of ReviewBot.15
Recommendation Lists
vs
Actual Reviewer
MRR = 1/3
MRR
RevFinder 0.6 0.55 0.31 0.40
ReviewBot 0.25 0.30 0.22 0.07
Ran
k of
C
orre
ct R
evie
wer
s Android OpenStack Qt LibreOffice●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●
●
●
●
●●●●●
●●●
●●●●●●●●●●●●
●
●●●●●●
●●●
●●●●
●
●
●●●
●●●●●●
●
●●●●●
●
●●
●
●●●●●●●
●
●●●●
●
●●
●
●●●●●●●●●●
●
●●●
●
●●●
●
●●●●●●
●
●
●
●●
●
●●●
●
●
●●
●
●●●
●●
●●
●
●
●●●●
●
●●●●●
●●
●
●
●●
●
●●●●●●●●●●●●
●
●
●●●●
●
●●
●
●●●●●●●●●
●
●
●
●●●
●
●
●
●
●
●●●
●
●●
●
●
●●
●●●●●
●
●●●
●
●
●●
●
●●
●
●
●●●●
●
●●●●●●●
●●
●●●●●
●●
●
●
●●●●●●●●●
●
●●●●
●
●
●●●●●●●●●●●●●●
●●
●
●
●●
●
●●
●
●●
●
●
●●
●●
●●●●●●
●
●●●
●●
●
●
●
●
●●●
●
●●●●●●●●
●
●●●●●●
●●
●●
●
●●●●
●
●
●
●●●
●
●
●
●
●
●●
●
●●●
●
●
●●●
●
●
●●●●●
●
●
●
●●●●
●
●
●
●
●●●●●●●●●●
●
●
●
●●●
●●●
●●●●●●●●●●●
●
●
●
●
●●●●
●
●●
●●
●
●
●
●
●●●●●●●
●
●
●
●
●●
●●
●
●
●
●
●
●●●●●
●
●●●
●
●●●●●
●
●
●
●
●●●
●
●
●
●●●●●●●
●
●
●
●●
●
●●●●●
●●
●
●●●●
●
●●●●●
●
●
●●
●
●
●
●●●●●
●
●●●●●●
●
●●●
●●
●●●●
●
●●
●
●
●●
●
●
●●●●●
●
●
●●
●
●●●●●●●●●●
●
●●●●●
●●
●●●
●●
●●
●
●
●●
●
●
●
●
●●
●
●●
●●●●●
●1
515
50100200
RevFinder ReviewBot RevFinder ReviewBot RevFinder ReviewBot RevFinder ReviewBotApproach
Rank
RevFinder ReviewBot RevFinder ReviewBot RevFinder ReviewBot RevFinder ReviewBot
MRR = 1/1
Empirical Evaluation: Ranking Performance
Mean Reciprocal Rank measures an average rank of correct reviewer that an approach can recommend.
RevFinder recommended the correct code-reviewers with a median rank of 2 - 8. The code-reviewers ranking of RevFinder
is 3 times better than that of ReviewBot.15
Recommendation Lists
vs
Actual Reviewer
MRR = 1/3
MRR
RevFinder 0.6 0.55 0.31 0.40
ReviewBot 0.25 0.30 0.22 0.07
Ran
k of
C
orre
ct R
evie
wer
s Android OpenStack Qt LibreOffice●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●
●
●
●
●●●●●
●●●
●●●●●●●●●●●●
●
●●●●●●
●●●
●●●●
●
●
●●●
●●●●●●
●
●●●●●
●
●●
●
●●●●●●●
●
●●●●
●
●●
●
●●●●●●●●●●
●
●●●
●
●●●
●
●●●●●●
●
●
●
●●
●
●●●
●
●
●●
●
●●●
●●
●●
●
●
●●●●
●
●●●●●
●●
●
●
●●
●
●●●●●●●●●●●●
●
●
●●●●
●
●●
●
●●●●●●●●●
●
●
●
●●●
●
●
●
●
●
●●●
●
●●
●
●
●●
●●●●●
●
●●●
●
●
●●
●
●●
●
●
●●●●
●
●●●●●●●
●●
●●●●●
●●
●
●
●●●●●●●●●
●
●●●●
●
●
●●●●●●●●●●●●●●
●●
●
●
●●
●
●●
●
●●
●
●
●●
●●
●●●●●●
●
●●●
●●
●
●
●
●
●●●
●
●●●●●●●●
●
●●●●●●
●●
●●
●
●●●●
●
●
●
●●●
●
●
●
●
●
●●
●
●●●
●
●
●●●
●
●
●●●●●
●
●
●
●●●●
●
●
●
●
●●●●●●●●●●
●
●
●
●●●
●●●
●●●●●●●●●●●
●
●
●
●
●●●●
●
●●
●●
●
●
●
●
●●●●●●●
●
●
●
●
●●
●●
●
●
●
●
●
●●●●●
●
●●●
●
●●●●●
●
●
●
●
●●●
●
●
●
●●●●●●●
●
●
●
●●
●
●●●●●
●●
●
●●●●
●
●●●●●
●
●
●●
●
●
●
●●●●●
●
●●●●●●
●
●●●
●●
●●●●
●
●●
●
●
●●
●
●
●●●●●
●
●
●●
●
●●●●●●●●●●
●
●●●●●
●●
●●●
●●
●●
●
●
●●
●
●
●
●
●●
●
●●
●●●●●
●1
515
50100200
RevFinder ReviewBot RevFinder ReviewBot RevFinder ReviewBot RevFinder ReviewBotApproach
Rank
RevFinder ReviewBot RevFinder ReviewBot RevFinder ReviewBot RevFinder ReviewBot
MRR = 1/1
Empirical Evaluation
Accuracy
“Does RevFinder accurately recommend reviewers?”
Measure: Top-k Accuracy
Ranking Performance
“Does RevFinder provide a better ranking of correct reviewers?”
16
On average, RevFinder can correctly recommend reviewers for 79% of
reviews with a top-10 recommendation.
Measure: Mean Reciprocal Rank (MRR)
Empirical Evaluation
Accuracy
“Does RevFinder accurately recommend reviewers?”
Measure: Top-k Accuracy
Ranking Performance
“Does RevFinder provide a better ranking of correct reviewers?”
16
On average, RevFinder can correctly recommend reviewers for 79% of
reviews with a top-10 recommendation.
Measure: Mean Reciprocal Rank (MRR)
On average, RevFinder can recommend reviewers with a median rank of 4.
Empirical Evaluation
Accuracy
“Does RevFinder accurately recommend reviewers?”
Measure: Top-k Accuracy
Ranking Performance
“Does RevFinder provide a better ranking of correct reviewers?”
16
On average, RevFinder can correctly recommend reviewers for 79% of
reviews with a top-10 recommendation.
Measure: Mean Reciprocal Rank (MRR)
On average, RevFinder can recommend reviewers with a median rank of 4.
We can use a similarity of previously reviewed file paths to recommend code-reviewers for new code changes.
Summary
17
Code-Reviewer Assignment ProblemWho should review
my code?
Code change
Gerrit Code Review
Invite Reviewers
New Developer
4
Code-Reviewer Assignment ProblemWho should review
my code?
Code change
Gerrit Code Review
Invite Reviewers
New Developer
4
Code-Reviewer Assignment ProblemWho should review
my code?
Code change
Gerrit Code Review
Invite Reviewers
New Developer
4
5
How do reviews with code-reviewer assignment problem impact reviewing time?
Studied Projects
Sample Reviews 357 363 378 363
#Reviews w/ Assignment Problem 10% 5% 30% 4%
Reviews with code-reviewer assignment problem significantly take 12 days longer to approve a code change.
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systems
Revie
wing
tim
e (D
ays)
Android OpenStack Qt LibreOffice
010
2030
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systems
Revie
wing
tim
e (D
ays)
Android OpenStack Qt LibreOffice
010
2030
Reviewing Time (Days)
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systems
Revie
wing
tim
e (D
ays)
Android OpenStack Qt LibreOffice
010
2030
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systems
Revie
wing
tim
e (D
ays)
Android OpenStack Qt LibreOffice
010
2030
Review w/ Assignment Problem Review w/o Assignment Problem
18 Days Longer 13 Days Longer 9 Days Longer 6 Days Longer
Code-Reviewer Assignment ProblemWho should review
my code?
Code change
Gerrit Code Review
Invite Reviewers
New Developer
4 5
How do reviews with code-reviewer assignment problem impact reviewing time?
Studied Projects
Sample Reviews 357 363 378 363
#Reviews w/ Assignment Problem 10% 5% 30% 4%
Reviews with code-reviewer assignment problem significantly take 12 days longer to approve a code change.
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systems
Revie
wing
tim
e (D
ays)
Android OpenStack Qt LibreOffice
010
2030
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systems
Revie
wing
tim
e (D
ays)
Android OpenStack Qt LibreOffice
010
2030
Reviewing Time (Days)
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systems
Revie
wing
tim
e (D
ays)
Android OpenStack Qt LibreOffice
010
2030
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systemsRe
viewi
ng ti
me
(Day
s)
Android OpenStack Qt LibreOffice
010
2030
Review w/ Assignment Problem Review w/o Assignment Problem
18 Days Longer 13 Days Longer 9 Days Longer 6 Days Longer
5
How do reviews with code-reviewer assignment problem impact reviewing time?
Studied Projects
Sample Reviews 357 363 378 363
#Reviews w/ Assignment Problem 10% 5% 30% 4%
Reviews with code-reviewer assignment problem significantly take 12 days longer to approve a code change.
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systems
Revie
wing
tim
e (D
ays)
Android OpenStack Qt LibreOffice
010
2030
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systems
Revie
wing
tim
e (D
ays)
Android OpenStack Qt LibreOffice
010
2030
Reviewing Time (Days)
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systems
Revie
wing
tim
e (D
ays)
Android OpenStack Qt LibreOffice
010
2030
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systemsRe
viewi
ng ti
me
(Day
s)
Android OpenStack Qt LibreOffice
010
2030
Review w/ Assignment Problem Review w/o Assignment Problem
18 Days Longer 13 Days Longer 9 Days Longer 6 Days Longer
Code-Reviewer Assignment ProblemWho should review
my code?
Code change
Gerrit Code Review
Invite Reviewers
New Developer
4
7
RevFinder: Overview
R1 R2
Changed Files- src/video/camera.java - src/video/player.javaCode-Reviewers
Reviewed by John
Changed Files
- src/devices/screen.java
Code-Reviewers
Reviewed by Pat
R3 (New)
Changed Files
- src/video/recorder.java
Code-Reviewers
?
RevFinder
Review History
Find reviewers who have reviewed the most similar file paths
5
How do reviews with code-reviewer assignment problem impact reviewing time?
Studied Projects
Sample Reviews 357 363 378 363
#Reviews w/ Assignment Problem 10% 5% 30% 4%
Reviews with code-reviewer assignment problem significantly take 12 days longer to approve a code change.
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systems
Revie
wing
tim
e (D
ays)
Android OpenStack Qt LibreOffice
010
2030
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systems
Revie
wing
tim
e (D
ays)
Android OpenStack Qt LibreOffice
010
2030
Reviewing Time (Days)
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systems
Revie
wing
tim
e (D
ays)
Android OpenStack Qt LibreOffice
010
2030
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systemsRe
viewi
ng ti
me
(Day
s)
Android OpenStack Qt LibreOffice
010
2030
Review w/ Assignment Problem Review w/o Assignment Problem
18 Days Longer 13 Days Longer 9 Days Longer 6 Days Longer
Code-Reviewer Assignment ProblemWho should review
my code?
Code change
Gerrit Code Review
Invite Reviewers
New Developer
4
7
RevFinder: Overview
R1 R2
Changed Files- src/video/camera.java - src/video/player.javaCode-Reviewers
Reviewed by John
Changed Files
- src/devices/screen.java
Code-Reviewers
Reviewed by Pat
R3 (New)
Changed Files
- src/video/recorder.java
Code-Reviewers
?
RevFinder
Review History
Find reviewers who have reviewed the most similar file paths
5
How do reviews with code-reviewer assignment problem impact reviewing time?
Studied Projects
Sample Reviews 357 363 378 363
#Reviews w/ Assignment Problem 10% 5% 30% 4%
Reviews with code-reviewer assignment problem significantly take 12 days longer to approve a code change.
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systems
Revie
wing
tim
e (D
ays)
Android OpenStack Qt LibreOffice
010
2030
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systems
Revie
wing
tim
e (D
ays)
Android OpenStack Qt LibreOffice
010
2030
Reviewing Time (Days)
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systems
Revie
wing
tim
e (D
ays)
Android OpenStack Qt LibreOffice
010
2030
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systemsRe
viewi
ng ti
me
(Day
s)
Android OpenStack Qt LibreOffice
010
2030
Review w/ Assignment Problem Review w/o Assignment Problem
18 Days Longer 13 Days Longer 9 Days Longer 6 Days Longer
7
RevFinder: Overview
R1 R2
Changed Files- src/video/camera.java - src/video/player.javaCode-Reviewers
Reviewed by John
Changed Files
- src/devices/screen.java
Code-Reviewers
Reviewed by Pat
R3 (New)
Changed Files
- src/video/recorder.java
Code-Reviewers
?
RevFinder
Review History
Find reviewers who have reviewed the most similar file paths
Code-Reviewer Assignment ProblemWho should review
my code?
Code change
Gerrit Code Review
Invite Reviewers
New Developer
4
Empirical Evaluation
Accuracy
“Does RevFinder accurately recommend reviewers?”
Measure: Top-k Accuracy
Ranking Performance
“Does RevFinder provide a better ranking of correct reviewers?”
15
On average, RevFinder can correctly recommend reviewers for 79% of
reviews with a top-10 recommendation.
Measure: Mean Reciprocal Rank (MRR)
On average, RevFinder can recommend reviewers with a median rank of 4.
We can use a similarity of previously reviewed file paths to recommend code-reviewers for new code changes.
5
How do reviews with code-reviewer assignment problem impact reviewing time?
Studied Projects
Sample Reviews 357 363 378 363
#Reviews w/ Assignment Problem 10% 5% 30% 4%
Reviews with code-reviewer assignment problem significantly take 12 days longer to approve a code change.
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systems
Revie
wing
tim
e (D
ays)
Android OpenStack Qt LibreOffice
010
2030
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systems
Revie
wing
tim
e (D
ays)
Android OpenStack Qt LibreOffice
010
2030
Reviewing Time (Days)
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systems
Revie
wing
tim
e (D
ays)
Android OpenStack Qt LibreOffice
010
2030
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systemsRe
viewi
ng ti
me
(Day
s)
Android OpenStack Qt LibreOffice
010
2030
Review w/ Assignment Problem Review w/o Assignment Problem
18 Days Longer 13 Days Longer 9 Days Longer 6 Days Longer
7
RevFinder: Overview
R1 R2
Changed Files- src/video/camera.java - src/video/player.javaCode-Reviewers
Reviewed by John
Changed Files
- src/devices/screen.java
Code-Reviewers
Reviewed by Pat
R3 (New)
Changed Files
- src/video/recorder.java
Code-Reviewers
?
RevFinder
Review History
Find reviewers who have reviewed the most similar file paths
Code-Reviewer Assignment ProblemWho should review
my code?
Code change
Gerrit Code Review
Invite Reviewers
New Developer
4
Empirical Evaluation
Accuracy
“Does RevFinder accurately recommend reviewers?”
Measure: Top-k Accuracy
Ranking Performance
“Does RevFinder provide a better ranking of correct reviewers?”
15
On average, RevFinder can correctly recommend reviewers for 79% of
reviews with a top-10 recommendation.
Measure: Mean Reciprocal Rank (MRR)
On average, RevFinder can recommend reviewers with a median rank of 4.
We can use a similarity of previously reviewed file paths to recommend code-reviewers for new code changes.
5
How do reviews with code-reviewer assignment problem impact reviewing time?
Studied Projects
Sample Reviews 357 363 378 363
#Reviews w/ Assignment Problem 10% 5% 30% 4%
Reviews with code-reviewer assignment problem significantly take 12 days longer to approve a code change.
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systems
Revie
wing
tim
e (D
ays)
Android OpenStack Qt LibreOffice
010
2030
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systems
Revie
wing
tim
e (D
ays)
Android OpenStack Qt LibreOffice
010
2030
Reviewing Time (Days)
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systems
Revie
wing
tim
e (D
ays)
Android OpenStack Qt LibreOffice
010
2030
Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem
Studied systemsRe
viewi
ng ti
me
(Day
s)
Android OpenStack Qt LibreOffice
010
2030
Review w/ Assignment Problem Review w/o Assignment Problem
18 Days Longer 13 Days Longer 9 Days Longer 6 Days Longer
7
RevFinder: Overview
R1 R2
Changed Files- src/video/camera.java - src/video/player.javaCode-Reviewers
Reviewed by John
Changed Files
- src/devices/screen.java
Code-Reviewers
Reviewed by Pat
R3 (New)
Changed Files
- src/video/recorder.java
Code-Reviewers
?
RevFinder
Review History
Find reviewers who have reviewed the most similar file paths
Empirical Evaluation
Accuracy
“Does RevFinder accurately recommend reviewers?”
Measure: Top-k Accuracy
Ranking Performance
“Does RevFinder provide a better ranking of correct reviewers?”
15
On average, RevFinder can correctly recommend reviewers for 79% of
reviews with a top-10 recommendation.
Measure: Mean Reciprocal Rank (MRR)
On average, RevFinder can recommend reviewers with a median rank of 4.
We can use a similarity of previously reviewed file paths to recommend code-reviewers for new code changes.
Code-Reviewer Assignment ProblemWho should review
my code?
Code change
Gerrit Code Review
Invite Reviewers
New Developer
4