A Qualitative Study on Performance Bugs (MSR 2012)
42
Shahed Zaman Ahmed E. Hassan SAIL, Queen’s University A Qualitative Study on Performance Bugs Bram Adams MCIS, École Polytechnique de Montréal
-
Upload
bram-adams -
Category
Technology
-
view
434 -
download
1
description
Software performance is one of the important qualities that makes software stand out in a competitive market. However, in earlier work we found that performance bugs take more time to fix, need to be fixed by more experi- enced developers and require changes to more code than non-performance bugs. In order to be able to improve the resolution of performance bugs, a better understanding is needed of the current practice and shortcomings of reporting, reproducing, tracking and fixing performance bugs. This paper qualitatively studies a random sample of 400 performance and non-performance bug reports of Mozilla Firefox and Google Chrome across four dimensions (Impact, Context, Fix and Fix validation). We found that developers and users face problems in reproducing performance bugs and have to spend more time discussing performance bugs than other kinds of bugs. Sometimes performance regressions are tolerated as a trade- off to improve something else. http://sail.cs.queensu.ca/publications/pubs/MSR2012_Zaman.pdf
Transcript of A Qualitative Study on Performance Bugs (MSR 2012)
Shahed Zaman Ahmed E. HassanSAIL, Queen’s University
A Qualitative Study onPerformance Bugs
Bram%AdamsMCIS,&École&Polytechnique&de&Montréal
Bugs, Bugs, Bugs!
2
Costly Affect%
Reputa4on
Bugs, Bugs, Bugs!
2
482%bugs/week
Most%research%treats%all%bugs%equally!%
Does%this%make%sense?
Bugs, Bugs, Bugs!
4
Performance%Bugproblem&in&efficiency/speed&at&which&a&system&performs&at&runDEme=
5hHp://www.pcmag.com/arEcle2/0,2817,2325772,00.asp
systemEkilling%
performance%problems
Security versus Performance Bugs:
A Case Study on
Shahed&Zaman,&Bram&Adams,&Ahmed&E.&HassanSoUware&Analysis&and&Intelligence&Lab&(SAIL),&Queen’s&University
6
MSR%2011%Quan4ta4ve%Study
Performance&bugs&are&fixed&by&more&experienced&
developers
People
Performance&bug&fixes&are&more&complex
Bug%FixTime
&&&&&Performance&bugs&take&more%4me&to&fix
7
Why?
8
Shahed Zaman Ahmed E. HassanSAIL, Queen’s University
A Qualitative Study onPerformance Bugs
Bram%AdamsMCIS,&École&Polytechnique&de&Montréal
Chrome®
44,997%bugs
510%performance%bugs
295,198%bugs
7,603%performance%bugs
Performance%Bugs NonEperformance%bugs
100%Bugs
100%Bugs
=
200%Bugs
+
100%Bugs
100%Bugs
=
200%Bugs
+
Study Setup
11
bug%reports
‘slow’,%‘perf’,%
‘hang’
Our%heuris4cs%have:
100&±&10%%precision83&±&10%%recall
Our%heuris4cs%have:
90&±&10%%precision97&±&10%%recall
12
analyze
13
comments
bug%report
+%a]achments
14
update&taxonomyanalyze
think
14
IterativeProcess
update&taxonomyanalyze
think
4 Dimensions(20 subdimensions)
impact&on&the&user
context&of&the&bug
bug&fix
bug&fix&validaEon
4 Dimensions(20 subdimensions)
impact&on&the&user
context&of&the&bug
bug&fix
bug&fix&validaEon
Findings
is®ression
blocks&release
WFM&after&a&long&Time
People&Threaten&to&switch
0 5 10 15 20 25
PerformanceNonDPerf.
%
sta4s4cally%
significant%
difference
No Traceability
17
WorksForMe
Findings
is®ression
blocks&release
WFM&after&a&long&Time
scares&users&away
0 5 10 15 20 25
PerformanceNonDPerf.
%
4 Dimensions(20 subdimensions)
impact&on&the&user
context&of&the&bug
bug&fix
bug&fix&validaEon
Findingshas&measurements
has&test&cases
has&stack&traces
Replication&Problems
Reported&by&Member
Duplicate&Bugs
0 10 20 30 40 50 60
PerformanceNonDPerf.
%
21
automa4c%performance%
regression%tes4ng
21
automa4c%performance%
regression%tes4ngperformance&
regression&idenEfied&automaEcally
solicits%comparison%to%
other%browsers
22
Findingshas&measurements
has&test&cases
has&stack&traces
has&replication&problems
Reported&by&Member
Duplicate&Bugs
0 10 20 30 40 50 60
PerformanceNonDPerf.
%
Replication Problems
24
incomplete&report
<empty><moreempty>
plugins/addDons
Hard to Track Bugs ofExternal Applications
25
Findingshas&measurement&data
has&test&cases
has&stack&traces
has&replication&problems
is&reported&by&developer
is&a&duplicate
0 10 20 30 40 50 60
PerformanceNonDPerf.
%
4 Dimensions(20 subdimensions)
impact&on&the&user
context&of&the&bug
bug&fix
bug&fix&validaEon
Findings
has&discussion
Depends&on&Other&Bugs
Blocks&Other&Bugs
0 10 20 30 40 50
PerformanceNonDPerf.
%
Performance Bugs Require Collaboration
29
understand&the&problem
idenEfy&bug&fix
reach&consensus
me&too
Findings
has&discussion
depends&on&other&bug
blocks&other&bug
0 10 20 30 40 50
PerformanceNonDPerf.
%
Rebasing Performance
31
adjust&baseline
4 Dimensions(20 subdimensions)
impact&on&the&user
context&of&the&bug
bug&fix
bug&fix&validaEon
33
NO&difference&between&patch&discussions&and&reviews&of
performance&and&nonDperformance&bugs
is®ression + +blocks&release + +WFM&aUer&a&long&Eme + +scares&users&away + +
has&measurements + +has&test&cases + +contains&stacktrace + +has&replicaEon&problems + +is&reported&by&a&developer ? +is&a&duplicate E E
has&discussion + +depends&on&other&bug + +blocks&other&bug + E
impact&on&the&user
context&of&the&bug
bug&fix
is®ression + +blocks&release + +WFM&aUer&a&long&Eme + +scares&users&away + +
has&measurements + +has&test&cases + +contains&stacktrace + +has&replicaEon&problems + +is&reported&by&a&developer ? +is&a&duplicate E E
has&discussion + +depends&on&other&bug + +blocks&other&bug + E
impact&on&the&user
context&of&the&bug
bug&fix
MSR%2011%Quan4ta4ve%Study%(reprise)
Performance&bugs&are&fixed&by&more&experienced&
developers
Performance&bug&fixes&are&more&complex
&&&&&Performance&bugs&take&more%4me&to&fix
36
replication problems
more dependencies
more discussion
WFM after a long time
more release blocking
users leaving
Threats to Validity
37
human&bias
only&4&dimensions&&&20&subDdimensions
only&2&subjectsystems
‘slow’,%‘perf’,%
‘hang’
heurisEcs&for&performance&bug&idenEficaEon
Chrome®
44,997%bugs510%performance%bugs
295,198%bugs7,603%performance%bugs
Performance%Bugs NonEperformance%bugs
100%Bugs
100%Bugs=
200%Bugs
+100%Bugs
100%Bugs=
200%Bugs
+
Study Setup
no traceability
no reproducabilitydiscussion!
4 Dimensions(20 subdimensions)
impact&on&the&user
context&of&the&bug
bug&fix
bug&fix&validaEon
is®ression + +blocks&release + +WFM&aUer&a&long&Eme + +scares&users&away + +has&measurements + +has&test&cases + +contains&stacktrace + +has&replicaEon&problems + +is&reported&by&a&developer ? +is&a&duplicate E Ehas&discussion + +depends&on&other&bug + +blocks&other&bug + E
impact&on&the&user
context&of&the&bug
bug&fix
MSR%2011%Quan4ta4ve%Study%(reprise)Performance&bugs&are&fixed&
by&more&experienced&developers
Performance&bug&fixes&are&more&complex
&&&&&Performance&bugs&take&more%4me&to&fix
37
replication problems
more dependencies
more discussion
WFM after a long time
more release blocking
users leaving
