Verification at scale: Fitting static code analysis into continuous integration
Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software
-
Upload
moritz-beller -
Category
Software
-
view
714 -
download
0
Transcript of Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software
Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software
Moritz Beller @Inventitech
Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software
Moritz Beller @Inventitech
Radjino Bholanath, Andy Zaidman
Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software
Moritz Beller @Inventitech
Radjino Bholanath, Andy Zaidman
Shane McIntosh
Automatic Static Analysis Tools (ASATs)
Image: https://pixabay.com/static/uploads/photo/2015/06/12/18/31/cute-807306_960_720.png
Automatic Static Analysis Tools (ASATs)
Image: https://pixabay.com/static/uploads/photo/2015/06/12/18/31/cute-807306_960_720.png
Automatic Static Analysis Tools (ASATs)
Image: https://pixabay.com/static/uploads/photo/2015/06/12/18/31/cute-807306_960_720.png
Automatic Static Analysis Tools (ASATs)
Image: https://pixabay.com/static/uploads/photo/2015/06/12/18/31/cute-807306_960_720.png
Automatic Static Analysis Tools (ASATs)
Image: https://pixabay.com/static/uploads/photo/2015/06/12/18/31/cute-807306_960_720.png
Automatic Static Analysis Tools (ASATs)
Image: https://pixabay.com/static/uploads/photo/2015/06/12/18/31/cute-807306_960_720.png
RQ1: How Prevalent Are ASATs?
Image: http://www.valueinvestasia.com/wp-content/uploads/2015/03/odd-one-out.jpg
Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
RQ1: How Prevalent Are ASATs?
Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
RQ1: How Prevalent Are ASATs?
Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
RQ1: How Prevalent Are ASATs?
Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
RQ1: How Prevalent Are ASATs?
Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
RQ1: How Prevalent Are ASATs?
Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
1) (Automated) investigation of repository information is an approximation of real ASAT use
RQ1: How Prevalent Are ASATs?
Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
1) (Automated) investigation of repository information is an approximation of real ASAT use
2) We cannot infer how a project uses ASATs from a repository analysis alone
RQ1: How Prevalent Are ASATs?
General Defect Classification (GDC)
RQ1: 9
Checkstyle, FindBugs, PMD, ESLint, JSCS, JSHint, JSL, PYLint, RuboCop
General Defect Classification (GDC)
1,825
RQ1: 9
Checkstyle, FindBugs, PMD, ESLint, JSCS, JSHint, JSL, PYLint, RuboCop
RQ2.1: How Popular Are Certain ASATs?
Tool Language Configuration Files
Checkstyle Java 18,785
FindBugs Java 2,090
PMD Java 7,458
ESLint JavaScript 4,435
JSCS JavaScript 11,677
JSHint JavaScript 108,770
JSL JavaScript 862
Pylint Python 4,071
RuboCop Ruby 10,066
Total - 168,405
RQ2.1: How Popular Are Certain ASATs?
Tool Language Configuration Files
Checkstyle Java 18,785
FindBugs Java 2,090
PMD Java 7,458
ESLint JavaScript 4,435
JSCS JavaScript 11,677
JSHint JavaScript 108,770
JSL JavaScript 862
Pylint Python 4,071
RuboCop Ruby 10,066
Total - 168,405
35% 65%
ASATs perform poorly at finding functional defects. Wagner et al.
RQ 2.2: Which Rules Do Developers Enable?
RQ 2.2: Which Rules Do Developers Enable?
0%
10000000%
20000000%
30000000%
40000000%
50000000%
60000000%
70000000%
80000000%
90000000%
100000000%
Checkstyle ESLint FindBugs JSCS JSHint JSL PMD Pylint RuboCop
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
0%
10000000%
20000000%
30000000%
40000000%
50000000%
60000000%
70000000%
80000000%
90000000%
100000000%
Checkstyle ESLint FindBugs JSCS JSHint JSL PMD Pylint RuboCop
RQ 2.2: Which Rules Do Developers Enable?
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
RQ 2.2: Which Rules Do Developers Enable?
We: This is great!
Image: Phil Skipper, The Potentialist,http://i2.wp.com/thepotentiality.com/assets/media/Kurt-Enthusiasm.jpg
RQ 2.2: Which Rules Do Developers Enable?
We: This is great!
Image: Phil Skipper, The Potentialist,http://i2.wp.com/thepotentiality.com/assets/media/Kurt-Enthusiasm.jpg
RQ 2.2: Which Rules Do Developers Enable?
ASAT Developers*: Don't care.
Image: NiBiS, http://bidab.nibis.de/PICT/traurig.jpg
RQ 2.2: Which Rules Do Developers Enable?
ASAT Developers*: Don't care.
Image: NiBiS, http://bidab.nibis.de/PICT/traurig.jpg
RQ 2.3: How Good Is The Default?
But, typically only have one change from the default …
Most ASAT configurations deviate from the default.
RQ 2.3: How Good Is The Default?
But, typically only have one change from the default … ● - Addition
Most ASAT configurations deviate from the default.
RQ 2.3: How Good Is The Default?
But, typically only have one change from the default … ● - Addition● - Deletion
Most ASAT configurations deviate from the default.
RQ 2.3: How Good Is The Default?
But, typically only have one change from the default … ● - Addition● - Deletion● - Re-configuration/Custom analysis
Most ASAT configurations deviate from the default.
● Why do projects favor certain GDC rule categories from ASATs?
● Can ASAT developers better fit their default configurations to their users' needs?
RQ2: Open Questions
● Why do projects favor certain GDC rule categories from ASATs?
● Can ASAT developers better fit their default configurations to their users' needs?
● Do 'dynamic' languages require more ASAT use?
RQ2: Open Questions
● Why do projects favor certain GDC rule categories from ASATs?
● Can ASAT developers better fit their default configurations to their users' needs?
● Do 'dynamic' languages require more ASAT use?
RQ2: Open Questions
RQ3: How Do ASAT Configurations Evolve?
Image: Daimler AG, http://5komma6.mercedes-benz-passion.com/wp-content/uploads/2013/06/s-class-lineup.jpg
● Why do ASAT configurations not typically evolve?
● How are ASATs used in a CI-environment?
RQ 3: Open Questions
Moritz Beller
@Inventitech
Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software
Moritz Beller
@Inventitech
Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software
Moritz Beller
@Inventitech
Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software
Moritz Beller
@Inventitech
Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software