Complement Software Testing with Static Analysis
23
Introducing Static Analysis in a Mature Codebase John Ruberto
-
Upload
john-ruberto -
Category
Technology
-
view
530 -
download
0
description
Tells the story how we introduced static analysis to complement unit testing & code review, with a mature codebase. Previous attempts at using static analysis failed because the tools would find many "errors" that were not really interesting (coding style, etc.). Also, the tools would find many errors that existed for a long time, and were not problems for customers. This time, we concentrated on 2 things: - new code only (nightly scans - fix code that was checked in yesterday) - errors that result in run-time error Results: - Adoption by our developers - Better code quality - Developers wrote more resilient code the first time.
Transcript of Complement Software Testing with Static Analysis
Introducing Static Analysis in a Mature Codebase
John Ruberto
Born out of failure Telcom EMS/NMS system: 6M LOC C++
Very successful in the market 400K issues found through static analysis Many false positives, each issue became a debate
Financial Web App: 1M LOC Java and javascript Very successful in the market 20K issues found through static analysis Each one to be reviewed, owner identified,
communicated Effort to manage the pile became overwhelming
How we did it The case for change What can static analysis do for you? Process Design Tool Selection Deployment Change Management Backlog Reduction
Why use Static Analysis? Root Cause Analysis: defects found in system
test
Coding errors38%
Coding:Exception Handling19%
Missing Code9%
Requirements: Missing
9%
Design: Usability6%
Coding: Missing6%
Design: Func-tional4%
Coding: Browser Dif -ferences
4%Requirements: Wrong
3%
Content Localization1%
How to improve our coding practices Brain storm with developers:
Improve our unit test & code review practices Introduce Static Analysis
According to Jones*, 97% of defects can be removed by using these practices: Code Review, Static Analysis, Unit Test
* Jones, Capers, 2011, Software Quality in 2011, A survey of the State of
the Art.
What can Static Analysis do for you? Example of a defect caught by Static Analysis Potential Null Pointer Exception:
boolean foo = bar.status();
if (bar != null) { … // do something here, but too late
Process Design Minimize feedback loop
Creation -> Detection Focus on new issues Alert developers privately
Focus on developer productivity Track issue status Efficient management of issues
Process Design
Process Design, iteration 2
Tool Selection Tool Requirements
Effectiveness with our development language False positive rate with our code base Compatibility with our development environment Ability to uniquely identify defects Either a built-in defect management system, or
the ability to integrate with our system through an API
Tool Evaluation Criteria Efficacy of the checkers
How powerful the tool is for finding interesting defects
Interesting = defects that would result in runtime error
False positive rate Tool indicates a defect, where there is no defect
False negative rate Tool doesn’t find a defect, which indeed exists
Tool Evaluation - Results Two tools with complimentary results: Coverity &
Findbugs
Deployment Integrated into our build system Initial scans: establish baseline Scripting for issue assignment & notification Training Follow up, fixing glitches, etc.
4 weeks of effort
Automated Issue Management Each nightly scan results in new issues &
issues that no longer appear For each new issue:
Look up the developer that checked in that file Assign the issue to the developer Notify via email, with link to issue in the tool
For issues no longer appearing in the scans Mark them fixed in the defect system
Weekly Reports Regular communication is important for people to see
the value, and keep the change alive Focus on action
Monthly Reports Focus on Return on Investment Reinforce the “new code” focus
1/3/081/1
1/08
1/19/0
81/2
7/082/4
/082/1
2/08
2/20/0
82/2
8/083/7
/083/1
5/08
3/23/0
83/3
1/084/8
/084/1
6/08
4/24/0
85/2
/085/1
0/08
5/18/0
85/2
6/086/3
/086/1
1/08
0
100
200
300
400
500
600
700
800
900
1000
FoundFixed
Backlog Reduction First, build confidence in tool & issues Then, let the games begin
During the Olympics: 10 pts each major 5 pts each moderate 1 pt each minor &
each close/no fix Awards: Gold, Silver, Bronze
Lottery: 1 “ticket” for each fix Regular drawings Regular posting of
winners & leaderboard
Image: Creative Commons: http://www.flickr.com/photos/garryknight/
Results: 0 backlog
Jan Feb Mar Apr May Jun Jul Aug Sep0
500
1000
1500
2000
2500
3000
3500
4000
4500
Static Analysis Issue Status
TotalFixed
Isue
s
Focus on new issues first, lead to confidence in the tools and effort to eliminate the backlog
Results: better code Developers learn the anti-patterns: learn to
write more resilient code
Month 1 Month 2 Month 3 Month 4 Month 5 Month 60
50
100
150
200
250Defects Detected by Static Analysis
Results: better quality Better experience for customers
Fewer failures occurring in production
Improved Code Quality > 97% - 100% of issues fixed before customer
release
Reduced Technical Debt Eliminated over 3000 issues
Five Tips Engage Developers through the entire process
(after all, we are doing this for developers) Choose the right tool for your codebase Focus on new code, let “legacy” follow Focus on developer productivity, not finding
fault Automate issue management
Triage, Assignment, Verification, Closure
Summary of benfits 100% code coverage, every night Common errors found cheaply, supplements
code review and unit test Feedback loop results in better coding skills Identifies the exact line of code with the error Objective closure criteria, its fixed or not fixed.
