Actionable Ideas for Improving Software Quality · 4 We’ve been trying to improve quality for 30+...
Transcript of Actionable Ideas for Improving Software Quality · 4 We’ve been trying to improve quality for 30+...
V2.2 | 2017-10-02
Actionable Ideas for Improving Software Quality
2
Software Trends
Actionable Ideas for Improving Software Quality
• Hundreds of files• Thousands of Lines of Code• Lots of “connections”• Lots of safety concerns
SOFTWAREEMBEDDED SOFTWARE
+SIZE
ENTERPRISESOFTWARE
+Connections
• Thousands of files• Millions of Lines of Code• Few “connections”• Few safety concerns
+Safety
3
Management Stress
Larger Code Bases
Longer Life Cycles
Larger and Distributed Teams
Development Stress
Many bugs are found close to release
Code changes break existing features
Full testing takes a long time
Business Stress
Feature Releases are Slow
Customers Report Bugs
Things Could Be Better
Software Stress Points
Actionable Ideas for Improving Software Quality
4
We’ve been trying to improve quality for 30+ years
1982, let’s do a better job defining the problem> Early Modeling and Data Dictionary Tools
1983, let’s build a new language: Ada
1991, let’s build a new language: C++
1996, let’s draw pictures and get the code for free > Object Oriented Design, UML
2001, let’s build and integrate a little at a time> The Agile Manifesto, Continuous Integration
2002, let’s find bugs by modeling the code> Algorithmic Static Analysis
Tool Trends
Actionable Ideas for Improving Software Quality
5
What’s the Next Hot Trend?
Actionable Ideas for Improving Software Quality
Everyone Wants Better Quality Software
A Push-button Tool is probably not going to Solve the Problem
A new Development Paradigm is probably not going to Solve the Problem
Improved Testing is the Best way to Improve Software Quality
How can we Improve our Testing?
6
What’s the goal of testing?
Key Questions to ask
Actionable Ideas for Improving Software Quality
Who Builds and Runs Tests?
How Often do we Test?
Is our Testing Automated?
When are we Done Testing?
7
Common Answer
Application Correctness
How to Improve
Component Correctness
Benefits of Component Testing
Fewer Integration Bugs
Easier Re-Factoring of Modules
Why are we Testing?
What is the Goal of Testing?
8
Test Components in Isolation
Provides a formal check of component behavior, prior to integration
Allows Code to be more easily Re-Used and Re-Factored
Complements Functional Testing> All Component Tests might pass, but the system might not function!
For New Systems, concentrate on
Algorithms (e.g. fuel injection algorithm)
Low-Level building blocks (e.g. a message handler)
For Legacy Systems, concentrate on
Components with frequent changes and/or high bug counts
Components with high complexity
How to Implement Component Testing
What is the Goal of Testing?
9
Common Answer: The QA Group
How to Improve:
Make Everyone a “Tester”> Designers, Developers, Integrators, QA Engineers
Benefits
Spreads the Responsibility for Quality to Everyone
Leverages System Knowledge
Who’s Testing?
Who Builds and Runs Tests?
10
Implement Common Processes and Tools
Make Tests Easy to Build and Run
Deploy a automation server to process test jobs
Jenkins is an ideal platform,
Virtualization allows the infrastructure to scale
Make Tests and Results Easy to Understand
Publish results to the whole team, Track and publish testing trends
Make it possible to Debug Failing Tests
Testing framework must integrate with the debugger
How to Make Everyone A Tester
Who Builds and Runs Tests?
11
Common Answer: Periodically
How To Improve
Test Continuously, on every source change
Benefits of Continuous Test
Changes that break tests are never integrated
Tests are always passing on main branch
QA concentrates on Testing New Features …
When do we Test?
How Often do We Test?
12
Tests must be easy to run
A common platform is key
Tests must be small, atomic, and fast
Must be able to run independently
Parallel Testing must be possible
“We don’t have enough hardware” cannot be an excuse
Virtualization is a key building block
Invest in System-level, Simulation
How to Test Continuously
How Often do We Test?
13
Common Answer: When we hit the release date
How to Improve
Test Requirements not Code
Measure Code Coverage
Benefits
Test are more meaningful
Completeness is easy to measure
Estimates of remaining work are more accurate
Are we Done Yet?
When are we Done Testing?
14
Identify Requirements
What are we doing?
Identify Edge Cases for new Behavior
These are your tests!
Track Tests to Requirements
Ensure that all Requirements have Tests
Track Requirement changes to ensure Tests are Changed
How to Test Requirements
When are we Done Testing?
15
Measure Code Coverage
Coverage is a “free” metric
Measure during all flavors of Testing
Track Progress Over Time
Analyze Results
Develop a plan to improve coverage levels
How to Measure Testing Completeness
When are we Done Testing?
16
Common Answers: No idea, ask Bob
Correct Answer:
Push this button / Run this script
Benefits of Automated Testing
Any team member can run any test
Tests get re-used hundreds of thousands of times
How do I run this Test?
Is our Testing Automated and Repeatable?
17
Test Environment must be easy to setup
Automated Initialization of > Hardware, Software, Test Equipment
Test Purpose must be easily understood
NO code or script reading required
Tests must not be fragile
If a test fails “frequently” it must be improved
Test Results must be easy to interpret
Single Pass or Fail result
Tests must be able to be run manually
With the ability to debug
How to Implement Push Button Testing
Is our Testing Automated and Repeatable?
18
Summary
Actionable Ideas for Improving Software Quality
Unknown Complete
What’s our Code and Requirements Coverage?
No Idea Automatically
How do we Run Tests?
Per Release Per Change
How Often do we Test?
QA Everyone
Who Runs Tests?
19 © 2017. Vector Informatik GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V6.14 | 2017-04-18
For more information about Vectorand our products please visit
www.vector.com