10 signs your testing is not enough
Transcript of 10 signs your testing is not enough
10 signs your testing is not enough
How do we know?
How do we know if we tested enough?
• Frequently asked question
• Fiascos
• Assuring the minimum acceptable level of quality is a must
• Need to analyze if you should be investing more resources, time, or money in your software testing
• “Not enough” vs “Not Good Enough”
Background
Background
Background
Another horror story
Knight Capital
• Only one defect in the trading algorithm for Knight Capital caused a loss of $ 440 million in about 45 minutes.
• That loss is 3 times the company’s annual earnings.
• The shock that followed caused Knight Capital’s stock to lose 75% of its value in 1 business day.
Infinium shut their algorithm down after just three seconds of trading, but it had already lost more than $1 million.
Lack of understanding
Management misunderstanding
• We don’t have bugs!
• Testing is expensive!• It is an expense• What is the alternative?• Preventive by nature
• We don’t need testers!• We do DevOps!• We do Agile!• Testing and Dev – 2 different skill sets
Positive trend
Budget for software testing grows
How good is your testing?
1: Glitches
Excessive amount of Production bugs:
• Unknown• How long have they been there? 1, 2 or more Test cycles?• Why not discovered earlier?• Not only investing more in testing but also considering changing
Testing Strategy
• Known – Why?
• If found by Customers - Worst from business point of view • Bad PR • May have disastrous effects• Most expensive to fix
2: Bugs in UAT
Too much bugs discovered in Acceptance Testing:
• UAT supposed to build confidence in product
• Last stage before releasing • goal is to find deviations from intended design • Goal is NOT to find bugs
3: Bug counts grow
Are you aware of such trends?
• If bug counts grow you should analyze asap to find reasons
• If you don’t take measures your project/product might go in a Death Spiral
But do you have the Metrics to see trends?
• Basic Metrics are: • easy to apply • Great for analyzing trends
Quality Metrics
Quality Metrics
Project – Level Metrics
1. Requirements and Requirement Coverage – How to measure the coverage?
2. Defect Open and Close Rates
Quality Metrics
Quality Metrics
Department – Level Metrics
1. Mean Time To Detect (MTTD) and Mean Time To Repair – MTTD measures how long it takes QA professionals to find a problem, while the MTTR demonstrates the amount of time needed to effectively address it.
Quality Metrics
Quality Metrics
Company – Level Metrics
1. Number of Issues reported by Customers
4: Not investing enough
Investment in Testing is below industry average
• Always good to know what competition is doing
• You are investing less than similar companies
5: No clear criteria
No definition of what “enough” testing is:
• Coverage• Execution• Acceptable levels of defects
• Major• Minor
• Plan for fixing known defects• etc
6: Listen to Testers’ NO
You receive negative advice from Testers to go ahead:
• Always listen to employees
• Value of good tester
• Analyze the legitimated issues raised
7: Weak Prevention
Get the Team involved in:
• Requirements assessments
• Definition of coding standards/ good practices
• Reviews/Inspections – Static Testing
Prevention
8: No Dev involvement
You don’t have Unit testing in place?
• Software quality is a team responsibility
• Unit testing is natural to developers
• Find bugs earlier
Picture so far
Choices and effects
9: Time problems
9: Time problems
Testing periods are regularly reduced
9: Time problems
Deadlines problems regularly?
• How often does it happen that testing is squeezed
• Lack of time for testing is a symptom of deeper project, process and organizational issues.
• Make sure you make enough time for Testing
10: Testers keep quitting
Dissatisfaction of Test Team:
• Ultimate sign!
• Happy employees – better performance
• Can be financial
• Can be also professional• Not enough time for testing• Not enough resources• Frustration levels grow up
Testing is enough!
Make it even better!
1. Use testing as a high – level risk management
2. Test reusability
3. Automate
Challenges
in the testing…
Questions?