When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones...
Transcript of When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones...
![Page 1: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/1.jpg)
When Passion Obscures the Facts:
The Case forEvidence-Based Testing
Stuart Reid
Testing Solutions Group
117 Houndsditch
London EC3
UKTel: 0207 469 1500Fax: 0207 623 8459
www.testing-solutions.com
©Stuart Reid, 2010
![Page 2: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/2.jpg)
How not to write an abstract…?
For many years we have enjoyed (and sometimes endured) the debates between rival factions in the software testing community. These factions can be labelled into schools of testing, such as the context-driven, standard, analytic, and quality schools, although many simplify this to the context-driven versus the rest. Proponents are equally passionate both for and against, but passion is often used to conceal a lack of real evidence supporting the opposing positions. We see „evangelists‟ on both sides making claims and it is often those who speak the loudest, those who speak most persuasively, and those who appear the most passionate swaying opinion mainly by sheer force of personality.
This presentation will identify which claims are supported by valid evidence, which claims disagree with the available evidence, and those claims where there is currently insufficient evidence to reasonably support a claim one way or the other.
![Page 3: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/3.jpg)
Adult Testing Blogs…?
• “That is, let’s cry havoc and let slip the dogs of… testing.”
• “There are bad people in the world…”
• “I won't be put out if one of you bigger dogs gets on him first."
![Page 4: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/4.jpg)
Passion, charisma……facts?
• Have you believed everything you‟ve heard so far at this conference?
– who do you trust?
• Are you part of a „testing school‟?
– does it need defending from outsiders?
– or does your school defend the industry?
– does it make your views more extreme?
• How easily do you change your mind?
• Don‟t believe me – try using the evidence…
![Page 5: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/5.jpg)
Evidence-based medicine (EBM)
• EBM aims for the ideal that healthcare professionals should make "conscientious, explicit, and judicious use of current best evidence" in their everyday practice
– Wikipedia
Gathering the
evidence
Using the evidence
providersresearchers
![Page 6: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/6.jpg)
Evidence-based practice
what do you need to know?
finding the evidence
appraising the evidence
applying the results
![Page 7: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/7.jpg)
Medical Evidence - sources
1• meta-analysis (of randomized controlled trials)
2 • randomized controlled trials
3• controlled studies without randomisation
4• well-designed quasi-experimental studies
5• case studies, etc.
6• expert reports, opinions and/or clinical experience
![Page 8: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/8.jpg)
Gathering evidence
# studies
confidence in evidence
opinion
fact
testing is too immature to waitfor „facts‟ we don‟t need 100% confidence
![Page 9: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/9.jpg)
1
2
3
4
5
6
7
combined result
![Page 10: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/10.jpg)
Meta-analysis…..
…saves >10,000 babies from dying in the last 20 years
![Page 11: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/11.jpg)
Evidence-based testing
• EBT aims for the ideal that testing professionals should make "conscientious, explicit, and judicious use of current best evidence" in their everyday practice
– Stuart
Gathering the
evidence
Using the evidence
researchers testers
![Page 12: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/12.jpg)
Gathering the testing evidence
1 • meta-analyses
2 • academic research
3 • case studies
4 • literature searches
5 • surveys
6• expert opinion
7 • vendor reports
![Page 13: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/13.jpg)
Providing the testing evidence
• Academic evidence vs Practitioner evidence
• We need to know its:
– context - where is this evidence valid?
– level of confidence
• We need to be able to find it
– searchable tags
• In a standard format………?
![Page 14: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/14.jpg)
Evidence-based testing practice
what do you need to know?
finding the evidence
appraising the evidence
applying the results
![Page 15: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/15.jpg)
Finding & appraising the evidence
peer journals
academic conferences
text books / education personal experience
commercial conferences magazines
training courses / expert opinions anecdotes
![Page 16: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/16.jpg)
Confidence in the evidence
least confident
peer journals
academic conferences
text books / education
commercial conferences / magazines
training courses / expert opinions / anecdotes
most confident
![Page 17: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/17.jpg)
Complexity of evidence
easy to digest
peer journals
academic conferences
text books / education
commercial conferences / magazines
training courses / expert opinions / anecdotes
highly complex
![Page 18: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/18.jpg)
US Pre-schooler (2-5) diet advice
![Page 19: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/19.jpg)
US Kids (6-11) diet advice
![Page 20: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/20.jpg)
US Adult diet advice?
![Page 21: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/21.jpg)
More adult dietary advice
Best Antioxidants for Skin
The first signs of aging usually occur when there are wrinkles on the skin. There are various antioxidants for skin that can help in decreasing the aging process. The effects of antioxidants are evident after a long period of time. Consulting a skin doctor before you incorporate any of the antioxidant supplements in your diet is always the safest option you can take. Here are some best antioxidants for skin. Read more on antioxidants: how they work.
VitaminsSun skin creams and lotions are rich in vitamin E and this is so, because the vitamin E in these cosmetics prevents the damage of cell membranes. It is believed that when vitamin E is applied before sun exposure, it helps in lessening the skin redness, swelling and sun burn. A vitamin supplement that suits your skin is a good choice, as an antioxidant. Wheat germ, safflower, corn, soybean oil, mangoes, nuts and broccoli are some foods that are rich in vitamin E. Similarly, there are Vitamin C (ascorbic acid) and Vitamin A (1st antioxidant used for anti-aging) antioxidants, that one can use to control aging process. While vitamin C rich foods are cereals, beef, poultry, and fish vitamin A is found abundantly in potatoes, carrots, milk, egg yolks, and cheese. Read more on best antioxidant supplements.
Cochrane Review Meta-Analysis:“antioxidant supplements--vitamins A, C, and E; beta-carotene; and selenium--do not prevent heart disease or cancer, do not forestall death, and may even increase mortality risk slightly.”
From http://www.buzzle.com/articles/best-antioxidants.html
“Nice skin, shame she’s dead!”
![Page 22: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/22.jpg)
A naïve view of the „profession‟
capability
number of testers
![Page 23: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/23.jpg)
Actual IT „Professionals‟
capability
number of IT ‘professionals’ “Process Maturity Profile of the Software Community 2001 Year End Update”, SEI, 2002.
![Page 24: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/24.jpg)
Are testers more capable?
capability
number of testers/developers
![Page 25: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/25.jpg)
The profile for testers?
capability
number of testers
![Page 26: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/26.jpg)
Testers are not all the same…
20% 20%60%
capability
number of testers
![Page 27: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/27.jpg)
Complexity of evidence
easy to digest
peer journals
academic conferences
text books / education
commercial conferences / magazines
training courses / expert opinions / anecdotes
highly complex
![Page 28: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/28.jpg)
Usability of evidence
capability
complexity of evidence
the boundary of understanding
![Page 29: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/29.jpg)
Presenting the evidence to testers
Need to be provided with simple rules or guidelines
Should already be competently finding and appraising evidencePLUSShould consider contributing to the guidelines on current best evidence
May need to be trained to find and appraise
evidence
![Page 30: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/30.jpg)
So, is there any evidence already out there……….?
![Page 31: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/31.jpg)
Useful Questions/Evidence?
• Are formal reviews an efficient way of finding defects or are there better alternatives?
• Do testing tools save you money, cost you money or are they simply a distraction?
• Does using standards reduce the probability of project failure?
• Is a certified tester more effective than those without industry certification?
• Will exploratory testing detect more bugs than scripted approaches?
![Page 32: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/32.jpg)
For each Question…
confidence
foragainst
35% defect removal efficiency
![Page 33: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/33.jpg)
Formal reviews?
confidence
foragainst
85% max. defect removal efficiency
Capers Jones
Software Engineering Best Practices – Lessons from Successful Projects in the Top Companies, McGraw Hill, 2010.
Based on data from more than 13,000 projects 600 corporations and more than 30 government organizations.
![Page 34: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/34.jpg)
System Testing?
confidence
foragainst
40% max. defect removal efficiency
Capers Jones
![Page 35: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/35.jpg)
Automated Static Analysis?
confidence
foragainst
87% max. defect removal efficiency
Capers Jones
![Page 36: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/36.jpg)
Testing tools?
• EuroSTAR 2007 survey
– approx. 600 respondents
• 42% achieved significant benefit
• 33% had reasonable benefit
• 16% had some benefit
• 10% had no benefit from the tools
![Page 37: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/37.jpg)
Synergistic Approach?
confidence
foragainst
99% max. defect removal efficiency
Capers Jones
•Formal Reviews•System Testing•Static Analysis
![Page 38: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/38.jpg)
Standards?
confidence
foragainst
8.33% average defect removal
Capers Jones
![Page 39: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/39.jpg)
Certified testers?
confidence
foragainst
8.5% average defect removal
Capers Jones
![Page 40: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/40.jpg)
Certified testers?
• EuroSTAR 2007 survey
– approx. 600 respondents
• 35% said that they knew more about testing
• 26% got a better job or contract
• 17% were better recognised for their testing skills
• 1% reported higher pay
• 23% said it had no effect
![Page 41: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/41.jpg)
Exploratory testing?
• Will exploratory testing detect more bugs than scripted approaches?
– Yes, if there‟s no time to script and run tests
• Not a useful question
– the two approaches should ideally be used as complementary approaches
![Page 42: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/42.jpg)
Evidence-based testing
what do you need to know?
finding the evidence
appraising the evidence
applying the results
![Page 43: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/43.jpg)
Conclusions
• We should base our decisions on all the available evidence
– not restrict ourselves to one or two favourite sources (such as ourselves)
• More research is needed
– and the results need to be made accessible to all
• Testers are not a homogeneous group
– different capabilities have different needs
• It doesn‟t help the discipline if those with different views spend too much of their time arguing amongst themselves
![Page 44: When Passion Obscures the Facts: The Case for Evidence ......defect removal efficiency Capers Jones Software Engineering Best Practices –Lessons from Successful Projects in the Top](https://reader035.fdocuments.in/reader035/viewer/2022071018/5fd15f215956932c3a2d1416/html5/thumbnails/44.jpg)
An (academic) EBT Bibliography
• Procedures for Performing Systematic Reviews, Barbara Kitchenham , NICTA Technical Report 0400011T.1, July, 2004.
• Reviewing 25 Years of Testing Technique Experiments, N Juristo et al, Empirical Software Engineering, 9, 7–44, 2004.
• Evidence-Based Software Engineering for Practitioners T Dyba et al, IEEE Software, January / February 2005.
• In Search of What We Experimentally Know about Unit Testing, N Juristo et al, IEEE Software, November/December 2006.
• What Do We Know about Defect Detection Methods?, P Runeson et al, IEEE Software, May/June 2006.
• Software for Dependable Systems: Sufficient Evidence?, D Jackson, MIT, NSA HCSS, · Baltimore, MD, · May 9, 2007.
• Systematic literature reviews in software engineering – A systematic literature review, Information and Software Technology 51, 7–15, 2009.
• A systematic review on regression test selection techniques, E Engström et al, Information and Software Technology, 52, 14–30, 2010.