1 Copyright 1995-2014, Doran Jones, Inc. and Satisfice, Inc. Paul Holland, Doran Jones, Inc. pholland@doranjones.com www.doranjones.com

My Background Managing Director, Testing Practice at Doran Jones Independent S/W Testing consultant 4/2012 - 3/2014 16+ years testing telecommunications equipment and reworking test methodologies at Alcatel-Lucent 10+ years as a test manager Presenter at STAREast, STARWest, Lets Test, EuroSTAR and CAST Facilitator at 50+ peer conferences and workshops Teacher of S/W testing for the past 6 years Teacher of Rapid Software Testing Military Helicopter pilot Canadian Sea Kings

Attributions Over the past 10 years I have spoken with many people regarding Exploratory Testing and Metrics. Much of this talk comes from the Rapid Software Testing class developed by James Bach and Michael Bolton. For the rest, I cannot directly attribute any specific aspects to any individual but all of these people (and more) have influenced my opinions and thoughts on metrics: Cem Kaner, Ross Collard, Doug Hoffman, Scott Barber, John Hazel, Eric Proegler, Dan Downing, Greg McNelly, Ben Yaroch

What Im seeing in New York City C-level executives are aware that: They have quality issues The current approach of outsourcing doesnt work When you go with the lowest bidder, you often get the lowest skill level and lowest quality of work Software is becoming more: Expected by customers - everywhere Integrated with business needs Complicated (increasing risk of failure)

Important Change in Approach There is no such thing as a Best Practice in software testing: The word best means there is nothing better If something is the best we think it cant be improved Something that works well for one project may be a bad practice for another project or company There are only Good Practices for a particular context. All practices can be improved

Important Change in Approach Stop using the term Quality Assurance We cant A the Q The term comes from manufacturing which is a different industry with different challenges Software development is as much about relationships and social interaction as it is about technology Could you assure the quality of a meeting between intelligent and empowered individuals? Use Quality Assistance or Question Asker or simply Tester

Call this Checking not Testing Observe Evaluate Report Interact with the product in specific ways to collect specific observations. Apply algorithmic decision rules to those observations. Report any failed checks. means operating a product to check specific facts about it

Acquiring the competence, motivation, and credibility to Testing is create the conditions necessary to so that you help your clients to make informed decisions about risk. evaluate a product by learning about it through experimentation, which includes to some degree: questioning, study, modeling, observation and inference, including operating a product to check specific facts about it

Tacit Test Procedures Consistency Oracles Prospective Testing Learning and Teaching Commitment Management (inc. estimation) Recruiting Helpers Managing Testing Logistics Test Tooling and Artifact Development Test Framing Bug Advocacy & Triage Project Post Mortem Creating Archival Documentation Guiding Helpers Discovery of Curios, Issues & Risks Building the Test Team Designing Checks and Tests Playing with the Product Studying Results Galumphing Configuring Product & Tools Schedule Management Study Customer Feedback Relationship Building Making Failure Productive Sympathetic Testing Maintaining Personal Health and Motivation Team Leadership Quasi-Functional Testing Playing Programmer Testing w/Simulated Conditions Testing a Simulation Creating the Test Lab Studying Specs Managing Records Playing Business Analyst Opposition Research Testability Advocacy Cultivate Credibility

Testing vs. Checking TESTING (think what testers do): the process of evaluating a product by learning about it through experimentation, which includes to some degree: questioning, study, modeling, observation and inference. CHECKING (think fact checking): the process of making evaluations by applying algorithmic decision rules to specific observations of a product.

11 Exploratory Testing Is an approach to testing that emphasizes the personal freedom and responsibility of each tester to continually optimize the value of his work by treating learning, test design, test execution and result evaluation as mutually supportive activities that run in parallel throughout the project. (applicable to any test technique) (optimize how?)

12 Session Based Test Management SBTM allows Exploratory Testing to be managed in an effective way that stands up to scrutiny It allows planning, estimating, tracking and reporting of testing projects It does NOT involve counting test cases or reporting on pass/fail counts Read more online at www.satisfice.com and www.developsense.com

The Fallacy of Repeated Tests: Clearing Mines mines

Totally Repeatable Tests Wont Clear the Minefield mines fixes

Variable Tests are Therefore More Effective mines fixes

Agile Development Everyone on the SCRUM team must be able to perform all tasks for the team Test Driven Development (TDD) fulfils the testing needs of the project Acceptance Test Driven Development (ATDD) really fulfils the testing needs Designers can test their own code just as well as a tester

Agile Development The BEST expected result from an excellent automation framework is an awesome SANITY CHECK of the software Testing by testers is still required on most projects (some exceptions like Facebook) Dedicated testers should support SCRUM teams from within the team and/or as undone work

The Agile Testing Pyramid 18 Checking

A Better Testing Pyramid 19 From: Agile Testing: A Practical Guide for Testers and Agile Teams by Lisa Crispin and Janet Gregory. Automation is in the pyramid while manual testing is not. Manual testing is performed as required. Manual testing is a cloud and is not attached to the pyramid it can occur at any phase & be any size.

20 How often do you account for your progress? If you have any autonomy at all, you can risk investing some time in learning thinking refining approaches better tests Allow some disposable time Self-management is good!

21 Allow some disposable time If it turns out that youve made a bad investmentoh well If it turns out that youve made a good investment, you might have learned something about the product invented a more powerful test found a bug done a better job avoided going down a dead end for too long surprised and impressed your manager

22 Copyright 1995-2014, Doran Jones, Inc. and Satisfice, Inc.

23 Copyright 1995-2014, Doran Jones, Inc. and Satisfice, Inc.