Download - [Paul Holland] Trends in Software Testing

Transcript
Page 1: [Paul Holland] Trends in Software Testing

1

Copyright © 1995-2014, Doran Jones, Inc. and Satisfice, Inc.

Paul Holland, Doran Jones, Inc.

[email protected]

www.doranjones.com

Page 2: [Paul Holland] Trends in Software Testing

My Background Managing Director, Testing Practice at Doran Jones

Independent S/W Testing consultant 4/2012 - 3/2014

16+ years testing telecommunications equipment and reworking test methodologies at Alcatel-Lucent

10+ years as a test manager

Presenter at STAREast, STARWest, Let’s Test, EuroSTAR and CAST

Facilitator at 50+ peer conferences and workshops

Teacher of S/W testing for the past 6 years

Teacher of Rapid Software Testing

Military Helicopter pilot – Canadian Sea Kings

Page 3: [Paul Holland] Trends in Software Testing

Attributions

Over the past 10 years I have spoken with many people regarding Exploratory Testing and Metrics. Much of this talk comes from the Rapid Software Testing class developed by James Bach and Michael Bolton.

For the rest, I cannot directly attribute any specific aspects to any individual but all of these people (and more) have influenced my opinions and thoughts on metrics:

Cem Kaner, Ross Collard, Doug Hoffman, Scott Barber, John Hazel, Eric Proegler, Dan Downing, Greg McNelly, Ben Yaroch

Page 4: [Paul Holland] Trends in Software Testing

What I’m seeing in New York City

“C”-level executives are aware that: They have quality issues

The current approach of outsourcing doesn’t work

When you go with the lowest bidder, you often get the lowest skill level and lowest quality of work

Software is becoming more: Expected by customers - everywhere

Integrated with business needs

Complicated (increasing risk of failure)

Page 5: [Paul Holland] Trends in Software Testing

Important Change in Approach

There is no such thing as a “Best Practice” in software testing: The word “best” means there is nothing better

If something is the best we think it can’t be improved

Something that works well for one project may be a bad practice for another project or company

There are only “Good” Practices for a particular context.

All practices can be improved

Page 6: [Paul Holland] Trends in Software Testing

Important Change in Approach

Stop using the term “Quality Assurance” We can’t “A” the “Q”

The term comes from manufacturing which is a different industry with different challenges

Software development is as much about relationships and social interaction as it is about technology

Could you assure the quality of a meeting between intelligent and empowered individuals?

Use “Quality Assistance” or “Question Asker” or simply “Tester”

Page 7: [Paul Holland] Trends in Software Testing

Call this “Checking” not Testing

Observe Evaluate Report

Interact with the product in specific ways to collect specific observations.

Apply algorithmic decision rules to those observations.

Report any failed checks.

means

operating a product to check specific facts about it…

Page 8: [Paul Holland] Trends in Software Testing

Acquiring the competence, motivation, and credibility to…

Testing is…

create the conditions necessary to…

…so that you help your clients to make informed decisions about risk.

evaluate a product by learning about it through experimentation, which includes to

some degree: questioning, study, modeling, observation and inference, including…

operating a product to check specific facts about it…

Page 9: [Paul Holland] Trends in Software Testing

Tacit

Test Procedures Consistency

Oracles

Prospective Testing

Learning and

Teaching

Commitment

Management

(inc. estimation)

Recruiting

Helpers

Managing Testing

Logistics

Test Tooling and

Artifact Development

Test Framing

Bug

Advocacy

& Triage

Project

Post Mortem

Creating Archival

Documentation

Guiding Helpers

Discovery of

Curios, Issues &

Risks

Building the

Test Team

Designing

Checks and Tests

Playing with

the Product

Studying

Results

Galumphing

Configuring

Product & Tools

Schedule

Management

Study

Customer

Feedback

Relationship

Building

Making Failure

Productive

Sympathetic

Testing

Maintaining Personal

Health and Motivation

Team

Leadership

Quasi-Functional

Testing

Playing

Programmer

Testing w/Simulated

Conditions

Testing a Simulation

Creating the Test Lab

Studying Specs

Managing Records

Playing

Business Analyst

Opposition

Research

Testability Advocacy

Cultivate Credibility

Page 10: [Paul Holland] Trends in Software Testing

Testing vs. Checking

TESTING (think “what testers

do”): the process of evaluating a product by learning about it through experimentation, which includes to some degree: questioning, study, modeling, observation and inference.

CHECKING (think “fact

checking”): the process of making evaluations by applying algorithmic decision rules to specific observations of a product.

Page 11: [Paul Holland] Trends in Software Testing

11

Exploratory Testing Is…

an approach to testing…

that emphasizes the personal freedom and responsibility of each tester to continually optimize the value of his work…

by treating learning, test design, test execution and result evaluation as mutually supportive activities that run in parallel throughout the project.

(applicable to any test technique)

(optimize how?)

Page 12: [Paul Holland] Trends in Software Testing

12

Session Based Test Management

SBTM allows Exploratory Testing to be managed in an effective way that stands up to scrutiny

It allows planning, estimating, tracking and reporting of testing projects

It does NOT involve counting test cases or reporting on pass/fail counts

Read more online at www.satisfice.com and www.developsense.com

Page 13: [Paul Holland] Trends in Software Testing

The Fallacy of Repeated Tests: Clearing Mines

mines

Page 14: [Paul Holland] Trends in Software Testing

Totally Repeatable Tests Won’t Clear the Minefield

mines fixes

Page 15: [Paul Holland] Trends in Software Testing

Variable Tests are Therefore More Effective

mines fixes

Page 16: [Paul Holland] Trends in Software Testing

Agile™ Development

Everyone on the SCRUM team must be able to perform all tasks for the team

Test Driven Development (TDD) fulfils the testing needs of the project

Acceptance Test Driven Development

(ATDD) really fulfils the testing needs

Designers can test their own code just as well as a tester

Page 17: [Paul Holland] Trends in Software Testing

Agile™ Development

The BEST expected result from an excellent automation framework is an awesome SANITY CHECK of the software

Testing by testers is still required on most projects (some exceptions like Facebook)

Dedicated testers should support SCRUM teams from within the team and/or as “undone” work

Page 18: [Paul Holland] Trends in Software Testing

The Agile “Testing” Pyramid

18

Checking

Page 19: [Paul Holland] Trends in Software Testing

A Better “Testing” Pyramid

19

From: Agile Testing: A Practical Guide for Testers and Agile Teams by Lisa Crispin and

Janet Gregory.

Automation is in the pyramid

while manual testing is not.

Manual testing is

performed as required.

Manual testing is a cloud

and is not attached to the

pyramid – it can occur at

any phase & be any size.

Page 20: [Paul Holland] Trends in Software Testing

20

How often do you account for your progress?

If you have any autonomy at all, you can risk investing some time in learning

thinking

refining approaches

better tests

Allow some disposable time Self-management is good!

Page 21: [Paul Holland] Trends in Software Testing

21

Allow some disposable time

If it turns out that you’ve made a bad investment…oh well

If it turns out that you’ve made a good investment, you might have learned something about the product

invented a more powerful test

found a bug

done a better job

avoided going down a dead end for too long

surprised and impressed your manager

Page 22: [Paul Holland] Trends in Software Testing

22

Copyright © 1995-2014, Doran Jones, Inc. and Satisfice, Inc.

Page 23: [Paul Holland] Trends in Software Testing

23

Copyright © 1995-2014, Doran Jones, Inc. and Satisfice, Inc.