T-76.5613 Software Testing and Quality Assurance and... · HELSINKI UNIVERSITY OF TECHNOLOGY...

HELSINKI UNIVERSITY OF TECHNOLOGY

T-76.5613 Software Testing and Quality Assurance

Lecture 10, 11.10.2006

Exploratory Testing (ET)vs.

Test-Case Based (scripted) Testing

Juha ItkonenSoberIT

2Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY

Testing without predefined test casesManual testing

Based on experience, knowledge and skills of the testerWithout pre-documented test steps (detailed test cases)

Exploring the software or systemGoal is to expose quality-related informationAllowing tester to adjust plans and follow hunches

Based on the results and knowledge gained from the testing

Minimizing time spent on (pre)documentationTesting (execution) is not something that anyone masters

Exploratory Testing (ET) is


Exploratory Testing vs. Scripted Testing

ET is an approachMany testing techniques can be used in exploratory wayExploratory testing vs. scripted testing are the ends of a continuum

Freestyle exploratory “bug hunting”

Pure scripted(automated) testing

Manual scripts

High leveltest cases

Charteredexploratory testing


Scripted vs. Exploratory Testing

In scripted testing, tests are first designed and recorded. Then they may be executed at some later time or by a different tester.

In exploratory testing, tests are designed and executed at the same time, and they often are not recorded.

You build a mental model of the product while you test it. This model includes what the product is and how it behaves, and how it’s supposed to behave

Tests

Product

Tests

James Bach, Rapid Software Testing, 2002


Describing Exploratory Testing

In scripted testing, tests are first designed and recorded. Then they may be executed at some later time or by a different tester.

In exploratory testing, tests are designed and executed at the same time, and they often are not recorded.

James Bach, Rapid Software Testing, 2002

To the extent that the next test we do is influenced by the result of the last test we did, we are doing exploratory testing. James Bach, 2001

Exploratory testing involves simultaneously learning, planning, running tests, troubleshooting and reporting results. Cem Kaner, 2001


Characterizing Exploratory Testing

1. Tests are not defined in advance as detailed test scripts or test cases

Exploration with a general mission without specific step-by-step instructions on how to accomplish the missionAn exploratory tester uses any available information of the target of testing

requirements document, user manual, marketing brochure, …

2. Exploratory testing is guided by the results of previously performed tests and the gained knowledge from them

3. The focus in exploratory testing is on finding defects by exploration

Instead of systematically producing a comprehensive set of test cases for later use

4. Exploratory testing is simultaneous learning of the system undertest, test design, and test execution

5. The effectiveness of the testing relies on the tester’s knowledge, skills, and experience


ET and dimensions of testing

Pure ad hoc “bug hunting”

Completely scriptedtesting (automated)

No planningRigorously plannedon the level of individualtest cases

No tracking Coverage metrics andprogress tracking byplanned and executed testcases

No reporting or documentation

Detailed reports oneach executed test andall results, defects, issuesand comments

Destructive,revealing defects

Constructivedemonstrating benefitsbuilding confidence

ET


Exploring is asking questions and questioning

In exploratory testing tests are like questionsQuestioning skills are at the heart of ETExploratory tester proceeds by asking questions of the software under test

And uses the answers to guide testing and find the next questions

Lets play “20 questions”I think of some “thing” and you try to find out that “thing” by asking 20 questions that I can answer either ‘yes’ or ‘no’

Imagine having to write down those questions beforehand


Scripted vs. Exploratory TestsMine-field analogy

bugs fixes


Lateral thinking

Allowed to be distractedFind side paths and explore interesting areasPeriodically check your status against your mission


Dimensions of testing

Testing combines techniques that focus onTesters: who does the testingCoverage: what gets testedPotential problems: why you're testing (what risk you're testing for)Activities: how do you testEvaluation: how to tell whether the test passed or failed

All testing involves all five dimensionsHowever, each testing technique typically covers only one or two

Black Box Software Testing (Course Notes, Academic Version, 2004) www.testingeducation.org

Focus of the Exploratory Testing approach


ET as part of test lifecycle

Planning & Control

Prep

arat

ion

Spec

ificat

ion

Exec

utio

n

Com

plet

ion

Pol & Veenendal. Software Testing A guide to TMap Approach. 1998.

Planning & Control

Prep

arat

ion

Spec

ificat

ion

Exec

utio

n

Com

plet

ion

Planning & ControlPr

epar

atio

n

Spec

ificat

ion

Exec

utio

n

Com

plet

ion

Prep

arat

ion

Spec

ificat

ion

Exec

utio

n

Com

plet

ion

Planning & Control

ET


Well-timed test design

Early test designtest design finds faultsfaults found early are cheaper to fixmost significant faults found firstfaults prevented, not built inno additional effort

re-schedule test designtest design causes requirement changes

Not too early test case designDesign in implementation order

All tests do not have to be designed before implementation startsStart test design from the most completed and best understood features

Avoiding anticipatory test design and deprecated, incorrect test cases that are not based on the actual featuresTest planning and design must begin early, test case designnot necessarily


Information objectives

Exploratory testing is good for finding new informationYour testing can serve different information objectives

Find defectsMaximize bug countHelp managers make release decisionsBlock releasing premature softwareAssess qualityConform to regulationsFind safe scenarios to use the product…

A good exploratory tester will write down test ideas and use them in later test cycles

to create stronger and more efficient tests for the application in the future


Exploratory Function Testing

Use list of functions to give structure and high level guide to your testing

Requirements specificationFunctional specificationUser manual

Explore creatively each individual function and interactions of functions

Cover side paths, interesting and suspicious areasTests are designed simultaneously with test executionUse the list of functions to get back on track

Coverage and progress is planned and tracked by functions

Not by test cases


Exploratory testing vs. ad-hoc testing?

Exploratory testing can be a planned, goal driven and managed process – not an ad-hoc processTesting approach, goal, and coverage can be well plannedDetailed test log is created during the testing and defects are reported

Test report should be reviewable by another person in order to determine what was tested, how and what are the results

Exploratory test design and execution easily seem like ad-hoc activities

ET is ad hoc at the level of test case designAd hoc actually means “fit for purpose and situation”

NOT unmanaged, sloppy, vague, or randomSession-based testing is an approach to manage this creative and flexible process


Session Based Test Management (SBTM)

CharterTime BoxReviewable ResultDebriefing


Session-Based Testing

Enables planning and tracking exploratory testingWithout detailed test (case) designsUses time-boxed testing sessions as the planning and tracking mechanism instead of test cases and suites

Planning and tracking testing work in small chunks Dividing testing work in smaller tasksTracking testing work in time-boxed sessions

Can help getting testing done when resources are scarceEfficient – no unnecessary documentationUtilizes tester’s skills and experience during test executionAgile – testing is easy to focus to most important areas based on the test results and other information

Changes in requirements, increasing understanding, revealed problems, …


Charter

Designing the Charters is test planning and designBrief information / guidelines on:

What should be tested?Areas, components, features, …

Why do we test this?Goals, focus

How to test (approach)?Specific techniques or tactics to be usedTest data to be used

What problems to look for?

Might include guidelines on:Tools to useWhat risks are involvedDocuments to examineDesired output (information objectives) from the testing


Time Box

Focused test effort of fixed durationNo interruptions or other simultaneous tasks

Brief enough for accurate reportingBrief enough to allow flexible schedulingBrief enough to allow course correctionLong enough to get solid testing doneLong enough for efficient debriefingsBeware of overly precise timing

Short: 60 minutes (+-15)e.g. Normal: 90 minutes (+-15)

Long: 120 minutes (+-15)


Reviewable results

CharterTest notes

What was tested, howReproducibilityWhat needs more testing, new test ideas, etc.

Bugs and issuesData filesEffort breakdown

Duration (hours:minutes)Test design and execution (percent)Bug investigation and reporting (percent)Session setup (percent)Charter / opportunity (percent/percent)


Debriefing

The test lead reviews session notes to assure that he understands it and that it follows the process

Results of the sessionWhat was tested and howOther quality information

The tester answers any questionsSession metrics are checkedCharter may be adjustedSession may be extendedNew sessions may be charteredCoaching / mentoring happens


Exploratory Testing in Literature

No reliable scientific research resultsAcademic research on software testing focuses on test case design techniques, automation, and optimization


Literature study:Applicability of Exploratory Testing

When the test cases cannot be determined in advanceRapid feedback or learning of the product is neededTesting from the end-user viewpointProviding more diversity to scripted testing

When test scripts become “tired”, i.e., they are not detecting many defects anymore.

Exploring the size, scope, and variations of a found defect Regression testing based on defect reportsInvestigating the status of particular risksNot enough time for systematic approaches(When competent testers are not available)


Literature study:Claimed Benefits of Exploratory Testing

Utilises creativity and skills of a testerUtilizing experienceFinding important defectsYou see more when not following a script

Increased effectiveness in terms of number and importance of found defects

Optimized to find bugs (fast)Minimizing time spent on documentation (low-cost)

Simultaneous learning of the systemHelp testers to come up with more powerful tests

FlexibilityMinimized documentation before test executionRe-focusing on the most promising risk areasFollowing hunchesBeneficial when requirements or design changes

Ability to perform testing without finished or comprehensive documentationRapid flow of feedback from testing

To both developers and testers


Literature study:Identified Shortcomings of Exploratory Testing

Difficulty of tracking progressIndividual testersThe whole testing process

Tracking, planning, and measuring coverageExploratory testing has no ability to prevent defects through early test design


Results of a multiple case studyItkonen and Rautiainen. "Exploratory Testing: A Multiple Case Study", ISESE, 2005.

What practitioners from industry think about ETHow they use ETWhy they use ET

What benefits they feel it provides

What challenges they face using ET


Ways of utilizing ET

Session-based exploratory testingFunctional testing of individual featuresExploratory smoke testingFreestyle exploratory testing

Unmanaged ET as part of other dutiesExtending test cases

Outsourced exploratory testingAdvanced users, strong domain knowledgeReal use scenarios

Exploratory regression testingby verifying fixes or changes


Reasons for using ET

Writing detailed test cases for everything is difficult and laborious

The software can be used in so many ways or there are so many combinations between different features

With ET it is easy to test from the user’s viewpointET is a natural way of testing, which enables utilizing the testers’ experience and creativity during testingET enables quick feedback on new features from testers to developersET adapts well to changing situations

Where the requirements and the tested features change often, and the specifications are vague or incomplete

ET enables learning about the systemWhich can be utilized in other tasks, such as customer support and training


Perceived benefits of ET

ET is more versatile and goes deeper into the tested features

Testing things that would not be documented as test casesDependencies with existing features based on experience and knowledge

Test cases typically based on new requirementsRegression testing fixes

ET is an effective way of finding defectsRequires experienced testerEven if test cases are used the defects are found by exploringEasier to achieve a destructive attitudeIn the long run might be inefficient

ET gives a good picture of the overall quality of the system in a short timeET enables testing the look and feel of the system


Perceived shortcomings of ETCoverage was the biggest shortcoming

Planning and selecting what to testPrioritizing and allocating time of the domain expertsTracking the coverage of performed tests

Are all new features coveredAre all ways the user could use the system covered

Reliance on the expertise and skills of an individual tester

Human errorsDifferent ways of testing (also a strength)Difficulty of finding good domain experts

Repeatability of found defectsLong sequences of actions that lead to failureMemory leaks and other “non-deterministic” failures


Conclusions

ET is an accepted approach to testing in industryET seems to be an effective way of finding defects

At least when a session-based approach is used

Support for the benefit of utilizing testers’ creativity and experienceWe did not find any intentionally applied techniques or strategies used in ETUsing ET to learn the system for other purposes than testing was a new finding

Customer service and training

Biggest shortcoming is the challenges of planning and tracking test coverage

T-76.5613 Software Testing and Quality Assurance and... · HELSINKI UNIVERSITY OF TECHNOLOGY...

Documents

Transcript of T-76.5613 Software Testing and Quality Assurance and... · HELSINKI UNIVERSITY OF TECHNOLOGY...