T-76.5613 Software Testing and Quality Assurance and... · Without pre-documented test steps...

HELSINKI UNIVERSITY OF TECHNOLOGY

T-76.5613 Software Testing and Quality Assurance

Lecture 6, 26.9.2007

Exploratory Testing (ET)vs.

Test-Case Based (scripted) Testing

Juha ItkonenSoberIT

2Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY

Testing without predefined test casesManual testing

Based on experience, knowledge and skills of the testerWithout pre-documented test steps (detailed test cases)

Exploring the software or systemGoal is to expose quality-related informationAllowing tester to adjust plans and follow hunches

Based on the results and knowledge gained from the testing

Minimizing time spent on (pre)documentationTesting (execution) is not something that anyone masters

Exploratory Testing (ET) is


Motivation

ET is widely used approach in the industryBased on our earlier researchPractitioner reports in the literature and seminars

Anecdotal results of high defect detection efficiency of ET exist

Reported by consultants and practitioners

Existing research ignores testing not based on designing test cases beforehand

Test-case based testing (TCT) approach in empirical research literature is mainly visible as test case design techniques

Besides test cases, many other factors have strong effect on the results of testing

E.g. different testers find different defects even if using the same technique


Exploratory Testing vs. Scripted Testing

ET is an approachMany testing techniques can be used in exploratory wayExploratory testing vs. scripted testing are the ends of a continuum

Freestyle exploratory “bug hunting”

Pure scripted (automated) testing

Manual scripts

High level test cases

Chartered exploratory testing


Scripted vs. Exploratory Testing

In scripted testing, tests are first designed and recorded. Then they may be executed at some later time or by a different tester.

In exploratory testing, tests are designed and executed at the same time, and they often are not recorded.

You build a mental model of the product while you test it. This model includes what the product is and how it behaves, and how it’s supposed to behave

Tests

Product

Tests

James Bach, Rapid Software Testing, 2002


Characterizing Exploratory Testing

1. Tests are not defined in advance as detailed test scripts or test cases

Exploration with a general mission without specific step-by-step instructions on how to accomplish the missionAn exploratory tester uses any available information of the target of testing

requirements document, user manual, marketing brochure, …

2. Exploratory testing is guided by the results of previously performed tests and the gained knowledge from them

3. The focus in exploratory testing is on finding defects by exploration

Instead of systematically producing a comprehensive set of test cases for later use

4. Exploratory testing is simultaneous learning of the system under test, test design, and test execution

5. The effectiveness of the testing relies on the tester’s knowledge, skills, and experience


ET and dimensions of testing

Pure ad hoc “bug hunting”

Completely scripted testing (automated)

No planningRigorously planned on the level of individual test cases

No tracking Coverage metrics and progress tracking by planned and executed test cases

No reporting or documentation

Detailed reports on each executed test and all results, defects, issues and comments

Destructive, revealing defects

Constructive demonstrating benefits building confidence

ET


Exploring is asking questions and questioning

In exploratory testing tests are like questionsQuestioning skills are at the heart of ETExploratory tester proceeds by asking questions of the software under test

And uses the answers to guide testing and find the next questions

Lets play “20 questions”I think of some “thing” and you try to find out that “thing” by asking 20 questions that I can answer either ‘yes’ or ‘no’

Imagine having to write down those questions beforehand


Scripted vs. Exploratory Tests Mine-field analogy

bugs fixes


Lateral thinking

Allowed to be distractedFind side paths and explore interesting areasPeriodically check your status against your mission


Dimensions of testing

Testing combines techniques that focus onTesters: who does the testingCoverage: what gets testedPotential problems: why you're testing (what risk you're testing for)Activities: how do you testEvaluation: how to tell whether the test passed or failed

All testing involves all five dimensionsHowever, each testing technique typically covers only one or two

Black Box Software Testing (Course Notes, Academic Version, 2004) www.testingeducation.org

Focus of the Exploratory Testing approach


ET as part of test lifecycle

Planning & Control

Prep

arat

ion

Spec

ificat

ion

Exec

utio

n

Com

plet

ion

Pol & Veenendal. Software Testing A guide to TMap Approach. 1998.

Planning & Control

Prep

arat

ion

Spec

ificat

ion

Exec

utio

n

Com

plet

ion

Planning & Control

Prep

arat

ion

Spec

ificat

ion

Exec

utio

n

Com

plet

ion

Prep

arat

ion

Spec

ificat

ion

Exec

utio

n

Com

plet

ion

Planning & Control

ET


Information objectives

Exploratory testing is good for finding new informationYour testing can serve different information objectives

Find defectsMaximize bug countHelp managers make release decisionsBlock releasing premature softwareAssess qualityConform to regulationsFind safe scenarios to use the product…

A good exploratory tester will write down test ideas and use them in later test cycles

to create stronger and more efficient tests for the application in the future


Exploratory Function Testing

Use list of functions to give structure and high level guide to your testing

Requirements specificationFunctional specificationUser manual

Explore creatively each individual function and interactions of functions

Cover side paths, interesting and suspicious areasTests are designed simultaneously with test executionUse the list of functions to get back on track

Coverage and progress is planned and tracked by functions

Not by test cases


Exploratory testing vs. ad-hoc testing?

Exploratory testing can be a planned, goal driven and managed process – not an ad-hoc processTesting approach, goal, and coverage can be well plannedDetailed test log is created during the testing and defects are reported

Test report should be reviewable by another person in order to determine what was tested, how and what are the results

Exploratory test design and execution easily seem like ad-hoc activities

ET is ad hoc at the level of test case designAd hoc actually means “fit for purpose and situation”

NOT unmanaged, sloppy, vague, or randomSession-based testing is an approach to manage this creative and flexible process


Session Based Test Management (SBTM)

CharterTime BoxReviewable ResultDebriefing


Charter

Designing the Charters is test planning and designBrief information / guidelines on:

What should be tested?Areas, components, features, …

Why do we test this?Goals, focus

How to test (approach)?Specific techniques or tactics to be usedTest data to be used

What problems to look for?

Might include guidelines on:Tools to useWhat risks are involvedDocuments to examineDesired output (information objectives) from the testing


Time Box

Focused test effort of fixed durationNo interruptions or other simultaneous tasks

Brief enough for accurate reportingBrief enough to allow flexible schedulingBrief enough to allow course correctionLong enough to get solid testing doneLong enough for efficient debriefingsBeware of overly precise timing

Short: 60 minutes (+-15)e.g. Normal: 90 minutes (+-15)

Long: 120 minutes (+-15)


Reviewable results

CharterTest notes

What was tested, howReproducibilityWhat needs more testing, new test ideas, etc.

Bugs and issuesData filesEffort breakdown

Duration (hours:minutes)Test design and execution (percent)Bug investigation and reporting (percent)Session setup (percent)Charter / opportunity (percent/percent)


Debriefing

The test lead reviews session notes to assure that he understands it and that it follows the process

Results of the sessionWhat was tested and howOther quality information

The tester answers any questionsSession metrics are checkedCharter may be adjustedSession may be extendedNew sessions may be charteredCoaching / mentoring happens


Session-Based Testing

Enables planning and tracking exploratory testingWithout detailed test (case) designsUses time-boxed testing sessions as the planning and tracking mechanism instead of test cases and suites

Planning and tracking testing work in small chunks Dividing testing work in smaller tasksTracking testing work in time-boxed sessions

Can help getting testing done when resources are scarceEfficient – no unnecessary documentationUtilizes tester’s skills and experience during test executionAgile – testing is easy to focus to most important areas based on the test results and other information

Changes in requirements, increasing understanding, revealed problems, …


Exploratory Testing in Literature

Lack of reliable scientific research resultsAcademic research on software testing focuses on test case design techniques, automation, and optimization


Literature study: Applicability of Exploratory Testing

When the test cases cannot be determined in advanceRapid feedback or learning of the product is neededTesting from the end-user viewpointProviding more diversity to scripted testing

When test scripts become “tired”, i.e., they are not detecting many defects anymore.

Exploring the size, scope, and variations of a found defect Regression testing based on defect reportsInvestigating the status of particular risksNot enough time for systematic approaches(When competent testers are not available)


Literature study: Claimed Benefits of Exploratory Testing

Utilises creativity and skills of a testerUtilizing experienceFinding important defectsYou see more when not following a script

Increased effectiveness in terms of number and importance of found defects

Optimized to find bugs (fast)Minimizing time spent on documentation (low-cost)

Simultaneous learning of the systemHelp testers to come up with more powerful tests

FlexibilityMinimized documentation before test executionRe-focusing on the most promising risk areasFollowing hunchesBeneficial when requirements or design changes

Ability to perform testing without finished or comprehensive documentationRapid flow of feedback from testing

To both developers and testers


Literature study: Identified Shortcomings of Exploratory Testing

Difficulty of tracking progressIndividual testersThe whole testing process

Tracking, planning, and measuring coverageExploratory testing has no ability to prevent defects through early test design


Results of a multiple case study Itkonen and Rautiainen. "Exploratory Testing: A Multiple Case Study", ISESE, 2005.

What practitioners from industry think about ETHow they use ETWhy they use ET

What benefits they feel it provides

What challenges they face using ET


Ways of utilizing ET

Session-based exploratory testingFunctional testing of individual featuresExploratory smoke testingFreestyle exploratory testing

Unmanaged ET as part of other dutiesExtending test cases

Outsourced exploratory testingAdvanced users, strong domain knowledgeReal use scenarios

Exploratory regression testingby verifying fixes or changes


Reasons for using ET

Writing detailed test cases for everything is difficult and laborious

The software can be used in so many ways or there are so many combinations between different features

With ET it is easy to test from the user’s viewpointET is a natural way of testing, which enables utilizing the testers’ experience and creativity during testingET enables quick feedback on new features from testers to developersET adapts well to changing situations

Where the requirements and the tested features change often, and the specifications are vague or incomplete

ET enables learning about the systemWhich can be utilized in other tasks, such as customer support and training


Perceived benefits of ET

ET is more versatile and goes deeper into the tested features

Testing things that would not be documented as test casesDependencies with existing features based on experience and knowledge

Test cases typically based on new requirementsRegression testing fixes

ET is an effective way of finding defectsRequires experienced testerEven if test cases are used the defects are found by exploringEasier to achieve a destructive attitudeIn the long run might be inefficient

ET gives a good picture of the overall quality of the system in a short timeET enables testing the look and feel of the system


Perceived shortcomings of ETCoverage was the biggest shortcoming

Planning and selecting what to testPrioritizing and allocating time of the domain expertsTracking the coverage of performed tests

Are all new features coveredAre all ways the user could use the system covered

Reliance on the expertise and skills of an individual tester

Human errorsDifferent ways of testing (also a strength)Difficulty of finding good domain experts

Repeatability of found defectsLong sequences of actions that lead to failureMemory leaks and other “non-deterministic” failures


Conclusions

ET is an accepted approach to testing in industryET seems to be an effective way of finding defects

At least when a session-based approach is used

Support for the benefit of utilizing testers’ creativity and experienceWe did not find any intentionally applied techniques or strategies used in ETUsing ET to learn the system for other purposes than testing was a new finding

Customer service and training

Biggest shortcoming is the challenges of planning and tracking test coverage

HELSINKI UNIVERSITY OF TECHNOLOGY

Results from a controlled student experiment

What is the benefit of designing the test cases beforehand?

Juha Itkonen, Mika V. Mäntylä and Casper Lassenius

"Defect Detection Efficiency: Test Case Based vs. Exploratory Testing", in proceedings of the International Symposium on Empirical Software Engineering and

Measurement, 2007.


Research problem

Do testers performing manual functional testing

with pre-designed test cases find more or different defects compared to testers working

without pre-designed test cases?


Research questions

How does using pre-designed test cases affect

1. the number of detected defects?

2. the type of detected defects?

3. the number of false defect reports?

False reports are:• incomprehensible• duplicate• reporting a non-existent defect


Total number of detected defects

Found defects per subject

Testing approach

Feature Set

Number of defects

x̄ σ FS A 44 6.28 2.172 FS B 41 7.82 2.522

ET

Total 85 7.04 2.462 FS A 43 5.36 2.288 FS B 39 7.35 2.225

TCT

Total 82 6.37 2.456 x̄ = mean and σ = standard deviation

Difference shows 0.7 defects more in the ET approachT-test significance value of 0.088

Not statistically significant


Box-plot of the defect counts

Slightly different defect counts between the two feature setsNo significant difference between the two approachesSome subjects revealed very high number of defects in Feature set B

Both with ET and TCT

FS BFS A

15

10

5

0

Det

ecte

d de

fect

s pe

r sub

ject

TCTET

Method


Defect Types

Compared to TCT, ET detectedMore defects that were obvious to detect More defects that were difficult to detectMore user interface and usability issuesLess technical defectsMore low severity defects

These are descriptive rather than conclusive findingsNot statistically significant


False defect reports

False defects per subject

Testing Approach

Feature Set

x̄ σ FS A 1.00 1.396 FS B 1.05 1.191 ET Total 1.03 1.291 FS A 1.64 1.564 FS B 2.50 1.867 TCT Total 2.08 1.767

TCT produced on average 1.05 more false reports than ET

per subject and testing sessionDifference is statistically significant with a significance value 0.000

Mann-Whitney U test

False reports are:• incomprehensible• duplicate• reporting a non- existent defect


Conclusions

1. The data showed no benefits from using pre- designed test cases

in comparison to exploratory testing approach

2. There was no significant difference in terms of defect type, severity, or detection difficulty

Defect type distributions indicate certain defect types might be better detected by ET

3. Test-case based testing produced more false defect reports

Perhaps test cases made testing more straightforward and mechanic leading, e.g., to higher number of duplicate reports

T-76.5613 Software Testing and Quality Assurance and... · Without pre-documented test steps...

Documents

Transcript of T-76.5613 Software Testing and Quality Assurance and... · Without pre-documented test steps...