Black-Box Testing Techniques - · PDF fileHELSINKI UNIVERSITY OF TECHNOLOGY T-76.5613 Software...

HELSINKI UNIVERSITY OF TECHNOLOGY

T-76.5613 Software Testing and Quality Assurance

Lecture 5, 24.9.2007

Black-Box Testing Techniques

Juha ItkonenSoberIT

2Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY

Black-box test case design techniques

Basic techniquesEquivalence partitioning Boundary value analysis

Function testingTesting heuristicsState transition testingSpecification testingError guessing and Failure modesScenario testing

Soap opera testing


Equivalence partitioning and boundary value analysis (i.e. domain testing)

To avoid unnecessary testing, partition (divide) the range of inputs into groups of equivalent tests. Then treat an input value from the equivalence class as representative of the full group.We treat two tests as equivalent if they are so similar to each other that it seems pointless to test both.Boundaries mark the point or zone of transition from one equivalence class to another. These are good members of equivalence classes to use because the program is more likely to fail at a boundary.

invalid valid invalid

0 1 100 101


Why not test only the boundaries?

If you do boundaries only, you have covered all the partitions as well

technically correct and may be OK if everything works correctlyboundaries can be analyzed wrongly, there can be hidden boundaries

if the test fails, is the whole partition wrong, or is a boundary in the wrong place – have to test mid-partition anyway

testing only extremes may not give confidence for typical use scenarios (especially for business people or users)boundaries may be harder (more costly) to set up


Exercise: Design test cases Use equivalence classes and boundaries


Exercise: Design test cases Some sample test cases

All: just one slide, no slides, many slidesCurrent slide: first, last, middle, selected, not selectedSelection: no selection, one slide, many slides, dis-continuous selection, all slides, …Slides:

One, first, lastSemicolon separated, two, many, consecutive not consecutiveNormal ranges, open ranges (e.g. ‘-5’ or ‘4-’), all slides, multiple ranges, duplicate and intersecting rangesRanges that exceed the number of slides, etc. Invalid slide numbersEmpty, invalid strings

Slides per page: One, two, four, six, odd number, big even number

Different amount of slides in the presentation (one, odd, even) Order: Horizontal and vertical

Combinations with other settings?


Function testing

Testing one function at a time“Functional unit testing”

Can be performed based on various informationGoing through the user interface or other interface commandsUser manual (draft)Functional specifications and requirements

Various techniques can be utilized for testing each function

Equivalence classes, boundary values, combinations, …


Function testing tasks

Build a list of functionsIdentify each function and sub-function

Determine how to recognize correct behaviourIdentify variables, equivalence classes, and boundariesIdentify environmental variables

that may constrain the function under test

Check that the function does what it's supposed to do and does not do what it's not supposed to do


Testing individual functions

Input variables of the functionOutputs of the functionPossible scope of the function

E.g. apply to selection, page, whole document

Options of the functionSettings, configuration, etc.

Circumstances under which the function behaves differently

Modes, or status of the systemDifferent ways of performing the same functionOther functions that interact with the tested function


Benefits of function testing

Functional coverageCover every feature and optionFunctional specification or function list

Useful for initial testing of the productIntroduction to the capabilities of the productReveals serious problems that should be addressed early


Shortcomings of function testing as a primary method

Misses interaction and relationships between functions

Feature interactionsmisses exploration of the benefits offered by the program

Is not necessarily powerful for validationWhether the software actually solves the customer’s problemWhether the users can actually achieve the benefitsEmphasizes isolated testing of individual itemsLoad, performance, reliability, etc. not covered

Easily focuses on the typical caseswithout considering boundaries, special cases or tests of special data


Test Heuristics

A heuristicA rule of thumbA way of making an educated guessUseful rules that are not guaranteed to lead to the right result

Heuristics can be used to generate ides for good testsHeuristics are suggestions that must be used wisely

Do not blindly follow heuristics without understanding them

Experienced testers collect heuristics that help them make good guesses when designing tests


Examples of general testing heuristics

Test at the boundariesTest exceptional casesTest with realistic data and scenariosTest every error messageAvoid redundant testsTest configurations that are different from the programmers’Run tests that are annoying to set up…


CICCUPP – Consistency heuristics help recognizing defects in software

Consistent with History: Present function behavior is consistent with past behavior.Consistent with our Image: Function behavior is consistent with an image that the organization wants to project. Consistent with Comparable Products: Function behavior is consistent with that of similar functions in comparable products.Consistent with Claims: Function behavior is consistent with what people say it’s supposed to be.Consistent with User’s Expectations: Function behavior is consistent with what we think users want.Consistent within Product: Function behavior is consistent with behavior of comparable functions or functional patterns within the product.Consistent with Purpose: Function behavior is consistent with apparent purpose.

Suorce: Black Box Software Testing, Cem Kaner


State transition testing

Uses a model that shows:states the software may occupytransitions between the states events which cause the transitionsactions that result from the transitions

Wait for card Wait for PINCancel

Return card

Card inserted

Ask for PIN

Invalid PIN

Ask for PIN

Valid PIN

Ask amount


State transition table

Current state Event Action Next state

Wait for card Card inserted Ask for PIN Wait for PIN

Wait for pin Cancel Return card Wait for card

Wait for pin Valid PIN Ask amount Wait for Amount

Wait for pin Invalid PIN Ask for PIN Wait for pin

Wait for amount … … …

… … … …


State transition testing coverage criteria

Simple criteriaCover all statesCover all eventsCover all transitionsCover all paths (infinite if loops possible)

Switch coverageAll single transitions (0-switch)Transition pairs (1-switch)Transition triples (2-switch)etc.

A more ”complete” test set will test for possible invalid transitions

use state transition table to identify invalid transitions


State transition table with invalid transitions

Current state Event Action Next state

Wait for card Card inserted Ask for PIN Wait for pin

Wait for card Cancel Wait for card

Wait for card Valid PIN Wait for card

Wait for card Invalid PIN Wait for card

Wait for card Valid amount Wait for card

Wait for card Sufficient balance Wait for card

Wait for card Insufficient balance Wait for card

Wait for pin Card inserted Wait for pin

Wait for pin Cancel Return card Wait for card

Wait for pin Valid PIN Ask amount Wait for amount

Wait for pin Invalid PIN Ask for PIN Wait for pin

Wait for pin Valid amount Wait for pin

Wait for pin Sufficient balance Wait for pin

Wait for pin Insufficient balance Wait for pin

Wait for amount Card inserted Wait for amount

Wait for amount Cancel Wait for amount

Wait for amount Valid PIN Wait for amount

Wait for amount Invalid PIN Wait for amount

Wait for amount Valid amount Get amount Check balance

Wait for amount Sufficient balance Wait for amount

Wait for amount Insufficient balance Wait for amount

Check balance Card inserted Check balance

Check balance Cancel Check balance

Check balance Valid PIN Check balance

Check balance Invalid PIN Check balance

Check balance Valid amount Check balance

Check balance Sufficient balance Give money Dispense notes

Check balance Insufficient balance Return card Wait for card

… … … …


Specification testing

Testing based on some sort of specification document

Requirements specificationFunctional specificationUser manualEtc.

Discovering what claims are being made testing them against the product


Specification testing examples

Using context-free questionsWho? What? When? Where? How? Why?

How does the spec say what it saysAmbiguity

Are multiple interpretations possible?

Adequacy Does it provide enough information for programming, documentation and testing?

CompletenessTo what extent does it cover the feature set, use cases, etc.?

Whenever there is ambiguity, there is a strong opportunity for a defect


Error guessing and Failure model analysis

Always worth includingComplements systematic techniquesCan find some faults that systematic techniques can missIs based on experience and previous or typical defects


Error guessing: deriving test cases

People are not that imaginative about committing bugsThere tends to be the same bugs as usuallyLearning from typical errors

ConsiderPast failuresIntuitionExperienceBrain storming

”What is the craziest thing we can do?”

Lists and failure models in literature, defect catalogs

Create a defect catalog of your own List the most typical defects in your context


Test CatalogsTest catalogs are refined defect catalogsTest catalog is a check list of typical tests for a certain situationBased on experience on typical errors that developers make

Textual input/Numbersthe largest number that is supposed to be accepted the smallest number that is supposed to be accepted one larger than the largest number that is supposed to be accepted one smaller than the smallest number that is supposed to be accepted

M achine boundaries127128 (signed byte overflow) 255256 (unsigned byte overflow) 32767 (32 767) 32768 (32 768) (signed 16-bit overflow) 65535 (65 535) 65536 (65 536) (unsigned 16-bit overflow) 2147483647 (2 147 483 647) 2147483648 (2 147 483 648) (signed 32-bit word overflow) 4294967295 (4 294 967 295) 4294967296 (4 294 967 296) (unsigned 32-bit word overflow.)


Example of Failure Model Testing James A. Whittaker: How to Break Software

User interface attacksApply inputs that force all error messagesApply inputs that force the software to establish default valuesOverflow input buffersForce data structure to store too many or too few valuesInvestigate alternative ways to modify internal data constraintsEtc.

System interface attacksFill the system to its capacityForce media to be busy or unavailableVary file permissionsVary file content or corrupt fileEtc.


Scenario testing

A scenario is a hypothetical story, used to help a person think through a complex problem or system.Scenario Testing is a testing approach that is based on scenarios.Testing scenarios are complicated and realistic stories of real usage of the systemScenarios are carefully designed and documentedScenarios can be executed manually or automated

The goal is to focus on business needs and realistic use situations, instead of technical details and mechanical testing


Good Testing Scenarios

Based on a story about how the program is usedmotivations of the people involved.

MotivatingA stakeholder with influence would push to fix a revealed defect

CredibleNot only could happen in the real world; stakeholders would believe that something like it probably will happen

ComplexComplex use of the program Complex environment Complex set of data.

Results are easy to evaluateEspecially important for scenarios because they are complex

Cem

Kaner. “An Introduction to Scenario Testing”, STQE, October, 2003.http://www.kaner.com/pdfs/ScenarioIntroVer4.pdf


Scenarios can be used to

Learn the productConnect testing to documented requirementsBring requirements-related issues to the surfaceExpose failures to deliver desired benefitsExplore expert use of a programMake a bug report more motivating

Cem

Kaner. “An Introduction to Scenario Testing”, STQE, October, 2003.http://www.kaner.com/pdfs/ScenarioIntroVer4.pdf


Soap Opera Testing

Extreme Scenario TestingBuild a scenario based on real-life experienceExaggerate each aspect of it

for each variable, substitute a more extreme valueif a scenario can include a repeating element, repeat it lots of timesMake environment more hostile

increase or decrease memory, printer resolution, video resolution, etc.

Make the situation as complicated as possible

Create a real-life story that combines all of the elements into a test case narrative.

Black-Box Testing Techniques - · PDF fileHELSINKI UNIVERSITY OF TECHNOLOGY T-76.5613 Software...

Documents

Transcript of Black-Box Testing Techniques - · PDF fileHELSINKI UNIVERSITY OF TECHNOLOGY T-76.5613 Software...