Black box testing (an introduction to)

Università degli Studi dell’Aquila

L21: Black Box/Functional Testing

Henry Muccini

DISIM, University of L’Aquila

www.henrymuccini.com, [email protected]

The material in these slides may be freely reproduced anddistributed, partially or totally, as far as an explicitreference or acknowledge to the material author ispreserved.

Partially based on material from Alex Orso, Mauro Pezze’, and Michal Young.

White box testing

Representative values

Systematic vs Random Testing

Equivalence Partitioning

Category Partition Method

AGENDA

Category Partition Method

Where is the boundary between testing and other

(analysis/verification) methods?

What distinguishes software testing from “other” approaches

is execution. This requires the ability to:

launch the tests on some executable version (traceability problem), → launch the tests on some executable version (traceability problem),

and

→ analyse (observe and evaluate) the results (not granted for embedded

systems)

I am not saying testing is superior and we should disregard other

approaches: on the contrary, testing and other methods

should complement/support each other

Any software test campaign involves a trade-off

between

→ Limiting resource utilization (time, effort)

⇒ as few tests as possible

→ Augmenting our confidence

⇒

Augmenting our confidence

⇒ as many tests as possible

Two research challenges:

→ Determining a feasible and meaningful stopping rule

→ Evaluating test effectiveness (reliability, “coverage notion”,

....very tough problem

BLACK BOX/FUNCTIONAL TESTING

Functional testing: Deriving test cases from program

specifications

─ Functional refers to the source of information used in test case

design, not to what is tested

Also known as:

→ specification-based testing (from specifications)

→ black-box testing (no view of the code)

Functional specification = description of intended

program behavior

→ either formal or informal

(C) 2007 MAURO PEZZÈ & MICHAL YOUNG

Exploit some knowledge to choose samples that are more

likely to include “special” or trouble-prone regions of the

input space

→ Failures are sparse in the whole input space ...

→ ... but we may find regions in which they are dense

Desirable case: Each fault leads to failures that are dense

(easy to find) in some class of inputs

→ sampling each class in the quasi-partition selects at least one

input that leads to a failure, revealing the fault

→ seldom guaranteed; we depend on experience-based heuristics


Failure (valuable test case)

No failure

Failures are sparse

in the space of

possible inputs ...

... but dense in some

parts of the space

The s

pace o

f possib

le input

valu

es

(the h

aysta

ck)


If we systematically test some

cases from each part, we will

include the dense parts

Functional testing is one way of

drawing pink lines to isolate

regions with likely failures

The s

pace o

f possib

le input

valu

es

(the h

aysta

ck)

Random (uniform):

→ Pick possible inputs uniformly

→ Avoids designer bias

─ A real problem: The test designer can make the same logical mistakes and bad assumptions as the program designer (especially if they are the same person)logical mistakes and bad assumptions as the program designer (especially if they are the same person)

→ But treats all inputs as equally valuable


Defects are not distributed uniformly!

• Assume our printSum program fails if both

inputs are 0!

• Random sampling is unlikely to choose a = 0 • Random sampling is unlikely to choose a = 0

and b = 0!

We want bias!!

Systematic (non-uniform):

→ Try to select inputs that are especially valuable

→ Usually by choosing representatives of classes that are apt to fail often or not at all

Functional testing is systematic testing

•An independently testable feature (ITF) is a

functionality that can be tested independently of other

functionalities of the software under test.

•It need not correspond to a unit or subsystem of the

software. software.

•For example, a file sorting utility may be capable of

merging two sorted files, and it may be possible to test

the sorting and merging functionalities separately,

even though both features are implemented by much

of the same source code.

It is recommended to devise separate test cases for each functionality of the system, whenever possible.

The design of different tests for different functionalities can simplify the test generation problem

Thus, identifying functional features does not correspond to identifying single modules at the design level, but rather to suitably slicing the specifications to be able to attack their complexity incrementally

Feature-based view of the system

Functional testing uses the specification (formal or

informal) to partition the input space

→ E.g., specification of “roots” program suggests division

between cases with zero, one, and two real rootsbetween cases with zero, one, and two real roots

Test each category, and boundaries between boundaries between

categoriescategories

→ No guarantees, but experience suggests failures often lie at

the boundaries (as in the “roots” program)


•Try to select inputs that are especially valuable!

• Usually by choosing representatives of equivalence classes!

•Assumption: inputs in different categories behave inputs in different categories behave

equivalently equivalently

⇒⇒I can select I can select one input per categoryone input per category⇒⇒I can select I can select one input per categoryone input per category

Consider a software module that is intended to accept the name of a grocery item and a list of the different sizes the item comes in, specified in ounces. The specifications state that the item name is to be alphabetic characters 2 to 15 characters in length. Each size may be a value in the range of 1 to 48, whole numbers only. The sizes are to be entered in ascending order (smaller sizes first). A of 1 to 48, whole numbers only. The sizes are to be entered in ascending order (smaller sizes first). A maximum of five sizes may be entered for each item. The item name is to be entered first, followed by a comma, then followed by a list of sizes. A comma will be used to separate each size. Spaces (blanks) are to be ignored anywhere in the input.

Taken from: http://users.csc.calpoly.edu/~jdalbey/205/Resources/grocerystore.

Consider a software module that is intended to accept the name of a grocery item and a list of the different sizes the item comes in, specified in ounces.

The specifications state that the item name is to be alphabetic characters 2 to 15 characters in length. Each size may be a value in the range of 1 to 48, whole numbers only.

The sizes are to be entered in ascending order (smaller sizes The sizes are to be entered in ascending order (smaller sizes first). A maximum of five sizes may be entered for each item. The item name is to be entered first, followed by a comma, then followed by a list of sizes. A comma will be used to separate each size. Spaces (blanks) are to be ignored anywhere in the input.

Item name is alphabetic (valid)Item name is not alphabetic (invalid)Item name is less than 2 characters in length (invalid)Item name is 2 to 15 characters in length (valid)Item name is greater than 15 characters in length (invalid)Size value is less than 1 (invalid)Size value is in the range 1 to 48 (valid)Size value is greater than 48 (invalid)Size value is a whole number (valid)Size value is a decimal (invalid)Size value is numeric (valid)Size value is numeric (valid)Size value includes nonnumeric characters (invalid)Size values entered in ascending order (valid)Size values entered in nonascending order (invalid)No size values entered (invalid)One to five size values entered (valid)More than five sizes entered (invalid)Item name is first (valid)

…

# Test DataExpected

OutcomeClasses Covered

1 xy,1 T 1,4,7,9,11,13,16,18,20,22

2 AbcDefghijklmno,1,2,3 ,4,48 T 1,4,7,9,11,13,16,18,20,23

3 a2x,1 F 2

4 A,1 F 3

5 abcdefghijklmnop F 55 abcdefghijklmnop F 5

6 Xy,0 F 6

7 XY,49 F 8

8 Xy,2.5 F 10

9 xy,2,1,3,4,5 F 14

10 Xy F 15

11 XY,1,2,3,4,5,6 F 17

12 1,Xy,2,3,4,5 F 19

13 XY2,3,4,5,6 F 21

14 AB,2#7 F 12

Item name is alphabetic (valid)Item name is not alphabetic (invalid)Item name is less than 2 characters in length (invalid)Item name is 2 to 15 characters in length (valid)Item name is greater than 15 characters in length (invalid)Size value is less than 1 (invalid)Size value is in the range 1 to 48 (valid)Size value is greater than 48 (invalid)Size value is a whole number (valid)Size value is a decimal (invalid)Size value is numeric (valid)Size value is numeric (valid)Size value includes nonnumeric characters (invalid)Size values entered in ascending order (valid)Size values entered in nonascending order (invalid)No size values entered (invalid)One to five size values entered (valid)More than five sizes entered (invalid)Item name is first (valid)

…

The main idea is to partition the input domain of

function being tested, and then select test data for

each class of the partition.

The problem of all the existing techniques is the lack of The problem of all the existing techniques is the lack of

systematic.

Solution: the Category Partition Method

Steps:

�ITF (Independently testable feature)

�Identify parameters and Environmental conditions

� Parameters: Explicit input to the functional unit

Environmental Conditions: Characteristics of the � Environmental Conditions: Characteristics of the

system’s state

�Find categories that characterize each parameter and

environment condition.

�Every category should be partitioned into distinct choices

�Test Spec

Command: find (a pattern into a file)

Syntax: find <pattern> <file>

Function:Function: The find command is used to locate one or more instance of a given

pattern in a text file. All lines in the file that contain the pattern are written to

standard output. A line containing the pattern is written only once, regardless of the

number of times the pattern occurs in it.

The pattern is any sequence of characters whose length does not exceed the

maximum length of a line in the file .To include a blank in the pattern, the entire

pattern must be enclosed in quotes (“).To include quotation mark in the pattern ,two

quotes in a row (“ “) must be used.

Example:

find john myfile

display lines in the file myfile which contain john

find “john smith” in myfilefind “john smith” in myfile

display lines in the file myfile which contain john smith

find “john”” smith” in myfile

display lines in the file myfile which contain john” smith

Example:

find john myfile

display lines in the file myfile which contain john

find “john smith” in myfile

sizesize

quotequote emb blankemb blank

Parameters (explicit input to the functional unit)

find “john smith” in myfile

display lines in the file myfile which contain john smith

find “john”” smith” in myfile

display lines in the file myfile which contain john” smith

emb quotesemb quotes

Parameters:

Pattern size:

empty

single character

many character

longer than any line in the file

Quoting:Quoting:

pattern is quoted

pattern is not quoted

pattern is improperly quoted

Embedded blanks:

no embedded blank

one embedded blank

several embedded blanks

Embedded quotes:

no embedded quotes

one embedded quotes

several embedded quotes

File name:

good file name

no file with this name

Environments (characteristics of the system’s state):Environments Number of occurrence of pattern in file:

none

exactly one

more than one

Pattern occurrences on target line:

one

more than one

Total Tests frames:

1944

Pattern size : empty

Quoting : pattern is quoted

Embedded blanks : several embedded blanks

Embedded quotes : no embedded quoteEmbedded quotes : no embedded quote

File name : good file name

Number of occurrence of pattern in file : none

Pattern occurrence on target line : one

However, not all combinations of value classes

correspond to reasonable test case specifications!!!

• Properties

– [property A, B, …]

– A and B are property names

– E.g., [property Empty]– E.g., [property Empty]

• Selector expression

– [if A]

– E.g., [if Empty]

Parameters:

Pattern size:

empty [ property Empty ]

single character [ property NonEmpty ]

many character [ property NonEmpty ]

longer than any line in the file [ property NonEmpty ]

Quoting:

pattern is quoted [ property quoted ]

pattern is not quoted [ if NonEmpty ]

pattern is improperly quoted [ if NonEmpty ]

Embedded blanks:

no embedded blank [ if NonEmpty ]

one embedded blank [ if NonEmpty and Quoted ]

several embedded blanks [ if NonEmpty and Quoted ]

Embedded quotes:

no embedded quotes [ if NonEmpty ]

one embedded quotes [ if NonEmpty ]

several embedded quotes [ if NonEmpty ]

File name:

good file name

no file with this name

Environments: Total Tests frames:

678Number of occurrence of pattern in file:

none [ if NonEmpty ]

exactly one [ if NonEmpty ] [ property Match]

more than one [ if NonEmpty ] [ property Match ]


one [ if Match ]

more than one [ if Match ]

678

The label [error] indicates a value class that need be

tried only once, in combination with non-error values

of other parameters.

Parameters:

Pattern size:

empty [ property Empty ]

single character [ property NonEmpty ]

many character [ property NonEmpty ]

longer than any line in the file [ error ]

Quoting:

pattern is quoted [ property quoted ]

pattern is not quoted [ if NonEmpty ]

pattern is improperly quoted [ error ]

Embedded blanks:

no embedded blank [ if NonEmpty ]

one embedded blank [ if NonEmpty and Quoted ]

several embedded blanks [ if NonEmpty and Quoted ]

Embedded quotes:

no embedded quotes [ if NonEmpty ]

one embedded quotes [ if NonEmpty ]

several embedded quotes [ if NonEmpty ]

File name:

good file name

no file with this name [ error ]

Environments:

Number of occurrence of pattern in file:

Total Tests frames:

125

Total Tests frames:

40

[ single ]

Number of occurrence of pattern in file:

none [ if NonEmpty ]

exactly one [ if NonEmpty ] [ property Match]

more than one [ if NonEmpty ] [ property Match ]


one [ if Match ]

more than one [ if Match ]

12540

[ single ]

[ single ]

• Use a constraint solver

• Choose specific values that satisfy the

constraints

A completely different approach:

→We cannot realistically presume to find and remove the last

failure.

→Then, we should invest test resources to find out the

“biggest” ones.

“Goodness” here means to minimise the user

perceived faults, i.e.

→Try to find earlier those bugs that are more frequently

encountered (neglect “small” bugs)

→Not suitable for safety critical applications

Identify:

• Typical usage scenario

• Frequency of those scenarios

Test more, more frequent scenariosTest more, more frequent scenarios

Black box testing (an introduction to)

Education

Transcript of Black box testing (an introduction to)