Black Box Testing

SE 110 – Spring 2013

Software Testing: Black box testing

SE 110


Black-box testing

• Black box testing is done without knowledge of the internals of system under test.

• It is done from the customer’s view point and involves looking at the specifications only. – It requires a functional knowledge of the product to be

tested.

• Black box tests are convenient to administer as they use the complete finished product and do not require any knowledge of its construction.


Characteristics of black-box testing

• Done based on requirements.

• Addresses (should address) stated as well as implied requirements.

• Encompasses the end-user perspective.

• Checks for valid and invalid conditions / inputs.

• May or may not know the technology aspects of the product.


Typical errors found in black-box testing

• Incorrect or missing functionalities

• Interface errors

• Errors in data structures/data base access

• Behavior errors

• Performance errors

• Initialization and termination errors


Black-box vs. White-box testing

Black Box Testing White Box Testing

Has no access to program code

Has access to program code

Requires external perspective

Requires knowledge of program code

Set of techniques applicable to all other phases of testing

Typically applies only to unit testing, where code is involved


Black-box testing techniques

• Requirements-based testing• Positive and negative testing• Boundary value analysis• Decision tables• Equivalence partitioning• State-based testing• Compatibility testing• User documentation testing• Domain testing (leads to ad hoc testing)


General format for discussion of techniques

• Present some reasoning where applicable.

• List out one or two examples.

• Walk through the examples.

• Summarize the process for using the technique.


Requirements based testing• Done to ensure that all requirements in SRS are

tested.• Differentiates between implicit and explicit

requirements.• Review requirements first to ensure they are

consistent, correct, complete and testable.• Review enables translation of (some of) the

implied requirements to stated requirements.• A reviewed SRS tabulates requirements, along

with a requirements id and a priority.• This is the genesis of a Requirements

Traceability Matrix (RTM).


RTM: ExampleReq. ID Description Priority Test

conditionsTest case IDs

Phase of testing

BR-01 High Test_001 Unit, Component

BR-02 High Test_002 Unit, Component

BR-03 Medium Test_003, Test_004, Test_005

Integration

•Test condition: Different ways of testing a requirement

(different types of mapping)•Test case: Different conditions / scenarios for a given requirement•Phase of testing – helps in scheduling


Positive and negative testing

• Positive testing is done to check that the product does what it is supposed to.– Behaves correctly when given right inputs.– Maps to a specific requirement.– “Coverage” is defined better.

• Negative testing is done to show that the product does not fail when given unexpected inputs.– Tries to break the system.– No direct mapping to a specific requirement.– “Coverage” more challenging.


Boundary Value Analysis (BVA)

• Most defects come up near “boundaries”• Reasons from a white box perspective:

– Programmers tentativeness in using the right relational operator (< or < = ?)

– Multiple ways of implementing loops– Confusing array subscripts

• Reasons from a black box perspective:– Unclear requirements– Ambiguous or “it depends” mindset!


BVA: Example• Database starts with a pre-allocating number of buffers for

caching • Buffers filled up as needed• If full, buffers freed on a FIFO basis


BVA: Examples

• Look for any kind of gradation or discontinuity in data values that affects computation – the discontinuities are the boundary values, requiring thorough testing.

• Look for any internal limits like limits on resources (like the example of buffers given above). The behavior of the product at these limits should also be the subject of boundary value testing.

• Also include in the list of boundary values, documented limits on hardware resources. For example, if it is documented that a product will run with minimum 4MB of RAM, make sure you include test cases for the minimum RAM (i.e., 4MB in this case).

• The examples given above discuss boundary conditions for input data – the same analysis needs to be done for output variables also.


Decision tables

• A program’s behavior is characterized by several decision variables.– Input and output data can be expressed as Boolean

conditions (TRUE, FALSE, DON’T CARE). • Each decision variable specifies a Boolean

condition.• The distinct combinations of these decision

variables lead to different scenarios.• Each scenario occupies a row in the decision

table, and the row also has expected results.• One representative data point from each

scenario needs to be tested.


Decision tables: Example

• Taxpayers have a choice of either taking a standard deduction (SD) or itemizing their deductions.

• Various factors determine SD:– Single $4750– Married and filing a joint return $9500– Married and filing a separate return $7000– If filer or spouse is 65 years or older and additional

SD of $1000 is allowed– If filer or spouse is blind an additional SD of $1000 is

allowed


Decision tables: Example

Status Status of spouse

Age (<65 or >65)

Age of spouse

Blind (yes or No)

Spouse blind

SD amount

Single --- No --- No --- $4750

Married, filing separate return

Claimed SD

No --- No --- $7000

Married, filing joint return

--- Yes --- No No $10,500


Decision tables: Process

• Identify the decision variables.• Identify the possible values of each of the decision variables.• Enumerate the combinations of the allowed values of each of the

variables.• Identify the cases in which values assumed by a variable (or by sets

of variables) are immaterial for a given combination of other input variables.– Represent such variables by the Don’t Care symbol.

• For each combination of values of decision variables (appropriately minimized with the Don’t Care scenarios), list out the action or expected result.

• Form a table, listing on each but the last column one decision variable. On the last column, list the action item for the combination of variables in that row (including Don’t Cares, as appropriate)


Equivalence partitioning

• Generalization of BVA / decision table.

• Divide the (potentially infinite) set of values into a set of equivalence classes or partitions.

• One element of the class can act as a representative for the entire class.

• Results of the test for that one element is extrapolated to all the other elements.


Equivalence partitioning: Basic hypothesis

• Suppose an input domain is divided into equivalence class partitions P1, P2, .. PN.

• The behavior of the system is IDENTICAL for all inputs belonging to the same partition Pi.

• From a testing perspective, ONE input from each partition is sufficient for the purpose of testing.


Equivalence Classes (EqC)

• Single-domain Equivalence Classes– Equivalence classes are formed on the basis

of analysis of only one input domain (e.g., age).

• Multi-domain Equivalence Classes– Multiple domains (e.g., age and gender) can

be used together.– The total number of equivalence classes is a

Cartesian product of all the individual equivalence classes.


Single-Domain EqC: Example

• Credit hour requirements for students are given as follows:– 60 credit hours for M.Tech. – 60 credit hours for MS by Research– 72 credit hours for Ph.D.

What are the equivalence classes here?

0 to 60 credit hours60 to 72 credit hoursMore than 72 credit hours

M.Tech.MS by ResearchPh.D.


Multi-Domain EqC: Example

processMessage ( messageBuffer )

processMessage is the central component of a message processing software. The processor is responsible for receiving a message buffer and processing the message that is present inside the message buffer.

The first byte of the message buffer comprises the header. The header is interpreted as unsigned integer and indicates message type. Three message types are supported (0-2). The maximum size of the message body is 1023 bytes.

The actual processing logic of the message is implemented using third-party libraries that are available.

SE 110 – Spring 2013Body Equivalence Classes

Header Equivalence Classes

Multi-Domain EqC: Example

Header value

0 2 255H1 H2

B1

B2

INVALID

INVALIDINVALIDN

um

ber

of

byt

es i

n b

od

y

H2H1

B1 B2

1

1023

INVALID

VALID INVALID

B3

B3


Multi-domain EqC: Example

• The cross product of the partitions gives the exhaustive list of combinations of input to test.– Body (B1, B2, B3) and Header (H1,H2) would

give the combinations <B1,H1>, <B2,H1>,<B3,H1>, <B1,H2>, <B2,H2>,<B3,H2>

• Pick ONE test case for each combination.


Equivalence Class Test cases

Header value

0 2 255H1 H2

B1

B2

INVALID

INVALIDINVALIDN

um

ber

of

byt

es i

n b

od

y

1

1023

INVALID

VALID INVALID

B3

Test Case A total of 6 test cases are sufficient!


Equivalence partitioning: Process

• Choose criteria for doing the equivalence partitioning (range, list of values, etc.).

• Identify the valid equivalence classes based on the above criteria (number of ranges, allowed values, etc.) .

• Select a sample data from that partition.

• Write the expected result based on the requirements given.

• Identify special values, if any, and include them in the table.

• Check to have expected results for all the cases prepared

• If the expected result is not clear for any particular test case, mark appropriately and escalate for corrective actions. If you cannot answer a question, or find an inappropriate answer, consider whether you want to record this issue on your log.


State/graph based testing

• Useful for– Language processors or compilers, wherein

the syntax of the language automatically lends itself to a state machine.

– Workflow modeling where, depending on the current state and appropriate combination of inputs, specific workflows are carried out, resulting in new output and new state.

– Dataflow modeling, where the system is modeled as a state machine.


State-based testing: Example #1

• Validation of a number: – A number can start with an optional sign.– Optional sign can be followed by any number of digits, which, in turn,

can be followed by a decimal point. – If there is a decimal point, then there should be two digits after the

decimal. – Any number, whether or not it has a decimal point, should be terminated

by a blank.

1 32 4 5

6

sign

digit

digit

digit digit

blank blank

decimal point


State-based testing: Example #2

• An employee fills up leave application.• Information goes to an automated system which validates whether

the employee is eligible for leave. – If not, application is rejected. – If yes, information goes to manager who validates the leave and gives

the final approval.

EmployeeDesires Leave

VerifyEligibility

Leave ApplicationForm

ManagerEnsure

Feasibility

Reject

Ineligible

Eligible

Feasible

Not Feasible

Approve


State-based testing: Process

• Identify the grammar for the scenario. In the above example, we have represented the diagram as a state machine. In some cases, the scenario can be a context-free grammar, which may require a more sophisticated representation of a “state diagram”, with memory etc.

• Design test cases corresponding to each valid state-input combination.

• Design test cases corresponding to the most common invalid combinations of state-input.


Compatibility testing

• To ensure that the product works consistently with infrastructure components

• Could be parameters of hardware, OS, network…– Compatibility matrix (Example in next slide)

• Backward compatibility– Testing to ensure that product parameters that are created with

an older version of the product continue to work with the current version of the same product.

• Forward compatibility– Testing to ensure that the risk involved in product for future

requirements is minimized. – Examples: Testing a product with a beta version of the operating

system, early access version of the developer’s kit etc.


Compatibility matrix: Example

Server Application server

Web server

Client Browser MS Office

Mail server

Windows 2000 Advanced Server with SP4 Microsoft SQL Server 2000 with SP3a

Windows 2000 Advanced Server with SP4 and .Net framework 1.1

IIS5.0 Win2K Professional and Win 2K Terminal Server

IE 6.0 and IE 5.5 SP2

Office 2K and Office XP

Exchange 5.5 and 2K

Windows 2000 Advanced Server with SP4 Microsoft SQL Server 2000 with SP3a

Windows 2000 Advanced Server with SP4 and .Net framework 1.1

IIS 5.0 Win2K Professional and Win 2K Terminal Server

Netscape 7.0, 7.1, Safari and Mozilla

Office 2K and Office XP

Exchange 5.5 and 2K


User documentation testing

• To check if what is stated in the document is available in the product.

• To check if what is there in the product is available in the document.

• Documents include user guides, installation guides, set-up guides, read-me files, software release notes and on-line help.


Documentation testing: Benefits

• User documentation testing aids in highlighting problems that have been overlooked during reviews.

• High-quality user documentation ensures the consistency of documentation and product, thus minimizing possible defects reported by customers.

• Results in less difficult support calls. • Contributes to better customer satisfaction and better

morale of support staff.• New programmers and testers who join a project group

can use the documentation to learn the external functionality of the product.

• Customers need less training and can proceed more quickly to advanced training and product usage.– Reduced training costs for customers.


Domain testing


Domain testing

• Have “domain people” perform tests by using the software.

• Business flow determines the test, not “logic” or “steps”.

• Tests what the users do on a typical business day.

• Captures the typical problems faced by users (not necessarily captured in SRS).


When to use what…When you want to test scenarios that have… The most effective black box testing technique

is likely to be…

Output values are dictated by certain conditions depending upon values of input variables

Decision Tables

Input values being in ranges, with each range exhibiting a particular functionality

Boundary Value Analysis

Input values being divided into classes (like ranges, list of values, etc), with each class exhibiting a particular functionality

Equivalence Partitioning

Checking for expected and unexpected input values

Positive and negative testing

Workflows, process flows or language processors

Graph based testing

To ensure that requirements are tested and met properly

Requirements Based Testing

To test using the domain expertise rather than the product specification

Domain Testing

To ensure that the documentation is consistent with the product

Documentation Testing

Black Box Testing

Documents

Transcript of Black Box Testing