Black box software testing

1

01-06-2015

Black Box Software Testing

By Muhammad Soban & Muhammad Umair

2

Black box testing is a software testing techniques in whichfunctionality of the software under test (SUT) is tested without looking at the internal code structure, implementation details and knowledge of internal paths of the software.

This type of testing is based entirely on the software requirements and specifications.

Introduction Black Box Testing

3

In Black Box Testing we just focus on inputs and output of the software system.

The above Black Box can be any software system you want to test. For example : an operating system like Windows, a website like Google ,a DBMS like Oracle or even your own custom application.

Black Box Testing

4

Black box testing tends to find different kinds of errors than white box testing

Missing functions Usability problems Performance problems Concurrency and timing errors Initialization and termination errors

Unlike white box testing, black box testing tends to be applied later in the development process

Black Box Testing

5

Equivalence Partitioning

Black Box Testing Techniques

6

Typically the universe of all possible test cases is so large that you cannot try them all

You have to select a relatively small number of test cases to actually run

Which test cases should you choose?

Equivalence partitioning helps answer this question


7

Partition the test cases into "equivalence classes"

Each equivalence class contains a set of "equivalent" test cases

Two test cases are considered to be equivalent if we expect the program to process them both in the same way (i.e., follow the same path through the code)

If you expect the program to process two test cases in the same way, only test one of them, thus reducing the number of test cases you have to run


8

First-level partitioning: Valid vs. Invalid test cases


Valid Invalid

9

Partition valid and invalid test cases into equivalence classes


10

If an oracle is available, the test values in each equivalence class can be randomly generated. This is more useful than always testing the same static values.

Oracle: something that can tell you whether a test passed or failed

Test multiple values in each equivalence class. Often

you’re not sure if you have defined the equivalence classes correctly or completely, and testing multiple values in each class is more in-depth than relying on a single value.


Create a test case for at least one value from each equivalence class


11

12

When designing test cases, you may use different definitions of “equivalence”, each of which will partition the test case space differently

Example: int Add(n1, n2, n3, …) Equivalence Definition 1: partition test cases by the number of inputs

(1, 2, 3, etc.) Equivalence Definition 2: partition test cases by the number signs they

contain (positive, negative, both) Equivalence Definition 3: partition test cases by the magnitude of

operands (large numbers, small numbers, both) Etc.


13

Equivalence Partitioning Examples

Input Valid Equivalence Classes

Invalid Equivalence Classes

A integer N such that:-99 <= N <= 99

[-99, -10][-9, -1]0[1, 9][10, 99]

< -99> 99Malformed numbers{12-, 1-2-3, …}Non-numeric strings{junk, 1E2, $13}Empty value

Phone Number

Area code: [200, 999]

555-5555(555)555-5555555-555-5555200 <= Area code <= 999

Invalid format 5555555, (555)(555)5555, etc.

Area code < 200 or > 999Area code with non-numeric characters

14

When choosing values from an equivalence class to test, use the values that are most likely to cause the program to fail

Errors tend to occur at the boundaries of equivalence classes rather than at the "center"

If (200 < areaCode && areaCode < 999) { // valid area code } Wrong! If (200 <= areaCode && areaCode <= 999) { // valid area code } Testing area codes 200 and 999 would catch this error, but a center value

like 770 would not

In addition to testing center values, we should also test boundary values

Right on a boundary Very close to a boundary on either side

Boundary Value Analysis

15

Create test cases to test boundaries of equivalence classes

Boundary Value Analysis

16

Boundary Value Analysis Examples

Input Boundary Cases

A number N such that:-99 <= N <= 99

-100, -99, -98-10, -9-1, 0, 19, 1098, 99, 100

Area code: [200, 999] Area code: 199, 200, 201Area code: 998, 999, 1000

17

Numeric values are often entered as strings which are then converted to numbers internally [int x = atoi(str);]

This conversion requires the program to distinguish between digits and non-digits

A boundary case to consider: Will the program accept / and : as digits?

Boundary Value Analysis Examples

Char

ASCII

/ 0 1 2 3 4 5 6 7 8 9 :

47 48 49 50 51 52 53 54 55 56 57 58

18

Based on intuition, guess what kinds of inputs might cause the program to fail

Create some test cases based on your guesses

Intuition will often lead you toward boundary cases, but not always

Some special cases aren't boundary values, but are mishandled by many programs

Try exiting the program while it's still starting up Try loading a corrupted file Try strange but legal URLs: hTtP://Www.bYu.EDU/

Error Guessing

19

Functional testing This black box testing type is related to functional

requirements of a system e.g. login system working properly. It is done by software testers.

Non-functional testing This type of black box testing is not related to testing of a

specific functionality , but non-functional requirements such as scalability, usability, performance e.g. system under Low memory conditions

Types of Black Box Testing

20

Types of Functional Black Box Testing

21

Individual components are tested to ensure that they operate correctly

The size of a single module is small enough that we can locate an error fairly easily

It is usually performed by the developer.

Unit Testing(White Box Testing)

22

Incorrect initialization Precision inaccuracy Improper loop termination Improperly modified loop variables Incorrect arithmetic precedence Comparison of different data types Incorrect logical operators or precedence Incorrect comparison of variables Incorrect symbolic representation of an expression

Test cases in unit testing should uncover errors

23

A Stub is a small program routine that substitutes for a longer program, possibly to be loaded later or that is located remotely

A Driver is nothing more than a “main program” that accepts test-case data, passes such data to the component (to be tested), and prints relevant results

Drivers and stubs represent overhead. That is, both are software that must be written but that is not delivered with the final software product.

Unit Test Procedure

24

Unit-tested modules are combined into subsystems, which are then tested.

The goal here is to see if the modules can be integrated properly

Integration Testing

25

Integration Testing: Incremental Approach First combine only two components together and test

them. Remove the errors if they are there, otherwise combine another component to it and then test again, and so on until the whole system is developed

26

Integration Testing: Top-Down

Modules are integrated by moving downward through the control hierarchy beginning with the main control module.

Depth-first integration:M1, M2, and M5 would be integrated first. Next, M8 or M6 would be integrated. Then, the central and right-hand control paths are built.

Breadth-first integration:M2, M3, and M4 would be integrated first. The next control level M5, M6, and so on follows.

27

Integration Testing: Bottom-Up Begins construction and testing with

the components at the lowest level in the program structure.

1) Low-level components are combined into clusters (sometimes called builds) that perform specific software sub-functions.

2) A driver (a control program for testing) is written to coordinate test case input and output.

3) The cluster is tested.4) Drivers are removed and clusters

are combined moving upward in the program structure.

28

Test Case ID

Test Case Objective Test Case Description

Expected Result

1 Check the interface link between the Login and Mailbox module

Enter login credentials and click on the Login button

To be directed to the Mail Box

2 Check the interface link between the Mailbox and Delete Mails Module

From Mail box select the an email and click delete button

Selected email should appear in the Deleted/Trash folder

Integration Test Case:

29

System testing is the testing of a complete and fully integrated software product

The testing process is concerned with finding errors that result from unexpected interactions between subsystems and system components

It is also concerned with validating that the system meets its functional and non-functional requirements

System Testing

30

Alpha testing refers to the system testing carried out by the test team within the development organization

Beta testing is the system testing performed by a selected group of friendly customers

Acceptance testing is the system testing performed by the customer to determine whether to accept or reject the delivery of the system

Three main kinds of system testing

31

Installation Testing

Types of Non-Functional Black Box Testing

32

Check that software application is successfully installed and it is working as expected after installation

Make sure that users do not have any trouble when installing your software

Installation Testing

33

Required disk space is1MB, then make sure exactly 1MB

Test disk space requirement on different file system format. Like FAT16 will require more space than efficient NTFS or FAT32 file systems

Try to automate the routine to test the number of files to be written on disk

Verify registry changes on successful installation

Forcefully break the installation process in between and check system recovers to its original state without any issues

Installation Testing Tips With Some Broad Test Cases

34

If test cases are distributed from Internet, test cases should be included for

Bad network speed and broken connection Firewall and security related Size and approximate time taken Concurrent installation/downloads

Installation Testing Tips With Some Broad Test Cases

35

Determines a system's performance under real-life load conditions

Load testing identifies the bottlenecks in the system under various workloads and checks how the system reacts when the load is gradually increased

Load Test

36

The maximum operating capacity of an application

Determine whether current infrastructure is sufficient to run the application

Sustainability of application with respect to peak user load

Number of concurrent users that an application can support, and scalability to allow more users to access it

Load testing is commonly used for the Client/Server, Web based applications - both Intranet and Internet

Load testing usually identifies

37

Load Runner Web Load Astra Load Test Rad view's Web Load Studio, Rational Site Load Silk Performer

Load Testing Tools

38

Interoperability testing involves testing whether a given software program or technology is compatible with others and promotes cross-use functionality

For example, major studies have been conducted to determine how devices used for heart transplants interact with various microwave technologies. In software development, interoperability testing examines how applications function with other applications or services

Interoperability Testing

39

Stress testing Usability testing Security testing Recovery testing Reliability Testing Scalability testing Availability Testing Performance testing Compatibility testing Configuration Testing Documentation testing Maintainability Testing Localization testing and Internationalization testing

Other Non Functional Black Box Testing

40

(Dijkstra et al., 1972)

Testing can only show the presence of errors, not their absence

Black box software testing

Documents

Transcript of Black box software testing