Functional System Testing

Written by

Adam Carmi

Black Box Testing2

Outline

• Goal of testing• Test cases, test suites and test data• What is functional system testing?• Coverage• Functional testing techniques:

– Functional analysis– Equivalence partitioning– Boundary value analysis

Black Box Testing3

The goal of software testing

• The process of uncovering evidence of defects in software systems– Does not include efforts associated with tracking down

bugs and fixing them

• No amount of testing will improve the quality of a computer program– The more testing we do of a system, the more

convinced we might be of its correctness– Testing cannot in general prove a system works 100%

correctly

Black Box Testing4

Test cases

• The basic component of testing is a Test Case• In its most general form: (inputs, expected-result)

– inputs include system state, user commands and data values to be processed

– expected result includes visible/audible interface changes or changes in the system state

• Test cases are organized into Test Suites– functionality, security, performance, …

Black Box Testing5

Test case execution

• A running of the software (under test) that provides the inputs specified in the test case and observing the results and comparing them to those specified by the test case– If the actual result varies from the expected

result, then a failure has been detected

Black Box Testing6

Test data

• An effective test strategy requires careful acquisition and preparation of test data prior to testing– Testing can suffer if test data is poor

• Test data concerns:– Depth: quantity and size of data– Breadth: variance of data values and data types– Scope: completeness, relevance and accuracy of data

• Result of a query should be valid for the specific purpose of the query, and not due to a missing or inappropriate value

– Conditions: data should reflect specific “conditions” in the domain• Data that would otherwise arrive after performing specific operations

over time

• Test data and test results are expensive to construct

Black Box Testing7

Example: Test data for TVRS

Name: test1.db

Description: Violation records designed for validating violation lookup

Violation IDOffender’s first nameOffender’s last nameIssuing policeman ID

243567RachelJosef8700342

237812DanLevi6386541

264683DanPorat1346329

255245DinaJosef8245731

000345longFirstNlongLastN8700342

…………

Black Box Testing8

Specification Vs. Implementation

• The basic approaches to testing software are based on its specification and implementation

• White box testing – test cases and data are constructed based on the code that implements the software– Quality and correctness of computations is validated

– Will not be further discussed in this tutorial

• Black box testing – test cases and data are constructed based solely on the software’s specification

Black Box Testing9

Functional System Testing

• Testing of a completed application to determine that it provides all of the behaviors required of it– Testing of completed increments that provide some

degree of end-user functionality

• Search for defects that are variances between the actual operation of the system and the requirements for the system

• System is treated as a black box

Black Box Testing10

How much testing is adequate?

• Completely validating IEEE 754 floating-point division requires 264 test-cases!

float divide(float x, float y)

• From practical and economic perspectives, exhaustive testing is usually not possible– Which software pieces should we test?– Which test cases should we choose?

Black Box Testing11

Coverage

• Coverage is a measure of how completely a test suite exercises the capabilities of a piece of software– “Each line of code should be executed at least once”

– “One test case should be constructed from each specified requirement”

• It is necessary to use testing techniques that narrow down the number of test cases allowing the broadest testing coverage with the least effort

Black Box Testing12

Technique: Functional Analysis

• Analyze the expected behavior of the system according to its functional specification

• Generate a test procedure for each of the possible usage scenarios– Corresponds to use case scenarios

– Analyze how a change in one part of the system affects other parts

– “Grand tour” test cases: the result of one test case produces the data that is the input to the next test case

Black Box Testing13

Example: Functional Analysis I

Use Case: Remove Traffic Violation

1. Supervisor calls for deletion of the chosen Traffic Violation

2. TVRS prompts Supervisor for confirmation

3. Supervisor confirms

4. TVRS requests OffendersDB to delete the Traffic Violation from the offender’s record

5. OffendersDB approves that the Traffic Violation has been deleted

6. TVRS allows Supervisor to look up a new Traffic Violation as described in the “Lookup Traffic Violation” UC

Use Case: Remove Traffic Violation

1. Supervisor calls for deletion of the chosen Traffic Violation

2. TVRS prompts Supervisor for confirmation

3. Supervisor confirms

4. TVRS requests OffendersDB to delete the Traffic Violation from the offender’s record

5. OffendersDB approves that the Traffic Violation has been deleted

6. TVRS allows Supervisor to look up a new Traffic Violation as described in the “Lookup Traffic Violation” UC

Black Box Testing14

Example: Functional Analysis IITest case ID: 134543

Pre-conditions: 1. TVRS initialized with test1.db database

2. Violation 243567 displayed in the “Lookup Violation” dialog

Related use cases: “Lookup Traffic Violation”, “Remove Traffic Violation”

ActionExpected result

Press “Delete” buttonConfirmation dialog is displayed

Press the “Yes” button“Lookup Violation” dialog is displayed

Enter “243567” at “Violation ID” text field and press the “Search” button

A message dialog stating that violation “243567” is not stored in TVRS

Test results

Passed

Failed

Actual results:

Defect diagnosis:

Black Box Testing15

Example: Functional Analysis III

Verify effects of change

Filled when the test case is executed

How do we know that violation 243567 is

stored in the system?

How do we know that violation 243567 is

stored in the system?

Can a tester diagnose the cause of a defect?

Can a tester diagnose the cause of a defect?

In addition, a query could be run on the Offenders database

In addition, a query could be run on the Offenders database

Black Box Testing16

Technique: Equivalence Partitioning

• Identifies ranges of input and initial conditions that are expected to produce the same result

• A group of test cases form an equivalence class if:– They test the same feature/scenario– If one test reveals a fault, the other ones (probably) will

too– If a test does not reveal a fault, the other ones

(probably) will not either• It is adequate to use only a single representative of

the equivalence class

Black Box Testing17

Example: Equivalence Partitioning I

Input value specification for “Lookup Violation” form:Input value specification for “Lookup Violation” form:

Field nameValid values

Violation ID[0-9]{0, 9}

Offender’s first name[a-zA-Z]{0, 10}

Offender’s last name[a-zA-Z]{0, 10}

Black Box Testing18

Example: Equivalence Partitioning II

FieldValid equivalent

classes

Validrepresentative values

Invalid

equivalent

classes

Invalidrepresentative values

Violation IDKnown violation00243567ID < 0 or ID > 999999999

-1, 1234567890

Unknown violation32456720Non numeric ID23ab@

Empty“”

Offender’sfirst name

Unknown violationDavidCharacter# > 10Hasalongname

Single known violation

RachelInvalid characterad0@am

Many known violations

Dan

Empty“”

…………

Black Box Testing19

Example: Equivalence Partitioning III

• The number of test cases to choose from is reduced to(3 + 2) × (4 + 2) × (4 + 2) = 180

• The actual number can be further limited– Single invalid field per test case (3 × 4 × 4 + 6 = 54)– Importance of use case– Resources available– Most frequent input– Life-critical software– Infeasible test cases– Randomly– ...

Black Box Testing20

Technique: Boundary Value Analysis

• Based on experience / heuristics– Testing boundary conditions of equivalence classes is

more effective

• Choose input boundary values as equivalence classes representatives

• Choose inputs that invoke output boundary values• Examples:

– (0, 10] ⇒ validate using 0, 1, 2, 9, 10, 11

– Read up to 5 elements ⇒ validate reading 0, 1, 4, 5, 6 elements

Black Box Testing21

BVA as an equivalence partitioning extension

• Choose one (or more) arbitrary value(s) in each equivalence class

• Choose valid values exactly on lower and upper boundaries of equivalence class

• Choose invalid values immediately below and above each boundary (if applicable)

Black Box Testing22

General purpose test-suite construction technique

• May be used to obtain reasonable coverage with little effort– Use cautiously!– Unsuitable when values of different fields are related

1. While test cases can be added• Each new test case should include as many un-included valid non-boundary

equivalence class representatives as possible2. While test cases can be added

• Each new test case should include as many un-included valid boundary equivalence class representatives as possible

3. While test cases can be added• Each new test case should include a single invalid equivalence class

representative that has not been included before4. Manually replace/remove redundant or infeasible test-cases

1. While test cases can be added• Each new test case should include as many un-included valid non-boundary

equivalence class representatives as possible2. While test cases can be added

• Each new test case should include as many un-included valid boundary equivalence class representatives as possible

3. While test cases can be added• Each new test case should include a single invalid equivalence class

representative that has not been included before4. Manually replace/remove redundant or infeasible test-cases

Black Box Testing23

Example: Country Club I

DaySunday - ThursdayFriday - Saturday

Guest

status

VisitorMemberStudentVisitorMemberStudent

Age (years)

Admission fee

[0, 16)251020351030

[16, 60)502545702565

[60, 120]351530501545

SpecificationSpecification

Black Box Testing24

Example: Country Club II

FieldValid equivalent

classes

Validrepresentative values

Invalid

equivalent

classes

Invalidrepresentative values

DaySun - ThuMon, Sun, Thu

Fri - SatFri, Sat

Guest status

VisitorVisitor

MemberMember

StudentStudent

Age[0, 16)2, 0, 15Non-numeric value

14@a

[16, 60)34, 16, 59Age < 0 or

Age > 120

-1, 121

[60, 120]100, 60, 120

A combo box is used for choosing the day

and guest status

A combo box is used for choosing the day

and guest status

Black Box Testing25

Example: Country Club IIITest case IDDayGuest statusAgeResult

1MonVisitor225

2FriMember3425

3MonStudent10030

4SunVisitor025

5SatMember1610

6ThuStudent6030

7SunMember1510

8SatStudent12045

9ThuVisitor5950

10MonMember14@aInvalid age

11FriStudent-1Invalid age

12FriVisitor121Invalid age

valid

valid)boundary(

invalid