Testing the Programs CS4311 – Spring 2008 Software engineering, theory and practice, S. Pfleeger,...

Testing the Programs

CS4311 – Spring 2008Software engineering, theory and practice, S. Pfleeger, Prentice Hall ed.Object-oriented and classical software engineering, S. Schach, McGraw Hill ed.Software Engineering: a Practitioner’s Approach, Pressman, McGraw Hill ed.Software Requirements: Objects, Functions, and States, Davis, Prentice Hall ed.

Plan

• Types and classification of faults• Purpose of testing• Types and levels of testing• Test planning and implementation standards

• White-box and black-box testing• Strategies for defining test cases• When to stop testing• Integration testing strategies

Software Fault and Failures

• Ideal case: each time we run our program it works

• Real case: not the above• A is an incorrect intermediate state that may be entered during program execution

• A of the system is the result of one or more in some aspect of the system

Reasons for Failure

• Wrong specifications, missing requirements

• Requirements cannot be met

• Faulty design (system design or code design)

• Error in the program, algorithm not properly implemented, etc…

Verification Approaches for Implementation

• Non execution-based Code inspection Correctness proofs

• Automated theorem proving• Testing

Informal Approaches

• Examining the code code review Experts review code and documentation Experts look for misunderstanding, inconsistencies, other faults

• Code walkthrough Programmer presents the code to the review team

Team comments on correctness Informal process Focus on finding faults but not necessary fixing them

Formal Approach-1

• Code inspection (Fagan 1976) Similar to a walkthrough but more formal The review team checks the code and documentation against a prepared list of concerns

5 steps: • • Preparation• Inspection• Rework• Follow-up

Formal Approach-2

• Rule: if more than 5% of the material inspected has been reworked, then the team will go through another iteration

• In a 6000-line business data processing application, 93% of faults were found during inspections

• Formalism of inspections is better over walkthroughs

Code Correctness

• View the code as a statement of logical flow

• Test the correctness of the expression

• Automated techniques Symbolic execution

• Formal proof techniques

Symbolic Execution

• Involves simulated execution of the code using symbols instead of data variables

• The program execution is viewed as a sequence of states depending on input

•

• The program is correct if for each input state generates the correct output state

Automated Theorem Proving

Idea: develop a tool that takes as inputinput data and conditionsoutput data and conditionslines of code for the components to be tested

Tool either provides a proof of the correctness of the component or a counter-example

Very appealing, but the perfect theorem prover is an undecidable problem

Formal Proof Techniques

int k,s;int y[n];k=0;s=0;while(k<n) {s=s+y[k];k=k+1;

}Post-condition: s contains the sum of all elements of the array y

Proof of correctness is done by induction on n

Advantages and Disadvantages of Formal Techniques

• Logic formalism helps discovering algorithmic faults in the code

• But• The logical description of the code can be difficult to obtain and much longer than the code

• Does not identify faults in design, interfaces with other components etc…

• We are not guaranteed that the proof itself is correct

What is Testing?

• We could make sure our program does what it is supposed to do by proving the program is correct

• However, program correctness is the reverse of testing

• The goal of testing is to prove the existence of a fault

• • “Program testing can be a very effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence” (Dijkstra, 1972)

SpecificationsRequirements Design Code Environment

Support Documentation Other

Requirementsor

Specifications

Functionality

HW interface

SW interface

User interface

Functionaldescription

(Inter)-processCommunications

Data definition

Module design

LogicDescription

Error Checking

Standards

Logic

Computation

Data handling

Module

Interface/Implementation

Standards

Test HW

Test SW

Integration SW

Developmenttools

Missing Unclear Wrong Changed Better WayMode:Why?

Type:What?

Origin: Where?

Hewlett-Packard’s fault classification

Views of Software-1

• Textual Measures: program lengths, frequency and occurrence of tokens

• Syntactic Code treated as syntactic elements defined by grammar

Measures: statement counts, function calls, frequency of variable use

• Control flow Analysis of program execution relative to execution order of program elements and control flow relations

Identify paths through system and to assist in testing

Views of Software-2

• Data flow Analysis of program relative to data access behavior and data flow relations

DF graphs used to examine variable definitions and references

• Computation flow Program viewed as collection of computations treated as a trace of data states produced in response to input

Fault seeding and mutation analysis is used to analyze this view

• Functional Functions denoted as input/output pairs

Types of Tests

• Performance testing Determine the performance of the system such as execution time associated with various parts of the code

• Stress testing Designed to break software module and determine strengths and limitations

• Functional testing Exercise code with nominal inputs for which expected values are available

• Structural testing

Aimed at exercising internal logic of software system

Levels of Software Testing-1

• Unit/Component testing Verify implementation of software element

Ensure logic is complete and correct; works as designed

• Integration testing

Combine software elements and test until the entire system has been integrated

Ensure design objectives satisfied Trace each test to high-level design

Levels of Software Testing-2

• System testing Ensure software as a complete entity complies with operational requirements

Trace test to system requirements• Acceptance testing

Determine if test results satisfy acceptance criteria of project stakeholder

Ensure objectives of stakeholders satisfied Trace each test to stakeholder requirements

• Installation testing Perform testing with application installed on its target platform

Overview: Test planning

• • Design test cases• Write test cases• Test test cases• Execute test cases• Evaluate test result

Unit Testing

• Focus: elementary components of the software Modules Module packages

• Approaches (depends on testing goals) Black-box testing White-box testing

What is Black-box Testing?

• • No access to the internal or logical structure

• Determine if given input produces expected output.

Input Output

Advantage and Disadvantage

• Advantage:

• Disadvantage: in most real-world applications, impossible to test all possible inputs

• Need to define an efficient strategy to limit the drawback of black-box testing

Types of Black-box Testing

• • Boundary value analysis• Syntax-driven• Decision-table• Cause-effect

Equivalence Class Testing

• Partition test cases into classes such that:

• Every possible input belongs to one of the classes

• • If we demonstrate a fault in the code

for a given input, we should demonstrate a fault with any other input from the same class (with a high probability)

• Classes are identified by looking at boundary values for the variables of the application

Example 1

• If (x > y) then S1 else S2

Equivalence classes for values of x and y: x ≤ yx > y

Example 2

• The program specification states that the program accepts 4 to 10 inputs which are five-digit integers greater than 10,000.

What are the partitions?

Example 2-Partitions

4 ≤ x ≤ 10

10,000 ≤ y ≤ 99,999

x < 4 x > 10

y <10,000 y > 99,999

Number of input values

Input values

Boundary Value Analysis

• Examines boundaries of equivalence classes

• Test cases are on or just to one side of the boundary of an equivalence class

• Consider (r1, r2) given as an input or output specification Equivalence classes would yield values x such as: x ≤ r1, r1 < x < r2, x ≥ r2

Boundary value analysis: x = r1 –1, x = r1, x = r1 +1, x = r2 –1, x = r2, x = r2 +1, r1 < x < r2

Exercise

Given the following method specification:

/** Returns true iff key is contained in values. *///@ requires values != null;//@ ensures \result == (\exist int i; i >= 0 && i <

values.length;//@ values[i] ==

key);public static boolean contains(int[] values, int key)

Do boundary analysis to select a reasonable set of test cases.

Answer

values empty not empty

key not in array key in array

first last

Test suite: (<{}, 2>, <{2,33,24,5,66,7,22,6}, 9>, <{2,33,24,5,66,7,22,6}, 2>, <{2,33,24,5,66,7,22,6}, 33>, <{2,33,24,5,66,7,22,6}, 22>, <{2,33,24,5,66,7,22,6}, 6>)

Syntax-Driven Testing

• Applicable to systems that are defined by a given grammar, e.g., formula interpretation, file input, compilers

•

• Generate test cases such that each production rule is applied at least once.

Decision Table-Based Testing

• Applicable for requirements in if-then if C1 and C2 and…and Cn then Ak

•

• Number of columns: 2ⁿ (based on conditions)

• Number of rows: n + m, where m is number of actions

• For each set of conditions, there is a corresponding action

Example: Text Editor

Functions: copy, paste, boldface, underline, and select.

Copy selected

0 0 0 0 1 1 1 1

Paste selected

0 0 1 1 0 0 1 1

Text Highlighted

0 1 0 1 0 1 0 1

Copy text 1Paste text 1

Cause-Effect Graphs

• Decision table-based drawback Test all possible cases Results in combinatorial explosion

• Cause-effect graphs are inputs are outputs Create cause-effect graph Identify dependencies limiting the number of tested combinations

Example

• Consider an ATM• Causes:o C1: Command is credito C2: Command is debito C3: Account number is valido C4: Transaction amount is valid

• Effects:o E1: Print “invalid command”o E2: Print “invalid account number”o E3: Print “debit amount not valid”o E4: Debit accounto E5: Credit account

Example (Cont.)

The graph tells which causes affect which effects.

C1

C2

C3

C4

E1

E2

E3

E4

E5

\/

/\

/\

/\

/\

¬

¬

¬Causes: C1: Command is credit C2: Command is debit C3: Account number is valid C4: Transaction amount is validEffects: E1: Print “invalid command” E2: Print “invalid account number” E3: Print “debit amount not valid” E4: Debit account E5: Credit account

Example

C1

C2

C3

C4

E1

E2

E3

E4

E5

or

and

and

and

and

¬

¬

¬

Graph determines theTest cases by tracingBack the truth values ofSelected cause.

C1 0 1 X X 1C2 0 X 1 1 XC3 X 0 1 1 1C4 X X 0 1 1E1`

1 0 0 0 0

E2 0 1 0 0 0E3 0 0 1 0 0E4 0 0 0 1 0E5 0 0 0 0 1

Example (Cont.)

Graph determines the test cases by tracingback the truth values of selected cause.

C1 0 1 X X 1C2 0 X 1 1 XC3 X 0 1 1 1C4 X X 0 1 1E1 1 0 0 0 0E2 0 1 0 0 0E3 0 0 1 0 0E4 0 0 0 1 0E5 0 0 0 0 1

C1

C2

C3

C4

E1

E2

E3

E4

E5

\/

/\

/\

/\

/\

¬

¬

¬

For E1: (C1 C2) E1(C1 C2) E1

C1 C2 E1

Example (Cont.)

Graph determines the test cases by tracingback the truth values of selected cause.

C1 0 1 X X 1C2 0 X 1 1 XC3 X 0 1 1 1C4 X X 0 1 1E1 1 0 0 0 0E2 0 1 0 0 0E3 0 0 1 0 0E4 0 0 0 1 0E5 0 0 0 0 1

C1

C2

C3

C4

E1

E2

E3

E4

E5

\/

/\

/\

/\

/\

¬

¬

¬

For E2: C1 C3

Testing the Programs CS4311 – Spring 2008 Software engineering, theory and practice, S. Pfleeger,...

Documents

Transcript of Testing the Programs CS4311 – Spring 2008 Software engineering, theory and practice, S. Pfleeger,...