Testing and Debugging

Testing, Bug Fixing and Debugging the Code

Yordan DimitrovTelerik Corporationwww.telerik.

com

Testing

Testing is a means of detecting errors.

2

Developer Testing Unit testing Component testing Integration testing Regression testing System testing

3

Additional testing beta tests customer-acceptance tests performance tests configuration tests platform tests stress tests usability tests

4

Black and White

Black-box testing White-box (or glass-box) testing.

5

Role of Developer Testing in Software

Quality

Individual testing steps (unit test, component test, and integration test) typically find less than 50 percent of the errors present each.

The combination of testing steps often finds less than 60 percent of the errors present (Jones 1998).

6

Why is testing a hard activity?

Testing's goal runs counter to the goals of other development activities

Testing can never completely prove the absence of errors

Testing by itself does not improve software quality

Testing requires you to assume that you'll find errors in your code

7

How much time should be spent in developer

testing ?

8

Developer testing should probably take 8 to 25 percent of the total project time

Recommended Approach to Developer

Testing Test for each relevant requirement to make sure that the requirements have been implemented

Test for each relevant design concern to make sure that the design has been implemented

Use "basis testing" to add detailed test cases to those that test the requirements and the design

Use a checklist of the kinds of errors you've made on the project to date or have made on previous projects

9

Test First or Test Last?

Effort is the same Detect defects earlier

Forces you to think at least a little bit

Exposes requirements problems sooner

Run it when you want

10source flickr

Limitations of Developer Testing

Developer tests tend to be "clean tests“

Developer testing tends to have an optimistic view of test coverage

Developer testing tends to skip more sophisticated kinds of test coverage

11

Bag of Testing Tricks Incomplete Testing Structured Basis Testing Data-Flow Testing Equivalence Partitioning Error Guessing Boundary Analysis Classes of Bad Data Classes of Good Data Use Test Cases That Make Hand-Checks Convenient 12

Structured Basis Testing

Test each statement in a program at least once.

Compute the minimum number of test cases:

1. Start with 1 for the straight path through the

routine.

2. Add 1 for each of the following keywords, or their

equivalents: if, while, repeat, for, and, and or.

3. Add 1 for each case in a case statement. If the case

statement doesn't have a default case, add 1 more.

13

Sample Test Cases

14

Case

Test Description Test Data

1 Nominal case All boolean conditions are true

2 The initial for condition is false

numEmployees < 1

3 The first if is false m_employee[ id ].governmentRetirementWith-held >=MAX_GOVT_RETIREMENT

4 The second if is false because the first part of the and is false

not m_employee[ id ].WantsRetirement

5 The second if is false because the second part of the and is false

not EligibleForRetirement( m_employee[id] )

6 The third if is false not EligibleForPersonalRetirement( m_employee[ id ] )

Data-Flow Testing

The normal combination of data states is that a variable is defined, used one or more times, and perhaps killed.

The key to writing data-flow test cases is to exercise all possible defined-used paths:

• All definitions. Test every definition of every

variable—that is, every place at which any

variable receives a value.

• All defined-used combinations. Test every

combination of defining a variable in one place

and using it in another.

15

Sample Test Cases

Case Test Description

7 Define companyRetirement in line 12, and use it first in line 26.

This isn't necessarily covered by any of the previous test cases.

8 Define companyRetirement in line 17, and use it first in line 31.

This isn't necessarily covered by any of the previous test cases.

16

if ( Condition 1 ) {

x = a;} else {

x = b; } if ( Condition 2 ) {

y = x + 1; } else {

y = x - 1; }

Boundary Analysis

Case

Test Description

1 Case 1 is defined so that the true condition for m_employee[ ID ]. governmentRetirementWithheld < MAX_GOVT_RETIREMENT is the first case on the true side of the boundary. T

3 Case 3 is defined so that the false condition for m_employee[ ID ]. governmentRetirementWithheld < MAX_GOVT_RETIREMENT is on the false side of the boundary.

9 An additional test case is added for the case directly on the boundary in which m_employee [ ID ].governmentRetirementWithheld = MAX_GOVT_RETIREMENT.

17

Compound Boundaries

Case Test Description

10 A large group of employees, each of whom has a large salary (what constitutes "large" depends on the specific system being developed)—for the sake of example, we'll say 1000 employees, each with a salary of $250,000, none of whom have had any social security tax withheld and all of whom want retirement withholding.

11 A group of 10 employees, each of whom has a salary of $0.00.

18

Minimum and Maximum allowable values

Classes of Bad Data

Too little data

Too much data

The wrong kind of data

The wrong size of data

Uninitialized data

19

Case

Test Description

12 An array of 100,000,000 employees. Tests for too much data.

13 A negative salary. Wrong kind of data.

14 A negative number of employees. Wrong kind of data.

Classes of Good Data Nominal cases—middle-of-the-road,

expected values

Minimum normal configuration

Maximum normal configuration

Compatibility with old data

20

Case

Test Description

16 A group of one employee. To test the minimum normal configuration.

17 A group of 500 employees. To test the maximum normal configuration.

Typical Errors Which Classes Contain the Most Errors? Errors by Classification

The scope of most errors is fairly limited

Many errors are outside the domain of construction

Most construction errors are the programmers' fault

Clerical errors (typos) are a surprisingly common source of problems

Misunderstanding the design is a recurring theme in studies of programmer errors

Most errors are easy to fix

It's a good idea to measure your own organization's experiences with errors

21

Improving Your Testing

Planning to Test Retesting (Regression Testing) Automated Testing

23

Debugging

Debugging is a means of diagnosing and correcting the root causes of errors that have already been detected.

24

Finding a Defect

1. Stabilize the error

2. Locate the source of the errora. Gather the data

b. Analyze the data and form hypothesis

c. Determine how to prove or disprove the hypothesis

d. Prove or disprove the hypothesis by 2c

3. Fix the defect

4. Test the fix

5. Look for similar errors25

DEMO: Finding a Defect

26

The program is supposed to print a list of employees and their income-tax withholdings in alphabetical order.

Tips for Finding Defects (1)

Use all available data Refine the test cases Check unit tests Use available tools Reproduce the error several different ways

Generate more data to generate more hypotheses

Use the results of negative tests Brainstorm for possible hypotheses27

Tips for Finding Defects (2)

Narrow the suspicious region of the code

Be suspicious of classes and routines that have had defects before

Check code that’s changed recently Expand the suspicious region of the code

Integrate incrementally Check for common defects Talk to someone else about the problem

Take a break from the problem

28

Fixing a Defect Understand the problem before you fix it

Understand the program, not just the problem

Confirm the defect diagnosis Relax Save the original source code Fix the problem not the symptom Make one change at a time Add a unit test that expose the defect

Look for similar defects

29

Psychological Consideration

30

Your ego tells you that your code is good and doesn't have a defect even when you've seen that it has one.

How "Psychological Set" Contributes to Debugging Blindness

Psychological Distance

How "Psychological Distance" Can Help

31

First Variable Second Variable

Psychological Distance

stoppt stcppt Almost invisibleshiftrn shiftrm Almost nonedcount bcount Smallclaims1 claims2 Smallproduct sum Large

Tools Diff Tools Compiler Warning Messages

Set your compiler’s warning level to the highest

Treat warnings as errors

Initiate project wide standards Extended Syntax and Logic Checking

Profilers Test Frameworks/Scaffolding Debuggers

32

Questions?