Testing and Debugging

Testing, Bug Fixing and Debugging the Code

Yordan Dimitrov

Telerik Corporationwww.telerik.com

Team Leader, TeamPulse,

Telerik Corporation, Blog:http://blogs.telerik.com/

jordandimitrov/

http://codecourse.telerik.com/

Testing and Debugging

Testing, Bug Fixing and Debugging the Code

Yordan DimitrovTelerik Corporationwww.telerik.

com

Table of Contents What Is Testing? Seven Testing Principles Developer Testing

Developer vs. QA Testing

Debugging vs. Testing

Black-box vs. White-box Testing

Role of Developer Testing in Software Quality

Recommended Approach to Developer Testing

3

What Is Testing?

Main Test Activities Testing is not just running tests, but also: Planning and control Choosing test conditions Designing and executing test cases Checking results Evaluating exit criteria Reporting on the testing process

and system under test Finalizing or completing closure

activities6

Main Objectives in Testing

Testing pursues several objectives: Finding defects Gaining confidence about the level

of quality Providing information for decision-

making Preventing defects

7

Different Viewpoints / Different Objectives

Objectives of testing differ according to the point of view: Development testing – cause as

many failures as possible and fix them

Acceptance testing – confirm that the system works as expected

Assessment – assess the quality of the software and its readiness for release 8

Different Viewpoints / Different Objectives (2)

Objectives of testing differ according to the point of view: Maintenance testing – check for

new defects, introduced during development

Operational testing – assess system characteristics such as reliability or availability

9

Seven Testing Principles

Seven Testing Principles (1)

1.Testing shows presence of defects Testing can show that defects are

present Cannot prove that there are no

defects Appropriate testing reduces the

probability for defects

11

Seven Testing Principles (2)

2.Exhaustive testing is impossible All combinations of inputs and

preconditions are usually almost infinite number

Testing everything is not feasible Except for trivial cases

Risk analysis and priorities should be used to focus testing efforts

12

Seven Testing Principles (3)

3.Early testing Testing activities shall be started as

early as possible And shall be focused on defined

objectives

The later a bug is found – the more it costs!

13

Seven Testing Principles (4)

4.Defect clustering Testing effort shall be focused

proportionally To the expected and later observed

defect density of modules

A small number of modules usually contains most of the defects discovered Responsible for most of

the operational failures

14

Seven Testing Principles (5)

5.Pesticide paradox Same tests repeated over and over

again tend to loose their effectiveness Previously undetected defects

remain undiscovered

New and modified test cases should be developed

15

Seven Testing Principles (6)

6.Testing is context dependent Testing is done differently in

different contexts Example: safety-critical software is

tested differently from an e-commerce site

16

Seven Testing Principles (7)

7.Absence-of-errors fallacy Finding and fixing defects itself

does not help in these cases: The system built is unusable

Does not fulfill the users’ needs and expectations

17

Developer TestingTesting as a Priority of the Developer

Developer vs. QA Testing

Software is tested in numerous ways Some are typically performed by

developers

Some are more commonly performed by specialized test personnel QA Engineers

19

Developer Testing Developer testing refers to testing by the developer

Usually the following tests are priority of developer testing: Unit tests

Component tests

Integration tests

Sometimes regression tests and system tests are also included

20

Testing as a Priority of the QA Engineer

Numerous additional kinds of testing are performed by QA Engineers and rarely performed by developers:

21

Beta tests

Customer-acceptance tests

Performance tests

Configuration

tests

Platform tests

Stress tests

Usability tests

Etc.

Debugging vs. Testing Testing

A means of initial detection of errors

Debugging A means of diagnosing and

correcting the root causes of errors that have already been detected

22

Black-box vs. White-box Testing

Testing is usually broken into two broad categories: Black-box testing

White-box testing

24

Black-box Techniques Black-box techniques are a way to derive and select test conditions, test cases, or test data Based on an analysis of the test

basis documentation Also called specification-based or

behavioral techniques Tests are based on the way the

system is supposed to work

25

White-box Techniques White-box techniques

Also called structural or glass-box techniques

Based on an analysis of the structure of the component or system Information about how the software

is constructed?

E.g., code and detailed design information

Usually a priority of developers26

Role of Developer Testing in Software

Quality

Role of Developer Testing in Software

Quality Individual testing steps typically find less than 50 percent of the errors present each (Unit test, component test, and

integration test) The combination of testing steps often finds less than 60 percent of the errors present (Jones 1998)

28

Why is Testing a Hard Activity?

Testing's goal runs counter to the goals of other development activities

Testing can never completely prove the absence of errors

Testing by itself does not improve software quality

Testing requires you to assume that you'll find errors in your code

29

How Much Testing is Enough?

How much testing should be done is a matter of risk: Too much testing can delay the

product release and increase the product price

Insufficient testing hides risks of errors in the final product

30

How Much Time Should Be Spent in Developer

Testing ?

31

Developer testing should probably take 8 to 25 percent of the total project time

Recommended Approach to

Developer TestingGround Rules and Tips for Effective Development and Testing

Recommended Approach to Developer

Testing Test for each relevant requirement

Make sure that the requirements have been implemented

Test for each relevant design concern Make sure that the design has been

implemented

33

Recommended Approach to Developer

Testing (2) Use "basis testing" to add detailed test cases to those that test the requirements and the design

Use a checklist of the kinds of errors you've made on the project to date or have made on previous projects

34

Test First or Test Last?

Effort is the same

Detect defects earlier

Forces you to think at least a little bit

Exposes requirements problems sooner

Run it when you want

35Source: flickr

Limitations of Developer Testing

Developer tests tend to be "clean tests“

Developer testing tends to have an optimistic view of test coverage

Developer testing tends to skip more sophisticated kinds of test coverage

36

Bag of Testing Tricks Incomplete Testing Structured Basis Testing Data-Flow Testing Equivalence Partitioning Error Guessing Boundary Analysis Classes of Bad Data Classes of Good Data Use Test Cases That Make Hand-

Checks Convenient37

Structured Basis Testing

Test each statement in a program at least once

Compute the minimum number of test cases: Start with 1 for the straight path

through the routine

Add 1 for each of the following keywords, or their equivalents: if, while, repeat, for, and, and or

Add 1 for each case in a case statement If the case statement doesn't have a

default case, add 1 more

38

Sample

Statement1; <-- 1Statement2;

if ( x < 10 ) { <-- 2 Statement3;}

Statement4;

(1)Count "1" for the routine itself.(2)Count "2" for the if.

39

Sample Test Cases

40

Case

Minimum memory use Most readable output

1 Nominal case All boolean conditions are true

2 The initial for condition is false numEmployees < 1

3The first if is false

m_employee[ id ].governmentRetirementWith-held >=MAX_GOVT_RETIREMENT

4 The second if is false because the first part of the and is false

not m_employee[ id ].WantsRetirement

5 The second if is false because the second part of the and is false

not EligibleForRetirement( m_employee[id] )

6The third if is false

not EligibleForPersonalRetirement( m_employee[ id ] )

Data-Flow Testing

The normal combination of data states A variable is defined, used one or

more times, and perhaps killed

41Source: http://blog.radvision.com

Data-Flow Testing (2)

The key to writing data-flow test cases is to exercise all possible defined-used paths: All definitions

Test every definition of every variable

I.e., every place at which any variable receives a value

All defined-used combinations Test every combination of defining a

variable in one place and using it in another 42

Sample Test Casesif ( Condition 1 ) {

x = a;} else {

x = b; } if ( Condition 2 ) {

y = x + 1; } else {

y = x - 1; }

43

Case Test Description

1 Condition 1 = true, Condition 2 = true

2 Condition 1 = false, Condition 2 = false

?x = a; y = x – 1;

?x = b; y = x + 1;

Sample Test Cases

44

Case Test Description

7 Define companyRetirement in line 12, and use it first in line 26

This isn't necessarily covered by any of the previous test cases

8 Define companyRetirement in line 12, and use it first in line 31

This isn't necessarily covered by any of the previous test cases

Boundary Analysis

45

Case Test Description

1 Case 1 is defined so that the true condition for m_employee[ ID ]. governmentRetirementWithheld < MAX_GOVT_RETIREMENT is the first case on the true side of the boundary

3 Case 3 is defined so that the false condition for m_employee[ ID ]. governmentRetirementWithheld < MAX_GOVT_RETIREMENT is on the false side of the boundary

9 An additional test case is added for the case directly on the boundary in which m_employee [ ID ].governmentRetirementWithheld = MAX_GOVT_RETIREMENT

Compound Boundaries

Minimum and Maximum allowable values

46

Case Test Description

10 A large group of employees, each of whom has a large salary (what constitutes "large" depends on the specific system being developed)—for the sake of example, we'll say 1000 employees, each with a salary of $250,000, none of whom have had any social security tax withheld and all of whom want retirement withholding.

11 A group of 10 employees, each of whom has a salary of $0.00.

Classes of Bad Data

Too little data (Case 2-11)

Too much data The wrong kind of

data The wrong size of

data Uninitialized data

47

Case Test Description

12 An array of 100,000,000 employees. Tests for too much data.

13 A negative salary. Wrong kind of data.

14 A negative number of employees. Wrong kind of data.

Classes of Good Data Nominal cases—middle-of-the-road,

expected values Minimum normal configuration Maximum normal configuration Compatibility with old data

48

Case Test Description

16 A group of one employee. To test the minimum normal configuration.

17 A group of 500 employees. To test the maximum normal configuration.

Typical Errors Which Classes Contain the Most Errors?

Errors by Classification The scope of most errors is fairly

limited Many errors are outside the domain

of construction Most construction errors are the

programmers' fault Clerical errors (typos) are a

surprisingly common source of problems

49

Typical Errors (2) Errors by Classification

Misunderstanding the design is a recurring theme in studies of programmer errors

Most errors are easy to fix It's a good idea to measure your

own organization's experiences with errors

50Source: http://kathrynvercillo.com

Improving Your Testing

Planning to Test Retesting (Regression Testing) Automated Testing

51

Finding a Defect1.Stabilize the error

2.Locate the source of the errora)Gather the data

b)Analyze the data and form hypothesis

c)Determine how to prove or disprove the hypothesis

d)Prove or disprove the hypothesis by 2c

3.Fix the defect

4.Test the fix

5.Look for similar errors52

Finding a DefectDemo

Tips for Finding Defects (1)

Use all available data Refine the test cases Check unit tests Use available tools Reproduce the error several different ways

Generate more data to generate more hypotheses

Use the results of negative tests Brainstorm for possible hypotheses54

Tips for Finding Defects (2)

Narrow the suspicious region of the code

Be suspicious of classes and routines that have had defects before

Check code that’s changed recently Expand the suspicious region of the code

Integrate incrementally Check for common defects Talk to someone else about the problem

Take a break from the problem

55

Fixing a Defect Understand the problem before you fix it

Understand the program, not just the problem

Confirm the defect diagnosis Relax Save the original source code Fix the problem not the symptom Make one change at a time Add a unit test that expose the defect

Look for similar defects

56Source: http://www.movingseniorsbc.com

Your ego tells you that your code is good and doesn't have a defect even when you've seen that it has one.

How "Psychological Set" Contributes to Debugging Blindness

Psychological Considerations

57

Psychological Considerations (2)

58

Psychological Distance

How "Psychological Distance" can help

59

First Variable

Second Variable

Psychological Distance

stoppt stcppt Almost invisibleshiftrn shiftrm Almost nonedcount bcount Smallclaims1 claims2 Smallproduct sum Large

Testing and Quality Testing can give confidence in the quality of the software if it finds few or no defects

If defects are found, the quality increases when those defects are fixed

Lessons learnt from previous mistakes improve future performance

62

форум програмиране, форум уеб дизайнкурсове и уроци по програмиране, уеб дизайн – безплатно

програмиране за деца – безплатни курсове и уроцибезплатен SEO курс - оптимизация за търсачки

уроци по уеб дизайн, HTML, CSS, JavaScript, Photoshop

уроци по програмиране и уеб дизайн за ученициASP.NET MVC курс – HTML, SQL, C#, .NET, ASP.NET MVC

безплатен курс "Разработка на софтуер в cloud среда"

BG Coder - онлайн състезателна система - online judge

курсове и уроци по програмиране, книги – безплатно от Наков

безплатен курс "Качествен програмен код"

алго академия – състезателно програмиране, състезания

ASP.NET курс - уеб програмиране, бази данни, C#, .NET, ASP.NETкурсове и уроци по програмиране – Телерик академия

курс мобилни приложения с iPhone, Android, WP7, PhoneGap

free C# book, безплатна книга C#, книга Java, книга C#Дончо Минков - сайт за програмиранеНиколай Костов - блог за програмиранеC# курс, програмиране, безплатно

?

? ? ??

?? ?

?

?

?

??

?

?

? ?

Questions?

?

Testing and Debugging

http://academy.telerik.com

Free Trainings @ Telerik Academy

“High-Quality Programming Code" course @ Telerik Academy codecourse.telerik.com

Telerik Software Academy academy.telerik.com

Telerik Academy @ Facebook facebook.com/TelerikAcademy

Telerik Software Academy Forums forums.academy.telerik.com