Software Testing 101 or Why Should We Bother With Software...

Software Testing 101or

Why Should We Bother With Software Testing?Crash Course Lecture

Marat Akhin

Saint-Petersburg State Polytechnical University

December 8th, 2009

Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 1 / 51

Contents

1 Prelude

2 Basics

3 Design for testability

4 Coverage

5 Regression testing

6 Resume


What we WON'T cover today Heavy formal stu�

Heavy formal stu�

Input: T1,T2 � AST äëÿ ïðîãðàììû äî è ïîñëå ìîäèôèêàöèèOutput: M � ìíîæåñòâî ñâÿçàííûõ óçëîâ äëÿ T1 è T2

1: M← ∅2: Îòìåòèòü âñå óçëû T1 è T2 êàê ¾íåñâÿçàííûå¿

3: for x ∈ T1 do . Îáõîäèì T1 ñíèçó ââåðõ

4: if ∃y ∈ T2 : ¬isMatched(y) ∧match(x , y) = 1 then5: M←M ∪ (x , y)6: Îòìåòèòü x è y êàê ¾ñâÿçàííûå¿

7: end if

8: end for


What we WON'T cover today Details of testing techniques

Details of testing techniques


What we WON'T cover today Speci�c testing software

Speci�c testing software


What we WILL cover today Software testing 101

Software testing 101

Some basic things about software testing that you should know...

...if you want to be a good software testerShould you learn ¾Software testing 101¿ if you want to be a goodsoftware developer?YES

I hope you'll understand why in about one and a half hours =)




Some basic things about software testing that you should know......if you want to be a good software tester

Should you learn ¾Software testing 101¿ if you want to be a goodsoftware developer?YES





Some basic things about software testing that you should know......if you want to be a good software testerShould you learn ¾Software testing 101¿ if you want to be a goodsoftware developer?

YES





Some basic things about software testing that you should know......if you want to be a good software testerShould you learn ¾Software testing 101¿ if you want to be a goodsoftware developer?YES



What we WILL cover today Why should we bother with software testing?

Why should we bother with software testing?

Short answer: because I tell you that you should do so!

Long answer: because experience of thousands and thousands ofdevelopers all over the world indicate that software testing really helpsin the development of quality software. Otherwise � why is it used inalmost every software development methodology?


What we WILL cover today Why should we bother with software testing?

Why should we bother with software testing?

Short answer: because I tell you that you should do so!Long answer: because experience of thousands and thousands ofdevelopers all over the world indicate that software testing really helpsin the development of quality software. Otherwise � why is it used inalmost every software development methodology?


Prelude What question does testing answer?

What question does testing answer?

Does this software work?

NO, testing cannot answer that

Does this software fail?

YES, testing can (and should) answer that

Testing = Destruction


Prelude What ¾testing¿ is and what isn't

What ¾testing¿ is and what isn't

It is:

Best friend of Verification and Validation

What is the di�erence between the two?

Verification checks whether the software is correct and rightValidation checks whether the software complies to a set ofrequirements

In other words:

We verify that ¾we did it right¿

We validate that ¾we did the right thing¿




It isn't:

Way to ensure 100% guarantee of any kind

The software will never fail with NPEThe database will never deadlockCalculations are always correctTime requirements are always ful�lled

We ¾can¿ give 100% guarantee but only in the most trivial cases �and testing itself is usually not needed when we talk about trivial stu�We'll cover why 100% guarantee is impossible a little bit later




It isn't:



We ¾can¿ give 100% guarantee but only in the most trivial cases �and testing itself is usually not needed when we talk about trivial stu�

We'll cover why 100% guarantee is impossible a little bit later




It isn't:



We ¾can¿ give 100% guarantee but only in the most trivial cases �and testing itself is usually not needed when we talk about trivial stu�We'll cover why 100% guarantee is impossible a little bit later


Prelude Why testing is so di�cult

Why testing is so di�cult

Theorem

Software testing is all about PAIN

Proof.

Software engineering is all about PAINSoftware testing ⊂ Software engineering




Software Method Pain

Build-and-�x It just doesn't scale up

Waterfall Model We can't predict everything at the start

Extreme Programming Writing good test cases is very di�cult

Rapid Prototyping Throwing away the prototype is hard

Uni�ed Process Following its many rules is a pain

Dan Berry: ¾The Inevitable Pain of Software Development¿

Fred Brooks: ¾No Silver Bullet¿




Brian Kernighan

¾Debugging is twice as hard as writing the code in the �rst place.Therefore, if you write the code as cleverly as possible, you are, byde�nition, not smart enough to debug it.¿

Massimo Arnoldi (feat. Kent Beck)

¾Unfortunately at least for me (and not only) testing goes against humannature. If you realize the pig in you, you will see that you program withouttests.¿

We have a very serious problem

Now we understand that software testing is really hard. But we have to doit to make quality software.

A very simple question � what do we do?


Prelude What do we do?

What do we do?

Stick our teethLearn how to minimize the pain of testing

Understand how the testing worksUnderstand how software testing can help software developmentUnderstand how software development can help software testing

Believe in software testing


Basics A very basic scheme of software testing

A very basic scheme of software testing

Run the softwareCheck the run results hoping to �nd errors

aka bugsaka faultsaka defectsaka �awsaka failures

Let's clear up the de�nition of ¾software error¿, shall we?


Basics Model of software error

Model of software error

FAILURE

FAULT

ERROR

Failure � observable from the outside

incorrect program behaviourFault � corrupted program state causedby an errorError � developer's mistake in aprogram

Let's check this model on an example


Basics Model of software error (example)

Model of software error (example)

Find the error in the following Java program

1 int sumCollection( f ina l Collection <Integer > c) {

2 int sum = 0;

3 for ( int i : c) {

4 sum += i;

5 }

6 return sum;

7 }

Possible over�ow at line 4




c = {}

What happens?


2 int sum = 0;


4 sum += i;

5 }

6 return sum;

7 }

No fault and no failure � program works correctly




c = {1, 2, 3, 5}

What happens?


2 int sum = 0;


4 sum += i;

5 }

6 return sum;

7 }

No fault and no failure � program works correctly




c = {1, 2, 3, 5, +MAX_INT, -MAX_INT}

What happens?


2 int sum = 0;


4 sum += i;

5 }

6 return sum;

7 }

Program goes through corrupted state (fault is present)But no failure visible from the outside




c = {1, 2, 3, 5, +MAX_INT}

What happens?


2 int sum = 0;


4 sum += i;

5 }

6 return sum;

7 }

Program goes through corrupted state (fault is present)And it fails with incorrect results


Basics How to �nd a software error

How to �nd a software error

Model can be represented as:

a mental representation of ¾how thissoftware should work¿requirements from software technicalspeci�cationdi�erent test casesset of correct software outputsanother (a priori correct)implementation of the samespeci�cation




Testing run must ensure the following 3 properties to actually �nd an error:

Reachibility � test must execute (¾reach¿) the software errorCorruption � error must corrupt the program state to producesoftware faultPropagation � fault must persist through the program run and causea failure in program behaviour

Making these properties hold is a major pain in software testing




To soothe this pain, we have to develop a program that is:

Controllable � we can easily make the program go where and how wewant it to goObservable � we can easily understand what the program is doing atany given time

If we don't think about that before we start developing � chances are we'llnever be able to make the program controllable and observable.

We need ¾Design for testability¿


Design for testability Test-driven development

Test-driven development

Most well known technique of ensuring ¾Design for testability¿Created by Kent Beck back in 1999Two simple principles:

We write test cases �rstThen we make them work by writing code

Forces the developer to create and maintain a good softwarespeci�cationMakes the software controllable and observable:

Each test case corresponds to a speci�c control �ow trace � thus givingus controllabilityEach test case checks a set of software outputs � thus giving usobservability


Design for testability Controllability

Controllability

Simulation and stubbing

We can easily control our own code � but we can't do that withthird-party code like OS routines, incomplete code, etc.We have to simulate or stub (S&S) third-party code to control whatit is doing

S&S may be faster than third-party codeS&S can easily produce hard-to-induce error

Downwards scalability

Many bugs live in ¾the dark corners¿ of software limitsTo make them appear we have to reach these limits � and it is easierto do so when the limits are nice and smallWe scale the limits down to reach them during testing


Design for testability Observability

Observability

Assertions

Improve observability by propagating some errors into failuresAnd we can observe the error directly � not just the results of it

Invariant checkers

Extension of assertions to do a full-scale program state checkingInspect program state at some points during execution to ensure statecorrectnessOf course we run invariant checkers only during testing � they are tooexpensive to run during actual program run

Logging

Very important in any serious project to trace what and how theprogram is doing at any given timeProjects program traces into developer-friendly form


Coverage How to measure testing?

How to measure testing?

OK � we have ¾design for testability¿And we have some kind of testing in our project

Question is:

How do we measure software testing?

We need a way to estimate testing ¾e�ciency¿ � yes, this is anothermajor pain in software testing


Coverage Four types of coverage

Four types of coverage

We use di�erent coverage criteria to measure testing e�ciencyThere are 4 basic types of coverage:

Graph based coverageLogic based coverageInput partition based coverageSyntax based coverage

Paul Ammann, Je� O�utt, ¾Introduction to Software Testing¿


Coverage Graph based coverage

Graph based coverage

Cover some elements of some graph corresponding to the programIn other words, ¾Find a graph and cover it!¿Most common type of coverage due to simplicity (we're not talkingabout computational simplicity here)Examples include:

Statement coverageBranch coveragePath coverageData �ow coverage



Statement coverage

Cover every node in program's CFG (control�ow graph)CFG node = basic block � part of code withone entry point, one exit point and no jumpinstructions within itThe simpliest of all statement coverage criteria

In our example

Single test run is enough to achieve 100% statementcoverage



Branch coverage

Cover every branch in program's CFG (control�ow graph)Stronger than statement coverage � we test allpossible control �ow jumps (but we do it¾context-free¿)

In our example

We need at least two test runs to achieve 100%branch coverage



Path coverage

Cover every possible path in program's CFG(control �ow graph)Much stronger than branch coverage � we testall possible control �ow jumps in all possiblecombinations (i.e., ¾context-sensitive¿)Unfortunately, covering all execution paths isNP-complete task

In our example

We need at least four test runs to achieve 100%path coverage

If we had even one back-edge in our CFG...


Coverage Logic based coverage

Logic based coverage

Every conditional statement gives us two possible branches to coverBut consider this one:

i f (p != q && (p == null || !p.equals(q))) {

...

}

There are four logical branches to cover � we cannot cover themsimply by covering all branches


Coverage Logic based coverage

Logic based coverage

Let's take a look at these logical branches

p != q && (p == null || !p.equals(q)) Result

T F T T

T T - T

T F F F

F - - F

Much stronger than path coverageEven more ¾context-sensitive¿ than path coverageEven more ¾NP-complete¿ than path coverage


Coverage Input partition based coverage

Input partition based coverage

In a way � evolution of logic based coverage criteriaWe have input data domain D

It can be partitioned into several equivalence classes Ei

The equivalence classes must satisfy two main properties:

Ei ∩ Ej = ∅,∀i , j : i 6= j (equivalence classes must be disjoint)⋃i Ei = D (equivalence classes must completely cover input data

domain)

Seems pretty simple, doesn't it?


Coverage Input partition based coverage

Input partition based coverage

Main problem with input partitioning is ambiguityConsider the following example:

Program that determines whether the �le is sorted in ascending ordescending orderThat's easy!

E1 � �le is sorted in ascending order

E2 � �le is sorted in descending order

E3 � �le is not sorted

What about a �le with only one record?That's correct � partitioning fails

Easier to use than logic based coverageLess complex than other coverage criteriasPartitioning can only be done manually in most cases � outweights�rst two advantages


Coverage Syntax based coverage

Syntax based coverage

A ¾very other¿ kind of coverage criteriaAs you all know � software is usually writen in a programming languageAny programming language has some syntax rules or grammarWe can use that fact to test our software � let's see how we can dothat




A program is a valid string in a given grammarWhat if we mutate this string into another one?

We can get an invalid string � we are not interested in themIf we get another valid string � it is a mutant of the original program

A mutant exhibits di�erent behaviour from the original programA ¾perfect¿ test should pass only on the original program � andshould fail on any possible mutant (kill a mutant)




That is our coverage criteriaI.e., how many mutants a test killsThis is the main idea of mutation testing

A very powerful technique of estimating testing e�ciencyVery di�cult to apply by handVery computation-intensive � NP-complete at the very least


Regression testing What kind of testing is most widely used?

What kind of testing is most widely used?

We have ¾design for testability¿We know how to estimate testing e�ciencyAnd we have some kind of testing in our project

What kind of testing it is?

Let's answer some questions to �nd the truth


Regression testing What kind of testing is most widely used?

What kind of testing is most widely used?

Does a software change big and fast during its development?

Usually � NOWe maintain, correct, adapt and modify our software most of the timeUnfortunately, these small changes can break our software big time

What do we test during development?

We test modi�ed software in order to ensure it isn't broken by ourmodi�cationsWait... That is regression testing!

Regression testing is the main testing activity in any software developmentproject � or, at the very least, it should be


Regression testing Regression testing

Regression testing

How does it work?

1 We modify our program P into P′

2 We select tests T′that need to be run on P

′

3 We add new tests T′′that test new functionality

4 We run all these tests5 We analyze testing results

Steps (2)�(5) each contain some nasty pain that we have to bear


Regression testing Regression test selection

Regression test selection

How can we select tests that need to be run after a small change?

Conservative approach � run all tests (full regression testing)Cheap approach � run tests whose test requirements are related tothe changesSmart approach � �nd parts of program that are a�ected by thechanges and run corresponding tests

Many regression test selection techniques have been developedBut almost none of them found it's way to wide-spread use


Regression testing Regression test selection

Regression test selection

Why is that so?

We need to ensure 4 basic properties of RTS:

Inclusiveness � ability to include tests that are ¾modi�cationrevealing¿. Unsafe RTS techniques have inclusiveness less than 100%Precision � ability to omit tests that are not ¾modi�cation revealing¿E�ciency � RTS is e�cient if it takes less time to determine whattests to omit than running these testsGenerality � ability to work in most practical situations

It is very di�cult to ensure all these properties at the same timeRemember ¾Good. Fast. Cheap. Pick any two¿?


Regression testing Maintaining regression test suite

Maintaining regression test suite

Regression test suite tends to grow in time and spaceWe need a way of maintaining test suiteHow to decide what tests to add and what to remove?

Add a new test for every part of new functionalityAdd a new test for every �xed bugRemove tests that are duplicates of the other testsRemove tests that have never failed

What do we do when a test fails?


Regression testing Maintaining regression test suite

Maintaining regression test suite

There are several possible reasons of a regression test failure

The software has a regression bug � Must �x and retestThe input test values are no longer valid � Must delete or update thetestThe expected output is no longer valid � Must update the test

Usually very hard to decide which situation we have

We always have to consider adding additional resources vs removingexcessive tests


Regression testing Running regression tests

Running regression tests

That's simple � take all the selected tests and run them, right?Not quite � what order do we run them in?Does it matter?

Yes, it does:

We want to get testing results ASAPThat is why we have to prioritize tests:

Di�erent heuristicsCoverage-based prioritizationHistory-based selection


Resume Summary

Summary

Now you have a very basic glimpse of:

What software testing isHow it interacts with software developmentWhat are the sources of software testing pain and how to overcomethem

There is one piece of advice left to share with you

¾Those who cannot remember the past are condemned to repeat it¿George Santayana


Resume Main rule of software testing

Main rule of software testing

It is very simple:

Either do software testing right by the book or don't do it at allYes, it is a ¾win or lose¿ situation

If you do software testing less than 100% � you're wearing rose-colouredglasses

You have a fake feeling of safetyYou think that you ensure good software quality

You imagine you're a very good developer

Sometimes it's better not to test software at all


Resume Thank you for your time and patience

Thank you for your time and patience


Software Testing 101 or Why Should We Bother With Software...

Documents

Transcript of Software Testing 101 or Why Should We Bother With Software...