Formal Methods and Testing

Goal: software reliability

Use software engineering methodologies to develop the code.

Use formal methods during code development

What are formal methods?

Techniques for analyzing systems, based on some mathematics.

This does not mean that the user must be a mathematician.

Some of the work is done in an informal way, due to complexity.

Examples for FM

Deductive verification:Using some logical formalism, prove formally that the software satisfies its specification.

Model checking:Use some software to automatically check that the software satisfies its specification.

Testing:Check executions of the software according to some coverage scheme.

Typical situation:

Boss: Mark, I want that the new internet marketing software will be flawless. OK?

Mark: Hmmm. Well, ..., Aham, Oh! Ah??? Where do I start?

Bob: I have just the solution for you. It would solve everything.

Some concerns

Which technique? Which tool? Which experts? What limitations? What methodology? At which points? How expensive? How many people?

Needed expertise. Kind of training. Size limitations. Exhaustiveness. Reliability. Expressiveness. Support.

Badmouth

Formal methods can only be used by mathematicians.

The verification process is itself prone to errors, so why bother?

Using formal methods will slow down the project.

Some answers...

Formal methods can only be used by mathematicians.

Wrong. They are based on some math but the user should not care.

The verification process is itself prone to errors, so why bother?

We opt to reduce the errors, not eliminate them.Using formal methods will slow down the

project.Maybe it will speed it up, once errors are found

earlier.

Some exaggerations

Automatic verification can always find errors.

Deductive verification can show that the software is completely safe.

Testing is the only industrial practical method.

Our approach

Learn several methods (deductive verification, model checking, testing process algebra).

Learn advantages and limitations, in order to choose the right methods and tools.

Learn how to combine existing methods.

Emphasis

The process:Selecting the tools, Modeling,Verification, Locating errors.

Use of tools:Hands on. PVS, SPIN (but not in this course).

Visual notation:Statecharts, MSCs, UML.

Some emphasis

The process of selecting and using formal methods.

The appropriate notation. In particular, visual notation.

Hands-on experience with tools.

Where do we start?

Boss: Mark, can you verify this for me?

Mark: OK, first I have to ...

Things to do

Check the kind ofsoftware to analyze.

Choose methods and tools.

Express system properties.

Model the software.

Apply methods.Obtain verification

results.Analyze results.Identify errors.Suggest correction.

Different types of software

Sequential.Concurrent.Distributed.Reactive.Protocols.Abstract algorithms.Finite state.

Specification:Informal, textual, visual

The value of x will be between 1 and 5, until some point where it will become 7. In any case it will never be negative.

(1<=x<=5 U x=7) /\ [] x>=0

1<=x<=5 X=7

X>=0

Verification methods

Finite state machines. Apply model checking.

Apply deductive verification (theorem proving).

Program too big, too complicated.Apply testing techniques.

Apply a combination of the above!

Modeling

Use the program text.

Translate to a programming language embedded in some proof system.

Translate to some notation (transition system).

Translate to finite automata.

Use visual notation.

Special case: black box system.

What is testing?

Testing is not showing that there are no errors in the program.

Testing cannot show that the program performs its intended goal correctly.

So, what is testing?Testing is the process of executing the

program in order to find errors.A successful test is one that finds an error.

Some drawbacks of testing

There are never sufficiently many test

cases.

Testing does not find all the errors.

Testing is hard and takes a lot of time.

Testing is still a largely informal task.

Black-Box (data-driven, input-output) testing

The testing is not based on the structure of

the program (which is unknown).

In order to ensure correctness, every possible

input needs to be tested - this is impossible!

The goal: to maximize the number of errors

found.

White-Box testing

Is based on the internal structure of the

program.

There are several alternative criterions for

checking “enough” paths in the program.

Even checking all paths (highly impractical)

does not guarantee finding all errors (e.g.,

missing paths!)

Some testing principles

A programmer should not test his/her own program. One should test not only that the program does

what it is supposed to do, but that it does not do what it is not supposed to.

The goal of testing is to find errors, not to show that the program is errorless.

No amount of testing can guarantee error-free program.

Parts of programs where a lot of errors have already been found are a good place to look for more errors.

The goal is not to humiliate the programmer!

Inspections and Walkthroughs

Manual testing methods. Done by a team of people. Performed at a meeting

(brainstorming). Takes 90-120 minutes. Can find 30%-70% of errors.

Code Inspection

Team of 3-5 people. One is the moderator. He

distributes materials and records the errors.

The programmer explains the program line by line.

Questions are raised. The program is analyzed

w.r.t. a checklist of errors.

Checklist for inspections

Data declarationAll variables declared?Default values

understood?Arrays and strings

initialized?Variables with similar

names?Correct initialization?

Control flowEach loop terminates?DO/END statements

match? Input/outputOPEN statements

correct?Format specification

correct?End-of-file case handled?

Walkthrough

Team of 3-5 people. Moderator, as

before. Secretary, records

errors. Tester, play the role

of a computer on some test suits on paper and board.

Selection of test cases (for white-box testing)

The main problem is to select a good coveragecriterion. Some options are:

Cover all paths of the program. Execute every statement at least once. Each decision has a true or false value at least

once. Each condition is taking each truth value at

least once. Check all possible combinations of conditions

in each decision.

Cover all the paths of the program

Infeasible.Consider the flow diagram

on the left.It corresponds to a loop.The loop body has 5 paths.If the loops executes 20times there are 5^20

different paths!May also be unbounded!

Statement coverageExecute every statement at least once

By choosingA=2,B=0,X=3each statement will

be chosen.The case where the

tests fail is not checked!

IF (A>1)&(B=0) THEN X=X/A END;

IF (A=2)|(X>1) THEN X=X+1 END;

Now x=1.5

Decision coverageEach decision has a true and false outcome at least once.

Can be achieved using A=3,B=0,X=3 A=2,B=1,X=1

Problem: Does not test individual conditions. E.g., when X>1 is erroneous in second decision.

IF (A>1)&(B=0) THEN X=X/A; END;

IF (A=2)|(X>1) THEN X=X+1; END;

Decision coverage

A=3,B=0,X=3 IF (A>1)&(B=0) THEN X=X/A; END;

IF (A=2)|(X>1) THEN X=X+1; END;

Now x=1

Condition coverageEach condition has a true and false value at least once.

For example: A=1,B=0,X=3 A=2,B=1,X=0

lets each condition be true and false once.

Problem:covers only the path where the first test fails and the second succeeds.

IF (A>1)(A>1)&(B=0) THEN X=X/A END;

IF (A=2)|(X>1) THEN X=X+1 END;

Condition coverage

A=1,B=0,X=3 IF (A>1) (A>1) & (B=0) THEN X=X/A; END;

IF (A=2) | (X>1) THEN X=X+1; END;

Condition coverage

A=2,B=1,X=0

Did not check the first THEN part at all!!!

Can use condition+decision coverage.

IF (A>1)(A>1)&(B=0) THEN X=X/A; END;

IF (A=2)|(X>1) THEN X=X+1; END;

Multiple Condition CoverageTest all combinations of all conditions in each test.

A>1,B=0 A>1,B0 A1,B=0 A1,B 0

For second IF

A=2,X>1 A=2,X1 A 2,X>1 A 2,X1

IF (A>1)&(B=0) THEN X=X/A END;

IF (A=2)|(X>1) THEN X=X+1 END;

A smaller number of states that cover all cases:

A=2,B=0,X=4 A=2,B=1,X=1 A=1,B=0,X=2 A=1,B=1,X=1Note the X=4 in the firstcase: it is due to the factthat X changes beforebeing used!

IF (A>1)&(B=0) THEN X=X/A; END;

IF (A=2)|(X>1) THEN X=X+1; END;

Further optimization: not all combinations.For C /\ D, check (C, D), (C, D), (C, D).For C \/ D, check (C, D), (C, D), (C, D).

Test cases based on data-flow analysis

Partition the program into pieces of code with a single entry/exit point.

For each piece find which variables are set/used/tested.

Various covering criteria: from each set to each

use/test From each set to

some use/test.

x:=3

x>2?

y:=x+1

Variable set

Variable use

Variabletest

Test case design for black box testing

Equivalence partition Boundary value analysis Cause-effect graphs

Equivalence partition

Goals: Find a small number of test cases. Cover as much possibilities as you can.

Try to group together inputs for which the program would likely to behave the same.

Specificationcondition

Valid equivalenceclass

Invalid equivalenceclass

Example: A legal variable

Begins with A-Z Contains [A-Z0-9] Has 1-6 characters.

Specificationcondition

Valid equivalenceclass

Invalid equivalenceclass

Starting char

Chars

Length

Starts A-Z Starts other

[A-Z0-9] Has others

1-6 chars 0 chars, >6 chars

1 2

3 4

56 7

Equivalence partition (cont.)

Add a new test case until all valid equivalence classes have been covered. A test case can cover multiple such classes.

Add a new test case until all invalid equivalence class have been covered. Each test case can cover only one such class.

Specificationcondition

Valid equivalenceclass

Invalid equivalenceclass

AB36P (1,3,5) 1XY12 (2) A17#%X (4)

Specificationcondition

Valid equivalenceclass

Invalid equivalenceclass

Starting char

Chars

Length

Starts A-Z Starts other

[A-Z0-9] Has others

1-6 chars 0 chars, >6 chars

1 2

3 4

56 7

(6) VERYLONG (7)

Boundary value analysis

In every element class, select values that are closed to the boundary. If input is within range -1.0 -- +1.0,

select values -1.001, -1.0, -0.999, 0.999, 1.0, 1.001.

If needs to read N data elements, check with N-1, N, N+1. Also, check with N=0.