Formal Methods Goal: software reliability Use software engineering methodologies to develop the...

45
Formal Methods

Transcript of Formal Methods Goal: software reliability Use software engineering methodologies to develop the...

Page 1: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Formal Methods

Page 2: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Goal: software reliability

Use software engineering methodologies to develop the code.

Use formal methods during code development

Page 3: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

What are formal methods?

Techniques for analyzing systems, based on some mathematics.

This does not mean that the user must be a mathematician.

Some of the work is done in an informal way, due to complexity.

Page 4: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Examples for FM

Deductive verification:Using some logical formalism, prove formally that the software satisfies its specification.

Model checking:Use some software to automatically check that the software satisfies its specification.

Testing:Check executions of the software according to some coverage scheme.

Page 5: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Typical situation:

Boss: Mark, I want that the new internet marketing software will be flawless. OK?

Mark: Hmmm. Well, ..., Aham, Oh! Ah??? Where do I start?

Bob: I have just the solution for you. It would solve everything.

Page 6: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Some concerns

Which technique? Which tool? Which experts? What limitations? What methodology? At which points? How expensive? How many people?

Needed expertise. Kind of training. Size limitations. Exhaustiveness. Reliability. Expressiveness. Support.

Page 7: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Badmouth

Formal methods can only be used by mathematicians.

The verification process is itself prone to errors, so why bother?

Using formal methods will slow down the project.

Page 8: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Some answers...

Formal methods can only be used by mathematicians.

Wrong. They are based on some math but the user should not care.

The verification process is itself prone to errors, so why bother?

We opt to reduce the errors, not eliminate them.Using formal methods will slow down the

project.Maybe it will speed it up, once errors are found

earlier.

Page 9: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Some exaggerations

Automatic verification can always find errors.

Deductive verification can show that the software is completely safe.

Testing is the only industrial practical method.

Page 10: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Our approach

Learn several methods (deductive verification, model checking, testing process algebra).

Learn advantages and limitations, in order to choose the right methods and tools.

Learn how to combine existing methods.

Page 11: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Emphasis

The process:Selecting the tools, Modeling,Verification, Locating errors.

Use of tools:Hands on. PVS, SPIN.

Visual notation:Statecharts, MSCs, UML.

Page 12: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Some emphasis

The process of selecting and using formal methods.

The appropriate notation. In particular, visual notation.

Hands-on experience with tools.

Page 13: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Where do we start?

Boss: Mark, can you verify this for me?

Mark: OK, first I have to ...

Page 14: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Things to do

Check the kind ofsoftware to analyze.

Choose methods and tools.

Express system properties.

Model the software.

Apply methods.Obtain verification

results.Analyze results.Identify errors.Suggest correction.

Page 15: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Different types of software

Sequential.Concurrent.Distributed.Reactive.Protocols.Abstract algorithms.Finite state.

Page 16: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Specification:Informal, textual, visual

The value of x will be between 1 and 5, until some point where it will become 7. In any case it will never be negative.

(1<=x<=5 U x=7) /\ [] x>=0

1<=x<=5 X=7

X>=0

Page 17: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Verification methods

Finite state machines. Apply model checking.

Apply deductive verification (theorem proving).

Program too big, too complicated.Apply testing techniques.

Apply a combination of the above!

Page 18: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Modeling

Use the program text.

Translate to a programming language embedded in some proof system.

Translate to some notation (transition system).

Translate to finite automata.

Use visual notation.

Special case: black box system.

Page 19: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Part 2 (not for specification course)

Testing

Page 20: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

What is testing?

Testing is not showing that there are no errors in the program.

Testing cannot show that the program performs its intended goal correctly.

So, what is testing?Testing is the process of executing the

program in order to find errors.A successful test is one that finds an error.

Page 21: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Some drawbacks of testing

There are never sufficiently many test

cases.

Testing does not find all the errors.

Testing is hard and takes a lot of time.

Testing is still a largely informal task.

Page 22: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Black-Box (data-driven, input-output) testing

The testing is not based on the structure of

the program (which is unknown).

In order to ensure correctness, every possible

input needs to be tested - this is impossible!

The goal: to maximize the number of errors

found.

Page 23: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

White-Box testing

Is based on the internal structure of the

program.

There are several alternative criterions for

checking “enough” paths in the program.

Even checking all paths (highly impractical)

does not guarantee finding all errors (e.g.,

missing paths!)

Page 24: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Some testing principles

A programmer should not test his/her own program. One should test not only that the program does

what it is supposed to do, but that it does not do what it is not supposed to.

The goal of testing is to find errors, not to show that the program is errorless.

No amount of testing can guarantee error-free program.

Parts of programs where a lot of errors have already been found are a good place to look for more errors.

The goal is not to humiliate the programmer!

Page 25: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Inspections and Walkthroughs

Manual testing methods. Done by a team of people. Performed at a meeting

(brainstorming). Takes 90-120 minutes. Can find 30%-70% of errors.

Page 26: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Code Inspection

Team of 3-5 people. One is the moderator. He

distributes materials and records the errors.

The programmer explains the program line by line.

Questions are raised. The program is analyzed

w.r.t. a checklist of errors.

Page 27: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Checklist for inspections

Data declarationAll variables declared?Default values

understood?Arrays and strings

initialized?Variables with similar

names?Correct initialization?

Control flowEach loop terminates?DO/END statements

match? Input/outputOPEN statements

correct?Format specification

correct?End-of-file case handled?

Page 28: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Walkthrough

Team of 3-5 people. Moderator, as

before. Secretary, records

errors. Tester, play the role

of a computer on some test suits on paper and board.

Page 29: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Selection of test cases (for white-box testing)

The main problem is to select a good coveragecriterion. Some options are:

Cover all paths of the program. Execute every statement at least once. Each decision has a true or false value at least

once. Each condition is taking each truth value at

least once. Check all possible combinations of conditions

in each decision.

Page 30: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Cover all the paths of the program

Infeasible.Consider the flow diagram

on the left.It corresponds to a loop.The loop body has 5 paths.If the loops executes 20times there are 5^20

different paths!May also be unbounded!

Page 31: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Statement coverageExecute every statement at least once

By choosingA=2,B=0,X=3each statement will

be chosen.The case where the

tests fail is not checked!

IF (A>1)&(B=0) THEN X=X/A END;

IF (A=2)|(X>1) THEN X=X+1 END;

Now x=1.5

Page 32: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Decision coverageEach decision has a true and false outcome at least once.

Can be achieved using A=3,B=0,X=3 A=2,B=1,X=1

Problem: Does not test individual conditions. E.g., when X>1 is erroneous in second decision.

IF (A>1)&(B=0) THEN X=X/A; END;

IF (A=2)|(X>1) THEN X=X+1; END;

Page 33: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Decision coverage

A=3,B=0,X=3 IF (A>1)&(B=0) THEN X=X/A; END;

IF (A=2)|(X>1) THEN X=X+1; END;

Now x=1

Page 34: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Condition coverageEach condition has a true and false value at least once.

For example: A=1,B=0,X=3 A=2,B=1,X=0

lets each condition be true and false once.

Problem:covers only the path where the first test fails and the second succeeds.

IF (A>1)(A>1)&(B=0) THEN X=X/A END;

IF (A=2)|(X>1) THEN X=X+1 END;

Page 35: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Condition coverage

A=1,B=0,X=3 IF (A>1) (A>1) & (B=0) THEN X=X/A; END;

IF (A=2) | (X>1) THEN X=X+1; END;

Page 36: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Condition coverage

A=2,B=1,X=0

Did not check the first THEN part at all!!!

Can use condition+decision coverage.

IF (A>1)(A>1)&(B=0) THEN X=X/A; END;

IF (A=2)|(X>1) THEN X=X+1; END;

Page 37: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Multiple Condition CoverageTest all combinations of all conditions in each test.

A>1,B=0 A>1,B0 A1,B=0 A1,B 0

For second IF

A=2,X>1 A=2,X1 A 2,X>1 A 2,X1

IF (A>1)&(B=0) THEN X=X/A END;

IF (A=2)|(X>1) THEN X=X+1 END;

Page 38: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

A smaller number of states that cover all cases:

A=2,B=0,X=4 A=2,B=1,X=1 A=1,B=0,X=2 A=1,B=1,X=1Note the X=4 in the firstcase: it is due to the factthat X changes beforebeing used!

IF (A>1)&(B=0) THEN X=X/A; END;

IF (A=2)|(X>1) THEN X=X+1; END;

Further optimization: not all combinations.For C /\ D, check (C, D), (C, D), (C, D).For C \/ D, check (C, D), (C, D), (C, D).

Page 39: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Test cases based on data-flow analysis

Partition the program into pieces of code with a single entry/exit point.

For each piece find which variables are set/used/tested.

Various covering criteria: from each set to each

use/test From each set to

some use/test.

x:=3

x>2?

y:=x+1

Variable set

Variable use

Variabletest

Page 40: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Test case design for black box testing

Equivalence partition Boundary value analysis Cause-effect graphs

Page 41: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Equivalence partition

Goals: Find a small number of test cases. Cover as much possibilities as you can.

Try to group together inputs for which the program would likely to behave the same.

Specificationcondition

Valid equivalenceclass

Invalid equivalenceclass

Page 42: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Example: A legal variable

Begins with A-Z Contains [A-Z0-9] Has 1-6 characters.

Specificationcondition

Valid equivalenceclass

Invalid equivalenceclass

Starting char

Chars

Length

Starts A-Z Starts other

[A-Z0-9] Has others

1-6 chars 0 chars, >6 chars

1 2

3 4

56 7

Page 43: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Equivalence partition (cont.)

Add a new test case until all valid equivalence classes have been covered. A test case can cover multiple such classes.

Add a new test case until all invalid equivalence class have been covered. Each test case can cover only one such class.

Specificationcondition

Valid equivalenceclass

Invalid equivalenceclass

Page 44: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

AB36P (1,3,5) 1XY12 (2) A17#%X (4)

Specificationcondition

Valid equivalenceclass

Invalid equivalenceclass

Starting char

Chars

Length

Starts A-Z Starts other

[A-Z0-9] Has others

1-6 chars 0 chars, >6 chars

1 2

3 4

56 7

(6) VERYLONG (7)

Page 45: Formal Methods Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.

Boundary value analysis

In every element class, select values that are closed to the boundary. If input is within range -1.0 -- +1.0,

select values -1.001, -1.0, -0.999, 0.999, 1.0, 1.001.

If needs to read N data elements, check with N-1, N, N+1. Also, check with N=0.