SDSFoil no 1

V&V&S

Verification, Validation and Synthesis:

doing away with defects

Verification, Validation and Synthesis:

doing away with defects

SDSFoil no 2

Quality and quality assurance methods

Quality:

process quality = meeting the specification.

system quality = playing the role required by the environment.

Methods:

1. Constructive methods that aim to generate the right results in the first place

=> The SDL methodology, program generation.

2. Corrective methods that aim to detect errors and make corrections.

=> Verification and validation.

Quality:

process quality = meeting the specification.

system quality = playing the role required by the environment.

Methods:

1. Constructive methods that aim to generate the right results in the first place

=> The SDL methodology, program generation.

2. Corrective methods that aim to detect errors and make corrections.

=> Verification and validation.

SDSFoil no 3

Verification and Validation

Barry Boehm (Boehm, 1981):

Verification: To establish the truth of correspondence between a software product and its specification (from the Latin veritas, “truth”).

Validation: To establish the fitness or worth of a software product for its operational mission (from the Latin valere, “to be worth”).

Verification: Are we building the product right?

Validation: are we building the right product?”

Barry Boehm (Boehm, 1981):

Verification: To establish the truth of correspondence between a software product and its specification (from the Latin veritas, “truth”).

Validation: To establish the fitness or worth of a software product for its operational mission (from the Latin valere, “to be worth”).

Verification: Are we building the product right?

Validation: are we building the right product?”

SDSFoil no 4

Verification and Validation in TIMe

imple-mentation

instance

system

domain

config.

speci-fication

Application design

Framework design

Architecture design

needs needs

needs

Developers

Validate

validate

Market andusers

verify

verify

verify

SDSFoil no 5

Synthesize-design-implement.

Requirements

Design

Implementation

Results

Reviews analysis

Testing

Corrective methods (V&V)Defects and errors

Constructive methods

Techniques

SDSFoil no 6

Analysis

• Syntax analysis: by editor

• Static semantic analysis: by analyzer

• Dynamic semantic analysis: by simulation and reachability analysis (modelchecking)

• Verification: by manual inspection and checking formal relationships between models

• Validation: by manual inspection, by testing and formal compatibility checks

• Syntax analysis: by editor

• Static semantic analysis: by analyzer

• Dynamic semantic analysis: by simulation and reachability analysis (modelchecking)

• Verification: by manual inspection and checking formal relationships between models

• Validation: by manual inspection, by testing and formal compatibility checks

SDSFoil no 7

Static signal check 1

• Derive valid input and output signal sets from process graph

• Check signal sets against signal lists on each connection:

P2in = L2 + L3 + L5 P2out = L1 + L4 + L6

• Check signal destinations:

P2 out-to-p1 = L1 P2 out-to-p3 = L4 P2 out-to-p3 = L6

• Derive valid input and output signal sets from process graph

• Check signal sets against signal lists on each connection:

P2in = L2 + L3 + L5 P2out = L1 + L4 + L6

• Check signal destinations:

P2 out-to-p1 = L1 P2 out-to-p3 = L4 P2 out-to-p3 = L6

SDSFoil no 8

Static signal check 2

• Valid input and output signal sets against gate signal lists once for each type

• Signal lists against each other for each connection

• Signal destinations

• Valid input and output signal sets against gate signal lists once for each type

• Signal lists against each other for each connection

• Signal destinations

SDSFoil no 9

Static signal check 3

• Find logical signal routes by signal list partitioning conserving flows

• Signal sets against signal lists

• Signal destinations

• Find logical signal routes by signal list partitioning conserving flows

• Signal sets against signal lists

• Signal destinations

Who talks together here?Who talks together here?

SDSFoil no 10

Liveness = something good will eventually happen (system specific)

safety = nothing bad will ever happen (system independent)

1. Reachability analysis

Full (up to 10**5 states)

Controlled partial (up to 10**8 states)

Random simulation (larger)

2. Structural reasoning

Invariants

Logical inference on equations, rules, axioms

Model checking

3. Execution

Simulation

Testing

Liveness = something good will eventually happen (system specific)

safety = nothing bad will ever happen (system independent)

1. Reachability analysis

Full (up to 10**5 states)

Controlled partial (up to 10**8 states)

Random simulation (larger)

2. Structural reasoning

Invariants

Logical inference on equations, rules, axioms

Model checking

3. Execution

Simulation

Testing

Dynamic analysis

SDSFoil no 11

Reachability analysis

• Iteratively generate all reachable system states

• Analyse each state for

•deadlocks;

• incompleteness;

•other errors.

• Analyse the generated graph

•partial deadlock;

• liveness;

•etc.

• Iteratively generate all reachable system states

• Analyse each state for

•deadlocks;

• incompleteness;

•other errors.

• Analyse the generated graph

•partial deadlock;

• liveness;

•etc.

SDSFoil no 12

Are there any problems here?

[Gb,Sa,Qa]

[Ga, Sb,Qb]B

[Aa]

[Ra,Ea]A

[Rb,Eb]

[Ab]

1

Ra

Sb

3

Gb

Aa

5

Ea

Qb

1

Sa

Ga

8

Qa

1

1

Rb

Sa

3

Ga

Ab

5

Eb

Qa

1

1

Qb

8

Gb

Sb

A B

SDSFoil no 13

Step 1: Make transition charts

SDSFoil no 14

Step 2: Generate reachability graph

Find all possible behaviours considering that a signal transfer takes two steps:

•send

•consume

Find all possible behaviours considering that a signal transfer takes two steps:

•send

•consume

SDSFoil no 15

Findings

The errors:

• Unspecified receptions and deadlocks may occur

The reason:

• mixed initiatives not properly handled

The errors:

• Unspecified receptions and deadlocks may occur

The reason:

• mixed initiatives not properly handled

• Graph “explodes” because independent behaviours are interleaved and signals are queued

• Many paths lead to the same error

• Graph “explodes” because independent behaviours are interleaved and signals are queued

• Many paths lead to the same error

Is there a simpler way??Is there a simpler way??

SDSFoil no 16

Working directly in SDL

[Gb,Sa,Qa]

[Ga, Sb,Qb]B

[Aa]

[Ra,Ea]A

[Rb,Eb]

[Ab]

1

Ra

Sb

3

Gb

Aa

5

Ea

Qb

1

Sa

Ga

8

Qa

1

1

Rb

Sa

3

Ga

Ab

5

Eb

Qa

1

1

Qb

8

Gb

Sb

A BAssociation role behaviour =

the visible behaviour of one object when observed from another (associated) object.

Mixed initiative state = a state with inputs from two independent sources (i.e. association roles)

Association role behaviour = the visible behaviour of one object when observed from another (associated) object.

Mixed initiative state = a state with inputs from two independent sources (i.e. association roles)

invisible from B

mixed initiative

state

SDSFoil no 17

Role behaviour and input consistent roles

Deriving a role behaviour:

• Replace invisible inputs by “none” (or just make a mental note)

• Remove invisible outputs (or just ignore them)

The result is a process graph with only visible inputs and outputs. (or just a mental view on the original graph)

Input consistency:

• An invisible transition is a transition with a none input.

• An invisible transition is input consistent iff the next-state explicitly accepts all the visible inputs of the (present) state.

• The role is input consistent iff every invisible transition in the role is input consistent.

Deriving a role behaviour:

• Replace invisible inputs by “none” (or just make a mental note)

• Remove invisible outputs (or just ignore them)

The result is a process graph with only visible inputs and outputs. (or just a mental view on the original graph)

Input consistency:

• An invisible transition is a transition with a none input.

• An invisible transition is input consistent iff the next-state explicitly accepts all the visible inputs of the (present) state.

• The role is input consistent iff every invisible transition in the role is input consistent.

1

none

Sb

3

Gb

5

none

Qb

1

Sa

Ga

8

Qa

1

A

invisible transitions

1 and 3 are notinput consistentbecause 3 does not accept Sa

5 and 1 areinput consistentbecause 1 accepts more than 5

SDSFoil no 18

How to ensure input consistency ?

When checking - for each role:

• Find invisible transitions

• Check that every invisible transition is input consistent

• Consider transitions from mixed initiative states first.

When designing:

• for each state, in particular mixed initiative states, check for each role that all transitions are input consistent.

When checking - for each role:

• Find invisible transitions

• Check that every invisible transition is input consistent

• Consider transitions from mixed initiative states first.

When designing:

• for each state, in particular mixed initiative states, check for each role that all transitions are input consistent.

1

none

Sb

3

Gb

5

none

Qb

1

Sa

Ga

8

Qa

1

A

Mixed initiative state

1 and 3 are notinput consistent

5 and 1 areinput consistent

The intuition behind this is that the associated process will not immediately

observe an invisible transition and therefore the same visible outputs

should be accepted in both states

invisible transition

SDSFoil no 19

S-Rule: Input consistent process

Design SDL processes such that all their role behaviours, i.e. the behaviour visible to processes in their environment, are input consistent.

Design SDL processes such that all their role behaviours, i.e. the behaviour visible to processes in their environment, are input consistent.

In particular check the vicinity of mixed initiative statesIn particular check the vicinity of mixed initiative states

SDSFoil no 20

SDL and MSC

• The SDL behaviour shall “contain” all the MSC behaviours in addition the SDL behaviour may contain:

cases not treated by the MSCs

behaviour on other interfaces

behaviour in other services

• The SDL behaviour shall “contain” all the MSC behaviours in addition the SDL behaviour may contain:

cases not treated by the MSCs

behaviour on other interfaces

behaviour in other services

SDL (object) behaviour tree

MSCs

MSC (instance)behaviour tree

SDL

structure“contains”

SDSFoil no 21

Verifying SDL

• generate all possible traces defined by the MSCs

• check that all traces may execute in the global behaviour tree for the corresponding SDL (using the Validator)

• generate all possible traces defined by the MSCs

• check that all traces may execute in the global behaviour tree for the corresponding SDL (using the Validator)

SDL behaviour tree MSCsMSC behaviour treeSDL

SDSFoil no 22

Synthesizing behaviour 1

process ACSystem

User AC System

Code

OK

msc User_accepted

UnlockCard out

Idle

Door unlocked

User AC System

Code

NOK

msc User_rejected

Card out

Idle

Idle

Idle

IdleDoor unlocked

Code

CardOut

OK

Unlock

Code

CardOut

NOK

How to correct this?How to correct this?

SDSFoil no 23

...by combining and branching

Idle

IdleDoor unlocked

Code

CardOut

OK

Unlock

Code

CardOut

NOK

Idle

IdleDoor unlocked

CardOut

OK

Unlock

CardOut

NOK

Code

ValidateOK NOK

SDSFoil no 24

Validate PIN

wait for PIN

Idle

Process Type UserServer

DCL Cardid, PIN, zone integer;

Synthesizing behaviour 2

MSC EnterZone

User UserServer Authorizer

Card [cardid]

ValidatePIN

[cardid,PIN]

Authenticator

Givepin [PIN]

Enter PIN

exc NOKIllegal PIN

OKValidateRights

[cardid,zone]

exc NOKAccessDenied

OK

AccessGranted

ValidatePIN

(cardid, PIN)

GivePIN (PIN)

Enter PIN

Card (Cardid)

SDSFoil no 25

Synthesizing behaviour 3

MSC EnterZone

User UserServer Authorizer

Card [cardid]

ValidatePIN

[cardid,PIN]

Authenticator

Givepin [PIN]

Enter PIN

exc NOKIllegal PIN

OKValidateRights

[cardid,zone]

exc NOKAccessDenied

OK

AccessGranted

IdleValidateRights

Validate PIN

Process Type UserServer

Idle Idle

AccessGranted

OK

AccessDenied

NOK

ValidateRights

(cardid, zone)

OK

IllegalPIN

NOK

SDSFoil no 26

Synthesizing behaviour 4

General approach:

• Split each MSC into instances, and make corresponding SDL fragments.

• Combine all the SDL fragments for a given process into a more complete behaviour

• Remove non-determinism by combining and adding decisions or states

• Add unspecified behaviour details

• Add exceptions

• Consider all possibilities in each state: ensure that all roles are input consistent!

General approach:

• Split each MSC into instances, and make corresponding SDL fragments.

• Combine all the SDL fragments for a given process into a more complete behaviour

• Remove non-determinism by combining and adding decisions or states

• Add unspecified behaviour details

• Add exceptions

• Consider all possibilities in each state: ensure that all roles are input consistent!

SDSFoil no 27

Specification and design

•••

Specification

•••

Design

Verify MSC

Synthesize

Verify SDL

Decompose

Validate interface

SDSFoil no 28

Validating interfaces

• Static signal checks: are all output signals received?

• Aligning MSC: do both sides align with the same MSCs?

• Comparing role behaviours manually: are required roles contained in provided roles?

• Simulating with SDT: will the interface work for all simulated cases?

• Reachability analysis with SDT: will the validator find errors using state space exploration?

• Static signal checks: are all output signals received?

• Aligning MSC: do both sides align with the same MSCs?

• Comparing role behaviours manually: are required roles contained in provided roles?

• Simulating with SDT: will the interface work for all simulated cases?

• Reachability analysis with SDT: will the validator find errors using state space exploration?

structure

Type Type

SDSFoil no 29

Verify

Aspects:

• verifying MSC decomposition

• verifying SDL behaviour:

•object behaviour

•composition behaviour

Aspects:

• verifying MSC decomposition

• verifying SDL behaviour:

•object behaviour

•composition behaviour

context

structure

behaviour

Verifying SDL behaviourVerifying SDL behaviour

verifying MSC decomposition

Principle:

• every trace defined by the MSCs shall be possible in the SDL

Principle:

• every trace defined by the MSCs shall be possible in the SDL

aligning

SDSFoil no 30

MSC and testing

systemMSC

MSC

MSCsystemunder test

systemunder test

applicationinfrastructuresupport swhw

Abstract test suite

Executable test suite

TesterTester verdict

SDSFoil no 31

Testing what?Application (SDL).

Application support:

• Protocols.

• Input-output modules.

General support:

• Routing.

• Operating system.

• SDL runtime support.

• Error handling.

• Test and debug facilities.

Performance (non-functional):

• Real time response, delays.

• Traffic handling capacity.

• Load control.

Application (SDL).

Application support:

• Protocols.

• Input-output modules.

General support:

• Routing.

• Operating system.

• SDL runtime support.

• Error handling.

• Test and debug facilities.

Performance (non-functional):

• Real time response, delays.

• Traffic handling capacity.

• Load control.