V&V&S
-
Upload
ivana-goodwin -
Category
Documents
-
view
16 -
download
0
description
Transcript of V&V&S
SDSFoil no 1
V&V&S
Verification, Validation and Synthesis:
doing away with defects
Verification, Validation and Synthesis:
doing away with defects
SDSFoil no 2
Quality and quality assurance methods
Quality:
process quality = meeting the specification.
system quality = playing the role required by the environment.
Methods:
1. Constructive methods that aim to generate the right results in the first place
=> The SDL methodology, program generation.
2. Corrective methods that aim to detect errors and make corrections.
=> Verification and validation.
Quality:
process quality = meeting the specification.
system quality = playing the role required by the environment.
Methods:
1. Constructive methods that aim to generate the right results in the first place
=> The SDL methodology, program generation.
2. Corrective methods that aim to detect errors and make corrections.
=> Verification and validation.
SDSFoil no 3
Verification and Validation
Barry Boehm (Boehm, 1981):
Verification: To establish the truth of correspondence between a software product and its specification (from the Latin veritas, “truth”).
Validation: To establish the fitness or worth of a software product for its operational mission (from the Latin valere, “to be worth”).
Verification: Are we building the product right?
Validation: are we building the right product?”
Barry Boehm (Boehm, 1981):
Verification: To establish the truth of correspondence between a software product and its specification (from the Latin veritas, “truth”).
Validation: To establish the fitness or worth of a software product for its operational mission (from the Latin valere, “to be worth”).
Verification: Are we building the product right?
Validation: are we building the right product?”
SDSFoil no 4
Verification and Validation in TIMe
imple-mentation
instance
system
domain
config.
speci-fication
Application design
Framework design
Architecture design
needs needs
needs
Developers
Validate
validate
Market andusers
verify
verify
verify
SDSFoil no 5
Synthesize-design-implement.
Requirements
Design
Implementation
Results
Reviews analysis
Testing
Corrective methods (V&V)Defects and errors
Constructive methods
Techniques
SDSFoil no 6
Analysis
• Syntax analysis: by editor
• Static semantic analysis: by analyzer
• Dynamic semantic analysis: by simulation and reachability analysis (modelchecking)
• Verification: by manual inspection and checking formal relationships between models
• Validation: by manual inspection, by testing and formal compatibility checks
• Syntax analysis: by editor
• Static semantic analysis: by analyzer
• Dynamic semantic analysis: by simulation and reachability analysis (modelchecking)
• Verification: by manual inspection and checking formal relationships between models
• Validation: by manual inspection, by testing and formal compatibility checks
SDSFoil no 7
Static signal check 1
• Derive valid input and output signal sets from process graph
• Check signal sets against signal lists on each connection:
P2in = L2 + L3 + L5 P2out = L1 + L4 + L6
• Check signal destinations:
P2 out-to-p1 = L1 P2 out-to-p3 = L4 P2 out-to-p3 = L6
• Derive valid input and output signal sets from process graph
• Check signal sets against signal lists on each connection:
P2in = L2 + L3 + L5 P2out = L1 + L4 + L6
• Check signal destinations:
P2 out-to-p1 = L1 P2 out-to-p3 = L4 P2 out-to-p3 = L6
SDSFoil no 8
Static signal check 2
• Valid input and output signal sets against gate signal lists once for each type
• Signal lists against each other for each connection
• Signal destinations
• Valid input and output signal sets against gate signal lists once for each type
• Signal lists against each other for each connection
• Signal destinations
SDSFoil no 9
Static signal check 3
• Find logical signal routes by signal list partitioning conserving flows
• Signal sets against signal lists
• Signal destinations
• Find logical signal routes by signal list partitioning conserving flows
• Signal sets against signal lists
• Signal destinations
Who talks together here?Who talks together here?
SDSFoil no 10
Liveness = something good will eventually happen (system specific)
safety = nothing bad will ever happen (system independent)
1. Reachability analysis
Full (up to 10**5 states)
Controlled partial (up to 10**8 states)
Random simulation (larger)
2. Structural reasoning
Invariants
Logical inference on equations, rules, axioms
Model checking
3. Execution
Simulation
Testing
Liveness = something good will eventually happen (system specific)
safety = nothing bad will ever happen (system independent)
1. Reachability analysis
Full (up to 10**5 states)
Controlled partial (up to 10**8 states)
Random simulation (larger)
2. Structural reasoning
Invariants
Logical inference on equations, rules, axioms
Model checking
3. Execution
Simulation
Testing
Dynamic analysis
SDSFoil no 11
Reachability analysis
• Iteratively generate all reachable system states
• Analyse each state for
•deadlocks;
• incompleteness;
•other errors.
• Analyse the generated graph
•partial deadlock;
• liveness;
•etc.
• Iteratively generate all reachable system states
• Analyse each state for
•deadlocks;
• incompleteness;
•other errors.
• Analyse the generated graph
•partial deadlock;
• liveness;
•etc.
SDSFoil no 12
Are there any problems here?
[Gb,Sa,Qa]
[Ga, Sb,Qb]B
[Aa]
[Ra,Ea]A
[Rb,Eb]
[Ab]
1
Ra
Sb
3
Gb
Aa
5
Ea
Qb
1
Sa
Ga
8
Qa
1
1
Rb
Sa
3
Ga
Ab
5
Eb
Qa
1
1
Qb
8
Gb
Sb
A B
SDSFoil no 14
Step 2: Generate reachability graph
Find all possible behaviours considering that a signal transfer takes two steps:
•send
•consume
Find all possible behaviours considering that a signal transfer takes two steps:
•send
•consume
SDSFoil no 15
Findings
The errors:
• Unspecified receptions and deadlocks may occur
The reason:
• mixed initiatives not properly handled
The errors:
• Unspecified receptions and deadlocks may occur
The reason:
• mixed initiatives not properly handled
• Graph “explodes” because independent behaviours are interleaved and signals are queued
• Many paths lead to the same error
• Graph “explodes” because independent behaviours are interleaved and signals are queued
• Many paths lead to the same error
Is there a simpler way??Is there a simpler way??
SDSFoil no 16
Working directly in SDL
[Gb,Sa,Qa]
[Ga, Sb,Qb]B
[Aa]
[Ra,Ea]A
[Rb,Eb]
[Ab]
1
Ra
Sb
3
Gb
Aa
5
Ea
Qb
1
Sa
Ga
8
Qa
1
1
Rb
Sa
3
Ga
Ab
5
Eb
Qa
1
1
Qb
8
Gb
Sb
A BAssociation role behaviour =
the visible behaviour of one object when observed from another (associated) object.
Mixed initiative state = a state with inputs from two independent sources (i.e. association roles)
Association role behaviour = the visible behaviour of one object when observed from another (associated) object.
Mixed initiative state = a state with inputs from two independent sources (i.e. association roles)
invisible from B
mixed initiative
state
SDSFoil no 17
Role behaviour and input consistent roles
Deriving a role behaviour:
• Replace invisible inputs by “none” (or just make a mental note)
• Remove invisible outputs (or just ignore them)
The result is a process graph with only visible inputs and outputs. (or just a mental view on the original graph)
Input consistency:
• An invisible transition is a transition with a none input.
• An invisible transition is input consistent iff the next-state explicitly accepts all the visible inputs of the (present) state.
• The role is input consistent iff every invisible transition in the role is input consistent.
Deriving a role behaviour:
• Replace invisible inputs by “none” (or just make a mental note)
• Remove invisible outputs (or just ignore them)
The result is a process graph with only visible inputs and outputs. (or just a mental view on the original graph)
Input consistency:
• An invisible transition is a transition with a none input.
• An invisible transition is input consistent iff the next-state explicitly accepts all the visible inputs of the (present) state.
• The role is input consistent iff every invisible transition in the role is input consistent.
1
none
Sb
3
Gb
5
none
Qb
1
Sa
Ga
8
Qa
1
A
invisible transitions
1 and 3 are notinput consistentbecause 3 does not accept Sa
5 and 1 areinput consistentbecause 1 accepts more than 5
SDSFoil no 18
How to ensure input consistency ?
When checking - for each role:
• Find invisible transitions
• Check that every invisible transition is input consistent
• Consider transitions from mixed initiative states first.
When designing:
• for each state, in particular mixed initiative states, check for each role that all transitions are input consistent.
When checking - for each role:
• Find invisible transitions
• Check that every invisible transition is input consistent
• Consider transitions from mixed initiative states first.
When designing:
• for each state, in particular mixed initiative states, check for each role that all transitions are input consistent.
1
none
Sb
3
Gb
5
none
Qb
1
Sa
Ga
8
Qa
1
A
Mixed initiative state
1 and 3 are notinput consistent
5 and 1 areinput consistent
The intuition behind this is that the associated process will not immediately
observe an invisible transition and therefore the same visible outputs
should be accepted in both states
invisible transition
SDSFoil no 19
S-Rule: Input consistent process
Design SDL processes such that all their role behaviours, i.e. the behaviour visible to processes in their environment, are input consistent.
Design SDL processes such that all their role behaviours, i.e. the behaviour visible to processes in their environment, are input consistent.
In particular check the vicinity of mixed initiative statesIn particular check the vicinity of mixed initiative states
SDSFoil no 20
SDL and MSC
• The SDL behaviour shall “contain” all the MSC behaviours in addition the SDL behaviour may contain:
cases not treated by the MSCs
behaviour on other interfaces
behaviour in other services
• The SDL behaviour shall “contain” all the MSC behaviours in addition the SDL behaviour may contain:
cases not treated by the MSCs
behaviour on other interfaces
behaviour in other services
SDL (object) behaviour tree
MSCs
MSC (instance)behaviour tree
SDL
structure“contains”
SDSFoil no 21
Verifying SDL
• generate all possible traces defined by the MSCs
• check that all traces may execute in the global behaviour tree for the corresponding SDL (using the Validator)
• generate all possible traces defined by the MSCs
• check that all traces may execute in the global behaviour tree for the corresponding SDL (using the Validator)
SDL behaviour tree MSCsMSC behaviour treeSDL
SDSFoil no 22
Synthesizing behaviour 1
process ACSystem
User AC System
Code
OK
msc User_accepted
UnlockCard out
Idle
Door unlocked
User AC System
Code
NOK
msc User_rejected
Card out
Idle
Idle
Idle
IdleDoor unlocked
Code
CardOut
OK
Unlock
Code
CardOut
NOK
How to correct this?How to correct this?
SDSFoil no 23
...by combining and branching
Idle
IdleDoor unlocked
Code
CardOut
OK
Unlock
Code
CardOut
NOK
Idle
IdleDoor unlocked
CardOut
OK
Unlock
CardOut
NOK
Code
ValidateOK NOK
SDSFoil no 24
Validate PIN
wait for PIN
Idle
Process Type UserServer
DCL Cardid, PIN, zone integer;
Synthesizing behaviour 2
MSC EnterZone
User UserServer Authorizer
Card [cardid]
ValidatePIN
[cardid,PIN]
Authenticator
Givepin [PIN]
Enter PIN
exc NOKIllegal PIN
OKValidateRights
[cardid,zone]
exc NOKAccessDenied
OK
AccessGranted
ValidatePIN
(cardid, PIN)
GivePIN (PIN)
Enter PIN
Card (Cardid)
SDSFoil no 25
Synthesizing behaviour 3
MSC EnterZone
User UserServer Authorizer
Card [cardid]
ValidatePIN
[cardid,PIN]
Authenticator
Givepin [PIN]
Enter PIN
exc NOKIllegal PIN
OKValidateRights
[cardid,zone]
exc NOKAccessDenied
OK
AccessGranted
IdleValidateRights
Validate PIN
Process Type UserServer
Idle Idle
AccessGranted
OK
AccessDenied
NOK
ValidateRights
(cardid, zone)
OK
IllegalPIN
NOK
SDSFoil no 26
Synthesizing behaviour 4
General approach:
• Split each MSC into instances, and make corresponding SDL fragments.
• Combine all the SDL fragments for a given process into a more complete behaviour
• Remove non-determinism by combining and adding decisions or states
• Add unspecified behaviour details
• Add exceptions
• Consider all possibilities in each state: ensure that all roles are input consistent!
General approach:
• Split each MSC into instances, and make corresponding SDL fragments.
• Combine all the SDL fragments for a given process into a more complete behaviour
• Remove non-determinism by combining and adding decisions or states
• Add unspecified behaviour details
• Add exceptions
• Consider all possibilities in each state: ensure that all roles are input consistent!
SDSFoil no 27
Specification and design
•••
Specification
•••
Design
Verify MSC
Synthesize
Verify SDL
Decompose
Validate interface
SDSFoil no 28
Validating interfaces
• Static signal checks: are all output signals received?
• Aligning MSC: do both sides align with the same MSCs?
• Comparing role behaviours manually: are required roles contained in provided roles?
• Simulating with SDT: will the interface work for all simulated cases?
• Reachability analysis with SDT: will the validator find errors using state space exploration?
• Static signal checks: are all output signals received?
• Aligning MSC: do both sides align with the same MSCs?
• Comparing role behaviours manually: are required roles contained in provided roles?
• Simulating with SDT: will the interface work for all simulated cases?
• Reachability analysis with SDT: will the validator find errors using state space exploration?
structure
Type Type
SDSFoil no 29
Verify
Aspects:
• verifying MSC decomposition
• verifying SDL behaviour:
•object behaviour
•composition behaviour
Aspects:
• verifying MSC decomposition
• verifying SDL behaviour:
•object behaviour
•composition behaviour
context
structure
behaviour
Verifying SDL behaviourVerifying SDL behaviour
verifying MSC decomposition
Principle:
• every trace defined by the MSCs shall be possible in the SDL
Principle:
• every trace defined by the MSCs shall be possible in the SDL
aligning
SDSFoil no 30
MSC and testing
systemMSC
MSC
MSCsystemunder test
systemunder test
applicationinfrastructuresupport swhw
Abstract test suite
Executable test suite
TesterTester verdict
SDSFoil no 31
Testing what?Application (SDL).
Application support:
• Protocols.
• Input-output modules.
General support:
• Routing.
• Operating system.
• SDL runtime support.
• Error handling.
• Test and debug facilities.
Performance (non-functional):
• Real time response, delays.
• Traffic handling capacity.
• Load control.
Application (SDL).
Application support:
• Protocols.
• Input-output modules.
General support:
• Routing.
• Operating system.
• SDL runtime support.
• Error handling.
• Test and debug facilities.
Performance (non-functional):
• Real time response, delays.
• Traffic handling capacity.
• Load control.