Software engineering and standardisation section (TEC-EME) 1 Automatic test case generation from a...
-
Upload
makena-simon -
Category
Documents
-
view
213 -
download
0
Transcript of Software engineering and standardisation section (TEC-EME) 1 Automatic test case generation from a...
Software engineering and standardisation section (TEC-EME)
1Automatic test case generationfrom a formal model
LTG technology evaluation based ona B model of
a part of ATV onboard software
Julien ORSAT – Stagiaire from the University of Franche-Comté (France)
08/03 – 23/07/2004
Supervisor: Philippe CHEVALLEY
2Outline
B modelling for Leirios Test Generator
ATV software case study: Gyra functional unit
Gyra B formal model
LTG automatic test case generation
Conclusion
3Outline
B modelling for Leirios Test Generator B formal notation LTG requirement traceability LTG syntax restrictions
ATV software case study: Gyra functional Unit
Gyra B formal model
LTG automatic test case generation
Conclusion
4B formal notation
Used as input language for LTG Other supported input format : Statecharts/Statemate, UML
Part of the B method Code generation from the model Proof obligations to ensure correctness Several refinement steps from the model to the target code Created in the 90’s by J-R Abrial Applied in railway transports and smart card industry
LTG input: 1st step of the B method (“abstract machine”)
Text format Statecharts and UML are graphically used
5Some B concepts
Sets ; could be considered as base types GYRO_ID = {G1, G2, G3, G4} GYRO_MODE = {COARSE, FINE}
Relations ; to define sets of ordered pairs gyra_mode : GYRO_ID --> GYRO_MODE
Substitutions ; to modify variable values gyra_mode := gyra_mode <+ {(G3 |-> FINE)}
Operations ; to describe system behaviour Used by LTG to compute test cases
6B operation example
A B operation is considered atomic <+ is the relation overriding operator
cmd_report <-- setGyroMode(gyro, mode) =
PRE /* precondition */
gyro : GYRO_ID &
mode : GYRO_MODE & …
THEN /* body */
IF condition THEN
cmd_report := REJECTED
ELSE
cmd_report := ACCEPTED ||
gyra_mode := gyra_mode <+ { (gyro |-> mode) }
END
END
ProfileB reserved wordCommentaryBoolean expressionSubstitution
7LTG requirement traceability
Provided by LTG not part of B notation
Uses tags in the model Based on B commentaries IF (...) THEN
cmd_report := REJECTED /*@REQ: R52 @*/ELSE ...
Allows automatic traceability report generation Test case -> covered requirements Requirement -> covering test cases
Requirement coverage criteriafor generated test case evaluation
8LTG restrictions to the B notation
Syntax Real numbers
Difficulty to compute expected results
Sequences list-like data structures
“Such as” set description { element | predicates }
Single B machine only No machine inclusion, promotion …
9Outline
B modelling for Leirios Test Generator
ATV software case study: Gyra functional unit
Gyra Functional Unit ATV software structure Case study choice explanation
Gyra B formal model
LTG automatic test case generation
Conclusion
10Gyra Functional Unit
High level services to usea 4 GYRoscope Assembly (GYRA)
Hardware control Monitoring algorithms Processing algorithms
Designed and developed by EADS 89 requirements 13 Ada83 packages 7 000 code lines
Managed by an internalfinite state machine (FSM)
11ATV software structure
Gyra Functional Unit is part of software layer 3
LN 2 library
LN 1 library
LN 3:…
LN 3:Gyra FU
Device#n
Hardware
Software layers
28 FUs
… … …
LN 3:…
Device#1
Hardware interface
Communications
Applications
12Case study choice explanation
ATV spacecraft Uses current space software techniques
(software under development)
Gyra application Easy access to Gyra specification documents
Focus on the FSM Heavily used in ATV: 28 FSMs, 989 transitions Gyra FSM has an average size (38 transitions) FSMs are difficult/expensive to test
“Transition testing is one of the most, if not the most, demanding tasks of the product validation” EADS test plan
Would be valuable to automate test case generation
13Outline
B modelling for Leirios Test Generator
ATV software case study: Gyra functional Unit
Gyra B formal model Modelled Finite State Machine Requirement traceability
Automatic test case generation
Conclusion
14Modelled Finite State Machine
OFF_N
ON_N
ON_4ON_1
ON_2
ON_13
ON_14
ON_3
ON_24
ON_12
ON_23
ON_34
Transitions are activated by commands Redundant commands are accepted (requirement 16241) FSM modes are groups of FSM states (OFF mode, ON mode)
15Modelled FSM key figures
12 states
22 graphically represented transitions
+ 14 different reflexive transitions would be 102 more transitions
on the picture distinct from
command redundancy transitions
750 line model
NB: separated model for Survival modes(3 states, 2 transitions – 150 lines)
ON_x (11 states)
Example: GYRA_TR_MODE
According to EADS:1 transition
Modelling choice:11 different
transitions
ON_x (11 states)ON_x (11 states)
16Requirement traceability
Traceable Gyra requirements: 43 / 49
Non traceable requirements (6) : Initial values of variables (3),
No LTG tag for INITIALISATION part of the model High level requirements, refined by others (2) FSM mode shall be computed from FSM state (1)
That is to say not stored in memory – code inspection is required
17Outline
B modelling for Leirios Test Generator
ATV software case study: Gyra functional unit
Gyra B formal model
LTG automatic test case generation Test generation criteria Generated test cases Ada test program generation
Conclusion
18Test generation criteria
Model coverage criteria: 4 possibilities Decision coverage (DC) Condition/Decision coverage (C/DC) Modified condition/Decision coverage (MC/DC) Multiple condition coverage (MCC)
An LTG test case is made of 1. Preambule 2. Body + Identification 3. Postambule
With Gyra, test programs may end in any FSM state no need for a postambule
Softwarecriticality
19Generated test cases
179 functional abstract test cases generated = 179 operation activation sequences XML format (HTML and RTF translators)
Comparison to EADS test plan Chaining of several tests (8 large test sequences)
difficult to compare
Traceable requirements covered by LTG: 37/43 Difficulty with generic requirements (5)
Multiple tags for one requirement Were those requirements fully covered ?
1 requirement was not covered: some behaviours, related to 1 B operation, were not reached different generation criteria should solve the problem.
HTML example
20EADS test plan coverage
Gyra FSM state and transition coverage
OFF_N
ON_N
ON_4ON_1
ON_2
ON_13
ON_14
ON_3
ON_24
ON_12
ON_23
ON_34
21Coverage comparison
State coverage EADS test plan: 9/12 LTG: 12/12
Graphically represented transition coverage
EADS test plan: 14/22 LTG : 22/22
Reflexive transition coverage EADS: 13/14, tested on one state LTG: 13/14, tested on every state
uncovered transition is relatedto uncovered B operation
ON_x (11 states)
Example: GYRA_TR_MODE
In EADS test plan:1 test case
Our choice:11 test cases
22
Required to generate test programs from test cases:
Tagged test program pattern Relation mapping between B operations and Ada services
Test program generation on a simple model Test program example
Not enough time to generate test programs
Ada test program generation
23Outline
B modelling for Leirios Test Generator
ATV software case study: Gyra functional unit
Gyra B formal model
LTG automatic test case generation
Conclusion
24
179 Test Cases
LTG Test CaseGenerator
Test CaseGeneration
Test GenerationCriteria
LTGeXecutable
Script Builder
Test Execution Environment Test Driver Generation
Test DriverSchemas
49 Gyra functional requirements
B Model (750 lines)FunctionalModeling
LTGModel Animator
ModelValidation
LEIRIOS Test Generator™
done
to do
24
25Conclusion
Technology evaluation: B notation Particularly suited to model finite state machines Difficulty to model real time aspects like scheduling Added requirement traceability Some restrictions on the notation
Automatically generated test cases on the case-study 15 hour computation on a desktop PC, for 179 test cases Coverage is similar or better compared to EADS test plan
LTG product stability, documentation quality to be raised
Transition from research prototypeto commercial product is in progress
26Perspectives
Short-term perspectives Generate and execute Gyra FSM test programs
2 possibilities: by Estec or by EADS Structural coverage study Fault injection in the Gyra implementation to evaluate test
efficiency Generate tests for Gyra algorithms
Long-term perspectives LTG input format comparison for space projects
B notation Statecharts UML
Software engineering and standardisation section (TEC-EME)
27Automatic test case generationfrom a formal model
LTG technology evaluation based ona B model of
a part of ATV onboard software
Julien ORSAT – Stagiaire from the University of Franche-Comté (France)
08/03 – 23/07/2004
Supervisor: Philippe CHEVALLEY