Matlab-Simulink Using Simulink and Stateflow Automotive Applications
Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers
-
Upload
lionel-briand -
Category
Software
-
view
354 -
download
1
Transcript of Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers
.lusoftware verification & validationVVS
Effective Test Suites for !Mixed Discrete-Continuous
Stateflow Controllers Reza Matinnejad Shiva Nejati Lionel Briand SnT Center, University of Luxembourg
Thomas Bruckmann Delphi Automotive Systems, Luxembourg
Cyber Physical Systems (CPSs) Combination of computations (algorithms) and physical dynamics (differential equations)
2
Physical world Computation
Testing (Typical) Software
3
X = 10, Y = 30
Z = 20
Algorithms
Fail Pass Z = 10
Testing (CPS) Software
4
Algorithms + Differential Equations
Fail Z = 20
X = 10, Y = 30 S1(t) S2(t)
S3(t) Pass
Z = 20 S3(t)
S1t
S2t
S3t
S3t
Software Testing Challenges (CPS)
• Mixed discrete-continuous behavior (combination of algorithms and continuous dynamics)
• Inputs/outputs are signals (functions over time)
• Simulation is inexpensive but not yet systematically automated
• Partial test oracles
5
Generating effective test suites for Software used in !
Cyber-Physical Systems
6
Our Goal
Simulink/Stateflow
• A data flow-driven block diagram language
• Is widely used to develop Cyber Physical Systems
• Is executable
7
Stateflow
• A Statechart dialect integrated into Simulink
• Captures the state-based behavior of CPS software
• Has mixed discrete-continuous behavior
8
Generating effective test suites for mixed discrete-continuous
Stateflow controllers
9
Our Goal
Discrete Behavior What we typically think of software models
10
On
Off
On
Off
Speed < 10 Speed > 10
Discrete-Continuous Behavior What software models are actually being built using Stateflow
11
On
Off
CtrlSig
On
Off
Speed < 10 Speed > 10
tCtrlSig
tCtrlSig
Generating effective test suites for mixed discrete-continuous
Stateflow controllers
12
Our Goal
Test Suite Effectiveness (1) • Test suite size should be small because
• Test oracles cannot be fully automated
• Output signals need to be inspected by engineers
13
ModelSimulation
InputSignals
OutputSignal(s)
S3t
S2t
S1t
S3t
S2t
S1t
Test Case 1
Test Case 2
Test Suite Effectiveness (2) • Test suites should have a high fault revealing power
• Small deviations in outputs may not be recognized/important
• Test inputs that drastically impact the output signal shape are likely to have a higher fault revealing power
14
Test Output 1
TimeTime
CtrlSig
Faulty Model OutputCorrect Model Output
Test Output 2
Test Generation Algorithms!!
15
Our Approach
Test Generation Algorithms • Input-based Test Generation:
• Input Diversity Algorithm
• Coverage-based Test Generation:
• State Coverage Algorithm
• Transition Coverage Algorithm
• Output-based Test Generation:
• Output Diversity Algorithm
• Failure-based Algorithm
16
Input Diversity • Maximizing distances among input signals
17
Test Case 1
Test Case 2
Input Signal 1 Input Signal 2
S1t
S1t
S2t
S2t
Distance Between Signals
18
Time
Signal
Test Generation Algorithms • Input-based Test Generation:
• Input Diversity Algorithm
• Coverage-based Test Generation:
• State Coverage Algorithm
• Transition Coverage Algorithm
• Output-based Test Generation:
• Output Diversity Algorithm
• Failure-based Algorithm
19
Structural Coverage
• Maximizing the number of states/transitions covered
20
State Coverage Transition Coverage
1
4
2
3
1
4
2
3
Test Generation Algorithms • Input-based Test Generation:
• Input Diversity Algorithm
• Coverage-based Test Generation:
• State Coverage Algorithm
• Transition Coverage Algorithm
• Output-based Test Generation:
• Output Diversity Algorithm
• Failure-based Algorithm
21
Output Diversity • Maximizing distances among output signals
22
Test Case 1
Test Case 2
Output Signal
S3t
S3t
Failure-based Test Generation
23
Instability Discontinuity
0.0 1.0 2.0-1.0
-0.5
0.0
0.5
1.0
Time
Ctr
lSig
Output
• Maximizing the likelihood of presence of specific failure patterns in output signals
0.0 1.0 2.0Time
0.0
0.25
0.50
0.75
1.0
Ctr
lSig
Output
We developed our failure-based test generation algorithm using!
Meta-Heuristic Search
24
The Alternative Choice
25
Our ApproachExisting WorkTechnique
ModelChecking
- Require precisely definedoracles (user-specified assertions)
- Have been largely appliedto time-discrete models
- State-explosion problem!
- No need for automated test oracles
- Applicable to time-continuousand non-linear models
- Our algorithms are black-boxrandomized search: - non-memory intensive - can be parallelized
26
Failure-based Test Generation using Meta-Heuristic Search
Input Signals
Slightly Modifying Each Input Signal
Fitness Functions Capturing the Likelihood
of Presence of Failure Patterns in the Output Signals
Repeat
Until maximum resources spent
S Initial Candidate Solution
Search Procedure
R Tweak (S)
if Fitness (R) > Fitness (S)
S R
Return S
Output Stability !Fitness Function
• Sum of the differences of signal values for consecutive simulation steps
27
stability(sgo
) =kP
i=1|sg
o
(i ·�t)� sgo
((i� 1) ·�t)|
0.0 1.0 2.0-1.0
-0.5
0.0
0.5
1.0
Time
Ctr
lSig
Output
Output Continuity !Fitness Function
28
• Maximum of the minimum left or right derivatives for all the simulation steps
0.0 1.0 2.0Time
0.0
0.25
0.50
0.75
1.0
Ctr
lSig
Output
continuity(sg
o
) =
K�1max
i=1(min(|LeftDer(sg
o
, i)|, |RightDer(sgo
, i)|))
Comparing the!Test Generation Algorithms!
!
29
Evaluation
Research Questions
• RQ1 (Fault Revealing Ability)
• RQ2 (Fault Revealing Subsumption)
• RQ3 (Test Suite Size)
30
Experiment Setup • Three Stateflow models: two industrial and one publicly
available case study
31
75 (faulty models) * 100 (algorithm runs) *6 (generation algorithms) * 5 (different test suite sizes) =
225,000 test suites (in total)
Test Suite(size=3,5, 10,25,50)
{1.Fault
Seeding2.Generation
AlgorithmSF FaultySF
{75 75
Research Question 1!Fault Revealing Ability
How does the fault revealing ability of our proposed test generation algorithms
compare with one another?
32
1.0
0.0
0.5
Input Diversity
OutputDiversity
FaultRevealing
Rate
RQ1: Fault Revealing Ability
33
1. Output-based and coverage-based algorithms outperformed the input diversity algorithm
2. Output-based algorithms outperformed the coverage-based algorithms
3. Overall, output stability algorithm performed the best
Research Question 2!Fault Revealing Subsumption
Is any of our generation algorithms subsumed by other algorithms?
34
RQ2: Fault Revealing Subsumption
35
• For each of the 75 faulty models, we identified the best generation algorithm(s) for different test suite sizes (5, 10, 25, and 50)
Fault 1State Coverage
Transition Coverage
Output Diversity
Output Stability
Output Continuity
Fault 2 Fault 3 Fault 4
RQ2: Fault Revealing Subsumption (2)
36
1. The coverage-based algorithms found the least number of faults
2. Coverage-based algorithms are subsumed by output diversity algorithm when the test suite size increases (size = 25 , 50)
Research Question 3!Test Suite Size
What is the impact of the size of test suites generated by our generation algorithms on
their fault revealing ability?
37
RQ3: Test Suite Size
38
1. The fault revealing rates for output stability/continuity is very high for small test suites(size = 3,5) for Instability/Discontinuity failures
2. For Other failures, the ability of output diversity in revealing failures rapidly increases as the test suite size increases
DiscontinuityInstability Others
0.0
0.5
1.0
3 5 10 25 50
Test Suite Size
Faul
t Rev
ealin
g Ra
te M
ean
3 5 10 25 50 3 5 10 25 50
Output StabilityOuput Continuity State Coverage
Transition CoverageOutput Diversity
Lessons Learned
39
Lesson 1!Coverage-based algorithms are less
effective than output-based algorithms • The test cases resulting from state/transition coverage
algorithms cover the faulty parts of the models
• 97% state coverage and 81% transition coverage
• Cover faulty parts for 73 (out of 75) fault-seeded models
• However, they fail to generate output signals that are sufficiently distinct from the oracle signal, hence yielding a low fault revealing rate
40
Lesson 2!Combining Output-based Algorithms
41
• We suggest to divide the test suite size budget between output-based algorithms:
Output Continuity Output Stability Output Diversity
CoCoTest
42
.lusoftware verification & validationVVS
Effective Test Suites for !Mixed Discrete-Continuous
Stateflow Controllers Reza Matinnejad ([email protected]) Shiva Nejati Lionel Briand SnT Center, University of Luxembourg
Thomas Bruckmann Delphi Automotive Systems, Luxembourg
Lesson 1!Combing Output-based Algorithms
• We suggest to divide the test suite size budget between output stability, output continuity, and output diversity:
1. Allocate a small part of the test budget to output continuity
2. Share the rest of the budget between output stability and output diversity, by giving output diversity a higher share
44
Input / Output Vectors
45
0 5 10
50
150
250
Fuel
Lev
el S
enso
r
Fuel
Lev
el0 5 10
100.0
91.43
84.43
75.6270.01
66.1961.21
56.6654.3252.81
50
100
Time (s) Time (s)
Study subjects
46
Publicly AvailableName No. of
InputsHierarchical
States ParallelismNo. of States
SCPCASS
NoNo
2342
1316
2 No1 No
GCS Yes 8 10 0 Yes
No. of Transitions
2553
27
• SCPC: Supercharger Clutch Position Controller
• ASS: Auto Start Stop Control
• GCS: Guidance Control System
Fault Revealing Rate (FRR)
47
FRR(SF ,TS ) =
(1 91iq
ˆdist(sgi, gi) > THR
0 81iqˆdist(sgi, gi) <= THR
• FRR based on gi, output of the fault-free model, sgi, output of the fault-seeded model, and a threshold THR:
1. For continuous dynamic systems, the system output is acceptable when the deviation is small and not necessarily zero
2. It is more likely that manual testers recognize a faulty output signal when the signal shape drastically differs from the oracle.
RQ3: Test Suite Size
48
1. The fault revealing rates for output stability/continuity is very high for small test suites for Instability/Discontinuity
2. For “Other” failures, the ability of OD in revealing failures rapidly increases as the test suite size increases
Discontinuity
SCTC
ODOSOC* *+ +
--
Instability Others
0.0
0.5
1.0
3 5 10 25 50
**
* *
+
++
--
Test Suite Size
FRR
Mea
n
+-
-
3 5 10 25 50 3 5 10 25 50
* * *
-
--
+
+
+ - - -*
* *
+
++
+
+*
* *---