System Testing of Timing Requirements based on Use Cases and Timed Automata
-
Upload
lionel-briand -
Category
Software
-
view
43 -
download
0
Transcript of System Testing of Timing Requirements based on Use Cases and Timed Automata
![Page 1: System Testing of Timing Requirements based on Use Cases and Timed Automata](https://reader034.fdocuments.in/reader034/viewer/2022042707/58d1c0581a28ab705c8b458b/html5/thumbnails/1.jpg)
.lusoftware verification & validationVVS
System Testing of Timing Requirements based on Use Cases and Timed Automata
Chunhui Wang, Fabrizio Pastore, Lionel Briand
ICST’17, Tokyo, March 14th, 2017
University of Luxembourg, Luxembourg
{chunhui.wang, fabrizio.pastore, lionel.briand}@uni.lu
![Page 2: System Testing of Timing Requirements based on Use Cases and Timed Automata](https://reader034.fdocuments.in/reader034/viewer/2022042707/58d1c0581a28ab705c8b458b/html5/thumbnails/2.jpg)
Timing Requirements are crucial forsafety-critical systems
2
![Page 3: System Testing of Timing Requirements based on Use Cases and Timed Automata](https://reader034.fdocuments.in/reader034/viewer/2022042707/58d1c0581a28ab705c8b458b/html5/thumbnails/3.jpg)
Timing Requirements: Automotive Domain
Airbag should be fully deployed within 50ms whena crash event is detected.
3
![Page 4: System Testing of Timing Requirements based on Use Cases and Timed Automata](https://reader034.fdocuments.in/reader034/viewer/2022042707/58d1c0581a28ab705c8b458b/html5/thumbnails/4.jpg)
Timing Requirements: Automotive Domain
Car control system should signal overheating to driver whentemperature exceeds a safety threshold for a given time.
Most of safety criticalcomponents are not reliablewhen car is overheated(e.g. Airbag Control).
4
![Page 5: System Testing of Timing Requirements based on Use Cases and Timed Automata](https://reader034.fdocuments.in/reader034/viewer/2022042707/58d1c0581a28ab705c8b458b/html5/thumbnails/5.jpg)
Timing Requirements: overheating
•A Temperature Error can be qualified (signaled)if overheating is detected for at least 3100ms(avoid signal toggling)
•A Temperature Error should be qualified nolater than 4800ms after overheating isdetected
5
![Page 6: System Testing of Timing Requirements based on Use Cases and Timed Automata](https://reader034.fdocuments.in/reader034/viewer/2022042707/58d1c0581a28ab705c8b458b/html5/thumbnails/6.jpg)
Objectives
• Automatically generate test cases to stress timingrequirements with minimal modelling overhead
• Identification of input sequences that increase the likelihood that the system will break timing constraints
6
![Page 7: System Testing of Timing Requirements based on Use Cases and Timed Automata](https://reader034.fdocuments.in/reader034/viewer/2022042707/58d1c0581a28ab705c8b458b/html5/thumbnails/7.jpg)
Context• Functional Requirements are captured through an
analyzable form of use case specifications
• Timing Requirements are modeled by UML statecharts or Timed Automata
• Functional test cases automatically generated from use case specifications with UMTG
Wang, C., Pastore, F., Goknil, A., Briand, L. C., & Iqbal, Z. (2015, August). UMTG: a toolset to automatically generate system test cases from use case specifications. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (pp. 942-945). ACM.
BASIC FLOW1. The system REQUESTS temperature FROM temperature sensor.2. The system VALIDATE THAT the temperature is valid.3. The system SEND occupant status TO airbag control unit.……
ALTERNATIVE FLOWRFS 21. The system set temperature error as detected.
7
![Page 8: System Testing of Timing Requirements based on Use Cases and Timed Automata](https://reader034.fdocuments.in/reader034/viewer/2022042707/58d1c0581a28ab705c8b458b/html5/thumbnails/8.jpg)
THE ACTOR SENDTHE SYSTEM VALITHE SYSTEM DISTHE ACTOR SEND
THE ACTOR SENDTHE SYSTEM VALITHE SYSTEM DISTHE ACTOR SEND
THE ACTOR SENDTHE SYSTEM VALITHE SYSTEM DISTHE ACTOR SEND
Use Cases
Domain Model
Errors.size()== 0Status != null
t > 0 && t < 50
OCL constraints
Test Cases
Test Scenarios
UMTG [ISSTA’15]
8
![Page 9: System Testing of Timing Requirements based on Use Cases and Timed Automata](https://reader034.fdocuments.in/reader034/viewer/2022042707/58d1c0581a28ab705c8b458b/html5/thumbnails/9.jpg)
Context• Functional Requirements are captured through an
analyzable form of use cases specifications
• Timing Requirements are modeled by UML statecharts or Timed Automata
• Functional test cases automatically generated from use case specifications with UMTG
Wang, C., Pastore, F., Goknil, A., Briand, L. C., & Iqbal, Z. (2015, August). UMTG: a toolset to automatically generate system test cases from use case specifications. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (pp. 942-945). ACM. 9
![Page 10: System Testing of Timing Requirements based on Use Cases and Timed Automata](https://reader034.fdocuments.in/reader034/viewer/2022042707/58d1c0581a28ab705c8b458b/html5/thumbnails/10.jpg)
Strategy• Use Timed Automata to model Timing Constraints
• Automata are high-level, missing information about concrete test inputs that may lead to state transitions
• Rely upon Use Case Scenarios to identify the test inputs that trigger state transitions
• Rely upon edge coverage criteria to identify test sequences
• Integrate an ad-hoc metaheuristic search approach to build a testsuite that effectively stresses timing requirements by maximizingtest case diversity
10
![Page 11: System Testing of Timing Requirements based on Use Cases and Timed Automata](https://reader034.fdocuments.in/reader034/viewer/2022042707/58d1c0581a28ab705c8b458b/html5/thumbnails/11.jpg)
Test Generation combining Timed Automata and Use Case
Specifications (TAUC)
11
![Page 12: System Testing of Timing Requirements based on Use Cases and Timed Automata](https://reader034.fdocuments.in/reader034/viewer/2022042707/58d1c0581a28ab705c8b458b/html5/thumbnails/12.jpg)
Msg1?
Environment Automata
scenario1?[Error == true]
[Error == false]scenario2?
TimingReq Automata
ANALYSIS AND DESIGN 1
12
![Page 13: System Testing of Timing Requirements based on Use Cases and Timed Automata](https://reader034.fdocuments.in/reader034/viewer/2022042707/58d1c0581a28ab705c8b458b/html5/thumbnails/13.jpg)
Modelling• Automata that capture timing properties of the functionality
under test (timing requirement automata)
• E.g. qualification of temperature error (overheating)
• Automata that model the environment:
• Capture frequency of arrival of interrupts / messages / inputs from sensors
13
![Page 14: System Testing of Timing Requirements based on Use Cases and Timed Automata](https://reader034.fdocuments.in/reader034/viewer/2022042707/58d1c0581a28ab705c8b458b/html5/thumbnails/14.jpg)
Example: TimingReqAutomata• Automata that capture timing properties of the functionality
under test, e.g. qualification of of temperature error
14
x >= 3100
unDetected?isDetected = false, x = 0
detected?
isDetected = true, x = 0
isDetected = false, x = 0
isDetected = true, x = 0
x >= 6100
NotDetectedNotQualified DetectedNotQualified
DetectedQualifiedNotDetectedQualified
x <= 4800
x <= 8100
Invariant
Guard
Updates
unDetected?
detected?
isQualified = falseisQualified = trueA Temperature Error can be qualified if overheating is detected for at
least 3100ms.A Temperature Error should be qualified no later than 4800ms afteroverheating is detected
![Page 15: System Testing of Timing Requirements based on Use Cases and Timed Automata](https://reader034.fdocuments.in/reader034/viewer/2022042707/58d1c0581a28ab705c8b458b/html5/thumbnails/15.jpg)
• Automata that model the environment:
• Capture frequency of arrival of interrupts / messages / inputs from sensors
15x = 0
x > 41DMAInterrupt!
x = 0
x >= 50TimerInterrupt!
x <= 50
x = 0
x > 40OccupancyInfo!
x > 40CarInfo!
x > 40AirbagInfo!
Example: EnvironmentAutomata
![Page 16: System Testing of Timing Requirements based on Use Cases and Timed Automata](https://reader034.fdocuments.in/reader034/viewer/2022042707/58d1c0581a28ab705c8b458b/html5/thumbnails/16.jpg)
Msg1?
Environment Automata
scenario1?[Error == true]
[Error == false]scenario2?
TimingReq Automata
scenario1 scenario2 scenario3
ANALYSIS AND DESIGN 1
IDENTIFY FUNCTIONAL SCENARIOS 2
IDENTIFYDEPENDENCIES
3
Automated by UMTG
• Transitions are triggered by scenario outputs
• Scenarios can be executed only in specific states
• Useful to translate abstract test activities into concrete test inputs
16
![Page 17: System Testing of Timing Requirements based on Use Cases and Timed Automata](https://reader034.fdocuments.in/reader034/viewer/2022042707/58d1c0581a28ab705c8b458b/html5/thumbnails/17.jpg)
17
Coverage-based Test Generation
x >= 3100
unDetected?isDetected = false, x = 0
detected?
isDetected = true, x = 0
isDetected = false, x = 0
isDetected = true, x = 0
x >= 6100
NotDetectedNotQualified DetectedNotQualified
DetectedQualifiedNotDetectedQualified
x <= 4800
x <= 8100
unDetected?
detected?
isQualified = falseisQualified = true
![Page 18: System Testing of Timing Requirements based on Use Cases and Timed Automata](https://reader034.fdocuments.in/reader034/viewer/2022042707/58d1c0581a28ab705c8b458b/html5/thumbnails/18.jpg)
18
Coverage-based Test Generation
x >= 3100
unDetected?isDetected = false, x = 0
detected?
isDetected = true, x = 0
isDetected = false, x = 0
isDetected = true, x = 0
x >= 6100
NotDetectedNotQualified DetectedNotQualified
DetectedQualifiedNotDetectedQualified
x <= 4800
x <= 8100
unDetected?
detected?
isQualified = falseisQualified = true
Test CaseDetected
Wait 4800ms
unDetected
Wait 8100ms
![Page 19: System Testing of Timing Requirements based on Use Cases and Timed Automata](https://reader034.fdocuments.in/reader034/viewer/2022042707/58d1c0581a28ab705c8b458b/html5/thumbnails/19.jpg)
How to translate“error detected”
into a concrete test input?
By Reusing InformationAvailable in
Use Case Specifications
19
![Page 20: System Testing of Timing Requirements based on Use Cases and Timed Automata](https://reader034.fdocuments.in/reader034/viewer/2022042707/58d1c0581a28ab705c8b458b/html5/thumbnails/20.jpg)
How to translate“error detected”
to a concrete test input?
20
TemperatureSensor.allInstances()->forAll(t | t.temperature > 0 and t.temperature < 45)
UMTGgenerates the input
(e.g. temperature = 50)that leads to the detection of a
temperature error
![Page 21: System Testing of Timing Requirements based on Use Cases and Timed Automata](https://reader034.fdocuments.in/reader034/viewer/2022042707/58d1c0581a28ab705c8b458b/html5/thumbnails/21.jpg)
21
A scenario brings the system into a specific state(i.e. assigns values to variables)
![Page 22: System Testing of Timing Requirements based on Use Cases and Timed Automata](https://reader034.fdocuments.in/reader034/viewer/2022042707/58d1c0581a28ab705c8b458b/html5/thumbnails/22.jpg)
22
Errors.allInstances()->forAll(e | e.isQualified = false)
Precondition to execute the scenario: At least one error is in the states with the state variable isQualified = true
![Page 23: System Testing of Timing Requirements based on Use Cases and Timed Automata](https://reader034.fdocuments.in/reader034/viewer/2022042707/58d1c0581a28ab705c8b458b/html5/thumbnails/23.jpg)
23
x >= 3100
unDetected?
detected?
x >= 6100
NotDetectedNotQualified DetectedNotQualifiedx <= 4800
x <= 8100
unDetected?
detected?
isQualified = falseisQualified = true
NotDetectedQualified DetectedQualified
Scenario
isQualified = true
![Page 24: System Testing of Timing Requirements based on Use Cases and Timed Automata](https://reader034.fdocuments.in/reader034/viewer/2022042707/58d1c0581a28ab705c8b458b/html5/thumbnails/24.jpg)
24
• A scenario brings the system into a specific state (i.e. assign values to variables)
• A scenario can be executed only if the system has reached a specific state (i.e. requires that state variables have specific values)
![Page 25: System Testing of Timing Requirements based on Use Cases and Timed Automata](https://reader034.fdocuments.in/reader034/viewer/2022042707/58d1c0581a28ab705c8b458b/html5/thumbnails/25.jpg)
Msg1?
Environment Automata
scenario1?[Error == true]
[Error == false]scenario2?
TimingReq Automata
scenario1 scenario2 scenario3
ANALYSIS AND DESIGN 1
IDENTIFY FUNCTIONAL SCENARIOS 2
IDENTIFYDEPENDENCIES
3
25
![Page 26: System Testing of Timing Requirements based on Use Cases and Timed Automata](https://reader034.fdocuments.in/reader034/viewer/2022042707/58d1c0581a28ab705c8b458b/html5/thumbnails/26.jpg)
scenario1?[Error == true]
[Error == false]scenario2?
Msg1?
Timeliness Test Model
Environment Automata
C
scenario1!
Error := true
Scenario Automata
TimingReq Automata
4800ms
Scenario1
ScenarioX
8100ms
Message1…
GENERATE TIMELINESS TEST MODEL
4 GENERATE EXECUTION SCENARIOS
5
TAUC
26
![Page 27: System Testing of Timing Requirements based on Use Cases and Timed Automata](https://reader034.fdocuments.in/reader034/viewer/2022042707/58d1c0581a28ab705c8b458b/html5/thumbnails/27.jpg)
4800ms
Scenario1
ScenarioX
8100ms
Message1…
4800ms
Scenario1
ScenarioY
8100ms
Message2…
ScenarioX
ScenarioZ
8100ms
Message1…
Message3
Generated Test Suites
…4800ms
Scenario1
ScenarioX
8100ms
Message1…
Maximize test case diversity:• Execute more, diverse paths, including the same relevant edges• Execute paths with a diversity of input, interrupts, and messages• Increase the chances of timeliness failure detection
TAUC
27
![Page 28: System Testing of Timing Requirements based on Use Cases and Timed Automata](https://reader034.fdocuments.in/reader034/viewer/2022042707/58d1c0581a28ab705c8b458b/html5/thumbnails/28.jpg)
Case Study• BodySense, embedded system for detecting occupancy
status in a car
• Evaluation:
• Effectiveness in terms of fault detection ratecompared to current practice at IEE
28
![Page 29: System Testing of Timing Requirements based on Use Cases and Timed Automata](https://reader034.fdocuments.in/reader034/viewer/2022042707/58d1c0581a28ab705c8b458b/html5/thumbnails/29.jpg)
Effectiveness Evaluation• Derived 323 faulty version of BodySense by means of model
mutation• Each faulty version of BodySense models was generated by
executing a single mutation operatoron the original model
• Compare the Fault Detection Rate with the test suitesgenerated by• Random testing• Expertise-based manual testing
29
![Page 30: System Testing of Timing Requirements based on Use Cases and Timed Automata](https://reader034.fdocuments.in/reader034/viewer/2022042707/58d1c0581a28ab705c8b458b/html5/thumbnails/30.jpg)
30
Fault Detection Rate
25 50 75 100 122TAUC 85% 88% 91% 91% 91%
Random 7% 12% 22% 30% 40%
Manual 60%
![Page 31: System Testing of Timing Requirements based on Use Cases and Timed Automata](https://reader034.fdocuments.in/reader034/viewer/2022042707/58d1c0581a28ab705c8b458b/html5/thumbnails/31.jpg)
31
Evaluation of TAUC Test Suite
scenario1?[Error == true]
[Error == false]scenario2?
Original model
Mutate
scenario1?[Error == true]
[Error == false]scenario2?
scenario1?[Error == true]
[Error == false]scenario2?
…
scenario1?[Error == true]
[Error == false]scenario2?
TAUC
scenario1?[Error == true]
[Error == false]scenario2?
Msg1?
Environment Automata
C
scenario1!
Error := true
Scenario Automata
TimingReq Automata
4800ms
Scenario1
ScenarioX
8100ms
Message1…
4800ms
Scenario1
ScenarioY
8100ms
Message2…
ScenarioX
ScenarioZ
8100ms
Message1…
Message3
Test Suites
…
GENERATE TIMELINESS TEST MODEL
4
GENERATE EXECUTION SCENARIOS
5
IDENTIFYDEPENDENCIES
3
scenario1 scenario2 scenario3
IDENTIFY FUNCTIONAL SCENARIOS 2
Msg1?
Environment Automata
scenario1?[Error == true]
[Error == false]scenario2?
TimingReq Automata
ANALYSIS AND DESIGN 1
Timing Requirements are crucial forsafety-critical systems
Fault Detection Rate
25 50 75 100 122
TAUC 85% 88% 91% 91% 91%
Random 7% 12% 22% 30% 40%
Manual 60%