Example of a Complementary use of Model Checking and Agent-based Simulation Gabriel Gelman & Karen...
-
Upload
deirdre-kelley -
Category
Documents
-
view
220 -
download
1
Transcript of Example of a Complementary use of Model Checking and Agent-based Simulation Gabriel Gelman & Karen...
![Page 1: Example of a Complementary use of Model Checking and Agent-based Simulation Gabriel Gelman & Karen Feigh Georgia Institute of Technology & John Rushby.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d215503460f949f5d5c/html5/thumbnails/1.jpg)
Example of a Complementary use of ModelChecking and Agent-based Simulation
Gabriel Gelman & Karen Feigh Georgia Institute of Technology
& John Rushby
Stanford Research Institute
![Page 2: Example of a Complementary use of Model Checking and Agent-based Simulation Gabriel Gelman & Karen Feigh Georgia Institute of Technology & John Rushby.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d215503460f949f5d5c/html5/thumbnails/2.jpg)
2
Introduction
Increasing Complexity
Challenges in HMI
Leads to
Automation Surprises
Such as
Pilots AutomationPotential Issues
Model Checking
Simulation
Combine to leverage benefits
of both
System Behavior
To examine
Tackled by
HMI = Human-Machine Interaction
Agents
…
![Page 3: Example of a Complementary use of Model Checking and Agent-based Simulation Gabriel Gelman & Karen Feigh Georgia Institute of Technology & John Rushby.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d215503460f949f5d5c/html5/thumbnails/3.jpg)
3
Comparison: Model Checking/ Simulation
Simulation Model CheckingSophisticated models Simple models, few actions
Limited to scenarios Exhaustive state space search
Slow (one simulation takes time) Fast (millions of runs in seconds)
Time can be explicitly modeled No explicit modeling of time
High-Fidelity aircraft dynamics Cannot handle continuity (state explosion)
![Page 4: Example of a Complementary use of Model Checking and Agent-based Simulation Gabriel Gelman & Karen Feigh Georgia Institute of Technology & John Rushby.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d215503460f949f5d5c/html5/thumbnails/4.jpg)
4
Method: Connecting the Frameworks
Scenario NarrativeCreate Model & Specifications for Model Checking (SAL)
Analyze Using Model Checking (SAL)
Create Models & Metric Specifications for Simulation (WMC)
Analyze Using Simulation (WMC)
Extending the Counterexample Guided Abstraction Refinement (CEGAR) method
1. Verify that the action sequence predicted by MC to be problematic continues to be problematic
2. Refine MC prediction to include specific temporal relationships between events
![Page 5: Example of a Complementary use of Model Checking and Agent-based Simulation Gabriel Gelman & Karen Feigh Georgia Institute of Technology & John Rushby.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d215503460f949f5d5c/html5/thumbnails/5.jpg)
Automation Surprise Aviation Case Study
![Page 6: Example of a Complementary use of Model Checking and Agent-based Simulation Gabriel Gelman & Karen Feigh Georgia Institute of Technology & John Rushby.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d215503460f949f5d5c/html5/thumbnails/6.jpg)
6
Automation Surprise
“An Automation Surprise occurs when the automation behaves in a manner that is different from what the operator is expecting”, Palmer (1995) + Result of implementation of badly designed automation or lack of pilots’
training on system+ Introduction of highly automated aircraft (glass cockpits)
Starting with aircraft like B-757, B-737 and A320
Failure to activate Approach
Automatic Mode Changes
Sarter and Woods A320 study (80% surprised; n = 167)
![Page 7: Example of a Complementary use of Model Checking and Agent-based Simulation Gabriel Gelman & Karen Feigh Georgia Institute of Technology & John Rushby.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d215503460f949f5d5c/html5/thumbnails/7.jpg)
7
Case Study: Airbus Automatic Speed Protection
Flight Path Angle mode
engaged
Airspeed too fast
Overspeed Protection
Open mode engaged
Sequence on approach
FCU: Flight Control UnitV/S: Vertical Speed
FPA: Flight Path Angle
FCU altitude with respect to current
altitude
OPEN DESCENT
OPEN CLIMB
Higher
Lower
Note: During descent FCU altitude is usually set to Missed Approach altitude if Go Around required
![Page 8: Example of a Complementary use of Model Checking and Agent-based Simulation Gabriel Gelman & Karen Feigh Georgia Institute of Technology & John Rushby.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d215503460f949f5d5c/html5/thumbnails/8.jpg)
8
Sequence Automation Surprise
Instrument Landing System (ILS) Glideslope
Runway
1
2
Step 1: Aircraft is on ILS Glideslope and in FPA V/S modeStep 2: Air Traffic Control tells aircraft to level offStep 3: Aircraft tries to recapture ILS Glideslope with higher FPAStep 4: Because of steeper approach the speed exceeds Vmax
Step 5: Mode change to OP CLB because FCU alt higher than current alt
FPA = 3°
3
10° > FPA > 3°4
FCU Altitude = Go Around Altitudee.g. 5000ft
5
Altit
ude
Ground
FCU: Flight Control UnitFPA: Flight Path Angle
![Page 9: Example of a Complementary use of Model Checking and Agent-based Simulation Gabriel Gelman & Karen Feigh Georgia Institute of Technology & John Rushby.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d215503460f949f5d5c/html5/thumbnails/9.jpg)
Modeling Platforms
![Page 10: Example of a Complementary use of Model Checking and Agent-based Simulation Gabriel Gelman & Karen Feigh Georgia Institute of Technology & John Rushby.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d215503460f949f5d5c/html5/thumbnails/10.jpg)
10
Model Checking: SAL (Symbolic Analysis Laboratory)
+ Simple models are checked for a given property+ Reachable state space of a specification is explored+ Exhaustive exploration of action space
Symbolic Model Checking does not require to explore full space
(singe action or combination of actions)
Start
State 1
Initial Conditions
State 2
Actioni
List<Actions>
State OK
State NOT OK
State 3 Action
j
List<Actions>
Actio
n k
Actionx
Abstract System Model
Action1 ,…, Action
i ,…Actionj ,…Action
k
Trace of Actions
![Page 11: Example of a Complementary use of Model Checking and Agent-based Simulation Gabriel Gelman & Karen Feigh Georgia Institute of Technology & John Rushby.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d215503460f949f5d5c/html5/thumbnails/11.jpg)
Step Flight Mode Airspeed Altitude Flaps Max Speed Mental Model Pitch1 Other 200 3000 Retracted 400 Level -1/1002 V/S FPA 201 2989 Retracted 400 Descend -1/1003 V/S FPA 200 2988 Extended 180 Descend 04 OPEN CLB 201 2989 Extended 180 Descend 05 OPEN CLB 200 2990 Extended 180 Descend 1/506 OPEN CLB 190 3291 Extended 180 Descend 3/100
Case Study Modeled in SALAirplane: Flies (descending)Automation: Track ModePilot: Dials Descend
1
2Airplane: Flies (descending)Automation: VS/FPA modePilot: Extends Flaps Airplane: Flies with Flaps (descending)
(exceeds Vmax)Automation: Reverses ModePilot: Does nothing
3
4Airplane: Flies with Flaps (descending) Automation: OP CLB modePilot: Does nothing Airplane: Flies with Flaps (descending)
Automation: OP CLB modePilot: Does nothing
5
Note: Each step is a state transition, time
is not modeled
6AUTOMATION SURPRISE
• Alt increase from 2990 to 3291• Mental Model still in descend• Positive Pitch
11
FCU: Flight Control Unit
State
State Transition
Initial State (FCU Alt = 3201 feet)
![Page 12: Example of a Complementary use of Model Checking and Agent-based Simulation Gabriel Gelman & Karen Feigh Georgia Institute of Technology & John Rushby.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d215503460f949f5d5c/html5/thumbnails/12.jpg)
12
Simulation: WMC (Work Models that Compute)
Aircraft Work Model
Expectations
AutoSurprise
Human Agent
Mental Model
Pulls
Mental Model
Stores
Updateable World Representation
SIM Core
Scripted Events
Initial Conditions
Traces of Key Metrics
ResourcesActions
WMC Work Model
Agents
Altitude, Heading,
Speed, Vertical Speed
![Page 13: Example of a Complementary use of Model Checking and Agent-based Simulation Gabriel Gelman & Karen Feigh Georgia Institute of Technology & John Rushby.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d215503460f949f5d5c/html5/thumbnails/13.jpg)
13
Simulation Runs Based on MC Output
1. Verify that the action sequence predicted by SAL to be problematic continues to be problematic
2. Refine SAL's prediction to include specific temporal relationships between events
Step 2: Extend Flaps
Step 1: Arm Approach
Step 3: Monitor Speed
Becomes t = 5: Extend Flaps
t = 2: Arm Approach
t = 9: Monitor Speed
![Page 14: Example of a Complementary use of Model Checking and Agent-based Simulation Gabriel Gelman & Karen Feigh Georgia Institute of Technology & John Rushby.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d215503460f949f5d5c/html5/thumbnails/14.jpg)
14
Simulation States that Varied
ILS Glideslope
Runway
FPA = 3°
Altit
ude
Ground
STAR approach
Cruise
Level Off Altitude
Level Off Duration
Go Around Altitude
Flaps Extension
Speed
STAR: Standard Terminal Arrival RouteILS: Instrument Landing System
FPA: Flight Path Angle
![Page 15: Example of a Complementary use of Model Checking and Agent-based Simulation Gabriel Gelman & Karen Feigh Georgia Institute of Technology & John Rushby.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d215503460f949f5d5c/html5/thumbnails/15.jpg)
Results
![Page 16: Example of a Complementary use of Model Checking and Agent-based Simulation Gabriel Gelman & Karen Feigh Georgia Institute of Technology & John Rushby.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d215503460f949f5d5c/html5/thumbnails/16.jpg)
16
Meaningful Scenarios from Simulation Traces
OPEN DES
OPEN CLB
No Change
Simulation Traces Leads to
Automation Surprise
No Auto Surprise
![Page 17: Example of a Complementary use of Model Checking and Agent-based Simulation Gabriel Gelman & Karen Feigh Georgia Institute of Technology & John Rushby.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d215503460f949f5d5c/html5/thumbnails/17.jpg)
17
Overview of Scenarios in Simulation Output
SC Mode AS Description1 DES No Mode reversion before level off,
early flaps extension leads to overspeed2 CLB Yes --"--3 DES Yes* Mode reversion after level off,
early flaps extension leads to overspeed4** CLB Yes --"--5 DES Yes* After level off,
dive leads to overspeed on current flap configuration6 CLB Yes --"--
SC: ScenarioAS: Automation Surprise
(*) Possibly due to artifact(**) SAL Scenario
![Page 18: Example of a Complementary use of Model Checking and Agent-based Simulation Gabriel Gelman & Karen Feigh Georgia Institute of Technology & John Rushby.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d215503460f949f5d5c/html5/thumbnails/18.jpg)
18
Model Checking Matching CaseSA
L
WM
CUnknown time step
Action ValueExtend flaps 201 knotsLevel Off Altitude 3200 feetLevel Off Duration 100 secondsGA Altitude 3281 feet
![Page 19: Example of a Complementary use of Model Checking and Agent-based Simulation Gabriel Gelman & Karen Feigh Georgia Institute of Technology & John Rushby.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d215503460f949f5d5c/html5/thumbnails/19.jpg)
19
Scenario 4: OPEN CLB
1. Level off2. Return to glideslope (dive) 3. Flaps Extension 4. Sets max speed below
current speed (former max speed = 220 knots, max speed with flaps = 205 knots)
5. OPEN CLB engages 6. Aircraft climbs
Zoom
![Page 20: Example of a Complementary use of Model Checking and Agent-based Simulation Gabriel Gelman & Karen Feigh Georgia Institute of Technology & John Rushby.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d215503460f949f5d5c/html5/thumbnails/20.jpg)
20
Scenario 6: OPEN CLB
1. Level off2. Return to glideslope (dive)3. Overspeed from dive4. OPEN CLB engages5. Aircraft climbs
Zoom
![Page 21: Example of a Complementary use of Model Checking and Agent-based Simulation Gabriel Gelman & Karen Feigh Georgia Institute of Technology & John Rushby.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d215503460f949f5d5c/html5/thumbnails/21.jpg)
21
Preconditions for Scenarios
SC: ScenarioAS: Automation Surprise
• Go Around (GA) altitude fixed at 3291 feet (as in SAL)
• Flaps Extension speed fixed at 226 knots (as in SAL)
• Level Off altitude and duration varied
![Page 22: Example of a Complementary use of Model Checking and Agent-based Simulation Gabriel Gelman & Karen Feigh Georgia Institute of Technology & John Rushby.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d215503460f949f5d5c/html5/thumbnails/22.jpg)
22
Preconditions for Scenarios
• Go Around (GA) altitude fixed at 6000 feet
• Level Off altitude fixed at 7000 feet
• Level Off duration and Flaps Extension speed varied
SC: ScenarioAS: Automation Surprise
![Page 23: Example of a Complementary use of Model Checking and Agent-based Simulation Gabriel Gelman & Karen Feigh Georgia Institute of Technology & John Rushby.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d215503460f949f5d5c/html5/thumbnails/23.jpg)
Conclusion
![Page 24: Example of a Complementary use of Model Checking and Agent-based Simulation Gabriel Gelman & Karen Feigh Georgia Institute of Technology & John Rushby.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d215503460f949f5d5c/html5/thumbnails/24.jpg)
24
Next Step: Simulation Model Checking
+ Implement capability for new scenarios into model checking+ Make model checking model more detailed
Scenario NarrativeCreate Model & Specifications for Model Checking (SAL)
Analyze Using Model Checking (SAL)
Create Models & Metric Specifications for Simulation (WMC)
Analyze Using Simulation (WMC)
![Page 25: Example of a Complementary use of Model Checking and Agent-based Simulation Gabriel Gelman & Karen Feigh Georgia Institute of Technology & John Rushby.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d215503460f949f5d5c/html5/thumbnails/25.jpg)
25
Conclusion
+ Examined same scenario using both model checking and simulation
+ Simulation results show expansion of Model Checking results (more scenarios & comprises aircraft dynamics and time)
+ Method was shown how to use the two frameworks in conjunction to examine system behavior
Model Checking
Simulation
Intro Auto Surp Platforms Method Results Conclusion
![Page 26: Example of a Complementary use of Model Checking and Agent-based Simulation Gabriel Gelman & Karen Feigh Georgia Institute of Technology & John Rushby.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d215503460f949f5d5c/html5/thumbnails/26.jpg)
Questions & Comments Welcome Now
26