Automated)Design/Time)Analysis)) …...GOES/R • NOAA/NASA)GOES/R – Nextgeneraon)GOES) –...
Transcript of Automated)Design/Time)Analysis)) …...GOES/R • NOAA/NASA)GOES/R – Nextgeneraon)GOES) –...
![Page 1: Automated)Design/Time)Analysis)) …...GOES/R • NOAA/NASA)GOES/R – Nextgeneraon)GOES) – ConFnuity)of)GOES) – Improvementof)remotely)sensed)environmental)data – Mustbe)extremely)reliable)and](https://reader030.fdocuments.in/reader030/viewer/2022040919/5e954fa19ccc086fad3264df/html5/thumbnails/1.jpg)
Automated Design-‐Time Analysis for the GOES-‐R System
David Hall and Corina Păsăreanu SGT and CMU SV, NASA Ames Research Center
![Page 2: Automated)Design/Time)Analysis)) …...GOES/R • NOAA/NASA)GOES/R – Nextgeneraon)GOES) – ConFnuity)of)GOES) – Improvementof)remotely)sensed)environmental)data – Mustbe)extremely)reliable)and](https://reader030.fdocuments.in/reader030/viewer/2022040919/5e954fa19ccc086fad3264df/html5/thumbnails/2.jpg)
GOES
• GeostaFonary OperaFonal Environmental Satellites (GOES) – Operated by the NaFonal Oceanic and Atmospheric AdministraFon (NOAA)
– Provide conFnuous weather imagery and monitoring of meteorological and space environment data
– To protect life and property across the US
![Page 3: Automated)Design/Time)Analysis)) …...GOES/R • NOAA/NASA)GOES/R – Nextgeneraon)GOES) – ConFnuity)of)GOES) – Improvementof)remotely)sensed)environmental)data – Mustbe)extremely)reliable)and](https://reader030.fdocuments.in/reader030/viewer/2022040919/5e954fa19ccc086fad3264df/html5/thumbnails/3.jpg)
GOES-‐R
• NOAA/NASA GOES-‐R – Next generaFon GOES – ConFnuity of GOES – Improvement of remotely sensed environmental data – Must be extremely reliable and correct – SoOware intensive – Many interacFve components – … challenge to verificaFon
![Page 4: Automated)Design/Time)Analysis)) …...GOES/R • NOAA/NASA)GOES/R – Nextgeneraon)GOES) – ConFnuity)of)GOES) – Improvementof)remotely)sensed)environmental)data – Mustbe)extremely)reliable)and](https://reader030.fdocuments.in/reader030/viewer/2022040919/5e954fa19ccc086fad3264df/html5/thumbnails/4.jpg)
TesFng
– typically used to ensure soOware reliability – oOen manual and Fme-‐consuming – used late in the soOware life cycle – aOer the code has been wriRen – when it is expensive to fix discovered interacFon errors
![Page 5: Automated)Design/Time)Analysis)) …...GOES/R • NOAA/NASA)GOES/R – Nextgeneraon)GOES) – ConFnuity)of)GOES) – Improvementof)remotely)sensed)environmental)data – Mustbe)extremely)reliable)and](https://reader030.fdocuments.in/reader030/viewer/2022040919/5e954fa19ccc086fad3264df/html5/thumbnails/5.jpg)
Our Goal
• SystemaFc design Fme analysis • IdenFfying and correcFng errors at design Fme is easier and more cost effecFve
• Even if the system is already implemented – behavior idenFfied from the design specificaFons can be used to guide tesFng and assess completeness of the test cases
![Page 6: Automated)Design/Time)Analysis)) …...GOES/R • NOAA/NASA)GOES/R – Nextgeneraon)GOES) – ConFnuity)of)GOES) – Improvementof)remotely)sensed)environmental)data – Mustbe)extremely)reliable)and](https://reader030.fdocuments.in/reader030/viewer/2022040919/5e954fa19ccc086fad3264df/html5/thumbnails/6.jpg)
Analysis Approach
• The state transiFon behavior of the ground segment of the GOES-‐R
• Translated into MathWorks’ Stateflow notaFon – Natural mapping between GOES-‐R design documents and state-‐charts
– Translated using Polyglot – Analyzed using Ames JPF open-‐source verificaFon tool-‐set
![Page 7: Automated)Design/Time)Analysis)) …...GOES/R • NOAA/NASA)GOES/R – Nextgeneraon)GOES) – ConFnuity)of)GOES) – Improvementof)remotely)sensed)environmental)data – Mustbe)extremely)reliable)and](https://reader030.fdocuments.in/reader030/viewer/2022040919/5e954fa19ccc086fad3264df/html5/thumbnails/7.jpg)
Polyglot An extensible framework for defining different
semanFc variants of Statecharts • Provides modeling and analysis for mulFple Statechart formalisms
• Captures interacFons between components • Based on formal semanFcs that captures the variants of Statecharts
• Available from: hRps://wiki.isis.vanderbilt.edu/MICTES/index.php/PublicaFons
![Page 8: Automated)Design/Time)Analysis)) …...GOES/R • NOAA/NASA)GOES/R – Nextgeneraon)GOES) – ConFnuity)of)GOES) – Improvementof)remotely)sensed)environmental)data – Mustbe)extremely)reliable)and](https://reader030.fdocuments.in/reader030/viewer/2022040919/5e954fa19ccc086fad3264df/html5/thumbnails/8.jpg)
Polyglot -‐-‐ Tool Overview
Rhapsody
IMPORT Simulink/Stateflow
Pluggable SemanFcs
Generic ExecuFon Environment
UML Rhapsody
State machine model (structure) + proper;es (in Java)
EXPORT
Java Pathfinder
Stateflow
Data interface
Modeling / Intermediate RepresentaFon
ProperFes
ProperFes
Execu;on Control
[ISSTA 2011, NFM 2012, MoDeVVa 2012]
Java
![Page 9: Automated)Design/Time)Analysis)) …...GOES/R • NOAA/NASA)GOES/R – Nextgeneraon)GOES) – ConFnuity)of)GOES) – Improvementof)remotely)sensed)environmental)data – Mustbe)extremely)reliable)and](https://reader030.fdocuments.in/reader030/viewer/2022040919/5e954fa19ccc086fad3264df/html5/thumbnails/9.jpg)
Usage
1. User defines only the structure of the Statechart 2. The “pluggable” semanFcs are selected from a
defined set 3. Temporal properFes specified with paRern-‐based
system 4. Analysis performed with:
– SimulaFon (run the Java program) – Java Pathfinder: explicit state model checking – Symbolic Pathfinder: automated test-‐sequence generaFon
available from: hRp://babelfish.arc.nasa.gov/trac/jpf/wiki
![Page 10: Automated)Design/Time)Analysis)) …...GOES/R • NOAA/NASA)GOES/R – Nextgeneraon)GOES) – ConFnuity)of)GOES) – Improvementof)remotely)sensed)environmental)data – Mustbe)extremely)reliable)and](https://reader030.fdocuments.in/reader030/viewer/2022040919/5e954fa19ccc086fad3264df/html5/thumbnails/10.jpg)
Statechart review • Consider this Statechart* :
• Event “e” leads to S4 (UML), S5 (Rhapsody), or (S6) Stateflow
• UML semanFcs evaluate transiFon acFons at the end of a transiFon path, and Rhapsody semanFcs perform transiFon acFons when they are encountered.
S1
/ a = true
S2
e / a = false
S4 S5
S6 S3
[a] [!a]
e
![Page 11: Automated)Design/Time)Analysis)) …...GOES/R • NOAA/NASA)GOES/R – Nextgeneraon)GOES) – ConFnuity)of)GOES) – Improvementof)remotely)sensed)environmental)data – Mustbe)extremely)reliable)and](https://reader030.fdocuments.in/reader030/viewer/2022040919/5e954fa19ccc086fad3264df/html5/thumbnails/11.jpg)
Design Choice
Java as a common language representaFon
– Executable representaFon for the models, for quick validaFon and debugging
– Enables modular and extensible design for Polygot – Leverage JPF and SPF for model analysis and test-‐case generaFon
– Large acFon languages can be mapped to Java
![Page 12: Automated)Design/Time)Analysis)) …...GOES/R • NOAA/NASA)GOES/R – Nextgeneraon)GOES) – ConFnuity)of)GOES) – Improvementof)remotely)sensed)environmental)data – Mustbe)extremely)reliable)and](https://reader030.fdocuments.in/reader030/viewer/2022040919/5e954fa19ccc086fad3264df/html5/thumbnails/12.jpg)
Property SpecificaFon
• Earlier study by Dwyer et al. found 92% of real world specificaFons fall into a small category
• Property consists of 2 pieces: 1. Scope: when should property hold 2. PaRern: the condiFons that should be saFsfied
• Implemented as graphical extension to Simulink/Stateflow
![Page 13: Automated)Design/Time)Analysis)) …...GOES/R • NOAA/NASA)GOES/R – Nextgeneraon)GOES) – ConFnuity)of)GOES) – Improvementof)remotely)sensed)environmental)data – Mustbe)extremely)reliable)and](https://reader030.fdocuments.in/reader030/viewer/2022040919/5e954fa19ccc086fad3264df/html5/thumbnails/13.jpg)
PaRerns
• 5 total paRerns in 2 groups • Occurrence
– Absence: never true – Universality: always true – Existence: True at least once
• Order – Precedence: a state must precede another – Response: a state must follow another
![Page 14: Automated)Design/Time)Analysis)) …...GOES/R • NOAA/NASA)GOES/R – Nextgeneraon)GOES) – ConFnuity)of)GOES) – Improvementof)remotely)sensed)environmental)data – Mustbe)extremely)reliable)and](https://reader030.fdocuments.in/reader030/viewer/2022040919/5e954fa19ccc086fad3264df/html5/thumbnails/14.jpg)
Analysis with JPF / SPF
• Analysis and test-‐case generaFon is performed with the Symbolic Pathfinder (SPF), the symbolic execuFon module of Java Pathfinder
• Tests reachability, generates test-‐vectors of input sequences – Can be fed back to the original modeling tool or to Polyglot
![Page 15: Automated)Design/Time)Analysis)) …...GOES/R • NOAA/NASA)GOES/R – Nextgeneraon)GOES) – ConFnuity)of)GOES) – Improvementof)remotely)sensed)environmental)data – Mustbe)extremely)reliable)and](https://reader030.fdocuments.in/reader030/viewer/2022040919/5e954fa19ccc086fad3264df/html5/thumbnails/15.jpg)
Java Pathfinder (JPF)
extensible virtual machine framework for java bytecode verificaFon workbench to implement all kinds of verificaFon tools typical use cases:
soOware model checking (detecFon of deadlocks, races, assert errors) test case generaFon (symbolic execuFon) ... and many more
![Page 16: Automated)Design/Time)Analysis)) …...GOES/R • NOAA/NASA)GOES/R – Nextgeneraon)GOES) – ConFnuity)of)GOES) – Improvementof)remotely)sensed)environmental)data – Mustbe)extremely)reliable)and](https://reader030.fdocuments.in/reader030/viewer/2022040919/5e954fa19ccc086fad3264df/html5/thumbnails/16.jpg)
Symbolic Pathfinder (SPF)
combines symbolic execution, model checking and constraint solving applies to executable models and code handles dynamic data structures, loops, multi-threading, strings java pathfinder extension project jpf-symbc [TACAS’03, ISSTA’08, ASE’10] generates automatically input sequences to drive statecharts on different paths
Error Report
Systematic Analysis
Sym Exe Tree
Test Sequences
Constraint Solving
Java bytecode
symbolic pathfinder (spf)
![Page 17: Automated)Design/Time)Analysis)) …...GOES/R • NOAA/NASA)GOES/R – Nextgeneraon)GOES) – ConFnuity)of)GOES) – Improvementof)remotely)sensed)environmental)data – Mustbe)extremely)reliable)and](https://reader030.fdocuments.in/reader030/viewer/2022040919/5e954fa19ccc086fad3264df/html5/thumbnails/17.jpg)
Symbolic Pathfinder available from jpf distribuFon hRp://babelfish.arc.nasa.gov/trac/jpf/wiki/projects/jpf-‐symbc
![Page 18: Automated)Design/Time)Analysis)) …...GOES/R • NOAA/NASA)GOES/R – Nextgeneraon)GOES) – ConFnuity)of)GOES) – Improvementof)remotely)sensed)environmental)data – Mustbe)extremely)reliable)and](https://reader030.fdocuments.in/reader030/viewer/2022040919/5e954fa19ccc086fad3264df/html5/thumbnails/18.jpg)
ExecuFon with SPF
Pluggable SemanFcs
Generic ExecuFon Engine
UML Rhapsody Stateflow State machine + property
Execu;on engine
1. Input data
2. Inspect
3. Set
Java Pathfinder / Symbolic Pathfinder
Test vectors
Property reports
4. Generates
![Page 19: Automated)Design/Time)Analysis)) …...GOES/R • NOAA/NASA)GOES/R – Nextgeneraon)GOES) – ConFnuity)of)GOES) – Improvementof)remotely)sensed)environmental)data – Mustbe)extremely)reliable)and](https://reader030.fdocuments.in/reader030/viewer/2022040919/5e954fa19ccc086fad3264df/html5/thumbnails/19.jpg)
ApplicaFons
• Analysis of arbiter module for the Mars ExploraFon Rover [ISSTA 2011]
• InteracFon between Ares launch vehicle and Orion Crew ExploraFon Vehicle [NFM 2012]
![Page 20: Automated)Design/Time)Analysis)) …...GOES/R • NOAA/NASA)GOES/R – Nextgeneraon)GOES) – ConFnuity)of)GOES) – Improvementof)remotely)sensed)environmental)data – Mustbe)extremely)reliable)and](https://reader030.fdocuments.in/reader030/viewer/2022040919/5e954fa19ccc086fad3264df/html5/thumbnails/20.jpg)
Results for GOES-‐R
MathWorks/Stateflow: • GSSWRS
– Statechart with 8 states and 35 transiFons
– Polyglot: 759 + 36 Java LOC • GSTier:
– Statechart with 2 parallel state machines (6 states, 24 transiFons + 3 states, 6 transiFons)
– Polyglot: 701 + 36 Java LOC
![Page 21: Automated)Design/Time)Analysis)) …...GOES/R • NOAA/NASA)GOES/R – Nextgeneraon)GOES) – ConFnuity)of)GOES) – Improvementof)remotely)sensed)environmental)data – Mustbe)extremely)reliable)and](https://reader030.fdocuments.in/reader030/viewer/2022040919/5e954fa19ccc086fad3264df/html5/thumbnails/21.jpg)
Example Simplified Model
![Page 22: Automated)Design/Time)Analysis)) …...GOES/R • NOAA/NASA)GOES/R – Nextgeneraon)GOES) – ConFnuity)of)GOES) – Improvementof)remotely)sensed)environmental)data – Mustbe)extremely)reliable)and](https://reader030.fdocuments.in/reader030/viewer/2022040919/5e954fa19ccc086fad3264df/html5/thumbnails/22.jpg)
Analysis with SPF
• Ran up to depth 40 • Generates 1482 test cases • ProperFes:
– “absence” – Model should never be in state “Degraded” if command is “online”
![Page 23: Automated)Design/Time)Analysis)) …...GOES/R • NOAA/NASA)GOES/R – Nextgeneraon)GOES) – ConFnuity)of)GOES) – Improvementof)remotely)sensed)environmental)data – Mustbe)extremely)reliable)and](https://reader030.fdocuments.in/reader030/viewer/2022040919/5e954fa19ccc086fad3264df/html5/thumbnails/23.jpg)
Future Work
• Expand and robusFfy the tool • ComposiFonal verificaFon from JPF [FASE 2009] for increased scalability
• IdenFfy and analyze more properFes of ground system in the GOES-‐R project – Check conformance at different levels of abstracFon
– Measure coverage