Incremental Lifecycle Assurance of Critical Systems Poster (SEI … · 2015-10-23 · Incremental...
2
SEI Research Review 2015 Contact: Peter Feiler [email protected] Distribution Statement A: Approved for Public Release; Distribution is Unlimited Incremental Lifecycle Assurance of Critical Systems Critical System Assurance Challenge The traditional development lifecycle using existing methods of system engineering result in • Assurance-related post-unit test software rework at 50% of total system cost and growing • Labor-intensive system safety analysis without addressing software as major hazard source • High percentage of operator work arounds for software fixes due to high recertification cost NIST Study Current requirement engineering practice relies on stakeholders traceability and document reviews resulting in high rate of requirement change Incremental Lifecycle Assurance Goals • Improve requirement quality through coverage and managed uncertainty • Improve evidence quality through compositional analytical verification • Measurably reduce certification related rework cost through virtual integration and verification automation Impact and Alignment • AMRDEC Joint Multi-Role (JMR) Tech Demo: maturation of ACVIP for Future Vertical Lift (FVL) • Aerospace industry System Architecture Virtual Integration (SAVI) multi-year initiative • Standards: SAE AS-2C (AADL Requirements, Constraints), SAE S18 (ARP4761 System Safety) • Regulatory agencies: NRC, FDA, AAMI/UL Rolls Royce Study Managed awareness of requirement uncertainty can lead to 50% reduction in requirement changes U Minnesota Study Requirements often span multiple architecture layers Build the System Build the Assurance Case Requirements Engineering System Design Software Architectural Design System Architecture Validation Software Architecture Validation Component Software Design Code Development Integration Build Target Build Deployment Build Requirements Validation Design Validation Acceptance Test System Test Integration Test Architecture Modeling Analysis & Generation Three Dimensions of Requirement Coverage Fault Propagation Ontology *System Architecture Virtual Integration (SAVI) Aerospace industry initiative System interactions, state, behavior Design & operational quality attributes Utility (L,M) (M,M) (H,H) (H, L) (H,H) (H,H) (H,M) (H,L) Fault impact & contributors Unit Test Requirements error % Incomplete 21% Missing 33% Incorrect 24% Ambiguous 6% Inconsistent 5% Selection Low Precedence Medium Precedence High Precedence Weight 9 3 1 Precedence No experience of concept, or environment. Historically volatile. Some experience in related environments. Some historic volatility. Concept already in service. Low historic volatility. Incremental assurance through virtual system integration for early discovery Return on Investment study by SAVI* Priority focused architecture design exploration for high payoff Measurable improvement (Rolls Royce) Compositional verification and partitions to limit assurance impact Model Repository Architecture Model Component Models System Implementation System configuration FY15 Focus FY16 Focus Architecture-Led Incremental System Assurance (ALISA) Approach Semantically Consistent Unification of Modeling Concepts from Different Perspectives And their Use in Existing Practice Standards Architecture-centric Virtual System Integration (ACVIP) Incremental Lifecycle Assurance (ALISA) ALISA Workflow & Eclipse-based Workbench AMRDEC JMR Situational Awareness Requirements Case Study (Early Life Cycle) SAVI Multi-layered multi-dimensional aircraft verification (Multi-phase) Rolls Royce Engine Control Certification Study (Late Life Cycle) Measurement-driven Assurance Cost and Confidence Improvement through Incremental Lifecycle Assurance Goal, Intent, Requirement, Assumption, Claim Architecture-focused Requirements & Hazard Analysis Textual Requirements for a Patient Therapy System Importance of understanding system boundary We have effectively specified a system partial architecture Same Requirements Mapped to an Architecture Model Technical and Operational Validation in Actual Projects Access to Actual Project Information Anticipated Improvement Thresholds 25% Higher Requirement/Hazard Coverage 35% Higher Evidence Confidence 25% Reduced Uncertainty Impact 1. The patient shall never be infused with a single air bubble more than 5ml volume. 2. When a single air bubble more than 5ml volume is detected, the system shall stop infusion within 0.2 seconds. 3. When piston stop is received, the system shall stop piston movement within 0.01 seconds. 4. The system shall always stop the piston at the bottom or top of the chamber. Assessment of Potential for Proportional Recertification Cost Benefit and Risk of Partial Verification Measurably Increased Assurance Confidence Credit for Analytical Evidence Measurably Reduced Defect Leakage & Assurance Cost Apply COQualMO and SAVI ROI Architecture-led Contract- based Compositional Analysis & Verification Assurance Plan with Multi- valued Argumentation Logic Obstacle, Fault, Defect, Hazard, Vulnerability, Challenge Verification Method, Activity, Result, Evidence, Counter evidence Assurance & Qualification Improvement Strategy Assurance: Sufficient evidence that a system implementation meets system requirements Three Dimensions of Incremental Assurance Early Discovery leads to Rework Reduction Project Approach Patient Therapy System Infusion System Drug Delivery Hardware Pump System Pump Hardware Air Bubble Sensor Pump Controller 1 2 3 4 Reduce storage latency on customer DB to < 200ms. Deliver video in real time. Add CORBA middleware in < 20 person-months. Change Web user interface in < 4 person-weeks. Power outage at site 1 requires traffic redirected to site 2 in < 3 seconds. Network failure detected and recovered in < 1.5 minutes Credit Card transactions are secure 99.999% of time. Customer DB authorization works 99.999% of the time. Data Latency Transaction Throughput New Products Change COTS H/W Failure COTS S/W Failures Data Confidentiality Data Integrity Performance Modifiability Availability Security Guarantees Assumptions Environment Constraints/ Controls Resources Behavior State Input Output Implementation constraints Invariants Exceptional conditions Omission errors Commission errors Sequence errors Replication errors Concurrency errors Authorization errors Value errors Timing errors Rate errors Authentication errors System Under Control Behavior Actuator Sensor State Control System Behavior Output Input State Timing (H) Performance (M) Safety (H) Security (L) Reliability (L) Modifiability (L) Portability (M) Configurability (M) Design & Req Refinement Design & Req Refinement Compositional Verification Requirement Coverage Compositional Verification VA VA VA RS RS RS VA VA VA RS RS RS RS C C C Mission Requirements Function Behavior Performance Survivability Requirements Reliability Safety Security Operational & failure modes Resource, Timing & Performance Analysis Reliability, Safety, Security Analysis Architecture-led Requirement Specification Architecture-centric Virtual System Integration Static Analysis & Compositional Verification Incremental Assurance Plans & Cases throughout Life Cycle
Transcript of Incremental Lifecycle Assurance of Critical Systems Poster (SEI … · 2015-10-23 · Incremental...
SEI Research Review 2015
Contact: Peter Feiler [email protected]
Distribution Statement A:Approved for Public Release;Distribution is Unlimited
Incremental Lifecycle Assurance of Critical Systems
Critical System Assurance ChallengeThe traditional development lifecycle using existing methods of system engineering result in
• Assurance-related post-unit test software rework at 50% of total system cost and growing
• Labor-intensive system safety analysis without addressing software as major hazard source
• High percentage of operator work arounds for software �xes due to high recerti�cation cost
NIST StudyCurrent requirement engineering practice relies on stakeholders traceability and document reviews resulting in high rate of requirement change
Incremental Lifecycle Assurance Goals• Improve requirement quality through coverage and managed
uncertainty
• Improve evidence quality through compositional analytical veri�cation
• Measurably reduce certi�cation related rework cost through virtual integration and veri�cation automation
Impact and Alignment• AMRDEC Joint Multi-Role (JMR) Tech Demo: maturation of ACVIP
for Future Vertical Lift (FVL)
• Aerospace industry System Architecture Virtual Integration (SAVI) multi-year initiative
• Standards: SAE AS-2C (AADL Requirements, Constraints), SAE S18 (ARP4761 System Safety)
• Regulatory agencies: NRC, FDA, AAMI/UL
Rolls Royce StudyManaged awareness of requirement uncertainty can lead to 50% reduction in requirement changes
U Minnesota StudyRequirements often span multiple architecture layers
Build the System
Build the Assurance Case
RequirementsEngineering
SystemDesign
SoftwareArchitecturalDesign
SystemArchitectureValidation
SoftwareArchitectureValidation
ComponentSoftwareDesign
CodeDevelopment
IntegrationBuild
TargetBuild
DeploymentBuild
RequirementsValidation
DesignValidation
AcceptanceTest
SystemTest
IntegrationTest
ArchitectureModelingAnalysis &Generation
Three Dimensions of Requirement Coverage
Fault Propagation Ontology
*System Architecture Virtual Integration (SAVI) Aerospace industry initiative
System interactions, state, behavior Design & operational quality attributes
Utility
(L,M)
(M,M)
(H,H)
(H, L)
(H,H)
(H,H)
(H,M)
(H,L)
Fault impact & contributors
Unit Test
Requirements error %
Incomplete 21%
Missing 33%
Incorrect 24%
Ambiguous 6%
Inconsistent 5%
Selection
LowPrecedence
MediumPrecedence
HighPrecedence
Weight
9
3
1
Precedence
No experience of concept, or environment. Historically volatile.
Some experience in related environments. Some historic volatility.
Concept already in service. Low historic volatility.
Incremental assurance through virtual system integration for early discovery
Return on Investment study by SAVI*
Priority focused architecture design exploration for high payoff
Measurable improvement (Rolls Royce)
Compositional veri�cation and partitions to limit assurance impact
Model Repository
Architecture Model
Component Models
System Implementation
System con�guration
FY15Focus
FY16Focus
Architecture-Led Incremental System Assurance (ALISA) Approach
Semantically Consistent Uni�cation of Modeling Concepts from Different Perspectives And their Use in Existing Practice Standards
Architecture-centric Virtual System Integration (ACVIP)Incremental Lifecycle Assurance (ALISA)
ALISA Workflow & Eclipse-based Workbench
AMRDEC JMR Situational Awareness Requirements Case Study (Early Life Cycle) SAVI Multi-layered multi-dimensional aircraft verification (Multi-phase)
Rolls Royce Engine Control Certification Study (Late Life Cycle)
Measurement-driven Assurance Cost and Confidence Improvement through Incremental Lifecycle Assurance
Goal, Intent, Requirement,Assumption, Claim
Architecture-focused Requirements & Hazard Analysis
Textual Requirements for a Patient Therapy System
Importance of understanding system boundary We have effectively speci�ed a system partial architecture
Same Requirements Mapped to an Architecture Model
Technical and OperationalValidation in Actual Projects
Access to Actual Project Information
Anticipated Improvement Thresholds 25% Higher Requirement/Hazard Coverage
35% Higher Evidence Con�dence 25% Reduced Uncertainty Impact
1. The patient shall never be infused with a single air bubble more than 5ml volume.
2. When a single air bubble more than 5ml volume is detected, the system shall stop infusion within 0.2 seconds.
3. When piston stop is received, the system shall stop piston movement within 0.01 seconds.
4. The system shall always stop the piston at the bottom or top of the chamber.
Assessment of Potential for Proportional Recertification Cost
Bene�t and Risk of Partial Veri�cation
Measurably IncreasedAssurance Con�dence
Credit for Analytical Evidence
Measurably Reduced Defect Leakage & Assurance Cost
Apply COQualMO and SAVI ROI
Architecture-led Contract- based Compositional Analysis & Veri�cation
Assurance Plan with Multi-valued Argumentation Logic
Obstacle, Fault, Defect, Hazard,Vulnerability, Challenge
Verification Method, Activity, Result,Evidence, Counter evidence
Assurance & Quali�cation Improvement StrategyAssurance: Suf�cient evidence that a system implementation meets system requirements
Three Dimensions of Incremental Assurance
Early Discovery leads to Rework ReductionProject Approach
Patient Therapy System
Infusion System
Drug Delivery Hardware
Pump System
Pump Hardware
Air Bubble Sensor
Pump Controller
1 2
34
Reduce storage latency on customer DB to < 200ms.
Deliver video in real time.
Add CORBA middleware in < 20 person-months.
Change Web user interface in < 4 person-weeks.
Power outage at site 1 requires traf�c redirected to site 2 in < 3 seconds.
Network failure detected and recovered in < 1.5 minutes
Credit Card transactions are secure 99.999% of time.
Customer DB authorization works 99.999% of the time.
Data Latency
Transaction Throughput
New Products
Change COTS
H/W Failure
COTS S/W Failures
Data Con�dentiality
Data Integrity
Performance
Modi�ability
Availability
Security
GuaranteesAssumptions
Environment
Constraints/Controls
Resources
Behavior
StateInput Output
Implementation
constraints
Invariants
Exceptional
conditions
Omission errors Commission errors
Sequence errors
Replication errors
Concurrency errors
Authorization errors
Value errors
Timing errors
Rate errors
Authentication errors System Under Control
Behavior
Actuator Sensor
State
Control System
Behavior
Output Input
State
Timing (H) Performance (M) Safety (H)Security (L) Reliability (L) Modi�ability (L) Portability (M) Con�gurability (M)
Design & Req Re�nement
Design & Req Re�nement
Compositional Veri�cation
Requirement Coverage
Compositional Veri�cation
VA VA VA
RS RS RS VA VA VA
RS
RS RS RS
C
C
C
Mission Requirements
Function Behavior Performance
Survivability Requirements
Reliability Safety Security
Operational & failure modes
Resource, Timing & Performance
Analysis
Reliability, Safety, Security
Analysis
Architecture-led Requirement Speci�cation
Architecture-centric Virtual System
Integration
Static Analysis & Compositional
Veri�cation
Incremental Assurance Plans & Cases
throughout Life Cycle
Copyright 2015 Carnegie Mellon University
This material is based upon work funded and supported by the Department of Defense under Contract No. FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center.
Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Department of Defense.
NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.
This material has been approved for public release and unlimited distribution except as restricted below.
Internal use:* Permission to reproduce this material and to prepare derivative works from this material for internal use is granted, provided the copyright and “No Warranty” statements are included with all reproductions and derivative works.
External use:* This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other external and/or commercial use. Requests for permission should be directed to the Software Engineering Institute at [email protected].
* These restrictions do not apply to U.S. government entities.
Carnegie Mellon® is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.
DM-0002838
