DGTA-ADF Migrating to a Software Assurance Standard 2008 ADF Software Symposium FLTLT Patrick...
-
Upload
jerome-reed -
Category
Documents
-
view
214 -
download
0
Transcript of DGTA-ADF Migrating to a Software Assurance Standard 2008 ADF Software Symposium FLTLT Patrick...
DGTA-ADF
Migrating to a Software Assurance Standard
2008 ADF Software Symposium
FLTLT Patrick Redmond
SCI-DGTA
DGTA-ADF
Overview• The Problem with Legacy Software
• Migrating to a Software Assurance Standard
• The Potential Challenges- Bounding the Change- Low-Level Requirements- Traceability- Structural Coverage
• Case Study- MIL-STD-498 to DO-178B- Considered throughout.
DGTA-ADF
The Problem with Legacy Software• Numerous ADF platforms acquired where no software
assurance standard has been explicitly applied.
• Only MIL-STD-498 or DOD-STD-2167A.
• Development standards do not define how well software has to be constructed.
• During development, somebody made the decision to stop testing.
- Why?
• During development, somebody made the decision to review source code.
- Why?
• To what extent should the ADF rely on this software?
• To what extent does the ADF rely on this software?
DGTA-ADF
Migrating to a Software Assurance Standard• DGTA requires that a software assurance standard
be applied to legacy software systems.
• All development methods apply a level of assurance, but is that level acceptable?
• Applying a software assurance standard to each modification of in-service software progressively increases the integrity.
• Applies ‘Cancer Theory’ to Legacy Software
• Two methods:- Service History Argument and Application of Recognised
Standard e.g. RTCA DO-178B
- Software Assurance Task Matrix Negotiate a custom software assurance “standard” with DGTA
DGTA-ADF
How does migration work?
?
?
? ??
?? ?
?
?
? ?
?
?
?
?
?
? ?
Legacy Software of Unknown Integrity
Known Integrity
Software of Known(?) Integrity
DGTA considers the software to be compliant with the applied software assurance standard from this point forward.
Build x
Area Impacted byBuild x
Bui
ld y
DGTA-ADF
Service History Argument / Recognised Standard• Establish a Service History Argument that demonstrates that the
current software is acceptably safe.- Need to consider, among others:
Configuration Management Problem Reporting Change Control Relevance of Product Service History Operating Environments Safety Related Problems Design and Code Errors Error Rates
• No need to create a service history argument for in-service software, ADF has already determined that it is acceptably safe.
- Only applies to in-service, legacy software. Does not apply to acquisitions.- Does not apply if there is a substantial change of context in which the
software is used.
• Apply a recognised software assurance standard to the next modification.
• DGTA then consider the software to be compliant with that standard.
DGTA-ADF
Statement coverage.
High-level requirements coverage.
Software Assurance Task Matrix
Design Code Test
OFP B
OFP C
OFP D
Source code is verifiable.
Source code complies with low-level reqs.
Low-level requirements are defined.
High-level requirements are defined.
Decision coverage.
178B178B
Decision coverage.
Negotiate with SCI
Statement coverage plus untested code analysis.
DGTA-ADF
The Potential Challenges• Bounding the Change
- How much “re-assurance” needs to be done?
• Traceability
• Low-Level Requirements
• Structural Coverage
DGTA-ADF
The Easy Ones• Planning
• Additional Considerations- Tool Qualification
• Development of High-Level Requirements, Source Code and Executable Object Code
• Verification of High-Level Requirements
• Configuration Management
• Quality Assurance
DGTA-ADF
Bounding the Change• Software assurance standard only needs to be
applied to the scope of the modification.
• What is the scope of the modification?- The things that are changed, and- The things affected by the things that are changed.
• Change Impact Analysis:- Traceability- Memory Margin- Timing Margin- Data Flow- Control Flow- Input/Output- Development Environment and Process- Operational Characteristics- Certification Maintenance Requirements- Partitioning Analysis
DGTA-ADF
Traceability• Full traceability data was not recorded for many legacy
software systems.
• Software assurance standards generally require traceability to source code.
• When applying a software assurance standard to a legacy software system, how much traceability data needs to be generated?
• DGTA Position: Trace down and up once.
• Trace down from new or modified high-level requirements to affected and new low-level requirements, to affected and new code.
• Trace up from affected and new code to low-level requirements and to high-level requirements.
DGTA-ADF
Example – MIL-STD-498 to DO-178B• This example will be considered a number
of times.
• A legacy software system developed to MIL-STD-498 with typical artefacts.
• Can cause or contribute to Major hazards.
• Migrating to DO-178B Level C objectives.
DGTA-ADF
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4
Traceability
HLR2
LLR4
HLR1
LLR1 LLR2 LLR3
SCM1 SCM2 SCM3 SCM4 SCM5
New High-Level Requirement
New Low-Level Requirement
ModifiedLow-Level Requirement
New Source CodeModified Source Code
DGTA-ADF
Low-Level Requirements• Some legacy systems have no low-level requirements.
• Others have design descriptions that are not refined enough to be low-level requirements.
• Others have design descriptions that do not drive source code development (source code is developed from requirements).
• How many low-level requirements need to be defined or redefined?
• DGTA Position: Each affected low-level requirement and each low-level requirement identified by the down and up trace.
DGTA-ADF
Low-Level Requirements
HLR1 HLR2
LLR1 LLR2 LLR3 LLR4
SCM1 SCM2 SCM3 SCM4 SCM5
HLR2
LLR4LLR3
SCM2 SCM3 SCM4 SCM5
LLR1
DGTA-ADF
Low-Level Requirements• Verify that low-level requirements LLR1,
LLR3 and LLR4:- Comply with high-level requirements.- Are accurate and consistent.- Conform to standards.- Are traceable to high-level requirements.
• Ensure that low-level requirements LLR1, LLR3 and LLR4 are sufficiently refined to be directly translatable to source code.
- May need to further refine one low-level requirement into several.
• Leave LLR2 as is.
DGTA-ADF
Structural Coverage• Software assurance standards generally require
assessment of structural coverage in order to demonstrate that testing is complete.
• Purpose of Structural Coverage Objectives is to:- Identify shortcoming in requirements based test cases or
procedures,- Identify inadequacies in software requirements,- Identify dead code, and- Identify deactivated code.
• How do these measures apply to modification of legacy software systems?
• DGTA Position:- For requirements based measures: each new, modified or
affected requirement needs to be tested.- For structure based measures: each new or modified source
code module.
DGTA-ADF
Structural Coverage
HLR1 HLR2
LLR1 LLR2 LLR3 LLR4
SCM1 SCM2 SCM3 SCM4 SCM5
HLR2
LLR4LLR3
SCM2 SCM3 SCM4 SCM5
LLR1
HLR1
Statement Coverage
Normal Range and Robustness Tests
Normal Range and Robustness Tests
DGTA-ADF
Structural Coverage• Normal Range and Robustness Tests
- High-level requirements that have been added or modified.
- High-level requirements where a dependent low-level requirement has been added or modified.
- High-level requirements where the implementation (source code) of a dependent low-level requirement has been added or modified.
- Low-level requirements that have been added or modified.
- Low-level requirements where the implementation (source code) has been added or modified.
• Structural Coverage Objectives:- Structural coverage of all new or modified source
code modules.
DGTA-ADF
Additional Testing• Also need to consider:
- Data Dependencies Has the change impacted data that other functions rely
upon?- Control Flow Dependencies
Has the change inadvertently/adversely disrupted control flow or coupling?
- Timing Dependencies Has the change violated a timing constraint?
- Memory Space Dependencies Has the change violated memory constraints or used
memory space assigned to other functions?
DGTA-ADF
What about bug fixes?• A large part of in-service support is the
rectification of bugs.
• Bug fixes may not commence at the requirements level, may start with an identified fault in the source code.
• To what extent should software assurance standards be applied to bug fixes?
• DGTA Position: Trace up from the modified source code.
DGTA-ADF
LLR7LLR7LLR5LLR5
SCM8SCM8
HLR4HLR4HLR3HLR3
What about bug fixes?
LLR6 LLR8
SCM6 SCM7 SCM9 SCM10
Additional traceability data
Structural Coverage
Normal Range and Robustness TestsLow-Level Requirements
DGTA-ADF
Conclusions• DGTA requires the application of a software assurance
standard to legacy software systems.
• All development methods provide a level of assurance, writing it down can identify gaps.
• A software assurance standard can be applied either by:- A Product Service History Argument and application of a recognised
software assurance standard, or- A Software Assurance Task Matrix.
• The software assurance standard need only be applied to the current modification.
- Determine extent through change impact analysis.
• For legacy software systems, a large number of assurance objectives are probably already being achieved.
• Meeting a recognised software assurance standard will probably require additional effort in the areas of:
- Traceability,- Low-Level Requirements, and- Structural Coverage
DGTA-ADF
Questions?