Verification of avionics software to meet DO-178B/C · Verification of avionics software to meet...
-
Upload
hoanghuong -
Category
Documents
-
view
227 -
download
2
Transcript of Verification of avionics software to meet DO-178B/C · Verification of avionics software to meet...
Verification of avionics
software to meet DO-178B/C
What, why and how?
Software V&V. NMI. Bristol. 10th November 2015
Dr. Guillem Bernat
On-target Software Verification
Validation and Verification Validation: Checking that you are building the right thing
Verification: Checking that you are building the thing right
Software verification Process to determine that the software satisfies requirements
Does the software do what is supposed to do? On time?, all statements tested,
does not exceed stack limits, etc.
On-target: Information collected as close as possible to the final product.
Same object code as final system
Same target as final system
No models or assumptions needed
“Use evidence of the behaviour of the code running on target”
What is Worst-Case Execution Time
(WCET)?
WCET is longest possible execution time for a piece of
code on a specific hardware platform
An exact WCET is generally not feasible
Depends on SW architecture
SW structure
E.g. Data driven vs. structure driven programs
Hardware architecture
And other HW interactions
….
3
DO-178C 6.3.4: Reviews and Analyses of Source Code
These review and analysis activities … confirm that the Source Code satisfies these objectives:
…
f. Accuracy and consistency: The objective is to determine the correctness and consistency of the Source Code, including … worst-case execution timing, … The compiler (including its options), the linker (including its options), and some hardware features may have an impact on the worst-case execution timing and this impact should be assessed
Further clarified in DO-248B. FAQ#73
Note especially: Timing measurements by themselves cannot be used without an analysis demonstrating that the worst-case timing would be achieved, but processor behavior should be assessed.
Why measure WCET for aerospace applications?
4
Objective Activity Applicability by
Software Level
Description Ref Ref A B C D
Source Code complies with low-level
requirements
6.3.4.f 6.3.4
Manual analysis and measurement:
Review code to identify worst-case path
Devise test cases
Instrument code
Measure execution time
Consider the use of a "safety margin"
Identifying the worst-case path is difficult:
Source code might be misleading: Simple assignment statements may take a long time to execute
Complex-looking groups of statements might take little (or no) time to execute
Loop bounds can be difficult to determine
Mutually exclusive code paths
Measuring is difficult:
• Excluding interference from other tasks/partitions/cores
• Repeating timing measurements as problems are observed and investigated
What is current practice?
5
WCET analysis strategies
Definitions Root: the piece of code we are doing timing analysis for. E.g. Task, interrupt, etc.
Arguments of what is WCET High Watermark: The longest observed execution “end-to-end” of Root (e.g. task)
Requires finding out the test case that drives the worst case path and worst-
case hardware usage.
WCET *:Approximation to WCET by only partial instrumentation up to “n” levels
deep in the call tree.
WCET (deep): Analytical calculation of the longest execution time of Root. Based
on measurements of small sections of code and analytical path calculations.
Challenges to WCET collection… and ways to address them
Two main sources of challenges to collecting worst-case
execution time
System constraints
"Performance improvement" features
System constraints:
Limited target resources/limited data bandwidth…
… addressed by flexible instrumentation
"Performance improvement" features include:
"locally deterministic" hardware acceleration features (pipelines, branch prediction)
… collect multiple instances of timing data
cache memory (data, instruction)
… follow guidance of CAST-20
Multicore
… measure on a core-by-core basis, against no load and a high load
RTOS
… instrument context switches, demultiplex traces
8
Using hybrid analysis to improve measurements
WCET analysis process
9
Obtain HWM (High Water
Mark) – longest observed
execution time
Tool Qualification Issues
According to DO-330, tool qualification is required if: • The tool could fail to detect an existing error
• The tool's output could not be verified by another activity
• Processes are eliminated, reduced or automated by the tool
The "applicant" needs to provide: • PSAC
• TOR
• TAS
• TVR/TOVVR
• TQP
DO-330 process can be used to provide Qualification Kits for
both DO178-B and DO178-C… but….
13
Structural Code Coverage
Measurement Notes
Function
coverage
Each function has been called at least once
Not required by DO-178B/C
Call coverage Each function has been called at least once, and each
different function call has been encountered at least
once
Not required by DO-178B/C
Statement
coverage
Each statement in the code has been encountered at
least once
Required for DO-178B/C Level A,
B, C
Decision
coverage
Each decision in the code has evaluated true at least
once and evaluated false at least once, and each
function entry and exit point has been encountered at
least once
Required for DO-178B/C level A,
B
Condition
coverage
Each condition in the code has evaluated true at least
once and evaluated false at least once
Not required by DO-178B/C
Modified
Condition/
Decision
Coverage
Decision coverage plus each condition has been shown
to independently affect the outcome of its enclosing
decision
Required for DO-178B/C level A
MC/DC
Requirements based
testing
“flip” a condition and it
should affect the outcome
of the decision
Very hard to test from
requirements
Hard to define the “missing”
test vectors
Decisions
In “if”, “while”, …
In assignments
A= B && C;
In function calls
F(B&&C);
…
Case study 1
The problem On-target code coverage
Avionics display system
500 KLOC of Ada code
Tests on rig + host
Previous method required 15 builds due to overheads of instrumentation
6 days to run full suite x 15 = a lot of time
The solution Highly optimised target integration using RapiCover
Only one build, overall overhead below 15%.
Merging evidence for coverage from host build and on-target build
The benefit Reduce time and cost to perform structural code coverage
Code coverage performed at every single test build
(as opposed to once at the end)
Case study 2
The problem WCET analysis of engine management system
Large Ada code base
Customer own processor (new version of the CPU)
Internal static analysis tool could not be adapted to new processor
The solution Using built-in logic analyser interface to provide very low overhead tracing
mechanism
Develop a hardware tracing solution
Using RapiTime for detailed timing of blocks of code as part of “unit tests”.
Using RapiTime to compute WCET analytically using timings from unit tests
The benefit WCET analysis produced as a “side effect” of unit testing
Tool Qualification for DO178B.
Case study 3
The problem Code coverage on-target
Ethernet switch on aircraft. Mixed C/Ada code base.
Code coverage during “power cycles”.
Manual effort to review “coverage holes”
The solution Lightweight integration of RapiCover
Using LiveMaps recorded using RTBx
Using Justification workflow for addressing coverage holes
The benefits Fully automated structural code coverage analysis from on-target tests
Justifications used once, reused automatically for each build
Elimination of manual analysis of power-cycle code
Elimination of manual code reviews
Summary
Verification activities to satisfy DO178-B/C
Timing
Multiple definitions of what WCET is!.
Analytical WCET possible only when supported by design decisions
Structural code coverage
Very well understood
Future challenges, more demanding evidence from certification authorities.
More emphasis by certification authorities for evidence from tests “on-target”