Assurance techniques for code generators

19
Assurance techniques for code generators Ewen Denney USRA/RIACS, NASA Ames Bernd Fischer ECS, U Southampton

description

Assurance techniques for code generators. Ewen Denney USRA/RIACS, NASA Ames Bernd Fischer ECS, U Southampton. Assurance problem. Safety/mission-critical software requires assurance that it meets a certain level of “quality” What are the issues in assuring automatically generated code? - PowerPoint PPT Presentation

Transcript of Assurance techniques for code generators

Page 1: Assurance techniques  for code generators

Assurance techniques for code generators

Ewen DenneyUSRA/RIACS, NASA Ames

Bernd FischerECS, U Southampton

Page 2: Assurance techniques  for code generators

Assurance problem

• Safety/mission-critical software requires assurance that it meets a certain level of “quality”

• What are the issues in assuring automatically generated code?– Different forms of assurance– Different assurance techniques– Diverse generator paradigms

Page 3: Assurance techniques  for code generators

Forms of assurance

What exactly might we need to assure?

• Compliance with requirements

• Compliance with spec/model

• Certification standards

• Coding standards

• Absence of run-time errors

• Traceability

• Appropriate documentation

Minimize “automation surprises”

Correctness

Reliability

Legibility

Page 4: Assurance techniques  for code generators

Code generators in practicePractitioner survey carried out in March 2006

(Code Generators in Safety-critical Applications, J. Schumann, E. Denney); 23 responses from NASA and industry.

• How are ACGs used for safety-critical applications at NASA and in industry?

• Which are the primary application areas and domains?

• Which tools are used?• Challenges, benefits and problems?• How could ACGs be extended to be more

useful in safety-critical applications?

Page 5: Assurance techniques  for code generators

Tools and languages

The Big Three:

• Real-Time Workshop

• MatrixX

• SCADE

Page 6: Assurance techniques  for code generators

Domains and criticality levels

• Principle domains:– control– modeling/simulation

• Many highly critical applications

• ACG used for – production code (74%)– prototyping (52%)– simulation (48%)– testing (30%)– glue/interface code (30%)

Page 7: Assurance techniques  for code generators

System components

Page 8: Assurance techniques  for code generators

Weaknesses

• Steep Learning Curve– applicable problems, features, correct usage,

architecture, implied methodology, semantic ambiguities, …

– substantial impact on development process

• ACG customization– necessary in 1/3 of cases– often (2/3) done by tool vendor

• ACG bugs– in 2/3 of applications, bugs were found in ACG

Page 9: Assurance techniques  for code generators

Qualification• A code generator is qualified

– with respect to a given standard– for a given project

if there is sufficient evidence about the generator itself so that V&V need not be carried out on the generated code to certify it

• Must be done for every project, version• Can obtain verification credit• Generators are rarely qualified• Examples: ASCET-SE (IEC 61508), SCADE,

VAPS (DO-178B)

Page 10: Assurance techniques  for code generators

Certification and V&V

• Auto-generated code must be certified for safety-critical use

• Techniques used:– testing (90%)– static analysis (58%)– simulation (52%)– manual review (48%)

• No formal verification

• No review of generator code

Page 11: Assurance techniques  for code generators

Safety properties

Page 12: Assurance techniques  for code generators

Generator features

Page 13: Assurance techniques  for code generators

Domain-specific analyses

Mostly numeric issues:

• stability (root locus, Lyapunov)

• robustness

• convergence

• transience

Some domain-specific design rules:

• “forbidden” constructs

• block structure

Page 14: Assurance techniques  for code generators

Documentation

• Design information

• Code derivation

• Configuration management information(to “replay” generation)

• Safety information

• Tracing information

• Interface definitions, requirements

• User manuals

• Installation information

Should be customizable

Page 15: Assurance techniques  for code generators

Traceability

• Most important: model code

• Secondary: code V&V artifacts

Page 16: Assurance techniques  for code generators

Tool integration

Also

• workflow and process tools

• tools for integrating legacy code

Page 17: Assurance techniques  for code generators

Survey summary

• Integrated modeling, analysis, and simulation tools are most common in control domain

• In-house extensions common for modeling and verification issues

• Natural synergy between code generation and certification activities– perceived but not realized– autocode often treated like manual code

• Iterative customization of generator should be seen as integral part of development process

Page 18: Assurance techniques  for code generators

Assurance techniques

• Testing the generator (qualification)– for all specs, blocks, configurations, backends, …

• Post factum verification / certification– verify / certify generated programs individually

• Correctness by construction– generator inherently guarantees certain

properties

• Documentation

• Traceability

Page 19: Assurance techniques  for code generators

Discussion questions

• What are the interesting assurance artifacts, properties, etc. in your target domains?

• What are suitable notions of documentation, traceability, development process?

• What assurance techniques have you tried?

• How is the generative knowledge represented (templates, transformation rules, etc.) and how can it be combined with assurance information?

• Can we apply Design for Verification (D4V) to generators?