SOUND METHODS and EFFECTIVE TOOLSfor ENGINEERING MODELING and ANALYSIS

_________________by David Coppit, College of William and Mary,

and Kevin J. Sullivan, University of Virginia

Proceedings of the 25th International Conference on Software EngineeringPortland, Oregon - May 3-10, 2003

Presentation by Bryan E. Bloss - University of Central Florida, Nov. 2003

1. INTRODUCTION

Designing Software Tools for Formal Design________________________________________________________________________________________

Engineering modeling & analysis methodsare based on modeling languages fordescribing systems, with semantics formapping expressions (models) toestimates of system properties (results).

To be safe and effective, a modeling method requiresa language with a validated semantics; feature-rich,easy-to-use, dependable tools; and low engineering costs.

Hampered by shortcomings in software engineering & languages, today we lack adequate means to develop such methods.

Two Sub-Problems Addressed in this Paper:

- Finding a cost-effective way to ensure

semantic soundness of a complex method

- Using Package-Oriented Programming (POP) to

produce easy-to-use, functionally-rich tools from

available software packages (such as MS Office)

Results: A package-based tool “Galileo” is evaluated

favorably by NASA engineers, and development of

“Nova”, a similar tool based on a formal semantics,

proves the cost-effectiveness of a combined approach

2. FORMAL SEMANTICS & DEPENDABILITY

Why are they important?_____________________________________________________________________________________________________

Specification is more fundamental than implementation.

Without a formal specification:

- Validation is difficult

- No basis for a definitive user reference document

- Programmers are left to make uninformed semantic

decisions; unable to thoroughly test correct functioning.

Tools used in the design of safety critical systems should be treated as critical engineering components.

Our inability to develop low-cost, easy-to-use tools can thus be seen as a positive safety mechanism, but far from ideal.

Safety Example: 1996 alert from U.S. Nuclear Regulatory Commission warned of significant errors in several tools which had been adopted for use in nuclear reactor design & analysis

Another Example (not in paper): Crater analysis tool, used inappropriately during flight STS-107 to analyze foam damage

3. APPROACH & BASIC RESULTS

Developing the Galileo Tool for DFT Analysis___________________________________________________________________________________________

Observation: Most applications devote less than 10% of their code to the core function of the system!

90% is devoted to superstructure-- support functions such as text & graphical editing, data validation, etc.

Package-Oriented Programming (POP) is intended to save time in creating superstructure; frees more resources for the critical design activity: applying formal methods to define & validate the syntax and semantics of the modeling language

The Application: Dynamic Fault Trees (DFT)

Graphcal representation of everyconceivable sequence of eventsthat could cause a system to fail.

Each leaf is a basic event; internalgates define relationships leadingto system failures at upper levels.

Static trees model how eventcombinations lead to failures;Dynamic trees are order-sensitive.

(Illustrations fromCAIB Report)

3./4. CASE STUDY: RELIABILITY ENGINEERING

The Problems With Current DFT Languages___________________________________________________________________________________________

During development of Galileo tool, a non-trivial error was found in the underlying DFT language, DIFtree, where probability of a masked (hidden) failure wasn’t correctly computed

Also, DIFtree’s informal specification had left ambiguities on how to handle special cases; prior software implementations answered these questions inconsistently in different parts of the program.

And formal validation was time-consuming, due to lack of automation in available syntax & theorem-prover tools, and slow run-time perfomance

Worse, the theorem-prover tool required too much user expertise; guidance often needed from the tool’s author

5. The NOVA DFT Tool

Like Galileo, uses POP components from MS Office for fault tree editing:

- Word for text editing

- Visio graphical editor (enhanced for DFT modeling constructs)

- Excel for computational results

Will allow even more emphasis on formalization & validation than in Galileo

6. END-USER EVALUATION OF GALILEO

New version of Galileo tool funded by NASA Langley Research Center, to support new modeling & analysis constructs and be usable in practice

Featured in three workshops:

1st- Managers & engineers from several NASA divisions

2nd- Space Station engineers only

3rd- Space Shuttle engineers only

A short survey (34 questions) and an in-depth survey (77 questions) were offered to engineers, to evaluate user perceptions of usability & features

Feedback indicated that usability was same or better than other tools!

Also confirmed that dependability is crucial; a formal specificaltion of the modeling language was second only to a comprehensive test suite as a means for increasing trust.

____Evaluation of this Article ____

Strengths: A well-informed overview of the authors’ experience with developing a new software toolset for practical engineering, while emphasizing formal validation methods. Many implications for the design of other reliability-critical software applications.

Weaknesses: Little information on costs saved by POP method; little detail on formal proving methods. Also, the more advanced NOVA tool had not been user-tested at time of publication, so final results aren’t known.

____Questions? ____