NASA OSMA SAS ‘03

SAS03/FaultPrediction 1

NASA OSMA SAS ‘03

Software Requirements Analysis As Fault Predictor

Dolores R. WallaceSRS Technologies

Software Assurance Technology Centerhttp://satc.gsfc.nasa.gov

[email protected]

William M. WilsonSRS Technologies/SATC

[email protected]


Software Requirements Analysis As Fault Predictor

Presentation Outline1. Research Rationale & Objective2. Requested Project Data3. Planned Approach4. Data Acquisition & Analysis Obstacles5. Results & Findings6. Conclusions7. Recommendations


Research Rationale & Objective

Rationale• The earlier in the lifecycle that software faults can be

found the less expensive it is to correct them.• The earliest opportunity to preclude software faults is

during the development of the system's requirements.

Objective• To determine if software requirements analysis results can

be used as a predictor of software faults.


Requested Project Data

• Requirements Specification– Accessible in electronic media in MS Word format

for use with the ARM tool– Conforming to NASA-STD-2100– Specification statements identified by hierarchical

numbers


Requested Project Data – Cont.

• Verification Test Matrix– Requirements to Test or vice versa

- OR -• Traceability Matrix

– Requirements to Design Elements– Design Element To Software Component

- AND -• Test Plans

– Tests vs. Software Components.


Requested Project Data – Cont.

• Failure Reports/Data - Preferably in DDTS format– Including: test id, module id, CSCI id, release/build id,

suspected cause of failure and resolution (e.g., actual cause of failure - what was fixed)

• Configuration Control Reports– Baseline item changed -e.g., Requirement, Design,

Code– Reason/Source of change – e.g., RID, Failure Report,

PR, etc.


Planned Approach

1. Use ARM tool to analyze projects’ software requirements.

2. Collected projects’ failure data in a DDTS database.

3. Organize the data to represent same components for each project.

4. Conduct correlation analysis.5. Study results of step 4 to prove or disprove

hypothesis.


ARM TOOL Reports

Identifies and Counts:• Size of Requirements Document

– Lines of Text– Numbered Paragraphs– Number of Specification Statements– Number of Unique Specification Subjects

• Number of Specifications Containing– Weak, Optional and/or Incomplete Phrases.– Compound and/or Complex Statements.

• Requirements Document Structure– Depth and Distribution of Specifications


Data Acquisition & Analysis Obstacles

• Common Problems– No commonality or standardization of data or formats

across project.– Current projects are reluctant to provide data to outside

organizations.– Data from completed projects is incomplete, not in a

usable form or is not accessible.


Data Acquisition & Analysis Obstacles Cont.

• Specific Problems– No project staff available to guide through the mass of

documents in the collection– Documents locked on a WEB site– Documents in “.pdf” format– Variation of format and media within project– Complete set of data for a specific Build/Release not

available– Data held by support contractor and released only on a

“Project Need-To-Know” basis


Results & Findings

• Data provided by four projects– Projects A and B

• Major subsystems of operational information processing system

– Projects C and D• Flight instrumentation packages


Results & Findings Cont

Summary Of Project's Data

PROJECT Tot

al S

peci

ficat

ions

Wea

k Sp

ecifi

catio

ns

Inco

mpl

ete

Com

poun

d

Max

Dep

th

Spec

s at

Low

est L

evel

Tot

al P

robl

ems/

Cha

nge

Req

uest

s

Req

uire

men

ts

Des

ign

Cod

e

Tes

t

Doc

umen

tatio

n

Enh

ance

men

t

Oth

er

Non

e

A 849 256 10 241 5 226 3066* 0 183 2013 9 22 0 267 572B 226 47 13 39 5 38 1132** C 356 75 0 100 6 117 323*** 12 34 91 3 5 69 102 7D # 121##

* System Failure Reports** Details Not Yet Available*** Software Change Requests# System Detailed Design Spec not Requirement## Unformatted Text Problem reports


Conclusions

• Data from many more projects is required to improve the possibility of finding a number of data sets with sufficient commonality to support analysis.

• Analysis approach needs to be expanded to include determining if failures attributed to documentation and design flaws are in reality related to deficiencies in requirements.


Recommendations

Advocate and support:• The use of NASA-STD-2100 documentation

standard• The use of a format to report problems and

failures that is compatible with DDTS• The creation and use of centralized Center

repositories for data from completed projects.


Summary

• Lessons– Getting data requires strong, interested Civil

servant advocate– Analyzing data requires tedious effort, even

with automated tools• Future

– Analyze latest set of data– Attempt correlations for the 4 data sets

NASA OSMA SAS ‘03

Documents

Transcript of NASA OSMA SAS ‘03