Value-based Empirical Study Value-based Empirical Study for IV&Vfor IV&V

ISERN2005ISERN2005Masa Katahira, JAXAMasa Katahira, JAXA

Daniel Port, Univ. of HawaiiDaniel Port, Univ. of Hawaii

BackgroundBackground IV&V is a common method applied to safety-critical IV&V is a common method applied to safety-critical

software in order to gain quality and safety. NASA, software in order to gain quality and safety. NASA, ESA, and JAXA have started the study of strategic ESA, and JAXA have started the study of strategic IV&V together.IV&V together.

Main issue: strategic planning of IV&V activitiesMain issue: strategic planning of IV&V activities• Balancing safety vs. costBalancing safety vs. cost

Generating an Optimal Generating an Optimal StrategyStrategy

What is an optimal strategy with respect What is an optimal strategy with respect to cost?to cost?

1.1. For each attribute, pick technique that For each attribute, pick technique that gives maximum benefit gives maximum benefit

2.2. Order these pairs from max cost-benefit Order these pairs from max cost-benefit (RE reduction / cost) to lowest(RE reduction / cost) to lowest

Benefit(Ai,Tj) = before i after i j i jRE (A ) RE (A ,L ) RRCost(A ,L )

k N

after i J(i) i J(i) before i,Ji 1 i k 1

min RE (A ,T ) RRCost(A ,T ) RE (A )

RRL(Ai, TJ(i)) = ( REafter(Ai,TJ(i)) - REbefore(Ai)) / RRCost(Ai,TJ(i))

IV&V Assessment Strategies

0

2000

4000

6000

8000

10000

12000

14000

16000

18000

0 100 200 300 400 500 600

Cost

RE

highest CB

arbitrary

lowest cost

highest RE first, lowestcosthighest RE, max RE drop

Comparison of Strategies

strategic

Important ProblemsImportant Problems How much is enough IV&V to perform?How much is enough IV&V to perform? How can we make best use of IV&V efforts with How can we make best use of IV&V efforts with

limited resources (budget, schedule, project limited resources (budget, schedule, project constraints)?constraints)?

Must provide rationale for plans and budgetsMust provide rationale for plans and budgets Explain in a tangible way why IV&V is important Explain in a tangible way why IV&V is important

and its benefits to managers, customers, and and its benefits to managers, customers, and developersdevelopers

How can we select most cost-effective techniques How can we select most cost-effective techniques from hundreds of IV&V techniques?from hundreds of IV&V techniques?

Which organization should perform which Which organization should perform which techniques?techniques?

When should we stop IV&V?When should we stop IV&V? How can you assess IV&V Return On Investment?How can you assess IV&V Return On Investment?

IV&V Is clearly not Value neutral!

Current Case StudyCurrent Case Study System Characteristics Budget Limitation Environment

It must be addressed on evidence and past results, not just expert opinion and “best guess.”

We must incorporate a continuous improvement and control program based comparing expected results with actual results.

IV&V activity

Objectives Attributes Techniques

MODEL

Strategic Planning

Reduced RiskEffectivesCost

Real Issuesafter deliveryProblems

IncidentsAccidents

Planning is byexpert opinionand best guess

Current StudyCurrent Study

Collaboration Study has been just started Collaboration Study has been just started • Daniel Port, University of HawaiiDaniel Port, University of Hawaii• Masa Katahira, JAXA Masa Katahira, JAXA • Haruka Nakao, JAMSSHaruka Nakao, JAMSS

Expectation of ISERNersExpectation of ISERNers• Opinions for value based IV&VOpinions for value based IV&V• Suggestion for gathering the empirical data on IV&VSuggestion for gathering the empirical data on IV&V

We will show our data collection format if requestedWe will show our data collection format if requested

Back Up SlidesBack Up Slides

JAXA IV&V techniques structureJAXA IV&V techniques structure

Main consideration is built in the strategic planning Main consideration is built in the strategic planning mechanism which need to indicate the attributes having mechanism which need to indicate the attributes having system parameters (numeric numbers) and techniques system parameters (numeric numbers) and techniques having IV&V conditions (numeric numbers)having IV&V conditions (numeric numbers)

Target ObjectAttribute

Attribute

ObjectAttribute

Attribute

Technique

Technique

Technique

System Parameter

s

IV&V conditions

Ｓｙｓｔｅｍ　ＣｈａｒａｃｔＳｙｓｔｅｍ　Ｃｈａｒａｃｔｅｒｉｓｔｉｃｓｅｒｉｓｔｉｃｓ

13 characteristics13 characteristics (1) Autonomous Control

(2) Fault Tolerance

(3) Functional Role

(4) Dealing Data

(5) Relation of Hazard

(6) Hazard Control Function

(7) Execution Architecture

(8) Sub Architecture

(9) Time Criticality

(10) Number of Component

(11) Life Cycle

(12) Reuse

(13) Methodology

Functional

Architecture

Development Process

Potential risk for each Potential risk for each IV&V IV&V attributeattribute

(1) Autonomous Control

(2) Fault Tolerance

(3) Functional Role

(4) Dealing Data

(5) Relation of Hazard

(6) Hazard Control Function(7) Execution Architecture

(8) Sub Architecture

(9) Time Criticality

(10) Number of Component(11) Life Cycle

(12) Reuse

(13) Methodology

+ Autonomous O Not Autonomous

O 0FT + 1FT O 2FT

+ Central Controller + Device Controller+ Data Relay

+ System Data + Mission Data

O Cause of Hazard + Hazard ControllerO Indirectly Hazard Control

O Must Work Function + Must Not Work FunctionO Single Task + Multi Task

+ Sequence O Event Driven

O Hard Real-time + Soft Real-time

O Many + Not Many

+ In Development O In Operation

O Exist + None

+ Water Fall O Spiral

Characteristics parameter

+: Positive 0: Negative

11

00

11

00

00

00

00

00

00

00

00

11

(A5) Completeness of state transition

Total Potential risk if the attribute is not assessed :

Potential risk VALUE

3

IV&V conditionsIV&V conditions

8 IV&V 8 IV&V conditionsconditions

(1) A kind of target document

(2) Development phase

(3) Period for IV&V

(4) Knowledge of System or Operation(5) Developer Support

(6) Usability of source code

(7) Usability of electrical document(8) Size of document / LOC

Efficiency of Efficiency of IV&V IV&V techniquestechniquesEx. System A

Efficiency of Completeness Analysis(T6) for Completeness of state transition(T6)

(1) A kind of target document + Natural Language+ Flow ChartO Source Code

(2) Development phase + Requirement+ DesignO ManufactureO Test

(3) Period of time for IV&V + EnoughO Not Enough

(4) Knowledge of System or Operation

+ EnoughO Not Enough

(5) Support by Developer + EnoughO Not Enough

(6) Usability of source code + OKO NG

(7) Usability of e-document + OKO NG

(8) Size of document / LOC + MuchO Not Much

11

00

11

00

00

00

11

00

3