© 2013 Carnegie Mellon University Measuring Assurance Case Confidence using Baconian Probabilities...

© 2013 Carnegie Mellon University

Measuring Assurance Case Confidence usingBaconian Probabilities

Charles B. WeinstockJohn B. GoodenoughAri Z. KleinMay 19, 2013

2

Measuring Assurance Case ConfidenceWeinstock/Goodenough/Klein, May 2013


Copyright 2013Carnegie Mellon University and IEEEThis material is based upon work funded and supported by the Department of Defense under Contract No. FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center.NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.This material has been approved for public release and unlimited distribution.This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software Engineering Institute at [email protected]

mailto:[email protected]

3



The Problem

How confident in C1? Why?What does it mean to have confidence?What could be done to improve confidence? Why?

Ev2

Evidence

Ev3

Evidence

Ev1

Evidence

C1

The system is safe

C2

Hazard A has been eliminated

C3 Hazard B has

been eliminated

4



How is Confidence in a Hypothesis Increased?

A classic philosophical problem: •Determining the basis for belief in a hypothesis when it is impossible to

examine every possible circumstance covered by the hypothesis

Use Induction•Enumerative: number of confirming instances (Pascalian)

•Eliminative: variety of reasons for doubt (Baconian)

•Measuring support for a hypothesis/claim–Enumerative: support increases with number of confirmations–Eliminative: support increases with the number of excluded alternative

explanations, i.e., by eliminating reasons for doubting the claim•Defeaters: reasons for doubting a claim

5



A Baconian Theory of AC Confidence

Confidence is the degree of beliefWe grade “degree of belief” in terms of the number of eliminated defeaters (reasons for doubt)• i out of n reasons for doubt (i/n)

• 0/n – no confidence

• n/n – no remaining reasons for doubt

• 8/10 – residual doubt

Fundamental principle: Build confidence by eliminating reasons to doubt the validity of: •Claims (look for counter-examples and why they can’t occur)

• Evidence (look for reasons the evidence might be invalid and show those conditions do not hold)

• Inference rules (look for conditions under which the rule is not valid and why those conditions do not hold)

As reasons for doubt are eliminated, confidence grows (eliminative induction)

7



Citigroup Tower – New York

• The design passed all reviews• The wind tunnel (and other) tests

were successful• All parties involved were

convinced the building was safe• But…

• The design accounted for straight on wind, not quartering wind

• Still had adequate safety margins

• The design accounted for straight on wind, not quartering wind

• Still had adequate safety margins• But construction replaced welded

joints with bolted joints• The margin of safety was lost

C1.1

Citigroup Toweris safe

R2.1Unless thereis a design

error

UC3.2

Unless salient attributesof the design were

overlooked

IR2.2

If the design of thesystem is shownto be safe, then

the system is safe

UM4.1

But the model of thebuilding tested doesnot match the actual

building

Ev3.1Wind tunnel

testsshowing safe

design

UM4.2

But not all real-world windconditions were

simulated

C1.1

Citigroup Toweris safe

R2.1Unless thereis a design

error

IR2.2

If the design of thesystem is shownto be safe, then

the system is safe

UM4.1

But the model of thebuilding tested doesnot match the actual

building

UM4.2

But not all real-world windconditions were

simulated

Ev3.1Wind tunnel

testsshowing safe

design

UC3.2

Unless salient attributesof the design were

overlooked

8



Concerns with the Baconian Approach

•What if a relevant defeater has not been identified?•What if a defeater cannot be completely eliminated?•Surely not all defeaters are of equal importance. How is this handled?•Baconian probability seems rather weak in contrast to Bayesian or Pascalian

probability. What is being gained or lost?•The potential number of defeaters seems incredibly large for a real system. Is

this approach practical?

9



Unidentified Defeaters

Issue: Does the failure to identify a relevant defeater lead to an inflated sense of confidence?•The inability to identify some defeaters is itself a reason for doubt that needs

to be recognized in a case. •An assessment process for reviewing a case will need to take the possibility

of missing defeaters into account.•A similar problem exists with the use of assurance cases. How can you be

sure that your hazard analysis has covered all potential hazards?•We always admit the possibility that additional defeaters will be identified in

the future (this is what defeasible reasoning is all about).–When we say that we have “complete” confidence in a claim (i.e., that the

claim has Baconian probability n|n) we understand that this only reflects what we knew at a particular point in time.

The concepts of eliminative induction and defeasible reasoning help in developing sound and complete arguments.

10



Partially Eliminated Defeaters

Issue: How are defeaters that cannot be completely eliminated dealt with?•An argument should state a claim that, if true, serves to completely eliminate

an associated defeater.•There may be residual doubts about the truth of such a claim, in which case,

the same doubts apply to the defeater’s elimination– as an argument is refined, one eventually develops defeaters that are

“obviously” eliminated by associated evidence • If it is impossible to eliminate some lowest level defeaters, then the

associated doubt leads to incomplete elimination of a higher level defeater (and the claim that it defeats.)

A goal in developing a convincing argument is to formulate low level defeaters that can be eliminated by appropriate evidence.

11



Defeaters of Different (Relative) Importance

Issue: It seems reasonable that eliminating some defeaters would lead to higher levels of confidence than eliminating others (of lesser importance).• Independent of relative defeater importance, 1|2 (some doubts eliminated)

always represents more confidence than 0|2 (no doubts eliminated), and 2|2 (all doubts eliminated) always represents complete confidence

•Even though the hazards that are represented in a safety case have different likelihoods and impacts relative to one another, they all must be demonstrably mitigated in order to establish sufficient confidence that the system is safe.– Assessing the relative importance of hazards is not practically profitable.

•A system developer would not use the elimination of low impact defeaters to justify an increase in confidence any more than he would represent minimally unlikely and impactful safety hazards in a safety case.

12



Why Baconian Probability?

Issue: What is being gained and lost (over Pascalian or Bayesian probability) with this approach?•With eliminative induction we learn something concrete about why a system

works, and with enumerative induction, we may learn something statistical about the system.

•The Baconian approach allows us to articulate both the reasons why a system can fail and the reasons why the argument can be defective–Uneliminated defeaters serve to focus additional assurance efforts.

• the Baconian approach allows evaluation of a system prior to its operational use

•The Baconian approach avoids confirmation bias

The Baconian, Pascalian, and Bayesian approaches inform each other.•Make Pascalian claims about system reliability and then elucidate the

possibilities that would make you doubt the validity of the claim.•Take repeated samples to understand the operational reliability of the system.

13



Scalability and Practicality of the Approach

Issue: is this approach practical?•The number of defeaters relevant to an argument is quite large for a real

system.–But the amount of relevant argument and evidence for a real system is

inherently quite large.•We believe that the Baconian approach allows one to develop a more

thorough and cost-effective basis for developing confidence in system behavior than current methods.

•We hope that this approach will lead to more effective and focused assurance efforts.

This all remains to be seen. Our initial interactions with systems developers have been promising.

14



Summary

The identification of defeaters and how they are eliminated provides framework for assessing confidenceThe framework provides useful ways of thinking about assurance case deficiencies• Identification of deficiencies is the first step towards their mitigation•The end result is a stronger assurance case

The approach is still under development. We welcome your ideas

15



Contact Information

Charles B. WeinstockSenior Member of the Technical StaffSoftware Solutions DivisionTelephone: +1 412-268-7719Email: [email protected]

U.S. MailSoftware Engineering Institute4500 Fifth AvenuePittsburgh, PA 15213-2612USA

John B. GoodenoughSEI Fellow (retired)Software Solutions DivisionTelephone: +1 412-268-6391Email: [email protected]

Ari Z. KleinPh.D. Candidate – RhetoricSoftware Solutions DivisionTelephone: +1 412-268-7700Email: [email protected]

mailto:[email protected]

© 2013 Carnegie Mellon University Measuring Assurance Case Confidence using Baconian Probabilities...

Documents

Transcript of © 2013 Carnegie Mellon University Measuring Assurance Case Confidence using Baconian Probabilities...