Click to edit Master title style Click to edit Master subtitle style A PRACTICAL LOOK AT UNCERTAINTY...
-
Upload
dinah-campbell -
Category
Documents
-
view
216 -
download
0
description
Transcript of Click to edit Master title style Click to edit Master subtitle style A PRACTICAL LOOK AT UNCERTAINTY...
Click to edit Master title style
Click to edit Master subtitle style
A PRACTICAL LOOK AT UNCERTAINTY MODELING
Steve UnwinRisk & Decision Sciences Group
March 7, 2006
2
"The fundamental cause of trouble in the world today is that the stupid are cock-sure while the intelligent are full
of doubt.“
Bertrand Russell
3
Illuminating the Path• Visual Analytics Agenda - Recommendations
– Rec. 4.10: Develop new methods and technologies for capturing and representing information quality and uncertainty
– Rec. 4.11: Determine the applicability of confidence assessment in the identification, representation, aggregation, and communication of uncertainties in both the information and the analytical methods used in their assessment.
– Summary Rec: Develop methods and principles for representing data quality, reliability, and certainty measures throughout the data transformation and analysis process
4
Uncertainty Analysis as Resource to Visual Analytics
• VA Agenda
– Develop new methods and technologies for capturing and representing information quality and uncertainty
– Determine the applicability of confidence assessment in the identification, representation, aggregation, and communication of uncertainties in both the information and the analytical methods used in their assessment.
– Develop methods and principles for representing data quality, reliability, and certainty measures throughout the data transformation and analysis process
• UA Insight
– Probabilistic techniques• Elicitation methods• Aggregation methods• Information-theoretic approaches
– Nonprobabilistic alternatives• Dempster-Shafer• Possibility theory
– Uncertainty propagation techniques• Analytic• Numerical
– Risk communication• Risk representation• Decision-analysis methods
5
MEASURING UNCERTAINTY
CLASSICALMETHODS BAYESIAN
METHODS
NON-PROBABILISTIC
METHODS
6
Classical Statistics
• Focus on Aleatory Uncertainty– random variation inherent in the system
• Sampling produces confidence intervals• Need a sampling model
– Generally unavailable for many real-world complex situations
• Confidence intervals are not probability intervals– Propagation difficulties in even the simplest models
7
Bayesianism
• de Finetti, Ramsey, Savage (1920s-50s)• Subjectivism – Epistemic Probabilities
– Probability as a degree of belief• Classicists are coin tossers• Bayesians are believers
– What is the basis for forming probability?• “ Probabilities do not exist”
– Bruno de Finetti
8
Problems with Bayesianism• Because probabilities don’t exist, they have to be
created– but how?
• Bayes’ Theorem• Subjectivity is explicit
– judgment of evidence• Do probabilities really reflect the way we conceive
belief?– is probability theory a good theory of evidence?– what are the options?
9
One Option:Dempster-Shafer Theory
• Withholding belief distinct from disbelief• Seahawks or Steelers will win?• Set of possibilities: {sea, steel}• Probability theory:
– Weight of evidence attached to each exclusive possibility– p(sea), p(steel)
• D-S theory:– Weight of evidence attached to each subset– m(Ø), m(sea), m(steel), m(sea U steel)
• Allows: m(sea U steel) = 1, all other m=0– A compelling ignorance
10
Support and Plausibility
• Probability replaced by two belief measures:– Each calculated from weights of evidence– bel(sea) is the support for proposition ‘sea’– pl(sea) is the plausibility of ‘sea’– bel(sea) ≤ pl(sea)– Upper and lower “probabilities”
• Complete ignorance• SDU: bel(sea) = 0, pl(sea) = 1, i.e., complete
ignorance on the matter of proposition ‘sea’
11
Complementary Cumulative Belief Functions
ESD Sensor System On-Demand Failure Rate
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1.0E-04 1.0E-03 1.0E-02 1.0E-01 1.0E+00
Failure Rate per Demand
Belie
f Met
ric
Complementary Cumulative Support/ Belief
Complementary Cumulative Plausibility
Complementary Cumulative Probability
12
Possibility Theory
• Genesis in fuzzy sets• Possibility is an uncertainty measure that
mirrors the fuzzy set notion of imprecision
13
The Set of Tall Men
0
0.2
0.4
0.6
0.8
1
5' 8" 5' 9" 5' 10" 5' 11" 6' 0" 6' 1" 6' 2"
Height
Mem
bers
hip
to S
et TallVery Tall
m(h)
m'(h) = m2(h)
14
Possibility Theory
• 2-tier belief: possibility and necessity• nec(X) ≤ pos(X)• Distinctive combinatorial logic
– nec(X^Y) = min[nec(X), nec(Y)]– pos(XvY) = max[pos(X), pos(Y)]
• No conceptual connection to probability– although probability/possibility can co-exist
15
Possibilistic Interpretationof Intelligence Statements (Heuer)
Probability
Possibility
Chances are slight
Little chanceBetter than even
Highly likely
16
Experience with Nonprobabilistic Methods
• Not all good:– Standardization of belief metrics?– Treatment of dependences?– Treatment of conflicting evidence?– Computational demands?– Interpretation of results?– Incorporation into decision process?
• Plan B: Confront the problems with probabilistic methods
17
Principled Basis for Probability Formulation
• Analysts uncomfortable producing probabilities– justified discomfort
• Alternative:– Produce defensible basis for probability formulation based on
nonprobabilistic judgment• Maximize expression of uncertainty subject to judged
constraints• Borrow uncertainty metrics from:
– statistical mechanics– information theory
• Entropy = -∑i pi.ln pi – discrete probability distribution, pi
18
Application of Information-Theoretic Methods
• Two USNRC programs:– QUEST- SNL
• Quantitative uncertainty evaluation of source terms
– QUASAR – BNL• Quantitative uncertainty analysis of severe accident releases
• Both studies used the same form of input to the same deterministic models– non-probabilistic input
• expert-generated input parameter uncertainty ranges
• QUEST: Bounding analysis
• QUASAR: Information Theory used to generate probability distributions from bounds
• Probabilistic analysis internal to methodology – no elicitation of probability
19
Information Theory and the Preservation of Uncertainty
Uncertainty Bands
1.00E-04 1.00E-03 1.00E-02 1.00E-01 1.00E+00
QUEST
QUASAR
QUEST
QUASARI-131
Cs-137
Release Fraction
20
Uncertainty Analysis as Resource to Visual Analytics
• VA Agenda
– Develop new methods and technologies for capturing and representing information quality and uncertainty
– Determine the applicability of confidence assessment in the identification, representation, aggregation, and communication of uncertainties in both the information and the analytical methods used in their assessment.
– Develop methods and principles for representing data quality, reliability, and certainty measures throughout the data transformation and analysis process
• UA Insight
– Probabilistic techniques• Elicitation methods• Aggregation methods• Information-theoretic approaches
– Nonprobabilistic alternatives• Dempster-Shafer• Possibility theory
– Uncertainty propagation techniques• Analytic• Numerical
– Risk communication• Risk representation• Decision-analysis methods
21
Merit Criteriafor Uncertainty Analysis in Intel
• Makes the analyst’s job easier• Represents strength of evidence intuitively• Can reflect dissonant evidence• Appropriately propagates uncertainty from analyst
to decision-maker• Characterizes output uncertainty in a standardized
and interpretable way• Computationally tractable• Promotes insight