Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV...
-
Upload
buddy-roberts -
Category
Documents
-
view
215 -
download
2
Transcript of Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV...
![Page 1: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/1.jpg)
Do Humans make Good Observers – and can they
Reliably Fuse Information?
Dr. Mark BedworthMV Concepts Ltd.
![Page 2: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/2.jpg)
What we will cover:
• The decision making process• The information fusion context• The reliability of the process• Where the pitfalls lie• How not to get caught out• Suggestions for next steps
![Page 3: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/3.jpg)
What we will not cover:• Systems design and architectures• Counter-piracy specifics• Inferencing frameworks• Tracking• Multi-class problems• Extensive mathematics• In fact… most of the detail!
![Page 4: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/4.jpg)
Our objectives:• Understanding of the context of data fusion
for decision making• Quantitative grasp of a few key theories• Appreciation of how to put the theory into
practice• Knowledge of where the gaps in theory
remain
![Page 5: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/5.jpg)
Warning
This presentation containsaudience participation
experiments
![Page 6: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/6.jpg)
Decision Making
• To make an informed decision:– Obtain data on the relevant factors– Reason within the domain context– Understand the possible outcomes– Have a method of implementation
![Page 7: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/7.jpg)
Boyd Cycle• This is captured more formally as a
fusion architecture:– Observe: acquire data– Orient: form perspective– Decide: determine course of action– Act: put into practice
• Also called OODA loop
![Page 8: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/8.jpg)
OODA loop
Decide
Observe
ActOrient
![Page 9: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/9.jpg)
Adversarial OODA Loops
Owninformation
Adversaryinformation
Decide
Observe
ActOrient
Decide
Observe
OrientAct
Physical world
![Page 10: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/10.jpg)
Winning the OODA Game
• To achieve dominance:– Make better decisions– In a more timely manner– And implement more effectively
![Page 11: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/11.jpg)
Dominance History• Action dominance (-A)
– Longer range, more destructive, more accurate weapons
• Observation dominance (O-)– Longer range, more robust, more accurate
sensors• Information dominance (-O-D-)
– More timely and relevant information with better support to the decision maker
![Page 12: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/12.jpg)
Information DominancePart One: Orientation
“Having acquired relevant data;to undertake reasoning about the data within the domain context to form aperspective of the current situation;so that an informed decision cansubsequently be made”
![Page 13: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/13.jpg)
A number of approaches• Fusion of hard decisions
– Majority rule– Weighted voting– Maximum a posteriori fusion– Behaviour knowledge space
• Fusion of soft decisions– Probability fusion
![Page 14: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/14.jpg)
Reasoning Frameworks• Boolean
– Truth and falsehood
• Fuzzy (Zadeh)– Vagueness
• Evidential (Dempster-Shafer)– Belief and ignorance
• Probabilistic (Bayesian)– Uncertainty
![Page 15: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/15.jpg)
Probability theory• 0 ≤ P(H) ≤ 1• if P(H)=1
then H is certain to occur• P(H) + P(~H) = 1
either H or not-H is certain to occur (negation rule)
• P(G,H) = P(G|H) P(H) = P(H|G) P(G)the joint probability is the conditional probability multiplied by the prior (conjunction rule)
![Page 16: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/16.jpg)
Bayes’ Theorem
Posteriorprobability
Likelihood Priorprobability
Marginallikelihood
)()()|(
)|(XP
HPHXPXHP
![Page 17: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/17.jpg)
Perspective Calculation
• Usually the marginal likelihood is awkward to compute– But is not needed since it is independent
of the hypothesis– Compute the products of the likelihoods
and priors; then normalise over hypotheses
![Page 18: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/18.jpg)
Human Fusion Experiment (1)
• A threat is present 5% of the time it is looked for
• Observers A and B both independently look for the threat
• Both report an absence of the threat with posterior probabilities 70% and 80%
• What is the fused probability that the threat is absent?
![Page 19: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/19.jpg)
Human Fusion Experiment (2)
• Threat absent ≡ the hypothesis (H)• P(~H) = 0.05• P(H) = 0.95
• P(H|XA) = 0.70
• P(H|XB) = 0.80
• P(H|XA,XB) = ?
![Page 20: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/20.jpg)
Human Fusion Experiment (3)No threat
H=1.00
PriorP(H)=0.95
Report BP(H|XB)=0.80
Report AP(H|XA)=0.70
![Page 21: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/21.jpg)
Conditional Independence• Assume the data to be conditionally independent
given the class:
• Note that this does not necessarily imply:
)|()|()|,( HBPHAPHBAP
)()(),( BPAPBAP
![Page 22: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/22.jpg)
Conditionally Independent
Sensor 1 measurement
Sen
sor
2 m
easu
rem
ent
![Page 23: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/23.jpg)
Conditionally independent
Sensor 1 measurement
Sen
sor
2 m
easu
rem
ent
![Page 24: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/24.jpg)
Not conditionally independent
Sensor 1 measurement
Sen
sor
2 m
easu
rem
ent
![Page 25: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/25.jpg)
Not conditionally independent
Sensor 1 measurement
Sen
sor
2 m
easu
rem
ent
![Page 26: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/26.jpg)
Fusion: Product Rule (1)
• We require:
• From Bayes’ theorem:
),|( BAHP
),()()|,(
),|(BAP
HPHBAPBAHP
![Page 27: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/27.jpg)
Fusion: Product Rule (2)
• We assume conditional independence so may write:
),()()|()|(
),|(BAP
HPHBPHAPBAHP
![Page 28: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/28.jpg)
Fusion: Product Rule (3)
• Applying Bayes’ theorem again:
• And collecting terms:
),()(
)()()|(
)()()|(
),|(BAP
HPHP
BPBHPHP
APAHPBAHP
),()()(
)()|()|(
),|(BAPBPAP
HPBHPAHP
BAHP .
![Page 29: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/29.jpg)
Fusion: Product Rule (4)
• We may drop the marginal likelihoods again and normalise:
)()|()|(
),|(HP
BHPAHPBAHP
Posteriorprobability
Priorprobability
Posteriorprobability
Fused posteriorprobability
![Page 30: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/30.jpg)
Multisource Fusion Rule
• The generalisation of this fusion rule to multiple sources:
• This is commutative
11
)(
)|()|(
N
N
ii
HP
xHPXHP
![Page 31: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/31.jpg)
Commutativity of Fusion (1)
)()(
)|(
)(
)|(
)(
)|()|(
11
11
11
HPHP
xHP
HP
xHP
HP
xHPXHP
S
S
ii
R
R
ii
N
N
ii
![Page 32: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/32.jpg)
Commutativity of Fusion (2)
• The probability fusion rule commutes:– It doesn’t matter what the architecture is– It doesn’t matter if it is single stage or
multi-stage
![Page 33: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/33.jpg)
Experiment: Results
• Normalising gives:P(H|A,B) = 0.33 P(~H|A,B) = 0.67
59.095.0
80.070.0)(
)|()|(),|(
×
HPBHPAHP
BAHP
20.105.0
20.030.0)(~
)|(~)|(~),|(~
×
HPBHPAHP
BAHP
![Page 34: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/34.jpg)
Human Fusion Experiment (3)No threat
H=1.00
PriorP(H)=0.95
Report BP(H|XB)=0.80
Report AP(H|XA)=0.70
Fusion A,BP(H|XA,XB)=0.33
![Page 35: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/35.jpg)
Why was that so hard?
• Most humans find it difficult to intuitively fuse uncertain information– Not because they are innumerate– But because they cannot comfortably
balance the evidence (likelihood) with their predisposition (prior)
![Page 36: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/36.jpg)
Prior Sensitivity (1)
• If the issue is with the priors – do they matter?
• Can we ignore the priors?• Do we get the same final decision if
we change the priors?
![Page 37: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/37.jpg)
Prior Sensitivity (2)• If P(H|A) = P(H|B)• What value of P(H)
makes P(H|A,B) = 0.5?
22
2
)|(1)|(
)|()(
AHPAHP
AHPHP
![Page 38: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/38.jpg)
Prior Sensitivity (3)
0
0.1
0.20.3
0.4
0.5
0.6
0.70.8
0.9
1
0 0.2 0.4 0.6 0.8 1
P(H|A)=P(H|B)
P(H)
![Page 39: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/39.jpg)
Prior Sensitivity (4)
• Between 0.2 < P(H|A) < 0.8 the prior has a significant effect
• Carefully define the domain over which the prior is evaluated
• Put effort into using a reasonable value
![Page 40: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/40.jpg)
Sensitivity to Posterior Probability
• What about the posterior probabilities delivered to the fusion centre?
• Can we endure errors here?• Which types of errors hurt most?
![Page 41: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/41.jpg)
Probability Experiment (1)
• 10 estimation questions• Write down lower and upper bound• So that you are 90% sure it covers the
actual value• All questions relate to the highest
point in various countries (in metres)
![Page 42: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/42.jpg)
Probability experiment (2)
• Winner defined as:– Person with most answers correct– Tie-break decided by smallest sum of
ranges (for all 10 questions)
• Pick a range big enough• But not too big!
![Page 43: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/43.jpg)
The questions:-1. Australia2. Chile3. Cuba4. Egypt5. Ethiopia6. Finland7. Hong Kong8. India9. Lithuania10. Poland
![Page 44: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/44.jpg)
The answers:-1. Australia (2228m)2. Chile (6893m)3. Cuba (1974m)4. Egypt (2629m)5. Ethiopia (4550m)6. Finland (1324m)7. Hong Kong (958m)8. India (8586m)9. Lithuania (294m)10. Poland (2499m)
![Page 45: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/45.jpg)
Overconfidence (1)• Large trials show that most people get
fewer than 40% correct• Should be 90% correct!• People are often overconfident
(even when primed that they are being tested!)
![Page 46: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/46.jpg)
Overconfidence (2)
Actual probability
Decl
are
d p
robab
ility
overconfident
overconfident
underconfident
underconfident
wrong
wrong
![Page 47: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/47.jpg)
Confidence Amplification(1)
0 0.2 0.4 0.6 0.8 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Input class probability
Fus
ed
cla
ss p
rob
ab
ility
2 sensors3 sensors4 sensors5 sensors
Fu
sed
cla
ss p
rob
ab
ility
Input class probability
![Page 48: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/48.jpg)
Confidence Amplification(2)
![Page 49: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/49.jpg)
Veto Effect• If any local decision-maker outputs a
probability of close to zero for a class then the fused probability is close to zero– even if all the other decision-makers output a
high probability– about 40% of the response surface for two
sensors is either <0.1 or >0.9– this rises to 50% for three sensors and nearly
60% for four
![Page 50: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/50.jpg)
Moderation of probabilities
• If we suspect that the posterior probabilities are overconfident then we should moderate them– By building it into automatic techniques– By allowing for it if this is not possible
![Page 51: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/51.jpg)
Gaussian Moderation
• For Gaussian classifiers the Bayesian correction is analytically tractable
• By integrating over the mean and variance rather than taking the maximum likelihood value
![Page 52: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/52.jpg)
Student t-distribution(1)• For Gaussian data this is:
• Which is a “Student” t-distribution:
)|,(),|()|( 2
0
22 DPxPddDxP ii
2
2
22 1
ˆ)1(
)ˆ(.
21
)1(ˆ
2),ˆ,ˆ|(
N
ii
N
xN
N
N
NxP
![Page 53: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/53.jpg)
Student t-distribution(2)
-10 -5 0 5 100
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
Measurement value
Lik
elih
oo
d o
f da
taMeasurement value
Like
lihoo
d of
dat
a-10 -5 0 5 100
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
Measurement value
Lik
elih
oo
d o
f da
ta
![Page 54: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/54.jpg)
Student t-distribution(3)
-10 -5 0 5 100
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Measurement value
Pro
ba
bili
ty o
f cla
ss 1
Pro
babi
lity
of c
lass
1
-10 -5 0 5 100
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Measurement valueP
rob
ab
ility
of c
lass
1P
roba
bilit
y of
cla
ss 1
![Page 55: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/55.jpg)
Approximate Moderation(1)
• We can get a similar effect at the fusion centre using the posteriors– Convert back to “likelihoods” by dividing by the
prior– Add a constant to everything– Convert back to “posteriors” by multiplying by
the prior– Renormalise
![Page 56: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/56.jpg)
Approximate Moderation(2)
• How much to add depends on the source of the posterior probabilities– Correction factor for each source– Learned from data
![Page 57: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/57.jpg)
Other Issues
• Conditional independence not holding• Information incest• Missing data• Communication errors• Asynchronous information
![Page 58: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/58.jpg)
Information DominancePart Two: Decision
“Having reasoned about the datato form a perspective of the current situation; to make an informed decision which optimises the desirability of the outcome”
![Page 59: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/59.jpg)
Deciding what to do
“Decision theory is trivial, apart from the details”
• Select an action that maximises the expected utility of the outcome
![Page 60: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/60.jpg)
Utility functions?
• A utility function describes how desirable each possible outcome is– People are sometimes irrational– Desirability cannot be captured by a
single valued function– Allais paradox
![Page 61: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/61.jpg)
Utility Experiment(1)
1. Guaranteed €1 million2. 89% chance of €1 million
10% chance of €5 million1% chance of nothing
![Page 62: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/62.jpg)
Utility Experiment(2)
1. 89% chance of nothing11% chance of €1 million
2. 90% chance of nothing10% chance of €5 million
![Page 63: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/63.jpg)
Utility Experiment(3)
• If you prefer 1 to 2 on the first slideYou should prefer 1 to 2 on the second slide as well
• If not you are acting irrationally…
![Page 64: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/64.jpg)
Decision Theory• Assume we are able to construct a utility
function (or at least get our superior to define one!)
• Enumerate the possible actions– Use our fused probabilities to weight the utility of
the possible outcomes– Choose the action for which the expected utility
of the outcome is greatest
![Page 65: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/65.jpg)
Timing the decision
• What about timing?• When should the decision be made?
– If we wait then maybe the (fused) probabilities will be more accurate
– Or the action will be more effective
![Page 66: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/66.jpg)
Explore versus Exploit• By waiting you can explore the situation• By stopping you can exploit the situation• Stopping rule
– Sequential analysis– SPRT– Bayesian optimal stopping
![Page 67: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/67.jpg)
Experiment with timing
• I will show you 20 numbers• They are drawn from the same
(uniform) distribution• Select the highest value• But no going back• A bit like ¡Allá tú!
![Page 68: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/68.jpg)
Experiment with timing(1)
131
![Page 69: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/69.jpg)
Experiment with timing(2)
16
![Page 70: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/70.jpg)
Experiment with timing(3)
125
![Page 71: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/71.jpg)
Experiment with timing(4)
189
![Page 72: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/72.jpg)
Experiment with timing(5)
105
![Page 73: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/73.jpg)
Experiment with timing(6)
172
![Page 74: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/74.jpg)
Experiment with timing(7)
39
![Page 75: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/75.jpg)
Experiment with timing(8)
94
![Page 76: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/76.jpg)
Experiment with timing(9)
57
![Page 77: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/77.jpg)
Experiment with timing(10)
133
![Page 78: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/78.jpg)
Experiment with timing(11)
52
![Page 79: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/79.jpg)
Experiment with timing(12)
69
![Page 80: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/80.jpg)
Experiment with timing(13)
7
![Page 81: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/81.jpg)
Experiment with timing(14)
242
![Page 82: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/82.jpg)
Experiment with timing(15)
148
![Page 83: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/83.jpg)
Experiment with timing(16)
163
![Page 84: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/84.jpg)
Experiment with timing(17)
23
![Page 85: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/85.jpg)
Experiment with timing(18)
139
![Page 86: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/86.jpg)
Experiment with timing(19)
146
![Page 87: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/87.jpg)
Experiment with timing(20)
211
![Page 88: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/88.jpg)
The answer…
• How many people chose 242?• Balance between collecting data on
how big the numbers might be (exploration)and actually picking a big number(exploitation)
![Page 89: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/89.jpg)
The 1/e Law(1)
• Consider a rule of the form:
Observe M and remember the best value (V)
Observe remaining N-M and pick the first that exceeds V
![Page 90: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/90.jpg)
The 1/e Law(2)
• It can be shown that the optimum value for M is N/e
• And that for this rule the probability of selecting the maximum is at least 1/e
• Even for huge values of N
![Page 91: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/91.jpg)
Time Pressure (1)
• Individuals tend to make the decision too early
• Committees tend to leave the decision too late
![Page 92: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/92.jpg)
Time Pressure (2)
• Lecturers tend to overrun their time slot!
![Page 93: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/93.jpg)
Time Pressure (3)• Apologies for skipping over so much of the
detail• Some of the other areas that warrant
mention:– Game theory– Sensor management– Graphical models– Cognitive inertia– Inattentional blindness
![Page 94: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/94.jpg)
Please feel freeto contact me
www.mv-concepts.com
Or just come and introduce yourself…
![Page 95: Do Humans make Good Observers – and can they Reliably Fuse Information? Dr. Mark Bedworth MV Concepts Ltd. mark.bedworth@mv-concepts.com.](https://reader037.fdocuments.in/reader037/viewer/2022110211/56649ee65503460f94bf6420/html5/thumbnails/95.jpg)
Thank you!
Questions…