Use of available data to probe methods for decision under uncertainty Raphael T. Haftka...
-
Upload
kianna-gravitt -
Category
Documents
-
view
214 -
download
1
Transcript of Use of available data to probe methods for decision under uncertainty Raphael T. Haftka...
Use of available data to probe methods for decision under
uncertainty
Raphael T. Haftka ([email protected])
Department of Mechanical and Aerospace Engineering
University of Florida
Gainesville, Florida 32611
With: Raluca Rosca (UF), Efstratios Nikolaidis (University of Toledo)
Outline
• Why validating design methods is trickier than validating analysis models
• Domino tower data base• Invented design/decision problem based on
domino tower data• Performance of probabilistic and possibilistic
models and effect of inflation of variance with little data
• Conclusions
Validation of methods for design under uncertainty
• When data is missing, it may not be possible to validate a method. Must compare against other methods
• Choice of method may depend on amount of information
• I give simple instructions how to get to my home to people who do not know Gainesville. I tell locals what is the shortest route.
Gigerenzer’s Approach• Use existing data bases to compare methods for
making binary decisions (choose A or B)• Invent decision problem that can be based on data• Give decision maker part of data base to formulate
decision rule• Test performance of decision rule on entire data• Showed advantage of simple decision-making
methods• Gigerenzer and Todd (1999), Simple Heuristics
That Make Us Smart, Oxford University Press
Our objectives• Extend approach to non-binary decisions,
involving choice of variable (s)
• Focus on use of existing data sets for probing weaknesses rather than strengths– Experiments are more effective for disproving
rather than for proving
• Demonstrate lessons learned from validation
Domino tower data base
• Dominoes stacked until tower collapses
• One set of 50 towers built by Rosca
• Another set of 90 towers built by 16 students in a competition
Competition and Rosca’s histograms
• Analytical model of tower collapse fit best to Gamma distribution
• Data passed chi-test also for normal distribution
0 10 20 30 40 50 60 700
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
0.1
0 10 20 30 40 50 600
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
0.1
Guaranteed performance decision
• We use data to test decisions on level of guaranteed performance
• Company gains advantage over competitors by guaranteeing performance
• How to select guaranteed level?
• High target risks failure, low target risks loss of customers
Decision invented for domino data
• Rosca guarantees the height of the tower she will build
• She wins if she fulfills her guarantee and if competitor cannot build a tower higher by a handicap than her guarantee
• High guarantee risks failure to fulfill• Low guarantee risks failure due to competitor’s
tower exceeding guarantee plus handicap
Examples
• Handicap of 5, guarantee of 30 tiles
• Competitor wins by building 35, no matter what Rosca builds
• Rosca loses by building 29 even if competitor builds only 20
• Rosca wins by building 32 even if competitor builds 36
Test procedure
• Give decision maker (Rosca) five data points for both her and competitor results
• Decision maker fits distributions to each set of five points
• Guarantee selected on basis of distribution• Repeat 80 times for random quintets• Performance based on percentage of
success for all possible pairs (50x90)
Decision methods compared
• Probability based on two distributions
• Possibility based on triangular membership function spanning data
• Probability and possibility with inflated variance to compensate for scarce data
Optimum guarantees – all data
Probabilistic optimum (gamma)
Probabilistic optimum (normal)
Possibilistic optimum (triangular)
Handicap
guarantee Likelihood of success
guarantee Likelihood of success
guarantee Likelihood of success
2 32 0.307 33 0.318 33 0.318 5 31 0.411 32 0.433 32 0.433 8 29 0.563 30 0.536 31 0.513 11 27 0.640 29 0.615 29 0.615 15 26 0.724 27 0.726 28 0.737
Optimum guarantees are marked in bold. The optimum bid for a handicap of 11 is 28, corresponding to a likelihood of success of 0.6533.
Effect of inflation
Rosca Competitor Probabilistic optimum (normal fit)
Possibilistic optimum (triangular fit)
inflation factor for nhand=5
guarantee Likelihood of success
guarantee Likelihood of success
0 0 32 0.433 32 0.433 0 15 29 0.399 33 0.402 15 0 34 0.358 31 0.411 15 15 32 0.433 32 0.433
All data case; The probabilistic guarantee decreases when competitor inflation factor increases and increases when Rosca’s inflation factor increases while the possibilistic guarantee exhibits the opposite trend.
Five data points
Likelihood of success
for probabilistic optimum (normal fit)
Likelihood of success for possibilistic
optimum (triangular fit)
sample size=5
nhand Mean (of 80 runs) Mean (of 80 runs)
2 0.282 0.290 5 0.392 0.392 8 0.503 0.492
11 0.599 0.588 15 0.707 0.694
Conclusions from experiments
• Comparable performance of probability and possibility may indicate need to explain and improve probabilistic performance
• Adverse effect of inflation indicates need for theoretical study of this effect
• Probability downgraded importance of inflated mode of uncertainty and possibility emphasized it. This we only later derived analytically.
• Experiment surprised us and led us to useful insight into difference between the methods
Concluding remarks• We extended Gigerenzer’s method for using
existing data bases to test methods for decisions under uncertainty– Optimum value instead of binary decision– Repeated random partitioning of data
• Demonstrated that experiments can yield insights into decision making methods– Possible weaknesses in standard probabilistic approach
and inflation– Unreported difference between probability and
possibility