Robust Decision Making Robert Lempert RAND HDGC Seminar February 13, 2004.
New Methods to Bridge Long-Term Policy Analysis and Robust Decision Making Robert Lempert Director...
-
Upload
junior-glenn -
Category
Documents
-
view
214 -
download
0
Transcript of New Methods to Bridge Long-Term Policy Analysis and Robust Decision Making Robert Lempert Director...
New Methods to Bridge Long-Term Policy Analysis
and Robust Decision Making
New Methods to Bridge Long-Term Policy Analysis
and Robust Decision Making
Robert Lempert
DirectorRAND Frederick S. Pardee Center for Longer Range Global Policy and the Future Human Condition
BLOSSOM WorkshopEuropean Environmental Agency
April 30, 2008
2 4-30-08
Long-Term Decisions Present Difficult Challenges
Long-Term Decisions Present Difficult Challenges
• Long-term decisions occur
– When reflecting on potential events decades or more in the future
– Causes decision makers to choose near-term actions different than those they would otherwise pursue
• Long-term decisions occur
– When reflecting on potential events decades or more in the future
– Causes decision makers to choose near-term actions different than those they would otherwise pursue
• Long-term policy analysis often fails to persuade because
– Conditions of deep uncertainty prevail
– Short-term needs loom large
– Views of the future often anchored in the present
• Long-term policy analysis often fails to persuade because
– Conditions of deep uncertainty prevail
– Short-term needs loom large
– Views of the future often anchored in the present
3 4-30-08
Long-Term Decisions Present Difficult Challenges
Long-Term Decisions Present Difficult Challenges
• Long-term decisions occur
– When reflecting on potential events decades or more in the future
– Causes decision makers to choose near-term actions different than those they would otherwise pursue
• Long-term decisions occur
– When reflecting on potential events decades or more in the future
– Causes decision makers to choose near-term actions different than those they would otherwise pursue
“Missions to Mars” at
Disneyland’s Tomorrowland ca 1955
• Long-term policy analysis often fails to persuade because
– Conditions of deep uncertainty prevail
– Short-term needs loom large
– Views of the future often anchored in the present
• Long-term policy analysis often fails to persuade because
– Conditions of deep uncertainty prevail
– Short-term needs loom large
– Views of the future often anchored in the present
4 4-30-08
Scenarios Attractive for Long-Term Analysis, but Have Weakness in Contentious Public Debates
Scenarios Attractive for Long-Term Analysis, but Have Weakness in Contentious Public Debates
• At their best, scenarios can help decision makers
– Reduce overconfidence
– Challenge their mental models
– Overcome organizational and psychological barriers to considering threatening or inconvenient futures
• At their best, scenarios can help decision makers
– Reduce overconfidence
– Challenge their mental models
– Overcome organizational and psychological barriers to considering threatening or inconvenient futures
5 4-30-08
Scenarios Capture the Key Concept That a Multiplicity of Plausible Futures May Be as Close as We Get to the Truth
Scenarios Capture the Key Concept That a Multiplicity of Plausible Futures May Be as Close as We Get to the Truth
Rabbi Eliezer Ashkenazi (1580) chose to interpret the Tower of Babel story not as a challenge to divine power to which the Lord's response was to divide the human race but rather the opposite.
He saw the story as an attempt to establish a universal religious regime which God "was obliged to separate
…since the proliferation of doctrines aids and stimulates the investigator to attain the desired truths.”
6 4-30-08
Scenarios Attractive for Long-Term Analysis, but Have Weakness in Contentious Public Debates
Scenarios Attractive for Long-Term Analysis, but Have Weakness in Contentious Public Debates
• At their best, scenarios can help decision makers
– Reduce overconfidence
– Challenge their mental models
– Overcome organizational and psychological barriers to considering threatening or inconvenient futures
• But in contentious public debates, scenario methods can have difficulty
– Engaging mental models of diverse stakeholders
– Systematically informing decisions under uncertainty
– Addressing surprise and discontinuities
• At their best, scenarios can help decision makers
– Reduce overconfidence
– Challenge their mental models
– Overcome organizational and psychological barriers to considering threatening or inconvenient futures
• But in contentious public debates, scenario methods can have difficulty
– Engaging mental models of diverse stakeholders
– Systematically informing decisions under uncertainty
– Addressing surprise and discontinuities
7 4-30-08
OutlineOutline
• Robust decision making (RDM) provides framework for effective long-term analysis
• RDM’s “Scenario Discovery” approach offers useful scenario concept for public debates
• Recent work measures the impacts of these approaches with decision makers
• Robust decision making (RDM) provides framework for effective long-term analysis
• RDM’s “Scenario Discovery” approach offers useful scenario concept for public debates
• Recent work measures the impacts of these approaches with decision makers
8 4-30-08
RDM Views Scenarios As Part of Process Identifying and Building Consensus for Robust Strategies
RDM Views Scenarios As Part of Process Identifying and Building Consensus for Robust Strategies
Key Robust Decision Making Concepts:
• Construct ensemble of long-term scenarios that highlight key tradeoffs among near-term policy choices
• Consider near-term choices as one step in a sequence of decisions that evolve over time
• Use robustness criteria to compare alternative strategies
Key Robust Decision Making Concepts:
• Construct ensemble of long-term scenarios that highlight key tradeoffs among near-term policy choices
• Consider near-term choices as one step in a sequence of decisions that evolve over time
• Use robustness criteria to compare alternative strategies
– A robust strategy performs well compared to the alternatives over a wide range of plausible futures
– A robust strategy performs well compared to the alternatives over a wide range of plausible futures
9 4-30-08
New Technology Allows Computer to Serve As “Prosthesis for the Imagination”
New Technology Allows Computer to Serve As “Prosthesis for the Imagination”
• Robust Decision Making (RDM) is a quantitative decision analytic approach that
– Characterizes uncertainty with multiple, rather than single, views of the future
– Evaluates alternative decision options with a robustness, rather than optimality, criterion
• Robust Decision Making (RDM) is a quantitative decision analytic approach that
– Characterizes uncertainty with multiple, rather than single, views of the future
– Evaluates alternative decision options with a robustness, rather than optimality, criterion
10 4-30-08
New Technology Allows Computer to Serve As “Prosthesis for the Imagination”
New Technology Allows Computer to Serve As “Prosthesis for the Imagination”
• Robust Decision Making (RDM) is a quantitative decision analytic approach that
– Characterizes uncertainty with multiple, rather than single, views of the future
– Evaluates alternative decision options with a robustness, rather than optimality, criterion
– Iteratively identifies vulnerabilities of plans and evaluates potential responses
• Robust Decision Making (RDM) is a quantitative decision analytic approach that
– Characterizes uncertainty with multiple, rather than single, views of the future
– Evaluates alternative decision options with a robustness, rather than optimality, criterion
– Iteratively identifies vulnerabilities of plans and evaluates potential responses
Candidate strategy
Identify vulnerabilities
Assess alternatives for ameliorating vulnerabilities
• RDM combines key advantages of scenario planning and quantitative decision analysis in ways that
– Decision makers find credible
– Contribute usefully to contentious debates
• RDM combines key advantages of scenario planning and quantitative decision analysis in ways that
– Decision makers find credible
– Contribute usefully to contentious debates
11 4-30-08
Stylized Sustainability Example Summarizes RDM Approach
Stylized Sustainability Example Summarizes RDM Approach
What near-term actions can help ensure economic growth and environmental quality over the 21st century?
Economic growth rate (%)
1.0 2.0 3.0 4.0
–1.0
0
1.0
5.0
3.0
4.0
2.0
0
India since 1960
U.S. 1890-1930
U.S. since 1950
U.S. in 20th century
China since 1960
Brazil since 1980
Russia since 1993
Decoupling rate (%)
(Rate at which technology,
without regulation,
reduces emissions per
GDP)
Analysis suggests testing alternative strategies over this range of futures
12 4-30-08
Compare “Fixed” Near-Term Strategies Across Scenarios
Compare “Fixed” Near-Term Strategies Across Scenarios
Near term
Choose policies
Assume near-term policy continues until changed by future generations
Future decision-makers recognize
and correct our mistakes
Future
RAND MR-1626-RPC
13 4-30-08
1.0 2.0 3.0 4.0
–1.0
0
1.0
5.0
3.0
4.0
2.0
0
Conventionalworld
scenario
Strategy’sPerformanceStrategy’sPerformance
No regretMild
A lotOverwhelming
1.0 2.0 3.0 4.0
–1.0
0
1.0
5.0
3.0
4.0
2.0
0
Economic growth rate (%)
Decouplingrate (%)
Stay the CourseStay the Course
U.S. in 20th century
U.S. since 1950
Initial Scan Suggests No Fixed Emission Reduction Target Is Robust
Initial Scan Suggests No Fixed Emission Reduction Target Is Robust
StrategyVulner-abilities
Alternatives
1. Run simulation thousands of times
2. Display “scenario maps” showing deviation of proposed strategy from optimality over many futures
14 4-30-08
1.0 2.0 3.0 4.0
–1.0
0
1.0
5.0
3.0
4.0
2.0
0
Conventionalworld
scenario
Strategy’sPerformanceStrategy’sPerformance
No regretMild
A lotOverwhelming
1.0 2.0 3.0 4.0
–1.0
0
1.0
5.0
3.0
4.0
2.0
0
Economic growth rate (%)
Decouplingrate (%)
Stay the CourseStay the Course
U.S. in 20th century
U.S. since 1950
Initial Scan Suggests No Fixed Emission Reduction Target Is Robust
Initial Scan Suggests No Fixed Emission Reduction Target Is Robust
1.0 2.0 3.0 4.0
–1.0
0
1.0
5.0
3.0
4.0
2.0
0
Conventionalworld
scenario
1.0 2.0 3.0 4.0
–1.0
0
1.0
5.0
3.0
4.0
2.0
0
Crash EffortCrash Effort
U.S. in 20th century
U.S. since 1950
15 4-30-08
Craft Near-Term Adaptive Strategy That Aims to Balance Environmental and Economic Goals
Craft Near-Term Adaptive Strategy That Aims to Balance Environmental and Economic Goals
Present Future
NODoes the carrying capacity change?
Choose policies to maximize utility
Determine best policy to meet milestone
Select near-term milestone
YES
Is milestone achievable with
current approach?
Relax milestone
YES
NO
Implement policy
RAND MR-1626-RPC
16 4-30-08
Robust Strategy Reduces Uncertainty By Performing Well No Matter What Future Comes to Pass
Robust Strategy Reduces Uncertainty By Performing Well No Matter What Future Comes to Pass
Adaptive StrategyAdaptive Strategy
Economic growth rate (%)1.0 2.0 3.0 4.0
–1.0
0
1.0
5.0
3.0
4.0
2.0
0
No regretMild
A lotOverwhelming
U.S. in 19thcentury
U.S. since 1950
U.S. in 20thU.S. in 20thcenturycentury
Decoupling rate (%)
StrategyVulner-abilities Alternatives
17 4-30-08
OutlineOutline
• Robust decision making (RDM) provides framework for effective long-term analysis
• RDM’s “Scenario Discovery” approach offers useful scenario concept for public debates
• Recent work measures the impacts of these approaches with decision makers
• Robust decision making (RDM) provides framework for effective long-term analysis
• RDM’s “Scenario Discovery” approach offers useful scenario concept for public debates
• Recent work measures the impacts of these approaches with decision makers
18 4-30-08
Long-Term RDM Scenarios Highlight Trade-offs Among Near-Term DecisionsLong-Term RDM Scenarios Highlight
Trade-offs Among Near-Term Decisions1. Run simulation model for many different combinations of
uncertain input parameters
2. Identify those clusters of cases that highlight tradeoffs among near-term candidate strategies
1. Run simulation model for many different combinations of uncertain input parameters
2. Identify those clusters of cases that highlight tradeoffs among near-term candidate strategies
Candidate strategy
Identify vulnerabilities
Assess alternatives for ameliorating vulnerabilities
• Example future conditions highlighting near-term tradeoffs:
– 2007 Congressional Reauthorization of Terrorism Risk Insurance Act: In what situations would ending TRIA cost the taxpayer more than retaining the program?
– California water planning: Under what conditions would future climate change impacts suggest modifying current long-range water management plans?
• Example future conditions highlighting near-term tradeoffs:
– 2007 Congressional Reauthorization of Terrorism Risk Insurance Act: In what situations would ending TRIA cost the taxpayer more than retaining the program?
– California water planning: Under what conditions would future climate change impacts suggest modifying current long-range water management plans?
19 4-30-08
Scenario Discovery Implements This Concept for Computer-Assisted Scenario Development
Scenario Discovery Implements This Concept for Computer-Assisted Scenario Development
. . .. . ......
1. Indicate policy-relevant cases in database of simulation results
20 4-30-08
Scenario Discovery Implements This Concept for Computer-Assisted Scenario Development
Scenario Discovery Implements This Concept for Computer-Assisted Scenario Development
. . .. . ......Uncertain input
variable 2
1. Indicate policy-relevant cases in database of simulation results
2. Statistical analysis finds low-dimensional clusters with high density of these cases
Uncertain input variable 1
21 4-30-08
Scenario Discovery Implements This Concept for Computer-Assisted Scenario Development
Scenario Discovery Implements This Concept for Computer-Assisted Scenario Development
. . .. . ......Uncertain input
variable 2
1. Indicate policy-relevant cases in database of simulation results
2. Statistical analysis finds low-dimensional clusters with high density of these cases
3. Clusters represent scenarios and driving forces of interest to decision makers
Uncertain input variable 1
22 4-30-08
Scenario Discovery Implements This Concept for Computer-Assisted Scenario Development
Scenario Discovery Implements This Concept for Computer-Assisted Scenario Development
. . .. . ......Uncertain input
variable 2
1. Indicate policy-relevant cases in database of simulation results
2. Statistical analysis finds low-dimensional clusters with high density of these cases
3. Clusters represent scenarios and driving forces of interest to decision makers
Uncertain input variable 1
Density: • How many cases inside the
scenario are policy-relevant? (e.g. 75%)
Coverage: • How many of all the policy-
relevant cases do the scenarios include? (e.g. 82%)
Interpretability: • Is the number of scenarios
and driving forces sufficiently small to understand? (e.g. 1 scenario with two driving forces)
Density: • How many cases inside the
scenario are policy-relevant? (e.g. 75%)
Coverage: • How many of all the policy-
relevant cases do the scenarios include? (e.g. 82%)
Interpretability: • Is the number of scenarios
and driving forces sufficiently small to understand? (e.g. 1 scenario with two driving forces)
Approach provides measures of merit for scenario quality
23 4-30-08
Scenario Discovery May Improve Impact of Scenarios in Contentious Public Debates
Scenario Discovery May Improve Impact of Scenarios in Contentious Public Debates
• For instance, recent scenario discovery work on U.S. Federal terrorism insurance program was
– Based on a scenario not considered in the official budgetary analysis by government agencies
– Quoted on the floor of the United States Senate by a program supporter
– Criticized as “insidious” by program opponents
• But neither side in the debate could gain traction by quarrelling with our choice of scenario and its key driving forces
• For instance, recent scenario discovery work on U.S. Federal terrorism insurance program was
– Based on a scenario not considered in the official budgetary analysis by government agencies
– Quoted on the floor of the United States Senate by a program supporter
– Criticized as “insidious” by program opponents
• But neither side in the debate could gain traction by quarrelling with our choice of scenario and its key driving forces
24 4-30-08
OutlineOutline
• Robust decision making (RDM) provides framework for effective long-term analysis
• RDM’s “Scenario Discovery” approach offers useful scenario concept for public debates
• Recent work measures the impacts of these approaches with decision makers
• Robust decision making (RDM) provides framework for effective long-term analysis
• RDM’s “Scenario Discovery” approach offers useful scenario concept for public debates
• Recent work measures the impacts of these approaches with decision makers
25 4-30-08
How Does Climate Change Affect California’s Inland Empire Utilities Agency (IEUA)?
IEUA currently serves 800,000 people
– May add 300,000 by 2025
Water presents a significant challenge
26 4-30-08
How Does Climate Change Affect California’s Inland Empire Utilities Agency (IEUA)?
IEUA currently serves 800,000 people
– May add 300,000 by 2025
Water presents a significant challenge
IEUA’s 2005 long-range Urban Water Management Plan (UWMP)
– Aims to meet needs of growing population, but
– Did not address climate change
27 4-30-08
How Does Climate Change Affect California’s Inland Empire Utilities Agency (IEUA)?
IEUA currently serves 800,000 people
– May add 300,000 by 2025
Water presents a significant challenge
We conducted several analyses to help IEUA assess impact of climate change on their 2005 UWMP
– Traditional scenarios
– Probabilistic risk analysis
– Scenario Discovery
28 4-30-08
Conducted Workshops to Measure Impact of Alternative Analyses on IEUA
Conducted Workshops to Measure Impact of Alternative Analyses on IEUA
– Four IEUA workshops presented modeling results to participants including:
• Agency professional managers and technical staff
• Local elected officials
• Community stakeholders
– “Real-time” surveys measured participants’
• Understanding of concepts
• Willingness to adjust policy choices based on information presented
• Views on RDM, traditional scenarios, and probabilistic risk analysis
– Four IEUA workshops presented modeling results to participants including:
• Agency professional managers and technical staff
• Local elected officials
• Community stakeholders
– “Real-time” surveys measured participants’
• Understanding of concepts
• Willingness to adjust policy choices based on information presented
• Views on RDM, traditional scenarios, and probabilistic risk analysis
29 4-30-08
Disagree strongly
Agree somewhat
Agree somewhat
Agree somewhat
Agree strongly
Agree somewhat
Participants Ranked Scenario Discovery More Useful, But More Difficult to Understand
Participants Ranked Scenario Discovery More Useful, But More Difficult to Understand
Is easy to explain to decisionmakers
Provides information on how to improve plan
Provides results that can be used in planning
Scenario Discovery
Traditional Scenarios
Questionnaire item from first 3 workshops
– Traditional scenarios
• Gave IEUA much of the information they needed
• Emphasized the importance of achieving goals in IEUA’s plan
– Scenario discovery
• Provided more useful information for evaluating alternatives to plan
• Sparked discussion of adaptive strategies
30 4-30-08
Surveys Suggest RDM Analysis Changed Participants’ Views
Surveys Suggest RDM Analysis Changed Participants’ Views
Participants provided:
– Information on most effective RDM visualizations
After the workshop:
– 35% said consequences of bad climate change now appeared “more serious” than before
– 75% though the ability of IEUA planner to plan for and manage effects was “greater” than before
Overall, analysis:
– Increased support for near-term modifications to current IEUA plan
– Suggests that participants’ willingness to acknowledge a serious climate change threat increased after they felt more confident they could address the threat
Participants provided:
– Information on most effective RDM visualizations
After the workshop:
– 35% said consequences of bad climate change now appeared “more serious” than before
– 75% though the ability of IEUA planner to plan for and manage effects was “greater” than before
Overall, analysis:
– Increased support for near-term modifications to current IEUA plan
– Suggests that participants’ willingness to acknowledge a serious climate change threat increased after they felt more confident they could address the threat
31 4-30-08
Key ConceptsKey Concepts
Choose scenarios to highlight tradeoffs among near-term decisions
Otherwise number of potentially interesting scenarios remains unlimited
Use analytics to facilitate human creativity in designing policies robust across many futures
Measure scenarios’ impacts on decision makers to help improve process and methods
Designing measurements makes purpose clear
Can use framework for general thinking about long-term policy under deep uncertainty
Not just as basis for a modeling exercise
Choose scenarios to highlight tradeoffs among near-term decisions
Otherwise number of potentially interesting scenarios remains unlimited
Use analytics to facilitate human creativity in designing policies robust across many futures
Measure scenarios’ impacts on decision makers to help improve process and methods
Designing measurements makes purpose clear
Can use framework for general thinking about long-term policy under deep uncertainty
Not just as basis for a modeling exercise
32 4-30-08
For More InformationFor More Information
http://www.rand.org/international_programs/pardee/
Thank you!Thank you!