Modeling Framework to Support Evidence-Based Decisions
-
Upload
albert-simard -
Category
Business
-
view
1.307 -
download
0
description
Transcript of Modeling Framework to Support Evidence-Based Decisions
A Modelling Framework:
Supporting Evidence-Based Decisions
ModSimWorld
Montreal, Quebec June 8-9, 2009
2
Main MessagesMain Messages
• Models are needed to understand and predict the behavior of complex systems.
• Models are needed to fulfill an agency’s mandate and support its core business.
• Inadequate or incorrect use of models wastes resources, results in errors, and exposes an agency to liability.
• Models are needed to understand and predict the behavior of complex systems.
• Models are needed to fulfill an agency’s mandate and support its core business.
• Inadequate or incorrect use of models wastes resources, results in errors, and exposes an agency to liability.
Models should be used wisely
3
OutlineOutline
I. Underlying Concepts
Scientific underpinning
II. Decision Guide
Decision making
III. Glossary
Common understanding
I. Underlying Concepts
Scientific underpinning
II. Decision Guide
Decision making
III. Glossary
Common understanding
4
About ModelsAbout Models
• What are they?
– Simplified representations of reality.
– Transform data, information, and knowledge into outputs.
• Why do we use them?
– Reality is too complex
– Experiments are infeasible
– Predict consequences
– Increase understanding
• What are they?
– Simplified representations of reality.
– Transform data, information, and knowledge into outputs.
• Why do we use them?
– Reality is too complex
– Experiments are infeasible
– Predict consequences
– Increase understanding
....
Nonaka (2000)
Concepts
5
What is a Framework?What is a Framework?
“Structural outline of the components of an organization, system, or process and the relationships among them.”
Understanding Knowledge Services NRCan (2006)
Concepts
6
Framework ObjectivesFramework Objectives
• Support needs-driven and science-driven analysis.
• Promote dialogue among modelers, managers, & users.
• Reduce wasted time, effort, & money.
• Provide a basis for planning and action.
• Document and justify decisions.
• Support needs-driven and science-driven analysis.
• Promote dialogue among modelers, managers, & users.
• Reduce wasted time, effort, & money.
• Provide a basis for planning and action.
• Document and justify decisions.
Concepts
7
Framework DesignFramework Design
• Reflect modelling, management, and user perspectives.
• Balance efficiency and effectiveness with cost and effort.
• Applicable to both demand and supply approaches to modelling.
• Applicable to both logical and computational models
• Reflect modelling, management, and user perspectives.
• Balance efficiency and effectiveness with cost and effort.
• Applicable to both demand and supply approaches to modelling.
• Applicable to both logical and computational models
Concepts
8
Different PerspectivesDifferent Perspectives
What developers proposed
What managers funded
What stakeholders wanted
What users needed
Concepts
9
Supply & DemandSupply & Demand
Concepts
Supply: I have a model that solves your problem.
Demand: I have a problem that needs a model.
10
Modelling SystemModelling System
External Models
DevelopDevelop
Nature, Society
Internal Models
UseUseManageManage
Lost Models
ShareShare
PreservePreserve
Knowledge Management
Concepts
11
Modelling ProcessModelling Process
Modelling combines science & computers; judgement & experience; insight & intuition.
• Principles: effort, simple, data, knowledge, transparent, understandable.
• Complexity: Modelling is a dynamic feedback process with delays and uncertainty.
• Development: techniques are well-understood; management less understood and practiced.
• Use: Decision making under uncertainty, unknown elements, outcome probabilities.
Modelling combines science & computers; judgement & experience; insight & intuition.
• Principles: effort, simple, data, knowledge, transparent, understandable.
• Complexity: Modelling is a dynamic feedback process with delays and uncertainty.
• Development: techniques are well-understood; management less understood and practiced.
• Use: Decision making under uncertainty, unknown elements, outcome probabilities.
Concepts
12
Systems HierarchySystems Hierarchy
Data
ModelsDecision SupportInformatio
nKnowledge
ManagementManagement
Mandate
Business
Policies
Processes
Concepts
13
Data Data
A model and its data are inseparable; they succeed or fail as one.
– Data Needs: Situation may involve nature, the system, and/or intervention.
– Sampling: Statistics are essential to determine how much data is needed.
– Source: Ownership? Use rights? Privacy & security concerns?
– Scale: Time, space, and process scale must match the situation.
– Quality: Level of accuracy, detail, and completeness are needed?
A model and its data are inseparable; they succeed or fail as one.
– Data Needs: Situation may involve nature, the system, and/or intervention.
– Sampling: Statistics are essential to determine how much data is needed.
– Source: Ownership? Use rights? Privacy & security concerns?
– Scale: Time, space, and process scale must match the situation.
– Quality: Level of accuracy, detail, and completeness are needed?
Concepts
14
OutputsAcquisition ProcessingStorage
Knowledge
Organization
Society
Environment
Events
Economy
Channels
Access Search Retrieval
InterfaceProcessingDatabase
Inputs Audience UseMedia
InterfaceSystemDataModel
Outputs
Concepts
Information SystemInformation System
Interoperability IntegrationAvailability Utility
151. Common
•Flow-through (-)•Fixed (1:1)•Planning•Mechanistic•Automated•Data, facts
2. Complicated •Feedback•Linear (1:n)•Mathematics•Deterministic•Certainty•Explicit knowledge
3. Complex •Predictive feedback (+)•Non-linear (1:?)•Simulation•Stochastic•Uncertainty •Tacit knowledge
4. Chaotic
•Emergent•Disorganized•Scenario analysis•Mental•Reaction•Intuition
System:Behavior:Approach:Model:Decision:Basis:
Concepts
Models and Knowledge Models and Knowledge
16
OutlineOutline
I. Underlying Concepts
Scientific underpinning
II. Decision Guide
Decision making
III. Glossary
Common understanding
I. Underlying Concepts
Scientific underpinning
II. Decision Guide
Decision making
III. Glossary
Common understanding
17
Decision Guide - HierarchyDecision Guide - Hierarchy
• Phase: (3) demand, supply, project
• Stage: (7) approach, design, establish, develop, evaluate, implement, conclude
• Step: (34) screening, problem definition, suitability, knowledge, data
• Consideration (132): recurrence, importance, problem space, existence
• Phase: (3) demand, supply, project
• Stage: (7) approach, design, establish, develop, evaluate, implement, conclude
• Step: (34) screening, problem definition, suitability, knowledge, data
• Consideration (132): recurrence, importance, problem space, existence
18
Design
Acquire Data
Generate Knowledge
Approach
Out
Evaluation
IssueStart: (Demand)
Implementation
Outputs
Conclusion
End
Development(Modeller)
(Manager)
(User)
D1
D2
3
Applicability
S2
ModelStart (Supply) identification
S1
5
7
6
Establishment
4
(Manager)
(All)
Decision Guide - StagesGuide
19
Issue Model
Demand Supply
Project
End
Decision Guide PhasesDecision Guide Phases
Guide
20
Start (use)
Demand-drivenbackward chaining, closed question
Model
Supply-drivenforward chaining, open question
Start (model) Uses
Supply & DemandSupply & Demand
Guide
21
Issue
D1. Approach
D2. Design
Development
Acquire Data
Generate Knowledge
Out
Demand PhaseDemand Phase
Guide
22
Issue
Initial Screening
Problem Definition
Suitability
Knowledge Evaluation
Data Availability
Design
Recurrence ImportanceProblem spaceExistence
Business FunctionIntended use
Time available Situation
NeedsExistingGap
NeedsAttributesAccessibilityProcessing
Continue
Continue
Continue
Continue
Continue
Below threshold
Can’t define
Unsuitable
Excess gap
Inadequate
Generate?
Acquire ?
Yes
Yes
Out
No
No
D1.1
D1.2
D1.3
D1.4
D1.5
Guide Approach Stage
23
Decision Guide ConsiderationsDecision Guide Considerations
• Explains the question.
• Classify a situation or write a short description.
• Complete a statement template.
• Decide where to go next.
• Not a cookbook to be followed without interpretation.
• Compliments experience & judgement; doesn’t replace them.
• Explains the question.
• Classify a situation or write a short description.
• Complete a statement template.
• Decide where to go next.
• Not a cookbook to be followed without interpretation.
• Compliments experience & judgement; doesn’t replace them.
Guide
24
Development
Model
S1. Identification
S2. Applicability
Evaluation
Conclusion
Supply PhaseSupply Phase
Guide
25
Existing Model
Knowledge base
Suitability
Evaluation
Specification criteria (4)
Knowledge criteria (3)
Business lineFunction Development
S2.2
S2.3
S2.4
IdentificationSearchDescription
S1
modify
Data AvailabilityData criteria (5)
Data Acquisition
inaccessible
Conclusion
unsuitable
unjustified
Specifications
S2.1
Development criteria (3) Development
S2.5
Applicability Stage
Guide
26
Project
Phase
End
Design
Applicability
4. Development
5. Evaluation
6. Implementation
7. Project Conclusion
Out3. Project Establishment
Situation
Guide
27
Project Establishment
Hierarchy Relationships Indicators Review
Logic Computation Debugging Review
Attributes ConsistencyReview
Uncertainty Representation Review
Conceptualization
Construction
Evaluation
Verification
Validation
Inventory
exit
exit
exit
exit
4.2
4.3
4.4
4.5
continue
continue
continue
continue
Interaction
Awareness Understanding Consensus
4.1
Development Stage
Guide
28
OutlineOutline
I. Underlying Concepts
Scientific underpinning
II. Decision Guide
Decision making
III. Glossary
Common understanding
I. Underlying Concepts
Scientific underpinning
II. Decision Guide
Decision making
III. Glossary
Common understanding
29
GlossaryGlossary
• Background (introduction, methods, references).
• Taxonomy (organization, nature, risk analysis, content, modelling, concepts).
• Definitions - 650 terms from six sources.
• Links to taxonomy and related terms.
• Background (introduction, methods, references).
• Taxonomy (organization, nature, risk analysis, content, modelling, concepts).
• Definitions - 650 terms from six sources.
• Links to taxonomy and related terms.
Glossary
30
Sample DefinitionSample Definition
Model: Abstract and simplified construct or representation of reality in the form of a pattern, description, or definition that shows the essential structure, relationships, and workings of a concept, process, or system.
(see modelling approach, function, modelling methods, process, relationship, representation, system)
Glossary
31
Modelling Framework:
• Supports an agency’s business
• Facilitates horizontal integration
• Minimizes waste & inefficiency
• Maximizes likely success
• Documents & justifies decisions
“Using a clear blueprint first prevents chaos latter.”
Carla O’Dell (1998)