Measuring the Quality of Decisionmaking and Planning

Framed in the Context of IBC Experimentation

February 9, 2007

Evidence Based Research, Inc.

2Today’s Ground Rules

• IBC provides the framework for the discussion on measuring decisionmaking.

• This is not an IBC brief.• The briefer does not have the answers!• The purpose is to explore how one measures the quality

of decisionmaking in complex environments – this is a discussion session.

3Background Regarding IBC

Focus: • Visualization, operational assessment, and planning

tools for EBO.

Multiphased effort:• Phase 0: Proof of concept using ad hoc suite of tools.• Phase I: Demonstrated improvement potential of

federated tool sets developed by two contractors. • Phase II: Evolve tool set to expanded capabilities for

transition to JFCOM & COCOMS.

4Objectives of Phase II Experimentation

• Establish the operational utility of the tools.• Learn which tools can be helpful under what

circumstances and for what purposes.• Explore novel approaches throughout the spiral

development process.• Measure progress in increasing readiness of the

technology.• Facilitate early transition to operational users.

5Challenges

• Complex operational problem: – Wide range of situations, – Multiple knowledge domains, and– Poorly understood relationships.

• Major technical challenges:– Variation in maturity of underlying models and tools,– Limitations in availability of relevant data, and– Composability of federated capability from continually

expanding set of tools.

• Transition hurdles:– Operational risk due to high potential for limitations in

fidelity and possible misapplication.

6Option Exploration Tool (1)

• Comprehensive family of models.• Action-to-effect and effect-to-action.• Generates the distribution of plausible outcomes.• Visualization: assist leaders to understand and act.

7

DIME DimensionsPMESII Dimensions

Conceptual Models

Modeling Paradigm

Object Models

Causal Models

Statistical Models

Proposed Action

Desired Effect

Agent Based Simulation

Concept Map

Event-based Simulation

Differential Equations

Bayesian Network

Social Network

Petri net

Influence Diagram

Spread Sheet

Political/ Religious Model

Social/ Culture Model

Social Information Model

Political/Economic Model

Economic/Infrastructure Model

Enables “What-if” Analysis, Answers “Why”Enables “What-if” Analysis, Answers “Why”

Undesired Effect

Desired Effect

Military ModelReligious Model

Distribution authorized only to US Government Agencies and their contractors involved in Phase 2 of the Integrated Battle Command Program

Option Exploration Tool (2)

8 Campaign Planning Tools (1)

• Supports modular development of multiple lines of effort.• Automatically detects and displays interdependencies,

assumptions, resources, actions, duration of effects, metrics, and next state.

• Ability to modify plans, actions, interdependencies, models, and next states based on measured performance.

9

Political/ Strategy

Security Strategy

Political-Economic Strategy

Next StateActions

Assumptions

Resources

Measured Performance

Interdependencies

Allows the leader to visualize complex plansAllows the leader to visualize complex plansDistribution authorized only to US Government Agencies and their contractors involved in Phase 2 of the Integrated Battle Command Program

Campaign Planning Tools (2)

10

Network Centric Operations

Quality of Individual Sensemaking

Degree of Decision Synchronization

Degree of Effectiveness

Degree of Information “Share-ability”

Quality of Networking

Force

Quality of Individual Information Degree of Shared Information

Quality of Organic Information

C2 EffectorsValue Added Services

Qualityof

Inter-actions

InformationSources

Degree of Actions/ Entities SynchronizedC2 A

gility

Forc

e Agi

lity

Physical Domain

Social Domain

Information Domain

Cognitive Domain

Degree of Networking Net Readiness of Nodes

Individual Understanding

Degree of Shared SensemakingShared Awareness

Collaborative DecisionsIndividual Decisions

Shared Understanding

Individual Awareness

11

Decisionmaking Quality

• Two approaches:

– Quality of the process:• How the decision was made, and• Elements in the process.

– Quality of the decision:• Was it “right” or at least “not wrong”?• Did it have the desired effect?

12

Quality of the Process

• Correct sequence of actions.• Complete set of choices.• Assumes an observable process.• Repeatability.• Outcome is irrelevant.

13

Quality of the Decision

• Applicability to the solution space.• Compatibility with other SMEs.• Desired outcome/effects.• Outcome is relevant.

14

Measuring Quality

• The field needs a way to measure the quality of a product:– What does it mean to do good?– What does it mean to make a good decision?

• A simple problem:– Has a correct answer.

• A complex problem:– Has no right answer,– May have “not wrong” answers, and– Will have multiple perspectives.

15

Discussion Time

Thoughts?