Value Focused Metrics 26 ISMOR D.F. Davis A.D. Caldwell ddavis@gmu [email protected]

Peace Operations Policy Program, School of Public Policy

POPP

Value Focused Metrics26 ISMORD.F. Davis

A.D. [email protected]

[email protected]


POPPVFMValue Focused Metrics is a term stolen from Keeney’s Value Focused Thinking.

This use is not unique and a simple Google search brings out 99 other uses. Most of these are in the context of “Agile Human Resources,” “Agile Systems Design.” and “Project Management.”

Similar uses have been around since the early 1900s, but mostly since the maturing of Decision Science in the 1960s .

Value Focused Thinking by Ralph Keeney came out in 1992.


POPPInsight

“The purpose of computing is insight, not numbers.” Richard W. Hamming while at IBM labs

“The purpose of analysis is insight, not numbers.” Wilbur Payne while at TRADOC Analysis Command

“Our purpose is insight, not metrics.” Dave Davis while at ISMOR


POPP

RTO-TR-081Page 5-1

NATO Code of Best Practices For Command and Control Assessment

Metrics should flow from the process, not be separated from it.


POPP

The Code of Best Practice has adopted the following five levels of MoM (Measures of Merit):

• Measures of Policy Effectiveness (MoPE), which focus on policy and societal outcomes;• Measures of Force Effectiveness (MoFE), which focus on how a force performs its mission or the degree to which it meets its objectives;• Measures of C2 Effectiveness (MoCE), which focus on the impact of C2 systems within the operational context;• Measures of Performance (MoP), which focus on internal system structure, characteristics and behaviour; and • Dimensional Parameters (DP), which focus on the properties or characteristics inherent in the physical C2 systems.

RTO-TR-081-2

COBP


POPP

RTO-TR-081Page 5-3

COBPThis figure implies that metrics are nested. I’m not sure this is always the case.


POPPMPICE

SectorsGoals

Drivers/CapacitiesIndicators

Measures


POPP

0.0000.1000.2000.3000.4000.5000.6000.7000.8000.9001.000

HSI B

asel

ine

Econ

omic

s

Econ

Driv

ers

Econ

Insti

tutio

ns

Soci

al W

ell B

eing

SWB

Driv

ers

SWB

Insti

tutio

ns

Politi

cal

Pol D

river

s

Pol In

stitu

tions

Rule

of L

aw

RoL D

river

s

RoL I

nstit

ution

s

Secu

rity

Secu

rity D

river

s

Secu

rity I

nstit

ution

s

Phase I and Phase II Overall

Phase I

Phase II


POPP

0.0000.1000.2000.3000.4000.5000.6000.7000.8000.9001.000

Phase I and II Sectors

Phase I

Phase II


POPP

MoE Road Map

Mission

Objectives

Effects

Tasks

Enabling Tasks

MoEMeasure of Effectiveness

MoPMeasure of Performance

MoSMeasure of Success

Mission Statement

Lines of Operation

Decisive Points

Effects BasedPlanning

PrimaryIndicators

Campaign Plan

SecondaryIndicators

Subjective

Objective

Measures

RICHARDSON, H, 2004, MOE AND EBO IN THE HEADQUARTERS ARRC, IN THE CORNWALLIS CROUP IX: ANALYSIS FOR STABILIZATION AND COUNTER-TERRORIST OPERATIONS, THE CANADIAN PEACEKEEPING PRESS (HTTP://WWW.THECORNWALLISGROUP.ORG), P83


POPP

Schematic Overview of Campaign PlanCampaign Assessment

PHASE III

Safe and Secure Environment

Coalition Coherence

Infrastructure/

Humanitarian Sp

DP4

DP5

DP9

DP11

DP16DP8

DP1 DP15

DP10 DP21

DP19

PHASE IV

DP3

Mission/Phase

Endstate

Political /Legal

DP7

DP6

DP14 DP18

DP20

DP2

DP12 DP13 DP17 DP22

LOPs Decisive Pointstimeline

Measures of Effectiveness

? MOS



POPP

Phase 3 –Key Effects

ZNOG DEFEATED

ZNOG ISOLATED

HA OPS

PODs/LOCs

CONDITIONS FOR SSE

0% CE(Surrendered/Destroyed/W-drawn)

35% CE(< 1 Mech/ArmdDivs, < 1 InfDivs)

70% CE(< 2 Mech/ArmdDivs, < 2 InfDivs)

100% CE (relto A-Day)(2+ Mech/ArmdDivs, 2+ InfDivs)

OBJ GOLDs/ STEEL Secured Minimal ReinfPossible2 x LOCsSeverely Disrupted

Minimal ReinfExecuted (< or = 1 Bde)1 x LOC Disrupted

SigReinfExecuted (2 or more Bdes)2 x LOCsOpen

100% DPREsreceiving min stds#s of IOs/NGOs > Q-Day

75% DPREsreceiving min stds#s of IOs/NGOs < 90% Q-Day

50% DPREsreceiving min stds#s of IOs/NGOs < 80% Q-Day

> 50% DPREsreceiving min stds#s of IOs/NGOs < 70% Q-Day

MEE >85% / DOS >80%0 casualties / damage from atk

Unfettered LCC FoM

MEE 70-85% / DOS 70 -80%Rear cas< 10% of Close Cbtcas

Damage mitigated < 24 hrsLCC FoMInhibited

MEE 60-70% / DOS 60 -70%Rear cas< 15% of Close Cbtcas

Damage mitigated < 48 hrsLCC FoMDifficult

MEE <60% / DOS <60% Rear cas> 15% of Close Cbtcas

Damage mitigated > 72 hrsLCC FoMDenied

No SigDamage of Infra0 non-combatant cas

Minimal Damage of InfraMinimal non-combatant cas

SigDamage of InfraSigincidof non-combatant cas

Long Term Damage of InfraSignon-combatant cas

As of: 130900 Nov 03

LG

JB

LA

JA

LF



POPP

ChernoffFaces


POPP

1)Confusion between objectives, activities, inputs, outputs, outcomes, ways, means, ends, ...2)What influences what,3)Are metrics dependent on the objectives or are objectives dependent on the metrics?4)Roll up the metrics or display for mental integration5)Whose toy should we use?

Several Issues


POPPBumper Sticker

Metrics should be Organic to the Plan


POPPValue Focused MetricsMetrics should be organic to the objectives and flow from the plan, not come from separate lists or be tacked on at the end.

Value Focused Metrics (VFM) says: First identify your objectives, then determine how each Fundamental Objective can be served by a network or Means Objectives. This is the ‘Model’ of the overall intervention/plan or mission.

Once the structure and content of the Model is agreed, then identify what the interveners can do or control – what activities, projects and programs need to be put in place, and what Means Objectives are these expected to influence.

Finally, when the Model is enhanced by the activities, then and only then determine what Metrics should be associate with both Fundamental and Means Objectives.

Keep track of resources, but they are controls on the activities, not metrics.


POPP

Some definitions from Keeney (and some from Davis):

1)Objective (Keeney, p34): “..statement of something that one desires to achieve.”

a) Contextb) Objectc) Preference

2)Goal (K60) (An objective with values, standards or thresholds.)3)Fundamental Objectives: “Just Because”4)Means Objectives: “In order to accomplish X”5)Attribute (K100): “The degree to which an objective is achieved..”6)Metric (Davis): The definition of an attribute7)Measurement (Davis): The collected value of a metric.

Definitions


POPPWhat the METRICS will mean

Measurements associated with Metrics on the Fundamental Objectives indicate overall Outcomes, or mission achievement.

Are we accomplishing the overall effects that we are there to create?

Measurements associated with Metrics on the Means Objectives indicate how well the plan is being accomplished.

Are we using the right ways and means to accomplish our goals?

Metrics on Fundamental Objectives are for Mission Achievement.Metrics on Means Objectives are for Managerial Efficiency.

BOTH ARE VALID AND NECESSARYBUT they ARE NOT THE SAME THING!!!!


POPPThe Process

The modeling process starts with developing a Fundamental Objectives Hierarchy (FOH) and a Means Ends Objectives Network (MEON). A MEON is a model of causality, it is meant to show how a Fundamental Objective (FO) can be met. To do so, there are several requirements:

1)A list of domain elements must be collected – what are the things in this domain that impact or may impact the FO? For the intervention domain we are using the ETL/MPICE/CMPO. This yields CONTENT.2)These domain elements must be arrayed in a network showing both causal and correlation linkages. We are doing this using Microsoft Belief Networks (a freeware from MS that implements Bayesian Networks). This process provides us with STRUCTURE.3)Once the CONTENT has been arrayed in the STRUCTURE then the theories of change must be added. In the case of Bayes Nets, this theory is one of probabilities. This then gives us VALUE.4)Drivers/Activities/Programs/Projects can then be added to represent what the interveners can actually CONTROL. Activities may require RESOURCES.5)Many, but not necessarily all, objectives can have attributes, or METRICS attached.


POPPSources of the Elements

Catalogs – MPICEWorkshops and expert opinionAffinity Lists (Parnell)Old models“The Plan”


POPPWhat's a what

For each element in your list, ask first:

1)Why is this important?a) “In order to do this or that” – Add to list of Means

Objectivesb) “Just because, or That is the reason we are here” –

Add to list of Fundamental Objectivesc) “Its something that we do” – add to list of potential

activities2)If a Means objective, what does it influence? (potential link)3)For every objective, fundamental or means: How would we know that this objective has been accomplished? Candidate attributes or metrics.

Then, of course, ask: What have we forgotten?


POPP

Fundamental Objective Hierarchy

Means Ends Objective Network

Controls, Activities,Programs, Projects

The System

Metrics

Metrics

Resources


POPPStructure - Basic

The Yellow node is the Strategic (Fundamental) Objective. The Gray nodes are theMeans Objectives linked to represent how we think they support each other.


POPPStructure - Influences

The Green nodes are activities, things we can actively control


POPPStructure - Metrics

The Blue nodes are the metrics – the metrics depend on the objectivesand do not form their own, independent system.


POPPValue

Value depends on classifying the objective’s possible states. In this case Needs can be in either of the states Provided, Marginal, or Significant need persist.

Value is completed by assessing the local theories (probabilities) at each node. For example, the probability that Needs will be Provide For GIVEN that Shelter is Sufficient AND Education is Available AND Access to Water is Sufficient is .897. Whereas the probability that Needs will be Provided For GIVEN that Shelter is Available Below Standards AND Education is Marginal AND Access to Water is Marginal is only .526.

These theories are based on input

assessments by experts.


POPPConclusion

Metrics should be Organic to the Plan


POPPQuestions?

Value Focused Metrics 26 ISMOR D.F. Davis A.D. Caldwell ddavis@gmu [email protected]

Documents

Transcript of Value Focused Metrics 26 ISMOR D.F. Davis A.D. Caldwell ddavis@gmu [email protected]