Peace Operations Policy Program, School of Public Policy POPP Value Focused Metrics 26 ISMOR D.F....
-
Upload
cathleen-robbins -
Category
Documents
-
view
217 -
download
3
description
Transcript of Peace Operations Policy Program, School of Public Policy POPP Value Focused Metrics 26 ISMOR D.F....
Peace Operations Policy Program, School of Public Policy
POPP
Value Focused Metrics26 ISMORD.F. Davis
A.D. [email protected]
Peace Operations Policy Program, School of Public Policy
POPPVFMValue Focused Metrics is a term stolen from Keeney’s Value Focused Thinking.
This use is not unique and a simple Google search brings out 99 other uses. Most of these are in the context of “Agile Human Resources,” “Agile Systems Design.” and “Project Management.”
Similar uses have been around since the early 1900s, but mostly since the maturing of Decision Science in the 1960s .
Value Focused Thinking by Ralph Keeney came out in 1992.
Peace Operations Policy Program, School of Public Policy
POPPInsight
“The purpose of computing is insight, not numbers.” Richard W. Hamming while at IBM labs
“The purpose of analysis is insight, not numbers.” Wilbur Payne while at TRADOC Analysis Command
“Our purpose is insight, not metrics.” Dave Davis while at ISMOR
Peace Operations Policy Program, School of Public Policy
POPP
RTO-TR-081Page 5-1
NATO Code of Best Practices For Command and Control Assessment
Metrics should flow from the process, not be separated from it.
Peace Operations Policy Program, School of Public Policy
POPP
The Code of Best Practice has adopted the following five levels of MoM (Measures of Merit):
• Measures of Policy Effectiveness (MoPE), which focus on policy and societal outcomes;• Measures of Force Effectiveness (MoFE), which focus on how a force performs its mission or the degree to which it meets its objectives;• Measures of C2 Effectiveness (MoCE), which focus on the impact of C2 systems within the operational context;• Measures of Performance (MoP), which focus on internal system structure, characteristics and behaviour; and • Dimensional Parameters (DP), which focus on the properties or characteristics inherent in the physical C2 systems.
RTO-TR-081Page 5-2
COBP
Peace Operations Policy Program, School of Public Policy
POPP
RTO-TR-081Page 5-3
COBPThis figure implies that metrics are nested. I’m not sure this is always the case.
Peace Operations Policy Program, School of Public Policy
POPPMPICE
SectorsGoals
Drivers/CapacitiesIndicators
Measures
Peace Operations Policy Program, School of Public Policy
POPP
0.0000.1000.2000.3000.4000.5000.6000.7000.8000.9001.000
HSI B
asel
ine
Econ
omic
s
Econ
Driv
ers
Econ
Insti
tutio
ns
Soci
al W
ell B
eing
SWB
Driv
ers
SWB
Insti
tutio
ns
Politi
cal
Pol D
river
s
Pol In
stitu
tions
Rule
of L
aw
RoL D
river
s
RoL I
nstit
ution
s
Secu
rity
Secu
rity D
river
s
Secu
rity I
nstit
ution
s
Phase I and Phase II Overall
Phase I
Phase II
Peace Operations Policy Program, School of Public Policy
POPP
0.0000.1000.2000.3000.4000.5000.6000.7000.8000.9001.000
Phase I and II Sectors
Phase I
Phase II
Peace Operations Policy Program, School of Public Policy
POPP
MoE Road Map
Mission
Objectives
Effects
Tasks
Enabling Tasks
MoEMeasure of Effectiveness
MoPMeasure of Performance
MoSMeasure of Success
Mission Statement
Lines of Operation
Decisive Points
Effects BasedPlanning
PrimaryIndicators
Campaign Plan
SecondaryIndicators
Subjective
Objective
Measures
RICHARDSON, H, 2004, MOE AND EBO IN THE HEADQUARTERS ARRC, IN THE CORNWALLIS CROUP IX: ANALYSIS FOR STABILIZATION AND COUNTER-TERRORIST OPERATIONS, THE CANADIAN PEACEKEEPING PRESS (HTTP://WWW.THECORNWALLISGROUP.ORG), P83
Peace Operations Policy Program, School of Public Policy
POPP
Schematic Overview of Campaign PlanCampaign Assessment
PHASE III
Safe and Secure Environment
Coalition Coherence
Infrastructure/
Humanitarian Sp
DP4
DP5
DP9
DP11
DP16DP8
DP1 DP15
DP10 DP21
DP19
PHASE IV
DP3
Mission/Phase
Endstate
Political /Legal
DP7
DP6
DP14 DP18
DP20
DP2
DP12 DP13 DP17 DP22
LOPs Decisive Pointstimeline
Measures of Effectiveness
? MOS
RICHARDSON, H, 2004, MOE AND EBO IN THE HEADQUARTERS ARRC, IN THE CORNWALLIS CROUP IX: ANALYSIS FOR STABILIZATION AND COUNTER-TERRORIST OPERATIONS, THE CANADIAN PEACEKEEPING PRESS (HTTP://WWW.THECORNWALLISGROUP.ORG), P85
Peace Operations Policy Program, School of Public Policy
POPP
Phase 3 –Key Effects
ZNOG DEFEATED
ZNOG ISOLATED
HA OPS
PODs/LOCs
CONDITIONS FOR SSE
0% CE(Surrendered/Destroyed/W-drawn)
35% CE(< 1 Mech/ArmdDivs, < 1 InfDivs)
70% CE(< 2 Mech/ArmdDivs, < 2 InfDivs)
100% CE (relto A-Day)(2+ Mech/ArmdDivs, 2+ InfDivs)
OBJ GOLDs/ STEEL Secured Minimal ReinfPossible2 x LOCsSeverely Disrupted
Minimal ReinfExecuted (< or = 1 Bde)1 x LOC Disrupted
SigReinfExecuted (2 or more Bdes)2 x LOCsOpen
100% DPREsreceiving min stds#s of IOs/NGOs > Q-Day
75% DPREsreceiving min stds#s of IOs/NGOs < 90% Q-Day
50% DPREsreceiving min stds#s of IOs/NGOs < 80% Q-Day
> 50% DPREsreceiving min stds#s of IOs/NGOs < 70% Q-Day
MEE >85% / DOS >80%0 casualties / damage from atk
Unfettered LCC FoM
MEE 70-85% / DOS 70 -80%Rear cas< 10% of Close Cbtcas
Damage mitigated < 24 hrsLCC FoMInhibited
MEE 60-70% / DOS 60 -70%Rear cas< 15% of Close Cbtcas
Damage mitigated < 48 hrsLCC FoMDifficult
MEE <60% / DOS <60% Rear cas> 15% of Close Cbtcas
Damage mitigated > 72 hrsLCC FoMDenied
No SigDamage of Infra0 non-combatant cas
Minimal Damage of InfraMinimal non-combatant cas
SigDamage of InfraSigincidof non-combatant cas
Long Term Damage of InfraSignon-combatant cas
As of: 130900 Nov 03
LG
JB
LA
JA
LF
RICHARDSON, H, 2004, MOE AND EBO IN THE HEADQUARTERS ARRC, IN THE CORNWALLIS CROUP IX: ANALYSIS FOR STABILIZATION AND COUNTER-TERRORIST OPERATIONS, THE CANADIAN PEACEKEEPING PRESS (HTTP://WWW.THECORNWALLISGROUP.ORG), P87
Peace Operations Policy Program, School of Public Policy
POPP
ChernoffFaces
Peace Operations Policy Program, School of Public Policy
POPP
1)Confusion between objectives, activities, inputs, outputs, outcomes, ways, means, ends, ...2)What influences what,3)Are metrics dependent on the objectives or are objectives dependent on the metrics?4)Roll up the metrics or display for mental integration5)Whose toy should we use?
Several Issues
Peace Operations Policy Program, School of Public Policy
POPPBumper Sticker
Metrics should be Organic to the Plan
Peace Operations Policy Program, School of Public Policy
POPPValue Focused MetricsMetrics should be organic to the objectives and flow from the plan, not come from separate lists or be tacked on at the end.
Value Focused Metrics (VFM) says: First identify your objectives, then determine how each Fundamental Objective can be served by a network or Means Objectives. This is the ‘Model’ of the overall intervention/plan or mission.
Once the structure and content of the Model is agreed, then identify what the interveners can do or control – what activities, projects and programs need to be put in place, and what Means Objectives are these expected to influence.
Finally, when the Model is enhanced by the activities, then and only then determine what Metrics should be associate with both Fundamental and Means Objectives.
Keep track of resources, but they are controls on the activities, not metrics.
Peace Operations Policy Program, School of Public Policy
POPP
Some definitions from Keeney (and some from Davis):
1)Objective (Keeney, p34): “..statement of something that one desires to achieve.”
a) Contextb) Objectc) Preference
2)Goal (K60) (An objective with values, standards or thresholds.)3)Fundamental Objectives: “Just Because”4)Means Objectives: “In order to accomplish X”5)Attribute (K100): “The degree to which an objective is achieved..”6)Metric (Davis): The definition of an attribute7)Measurement (Davis): The collected value of a metric.
Definitions
Peace Operations Policy Program, School of Public Policy
POPPWhat the METRICS will mean
Measurements associated with Metrics on the Fundamental Objectives indicate overall Outcomes, or mission achievement.
Are we accomplishing the overall effects that we are there to create?
Measurements associated with Metrics on the Means Objectives indicate how well the plan is being accomplished.
Are we using the right ways and means to accomplish our goals?
Metrics on Fundamental Objectives are for Mission Achievement.Metrics on Means Objectives are for Managerial Efficiency.
BOTH ARE VALID AND NECESSARYBUT they ARE NOT THE SAME THING!!!!
Peace Operations Policy Program, School of Public Policy
POPPThe Process
The modeling process starts with developing a Fundamental Objectives Hierarchy (FOH) and a Means Ends Objectives Network (MEON). A MEON is a model of causality, it is meant to show how a Fundamental Objective (FO) can be met. To do so, there are several requirements:
1)A list of domain elements must be collected – what are the things in this domain that impact or may impact the FO? For the intervention domain we are using the ETL/MPICE/CMPO. This yields CONTENT.2)These domain elements must be arrayed in a network showing both causal and correlation linkages. We are doing this using Microsoft Belief Networks (a freeware from MS that implements Bayesian Networks). This process provides us with STRUCTURE.3)Once the CONTENT has been arrayed in the STRUCTURE then the theories of change must be added. In the case of Bayes Nets, this theory is one of probabilities. This then gives us VALUE.4)Drivers/Activities/Programs/Projects can then be added to represent what the interveners can actually CONTROL. Activities may require RESOURCES.5)Many, but not necessarily all, objectives can have attributes, or METRICS attached.
Peace Operations Policy Program, School of Public Policy
POPPSources of the Elements
Catalogs – MPICEWorkshops and expert opinionAffinity Lists (Parnell)Old models“The Plan”
Peace Operations Policy Program, School of Public Policy
POPPWhat's a what
For each element in your list, ask first:
1)Why is this important?a) “In order to do this or that” – Add to list of Means
Objectivesb) “Just because, or That is the reason we are here” –
Add to list of Fundamental Objectivesc) “Its something that we do” – add to list of potential
activities2)If a Means objective, what does it influence? (potential link)3)For every objective, fundamental or means: How would we know that this objective has been accomplished? Candidate attributes or metrics.
Then, of course, ask: What have we forgotten?
Peace Operations Policy Program, School of Public Policy
POPP
Fundamental Objective Hierarchy
Means Ends Objective Network
Controls, Activities,Programs, Projects
The System
Metrics
Metrics
Resources
Peace Operations Policy Program, School of Public Policy
POPPStructure - Basic
The Yellow node is the Strategic (Fundamental) Objective. The Gray nodes are theMeans Objectives linked to represent how we think they support each other.
Peace Operations Policy Program, School of Public Policy
POPPStructure - Influences
The Green nodes are activities, things we can actively control
Peace Operations Policy Program, School of Public Policy
POPPStructure - Metrics
The Blue nodes are the metrics – the metrics depend on the objectivesand do not form their own, independent system.
Peace Operations Policy Program, School of Public Policy
POPPValue
Value depends on classifying the objective’s possible states. In this case Needs can be in either of the states Provided, Marginal, or Significant need persist.
Value is completed by assessing the local theories (probabilities) at each node. For example, the probability that Needs will be Provide For GIVEN that Shelter is Sufficient AND Education is Available AND Access to Water is Sufficient is .897. Whereas the probability that Needs will be Provided For GIVEN that Shelter is Available Below Standards AND Education is Marginal AND Access to Water is Marginal is only .526.
These theories are based on input
assessments by experts.
Peace Operations Policy Program, School of Public Policy
POPPConclusion
Metrics should be Organic to the Plan
Peace Operations Policy Program, School of Public Policy
POPPQuestions?