Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015
-
Upload
support-for-improvement-in-governance-and-management-sigma-oecd -
Category
Government & Nonprofit
-
view
182 -
download
0
Transcript of Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015
![Page 1: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/1.jpg)
1
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
Institutional development of the Office of the Minister of State for Administrative
Reform (OMSAR)
Introduction to M&E of Public Policies: concepts and methods
Jaime Blasco 7-8 December 2015
BEIRUT
![Page 2: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/2.jpg)
2
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
1. Observation/experience
2. Intuition/common sense
3. Theory
4. Eminences
5. Consensus
… good will
… very useful, but not an ex-ante guarantee
of success
On doubt and skepticism
![Page 3: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/3.jpg)
3
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
1. Policy success consists of significantly
improving social, economic or
environmental conditions
Stating the obvious
![Page 4: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/4.jpg)
4
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
2. Success should not be conceived in
terms of activity, satisfaction, survival,
budget, adherence to the rule,
professional recognition or good relations
with other actors (though they may be
crucial for success, they are not success)
Stating the obvious
![Page 5: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/5.jpg)
5
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
3. Success hinges upon policy design,
implementation and context
Stating the obvious
![Page 6: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/6.jpg)
6
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
4. Success is a hypothesis to be tested,
not an apriorism (even if you have
resources, consensus, experts, time to
reflect, etc.)
Stating the obvious
![Page 7: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/7.jpg)
7
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
5. Without hypothesis testing (evaluation) ,
it is difficult to improve policy design,
reform & management (blind flying)
Stating the obvious
![Page 8: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/8.jpg)
8
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
1. Public intervention is motivated by the existence of a
condition or problem
2. The program is expected to ameliorate the problem
(from A to B, being B better than A)
3. Program is expected to do so throughout a set of
mechanisms (not wishful thinking)
Program theory
![Page 9: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/9.jpg)
9
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
Needs
Inputs
Activities
Outputs
Impacts
(Outcomes)
So
cie
ty
Pu
blic
inte
rve
ntio
n
What is
needed
• Mosquito nets
• Transportation
What the
program does
Free mosquito
net distribution
The products
Nets
distributed
Short-, medium
and long-run
benefits
Reduction in
infant mortality
The problem
High infant
mortality in
región X
What for?
How?
Implementation failure Flawed theory Flawed diagnose
Program theory
![Page 10: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/10.jpg)
10
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
Program theory
![Page 11: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/11.jpg)
11
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
Program theory
ICT capital
expenditure
ICT operational
costs
Re-
organisation
and other
intangible costs
Deployment of
eHealth
applications in
1ary and 2ary
care (eHR,
telemonitoring,
ePrescription,
etc)
Availability of
eService for
citizens
(information,
consultation,
booking,
reimbursement)
Use of eHealth
applications by
professionals
Use of eHealth
applications by
citizens
Cases/
procedures/
transactions
handled online
Clinical
outcomes
Safety and
quality of
care
Input/output
efficiency
Spillover on
ICT sector
Health
capital
Healthcare
sustainability
eHealth
Misuraca et al. 2013
Type I outputs: supply
Type II outputs: use
Outcomes: usefulness Inputs: cost
Readiness Intensity Impact B>C
![Page 12: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/12.jpg)
12
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
Program theory
ICT capital
expenditure
ICT operational
costs
Re-
organisation
and other
intangible costs
Availability of
eServices for
citizens and
business
eService/
applications of
employees and
for cross-
government
exchanges
Re-use of open
data
Use eServices
Cases/
procedures/
transactions
handled online
Cost savings
Effectiveness
for
constituencies
Input/output
efficiency
Spillover on
ICT sector
GDP and
productivity
Good
governance
eGovernment
Misuraca et al. 2013
Open data
available online
for re-use
Trust in
government
and
participation
![Page 13: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/13.jpg)
13
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
Program theory
![Page 14: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/14.jpg)
14
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
Uses of program theory
1. Making program design and expected mechanisms
explicit
2. Focus on outcomes (“what for”)
3. Identify weaknesses-> evaluation questions
4. Planning the evaluation (indicators, sources)
5. Non evaluative uses: Program design, advocacy
Program theory
![Page 15: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/15.jpg)
15
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
Program theory
![Page 16: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/16.jpg)
16
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
Needs
Inputs
Activities
Outputs
Outcomes
Process evaluation
Economic evaluation
Impact evaluation
Needs assessment
Design evaluation
Evaluation questions
![Page 17: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/17.jpg)
17
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
Needs assessment:
1. How many unemployed workers lack basic skills on
ICT?
2. What are existing levels of computing and
communications infrastructure at the national colleges
3. What specific project management capabilities do
managers lack and need most?
Evaluation questions
![Page 18: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/18.jpg)
18
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
Process evaluation
1. Actual users correspond to the intended target
population?
2. What are municipalities doing with the subsidies?
3. Is the program covering all the regions?
4. What changes have introduced to the plan during the
implementation process? (what’s the actual policy
design?)
Evaluation questions
![Page 19: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/19.jpg)
19
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
Evaluation questions
Implementation gaps
• Displacement (micropolitics)
• Incompleteness
• Adjustment
“Designing by doing”
![Page 20: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/20.jpg)
20
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
Impact evaluation:
1. Do participation in direct employment schemes
increase the probability of finding and retaining a job?
2. Do tax deduction schemes for older workers increase
savings for retirement?
3. Does parole reduce crime recidivism?
Evaluation questions
![Page 21: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/21.jpg)
21
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
Economic evaluation:
1. Do savings from reduced administrative burdens
overcome costs of a new e-service?
2. Is care for the dependent elderly at home more cost-
effective than insitutionalisation?
Evaluation questions
![Page 22: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/22.jpg)
22
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
Design evaluation:
1. Do telecommuting schemes for public officers increase
productivity?
2. What features of a telecommuting scheme increase the
chances to increase public officers’ productivity?
Evaluation questions
![Page 23: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/23.jpg)
23
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
Design evaluation:
http://ies.ed.gov/ncee/wwc/
https://educationendowmentfoundation.org.uk
/toolkit/
http://webarchive.nationalarchives.gov.uk/2
0120919132719/http://communities.gov.uk/doc
uments/localgovernment/pdf/142922.pdf
Evaluation questions
![Page 24: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/24.jpg)
24
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
21.40%
17.10%
0.00%
5.00%
10.00%
15.00%
20.00%
25.00%
2010 2012
Before
After
Distrust in bureaucracy
2013 2015
The elusive search for impacts
![Page 25: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/25.jpg)
25
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
64.1%
53.9%
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
90.0%
100.0%
Beneficiàries No beneficiàries
Cit
izen
sat
isfa
ctio
n
Standard Procedure E-procedure
The elusive search for impacts
![Page 26: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/26.jpg)
26
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
21.40%
17.10%
0.00%
5.00%
10.00%
15.00%
20.00%
25.00%
2010 2012
Before
After
IMPACT
COUNTERFACTUAL
20,40%
The elusive search for impacts
2013 2015
![Page 27: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/27.jpg)
27
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
64.1%
53.9%
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
90.0%
100.0%
Beneficiàries No beneficiàriesAlumnes
74,0% COUNTERFACTUAL
Standard Procedure E-procedure
Cit
izen
sat
isfa
ctio
n
The elusive search for impacts
![Page 28: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/28.jpg)
28
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
• Policy impact may be expressed as the subtraction:
Impact = Y1 – Y0
Where:
• Y1 is the factual: outcomes attained after the public policy • Y0 is the counterfactual: outcomes that would have been
attained in the absence of the intervention
Estimating impacts implies causal inferences
The elusive search for impacts
![Page 29: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/29.jpg)
29
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
Y1
Program
≈
No program
=Y0
Y2
Counterfactual hypothesis
The elusive search for impacts
![Page 30: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/30.jpg)
30
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
Impacte
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
1T-2011 2T-2011 3T-2011 4T-2011 1T-2012
Regulation
What has happened
You
th la
bo
r m
arke
t p
arti
cip
atio
n
The elusive search for impacts
![Page 31: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/31.jpg)
31
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
Conclusions
1. “What has happened”, by itself, is useful to know, but tells us nothing about the program’s impact
2. “Nothing” is “nothing”: the very same evolution of a problem (“what has happened”), may entail that the impact has been positive, zero, or negative
3. “Impact” stems from the comparison of “what has happened” and what would have happened”
The elusive search for impacts
![Page 32: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/32.jpg)
32
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
Impacte
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
1T-2011 2T-2011 3T-2011 4T-2011 1T-2012
The program
Factual Counterfactual
Main characters
The impact
The outcome
... The decision The measurement
N
You
th la
bo
r m
arke
t p
arti
cip
atio
n
The elusive search for impacts
![Page 33: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/33.jpg)
33
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
1. There is no such thing as an “impact indicator”. Estimating impacts entails analysis
2. Most often, robust analysis requires planning the evaluation before the program starts off, in order to collect the data you need
3. Sometimes, it requires modifying certain program procedures (selection procedures, not necessarily criteria)
4. Some impact questions do not have a robust answer, since there is no feasible identification strategy
Consequences
The elusive search for impacts
![Page 34: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/34.jpg)
34
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
The elusive search for impacts
Perspective Time Information
about...
Pros & cons
Retrospective
(ex-post)
Present Past • Most common
• Data restrictions
• Mistakes already made
Ex-ante Present Future • Pre-decision forecast
• Policy design benefits from
evaluation rationale
• Uncertainties about the future
undermine robustness
Prospective Present Future
retrospective
evaluations
• Setting the stage for a future robust
evaluation
• Policy design benefits from
evaluation rationale
![Page 35: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/35.jpg)
35
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
Methods
Impact estimations designs
Is it possible to use a comparison group?
Random assignment?
YES
YES NO
NO
Social experiment Discontinuous regression
Before/after
Time series
Matching
Dif in dif
Instrumental variables
![Page 36: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/36.jpg)
36
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
Methods
Garicano & Heaton, 2013
“The data do not suggest that agencies that substantially increased IT
over the sample period had superior improvements in clearance rates
compared with those that made little IT adjustment”
![Page 37: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/37.jpg)
37
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
Methods
Garicano & Heaton, 2013
MPI= 1 if high
level of IT
+ high
specialization
+ highly skilled
workers
“An agency implementing a combination of IT, specialization and
skilled workers experiences a roughly 5% lower offending rate than
does an agency with similar levels of IT that has not implemented
these other practices.
[This] is consistent with the presence of complementarities”
![Page 38: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/38.jpg)
38
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
Methods
COMPSTAT NYC
0.0
5.0
10.0
15.0
20.0
25.0
30.0
35.0
Murder rate (/100.000 people)
![Page 39: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/39.jpg)
39
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
Methods
COMPSTAT NYC
Real time mapping of crime (IT)
But also…
• A statement of the measurable goals of the NYPD
• Internal accountability through weekly COMPSTAT meetings
(officers accountable for understanding & reacting, not for results)
• Decentralization of command (authority and resources to precinct
commanders)
• Empowerment of middle managers
• Data-driven problem identification and strategy design
• Innovative problem solving tactics
Weisburd et al., 2003
![Page 40: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/40.jpg)
40
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
Methods
Micro or macro evaluations?
Accuracy vs. Scope
• Each concrete “program” has its own goals and peculiarities
• Aggregate measurements&counterfactuals are difficult to find
• Impact evaluations are more feasible at the micro level
• But the micro level has a very limited scope
• Start small & gradually replicate & try to scale up (with a better
grasp of theory and measurement issues)
• Estimate impacts at the micro level, and monitor at the macro level
![Page 41: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/41.jpg)
41
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
From counterfactualism to pragmatism
Pragmatism
Counterfactualism
(experimentalism)
Pragmatism
Epistemology Successionist causation Knowledge validity depends on its
pragmatic acceptability
Method Experimental & quasi-experimental Enlighten policy using information
structured with best of breed
techniques
Ontology Social complexity coped with by
controlling of covariates
Social world as power-play & chain
of influence that research tries to
penetrate to promote change
Objects Programs as instruments to solve
social problems
Programs as institutional realities
(such as services, targets, clients)
Progress Replication to accumulate evidence
on what works
Incremental improvements within
the political feasibility of policy
development
Utility Inform policy making for rational
choices amongst alternatives on the
basis of robust evidence
Evaluation geared to advancing the
practical goals of policy making
Pawson &Tilley, 1997; Misuraca et al. 2013
![Page 42: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/42.jpg)
42
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
1. Outcome measurement & monitoring (factual description)
2. Implementation analysis
3. Process tracing of impact structures
4. Qualitative interpretation of facts
5. Case studies
6. Precedent evaluations in the literature
7. Triangulation of imperfect & incomplete evidence
Pragmatism
![Page 43: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/43.jpg)
43
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
The scale does not make you thinner
• Evaluation is instrumental: success is not a robust report, but an improvement in policy effectiveness, efficiency, legitimacy, equity...
• The step from knowing to learning, & from learning to action is anything but automatic
Ask questions, get answers, adapt
Adapt!
![Page 44: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/44.jpg)
44
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
• Involve key stakeholders & decision-makers in the evaluation process
• Make knowledge trickle down to the whole organization
• Keep available recorded and enhance institutional memory of knowledge accrued
• Make knowledge penetrate the policy process
Adapt!
![Page 45: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/45.jpg)
45
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
• Think of programs as hypothesis, and test them(evaluate them)
• Think of wicked problems as challenges: innovate, test, (often fail), learn and adapt
• Escape from the illusion of completeness. Plan
your analysis & prioritize programs, questions and outcomes.
• Do not get frustrated with incomplete & imperfect knowledge. Not being able to know everything does not mean that we should not know anything.
Closing remarks
![Page 46: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/46.jpg)
46
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
• Escape from the illusion of an evaluation algorithm, model or template. Evaluation requires ad hoc approaches
• Struggle for the best knowledge you can produce given your constraints: a car is better than a bike, but a bike is better than a crashed car (whatever you do, do it properly).
• Think ex-ante about the ex post evaluation:
small changes in datasets and program procedures from the beginning allow for a robust, fast and cheap evaluation in the future.
Closing remarks
![Page 47: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/47.jpg)
47
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
• Impact evaluation is a powerful tool, but has
strict requirements: well specified programs, tangible and measurable goals, a feasible identification strategy & good quality data.
• Do not think only of impact evaluation: there are other forms of evaluation and knowledge production that are useful for decision-making
Closing remarks
![Page 48: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/48.jpg)
48
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
• Ask not what you can do for your evaluation but what evaluation can do for you: look out there for robust, relevant and summarized knowledge about what works, but pay attention to external validity. Context matters.
Closing remarks
![Page 49: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/49.jpg)
49
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
• Use pilot programs to generate robust knowledge.
• Look for opportunity windows to generate robust
knowledge generation.
• Link evaluation with management, planning & decision-making. Knowledge is not an end by itself.
Closing remarks
![Page 50: Jaime Blasco, training, working sessions and retreat, OMSAR Beirut 7-11 December 2015](https://reader031.fdocuments.in/reader031/viewer/2022030315/588169771a28ab80508b7931/html5/thumbnails/50.jpg)
50
A j
oin
t i
nit
iati
ve o
f th
e O
EC
D a
nd
th
e E
uro
pe
an
Un
ion
,
pri
nc
ipall
y f
ina
nced
by t
he
EU
Evaluation roadmap • Integrate &improve administrative datasets for
evaluation use. • Evaluate ex-ante evaluations in the design &
reform of major policies • Standardise a common monitoring & reporting
system for all programs • Perform impact evaluations for the major
programs at least once every four years • Perform pilot programs with experimental
designs • Plan and prioritise evaluations • Empower the unit for knowledge production
coordination & management • Link knowledge to decision-making processes
Closing remarks