New frontiersEvaluation methods

Theory of changeProject cycle and risk management

Jesper Johnsøn, CMI, U4Bergen, February 4, 2014

Frontiers…

Current plateau – (cross) country-level, bribery, fraud

• Evaluations of specific reforms• Integration with risk management• Proxy indicators

Imbalances and the “missing middle” • Imbalances: – So far, resources have been invested in standardised

diagnostic and advocacy tools, not learning and evaluation• Diagnostics: follow trends – indicators, data, monitoring• Evaluation: what works – question, design and methods,

indicators, data, judgement

• Missing middle

Output Outcome Impact Individual Sector Country

Level of results

Indicator targets

When and how to evaluate corruption costs/anti-corruption effects?

Diagnose corruption risk (does corruption matter, are patterns changing?) If yes, prepare for evaluation (how and how much does corruption matter, how effective is anti-corruption?)1. Logic model: Think through how the initiative is

intended to create results by contributing to behavioral, organisational, political, or societal change, expressed in a result chain or theory of change.

2. Monitor and evaluate: Work on increasing the evaluability of the reform by shaping design, establishing indicators, systematic data collection, baselines, comparison groups

INDICATORS

EVALUATION QUESTION, DESIGN,

METHODS

PROBLEM/DIAGNOSIS

DATA

ANALYSIS, COMPARISON, JUDGEMENT,

CHANGE

Overview of types of evaluations

Impact evaluations Programme evaluations Process evaluations

Focus: cause-and-effect, attribution, impact

Methods: counterfactual, comparison groups, statistical analysis, randomisation

Can answer question: What would have happened in the absence of the programme?

Focus: effectiveness, contribution, outcomes

Methods: comparative analysis, cost-effectiveness/benefit analysis, benchmarking

Can answer question: Has the programme been effective in achieving its stated objectives?

Focus: efficiency, implementation, internal mechanisms, outputs

Methods: workflow analysis, compare performed vs planned activities and outputs.

Can answer question:Is implementation done according to plan, and do results look attainable?

Clear objectivesIndicators

Mixed methodsTriangulation

ControlsRandomisation

ComparisonsBaselines

Trends

Code of Conduct example, level of results

Code of Conduct example, building an impact story

Diagnostics, evaluation, risk management and the project cycle• Framework for holistic use of different tools• Shows interlinkages between diagnostics, risk management and

evaluation tools

Diagnostics, evaluation, risk management and the project cycle• http://prezi.com/osddirkyzzpw/?

utm_campaign=share&utm_medium=copy&rc=ex0share

The Proxy Challenge

Challenges we confront• Linked, but not inseparable:

– Measuring changes in corruption levels– Evaluating whether anti-corruption efforts are successful

• Progress on measuring bribery and financial fraud, but two overall indicator problems remaining:– How to measure other types of corruption (patronage,

conflict of interest, abuse of power, etc.)– How to present a measure of overall corruption levels in a

country, region, sector, or organisation not biased towards measurable types of corruption, and can show trajectories of change

Why proxies?

• Good, existing “direct” indicators for bribery and financial fraud (need better application of methods)

• Can capture “missing middle” – provide needed innovation for some types of corruption, to capture trajectory of performance, and measure behavioural change

“Proxy indicators are alternatives to direct indicators that more directly measure the

phenomenon under study but that may be hard to operationalize or require overly costly data

collection”

Criteria

Standard criteria for indicator development (SMART)• Reflect de facto changes and behavioral changes• Prioritize sensitivity to context over standardization• Measure specific types of corruption• Ensure that changes in corruption trends are

attributable to reforms

“To assemble a body of promising ideas…”

• Clearly define the proxy indicator (or indicators) you suggest as a good measure of anti-corruption results

• Identify the type of corruption measured by the proxy indicator • Clarify how the proxy indicator reflects changes in corrupt behaviours• Present ideas for how the validity of the proxy indicator can be tested • Describe any relevant geographic, sectoral, or institutional context• Explain how the indicator can be combined with other proxy and

nonproxy (direct) indicators to obtain a better measurement of overall anti-corruption progress

• Comment on the strengths and weaknesses of the proxy indicator and the feasibility of its use by aid agencies, governments, civil society, and others for the purposes of monitoring and reporting