Www.3ieimpact.org Philip Davies Using Evidence for Policy and Practice Philip Davies International...

www.3ieimpact.orgPhilip Davies

Using Evidence for Policy and Practice

Philip Davies

International Initiative for Impact Evaluation [3ie]

BCURE Evidence-Informed Decision-Making Capacity Building Workshop1st and 2nd June 2015Pretoria, South Africa


Instrumental UseInvolves acting on research results in specific, direct ways.

Conceptual Use Involves using research results for general enlightenment; results influence actions, but in less specific, more indirect ways than in instrumental use

Symbolic Use Involves using research results to legitimate and sustain pre-determined positions.

How Evidence is Used in Policy Making


Instrumental, conceptual and symbolic uses of evidence are not mutually exclusive, but can operate in different ways at different stages in the policy cycle and under different political contexts.

How Evidence is Used in Policy Making


• Tested four variants of a means-tested conditional cash transfer paid to 16–18-year olds for staying in full- time education.

• Two levels of payment (£30 and £40) to either the young person or a primary carer (usually the mother);

• Combined with different levels of a retention bonus (£50 and £80) and an achievement bonus (£50 and £140).

• Male and female young people, and in both urban and rural areas.

• Comparison groups using propensity score matching

The Educational Maintenance AllowanceInstrumental Use of Evidence


Case Study - The Employment Retention and Advancement (ERA) Demonstration

The Policy Issue:

• How to retain, and advance, low-pay/no-pay workers in the labour market

Evidence Gathering:

• Systematic review of the global evidence on labour market and social welfare participation

• Secondary analysis of existing primary data (surveys and admin)

• Stakeholder analysis and involvement

• Empirical testing in six different labour market areas using experimental design (randomised controlled trial)


Case Study - The Employment Retention and Advancement (ERA)

Demonstration• Tested the likely impact and cost-effectiveness of:

A post-employment adviser service, Cash rewards for staying in work and for

completing trainingIn-work training support

• Random assignment of over 16,000 people from six regions of Britain: ERA programme or a business-as-usual control group.

• Seven years to complete – a strategic development evaluation

• But milestone data in real time from Year 1


[Extreme] Summary of Outcomes

The ERA Demonstration Project

• ERA evaluation had a differential effect on different types of low-pay/no-pay workers [Anticipate heterogeneity of impacts]

• Initial impacts in Year1 were not sustained• Initial small impacts in Year 1 had larger impacts in Years 3

and beyond• The timing of impact measurement/monitoring is important• Requires regular built-in M&E


Barriers to the Use of Evidence

• Policymakers’ lack of familiarity with the research process

• Researchers’ lack of familiarity with the policy process

• Trust (lack of trust) of policymakers in researchers (vice versa)

• Timeliness and availability of evidence

• Physical access to evidence

• Cognitive access to evidence (i.e. lack of understanding)

• Lack of clarity in the presentation of evidence


• Interactions between researchers and policymakers increases the prospects for research use by policymakers (Lavis et al, 2005).

• Early and ongoing involvement of relevant decision makers increase research utilisation (Lomas, 2000)

• Identify and use interpersonal networks and interactions

• Identify willing and able knowledge brokers

• Separate strategic from operational demands for evidence

• Get policy makers to own the evidence – not just the policy

Overcoming Barriers to the Use of Evidence


• Establish what research says and does not say

• Establish the policy messages and policy implications

• Use a 1:3:25 format

• Be clear - plain English summary

• Be persistent and opportunistic

Improving Communication of Evidence


The ‘One’ in the 1:3:25 Format

• A one page of main messages

• The lessons decision makers can take from the research

• Not a summary of findings

• Implications of findings

• No details of methodology


•These are the key findings of the study

•The classic Executive Summary

•Condensed to serve the needs of the busy decision maker

•Focus on how the study will be useful for policy

•Some brief mention of methodology

•Some implications of policy and practice

The ‘Three’ in the 1:3:25 Format


The ‘25’ in the 1:3:25 Format

This should include:• Context/Background• Approach (Methodology in Appendices, not text)• Results• Implications• Knowledge gaps• References• Additional resources• Appendices


Group Exercise

• Identify networks of policy makers and evidence providers in your work environment who might help to overcome barriers to using evidence

• Consider what actions might need to be taken in your work environment to get evidence into policy/practice

• Consider how the capacity to find, appraise and use evidence might be developed in your work environment


Thank youPhilip Davies

Email: [email protected]+44 (0)207 958 8350

Visit www.3ieimpact.org

mailto:[email protected]

Www.3ieimpact.org Philip Davies Using Evidence for Policy and Practice Philip Davies International...

Documents

Transcript of Www.3ieimpact.org Philip Davies Using Evidence for Policy and Practice Philip Davies International...