Assessing the impact of research on policy: A review of the literature for a project ... ·...

Assessing the impact of research on policy: A review of the literature

1

Assessing the impact of research on policy:

A review of the literature for a project on bridging research and policy through

outcome evaluation

Final report with references and appendices, February 2008

Annette Boaz, Siobhan Fitzpatrick and Ben Shaw

This literature review was conducted as part of the Defra Research Contract WR0808

‘Bridging Research and Policy Through Outcome Evaluation.’ The contract was awarded to a

team formed by Technopolis, GHK, King’s College London and Policy Studies Institute.

For further information about the review, please contact:

Dr Annette Boaz (Lecturer in Translational Research, KCL): [email protected]

Ben Shaw (Senior Research Fellow, PSI): [email protected]

King’s College London

King’s College London is one of the top 25 universities in the world (Times Higher 2007) and

the fourth oldest in England. A research-led university based in the heart of London, King’s

has 19,700 students from more than 140 countries, and 5,400 employees. King’s has an

outstanding reputation for providing world-class teaching and cutting-edge research. The

College is in the top group of UK universities for research earnings and has an annual income

of approximately £400 million. An investment of £500 million has been made in the

redevelopment of its estate. King’s has a particularly distinguished reputation in the

humanities, law, engineering, and social, health and natural sciences.

Policy Studies Institute

The Policy Studies Institute (PSI) is one of the UK’s leading independent research institutes,

highly regarded for rigorous and impartial evaluation of policy in the UK and Europe. PSI’s

Environment Group seeks to undertake interdisciplinary, policy-relevant research of the

highest quality, which recognises the interaction at every level between the environment, the

economy and society generally. Environmental research at PSI employs a wide range of

research methods, from formal modelling and scenario building to various styles of policy

analysis and evaluation.

Acknowledgements

The authors are grateful to the following experts in the field who read and commented on an

earlier draft of this report: Fred Carden (International Development Research Centre), Steve

Hanney (Brunel University), John Holmes (Oxford University), Sandra Nutley (University of

Edinburgh), John Young (Overseas Development Institute), Louise Shaxson (Independent

Consultant advising Defra) and Molly Morgan (SPRU, Defra). We would also like to thank

Lesley Grayson for editing the report and Alan Gomersall for conducting the database

searches.


1

Contents

Title:

Page:

Research Summary 2

Literature Review 6

1 Background to the project 6

2 Methods 8

3 Mapping the literature 9

4 Using the literature to answer the review question 14

4.1 How do people do it? 17

4.2 What are the advantages and disadvantages of different

approaches?

20

4.3 Which methods are most effective? 23

4.4 Which offer value for money? 25

4.5 Are there any specific issues to consider when evaluating

the impact of research on environmental policy?

26

5 What the literature doesn’t tell us 27

6 Conclusions 28

7 Bibliography 31

8 Bibliography of background documents 48

7 Appendices 50

A Data Extraction Sheets 51

B Review Methodology 369

C Data Tables 374

D Approaches and Methods for Gathering Data 379

E RAPID Outcome Assessment 390

F HERG Payback Model 392


2

Research Summary

How do you (best) evaluate the impact of research programmes on policy?

This research summary is for Defra officials interested in evaluating the impact of

research on policy. There is a growing interest in such evaluation within the research

councils, international organisations, the European Union and academia, and a

literature review was therefore commissioned by Defra to inform thinking in this

field. It draws on 156 UK and international papers and reports: 58 (37%) sources

report on evaluations of the impact of research on policy, while others are more

reflective and descriptive about the evaluative process and the range of methods

available.

This is a field of practice currently experiencing rapid development and examples of

recent innovative practice are captured in the review. The identified evaluations are

consistent in their use of a mixture of approaches, often involving a combination of

different qualitative methods, including case studies, in-depth interviews and

documentary analysis. These are sometimes used in conjunction with more

quantitative methods such as surveys.

Examples from other European Environment Ministries

The Finnish Environment Ministry scores all the research projects it funds against a set of evaluation criteria. They are scored by both the project leader and the Ministry supervisor. Any differences arising are explored through interviews with project leaders. The Netherlands Environment Ministry conducted two surveys (using external evaluators) of all policy makers who had commissioned research in a particular year. The compulsory questionnaire asked how, and the extent to which, the research had been used, and was followed up with interviews.

Methods and approaches used to evaluate the impact of research on policy

• Qualitative methods: semi-structured interviews, documentary analysis, field visits and observations

• Quantitative methods: surveys, bibliometrics and patent/new technology tracking

• Panels and peer review • Workshops and focus groups • Process tracking: historical tracing, positive utilisation narratives, tracing

post-research activity and impact logs • Literature review • Network mapping and analysis


3

Practice varies considerably between different countries and policy contexts. For

example, in the field of international development, there is an emphasis on qualitative,

participatory evaluations focussing on learning and service improvement. The main

method used for evaluating the impact of European Union (EU) research programmes

is panel reviews. There have been a number of recent reviews exploring alternative

methods for evaluating EU investments such as bibliometrics, econometrics and social

analysis. However, there is continued emphasis on the value of panels; these other

methods are generally viewed as useful complements rather than replacements.

Although panel reviews attract criticism for their reliance on experts, they do create a

sense of ownership through enabling the participation of key individuals from

different countries. This is particularly important in the European context and

underlines the importance of selecting methods that are both fit for purpose and

appropriate to the needs of key stakeholders.

While many evaluations track forward from a research project or programme to look

at its impacts on policy, others track back from policy to identify research use in the

policy development process. Approximately half of the evaluations in the review are

explicitly informed by a conceptual framework or model of the research-policy

process. While there is much in the literature about the advantages and disadvantages

of different approaches, there is very little about their effectiveness (in terms of

capturing impact) or costs.

Examples of existing frameworks HERG – The Health Economics Research Group Payback model is the best known framework for evaluating research impact (on a range of areas including policy). It has been used by a range of organisations including the UK Economic and Social Research Council and the NHS. However, it is a comprehensive (and therefore potentially costly) framework that may not be applicable in all settings. RIF – The Research Impact Framework assesses policy impact around five dimensions: the level of policy making, the type of policy, the nature of policy impact, policy networks and political capital. This framework generates brief one page impact narratives that could be more attractive and applicable than HERG in applied research environments.


4

The toolbox of methods used in evaluating the impact of research on policy seems to

be relatively transferable to different policy domains. However, it is suggested that

environmental policy typically involves a wide range of stakeholders and that the

views of stakeholders need to be heard in the evaluation process. Questions regarding

the allocation of responsibility for conducting the evaluations should be addressed.

Some papers highlight the benefits of using a blend of internal and external

evaluation.

Eight key dimensions of best practice emerge from the literature analysis undertaken

for this study. These are phrased as key questions to be addressed in order to identify

the most effective approach to evaluating the impact of research on policy. The

literature stresses the benefits of addressing these questions early in the research and

development process. For example, good quality systems need to be in place in order

to capture and track the use of research. Practical constraints relating to time, budget

and skills will inevitably guide any evaluation of research and development impacts

on policy. Responses to the eight dimensions below can be used as a basis for

developing an evaluation framework that is tailored to fit the particular organisational

context, resources and requirements.

8 key questions when designing a research impact evaluation

1. What is your conceptual framework? 2. What are the outcomes of interest? 3. What methods will best explore the outcomes of interest? 4. How do you address attribution? 5. What is the direction of travel for the evaluation? 6. Is this a mixed method approach, providing scope for triangulation? 7. Will the methods selected capture context and the complexity? 8. When might be the best time to conduct the evaluation?

These need to be considered in the light of the time, skills and resources available for the evaluation.


5

The literature suggests that more effective evaluation in this field could be achieved.

For example, reports argue for the development and application of existing conceptual

frameworks (including, for example, HERG) and innovative methods for evaluation

(advanced bibliometrics, survey methods and network mapping). The development of

new conceptual frameworks and methods is also advocated. Future evaluations might

focus on assessing the impact of interventions designed to promote research use,

including knowledge brokers, networks, and broader linkage and exchange

programmes. We are still a long way from providing a convincing answer to the

question: does our investment in research and development make a difference?

About the review

The Centre for Evidence & Policy at King’s College London, and the Environment

Group at Policy Studies Institute conducted a literature review to explore methods for

evaluating the impact of research on policy outcomes that might be appropriate to the

Sustainable Waste and Resource Management and the Sustainable Consumption and

Production research programmes. The review is part of a larger study conducted in

partnership with Technopolis and GHK Consulting, and focused on English language

publications over the period 1987 to 2007. The search strategy included four

elements: database searches, web searches, citation tracking and expert contacts. The

sources comprised ten key databases, three library catalogues and thirty relevant

organisations. Citation tracking involved following up references and contacts

emerging from documents identified through the initial stages of the search, and

proved to be a particularly fruitful source of relevant material. The final part of the

search aimed to ensure the review captured the most up-to-date knowledge on impact

assessment. This was achieved through holding discussions with the broader study

team responsible for the evaluation, and through asking expert contacts for feedback

on the draft report and for their advice regarding the inclusion of information from

any unpublished papers or work in progress. A total of 351 papers were identified

through the search, of which 156 met the selection criteria for inclusion in the review.


6

Literature Review

1. Background to project

There is a growing interest in evaluating the impact of research. This interest can be

discerned in UK research councils (10,15,381), international organisations (3,37,145),

the European Union (46,55,56) and academia (45,108,139). Furthermore, it can be

seen in a wide range of different disciplinary and policy domains. Recent evaluations

have considered both the broad impacts of research and its specific impacts on policy.

With these developments has come an interest in a wider variety of measures of

impact, reaching out to assess the non-academic influence of research investments

(100,120,144).

The factors motivating organisations and individuals to consider research impact are

manifold. They include accountability, performance, promotion of organisational

achievements, learning and moderating between competing stakeholders or interests

(10, 14). The pressure for accountability is particularly clear in a number of recent

documents (15,103). Increased government investment in research and development

in the UK (69) has been accompanied by a new programme of independent reviews to

evaluate the quality and use of science in government departments.

In recent years Defra has invested in better understanding the role of evidence in

policy making, with a focus on the demand for evidence-based policy (Defra, 2006),

the use of existing research within the department (Taig, 2004), the need to monitor

and evaluate policy (Defra, 2007) and the potential for bridging the gap between

research and policy through collaboration and advisory committees (Ashridge, 2006).

However, analysis within and for Defra has primarily focused on the process of

managing the evidence base. It does not refer specifically to evaluating research

impact on policy making, other than to conclude that assessing the success of science

policy interaction is important but challenging (Scott et al, 2005). A recent strategy

document concluded: ‘In order to strengthen the evaluative aspects of the programme

and to measure our progress to date in commissioning and using research to support

policy, we have tendered an outcome evaluation...the project will explore good

1 Each paper included in the review has been given a number to avoid long references in the text. The numbers can be used to find both the references and the data extraction sheets for each study.


7

practices drawn from international experiences in evaluating impacts of research on

attaining policy outcomes. It will apply these experiences to the wider sustainable

consumption and production and waste and resources evidence programmes.’ (Defra,

2007:31)

The Centre for Evidence & Policy at King’s College London and the Environment

Group at Policy Studies Institute were commissioned by Defra to conduct a literature

review to explore and examine methods for evaluating the impact of research on

policy outcomes that might be appropriate to the Sustainable Waste and Resource

Management (SWRM) and the Sustainable Consumption and Production (SCP)

research programmes. The review is part of a larger study conducted in partnership

with Technopolis and GHK Consulting that will lead to a new evaluation framework

for assessing the impact of Defra-funded research on policy outcomes. The literature

review is designed to ensure that the development of the evaluation framework is

informed by the best available evidence.

The review aimed to answer the following question:


To answer this question, the following sub-questions were addressed:

• How do people evaluate the impact of research programmes?

• What are the advantages and disadvantages of different approaches?

• Which methods are most effective?

• Which methods offer value for money?

• Are there any specific issues to consider when evaluating the impact of

research on environmental policy?

The review questions reflect the priorities expressed by Defra in the original invitation

to tender document.

This report begins with a brief description of the methods used to conduct the review.

The next section describes a mapping exercise that categorises the literature around a


8

number of key variables. While mapping is a relatively descriptive process, it

generates insights into the research landscape of research impact evaluation. The next

two sections outline and discuss what can be learnt from the literature with regard to

the review sub-questions, and outline gaps in the literature relevant to these questions.

The final section returns to the main review question: How do you (best) evaluate the

impact of research on policy?

2. Methods

The review focused on English language publications covering a period of twenty

years (1987-2007) and sought to identify:

• Evaluations of the impact of research on policy

• Reflective pieces about evaluation of the impact of research on policy

• Descriptions of methods for evaluating the impact of research on policy

• Reviews of methods for more general impact evaluations

• Evaluations of the effectiveness/value for money of different approaches

Particular effort was made to identify papers from other countries and papers with

specific reference or relevance to environmental, waste and sustainable consumption

and production policy. However, the majority of the papers found through these

additional searches were related to agriculture policy, and those that did refer

specifically to SCP or SWRM did not meet the inclusion criteria for the study, largely

because they did not address any of the review questions.

The search strategy included four elements: database searches, web searches, citation

tracking and expert contacts. The sources comprised ten key databases, three library

catalogues and thirty relevant organisations. Citation tracking involved following up

references and contacts emerging from the documents identified through the initial

stages of the search, and proved to be a particularly fruitful source of relevant

material. The final part of the search aimed to ensure that the review captured the

most up-to-date knowledge on impact assessment. This was achieved through holding

sessions with the broader study team responsible for the evaluation, and through

asking expert contacts for feedback on the draft report and for their advice regarding

the inclusion of information from any unpublished papers or work in progress.


9

A total of 351 papers were identified through the search, of which 156 met the

selection criteria for inclusion in the review. A data extraction sheet was completed

for each included study, and these are attached as Appendix A.

A detailed account of the review methodology can be found in Appendix B.

3. Mapping the literature

The first stage of the analysis involved mapping and categorising the literature around

a number of key variables: topic, country of origin, the way in which the papers and

reports address the review question, and the methodologies used or advocated for

conducting impact evaluations of research and development on policy. The data tables

for the figures one to four are attached in Appendix C. Figures 5 and 6 are derived

from the methods table (Appendix D).

The majority of the papers were identified through the internet search and from the

reference lists in papers and reports. Personal recommendations also made a strong

contribution. Although only 27 papers were identified through the database searches,

this approach did provide access to papers not identified through other sources,

reinforcing the value of a mixed method search strategy.

Figure 1: Method of Identification

Referenced, 55

Recommended, 25

Database, 27

Internet, 45

Journal search, 4


10

Figure 2: Paper includes:

A reflective piece about evaluation of

the impact of research on policy,

74

An evaluation of the impact of research

on policy, 58

A description of method (s) for

evaluating impact of research on policy,

42

Evaluation of effectiveness/ value

for money of methods, 13

A review of methods for evaluating impact more generally, 24

Other, 15

This review particularly focused on evaluations of the impact of research on policy.

Of the 156 studies, 58 reported on empirical studies of the relationship between

research and policy, of which 42 were specific evaluations of the impact of research

on policy. Most of the remainder were reflections on (or descriptions of) methods or

approaches to impact evaluation.

Figure 3: Field

Science, 14

Environment, 26International

Development, 26

Utilization literature, 13

Evaluation literature, 38Health, 35

Policy Research, 4


11

The review identified a wide-ranging literature, reflecting a cross-sectoral interest in

the issue of research impact on policy. In particular, there are large health and

international development literatures as well as a generic literature on utilisation and

evaluation. Particular effort was made to identify papers with topical relevance to

Defra, resulting in the identification of 37 environmental and science papers.

Figure 4: Country of OriginOther,

13

USA, 38

UK, 61

The Netherlands, 9

Switzerland, 3

Finland, 4

Canada , 20

Australia, 8

The majority of the papers come from the UK, USA and Canada, with smaller

numbers from Australia, the Netherlands, Finland and Switzerland. Studies were also

identified from a diverse range of ‘other’ countries including Brazil, Denmark,

Indonesia, Norway and Spain. The spread of countries is likely to reflect, to some

extent, the focus on English language publications.


12

Figure 5: Approaches and methodologies

0

10

20

30

40

50

60

Tracin

g po

st-re

sear

ch a

ctivit

y

Impa

ct log

Positiv

e ut

ilizat

ion n

arra

tives

Networ

k ana

lysis

Paten

ts/ N

ew te

chno

logies

Histor

ical tr

acing

Teleph

one

inter

view

User e

valua

tions

Field

visit

Liter

atur

e re

view

Wor

ksho

p

Surve

ys

Peer/

pane

l rev

iew

Bibliom

etric

s

Docum

enta

ry a

nalys

is

Case

study

ana

lysis

Semi-s

tructu

red

inter

views

To

tal n

um

ber

of

pap

ers

Discussed

Used

A wide range of research methods and approaches were applied to the study of

research impact. The most frequently mentioned and used were semi-structured

interviews, case studies and documentary analysis. This is consistent with another

recent review of impact studies (65a). However, most papers mentioned more than

one method. Interestingly, while bibliometric analysis was frequently cited in

empirical studies, this method was rarely used to measure the impact of research on

policy2. By contrast, telephone interviewing was used almost as frequently as it was

discussed. There were many more references to, and uses of, qualitative methods than

quantitative methods. A short description of each method can be found in table 1

(Appendix D).

2 See Hanney at al (2005) for an interesting discussion of the use of bibliometrics as a method for assessing the impact of a set of research studies by Alberti and colleague on subsequent research and on clinical practice.


13

Figure 6: Frameworks for structuring and interpreting data

02468

101214161820

Balanc

ed S

core

card

Corre

lation

Mat

rix

Resea

rch

Impa

ct Fra

mew

ork

ROAMEF

Result

s-ba

sed

Man

agem

ent

Logic

al Fra

mew

ork A

ppro

ach

RAPID o

utco

me

asse

ssm

ent

Simula

tion

Episod

e Stu

dies

Social

Ana

lysis

Bench

mar

king

Outco

me

Map

ping

HERG Pay

back

Mod

el

Econo

mic

Analys

is

To

tal n

um

ber

of

pap

ers

Discussed

Used

Many different interpretation frameworks are discussed in the literature but only a

small number are actually used in impact evaluations. In particular, RAPID Outcome

Assessment, the HERG Payback Model and economic analysis are applied in

empirical studies (although the last normally forms part of a wider analysis).

Descriptions of the HERG and RAPID models are attached as Appendices E and F as

these provide particularly useful insights into the process of research impact

evaluation. A short description of all the frameworks can be found in Table 1

(Appendix D).

While mapping is a relatively descriptive process, it generates insights into the

research landscape on research impact evaluation and provides a valuable backdrop to

the more detailed analysis below.


14

4. Using the literature to answer the review questions

This section of the report outlines and discusses what can be learnt from the literature

with regard to the review sub-questions before addressing the key question: How do

you (best) evaluate the impact of research on policy? The literature review has been

particularly timely as it taps into a rich seam of knowledge at a time of rapid change

in evaluative practice. Many of the key papers identified have been published within

the last couple of years and describe attempts to develop and apply new approaches to

evaluating research impact.

The relationship between knowledge and decision making has been a matter of

interest for hundreds, if not thousands, of years (Shadish et al, 1991). Much can be

learnt about the general relationships between knowledge and policy from the wider

literature, and this has been summarised in a series of reports and papers published by

the Research Unit for Research Utilisation (38,109,138,139). The diverse models of

the relationship between policy and research have consequences for the evaluation of

research impact. For example, it has been acknowledged (Kothari et al, 2005) that a

more instrumental model of research use is closely allied with immediate use, while

conceptual or enlightened use might be expected to occur in an eclectic fashion over a

long period of time.

So what do we mean by impact? An evaluator may ask many different questions

about a project or programme; ranging from has it worked to why and how it has (or

has not) worked and whether or not it has been acceptable to participants. The

particular focus of this review is the impact of research upon policy. Multiple terms

are used to describe research impact, including outcomes, benefit, payback,

translation, transfer, uptake and utilisation (11,25,50). The terms are often used

interchangeably, although there have been some efforts to distinguish between

outcomes and impacts. For example, the International Development Research Centre

(IDRC)3 argues that it is helpful to think in terms of behavioural outcomes rather than

longer term policy impacts. These different terms refer to a shared interest in change

that lies beyond the research process and its primary outputs. They are concerned

3 http://www.idrc.ca/en/ev-1-201-1-DO_TOPIC.html


15

primarily with policy outcomes or impacts rather than outputs or process indicators.

Outcomes have been defined as:

The end results of a programme for the people it was intended to serve.

(Weiss, 1998: 8)

With the interest in outcomes comes problems with attribution. The key question for

evaluators is therefore the extent to which impacts can be attributed to the use of

specific research outputs.

Figure 7: Looking at outcomes: losing attribution

Ability to quantify and establish attribution reduces with number of steps from outputs

The process by which the research is done might also have an impact on policy. For

example, research in collaborative projects may have an impact prior to the

production of research outputs.

Many approaches remain primarily concerned with the quantity and quality of

research outputs rather than outcomes (9,22,23). For example, bibliometrics are used

to quantify research publications, while peer review panels consider the quality of the

research produced by projects and programmes (often in relation to generic indicators

of research quality rather than its fit to wider programme objectives) (22,92).

Attribution becomes more difficult as the evaluator travels away from the initial

Input

(Research funding)

Final outcome

(Outcomes from implementation)

Intermediate outcome

(Implementation of strategic policy)

Initial outcome (Inclusion in

strategic policy

document)

Output

(Research findings/ reports.)


16

outputs, to look at different types of impact within the policy process. This has been

described as going ‘through the door’ rather than ‘to the door’ of the policy maker

(144). Only a few studies have focused on how policy is made ‘behind the door’ and

how research might contribute to that process. One study distinguishes between the

use of research conclusions and research recommendations (20).

Others have promoted a shift in language, preferring to focus on influence rather than

impact. Carden (26) stresses the difficulties in ascertaining attribution as the research

is often intertwined with too many other factors by the time impact has occurred. He

encourages evaluators to focus on changes in behaviour as indicators of broader

change. Nutley et al make a similar point:

Research use may thus be more about transformation than straightforward

application. There may, however, come a point where such refinements are so

extensive that it is no longer legitimate to refer to this process as ‘research use’

at all. (Nutley et al, 2007: 59)

It is perhaps not surprising that the task of quantifying and tracking research outputs

(using techniques such as basic bibliometrics) has often been considered to be a more

deliverable and thus attractive task. Furthermore, recent developments in the practice

of documenting and assessing citations have improved the potential contribution of

bibliometrics. One study has attempted to track citations in policy and professional

guidance and the mass media (104) and there is a call for more sophisticated analysis

to move beyond, for example, zero order approximations by which all impacts are

awarded the same value (144).

The evaluation process should be driven by what is meaningful in terms of outcomes,

not by what is measurable. Whilst this review focuses on the difficult task of

measuring outcomes, methods for evaluating processes are also included as it is clear

that understanding outcomes requires an exploration of processes, barriers and

facilitators as well as the end results.


17

4.1 How do people do it?

The literature (97,125) argues that evaluators should be driven by the need to identify

methods which are the most ‘fit for purpose’. This is not a straightforward task. Some

methods are very flexible and lend themselves to a wide variety of evaluative tasks.

For example, semi-structured interviews can be used to track forwards from a piece of

research or a programme, or backwards from a policy document. They can be used

with a wide range of stakeholders to explore a diversity of issues. Other methods,

such as network analysis, have more specific applications. A few approaches,

including cost-benefit analysis, require a particular set of technical skills in order to be

successfully applied to impact evaluations (92).

The approach selected also needs to be sensitive to the context in which the evaluation

will be undertaken (108,139). There are some interesting differences between the

approaches to evaluating the impact of research on policy in different countries and

contexts. One critical factor is the outcomes of interest. In the field of health care

there has been a considerable volume of empirical work exploring research impact.

However, the main focus of this work has been the impact on clinical practice rather

than policy (34,45,114). By contrast, evaluations of the impact of scientific research

seem to focus more directly on either changes in science policy or on the impact of

research on technological innovation (e.g. through patent analysis) and also on

national and regional/sub-regional economic development (28,59). To some extent,

these differences reflect particular departmental responsibilities: while the Department

of Health is primarily concerned with delivering services through the National Health

Service, the Department of Innovation, Universities and Skills has a more facilitative

role in promoting the role of science in the UK economy. A particular value of the

Health Economics Research Group (HERG) payback model that has emerged from

health services research in the past few years lies in its encouragement of a more

systematic and transparent consideration of the outcomes of interest for each

evaluation. While it does not require all evaluations to consider all dimensions, it

helps to put the dimensions in context and to ensure that those selected are the ones of

critical interest.

A considerable amount of impact evaluation is conducted in the field of international

development, where the emphasis seems to be on qualitative, participatory evaluations


18

with a focus on learning and service improvement. Most of the evaluations use case

studies or mixed method approaches (1,30,50,87) and participation is encouraged,

primarily through workshops. While engaging potential users in the process is

complex and time consuming, Carden argues that it has the potential to increase both

relevance and use (26b4). A further distinction between evaluation in international

development and other domains is the use of field visits, where evaluators observe

practice on the ground (50,135). This can generate valuable insights, particularly

when triangulated with data from other sources.

The reliability for policy purposes of the more qualitative data collected in both

international development evaluations and other fields seems to vary. For example,

storytelling has been used to explore and explain impacts (1,127). While stories can

be based on research evidence, they can also be largely anecdotal and highly

subjective. The focus on constructing positive utilisation narratives (1,39,72) strays

dangerously close to the line between evaluation and promotion. International

development evaluations pose important questions about who conducts evaluations

(professional evaluators, programme staff, service users), stakeholder involvement in

the evaluation and the value of a mixed method, qualitative approach.

By contrast, the US conceptual literature reflects the requirement outlined in the US

Government Performance and Results Act for greater use of quantitative methods and

performance indicators in impact evaluations (9). However, commentators have

struggled with the applicability of modelling and economic analyses to the evaluation

of research impact (132). Zilberman and Heiman (146) developed an economic

formula to assess the net benefit of research that led to innovations, but acknowledged

the empirical challenges of obtaining actual estimates. They suggest that the most

important contribution of impact assessments of research to policy may be

preventative (i.e. to screen wasteful policy proposals). The lack of evidence available

(146), the quality of existing data and the uncertainties characteristic of the research

process (82) make methods such as cost-benefit analysis (used so widely in industry

to explore costs involved and benefits generated), extremely difficult to apply (63).

Empirical evaluations conducted in the US tend to demonstrate continued reliance on

4 An ‘a, b, c or d’ after the number denotes a paper added after the consultation with experts.


19

a mixed method approach to overcome the limitations of individual methodologies by

a process of triangulation (80).

Panel reviews are the main method used for evaluating the impact of European Union

(EU) research programmes (9). There have been a number of recent reviews exploring

alternative methods for evaluating EU investments such as bibliometrics,

econometrics and social analysis (48). However, the value of panels continues to be

emphasised and other methods have been advocated as a complement rather than a

replacement. Although panel reviews attract criticism for their reliance on experts,

they do help to build ownership through the participation of key individuals from

different countries within the EU. This is particularly important in the European

context and underlines the importance of selecting methods that are both fit for

purpose and appropriate to the needs of key stakeholders. While panel reviews are the

predominant approach in the EU, other methods have been applied. One example is a

large scale internet survey of researchers funded as part of the 5th Framework

programme and conducted in 2004 (62).

A European Commission report on different approaches to evaluating energy research

in EU countries (46) identified a shift in practice away from project management

assessments (e.g. project delivered on time and to budget) towards an interest in the

impacts on science, policy and industry. While the paper described diverse practices

within Europe (for example, in Denmark there is a focus on evaluating the use of

research within target groups) the study also identified common interests across the

EU, including the need to demonstrate the additionality of research investments and a

commitment to developing meaningful quantitative indicators.

Practice varies considerably between different countries and policy contexts. There

are common themes regarding: the interest in developing and applying new methods,

such as network analysis (5,48,80,88); new understandings of change,including the

increased focus on evaluating behaviours (42); the need to develop existing methods

to ensure they are fit for purpose, such as new developments in citation analysis

(29,106); and the promotion of mixed method approaches. However, for evaluation to

be relevant and usable in different contexts, some diversity is likely to remain in

evaluation practice. For example, there is an emphasis on stakeholder engagement in


20

evaluation in international development. Similarly, whilst the use of peer panels in EU

evaluations may be complemented in the future by a wider range of methods, they are

unlikely to be replaced given their valuable role in building ownership and consensus.

The evaluation of research impact therefore calls upon a diverse range of methods

including; semi-structured interviews, documentary analysis, field visits and

observations, surveys, bibliometrics, patent/new technology tracking, panels and peer

review, workshops and focus groups, historical tracing, positive utilisation narratives,

literature review and network mapping and analysis.

4.2 What are the advantages and disadvantages of different approaches?

The wide range of different approaches used when analysing the impact of research

on policy are listed in Table 1 (Appendix D). Their advantages and disadvantages are

included in the table, along with a short description of each method. These can be

grouped as follows:

• Qualitative methods, including semi-structured interviews, documentary

analysis, field visits and observations. These methods are used to generate

rich descriptive and explanatory data that can be used to look at both

utilisation pathways and the policy context in which utilisation is or is not

taking place. However, they are often time- and resource-intensive, and it is

difficult to generalise their findings.

• Quantitative methods including surveys, bibliometrics and patent/new

technology tracking. These methods are used to quantify the impact of

research. They are suitable for repeat analyses and comparisons, can be used

to manage large amounts of data and can be very cost-effective. However,

they are often difficult to use in the analysis of research impact on policy.

Survey response rates can be poor and bibliometric data usually focus on the

quantity (rather than quality) of research outputs rather than outcomes.

• Panels and peer review. This is a relatively flexible and cost-effective

approach to evaluation. Experts and peers bring status and credibility to the

process and build ownership of the findings. These methods tend to be used to

enhance the quality of the research rather than to assess policy impact.


21

Furthermore, panels are open to accusations of lack of objectivity and are

heavily reliant on the quality of their membership.

• Workshops and focus groups. This is an interactive, consensus building

approach, but is not suitable for all topics (for example sensitive topics that

generate polarised views) and can lack rigour and objectivity.

• Process tracking including historical tracing, positive utilisation

narratives, tracing post-research activity and impact logs. These methods

focus on exploring the utilisation pathway. They rely heavily on the quality

and relevance of existing records (or record keeping in the case of impact

logs), documents and recall of participants in the process.

• Literature review. This method is usually used as part of a mixed method

study to scope a topic and place the impact evaluation in its wider context.

• Network mapping and analysis. This approach seeks to construct realistic

accounts of the nature of relationships between knowledge and policy. Other

research methods are used to generate the data required for constructing maps

of the networks that exist between individuals and organisations.

Different methods and approaches can produce very different assessments of the

relative impact of research, thus reinforcing the value of mixed method approaches.

The individual methods listed above are rarely used in isolation. For example, case

studies usually involve a combination of qualitative and sometimes quantitative

methods to explore utilisation (98,107). Most of the papers identified promote the idea

of mixed method approaches to exploring research impact, whether as part of a case

study or not (6,146). In particular, they argue for a combination of qualitative and

quantitative approaches, acknowledging the different contributions the two

approaches can make.

In addition to selecting methods, there is some question about who should be involved

in conducting the evaluation. For example, user evaluations involve stakeholders in

producing the study, while expert and peer panels also represent a more interactive,

consensus building approach. These approaches have the advantage of building

ownership and fostering individual and organisational learning. However, while these

approaches can be cost-effective, there can be challenges in terms of quality,


22

objectivity and bias (although it is important to note that not all external reviews are

objective just as not all internal reviews are subjective). In a review of methods for

assessing research impact, Hovland (72) suggests that a strong evaluation would

include elements of both internal and external review. The potential of a mixed

approach is echoed elsewhere for practical and methodological reasons. For example,

it has been argued (144) that internal evaluators would be in a better position than

external evaluators to keep a record of impacts, both in the short term and over a

longer time period. External evaluators could then be called upon at a critical point in

time when impacts might be expected. Using both internal and external evaluators can

also bring different perspectives to bear on the evaluation (26b).

Most of the identified evaluation approaches can be used for either forward tracking

from a piece of research or backward tracking from a policy change or document,

although the former (104) is more common than the latter (68). Hanney et al (65a)

suggest that tracking forwards can identify a greater level of impact than tracking

backwards. They suggest that this might be due in part to the reliance on self-reported

data from lead investigators in forward tracking studies. However, there are some

examples where evaluators have sought to track in both directions in order to

construct a rich account of the relationship between research and policy. For example,

the RAPID framework developed and applied by the Overseas Development Institute

(71,82,87) draws upon episode studies of specific policy changes, case study analysis

of specific research projects, and outcome mapping approaches. Information is

collected using a range of methods including: literature review and documentary

analysis; workshops with key staff, partners and actors; stakeholder interviews to

triangulate workshop outputs; field visits; and a debriefing session to discuss initial

findings.

One of the challenges of applying standard research methods is that they have not

been developed for the particular purpose of evaluating the impact of research on

policy. The same can be said for some of the data interpretation frameworks,

including the various approaches to economic analysis, outcome mapping,

benchmarking, simulations, episode studies and social analysis. A number of project

management tools do include evaluation as a critical stage in the life cycle of a

project. These include log frames (49,117), ROAMEF (Rationale, Objectives,


23

Appraisal in detail, Monitoring, Evaluation and Feedback) (37,56) and outcome

mapping (52,87). The key strength of these approaches is that evaluation is embedded

within the project management process, thus ensuring that it is not seen as a separate,

stand alone activity, and is considered from the beginning of the project planning

process. However, these different approaches do not provide detailed guidance on

how to capture learning about processes and how to measure complex outcomes.

Thus, despite their useful elements (see Table 1 in Appendix D), there are

considerable challenges in applying these tools to the evaluation of research impact on

policy.

Adopting and/or adapting a conceptual framework or model of the relationship

between research and policy is, however, considered to be a valuable step in the

development of a plan for studying the impact of research upon policy (100), and a

number of models have been developed specifically for this purpose. These include

the Health Economics Research Group (HERG) payback model, RAPID outcome

assessment and the Research Impact Framework (RIF). For example, the HERG

payback model explores five different impact domains: knowledge production,

research targeting and capacity building, informing policy and product development,

health (and health sector) benefits and broader economic benefits. There is much to be

learnt from this approach, particularly as it has now been applied to a wide range of

policy fields, such as health services and labour market research (65,144). The

application of revised citation analysis techniques and a mixed method approach,

combined with a clear conceptual framework (often used alongside a logic model to

assess outputs and outcomes), makes the approach attractive in different countries and

contexts. The RIF assesses policy impact around five dimensions: the level of policy-

making, the type of policy, the nature of policy impact, policy networks and political

capital. This framework generates brief one-page impact narratives that could be more

attractive and applicable in applied research environments (84a). Brief details of each

of these models can be found in Table 1 (Appendix D). A one page description of the

RAPID and HERG approaches can be found in Appendix E and F.

4.3 Which methods are most effective?

Only 14 studies reflect directly on the effectiveness of different methods for

evaluating the impact of research, and in only one of these was the effectiveness of


24

different approaches a central concern. Hanney et al (63) compared survey methods

and case studies as methods for evaluating the impact of research investments. They

conducted an empirical comparison of questionnaires and case studies, and found that

questionnaires could provide a relatively robust approach to monitoring a broad

research portfolio, but would benefit from a small number of accompanying case

studies that would help to verify the information given and to test and improve the

questionnaire for future use. This analysis was repeated with similar results as part of

an evaluation of the impact of the NHS HTA programme (65a). The comparative

study (63) emphasises the need to develop realistic approaches that fit with the scope

of the evaluative task and the resources available.

In the remaining 12 papers, observations were generally limited to short paragraphs of

text in the background or conclusions sections. These papers are consistent in their

analysis, concluding that in order to gain meaningful insights into research impact,

evaluators need to adopt in-depth, exploratory methods such as semi-structured

interviews and case studies (143). Williams and Rank (142) believe case studies

represent the best method for evaluating the impact of research and development,

while Jones et al (77) conclude that there is no substitute for interviewing individuals,

particularly within the policy arena. However, while agreeing that evaluators need to

go beyond the use of surveys and quantitative indicators in appraising research

impact, Anderson (6) argues that in-depth case studies can be prohibitively expensive.

Kilpatrick et al (79) consider the costs and benefits of a range of approaches to

measuring research impact, noting that methods have been appropriated from

evaluations in other fields or from more general social science research. They find

that none of the ‘borrowed’ methods completely meets the theoretical needs of

analysts assessing research impact, and conclude that while use of a combination of

methods may overcome individual shortcomings, this is likely to be a more costly

option. Other commentators also warn against the use of single methods, in particular,

bibliometrics and econometrics (29) and argue that there is a need to find new (100)

or ‘hybrid’ (143) methodologies


25

4.4 Which offer value for money?

The review found very little data on the value for money offered by the different

approaches identified. At most, papers include short paragraphs reflecting on the

methods used and their acceptability to participants. Typically (as with other

research), the costs of conducting impact studies are not reported. The general theme

emerging from reflective accounts seems to be that the most cost-effective and

practicable approaches, such as surveys, are the least likely to yield insights into the

process of research utilisation (particularly when conducted in isolation). There is a

suggestion in one study (36) that telephone surveys might offer a cost-effective

‘middle ground’ between postal surveys and in-depth interviews: participants

concluded that telephone interviews had offered an efficient use of their time as

respondents, and had additional value as a method for identifying individuals for in-

depth interview.

Two papers (38,63) in particular, conclude that there is unlikely to be sufficient time

or resources to consider the impact of a whole portfolio of research. Evaluators will

have to consider both issues of sampling (i.e. which research to focus on from within

a programme) and the costs of potentially appropriate methods. For example, Hanney

et al (63:189) conclude that while in-depth case studies generate valuable insights,

they are ‘inevitably fairly resource intensive and would not be appropriate for

extensive surveying or regular monitoring of an R&D portfolio’.

Beacham et al (11) also suggest that the comprehensive approach supported by many

commentators may well be too complex and expensive relative to the potential

benefits, given the intricacy of the evaluative task. In this respect, it is perhaps worth

keeping in mind some of the quicker, well established and more cost-effective

methods, such as using an expert panel to provide an informed opinion about impact,

developing a file to record personal testimonies, or developing proxy measures of

impact. Davies and Dart (39) suggest consideration of the ‘most significant change’

technique, which focuses on collecting stories at the field level for discussion by a

panel and collation in a report. Overall, the frequent recommendation to use a variety

of evaluation methods is likely to add to the cost of impact studies. The suggestion

that new methods need to be devised also has cost implications, given that

methodological development work is time- and resource-intensive.


26

4.5 Are there any specific issues to consider when evaluating the impact of research

on environmental policy?

The toolbox of methods used in evaluating the impact of research on policy seems to

be relatively transferable to different policy domains. Hanney et al have conducted

significant work in applying their model (HERG payback) to different policy

domains. Based on their conclusions, two issues of transfer emerge. The first concerns

the nature of the policy outcomes to be measured. Much environmental policy is less

concerned with service delivery than with changes in attitudes and behaviour. Unlike

health, there is no clear service sector or service delivery component to policy

development. A second dimension concerns politically charged topics. In some policy

areas, evaluators need to be very sensitive to the status of the topic in political terms.

Some key debates may not feature in the paper trail of publicly available policy

documents or may not be comfortable topics for policy makers to discuss. Finally,

some areas are clearly more mature in policy terms and evaluators might therefore

expect to identify more policy activity. Sustainability is a particularly challenging area

of policy development, given its relative youth, its political profile and its remit to

change ‘hearts and minds’ as a driver for change rather than focusing directly on

providing services. These issues have more implications for the conceptual model of

the policy process used to guide the evaluation, rather than for the methods used

(although the choice of model is likely to have methodological implications).

Kivimaa et al (79a) suggest a further dimension of environmental policy that has

implications for the evaluation of research impact. They argue that environmental

policy typically involves a wide range of stakeholders and that the views of these

stakeholders need to be heard in the evaluation process. They also highlight the inter-

and trans-disciplinary nature of environmental research that needs to be taken into

consideration in the design of research impact evaluation. Finally, a review of 33

organisations (70a) gave two specific examples of evaluations of environmental

research impact on policy. Projects funded by the Finnish Environment Ministry are

scored against a set of evaluation criteria by both the project leader and the Ministry

supervisor. The Netherlands Environment Ministry has conducted two surveys (using

external evaluators) of all policy makers who had commissioned research in a


27

particular year. The compulsory questionnaire included questions on how, and the

extent to which, it had been used. The questionnaire was followed up with interviews.

5. What the literature doesn’t tell us

The review found very few papers focusing on the costs and effectiveness of different

approaches to evaluating the impact of research on policy. Only one study focused on

comparing the effectiveness of two popular methods (case studies and survey

methods). However, authors were more reflective about the advantages and

disadvantages of different approaches and this information has been distilled in Table

1 (Appendix D).

The literature also identified fewer than expected empirical evaluations of the impact

of research on policy. It is unclear whether this reflects a low level of activity, a

failure to publish evaluations or the reporting of much of this activity in the grey (non-

academic, report) literature that can be difficult to identify. The extensive internet

search conducted as part of the review did provide access to grey literature so low

numbers and lack of publication of evaluations may be more likely explanations.

Those concrete examples of evaluations that were identified generated powerful

insights relevant to this review and this literature is anticipated to grow in the future.

At present, the majority of studies reflect on the relationship between research and

policy rather than the impact of research on policy and, although this is an interesting

and insightful literature, it was not the focus of this review.

Finally, while there is considerable reflection in the literature on the need to develop

and test new methods for evaluating the impact of research on policy, or to modify

methods used in other contexts, there are fewer empirical examples. To some extent,

new methods are likely to be needed to reflect changing conceptual models of the

relationship between knowledge and policy. For example, Gabbay et al (2003) have

mapped the relationship between knowledge and practice as a mass of circles and

boxes connected by complex webs of lines and arrows, which reflects the more

realistic view of the non-linear processes of research influence and uptake, and

represents a world apart from the neat circular and linear ROAMEF and log frames

models. Innovations in technology, such as mapping software, make it possible to at

least attempt to capture the complexity of policy making on paper. This is a field in


28

transition and there may well be applications of innovative methodological practice in

development.

6. Conclusions

How do you (best) evaluate the impact of research on policy?

Although the literature is thin on empirical examples, it is full of reflective pieces.

This is a field of practice currently experiencing rapid development and examples of

innovative practice have recently been published (21,41,144). For example, new

technological developments will now support more complex approaches to

bibliometrics and network analysis. E-mail (79a) and the web (62) offer alternative

vehicles for contacting and surveying participants. Eight key dimensions of best

practice emerge from the literature analysis undertaken for this study. These are

phrased as questions that need to be addressed in order to identify the most effective

approach to evaluating the impact of research on policy. The literature stresses that

the earlier these questions can be addressed in the research and development process,

the better the evaluation will be (97,121). For example, good quality systems need to

be in place in order to capture and track the use of research. Practical constraints in

terms of time, budget and skills will inevitably guide any evaluation of research and

development impacts on policy. Responses to the eight dimensions below can be used

as a basis for developing an evaluation framework that is tailored to fit the

organisational context, resources and requirements.

1. What conceptual understanding of the relationship between knowledge and

policy is informing the evaluation? One way of thinking about this is to

consider the current balance between different types of research use –

enlightened, instrumental, interactive and political/tactical research (Weiss,

1979) – within the organisation and how this relates to where the organisation

would like to be.

2. What are the outcomes of interest? For example, the HERG impact

dimensions, to the door/through the door, positive and negative uses? To what

extent is the focus on internal policy use or on a wider network of potential

uses and users? For SCP these outcomes might be about changes in knowledge

and attitudes rather than informing legislation or introducing new services.


29

3. What methods might be used to explore the outcomes of interest? For

example:

• achieving impact on policy documents (documentary analysis, citation

analysis)

• success of networks (network mapping)

• being responsive to policy needs (interviews, documentary analysis)

• involving policy makers in agenda setting (interviews, documentary

analysis, workshops)

4. How does the evaluation address issues of attribution? Defra has expressed a

particular interest in capturing the policy outcomes of research (as opposed to

the outputs of the research process). However, there are implications for the

degree of confidence it is possible to have in the results (see Figure 7). It is

therefore critical that qualitative methods are used to understand the pathways

and processes, and to describe perceived outcomes alongside attempts to

develop meaningful outcome measures.

5. What is the direction of travel for the evaluation?

• Backwards from policies/policy makers?

• Forwards from research reports/research programmes/researchers?

• Both directions?

6. Is this a mixed method approach, providing scope for triangulation? Are

• The persuasive power of metrics, and

• The explanatory power of qualitative methods

used to best effect?

7. Will the methods selected capture the context and complexity of the research

utilisation pathways, therefore helping to understand how (and whether)

change has occurred? Will they capture changes in behaviours and

relationships? There are valuable insights to be gained from the wider

evaluation literature. For example, Theory Based Evaluation builds evaluation


30

from the logic behind a project or programme (Weiss, 1995) and Realistic

Evaluation argues that a greater focus on understanding contexts and

mechanisms will lead to more meaningful appreciation of outcomes. (Pawson

and Tilley, 1997).

8. Does the timing of the evaluation offer sufficient time for change to occur,

without compromising the likely recall capacities of respondents? Are there

mechanisms that might be put in place to look at outcomes in the more

immediate and longer terms? Is the evaluation conducted after a policy change

has occurred (ex post) or is it predictive of potential use (ex ante)?

Demand for a more rigorous demonstration of the impact of research and

development investments is growing. A National Audit report found:

Evaluation of the quality of the research process is well established. However,

there is a strong and developing emphasis on evaluation to encompass research

relevance and value for money, as the link between research results and policy

formulation increasingly becomes the focus of attention. (103: 2)

The evaluations identified in this review are consistent in their use of mixed method

approaches, often using a combination of different qualitative methods such as in-

depth interviews and documentary analysis. Approximately half of these are explicitly

informed by a conceptual framework or model of the research-policy process. The

descriptive and reflective papers argue that better evaluation in this field could be

achieved. They argue for the development and application of conceptual frameworks

(including, for example, HERG and RAPID but also encouraging the development of

new models) and innovative methods for evaluation (advanced bibliometrics,

developments in survey methods, network mapping and again encouraging the

development of new approaches). Future evaluations might try to assess the impact of

interventions designed to promote research use, such as knowledge brokers, networks,

and broader linkage and exchange programmes. We are still a long way from

providing a convincing answer to the question: does our investment in research and

development make a difference?


31

Bibliography

1. Acosta A and Douthwaite B (2005) Appreciative Inquiry: an approach for learning and change based on our own best practices, ILAC Brief 6, Rome: Institutional Learning and Change. http://www.cgiar-ilac.org/downloads/Briefs/Brief6Proof2.pdf.

2. Adamo A (2002) Strategic evaluation of policy influence: what

evaluation reports tell us about public policy influence by IDRC-supported research. Ottawa: International Development Research Centre. http://www.idrc.ca/uploads/user-S/10359910880strategic.pdf.

3. Adamo A (2003) Influencing public policy through IDRC-supported

research: synthesis of document reviews. Ottawa, International Development Centre, Evaluation Unit. http://www.idrc.org.sg/en/ev-31926-201-1-DO_TOPIC.html.

4. Allen P, Peckham S, Anderson S and Goodwin N (2007)

Commissioning research that is used: the experience of the NHS Service Delivery and Organisation Research and Development Programme. Evidence & Policy 3(1): 119-134.

5. Almeida C and Bascálo E (2006) Use of research results in policy

decision-making, formulation, and implementation: a review of the literature. Cadernos de Saúde Pú blica 22(Suppl): S7-S19. http://www.scielosp.org/pdf/csp/v22s0/02.pdf.

6. Anderson R (2006) The policy impact of population health surveys: an

illustration of the measurement challenges using the NSW Health Survey. Evidence & Policy 2(2): 167-183.

7. Annema J A, van Wee B, van Hoek T and van der Waard J (1999)

Evaluation of Dutch public investment plans. Environmental Impact Assessment Review 19(3): 305-317.

8. Arnold E (2004) Evaluating research and innovation policy: a systems

world needs systems evaluations. Research Evaluation 13(1): 3-17. 9. Arnold E, Muscio A and Zaman R (2005) Reviews reviewed: lessons

from the first six international panel reviews 1999-2004,. Brighton: Technopolis, for the Engineering and Physical Sciences Research Council. http://www.epsrc.ac.uk/CMSWeb/Downloads/Publications/Other/InternationalReviewEvaluation.pdf.


32

10. Arts and Humanities Research Council (2007) AHRC impact strategy.

Bristol: Arts and Humanities Research Council. http://www.ahrc.ac.uk/images/AHRC_impact_strategy.pdf.

11. Beacham B, Kalucy L and McIntyre E (2005) Understanding &

measuring research impact. FOCUS on... (2) 12pp. http://www.phcris.org.au/publications/focuson/index.php.

12. Bekker M P M, Putters K and van der Grinten T E D (2004) Exploring

the relation between evidence and decision-making: a political-administrative approach to health impact assessment. Environmental Impact Assessment Review 24(2): 139-149.

13. Berryman J (2006) What defines 'enough' information? How policy

workers make judgements and decisions during information seeking: preliminary results from an exploratory study. Information Research 11(4) unpaged. http://informationr.net/ir/11-4/paper266.html.

14. Big Lottery Fund (2005) Increasing the impact of research, evaluation

and learning. London: Big Lottery Fund. http://www.biglotteryfund.org.uk/er_eval_increase_impact_report.pdf.

15. Biotechnology and Biological Sciences Research Council (2005)

BBSRC strategy for evaluating research programmes, SB 44/2005. Swindon: BBSRC. http://www.bbsrc.ac.uk/organisation/structures/boards/strategy/2005/0511_evaluating_research.pdf.

16. Boaz A (2006) Systematic reviews as a source of evidence for policy:

an in-depth exploration of a review on mentoring. PhD Thesis. London. 17. Bouma J and Droogers P (2007) Translating soil science into

environmental policy: a case study on implementing the EU soil protection strategy in the Netherlands. Environmental Science and Policy 10(5): 454-463.

18. Bozeman B (2003) Public value mapping of science outcomes: theory

and method. A monograph of the Public Value Mapping Project of the Center for Science, Policy and Outcomes. Washington, DC: Columbia University, Center for Science, Policy and Outcomes, and Atlanta: Georgia Tech, School of Public Policy. http://www.cspo.org/home/cspoideas/know_flows/Rock-Vol2-1.PDF.

19. Brehaut J D and Juzwishin D (2005) Bridging the gap: the use of

research evidence in policy development, HTA Initiative 18. Edmonton: Alberta Heritage Foundation for Medical Research. http://www.ihe.ca/documents/hta/HTA-FR18.pdf.

20. Brofoss K E (1998) The Research Council of Norway's use of research


33

evaluation: an assessment of research evaluation as a strategic tool. Research Evaluation 7(3): 134-140.

21. Bumgarner R, Hattaway D, McGann J and Wise H (2006) Center for

Global Development: evaluation of impact. Washington, DC: Arabella Philanthropic Investment Advisors. http://www.arabellaadvisors.com/CGDreport/CGDWeb.pdf.

22. Butler L and McAllister I (2007) Metrics or peer review? Evaluating

the 2001 UK Research Assessment Exercise in political science. Canberra: Australian National University, Research School of Social Sciences. http://repp.anu.edu.au/papers/2007_ukresearchassess.pdf.

23. Buxton M, Hanney S, Packwood T, Roberts S and Youll P (2000)

Assessing benefits from Department of Health and National Health Service research & development. Public Money & Management 20(4): 29-34.

24. Campbell S, Benita S, Coates E, Davies P and Penn G (2007) Analysis

for policy: evidence-based policy in practice. London: Government Social Research Unit. http://www.gsr.gov.uk/downloads/resources/pu256_160407.pdf.

25. Carden F (2004) Issues in assessing the policy influence of research.

International Social Science Journal 56(179): 135-151.

26. Carden F (2005) Making the most of research: The influence of IDRC-supported research on policy processes, Paper for the International conference 'African Economic Research Institutions and Policy Development: Opportunities and Challenges', Dakar, January 28-29 2005. http://www.idrc.ca/uploads/user-S/11085518871Making_the_Most_of_Research.pdf.

26a. Carden, F (2007) Context Matters – the influence of IDRC-supported

research on policy processes, in Ayuk, E T and Marouani, M A (2007) The policy Paradox in Africa: Strengthening links between economic research and policymaking. New Jersey: Africa World Press and Ottawa: International Development Research Centre:

26b. Carden, F (Forthcoming, 2008) Using comparative data: A systems

approach to a multiple case study, chapter for a book on case methods 26c. Carden, F (2008) The Episode Studies Approach. in Trussie, D The

Politics of Trade: the role of research in trade policy and negotiation. Amsterdam and Ottawa: Martin Nijhoff-Brill and International Development Research Centre.

26d. Carden, F (Forthcoming, 2008) Knowledge to Policy:Making the most

of development research. Ottawa: Sage & International Development Research Centre (IDRC)


34

27. Cash D and Clark W (2001) From science to policy: assessing the assessment process, Faculty Research Working Paper RWP01-045. Cambridge, MA: Harvard University, John F Kennedy School of Government. http://belfercenter.ksg.harvard.edu/publication/by_type/working_paper.html.

28. CGIAR Science Council (2006) Impact assessment of policy-oriented

research in the CGIAR: a scoping study report. Rome: CGIAR Science Council Secretariat. http://www.sciencecouncil.cgiar.org/publications/pdf/CG_SC_PORIA-pr2(enabl).pdf.

29. Council for the Humanities, Arts and Social Sciences (2005) Measures

of quality and impact of publicly funded research in the humanities, arts and social sciences, CHASS Occasional Paper 2. Canberra: Council for the Humanities, Arts and Social Sciences. http://www.chass.org.au/papers/pdf/PAP20051101JP.pdf.

30. Christiansen K with Hovland I (2003) The PRSP initiative:

multilateral policy change and the role of research, Working Paper 216. London, Overseas Development Institute. http://www.odi.org.uk/publications/working_papers/wp216.pdf.

31. Church C (2005) Mind the gap: policy development and research on

conflict issues. Londonderry: University of Ulster, International Centre of Excellence for Conflict and Peace Studies. http://www.incore.ulst.ac.uk/policy/rip/RIP.pdf.

32. Clark W, Mitchell R, Cash D and Alcock F (2002) Information as

Influence: how institutions mediate the impact of scientific assessments on global environmental affairs, Faculty Research Working Paper RWP02-044. Cambridge, MA: Harvard University, John F Kennedy School of Government. http://ksgnotes1.harvard.edu/research/wpaper.nsf/rwp/RWP02-044/$File/rwp02_044_clark.pdf.

33. Committee on Science, Engineering, and Public Policy (2001)

Implementing the Government Performance and Results Act for research. a status report. Washington, DC: National Academy Press. http://www.nap.edu/catalog.php?record_id=10106.


35

34. Council for Medical Sciences (2002) The societal impact of applied

health research: towards a quality assessment system. Amsterdam: Royal Netherlands Academy for Arts and Sciences. http://www.knaw.nl/publicaties/pdf/20021098.pdf.

35. Cozzens S (1995) US research assessment: recent developments.

Scientometrics 34(3): 351-362.

36. Cranfield S and Ward H (2002) 'Managing Change in the NHS' (2001) publications: report of a study on the uptake and use of two SDO publications. London: National Co-ordinating Centre for the NIHR Service Delivery and Organisation Programme (NCCSDO). http://www.sdo.lshtm.ac.uk/files/adhoc/change-management-survey.pdf.

37. Cunningham P, Boden M, Glynn S and Hills P (2001) Measuring and

ensuring excellence in government science and technology: international practices. France, Germany, Sweden and the Unired Kingdom. Manchester: University of Manchester PREST (Policy Research in Engineering, Science and Technology). http://www.mbs.ac.uk/Research/engineeringpolicy/publications/documents/CSTA-PREST.pdf.

38. Davies H, Nutley S and Walter I (2005) Approaches to assessing the

non-academic impact of social science research: report of the ESRC symposium on assessing the non-academic impact of research, St Andrews: University of St Andrews, Research Unit for Research Utilisation. http://www.odi.org.uk/rapid/Meetings/ESRC/Index.html.

39. Davies R and Dart J (2005) The 'Most Significant Change' (MSC)

technique: a guide to its use. London: CARE International et al. http://www.mande.co.uk/docs/MSCGuide.pdf.

40. Deelstra Y, Nooteboom S G, Kohlmann H R, van den Berg J and

Innanen S (2003) Using knowledge for decision-making purposes in the context of large projects in the Netherlands. Environmental Impact Assessment Review 23(5): 517-541.

41. Delanghe H and Muldur U (2007) Ex-ante impact assessment of

research programmes: the experience of the European Union's 7th Framework Programme. Science and Public Policy 34(3): 169-183.

42. Earl S and Carden F (2002) Learning from complexity: the

International Development Research Centre's experience with Outcome Mapping. Development in Practice 12(3/4): 518-524.

43. Earl S, Carden F, Smutylo T (2001) Outcome Mapping: building

learning and reflection into development programs. Ottawa: International Development Research Centre. http://www.idrc.ca/en/ev-9330-201-1-DO_TOPIC.html.


36

44. Edwards K (2001) PCRs and policy influence. What project

completion reports have to say about public policy influence by Centre-supported research: final report. Ottawa: International Development Research Centre, Evaluation Unit. http://www.idrc.ca/uploads/user-S/10359905080pcr_short_final.pdf.

45. Elliott H and Popay J (2000) How are policy makers using evidence?

Models of research utilisation and local NHS policy making. Journal of Epidemiology and Community Health 54(6): 461-468.

46. European Commission (2005) Assessing the impact of energy

research, EUR 21354. Luxembourg: Office for Official Publications of the European Communities. http://ec.europa.eu/research/energy/pdf/erevia_en.pdf.

47. Evans J P (2006) Lost in translation? Exploring the interface between

local environmental research and policymaking. Environment and Planning A 38(3): 517-532.

48. Fahrenkrog G, Polt W, Rojo J, Tübke A and Zinöcker K (eds) (2002)

RTD Evaluation Toolbox: assessing the socio-economic impact of RTD-policies, IPTS Technical Report Series, EUR 20382. Seville: European Commission, Joint Research Centre, Institute for Prospective Technological Studies. http://ftp.jrc.es/eur20382en.pdf.

49. Fisher E and Holland J D (2003) Social development as knowledge

building: research as a sphere of policy influence. Journal of International Development 15(7): 911-924.

50. Flint M E S (1998) Natural resources research evaluation synthesis,

Evaluation Report EV:604. London: Department for International Development. http://www.dfid.gov.uk/aboutdfid/performance/files/ev604.pdf.

51. Gardner B L (1999) Returns to policy-related social science research

in agriculture, Impact Assessment Discussion Paper No. 9. Washington, DC: International Food Policy Research Institute. http://www.ifpri.org/impact/iadp09.pdf.

52. Garfinkel M S, Sarewitz D, Porter A L (2006) A societal outcomes

map for health research and policy. American Journal of Public Health 96(3): 441-446.

53. General Accounting Office (2003) Pipeline safety: systematic process

needed to evaluate outcomes of research and development program, GAO-03-746. Washington, DC: United States General Accounting Office. http://www.gao.gov/new.items/d03746.pdf.


37

54. Government Accountability Office (2007) US public diplomacy: actions needed to improve strategic use and coordination of research, GAO-07-904. Washington, DC: United States Government Accountability Office. http://www.gao.gov/new.items/d07904.pdf.

55. Georghiou L (1995) Assessing the Framework Programmes: a meta-

evaluation. Evaluation 1(2): 171-188.

56. Georghiou L and Davis E (eds) (1988) Evaluation of R & D: a policymaker's perspective: proceedings of the International Workshop on Assessment and Evaluation, Queen Elizabeth II Conference Centre London, London: HMSO.

57. Gerhardus A, Dujardin M, Kiet P H T, Siddiqi S and Sauerborn R

(2000) A methodology to assess the use of research for health policy development, Discussion Paper 04. Heidelberg: University of Heidelberg Medical School, Department of Tropical Hygiene and Public Health. For availablility, contact: [email protected].

58. Gerhardus A and Kielmann K (2000) The use of research for decision-

making in the health sector: the case of 'Shared Care' in Burkina Faso, in: Lessons in research to action: case studies from seven countries. Geneva: Council on Health Research for Development: 19-27. http://www.cohred.org/main/CommonCategories/content/696.pdf.

59. Godin B and Doré C (2003) Measuring the impacts of science: beyond

the economic dimension. Montreal: Canadian Science and Innovation Indicators Consortium. http://www.csiic.ca/PDF/Godin_Dore_Impacts.pdf.

60. Government of Western Australia (1997) Appendix B: PIs for policy

development and advice, in: Preparing performance indicators: a pratical guide. Perth: Government of Western Australia, Department of the Premier and Cabinet, Public Sector Management Office. http://www.dpc.wa.gov.au/psmd/pubs/psrd/preppis.html.

61. Grosse S D, Teutsch S M and Haddix A C (2007) Lessons from cost-

effectiveness research for United States public health policy. Annual Review of Public Health 28: 365-391.


38

62. Guy K, Amanatidou E and Psarra F (2005) Framework Programme 5

(FP5) impact assessment: a survey conducted as part of the five-year assessment of European Union research activities (1999-2003). Science and Public Policy 32(5): 349-366.

63. Hanney S, Davies A and Buxton M (1999) Assessing benefits from

health research projects: can we use questionnaires instead of case studies? Research Evaluation 8(3) pp189-199

64. Hanney S R, Gonzalez-Block M A, Buxton M J and Kogan M (2003)

The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Research Policy and Systems 1(2). http://www.health-policy-systems.com/content/1/1/2.

65. Hanney S R, Grant J, Wooding S and Buxton M J (2004) Proposed

methods for reviewing the outcomes of health research: the impact of funding by the UK's ‘Arthritis Research Campaign’. Health Research Policy and Systems 2(4). http://health-policy-systems.com/content/2/1/4.

65a. Hanney S, Buxton M, Green C, Coulson D and Raftery J (2007) An

assessment of the impact of NHS Health Technology Assessment Programme. NHS Health Technology Assessment Programme. Vol 11, number 53. http://www.ncchta.org/fullmono/mon1153.pdf

66. Healy R G and Ascher W (1995) Knowledge in the policy process:

incorporating new environmental information in natural resources policy making. Policy Sciences 28(1): 1-19.

67. Herrick, C and Sarewitz D (2000) Ex post evaluation: a more effective

role for scientific assessments in environmental policy. Science, Technology, & Human Values 25(3): 309-331.

68. Hilderbrand M and Simon J (2000) The role of research in child health

policy and programs in Pakistan, in: Lessons in research to action: case studies from seven countries. Geneva: Council on Health Research for Development: 77-86. http://www.cohred.org/main/CommonCategories/content/696.pdf.

69. Her Majesty’s Treasury, Department for Education and Skills, Office

of Science and Technology and Department of Trade and Industry (2002) Cross-cutting review of science and research: final report. London, HM Treasury. http://www.hm-treasury.gov.uk/media/E/7/science_crosscutter.pdf.


39

70. Holmes J (2005) The use of science in policy and regulation: baseline

review. Swindon: Environmental Research Funders Forum. http://www.erff.org.uk/reports/events.asp.

70a. Holmes, J (2007) Study of dissemination and implementation of

research in SKEP member organizations. Scientific Knowledge for Environmental Protection (SKEP) http://www.skep-era.net/site/82.asp

71. Hooton N., Lee-Smith D, Nasinyama G and Romney D (2006)

Learning lessons from the Kampala urban agriculture policy process, Process and Partnership for Pro-Poor Policy Change Project report. International Livestock Research Institute, and London: Overseas Development Institute. http://www.pppppc.org/content/files/documents/Kampala%20report%20final%20WEB.pdf.

72. Hovland I (2007) Making a difference: M&E of policy research,

Working Paper 281. London, Overseas Development Institute. http://www.odi.org.uk/publications/working_papers/wp281.pdf.

73. International Food Policy Research Institute (2002) Impact evaluation:

assessing the impact of policy-oriented social science research, Summary reprt of an IFPRI symposium workshop. Washington, DC, IFPRI. http://www.ifpri.org/pubs/ib/ib5.pdf.

74. Innvær S, Vist G, Trommald M and Oxman A (2002) Health policy-

makers' perceptions of their use of evidence: a systematic review. Journal of Health Services Research and Policy 7(4): 239-244.

75. Jensen F S and Koch N E (2004) Twenty-five years of forest recreation

research in Denmark and its influence on forest policy. Scandinavian Journal of Forest Research 19(Suppl. 4): 93-102.

76. Johnston R (1995) Research impact quantification. Scientometrics

34(3): 415-426.

77. Jones S A, Fischhoff B and Lach D (1999) Evaluating the science-policy interface for climate change research. Climatic Change 43(3): 581-599.

78. Kalpazidou-Schmidt E (2006) RTD evaluation and policy in the

European Research Area. Evidence & Policy 2(2): 185-209.


40

79. Kilpatrick H E (1998) Some useful methods for measuring the benefits

of social science research, Impact Assessment Discussion Paper No. 5. Washington, DC: International Food Policy Research Institute. http://www.ifpri.org/impact/iadp05.pdf.

79a. Kivimaa, P, Mela, H, and Furman, E (2007) Approaches and practices

in mid-term and ex-post evaluation of environmental research programmes. Scientific Knowledge for Environmental Protection (SKEP) Work Package 3, 28th June 2007 http://www.skep-era.net/site/files/WP3%20final_eval.v1.2.pdf

80. Kostoff R N (1994) Assessing research impact: US government

retrospective and quantitative approaches. Science and Public Policy 21(1): 13-22.

81. Kostoff R N (1995) Research requirements for research impact

assessment. Research Policy 24(6): 869-882.

82. Kostoff R N (1995) Federal research impact assessment: axioms, approaches, applications. Scientometrics 34(2): 163-206.

83. Kuhlmann S (1995) German government department's experience of

RT&D programme evaluation and methodology. Scientometrics 34(3): 461-471.

84. Kuruvilla S, Mays N, Pleasant A and Walt G (2006) Describing the

impact of health research: a Research Impact Framework. BMC Health Services Research 6(134). http://www.biomedcentral.com/content/pdf/1472-6963-6-134.pdf.

84a. Kuruvilla S, Mays N and Walt G (2007) Describing the impact of

health services and policy research, Journal of Health Services Research and Policy. Vol 12 (Suppl 1) 23-31

85. Lavis J N, Ross S E, Hurley J E et al (2002) Examining the role of

health services research in public policy-making. Milbank Quarterly 80(1): 125-154.

86. Lavis J N, Ross S E, McLeod C and Gildiner A (2003) Measuring the

impact of health research. Journal of Health Services Research & Policy 8(3): 165-170.

87. Leksmono C, Young J, Hooton N, Muriuki H and Romney D (2006)

Informal traders lock horns with the formal milk industry: the role of research in pro-poor dairy policy shift in Kenya, Working Paper 266. London: Overseas Development Institute. http://www.odi.org.uk/publications/working_papers/wp266.pdf.

88. Lewis J M (2005) A network approach for researching partnerships in


41

health. Australia and New Zealand Health Policy 2(22). http://www.anzhealthpolicy.com/content/2/1/22.

89. Lindquist E A (2001) Discerning policy influence: framework for a

strategic evaluation of IDRC-supported research. Victoria, BC: University of Victoria, School of Public Administration. http://www.idrc.ca/uploads/user-S/10359907080discerning_policy.pdf.

90. Link A (1995) Economic performance measures for evaluating

government-sponsored research. Scientometrics 36(3): 325-342.

91. Luukkonen T (1998) The difficulties in assessing the impact of EU Framework Programmes. Research Policy 27(6): 599-610.

92. Luukkonen-Gronow T (1987) Scientific research evaluation: a review

of methods and various contexts of their application. R&D Management 17(3): 207-221.

93. Luukkonen-Gronow T (1989) The impact of evaluation data on policy

determination, in: D Evered and S Harnett (eds) The evaluation of scientific research. London: Ciba Foundation: 234-246.

94. Mackinnon J and Reinikka R (2002) How research can assist policy:

the case of economic reforms in Uganda. World Bank Research Observer 17(2): 267-292.

95. Marburger J H (2005) Wanted: better benchmarks. Science 308(5725):

1087.

96. McNie E C (2006) Reconciling the supply of scientific information with user demands: an analysis of the problem and review of the literature. Environmental Science and Policy 10(1): 17-38.

96a. Meagher L, Lyle C and Nutley S (submitted 2008) Flows of

knowledge, expertise and influence: a method for assessing policy and practice impacts from social science research, Submitted to Research Evaluation 21 January 2008

97. Merkx F, van der Weijden I, Oostveen A M, van den Besselaar P and

Spaapen J (2007) Evaluation of research in context: quick scan of an emerging field. The Hague: Rathenau Instituut and EriC (Evaluating Research in Context). http://www.rathenau.nl/showpageBreed.asp?steID=1&ID=2963.


42

98. Michelson E S (2006) Approaches to research and development

performance assessment in the United States: an analysis of recent evaluation trends. Science and Public Policy 33(8): 546-560.

99. Milner S J, Bailey C, Deans J and Pettigrew D (2005) Integrated

impact assessment in the UK: use, efficacy and future development. Environmental Impact Assessment Review 25(1): 47-61.

100. Molas-Gallart J. and Tang P (2007) Policy and practice impacts of

ESRC funded research: case study of the ESRC Centre for Business Research. Swindon: Economic and Social Research Council.

101. Molas-Gallart J., Tang P. and Morrow S (2000) Assessing the non-

academic impact of grant-funded socio-economic research: results from a pilot study. Research Evaluation 9(3): 171-182.

102. National Audit Office (2003) Getting the evidence: using research in

policy making, HC 586-I, session 2002-2003. London: Stationery Office. http://www.nao.org.uk/publications/nao_reports/02-03/0203586-i.pdf.

103. National Audit Office (2003) An international review on governments'

research procurement strategies. A paper in support of Getting the evidence: Using research in policy making, HC 586-II, session 2002-2003. London: Stationery Office. http://www.nao.org.uk/publications/nao_reports/02-03/0203586-ii.pdf.

104. National Health Service SDO (2007) The impact of the NHS Service

Delivery and Organisation Research and Development Programme 2001-2006. London: National Co-ordinating Centre for the NIHR Service Delivery and Organisation Programme (NCCSDO). http://www.sdo.lshtm.ac.uk/files/adhoc/impact-study-full.pdf.

105. National Research Council (2007) Prospective evaluation of applied

energy research and development at DOE (Phase Two). Washington, DC: National Academies Press. http://books.nap.edu/catalog.php?record_id=11806.

106. Nederhof A (2006) Bibliometric monitoring of research performance

in the social sciences and the humanities: a review. Scientometrics 66(1): 81-100.

107. Norton G W and Alwang J (1998) Policy for plenty: measuring the

benefits of policy-oriented social science research, Impact Assessment Discussion Paper No. 6. Washington, DC: International Food Policy Research Institute. http://www.ifpri.org/impact/iadp06.pdf.

108. Nutley S, Percy-Smith J and Solesbury W (2003) Models of research

impact: a cross sector review of literature and practice, Building Effective


43

Research 4. London: Learning and Skills Research Centre. http://www.lsrc.ac.uk/publications/index.asp.

109. Office of Science and Innovation (2006) Science review of the

Department for Environment, Food and Rural Affairs. London: Office of Science and Innovation. http://www.berr.gov.uk/files/file35751.pdf.

110. Orosz E (1994) The impact of social science research on health policy.

Social Science & Medicine 39(9): 1287-1293.

111. Paarlberg R L (1999) External impact assessment of IFPRI’s 2020 Vision for Food, Agriculture, and the Environment Initiative, Impact Assessment Discussion Paper No. 10. Washington, DC: International Food Policy Research Institute. http://www.ifpri.org/impact/iadp10.pdf.

112. Pang T, Sadana R, Hanney S, Bhutta Z A, Hyder A A and Simon J

(2003) Knowledge for better health: a conceptual framework and foundation for research systems. Bulletin of the World Health Organization 81(11): 815-820.

113. Park D (1998) A proposal for measuring the benefits of policy-oriented

social science research, Impact Assessment Discussion Paper No. 3. Washington, DC: International Food Policy Research Institute. http://www.ifpri.org/impact/iadp03.pdf.

114. Peckham S, Willmott M, Allen P, Anderson S and Goodwin N

(forthcoming 2008) Assessing the impact of the NHS Service Delivery and Organisation Research Programme. Submitted to Evidence & Policy.

115. Percy-Smith J, Speller V and Nutley S (2006) Evidence Informed

Policy and Practice: a review of approaches used in health improvement in Scotland. Edinburgh: NHS Health Scotland. http://www.healthscotland.com/documents/1318.aspx.

116. Pestieau C (2003) Evaluating policy research, Research Paper W22.

Ottawa: Canadian Policy Research Networks. http://www.cprn.org/documents/24336_en.pdf.

117. Pielke R A (1995) Usable information for policy: an appraisal of the

US Global Change Research Program. Policy Sciences 28(1): 39-77.


44

118. PREST (Policy Research in Engineering, Science and Technology)

(2006) Smart innovation: a practical guide to evaluating innovation programmes. Manchester: University of Manchester, PREST. http://www.mbs.ac.uk/Research/engineeringpolicy/publications/documents/SMARTInnovation.pdf.

119. Research Council Economic Impact Group (2006) Increasing the

economic impact of research councils: advice to the Director General of Science and Innovation, DTI from the Research Council Economic Impact Group, 06/1678. London: Department of Trade and Industry. http://www.berr.gov.uk/files/file32802.pdf.

120. Research Councils UK (2007) Increasing the economic impact of the

research councils. Swindon: Research Councils UK. http://www.rcuk.ac.uk/cmsweb/downloads/rcuk/publications/ktactionplan.pdf.

121. Rider Smith D with Sutherland A (2002) Institutionalizing IMPACT

Orientation: building a performance approach that enhances the impact orientation of research organizations. London: University of Greenwich, Natural Resources Institute. http://www.parcinfo.org/documents/PARC%20Library/Institutionalizing%20Impact%20Orientation.pdf.

122. Rigby E (2002) Assessing impacts of social science research. Paper

presented at Social Science Research Impact: Mapping and Measurement, International Social Science Comparisons Workshop, Berlin-Brandenburgische Akademie der Wissenschaften, Berlin, 14-15 March 2002. Berlin. http://www.stage-research.net/SPSG/paris-issc/BerlinRigby.html.

123. Riggirozzi M P and Tuplin T (2004) The influence of research on

policy: MIMAP Philippines. Ottawa: International Development Research Centre, Evaluation Unit. http://www.idrc.ca/en/ev-57585-201-1-DO_TOPIC.html.

124. Ryan J G (1999) Assessing the impact of rice policy changes in Viet

Nam and the contribution of policy research, Impact Assessment Discussion Paper No. 8. Washington, DC: International Food Policy Research Institute. http://www.ifpri.org/impact/iadp08.pdf.

125. Ryan J G (2002) Synthesis report of workshop on assessing the impact

of policy-oriented social science research in Scheveningen, the Netherlands, November 12-13, 2001, Impact Assessment Discussion Paper No. 15. Washington, DC: International Food Policy Research Institute. http://www.ifpri.org/impact/iadp15.pdf.

126. Ryan J G and Garrett J L (2003) The impact of economic policy

research: lessons on attribution and evaluation from IFPRI, Impact Assessment Discussion Paper No. 20. Washington, DC: International Food


45

Policy Research Institute. http://www.ifpri.org/impact/iadp20.pdf.

127. Sayce K and Norrish P (2006) Perceptions and practice: an anthology

of impact assessment experiences. Wageningen: Technical Centre for Agricultural and Rural Cooperation. http://www.anancy.net/uploads/file_en/impact%20assessment.pdf.

128. Science Advisory Council (2007) CSA response to SAC

recommendations on the End to End Review of Science into Policy in Defra, SAC (07) 02. London, Department for Environment, Food and Rural Affairs. http://www.defra.gov.uk/science/documents/papers/2007/SAC(07)02.pdf.

129. Science and Technology Committee (2006) Research council support

for knowledge transfer: government response to the Committee's third report of session 2005-06: fifth special report of session 2005-06, HC 1653. London, Stationery Office. http://www.publications.parliament.uk/pa/cm200506/cmselect/cmsctech/1653/1653.pdf.

130. Scott A (2000) The dissemination of the results of environmental

research, Environmental Issues Series No. 15, Office for Official Publications of the European Communities: Luxembourg. http://reports.eea.europa.eu/Environmental_issues_No_15/en.

131. Scott A, Steyn G, Geuna A, Brusoni S and Steinmueller E (2001) The

economic returns to basic research and the benefits of university-industry relationships: a literature review and update of findings. Brighton: University of Sussex, SPRU-Science and Technology Policy Research. http://www.sussex.ac.uk/spru/documents/econreturnsost.pdf.

132. Smith V H (1998) Measuring the benefits of social science research,

Impact Assessment Discussion Paper No. 2. Washington, DC: International Food Policy Research Institute. http://www.ifpri.org/impact/iadp02.pdf.

133. Soper B and Hanney S R (2007) Lessons from the evaluation of the

UK's NHS R&D Implementation Methods Programme. Implementation Science 2(7). http://www.implementationscience.com/content/pdf/1748-5908-2-7.pdf.

134. Spilsbury M J and Kaimowitz D (2000) The influence of research and

publications on conventional wisdom and policies affecting forests. Unasylva 51(203): 3-10. http://www.fao.org/docrep/x8080e/x8080e02.htm#P0_0.

135. Thiele G, Devaux A, Velasco C and Manrique K (2006) Horizontal

evaluation: stimulating social learning among peers. ILAC Brief 13, Rome: Institutional Learning and Change. http://www.cgiar-ilac.org/downloads/Briefs/ILAC_Brief13.pdf.


46

136. Tress B, Tress G and Fry G (2005) Integrative studies on rural

landscapes: policy expectations and research practice. Landscape and Urban Planning 70(1-2): 177-191.

136a. Trostle J, Bronfman M and Langer A (1999) How do researchers

influence decision-makers? Case studies of Mexican policies, Health Policy and Planning; 14 (2): 103-114

137. UK Evaluation Forum (2006) Medical research: assessing the benefits

to society. London: The Academy of Medical Sciences, Medical Research Council and Wellcome Trust. http://www.wellcome.ac.uk/assets/wtx031817.pdf.

138. Walter I, Nutley S and Davies H (2003) Research impact : a cross

sector review – literature review. St Andrews: St Andrews University, Research Unit for Research Utilisation. http://www.ruru.ac.uk/discussion_papers.html.

139. Walter I. Nutley S and Davies H (2004) Assessing research impact:

report of seminar, 15-16 January 2004. St Andrews: University of St Andrews, Research Unit for Research Utilisation. http://www.ruru.ac.uk/discussion_papers.html.

140. Warburton D (2006) Evaluation of Your Health, Your Care, Your Say:

an independent report commissioned by the Department of Health. London: Department of Health. Available via http://www.dh.gov.uk/en/Publicationsandstatistics/index.htm.

141. Weiss C H (2003) Studying research utilization, Paper prepared for

IDRC meeting, Ottawa, March 24-25, 2003. http://www.idrc.ca/en/ev-66465-201-1-DO_TOPIC.html.

142. Williams D and Rank A D (1998) Measuring the economic benefits of

research and development: the current state of the art. Research Evaluation 7(1): 17-30.


47

143. Wooding S, Hanney S, Buxton M and Grant J (2004) The returns from

arthritis research: a report prepared by RAND Europe for the Arthritis Research Campaign. Volume 1: approach, analysis and recommendations. Cambridge: RAND Europe. http://www.rand.org/pubs/monographs/MG251/.

144. Wooding S, Nason E, Klautzer L, Rubin J, Hanney S and Grant J

(2007) Policy and practice impacts of research funded by the Economic and Social Research Council. a case study of the Future of Work programme, approach and analysis. Cambridge: RAND Europe. http://www.rand.org/pubs/technical_reports/2007/RAND_TR435.pdf.

145. World Bank (2004) Monitoring and evaluation: some tools, methods

and approaches. Washington, DC, The World Bank. Available via http://www.worldbank.org/ieg/.

146. Zillberman D and Heiman A (1999) The value of economic research,

Impact Assessment Discussion Paper No. 7. Washington, DC: International Food Policy Research Institute. http://www.ifpri.org/impact/iadp07.pdf.


48

Bibliography of background documents

Ashridge Centre for Business and Society(2006) Scoping study for an

organisational model for SCP evidence: managing the SCP evidence base from a relational perspective. Berkhamsted: Ashridge Centre for Business and Society. http://theknowledgebridge.com/downloads/Ashridge%20Report%20-%20final-1.pdf.

Department for Environment Food and Rural Affairs (2006) Our approach to

evidence and innovation. London, Defra. http://www.defra.gov.uk/science/how/documents/EvidenceAndInnovation.pdf.

Department for Environment Food and Rural Affairs (2007) Waste and

resources evidence strategy 2007-2011. London, Defra. http://www.defra.gov.uk/environment/waste/wip/research/pdf/wres-2007-2011.pdf.

Gabbay J, le May A, Jefferson H, Webb D, Lovelock R, Powell J, Lathlean J. (2003) A case study of knowledge management in multi-agency consumer-informed 'communities of practice': implications for evidence-based policy development in health and social services. Health 7(3): 283-310.

Hanney S R, Home P D, Frame I, Grant J, Green P, and Buxton M J (2005) Identifying the impact of diabetes research. Diabetic Medicine. 23: 176-184

Kothari A, Birch S, Charles, C (2005) ‘Interaction’ and research utilisation in health policies and programs: does it work? Health Policy 71(1): 117-125.

Nutley, S, Walter, I, and Davies, H (2007) Using evidence: How research can inform public services. Bristol: The Policy Press.

Pawson, R and Tilley, N (1997) Realistic evaluation Sage: London. 235pp

Scott A, Holmes J, Steyn G, Wickham S and Murlis J (2005) Science Meets Policy in Europe. Report for the science meets policy conference, London, November 2005 of research undertaken by The Knowledge Bridge for the UK Department for Environment, Food and Rural Affairs (Defra), November 2005. http://www.theknowledgebridge.com/downloads/SMP_2005_vol_2_print_version.pdf

Shadish W, Cook T and Leviton C (1991) Foundations of programme

evaluation: theories of practice. Thousand Oaks, CA: Sage. Taig T (2004) The development and use of scientific advice in Defra: a report

produced for Defra. TTAC Limited. http://www.defra.gov.uk/science/documents/papers/2004/SAC%20_04_%2005_Tony_Taig.pdf

Weiss C (1998) Evaluation: methods for studying programs and policies: 2nd

edition. London: Prentice Hall International. Weiss C (1995) Nothing as practical as a good theory: exploring theory based

evaluation for comprehensive community initiatives for children and families. In New


49

approaches to evaluating community initiatives, eds Connell J P, Kibisch A C, Schorr L B and Weiss C H, pp65-92. Washington DC: Aspen Institute

Weiss C (1979) The Many Meanings of Research Utilization. Public

Administration Review. 39(5): 426-431.


50

APPENDICES


51

Appendix A: Data extraction sheets

Data extraction sheet

ID reference number 01 Method of identification (e.g. database)

Database

Details of publication Author Title Source (journal/conference etc) Year/volume/pages Country of origin Institutional affiliation

Acosta A and Douthwaite B Appreciative Inquiry: an approach for learning and change based on our own best practices ILAC Brief 6 http://www.cgiar-ilac.org/downloads/Briefs/Brief6Proof2.pdf July 2005 USA Consultative Group on International Agricultural Research, Institutional Learning and Change

Field (e.g. environmental research)

International Development

Type of paper (conceptual, empirical, descriptive)

Descriptive

Research question/aim

Outlines the methodology and advantages/disadvantages of Appreciative inquiry

Study design

Theory

Analysis

Findings/key points

Appreciative Inquiry is a methodology to assess the impact of research by capturing positive features of a project or organisation. It is based on the assumption that learning from past successes can help develop future success. It also promotes knowledge sharing and motivation within an organisation. Data are obtained through interviews and ‘story-sharing’. The five steps are: 1. Define focus and scope of enquiry; 2. Elicit success stories; 3. Determine future goals; 4. Map routes to desired goals; 5. Make it happen.

Paper includes An evaluation of the impact of research


52

on policy A reflective piece about evaluation of the impact of research on policy A description of method(s) for evaluating impact of research on policy Evaluation of effectiveness/value for money of methods A review of methods for evaluating impact more generally Others of relevance (give reason)

�


53


ID reference number 02 Method of identification (e.g. database) Internet search Details of publication Author Title Source (journal/conference etc) Year/volume/pages Country of origin Institutional affiliation

Adamo A Strategic evaluation of policy influence: what evaluation reports tell us about public policy influence by IDRC-supported research International Development Research Centre, Evaluation Unit http://www.idrc.ca/uploads/user-S/10359910880strategic.pdf April 2002 Canada International Development Research Centre




Empirical


What is ‘policy influence’, what has IDRC research accomplished so far, and how?

Study design

Review of 23 2001-2002 evaluation reports – these were informed by researchers’ narratives

Theory

Analysis

Findings

The paper finds that many evaluations of policy ‘influence’ are anecdotal, with varied interpretations of key terms (i.e. impact, outcome, influence). It indicates that influence (i.e. strengthened policy dialogue) is an intermediate outcome, whereas impact (i.e. development of policy) is more long-term. The paper recommends that evaluations needs to consider the process of policy influence in order to capture intended and unintended outcomes (i.e. dissemination, capacity building, knowledge generation). Furthermore, measures of influence should consider the level of


54

policy (national, regional), as well as intended and actual outcomes.

Paper includes An evaluation of the impact of research on policy A reflective piece about evaluation of the impact of research on policy A description of method(s) for evaluating impact of research on policy Evaluation of effectiveness/value for money of methods A review of methods for evaluating impact more generally Others of relevance (give reason)

� �


55



Adamo A Influencing public policy through IDRC-supported research: synthesis of document reviews International Development Ressearch Centre, Evaluation Unit http://www.idrc.org.sg/en/ev-31926-201-1-DO_TOPIC.html 2003 Canada International Development Research Centre




Empirical


How can a research centre increase its influence on public policy? What has IDRC achieved so far? What factors and mechanisms facilitate influence?

Study design

This synthesis of IDRC research project document reviews is the first of a three-part project, which will use regional case studies and stakeholder/expert workshops

Theory

Analysis

Findings

Document reviews help to situate impact evaluations within the context of development research. One of the strengths of document review is in the way programmes and projects express their intention to influence policy. However, they are descriptive, not necessarily representative, and they are ‘imperfect tools’ of evaluation (p4). The document reviews compare intent to influence with intensity of strategy (spending, dissemination etc). Recommends that impact evaluations are process-oriented and consider the scale of


56

policy influence (i.e. national, regional). Page 32 describes eight potential types of policy influence, of which only one involves direct influence on policy formulation (indirect includes strengthened capacity of researchers/ decision makers, changes in attitudes). However, it does not propose a methodology to capture these impacts.


� �


57


ID reference number 04 Method of identification (e.g. database) Journal search Details of publication Author Title Source (journal/conference etc) Year/volume/pages Country of origin Institutional affiliation

Allen P, Peckham S, Anderson S and Goodwin N Commissioning research that is used: the experience of the NHS Service Delivery and Organisation Research and Development Programme Evidence & Policy 2007 3(1) pp119-134 UK National Coordinating Centre for the NHS SDO R&D programme


Health


Empirical


What can be learnt about knowledge transfer from the first five years of the Service Delivery Organisation (SDO) programme?

Study design

Theory

Analysis

Findings

Strresses the importance of moving away from ‘a simple characterization of two communities (research and policy) and the focus on how research evidence is used to a broader view of how research and policy come together’ (p130). It may be more helpful to think about how research is developed, how policy agendas are set and how policy is implemented than how knowledge is transferred from research findings to decision makers. The authors see a key role for interactions between decision makers, commissioners and researchers.

Paper includes An evaluation of the impact of research on policy


58

A reflective piece about evaluation of the impact of research on policy A description of method(s) for evaluating impact of research on policy Evaluation of effectiveness/value for money of methods A review of methods for evaluating impact more generally Others of relevance (give reason)

Interesting account of what might need to be done to promote research use (and thus what might need to be evaluated).


59



Referenced

Details of publication Author Title Source (journal/ conference etc) Year/volume/pages Country of origin Institutional affiliation

Almeida C and Báscolo E Use of research results in policy decision-making, formulation, and implementation: a review of the literature Cadernos de Saúde Pública 2006 22(Suppl) ppS7-S19 http://www.scielosp.org/pdf/csp/v22s0/02.pdf Brazil Escola Nacional de Saúde Pública Sergio Arouca


Health


Conceptual


Theoretical reflections on knowledge transfer and utilisation of knowledge in public policy making.

Study design

Literature review

Theory

‘Two communities’, ‘linkages’, and ‘engineering’ approaches to utilisation

Analysis

Findings/key points

Health services research is a problem-oriented field and, as such, utilisation is most commonly instrumental. Evaluations of the influence of policy on research depend upon the definitions of policy and the policy process (p9), and an understanding of other contributing sources of information. Bowen and Zwi (p12) suggest that evidence interacts with context prior to utilisation within policy. This paper concludes that the greatest influence of research is in shaping the terms of debate – further influence is contingent on ideology, political and contextual factors. It highlights the importance of analysis examining key actors, decision making processes (how information is gathered, evaluated, what motivates decisions, idea ‘diffusion’ and stakeholder analysis) (p15).


60

Bowen and Zwi propose forward analysis from the research (sourcing, using and implementing) (p16)


�


61


ID reference number 06 Method of identification (e.g. database) Database search Details of publication Author Title Source (journal/conference etc) Year/volume/pages Country of origin Institutional affiliation

Anderson R The policy impact of population health surveys: an illustration of the measurement challenges using the NSW Health Survey Evidence & Policy 2006 2(2) pp167-183 Australia University of Technology, Sydney


Health


Empirical


Do large-scale population health surveys produce research evidence that is used by policy decision makers?

Study design

Case study of New South Wales Health Survey Program, Australia. Sub-analysis of survey of users, telephone survey of respondents, series of case studies.

Theory

Analysis

Findings/key points

Health surveys can inform multiple policy decisions, and are an essential ingredient of further research (direct and indirect impacts). The paper finds that general survey data have a wider range of potential uses (and users) and a potentially longer ‘shelf life’ than one-off primary research. A range of policy/ practitioners who utilised the data are identified – approximately two-thirds used the data to inform planning policy or to evaluate existing policy, usually in conjunction with other information sources. Although the study hoped to develop quantitative indicators, it found these to be unrealistic and unrepresentative (too small a sample). The paper raises issues about who to


62

identify as key actors – initial users (data analysts, report writers) or those closer to policy processes. For a valid assessment of impact on policy, the methodology needs to go beyond survey of decision makers and quantitative indicators, and author recommends the use of semi-structured interviews in addition (see also Lavis et al, 2003). In-depth case studies ‘can be prohibitively expensive’ (p177). Beyond measuring the perceived impacts on policy, the attribution of particular benefits would require very strong evidence about intervening causal relationships: research utilisation, associations between policies and interventions, associations between health benefits and economic benefits (p179).

Paper includes; An evaluation of the impact of research on policy A reflective piece about evaluation of the impact of research on policy A description of method(s) for evaluating impact of research on policy Evaluation of effectiveness/value for money of methods A review of methods for evaluating impact more generally Others of relevance (give reason)

� � �


63



Annema J A, van Wee B, van Hoek T and van der Waard J Evaluation of Dutch public investment plans Environmental Impact Assessment Review 1999 19(3) pp305-317 The Netherlands National Institute of Public Health and Environment


Evaluation literature


Empirical


Proposes a methodology for ex-ante economic, social and ecological policy evaluation.

Study design

Theory

Analysis

Three stage methodology: 1. identify policy tasks based on prognosis of goals; 2. evaluate legitimacy, benefits, cost-effectiveness, uncertainties and risks, and alternatives; 3. compare policy options.

Findings

Clarity of policy objectives is crucial to evaluation. The authors reflect that this methodology allows for both positive and negative appraisals of economic, social and ecological impacts. Increasing the number of indicators increases the quality of appraisal but risks over-emphasising certain impacts. However, it also risks bias against non-quantifiable indicators.

Paper includes An evaluation of the impact of research on policy A reflective piece about evaluation of the impact of research on policy A description of method(s) for evaluating


64

impact of research on policy Evaluation of effectiveness/value for money of methods A review of methods for evaluating impact more generally Others of relevance (give reason)

�


65


ID reference number 08 Method of identification (e.g. database) Journal search Details of publication Author Title Source (journal/ conference etc) Year/volume/pages Country of origin Institutional affiliation

Arnold E Evaluating research and innovation policy: a systems world needs systems solutions Research Evaluation 2004 13(1) pp3-17 UK Technopolis/University of Brighton




Conceptual


What does the shift to a systems perspective mean for the scope and practice of R&D evaluation?

Study design

Theory

Analysis

Findings

There is increasing systems thinking in innovation and management, but evaluation is still focused on individual projects and programmes. Evaluation needs to consider the macro, meso and micro levels of impact.

Paper includes An evaluation of the impact of research on policy A reflective piece about evaluation of the impact of research on policy A description of method(s) for evaluating impact of research on policy Evaluation of effectiveness/value for money of methods A review of methods for evaluating impact more generally


66

Others of relevance (give reason)

� Emphasis on micro-, meso- and macro-level evaluation


67


ID reference number 09 Method of identification (e.g. database) Internet search Details of publication Author Title Source (journal/conference etc) Year/volume/pages/ Country of origin Institutional affiliation

Arnold E, Muscio A and Zaman R Reviews reviewed: lessons from the first six international panel reviews, 1999-2004 Technopolis, for Engineering and Physical Sciences Research Council. Available via: http://www.epsrc.ac.uk/default.htm January 2005 UK Technopolis




Empirical


A review of international panel evaluations of science quality

Study design

Draws on US Government Performance and Results Act, French CNER(National Committee for the Evaluation of Research), Finnish and Swedish practice

Theory

Analysis

Findings/key points

The high status of panellists enables reviewers to be critical. However, the consequences of panel reviews are rarely visible, even if they can be said to influence debate and science policy. It is common practice for panellists to read background documentation prior to a field visit, and to use a combination of methods to triangulate findings (in this case, a survey). The effectiveness/ appropriateness of the six-year cycle of review varies between disciplines, e.g. computer science is faster changing than physics (p13). The GPRA (1993) ‘has influenced practice worldwide to move towards clearer, more quantitative evaluations and performance indicators’


68

(p16), and its low cost ($50,000 per field but only low because most staff worked pro bono). The Academy of Finland (p17) reviews quality and impact of science every three years, based on secondary data complemented by knowledge of council members. The Swedish Research Council (p18-19) evaluates quality of research in international context. French CNER (p19) uses peer review, bibliometrics, event analysis and inspection. US, French and Finnish examples suggest careful use of bibliometrics is useful to inform reviews. The US and French examples highlight the benefits of triangulation.


� �


69


ID reference number 10 Method of identification (e.g. database) Internet Details of publication Author Title Source (journal/conference etc) Year/volume/pages Country of origin Institutional affiliation

Arts and Humanities Research Council AHRC impact strategy Arts and Humanities Research Council. Available via http://www.ahrc.ac.uk 2007 UK Arts and Humanities Research Council




Descriptive


Strategy document outlining a framework for describing the range of impacts achieved by research and identifying the principles and mechanisms for measuring/assessing these impacts.

Study design

Theory

Analysis

Findings

The evaluation framework is driven by accountability, value-for-money and organisational strategy, and develops a narrative of intrinsic and instrumental impacts, supported by metrics where appropriate. It is underpinned by the concept of the research landscape rather than the evaluation of individual awards; by a research life cycle rather than end-of-award evaluations; by the use of multiple evaluation elements (metrics of activity, impact assessment and assessment of quality or excellence); and by knowledge exchange. ‘The measurement of inputs, outputs and outcomes becomes more difficult as they move from being tangible and objective to become less tangible and subjective’


70

(p9). Case study analysis can demonstrate a range of outcomes, including economic and innovation. Evaluations should first identify and describe impacts, and only quantify impacts where this is possible.


� �


71



Referenced

Details of publication Author Title Source (journal/conference etc) Year volume/pages Country of origin Institutional affiliation

Beacham B, Kalucy L and McIntyre E Understanding and measuring research impact FOCUS on… www.phcris.org.au/publications/focuson/index.php. 2005 (2) 12pp Australia Flinders University, Adelaide


Health


Conceptual


Literature review of models to measure, and issues in measuring research impact.

Study design

Theory

Instrumental, conceptual, symbolic utilisation of research.

Analysis

Findings/key points

Multiple terms refer to research impact: ‘benefit’, ‘payback’, ‘translation’, ‘transfer’, ‘uptake’, ‘utilisation’. Measuring the impact of research can be a) in a variety of settings (policy, practice, social behaviour); b) across types of research impact (instrumental, conceptual, symbolic) and of associated process factors; and c) assessing the effects of new knowledge above the value of old, previously ignored/unrecognised research (p5). Impact refers to the value and benefits associated with using knowledge derived through research and conducting research. The paper proposes a three step process for assessing research impact: 1. identify target audience; 2. select appropriate category of measures (i.e. producer-push, user-pull or exchange); 3. select measures (process measures - bibliometrics, intermediate outcome measures – survey of attitudes, and outcome measures). ‘Meaningful approaches to measuring research impact rely on understanding the main ways


72

research can be used and the range of forms in which research impact occurs. Ideally they should also acknowledge the contribution of activity occurring at different stages of the research cycle and in both the researcher and user settings’ (p8). Describes the HERG Payback model (p6) and the input/output model (p9) to illustrate the diversity of impacts in practice. A more comprehensive approach to measuring impact may be too complex and expensive relative to potential benefits, but can use methodologies including expert panel, personal testimonies log, detailed case study assessments, proxy measures of impact. Page 10 outlines alternative measures and data sources for producer-push, user-pull and exchange processes, including surveys, documentary analysis, surveys and semi-structured interviews.


� � �


73


ID reference number 12 Method of identification (e.g. database) Database Details of publication Author Title Source (journal/conference etc) Year/volume/pages Country of origin Institutional affiliation

Bekker M P M, Putters K and van der Grinten T E D Exploring the relation between evidence and decision-making. A political-administrative approach to health impact assessment. Environmental Impact Assessment Review 2004 24(2) pp139-149 The Netherlands Institute for Health Policy and Management


Health


Empirical


How are policy decisions taken? How does this affect Health Impact Assessment?

Study design

Case study of smoking policy.

Theory

Rational model of research utilisation; incremental model (Lindblom, 1959, 1979); mixed methods (Etzioni, 1967).

Analysis

Findings/key points

Health Impact Assessment is a prospective evaluation of policy impact, (incorporating cost-benefit analysis). The utilisation of research can be problem-solving, enlightenment or knowledge creep. The paper also observes that decision makers justify the use of research more in hindsight, using different lines of reasoning. The importance of communication increases with conflicting arguments. The case study of impact (smoking policy) is actor-centred on who had decision making power, and the processes of events through which decisions were made. The paper recommends that evaluations of decision making and utilisation should


74

consider the values, beliefs and ideologies of individual decision makers and institutions.


�


75



Berryman J What defines ‘enough’ information? How policy workers make judgement and decisions during information seeking: preliminary results from an exploratory study. Information Research http://informationr.net/ir/11-4/paper266.html. 2006 11(4) 16pp Australia University of Technology, Sydney


Utilisation literature


Empirical


An ex-post evaluation of how policy makers make policy decisions, and how they judge what is ‘enough’ information?

Study design

33 semi-structured interviews with public sector policy workers in Australia (interviewed individually and in pairs).

Theory

Foster (2004) on non-linear information seeking behaviour.

Analysis

Findings/key points

The study uses a background literature review to contextualise the information, and analysis of interview transcripts to understand key themes and interrelated factors. The study finds that judgement and decision making are a multi-stage and fluid process, also affected by context-specific influencing factors. The paper concludes with the importance of developing an evaluation framework.

Paper includes An evaluation of the impact of research on policy A reflective piece about evaluation of the impact of research on policy

�


76

A description of method(s) for evaluating impact of research on policy Evaluation of effectiveness/value for money of methods A review of methods for evaluating impact more generally Others of relevance (give reason)


77


ID reference number 14 Method of identification (e.g. database) Personal recommendation Details of publication Author Title Source (journal/conference etc) Year/volume/pages Country of origin Institutional affiliation

Big Lottery Fund Big Lottery Fund: increasing the impact of research, evaluation and learning. Big Lottery Fund. Available via: http://www.biglotteryfund.org.uk. December 2005 UK William Solesbury Associates




Empirical


Evaluation of the effectiveness of the dissemination of evaluation and research undertaken by the Big Lottery Fund.

Study design

Review of theory and practice relevant to dissemination, and ten case studies (using documentary analysis and 50 interviews with Big Lottery Fund staff) to assess practice.

Theory

Analysis

Findings/key points

The paper describes the methods used (documentary analysis and in-depth interviews to inform ten case studies). The literature review was used to develop a framework for understanding impacts (from which a check list of good practices and key dimensions was derived). These were then used to guide the evaluation. The paper does not reflect on the approach taken.


�


78



79



Biotechnology and Biological Sciences Research Council BBSRC strategy for evaluating research programmes, SB 44/2005 BBSRC. Available via: http://www.bbsrc.ac.uk. 2005 UK Biotechnology and Biological Sciences Research Council


Science


Descriptive


Outline of institutional evaluation strategy.

Study design

Theory

Analysis

Findings/key points

Evaluation is useful for accountability, to aid funding decisions, and for performance management. Evaluation distinguishes between overall and scheme objectives; activities/processes; and immediate/intermediate/ultimate impacts. The approach to evaluation defines the purpose and objectives of evaluation, its scope and designing the methodology (triangulation of methods including: peer review, survey, bibliometrics, benchmarking, historical tracing, social analysis and economic analysis) (p9). The document reflects that peer review and questionnaires are the two most commonly used methods, and that historical tracing, social analysis and economic analysis are the most resource intensive and thus more expensive.

Paper includes


80

An evaluation of the impact of research on policy A reflective piece about evaluation of the impact of research on policy A description of method(s) for evaluating impact of research on policy Evaluation of effectiveness/value for money of methods A review of methods for evaluating impact more generally Others of relevance (give reason)

� �


81



Boaz A Systematic reviews as a source of evidence for policy: an in-depth exploration of a review on mentoring PhD thesis December 2006 UK University of London


Policy research


Empirical


Exploration of the use of a systematic review on mentoring.

Study design

Case study and literature review.

Theory

Analysis

Findings/key points

Thirty-three semi structured interviews were conducted, in combination with observations, documentary analysis and survey methods. While it is not possible to generalise from a single case, it is useful in shedding light on the utilisation process. The study concludes by suggesting that the case study approach could be replicated with other reviews and the results compared and discussed. It also argues that while case studies have traditionally been used to explore research utilisation, other methods used elsewhere in social science research might also be promising.


�


82



83



Bouma J and Droogers P Translating soil science into environmental policy: a case study on implementing the EU soil protection strategy in the Netherlands Environmental Science and Policy 2007 10(5) pp454-463 The Netherlands Wangeningen University


Environment


Empirical


How has soil science research contributed to soil policy? How can this process of utilisation be improved?

Study design

Case study analysis

Theory

Analysis

Findings/key points

The study explores the context of research, and factors that may facilitate or hinder influence on policy, i.e. communication, relevance and scales (local). However, there is no reflection on the methodology used.

Paper includes An evaluation of the impact of research on policy A reflective piece about evaluation of the impact of research on policy A description of method(s) for evaluating impact of research on policy Evaluation of effectiveness/value for money of methods A review of methods for evaluating

�


84

impact more generally Others of relevance (give reason)


85



Internet search


Bozeman B Public value mapping of science outcomes: theory and method. A monograph of the Public Value Mapping Project of the Center for Science, Policy and Outcomes Columbia University, Center for Science, Policy and Outcomes, and Georgia Tech, School of Public Policy http://www.cspo.org/home/cspoideas/know_flows/Rock-Vol2-1.PDF. 2003 USA Center for Science, Policy, and Outcomes


Science


Descriptive


Develops Public Value Mapping (PVM) as a tool to understand the broader social impacts of large scale science programmes.

Study design

Theory

Public failure theory (normative, p23) and churn model of innovation (explanatory, p24).

Analysis

Findings/key points

PVM is a twelve-step conceptual tool to assess the broader social impacts of science. It can be applied either formatively or summatively, and is focused on the outcomes of the knowledge value collective (social indices and indicators). Analysis focuses on scientific and human capital, guiding policy, network linkages, environmental resources available, and the ability to utilise knowledge. In-depth case study and historical analysis are always elements of PVM; other methodologies are dependent on context, including surveys, focus groups, output/impact analysis, expert opinion, patent and citation analysis. Outcome analysis also considers who benefits, as well as how. The search for impacts/outcomes begins only after goals have been noted. Page 15 notes a widespread interest in the USA,


86

in assessing impacts of research by the quantity of publications, and the use of econometric methods to evaluate funding policy, but PVM tries to move away from placing market values on outcomes.


�


87


ID reference number 19 Method of identification (e.g. database) Referenced Details of publication Author Title Source (journal/conference etc) Year/volume/pages Country of origin Institutional affiliation

Brehaut J D and Juzwishin D Bridging the gap: the use of research evidence in policy development HTA Initiative 18 http://www.ihe.ca/documents/hta/HTA-FR18.pdf. 2005 Canada Alberta Heritage Foundation for Medical Research


Health


Conceptual


What are the challenges of using research evidence to inform healthcare policymaking? Develops the STEEPLE framework to identify/facilitate the use of evidence in health care policy making.

Study design

Workshops and peer review to help revise adapted methodology.

Theory

Analysis

Findings/key points

The paper proposes STEEPLE (Social and systems demographics, Technology, Environment, Economics, Politics, Legislation, Ethics), a descriptive framework to help decision makers conceptualise the type and quality of evidence to use in policy decision making. The methodology differs from the UK PESTLE model, in its focus on decision making (competing sources of information) within the Politics component, and the inclusion of a category for ethical analysis. The paper concludes that ‘Insofar as decision makers can often make good policy decisions, the process works. Therefore, the model is useful, but it is not the whole


88

answer’ (p23). Paper includes An evaluation of the impact of research on policy A reflective piece about evaluation of the impact of research on policy A description of method(s) for evaluating impact of research on policy Evaluation of effectiveness/value for money of methods A review of methods for evaluating impact more generally Others of relevance (give reason)

� A descriptive framework to evaluate the utilisation of evidence in policy


89



Brofoss K E The Research Council of Norway’s use of research evaluation: an assessment of research evaluation as a strategic tool Research Evaluation 1998 7(3) pp134-140 Norway The Royal Norwegian Embassy




Empirical


An examination of the policy context, evaluation portfolio and utilisation profile, and interaction between different types of research use (political, conceptual etc).

Study design

Examines 27 of 30 evaluation reports commissioned in 1993-94, and asks ‘respondents’ (assumption that this is in-depth interviews).

Theory

Weiss with Bucuvalas (1980) on instrumental, conceptual and political use of research.

Analysis

Five-dimensional analysis of research: scientific quality; relevance to society; research organisation; efficiency; and use of research results. Uses a correlation matrix (p139) to assess utilisation.

Findings/key points

Analysis found that the uptake of research recommendations was greater by individual programmes than by the Board. The paper highlights the differences between the use of research conclusions, and the use of research recommendations.


� �


90



91



Personal recommendation


Bumgarner R, Hattaway D, McGann J and Wise H Center for Global Development: evaluation of impact Arabella Analytical Report http://www.arabellaadvisors.com/CGDreport/CGDWeb.pdf. December 2006 USA Center for Global Development


International development


Empirical


Evaluation of CGD think-tank: Does CGD’s research agenda meet its policy making targets? Is the research high quality, and does it influence further research? Is the communications strategy achieving desired impact? Has the CGD developed appropriate partnerships?

Study design

150 in-depth interviews and 1,200 survey responses. Documentary analysis (including website material) and case study analysis.

Theory Analysis Findings/key points

86% of 1,200 survey respondents reported using CGD work in their policy analysis. The paper reports the rigour and relevance (timeliness, robustness) of research, with the implication that these factors have increased impact. It identifies successful research as those that ‘could catch policymakers’ attention’ (p12). Research was reported to have already had an effect (in development assistance locally etc) but potential collaborations/partnerships would increase impact on research. The three case studies explored how the research gained policy attention (i.e. described context, highlighted concept, demonstrated benefits, enabled policy maker to draft a plan). The authors note, however, that not all research projects are designed primarily to influence policy, and this will have implications for evaluating against original objectives. The paper finds that consistent thematic research (staying


92

power) and better communication can enhance influence/impact. The evaluation assumes that ‘impact’ is visible through use in policy debate, policy papers and citations (p71). The only measure/data regarding impact on policy was through interviews. Also noted the strength of relevance ‘very important, crucial’ etc. Impact is difficult to ‘prove’ but the large numbers of interviews addressed this challenge (p118) and evidence was then cross checked with publication dates, and other interviewees. It is ‘important to distinguish between outputs and impact. The degree of policy impact – the contribution that specific research products or outputs makes to actual policy decisions – is contingent upon numerous factors, including the format of the product, timing, distribution channels and outreach strategy’ (p118).


� �


93



Internet search


Butler L and McAllister I Metrics or peer review? Evaluating the 2001 UK Research Assessment Exercise in political science Australian National University http://repp.anu.edu.au/papers/2007_ukresearchassess.pdf June 2007 Australia Australian National University, Research School of Social Sciences




Empirical


Can citations substitute for peer evaluation in evaluating research quality?

Study design

Data analysis of the 2001 RAE (4,400 journal/book/chapter submissions).

Theory

Analysis

Findings

The paper compares citation analysis with RAE outcomes, and finds that metrics report on the whole, similar results for RAE funding as peer review. Metrics are a more defensible and transparent evaluation method, and a lower cost.

Paper includes An evaluation of the impact of research on policy A reflective piece about evaluation of the impact of research on policy A description of method(s) for evaluating impact of research on policy


94

Evaluation of effectiveness/value for money of methods A review of methods for evaluating impact more generally Others of relevance (give reason)

�


95



Buxton M, Hanney S, Packwood T, Roberts S and Youll P Assessing benefits from Department of Health and National Health Services research and development Public Money & Management 2000 20(4) pp29-34 UK Brunel University


Health


Empirical


To describe the methods used to provide a retrospective assessment of the benefits of the North Thames R&D programme and to use this experience to develop a plan for how such evaluation and monitoring might be undertaken on a regular basis.

Study design

‘Experience has shown case studies to be very illuminating when conducting payback assessment.’ However, detailed case studies were considered to be too resource intensive for regular monitoring so a series of questionnaires were devised with the selective use of case studies (p30). The postal questionnaire was designed around the dimensions of the HERG Payback model.

Theory

Analysis

Findings/key points

There is a need for multi-dimensional categorisation of impact as ‘publications on their own do not provide a sufficient indicator of broader impacts on policy or health benefits’ (p33). Gathering information directly from researchers via a questionnaire based on the payback categories did seem potentially feasible.


96

However, case studies also had a role to play in generating stories and verifying information in the self-completion questionnaires (and in offering suggestions for improving the questionnaire). Overall comparison of questionnaire and case study scores found that questionnaires slightly underestimated impact (p31). External data (such as citation analyses) can also play a role, although application is limited.


�


97



Referenced


Campbell S, Benita S, Coates E, Davies P and Penn G Analysis for policy: evidence-based policy in practice Government Social Research Unit http://www.gsr.gov.uk/resources/publications.asp 2007 UK Government Social Research Unit


Policy research


Empirical


What is evidence and how is it used in policy-making?

Study design

Interviews and discussion groups with 42 policy-makers.

Theory

Analysis

Findings

The findings indicate reluctance among policy makers to use scientific evidence as a basis for policy making. Respondents question the ability of research to point to clear, unambiguous conclusions as well as uncertainty about what constitutes evidence, and unfamiliarity with the relative merits and explanatory value of different methods.

Paper includes An evaluation of the impact of research on policy A reflective piece about evaluation of the impact of research on policy A description of method(s) for evaluating impact of research on

�


98

policy Evaluation of effectiveness/value for money of methods A review of methods for evaluating impact more generally Others of relevance (give reason)


99



Carden F Issues in assessing the policy influence of research International Social Science Journal 2003 56(1) pp135-151 Canada International Development Research Centre




Conceptual


What are the gaps in evaluation study? How can the potential for research influence be increased?

Study design

Theory

Overview of knowledge utilisation models. Caplan (1979) cultural differences between researchers and policy makers. Weiss (2003) influence on policy makers versus influence on policy system.

Analysis

Findings

The paper summarises the key findings from a large-scale IDRC project which included literature review, workshops, interviews with staff and key experts, and selected case studies. It identifies tensions between the generalisability of science versus policy-relevant research. The main challenges of supply-side policy influence are capacity building, ownership, persistence, intent, communication and dissemination. The author notes that it is useful to incorporate evaluation early in the research process (ideally from the outset), and observes issues in defining ‘impact’, attribution (p136) and the timing of evaluation (p137). It is necessary to


100

consider the process of influence in order to identify critical success factors and the development of new stakeholder relationships. ‘Development is a much longer-term process, especially as far as policy influence is concerned. As a result the issue of what to evaluate is problematic’ (p136). Evaluations conducted by programme-staff/research producers have a tendency to elevate their own attribution in policy influence, and to ignore unintended consequences. Flyvberg (2001, p38) and Yin (1994) argue that case studies are the approach best suited to understanding context and the relationships that are part of development’ (p137). They are particularly useful in answering ‘how’ and ‘why’ questions, whereas surveys can answer ‘what’. The paper notes that ‘while policy influence could de defined broadly, it appears most feasible to focus the evaluation on the effects of IDRC-supported research on public policy, at global, national, and local levels’ (p141). With reference to an IDRC-wide review of impact evaluation, ‘this should be a 10 or 15-year retrospective examination of policy influence’ (p144).


�


101



Carden F Making the most of research: the influence of IDRC-supported research on policy processes Paper for the International Conference ‘African Economic Research Institutions and Policy Development: Opportunities and Challenges’. Available via: http://www.idrc.ca/index_en.html 2005 Canada International Development Research Centre





When is research most likely to influence policy? What factors inform the process of influence on policy?

Study design

Uses 22 IDRC case studies to illustrate arguments (but not as a methodology)

Theory

Analysis

IDRC framework (developed in consultation with IDRC staff) of the three types of influence on policy: 1.expanding policy capacities; 2. broadening policy horizons; 3. affecting policy regimes.

Findings/key points

The IDRC framework highlights the fact that policy influence goes beyond changing policies themselves. The author refers to policy influence not impact: by the time impact occurs, research is so entwined with other factors it is impossible to attribute causality (p4). The ideal conditions for policy influence are a demand from policy makers for research to inform policy creation. The paper highlights institutional stability, the capacity of policy makers to use research, degree of government control


102

(decentralised vs tight control) and economic pressures as key factors that shape policy influence.


�


103


ID reference number 26a Method of identification (e.g. database) Personal recommendation Details of publication Author Title Source (journal/conference etc) Year/volume/pages Country of origin Institutional affiliation

Carden, F Context Matters – the influence of IDRC-supported research on policy processes In Ayuk, E T and Marouani, M A (2007) The policy Paradox in Africa: Strengthening links between economic research and policymaking. Africa World Press: New Jersey and International Development Research Centre: Ottawa 2007 Canada IDRC




Empirical


Study of research and its relationship to decision making and content.

Study design

22 case studies of IDRC funded research (44)

Theory

IDRC think that it is more useful to look at policy influence rather than policy impact (as only those within a system are in a position to achieve impact). Testing a framework of three dimensions of influence: expanding policy capacities, broadening policy horizons and affecting policy regimes.

Analysis

Data was analysed based on categories identified through workshop discussions on how IDRC supports research: ‘what we do, where we work and how we work.’

Findings/key points

Value of having a framework for understanding the influence of research on policy and decision making to inform evaluation, monitoring and planning. Suggests that the framework would benefit from further testing and revising in different settings. No single factor is


104

key to promoting influence, instead need to map an interplay of capacities, contexts and conditions (on an on going basis)


�


105


ID reference number 26b Method of identification (e.g. database) Personal recommendation Details of publication Author Title Source (journal/conference etc) Year/volume/pages Country of origin Institutional affiliation

Carden, F Using comparative data: A systems approach to a multiple case study Chapter for a book on case methods (Forthcoming, 2008) Canada International Development Research Centre (IDRC)




Empirical


Description of a study of the influence of research on policy using a multiple case study approach

Study design

25 case studies of IDRC research use were identified in a range of different countries (including Ukraine and Guatemala). Two pilots carried out at the start. 22 cases were completed in the first twelve months and two more were completed twelve months later and were integrated into the analysis. One was not completed. Six background studies were also undertaken, including a literature review and a project completion report review (44).

Theory

Analysis

Findings/key points

On-going issues for evaluators: attribution, impact and timing. Challenges included selecting cases, identifying (local) evaluators, timing, consistency, the generation of a considerable volume of data to analyse (collectively), and the need to engage partners from the beginning. Involving


106

users made the evaluation much more complicated, but had the potential to dramatically increase relevance and use (two key goals of evaluation)


�


107


ID reference number 26c Method of identification (e.g. database) Personal recommendation Details of publication Author Title Source (journal/conference etc) Year/volume/pages Country of origin Institutional affiliation

Carden, F The Episode Studies Approach In Trussie, D The Politics of Trade: the role of research in trade policy and negotiation. Amsterdam and Ottawa: Martin Nijhoff-Brill and IDRC. 2008 Canada International Development Research Centre (IDRC)




Descriptive


Introduction to episode studies and presentation of a framework for understanding the findings from episode studies

Study design

Theory

Analysis

Findings/key points

Episode studies have a strong narrative appeal – telling the history of events around a policy change. It proposes a framework for understanding episodes of policy influence that addresses three dimensions: context, research approach and the research itself.


�


108



109


ID reference number 26d Method of identification (e.g. database) Personal recommendation Details of publication Author Title Source (journal/conference etc) Year/volume/pages Country of origin Institutional affiliation

Carden, F Knowledge to Policy:Making the most of development research Sage & International Development Research Centre (IDRC) Forthcoming, 2008 Canada IDRC




Empirical


The book is in three parts: part one details the evaluation findings, part two describes each of the research cases examined in the evaluation and part three describes the evaluation methodology

Study design

The study employed a multiple case study approach 25 case studies of IDRC funded research (44) were commissioned in different countries and contexts. Of those, 22 were completed within 12 months and one additional case reached was completed the following year

Theory

Analysis

Findings/key points

The book contains the most detailed account of the IDRC multiple case study evaluation of its activities. The evaluation was designed and conducted for IDRC’s own use—to explore, explain and improve how IDRC’s support for research can influence public policy in developing countries, but it has attracted


110

wider attention. Data were collected, discussed and analysed as far as possible in collaboration with the users—researchers, staff and executives in IDRC. The depth of data collection in each case and the range of fields of research and regions of undertaking provided a foundation on which to develop a common framework for the purposes of the Centre’s work.


� �


111



Referenced


Cash D and Clark W From science to policy: assessing the assessment process Faculty Research Working Paper RWP01-045. Available via: http://belfercenter.ksg.harvard.edu/publication/ 2001 USA John F Kennedy School of Government, Harvard University


Environment


Conceptual


Assessment of the pros/cons of the Global Biodiversity Assessment

Study design

Theory

Policy windows (Kingdon, 1995)

Analysis

Findings

Compares the value of assessments/ evaluations as evidence: salient, credible and legitimate. The report recommends that assessments should be politically sensitive (context); address the needs of potential users (participation); be a process of communication; and that global and local levels be connected. The discussion notes the interest, capacity and openness of stakeholders themselves (p7), boundary organisations (p8) and network entry points and nodes (p11), which have implications for the scope of evaluation.

Paper includes; An evaluation of the impact of research on policy A reflective piece about evaluation of the impact of research on policy

�


112



113



Internet search


CGIAR Science Council Impact assessment of policy-oriented research in the CGIAR: a scoping study report CGIAR Science Council Secretariat http://www.sciencecouncil.cgiar.org/publications/ 2006 USA Consultative Group on International Agricultural Research


Science


Conceptual


A review of existing CGIAR ex-post policy impact assessment methods: what has been achieved so far and what needs to be done?

Study design

General literature review, a taxonomy to distinguish between types of research and intended pathways, and workshop with panel of experts.

Theory

Analysis

Findings

Quantitative methods can track historical trends in rates of return (most used for economic analyses: impact); qualitative methods can help understand the use of research outputs in policy processes (influence). For CGIAR it is important to understand how policy influence can be translated into meeting long term goals (i.e. policy outcomes). Transparency of assessment methods and how conclusions are drawn needs to be improved. Separating policy making from implementation is not appropriate, as without implementation there is unlikely to be impact. The paper reflects that most evaluation studies fail to justify attribution and value-added of specific studies, especially with confirmatory research. In these cases, ‘it may be useful to refer


114

to the contribution of information from research to policy-making processes rather than attempting to ‘attribute’ the effect or influence on particular bodies’ (p6). The majority of assessments consider diffusion (intermediary to impact influence). The authors propose citation and survey methods to assess diffusion; surveys and interviews of those involved in policy making to assess influence or outcomes (although additional analysis is necessary to establish relative benefits); and econometric models to quantify impact assessments. ‘The use of policy-oriented information is neither observable nor easy to track’ (p11). Within existing CGIAR assessments, interviews are the most commonly used methodology (18), followed by citation analysis (8) and surveys (6). Garrett and Islam (1998) suggest that social science evaluation should consider only outputs, processes and potential outcomes, rather than actual policy outcomes, because of the difficulty in attribution (p31). Annex B indicates some of the empirical issues in conducting an impact analysis of research: scale and scope, timing (jumps, lags and horizons), supply- or demand-side approaches, the importance of surprise, attribution and counterfactuals, the choice of indicators, sampling, and whether ex-ante or ex-post.


�


115



Council for the Humanities, Arts and Social Sciences Measures of quality and impact of publicly funded research in the humanities, arts and social sciences CHASS Occasional Paper 2 http://www.chass.org.au/papers/ 2005 Australia Council for the Humanities, Arts and Social Sciences




Descriptive


Develops a ‘fairer’ way of evaluating research that focuses on quality and impact (not outputs).

Study design

Consultation (200 responses), survey of literature and practice, case studies and focus groups.

Theory

Analysis

Findings/key points

The CHASS model to assess the impact of research groups has received ‘uncommon acceptability’ among participants and meets five key criteria for acceptability: simplicity; familiarity; reliability; flexibility; and universality. In assessing a research unit, three components should be examined: quality (best assessed by peer review and key metrics); impact (highly discipline specific – panel evaluations from academe and users of research, should assess selected cases using an audit model); and capability (capacity to contribute to future research – again panel evaluations). Suggests minimum-maximum percentage weightings for each component to ensure flexibility of model


116

to meet objectives. Bibliometric measures alone are not enough; these are best used as supportive measures (p10). ‘The best way to evaluate research is through assessment by peers and experts’ (p10). The paper suggests that impact indicators should be measured against the following criteria: knowledge diffusion; knowledge production; knowledge relationships; knowledge engagement (p12). ‘It is impossible to quantify the total ‘contribution’ of the humanities, the arts and the social sciences to society’ (p15). Page 18 presents a linear conceptual map of origin-production-outputs-outcomes. Page 24 maps the difference between curiosity-driven, panel-reviewed academic research, and problem-driven, professionally evaluated professional research: the value of particular research depends upon how the end-user is likely to be influenced and its perceived appropriateness. ‘Most evidence indicates that evaluation of quality is best achieved through appropriately informed qualitative judgement’ (p32). Outlines the HERG payback model (p42) as an example of impact evaluation methodology. It is important to distinguish between the level at which judgements are made, and the level at which they are reported (i.e. RAE assesses projects, but reports departments). Both quality and impact are important considerations, and are in some ways inseparable (p54). Page 69 recommends extended citation-based indicators given the difficulties in identifying suitable quality metrics.

Paper includes; An evaluation of the impact of research on policy A reflective piece about evaluation of the impact of research on policy A description of method(s) for evaluating impact of research on policy Evaluation of effectiveness/value for

� �


117

money of methods A review of methods for evaluating impact more generally Others of relevance (give reason)

�


118



Christiansen K with Hovland I The PRSP Initiative: multilateral policy change and the role of research Working Paper 216. Available via: http://www.odi.org.uk/publications/ August 2003 UK Overseas Development Institute




Empirical


A case study of the adoption of the PRSP initiative, and the role of research and other factors in this process.

Study design

Literature review and documentary analysis, 33 in-depth interviews (from a range of UK and Washington-based organisations), and circulation of working paper.

Theory

Weiss (research utilisation models) and Lipsky, 1980 (street-level bureaucrats).

Analysis

The relative contribution of research is assessed using a three-dimensional framework: policy context (politics and institutions); evidence (credibility and communication); and links (influence and legitimacy).

Findings/key points

The case study examines context, evidence and links to identify factors that facilitated research impact (i.e. networks, communication, credibility, linkages, conducive contexts), and maps key players/stakeholders. Analysis finds that both informal and formal networks contributed to the speed of acceptance of ideas within international development policy field. The paper tracks key events, and notes references within relevant publications. It concludes that communication and rigorous evidence increase the probability of research


119

uptake. Furthermore, researchers are likely to have an indirect influence via networks, and evidence can change the policy context (p38).


�


120



Referenced


Church C Mind the gap: policy development and research on conflict issues International Centre of Excellence for Conflict and Peace Studies http://www.incore.ulst.ac.uk/policy/rip/RIP.pdf 2005 UK University of Ulster, Londonderry


Policy research


Empirical


To what extent is research on conflict issues being used in policy development?

Study design

Literature review, questionnaire to researchers, interviews with policy makers.

Theory

Weiss (1980) and Pelz (1978) on types of utilisation: conceptual/enlightenment; symbolic/legitimative; instrumental/ engineering.

Analysis

Findings/key points

Within this study, most of the analysis relates to interview data ‘due to the subjective nature of the topic, the importance of nuance in language and ideas, and the paucity of literature specific to conflict issues, as well as the limited success of the researcher questionnaire’ (p10). The authors found that conflict research primarily influences the policy context through an enlightenment function, with instrumental utilisation a distant second. Many factors impede the accurate measurement of the influence of research on policy, including: definition of policy; language (impact, use, dissemination, influence); and discerning cause and effect. Molas-Gallart et al (2000, p1): ‘When compared with the physical, engineering and


121

medical sciences, [social science research’s] contribution is likely to be more indirect, and more difficult to observe.’ ‘As with anything subjective, there is room for error (of judgement)’ (p13). The paper also notes that policy makers are likely to underestimate the influence of research (i.e. may be indirect through personal advice). The remainder of the paper then discusses how to bridge the policy-research gap.


� �


122



Internet search


Clark, W, Mitchell R, Cash D and Alcock F Information as influence: how institutions mediate the impact of scientific assessments on global environmental affairs Faculty Research Working Paper RWP02-044 http://belfercenter.ksg.harvard.edu/publication/ 2002 USA Harvard University, John F Kennedy School of Government


Environment


Conceptual


How have scientific assessments influenced the development of global environmental issues?

Study design

Theory

Analysis

Uses a framework for understanding: 1. Changes in the issue domain; 2. Information as a source of instability; 3. How does information have influence?; 4. When and which information has influence?

Findings

Science rarely influences policy directly, but can exert instead an indirect influence on the long term development of an issue via: who actively participates in discussions; how discussions and negotiations are framed (local, national); what policy goals, management options and technical knowledge are emphasised; and the relative visibility of the policy issue. It is difficult to establish attribution (p11) It is important to identify factors that, when influenced by information, drive issue development and consequently, policy development (p12). The paper finds that the following factors affect the potential influence of a particular piece of research:


123

embeddedness/neutrality of author; boundary spanning; and saliency/credibility/legitimacy and trade off between factors, which will have implications for the scope of evaluation. The authors conclude that even where information can completely alter the issue domain, it may still have little influence.


�


124



Internet search


Committee on Science, Engineering, and Public Policy (COSEPUP) Implementing the Government Performance and Results Act for research National Academies Press http://www.nap.edu/catalog.php?record_id=10106 2001 USA COSEPUP


Science


Empirical


A ‘snapshot’ evaluation of federal research programmes in response to the GPRA to identify best practice and issues in communication.

Study design

Panel meetings, focus groups (to inform scoping and questions), case study analysis of performance reports of five government agencies’ strategic and performance plans.

Theory

Analysis

Findings

GPRA compliance has added substantially to the costs of planning and evaluation. Evaluations also emphasise the role of communication, both within and between different agencies. The report finds that the most effective technique for evaluating research programmes is expert review using the criteria of quality, relevance and leadership.

Paper includes; An evaluation of the impact of research on policy A reflective piece about evaluation of the impact of research on policy A description of method(s) for


125

evaluating impact of research on policy Evaluation of effectiveness/value for money of methods A review of methods for evaluating impact more generally Others of relevance (give reason)

�


126



Referenced


Council for Medical Sciences The societal impact of applied health research: towards a quality assessment system. Royal Netherlands Academy for Arts and Sciences http://www.knaw.nl/publicaties/pdf/20021098.pdf 2002 The Netherlands Council for Medical Sciences


Health


Descriptive


A description of experiences of evaluating the societal impact of applied health research.

Study design

Overview of current impact evaluations and reflection on different approaches.

Theory

Analysis

Findings

Evaluations need to evaluate impact with regard to organisational mission. Bibliometrics alone are not sufficient indicators of impact, but are useful for considering the impact on quality assurance procedures (guidelines etc), and the impact of research on health care policy. In this respect it is possible to distinguish ex-ante and ex-post indicators of policy relevance. Ex-ante measures might look at the relevance of research questions and research to policy problems, and ex-post measures might look at levels of government funding, the extent to which findings reach relevant stakeholders, evidence of use and evaluation of a research group’s implementation strategy. Evaluations can also look for impacts on other products such as new technologies. It is important to remember that the independence/quality of research should not be compromised by attempts to promote use. The


127

report lists potential indicators (p26), and concludes that there is a need for a single widely accepted method for evaluating societal impact of applied health research.

Paper includes An evaluation of the impact of research on policy A reflective piece about evaluation of the impact of research on policy A description of method (s) for evaluating impact of research on policy Evaluation of effectiveness/ value for money of methods A review of methods for evaluating impact more generally Others of relevance (give reason)

�


128


ID reference number 35 Method of identification (e.g. database) Keith Pavitt Library search Details of publication Author Title Source (journal/conference etc) Year/volume/pages Country of origin Institutional affiliation

Cozzens S US research assessment: recent developments Scientometrics 1995 34(3) pp351-362 USA Office of Policy Support




Descriptive

Research question/ Aim

How has the Government Performance and Results Act (GPRA) affected programme evaluation?

Study design

Theory

Analysis

Findings

The GPRA emphasises multiple performance indicators, and interpretation within a descriptive framework. It measures performance against a specified goal, i.e. quantity, quality, timeliness, cost and outcome. Measures are in terms of ‘significant progress’ towards a goal if, for example, research is published in peer reviewed journals, and ‘outstanding progress’ if, for example, the research is considered to be the most influential in the field.


�


129



130



Referenced


Cranfield S and Ward H ‘Managing change in the NHS’ (2001) publications: report of a study on the uptake and use of two SDO publications National Co-ordinating Centre for the National Institute for Health Research Service Delivery and Organisation Programme (NCCSDO) http://www.sdo.lshtm.ac.uk/files/adhoc/change-management-survey.pdf 2002 UK NCCSDO


Health


Empirical


Assessment of the impact of two SDO publications.

Study design

Data on requests for the publications, their dissemination and numbers of downloads from the NCCSDO website and structured interviews with 21 individuals from team, service and organisational leadership levels conducted by telephone between December 2001 and March 2002.

Theory

Analysis

Notes from the interviews were transcribed and analysed.

Findings

Participants were asked questions about awareness and dissemination, appropriateness to target audience, methods of use and the quality and usefulness of the publications. ‘Telephone interviewing proved a useful means of making economical use of respondents time and conducting short very focused interviews was popular’ (p18)


�


131


�


132



Keith Pavitt Library search


Cunningham P, Boden M, Glynn S and Hills P Measuring and ensuring excellence in government science and technology: international practices: France, Germany, Sweden and the United Kingdom PREST (Policy Research in Engineering, Science and Technology). Available via: www.mbs.ac.uk/Research/engineeringpolicy/index.aspx 2001 UK PREST, University of Manchester




Descriptive


International comparison of evaluation methods by government and research institutes.

Study design

Theory

Analysis

Country-specific reports

Germany, Sweden and France largely conduct ex-ante assessments of research. Impact assessments (especially in Germany) integrate qualitative and quantitative methods (p46). Sweden uses an expert panel to comment on the value of research (five increments: excellent, very good, good, fair, poor). There are no overarching guidelines/methodologies to measure impact in Germany, Sweden or France. The UK has detailed research and development assessment guidelines but these are impractical in application. The principles of ROAME analysis are widely accepted, especially for ex-post evaluations.

Paper includes An evaluation of the impact of research on policy A reflective piece about evaluation of the impact of


133

research on policy A description of metho(s) for evaluating impact of research on policy Evaluation of effectiveness/ value for money of methods A review of methods for evaluating impact more generally Others of relevance (give reason)

�


134



Referenced


Davies H, Nutley S and Walter I Approaches to assessing the non-academic impact of social science research: report of the ESRC symposium on assessing the non-academic impact of research Research Unit for Research Utilisation www.odi.org.uk/rapid/Meetings/ESRC/Index.html 2005 UK University of St Andrews




Conceptual

Research question/ aim

Explores why it is important to assess the impact of research on public policy, and potential ways of doing so.

Study design

Theory

Analysis

Presents impacts on HERG spider diagrams (of knowledge production, research capacity building, policy/product development, sector benefits, wider societal benefits).

Findings

Indirect influence is more important in the long run and impact should not be seen as an end-stage. Raises issues of where to look for impacts, when to look and how to assess attribution. Approaches to assessing impact need to be purposeful, pragmatic and cognisant of the complexities involved. Comprehensive impact assessment of an entire portfolio is time consuming and costly. Different stakeholders may want information on impacts for different reasons (p4). It is important to distinguish between the type of research (basic/applied, what/how etc) and to be realistic about expected impacts, which can be reciprocal between research and policy (p16). Impact is a function of research itself and the context within


135

which findings are delivered, and does not equate to the ‘value of research’. Study participants highlighted the folowing as ways of providing an informed opinion about overall impact: international expert panel; an impact ‘log’; learning from key case studies; and developing proxy measures of impact by monitoring process.


�

�


136



Referenced


Davies R and Dart J The ‘Most Significant Change’ (MSC) technique: a guide to its sse Care International et al http://www.mande.co.uk/docs/MSCGuide.pdf April 2005 UK Care International




Descriptive


How can the MSC technique be used to monitor and evaluate social change programmes?

Study design

Theory

Analysis

Findings/key points

MSC is a participatory technique that can be used to monitor and evaluate social change programmes (applicable across sectors). The process involves collecting stories of ‘significant change’ at field level. Panels then select from these. The ‘stories’ (no pre-defined indicators but broad domains such as ‘changes in peoples’ lives’) focus on intermediate outcomes and impact. After a period of, for example, a year, the stories are collated into a report. The authors reflect that this methodology encourages capacity building and analysis, as panels have to decide which changes are more important.



137

A description of method(s) for evaluating impact of research on policy Evaluation of effectiveness/ value for money of methods A review of methods for evaluating impact more generally Others of relevance (give reason)

� �


138



Deelstra Y, Nooteboom S G, Kohlmann H R, van der Berg J and Innanen S Using knowledge for decision-making purposes in the context of large projects in The Netherlands Environmental Impact Assessment Review 2003 23(5) pp517-541 The Netherlands DHV Management Consultants




Empirical


Assesses the contribution of impact assessments in decision making, barriers to uptake, and means of evaluating the use of research.

Study design

Two case studies

Theory

Model of Socially Acceptable Decisions (p523) which are consensus-oriented rather than conflict-oriented

Analysis

Findings/key points

Agreement about decisions is not the same as agreement about policies. An effective impact assessment would account for as many views and processes as possible. The paper suggests ‘ex-post satisificing’ (the extent to which involved actors are satisfied with the result of the policy process). It examines the theory about how decision makers consider knowledge (problem solving or enlightenment) and windows of opportunity. Whilst the report does not account for data/information collection, it does describe the policy dialogue within networks and at meetings, and the involvement of various actors.


�


139


�


140



Delanghe H and Muldur U Ex-ante impact assessment of research programmes: the experience of the European Union’s 7th Framework Programme Science and Public Policy 2007 34(3) pp169-183 Belgium European Commission




Empirical


What can be learned from the impact assessment of the 7th Framework Programme for R&D?

Study design

Theory

Analysis

Findings/key points

The EU has practised ex-post evaluations in this field for over ten years, but ex-ante only since 2005. A 2004 evaluation of the programme found that the quality of evaluations could be increased through better assessments of the trade-offs and inter-linkages between impacts, improved quantification and improved understanding of time horizons. The Commission Impact Assessment considers economic, social and environmental impacts, and is looking to develop an impact assessment for research policy. Page 174 proposes three frameworks to assess macro-economic impact. The Framework Programme assessment uses macro-economic and quantitative measures (patents and participation counts) to measure change over a specified time period. The 7th


141

Framework Programme focuses on four principal components: ideas, capacities, people and co-operation. Ex-post evaluations are systematically used to gain insights into innovation impact.


�


142



Earl S and Carden F Learning from complexity: the International Development Research Centre’s experience with Outcome Mapping Development in Practice 2002 12(3/4) pp518-524 Canada International Development Research Centre




Descriptive


What is Outcome Mapping as an evaluation methodology?

Study design

Theory

Analysis

Findings

Outcome Mapping (OM) is a participatory planning tool (but can be used for ex-post evaluations) to describe outcomes, such as changes in behaviour, relationships, activities/actions by boundary organisations. The authors argue that OM is a robust methodology to think strategically and holistically about desired results. The primary methodology for data collection is a stakeholder workshop. OM fosters learning by incorporating self-assessment and reflection throughout the planning, monitoring and evaluation stages, and assesses contribution not attribution. It is suitable where evaluation is primarily for learning and improvement.



143


�


144



Earl S, Carden F and Smutylo T Outcome Mapping: building learning and reflection into development programs International Development Research Centre http://www.idrc.ca/en/ev-9330-201-1-DO_TOPIC.html 2001 Canada IDRC




Descriptive


Outlines the methodology and goals of Outcome Mapping (OM), and how it can be used.

Study design

Theory

Analysis

Findings

OM can be used at both the project and programme level to evaluate changes in the behaviour of boundary organisations. It is designed primarily as a tool for performance monitoring and learning, and monitors progress towards indicators. Pages 12 and 13 outline the OM framework and how to conduct an OM workshop (primary methodology). Records findings of discussions/ workshops on a strategy matrix (p39).



145


�


146



Edwards K PCRs and policy influence. What project completion reports have to say about public policy influence by Centre-supported research: final report International Development Research Centre Evaluation Unit. Available via: http://www.idrc.ca/index_en.html 2001 Canada IDRC




Empirical


Study design

Review of a selection of project completion reports to explore what they report about: how and to what extent the research commissioned has influenced public policy; the nature and type of impact; the mechanisms and approaches supporting influence; and the barriers and facilitators to influence

Theory

Analysis

A total of 100 project completion reports were selected from the IDRC database for analysis, focusing on the information given on: ‘areas in which the project can reasonably be expected to have a significant, positive development impact, especially public policy.’ Only looked at 75 due to time constraints. Information was taken from the responses to a wide variety of questions. It is collated in tables and analysed to identify trends emerging from across the PCRs.

Findings

Reflections on the use of project report analysis. Challenges included the lack of specific information provided about


147

policy influence, and the fact that much of the information is ambiguous and unconfirmed and concerns potential rather than actual impact. There are also considerable gaps in the data. The author has more confidence in the information given about the links established between research and policy, about the range of mechanisms and approaches, and the factors that impact on policy influence


�


148



Elliot H and Popay J How are policy makers using evidence? Models of research utilisation and local NHS policy making Journal of Epidemiology and Community Health 2000 54(6) pp461-468 UK University of Salford


Health


Empirical


What are the factors that enable or impede evidence-based policy making?

Study design

Qualitative study of local policy makers within the NHS, using literature review, documentary analysis and case studies.

Theory

Weiss (problem-solving and interactive models of research utilisation) and Giddens (dialogical model).

Analysis

Findings/key points

The direct influence of research evidence is tempered by factors such as financial constraints and timescales. Research is more likely to impact on policy indirectly, including shaping policy debate and mediating dialogue between producers and users. Uses a literature review and nine case studies to better understand the use of research evidence in NHS policy making. Documentary analysis used to identify case studies from research priorities; case studies informed by in depth interviews, documentary analysis, meeting observations and historical tracing. Research can be used to ‘answer’ immediate policy questions, illuminate wider policy issues and develop new relationships between research users and


149

producers. Paper includes; An evaluation of the impact of research on policy A reflective piece about evaluation of the impact of research on policy A description of method(s) for evaluating impact of research on policy Evaluation of effectiveness/value for money of methods A review of methods for evaluating impact more generally Others of relevance (give reason)

� �


150



Referenced


European Commission Assessing the impact of energy research EUR 21354 http://ec.europa.eu/research/energy/pdf/erevia_en.pdf 2005 Belgium European Commission


Environment


Descriptive


Compares evaluation procedures in Member States.

Study design

Summary of six meetings/presentations over 16 months with EREVIA Group (international representation)

Theory

Analysis

Findings/key points

Research evaluation traditionally considered whether research outputs were achieved within time/budget. Now it considers efficiency, cost-effectiveness, transparency and impacts on science/industry/policy (purposes of accountability). Most member states prioritise qualitative aspects of evaluation, but it is possible to quantify impacts as long as projects have well-defined and measurable objectives : quantification is generally sought but not always possible (p14). Impact assessments should account for local context, goals and the areas on which research is likely to have an effect. In France, evaluations examine efficiency of action (link between input/output), effectiveness (link between objectives and results) and utility and sustainability (link between societal/economic/environmental problem areas and impacts). Denmark measures use of results: use within target groups; dissemination to relevant groups; objective evaluation of usefulness of project resul. The overall impact of a project is rated


151

by assigning a score for each criterion, and making a judgement on the sum of the scores (pp15-16). Common methodological issues in member states include how to link indicators with impact assessment, how to progress from qualitative to quantitative analysis, and how to assess outputs/inputs in long term research (p22). Problems will arise if evaluation indicators do not link with ex-ante programme objectives. Evaluations need to measure ‘additionality’ by comparing actual outcomes with what would have happened without the research.


� � �


152



Evans J P Lost in translation? Exploring the interface between local environmental research and policy making Environment and PlanningA 2006 38(3) pp517-532 UK University of Birmingham


Environment


Empirical


Explores the intersection of environmental science and policy making

Study design

‘A detailed case study of an end user orientated research project’ (p518), drawing on data collected between 1998 and 2004 using semi-formal interviews, documentary analysis, observations of meetings and field visits. The research project was one of 41 funded as part of the Natural and Environmental Research Council-funded Urban Regeneration and the Environment (URGENT) programme.

Theory

Analysis

Findings

The study design is the main dimension of interest (see above details of the case study approach used). No reflection on the method apart from to comment that it is dangerous to extrapolate from a single case.

Paper includes; An evaluation of the impact of research on policy A reflective piece about evaluation of the impact of research on policy A description of method(s) for evaluating impact of research on policy

�


153



154


ID reference number 48 Method of identification (e.g. database) British Library catalogue search Details of publication Author Title Source (journal/conference etc) Year/volume/pages Country of origin Institutional affiliation

Fahrenkrog G, Polt W, Rojo J, Tübke A and Zinöcker K RTD Evaluation Toolbox: assessing the socio-economic impact of RTD policies IPTS Technical Report Series, EUR 20382 http://ftp.jrc.es/eur20382en.pdf 2002 Spain European Commission, Joint Research Centre, Institute for Prospective Technological Studies




Descriptive


Report of the methodologies for evaluating the socio-economic impact of research and technological development policies and examples of good practice from European Member States.

Study design

Theory

Analysis

Findings/key points

The practice of evaluation has developed from purely quantifying policy impacts towards facilitating mutual learning in the policy process. Current EU evaluation practice focuses more on monitoring than impact assessment, and relies upon panel evaluation. Different methods should be complements, not substitutes. ‘Any attempt to evaluate a…programme in isolation, without reference to controls and comparisons, would have to be regarded as bad practice’ (p25). Details the use of econometric and social analysis (p140), expert panels and peer review (p163) (can be both ex-ante and ex-post),


155

network analysis (p183) and benchmarking (p201).


�


156


ID reference number 49 Method of identification (e.g. database) BLDS Library search Details of publication Author Title Source (journal/conference etc) Year/volume/pages Country of origin Institutional affiliation

Fisher E and Holland J D Social development as knowledge building: research as a sphere of policy influence Journal of International Development 2003 15(7) pp911-924 UK University of Wales




Descriptive


Looks at the capacity building initiative that tried to generate knowledge as a resource for policy formulation.

Study design

Description of the design and implementation of a project (entitled Project Magnolia) designed to promote the use of knowledge in policy in ‘Southern’ countries. Reflection on the methodology for evaluating projects.

Theory

Analysis

Findings

The project was evaluated through an Outcome to Purpose Review (OPR). This is an internal DFID management tool for monitoring progress that uses the logical framework methodology.A lesson learning workshop was also held, which used brainstorming techniques with a range of actors from the project. The authors were concerned that the OPR approach was not likely to capture the nuances of the process and suggest the need for ‘a methodology that documents and generates a narrative about the project process, rather than ticking off points against a log-frame’ (p920).

Paper includes


157


� A reflective piece about an attempt to promote the use of knowledge in policy making, with a short section commenting on methodological approaches to evaluating projects designed to promote the use of knowledge in policy.


158



Internet search


Flint M E S Natural resources research evaluation synthesis Evaluation Report EV:604. Available via: http://www.dfid.gov.uk/aboutdfid/evaluation.asp December 1998 UK Department for International Development

Field (e.g. environmental research) International development Type of paper (conceptual, empirical, descriptive)

Empirical


Three ex-post evaluation studies to determine success or failure of the research projects.

Study design

Field visit, interviews, documentary analysis, expert review of findings.

Theory

Analysis

Findings/key points

Projects were found to be partially successful: they achieved technical success but did not impact on policy or practice (i.e. successful outputs but less successful uptake and impacts – impacts on poverty, health etc were graded A+ to D). ‘A recent review by the Consultative Group on International Agricultural Research (CGIAR, 1997) of 87 ex post impact assessments concluded that there was a need for systematically and reliably collected information on intermediate products and indirect outcomes (4.6). At present, very little is known about the impact of CGIAR research on end-users, food security, poverty or natural resources. Until relatively recently, assessment of CGIAR and other research programmes tended to be limited to a peer review of scientific aspects.’ Stakeholder needs should be assessed as part of the evaluation. Research that is intended to influence policy should be explicit about the policy context and trends, and ways in which research can influence policy. Existing knowledge must be critically examined to assess what new policy-relevant information can be expected. The importance of external factors needs


159

to be explicitly addressed by specialists as part of the research project design. The evaluations quantified the impact of the three projects on outputs, purpose and goal in a matrix on a scale of 1-5 (p12). The report also discusses probable factors affecting uptake and utilisation of research findings.


�


160



Gardner B L Returns to policy-related social science research in agriculture Impact Assessment Discussion Paper 9 http://www.ifpri.org/impact/iadp09.pdf May 1999 USA International Foof Policy Research Institute


Environment


Empirical


What is the value of policy-related social science research impact on specific policy change?

Study design

Three case studies.

Theory

Uses Bayesian theory to assess the ex-ante value of research, estimated from decision makers’ prior probabilities of the state of the world and the likelihood of correct and incorrect research findings.

Analysis

Prospective evaluation, starting from the research.

Findings

The evaluations quantify the value of research as the expected gain from policy decisions influenced by the information generated. The extent to which uncertainty is reduced is a good proxy measure for the ex-ante value that can be attributed to research. Measures this impact by accounting for changes in political actions both of decision makers, and those seeking to influence government. However, this approach only measures economic value and not scientific contribution.

Paper includes; An evaluation of the impact of research on policy

�


161


�


162



Garfinkel M S, Sarewitz D and Porter A L A societal outcomes map for health research and policy American Journal of Public Health 2006 96(3) pp441-446 USA J Craig Venter Institute


Health


Conceptual


Can ‘road mapping’ improve research and policy linkages with health outcomes?

Study design

Road mapping and technology assessment of case study (perinatal health). Scientometrics and text mining of literature.

Theory

Analysis

Uses both research and policy recommendations as inputs, working towards a specified outcome. Can map gaps and knowledge, and helps decision makers to compare outcomes and tradeoffs.

Findings

Uses ‘road mapping’ (a graphical overview of potential solutions over time) to identify research and policy combinations that could potentially influence health outcomes in general. The ex-ante methodology uses both research and policy recommendations as inputs towards a specific outcome. The method draws upon the principles of Outcome Mapping (although unlike OM, does not refer to a workshop). The authors reflect that road mapping can help identify both gaps and knowledge, and enable decision-makers to compare these trade-offs. Furthermore, the method contributes


163

to a more open policy debate. Paper includes; An evaluation of the impact of research on policy A reflective piece about evaluation of the impact of research on policy A description of method(s) for evaluating impact of research on policy Evaluation of effectiveness/value for money of methods A review of methods for evaluating impact more generally Others of relevance (give reason)

�


164


ID reference number 53 Method of identification (e.g. database) Internet search Details of publication Author Title Source (journal/ conference etc) Year/volume/pages Country of origin Institutional affiliation

General Accounting Office Pipeline safety: systematic process needed to evaluate outcome of research and development program GAO-03-746 http://www.gao.gov/new.items/d03746.pdf 2003 USA General Accounting Office




Empirical


How does OPS distribute funding between priorities? What are expert views of priorities? How does the OPS evaluate R&D Outcomes?

Study design

Documentary analysis (including of events and workshops); interviews with policy officials, stakeholders and field experts (selection criteria: knowledge, balance and independence) – 49 of 55 responded.

Theory

Analysis

Findings/key points

The evaluation mapped expert opinion about strategic priorities against policy (and funding) priorities, and found that expert review helped the process of aligning priorities. The category of R&D (theme, subject) was tabled against the intended goals (p20). The authors reflect that a systematic evaluation process is needed to demonstrate progress towards achieving outcomes. Attempts to evaluate research outcomes have been limited, and have tended to be project-specific. Although ‘success stories’ are reported, this paper finds them to be selective and not an adequate assessment of programme-wide performance (p27).


165

However, best practices do help agencies systematically evaluate research outcomes (setting goals, and measuring progress towards these).


� �


166



Government Accountability Office US public diplomacy: actions needed to improve strategic use and coordination of research GAO-07-904 http://www.gao.gov/new.items/d07904.pdf July 2007 USA Government Accountability Office




Empirical


How do US government agencies use research to support their strategic communication objectives? How do agencies identify end-user needs, assess end-user satisfaction and coordinate the sharing of available research?

Study design

Interviews, documentary analysis of the scope and quality of research, case study analysis of specific communication efforts, international comparison of different approaches, and a workshop with senior stakeholders.

Theory

Analysis

Findings/key points

In addition to the above methods, the evaluation uses media monitoring to support communications strategies. The authors reflect that once target audiences are identified, the use of surveys and focus groups can be used to gauge awareness and perceptions of programmes.



167


� Useful methodology to consider the process of research influence on policy.


168



Georgihou L Assessing the framework programmes: a meta evaluation Evaluation 1995 1(2) pp171-188 UK PREST (Policy Research in Engineering, Science and Technology), University of Manchester




Descriptive


Review of the experience of evaluating the EU R&D framework programmes, 1985-1995. Outlines a range of methods used and comments on their appropriateness and interactions with the policy system.

Study design

Theory

Analysis

Findings/key points

DG XII has used panel evaluations with professional support through detailed additional studies (when the time and resources are available). The author reflects that the use of review panels is appropriate in the EU context as it can offer a national balance (with representatives from 6-8 countries) that a single study team cannot, and is also considered to be fair and independent. However, panellists have found it easier to comment on the technical quality of the research, than its potential impacts (because it is too early to look at impacts at the stage of review; because supporting work for the evaluation is conducted too quickly; and because panel members can


169

lack skills in evaluation). The EU has also commissioned national impact studies conducted by professional evaluators rather than panels. UK IMPACT involved a large scale survey of Framework participants, interviews and statistical data. The impact studies have been constrained by resources (often relying on centrally designed surveys). The EU has also conducted horizontal evaluations on key issues or themes. For evaluations to be used, the author considers credibility, absorbability and steer-ability (a mechanism to implement findings) to be important.


�


170



Georghiou L and Davis E (eds) Evaluation of R&D: a policymaker’s perspective. Proceedings of the International Workshop on Assessment and Evaluation, 17/18 Nov 1988 HMSO 1988 UK Department of Trade and Industry




Descriptive


What is the process of R&D evaluation, and how are these evaluations useful to policy makers?

Study design

Theory

Analysis

Findings/key points

It is important to understand the motivation for evaluation, the scope, and whether it is ex-ante, real-time or ex-post evaluation. Highlights the National Science Foundation’s use of surveys/interviews to gauge the extent to which industrial practice is based on research. Also discusses horizontal evaluation (European) and ROAME (DTI) (p12). One would expect different outcomes depending on the timing of the evaluation. Page 19 suggests a five year time frame for industrially oriented research. The DTI experience is that evaluation should be integral to decision-making.



171


�


172


ID reference number 57 Method of identification (e.g. database) Referenced Details of publication Author Title Source (journal/ conference etc) Year/volume/pages Country of origin Institutional affiliation

Gerhardus A, Dujardin M, Kiet, P H T, Siddiqi S and Sauerborn R A methodology to assess the use of research for health policy development Discussion Paper 04 2000 Germany University of Heidelberg, Medical School


Health


Descriptive


Develops a model to map research-policy flows.

Study design

Analysis of nine case studies (no details in paper) to develop methodology.

Theory

Analysis

Findings/key points

Research findings are often distorted, modified or diluted before they enter health policy – with the rare exception of those cited in policy papers. Looks to develop indicators to compare the use of research before and after an intervention to enhance utilisation. ‘Research results can take many paths to its users and it is difficult to trace these paths’ (p3), and the complexity of other influences on policy. The authors reflect upon limitations in interviewing (decision makers may be unaware that their information is based on research). The model maps flows between different research producers to users, either directly or via a research broker (p5). The authors recommend that the impact of research has to be assessed in relation to other factors. Indicators to measure research utilisation include: access, communication, knowledge, use, and influence on points of view/policy


173

issues. The methodology includes documentary analysis and interviews, giving a point score according to the weighting of which part of specific research was used (p8). The authors reflect that point scoring can highlight weak areas for improvement.


�


174



Gerhardus A and Kielmann K The use of research for decision-making in the health sector: the case of ‘Shared Care’ in Burkina Faso In: Lessons in research to action and policy: case studies from seven countries. Available via: http://www.cohred.org/main/ 2000 Switzerland Council on Health Research for Development (COHRED)


Health


Empirical


What are the push and pull factors of research non-utilisation?

Study design

Documentary analysis to identify key stakeholders and the Shared Care agenda; semi-structured interviews and focus groups.

Theory

Analysis

Findings/key points

The authors explored the hypothesis that the rationale behind the Shared Care agenda was based on study findings. The evaluation assessed how findings were communicated and to whom, the involvement of decision makers in the development of research, the validity of research, relationships between key stakeholders, and external factors such as cultural issues. The study showed that even where research results are well known and trusted as valid by decision makers, they may not be acted upon unless issues of context, communication and ownership are addressed.

Paper includes


175


� �


176



Internet search

Details of publication Author Title Source (journal/conference etc) Year/volume/pages/ Country of origin Institutional affiliation

Godin B and Doré C Measuring the impacts of science: beyond the economic dimension Canadian Science and Innovation Indicators Consortium, Montreal www.csiic.ca/PDF/Godin_Dore_Impacts.pdf 2003 Canada CSIIC


Science


Conceptual


What are the non-economic impacts of scientific research, and how can measurement of these influences be approached?

Study design

Interviews with research users and producers

Theory

Analysis

Findings/key points

‘Most of the literature is concerned with the definition of the right approach to be used in the evaluation of impact or simply with the description of the available methods to do so’. ‘For many, the concern in measuring non-economic impact depends on a better knowledge of the mechanisms of research transfer’ (p3). The authors propose 11 categories of impact (p6) of which policy is one. ‘Political impact has to do with the way knowledge influences policy-makers and policies: the interest and attitudes of politicians, administrators and citizens towards a question of public interest involving science and technology, public action (law-jurisprudence-ethics, policies, programs-regulation-norms, standards) and citizen participation in scientific and


177

technical decisions.’ (p7). The report identifies three challenges in approaching an impact evaluation: 1. distinguishing between output, impact and influence; 2. identifying transfer mechanisms through which science translates into impact; 3. developing appropriate and reliable instruments and indicators. Evaluations need to account for the presence of change, and its quantity, duration and frequency.


�


178



Government of Western Australia Preparing performance indicators: a practical guide http://www.dpc.wa.gov.au/psmd/pubs/psrd/preppis.html 1997 Australia Public Sector Management Office, Department of the Premier and Cabinet




Descriptive


A guide to developing performance indicators and how to link outputs and outcomes.

Study design

Theory

Analysis

Findings/key points

Performance Indicators are largely for monitoring and managing performance, and they are useful for accountability and budget management. They measure efficiency with regard to outputs, and effectiveness with regard to outcomes. Evaluations need to ascertain whether the desired outcome relates to behavioural change (i.e. regulations, advice), satisfying community needs (specific goods and services) or revenue raising. Indicators need to assess the extent to which predetermined targets are achieved, trends over time, performance relative to similar agencies, performance relative to predetermined benchmarks. In calculating overall efficiency, it may be necessary to weight measures. Even though policy work is a support activity, policy agencies do need performance measures. ‘Since it


179

is often not pertinent to assess policy outcomes at an agency level, assessment needs to focus on policy inputs, processes and outputs as well as the extent to which policy advice is accepted.’ (Appendix B). The report suggests assessing the effectiveness of policy advice by focusing on policy inputs, processes and outputs as well as the extent to which policy advice is accepted (assess quality and output quantity of policy advice). The authors recommend using panel/peer review to do so.

Paper includes An evaluation of the impact of research on policy A reflective piece about evaluation of the impact of research on policy A description of method(s) for evaluating impact of research on policy Evaluation of effectiveness value for money of methods A review of methods for evaluating impact more generally Others of relevance (give reason)

� (taking policy advice to be research)


180


ID reference number 61 Method of identification (e.g. database) Database search Details of publication Author Title Source (journal/ conference etc) Year/volume/pages Country of origin Institutional affiliation

Grosse S D, Teutsch S M and Haddix A C Lessons from cost-effectiveness research for United States public health policy Annual Review of Public Health 2007 vol 28 pp365-391 USA Centers for Disease Control and Prevention


Health


Conceptual, empirical (four case studies)


Review of the methods for, use of, and implications of findings from cost-effectiveness policy research, and the extent to which this has influenced policy (focuses on economics from a societal evaluation perspective as this is most relevant to policy decisions)

Study design

Theory

Analysis

Findings/key points

‘Reliable evidence of effectiveness in terms of health outcomes [is] frequently lacking, but valuing different types of outcomes is an inherently subjective, value-laden process in which economic methods need to be combined with broad stakeholder input and deliberation’ (p366). Although economic evaluations are not a requirement of policy-relevant research, they can lead to more informed decisions. However, in certain policy areas (i.e. public health) ‘policy makers should resist the idea that cost-effectiveness should be a factor in decision making’ (p380). Cost-effectiveness analysis can be used to


181

assess the value of preventative services, programmes and alternatives. Finds that the influence of economic evaluations is often ‘difficult to ascertain. They can be an important component of decision making along with evidence of effectiveness and contextual information’ (p382).

Paper includes An evaluation of the impact of research on policy A reflective piece about an evaluation of the impact of research on policy A description of methods for evaluating impact of research on policy Evaluation of effectiveness/value for money of methods for evaluating the impact of research on policy A review of methods for evaluating the impact of research on policy A review of methods for evaluating impact more generally Specific reference to/relevance to environmental policy Others of relevance (give reason)

� A reflective piece about the relevance and utilisation of cost-effectiveness research for policy.


182



Guy K, Amanatidou E and Psarra F Framework Programme 5 impact assessment: a survey conducted as part of the five-year assessment of European Union research activities (1999-2003) Science and Public Policy 2005 32(5) pp349-366 UK Wise Guys Ltd/European Commission




Empirical


A report of the EU 2004 survey to assess the impact of projects and actions during the Fifth Framework Programme (FP5).

Study design

Internet questionnaire with 1,700 responses to general questions, and 1,000 responses to questions about specific projects.

Theory

Analysis

Findings/key points

Impacts were assessed in a ‘three circle’ model: ‘first circle’ impacts on the research teams themselves; ‘second circle’ impacts on the parent organisations of these teams; and ‘third circle’ impacts on broader user communities and environments. Goals of research projects include policy influence, but also broader management/ commercial objectives. Participants were asked to rank the importance of objectives for each ‘circle’, and also to rank (scale of 1-5) how they perceived the impact of research on user groups. The authors note that attainment of important goals does not always lead to large impacts. Equally, modest attainments of less important goals can


183

have large impact. Analysis of responses found significant over-reporting of impact/attribution to participants’ own research. ‘A questionnaire addressed only to participants [is not] the most effective way of collecting information on the impacts of participation on anybody other than the participants themselves’. Questionnaires can also be very revealing about goal attainment and the impacts of participation, in particular, user satisfaction.


�


184



Hanney S, Davies A and Buxton M Assessing benefits from health research projects: can we use questionnaires instead of case studies? Research Evaluation 1999 8(3) pp189-199 UK Brunel University


Health


Empirical


An empirical comparison of the methodological appropriateness and reliability of case study and questionnaire methodologies, with regard to their use within the HERG Payback model

Study design

Compares a questionnaire devised to evaluate North Thames NHS research, with a case study approach.

Theory

Analysis

Findings/key points

The authors found that questionnaires could provide a relatively robust mechanism for routine monitoring of a broad portfolio, but would benefit from a small number of accompanying case studies that would help to verify the information given and to test and improve the questionnaire for future use. On p190 the authors reflect that assessing research impacts on policy creates some conceptual and practical problems, even where case studies are used. Robertson (1995) attempted cost-benefit analysis of transport research and claimed that the benefits can be aggregated as a monetary value (p191). However, the authors conclude that a non-aggregated multidimensional approach is most


185

helpful when evaluating health research from a portfolio. Large scale case studies can be resource-intensive. Page 191 notes a 70% response rate from the questionnaire sample. Analysis of the questionnaire findings involved some bibliometrics, and a scoring methodology for some of the descriptive answers. An average of 30 hours was used for each case study (including documentary analysis, and interviews with researchers and users of research p194). The authors reflect that case studies enable the exploration of specific issues, and that a questionnaire cannot (p196).


�


186



Hanney S R, Gonzalez-Block M A, Buxton M J and Kogan M The utilisation of health research in policy-making: concepts, examples and methods of assessment Health Research Policy and Systems 2003 1(2) http://www.health-policy-systems.com/content/1/1/2 UK Brunel University


Health


Conceptual


What are the issues in health research utilisation? How can the process of research utilisation within policy making be assessed?

Study design

Theory

Rational/incremental/garbage-can theory. Enlightenment, problem-solving, political, tactical and knowledge-driven utilisation models.

Analysis

Findings/key points

The WHO Health Research Systems Analysis Initiative reviews the issues relating to research utilisation, and makes recommendations about appropriate methods for assessing such utilisation. There is an increasing focus on systems analysis. ‘A full review of the many meanings of research impact reveals that there may be more utilisation in policy-making than is sometimes recognised’ (p2). ‘Developing a conceptual framework of the processes of utilisation should assist with the formulation of assessment tools that reveal the full picture of the way research is used in policy-making’ (p2). The WHO is


187

increasingly interested in assessing the role of research utilisation in policy-making (Hanney et al, 2002). Page 3 – the prime focus of assessment of research impact on policy making should be the policy maker (as it is he who makes policy decisions). Such assessments should be part of the assessment of wider research utilisation (i.e. by industry, medical practitioners, the public) as these also influence policy (p3). The report defines research-informed policies as secondary outputs from research, but not the final desired outcomes (which are benefits to health). Page 5 emphasises that not all health knowledge utilisation goes through a policy making stage, or does so after partial translation of findings into practice (i.e. clinical guidance follows practice). The utilisation of technical research is less influenced by political persuasion than is the use of social science research (p6). Different types of research are likely to be used for different policy making purposes (administrative, legislative, clinical, applied, basic research). Suggests that policy networks are a useful framework to study utilisation (interactive model). The cited example of impact analysis (in South Africa and Zambia) drew on documentary analysis, key interviews, media analysis and secondary analysis. Data analysis included timelines of reform, stakeholder analysis, policy mapping, impact analysis and a review process. Network analysis is most suited to studies of research centres as opposed to individual research projects. The report suggests that documentary analysis and in-depth interviews are the two most frequently used methods (and usually together), with surveys useful for analysis of impact at local level policy making, and bibliometrics for broader studies (p12). The report also discusses different situations whereby research is more likely to be taken into account (i.e. through intermediary organisations, good


188

practice dissemination). The authors do not reflect on how this should be incorporated into impact assessment, but suggest it is nonetheless important for institutional learning. The purposes of assessing the utilisation of research include: accountability, future prioritisation and learning. Assessment methods have to be developed to reflect the purpose of assessment, analysis about the types of research, interfaces and receptors, and utilisation possibilities; and the roles of ex-post and real-time assessments (p20). Suggests scales of describing the level of research utilisation: 1. consistency of policy with research findings; 2. degree of influence of research on policy agenda setting; 3. degree of influence of research on policy formulation; 4. degree of influence of research on policy implementation. The authors recommend triangulation to enable analysis of contribution (if not attribution).


� � �


189



Hanney S R, Grant J, Wooding S and Buxton M J Proposed methods for reviewing the outcome of health research: the impact of funding by the UK’s ‘Arthritis Research Campaign’ Health Research Policy and Systems 2004 2(4) http://health-policy-systems.com/content/2/1/4 UK Brunel University


Health


Descriptive and empirical.


Description of how the researchers plan to carry out an evaluation of the impact of the Arthritis Research Campaign and three pilot studies.

Study design

Theory

Analysis

Findings

The researchers plan to use case study analysis drawing on multiple sources of evidence (converging on the same issues and aiming to triangulate between sources). These case studies include: documentary and literature review, semi-structured interviews with key informants and bibliometric analysis. An evaluation framework was used to organise the case studies in the assessment of outcomes. The framework has two elements: The payback model is intended to assist assessment rather than to provide an accurate description of the relationship between knowledge and policy. The study also used a logic model for assessing the outputs and outcomes. Each


190

of the 16 cases will be written up individually as a narrative based on the stages of the logic model to shed light on processes involved and contribute to cross case analysis. The cross case analysis will attempt to identify factors associated with outcomes


�


191



Hanney S, Buxton M, Green C, Coulson D and Raftery J An assessment of the impact of NHS Health Technology Assessment Programme NHS Health Technology Assessment Programme http://www.ncchta.org/ fullmono/mon1153.pdf 2007, Vol 11, number 53, pp UK


Health


Empirical


How should the impact of the NHS HTA programme be measured? What models are available and what are their strengths and weaknesses? What has been the impact of the first 10 years of the HTA programme (1993-2003)? What factors seem to be associated with the research making an impact?

Study design

A literature review and a multi-method evaluation (documentary analysis, survey of lead researchers and 16 detailed case studies) of the NHS HTA programme. The case studies were selected using stratified random sampling.

Theory

HERG payback framework

Analysis

Each case study was written up using the HERG payback framework

Findings/key points

The literature review identified 200 papers and looked closely at 46 of them (5 conceptual or methodological approaches, 23 empirical evaluations of impact and 18 papers that were both conceptual and empirical). The methods identified were: documentary analysis,


192

questionnaires, interviews and detailed case studies. The conceptual approaches included the HERG payback model, the ESRC framework for assessing non-academic impact of socio-economic research and the model developed by Lavis and colleagues to look at the impact of health research on policy. The payback model was considered to be the most widely applied and appropriate model – particularly as it seemed to draw on key elements in alternative approaches. The framework proved particularly useful in the collection and organisation of material. The blend of quantitative and qualitative data collection methods seemed generated valuable insights. This study developed methods for scoring the impact of a research project, based firstly on database information and questionnaires and then compared with the detailed case studies. The stratified random sampling approach to case study selection seemed to work well. The case studies were scored for impact before and after interview to assess whether there was an information gain from conducting interviews. The authors conclude that there is a need for more payback case studies based on selected projects within research programmes, for further refinement of the payback framework and that it would also be useful to collate health impact studies over time (to help capture impacts over time) and to conduct them in a consistent way to facilitate comparison.

Paper includes An evaluation of the impact of research on policy A reflective piece about evaluation of the impact of research on policy A description of method(s) for evaluating impact of research on policy Evaluation of effectiveness/value for money of methods

� �


193

A review of methods for evaluating impact more generally Others of relevance (give reason)

�


194



Healy R G and Ascher W Knowledge in the policy process: incorporating new environmental information in natural resources policy making Policy Sciences 1995 28(1) pp1-19 USA Duke University


Environment


Conceptual


Explores the utilisation of information in natural resources policy making, and how new sources of information might advance policy. Explores the impact of research on the process of policy making.

Study design

Theory

Analysis

Findings/key points

The report describes the historical context of policy, prior use of valuation studies and influences from other sources of information. Information is increasingly used in the decision making process prior to actual decision making. Valuation (costing) of different policy options enables information trade-offs. Increasing the use of information increases the demand for technical analytical routines to utilise the information. The authors also observe difficulties in accounting for preferences and interests. Although new information may change policy outcomes, there is little reason to believe that it makes the decision making process either shorter or smoother.

Paper includes


195


�


196



Herrick C and Sarewitz D Ex post evaluation: a more effective role for scientific assessments in environmental policy Science, Technology & Human Values 2000 25(3) pp309-331 USA PERI Environmental Associates and Center for Science, Policy and Outcomes


Environment


Empirical (conceptual discussion).


How are predictive scientific assessments limited in their impact on policy?

Study design

Three case studies (although no further methodology is detailed) of acid rain, global climate change and air quality standards.

Theory

Analysis

Findings

The report finds that the impact of research on policy is greatest when research findings generate widespread public support, which in turn stimulates a policy response, e.g. climate change. The authors recommend that scientific assessments be used less as a guide for new policy, and more as a tool to support, monitor and assess existing policy.

Paper includes; An evaluation of the impact of research on policy A reflective piece about evaluation of the impact of research on policy A description of method(s) for evaluating impact of research on policy

�


197



198



Hilderbrand M and Simon J The role of research in child health Policy and programs in Pakistan In: Lessons in research to action and policy: case studies from seven countries. Available via: http://www.cohred.org/main/. 2000 Switzerland Council on Health Research and Development (COHRED)


Health


Empirical


Ex-post evaluations of the utilisation of research in child health policy.

Study design

Open-ended interviews (16 decision makers, 22 researchers across types of research: clinical, community health and demographic), study review and documentary analysis. Focuses on three programmes.

Theory

Analysis

Considered four major sets of factors thought to be important in shaping the role of research: the environment; availability of relevant research; demand for research by decision-makers; and linkage/dissemination strategies.

Findings/key points

Individual programmes self-report to have close research-policy links, but the general perception from this report is of poor usage overall (most interviewees could cite instances whereby research had affected a policy or programme, but frequently the same examples). The authors found limiting factors to be the environment (not conducive to utilisation); poor availability of relevant high quality research; weak research


199

‘communities’; demand-side constraints (policy makers were limited by a demand for quick results); and poor linkages. The report also highlights the importance of research users and producers collaborating from the outset to stimulate demand and usage.


�


200



Referenced


Her Majesty’s Treasury et al Cross-cutting review of science and research: final report HM Treasury http://www.hm-treasury.gov.uk/media/E/7/science_crosscutter.pdf. 2002 UK HM Treasury


Science


Descriptive


Reviews the funding mechanisms and use of scientific research in government.

Study design

Theory

Analysis

The report is divided into three parts: university science research funding,; knowledge transfer; and science in government departments

Findings

The review recommends that individual departments be held fully accountable for value for money and impact/use of research. It also recommends a review of funding mechanisms (current practice does not encourage cross-cutting research, also short-term culture contradicts scientific goals) and increased funding through the research councils. The Chief Scientific Adviser has proposed a rolling review of knowledge utilisation, to include vertical and horizontal evaluation of research use (p98).



201


� Useful overview of the utilisation of scientific research in government


202



Holmes J The use of science in policy and regulation: baseline review Environmental Research Funders Forum http://www.erff.org.uk/reports/events.asp. 2005 UK Oxford University


Environment


Descriptive


A review of the ERFF work package to improve the use of science in policy-making, implementation, decision making and legislation.

Study design

Informed by face-to-face and telephone discussions, documentary analysis.

Theory

Analysis

Findings/key points

This strategy document outlines a general recognition of the need to enhance environmental policy-research interaction (citing barriers to include differences in motivations, cultures, timeframes and reward structures) through collaboration/ interaction, communication, question framing and transparency (maintaining clear distinctions between values and ‘facts’, and the different roles of each in decision making). Some research communities tend to see excellence and usefulness as conflicting requirements (point 34). Roles and mechanisms for the definition of research programmes (including the need for better mechanisms to enhance policy impact) need to be developed).



203


� Useful considerations for evaluation of environmental research.


204



Holmes, J Study of dissemination and implementation of research in SKEP member organizations. Scientific Knowledge for Environmental Protection (SKEP) http://www.skep-era.net/site/82.asp 2007 UK


Environmental research


Empirical


Explore approaches to and experiences of research dissemination and implementation by SKEP member organizations.

Study design

95 interviews in 33 organizations and a review of the literature

Theory

Analysis

Findings/key points

Study explored planning and management, communication of results, interpreters and intermediaries, engagement with stakeholders and evaluation. In terms of evaluation of dissemination and implementation activities, most of the organizations did not have a formal system. They rely on informal feedback or ‘count things that can be readily measured’ (p20). Challenges identified included; the difficulty in tracing the uptake of research in policy making, the time lag to utilization and the fact that research often leads to conceptual use which is harder to capture than instrumental use. Two organizations had used more systematic


205

approaches to evaluation: Environmental projects funded by the Finnish Environment Ministry are scored against a set of evaluation criteria by the project leader and the Ministry supervisor. There is usually a good match, but where there is not the project leader is interviewed to explore the differences. The Netherlands Environment Ministry has carried out two surveys (using external evaluators) of all policy makers (they were required to respond) who had commissioned research in a particular year. The questionnaire included questions on how and the extent to which is had been used. The questionnaire was followed up with interviews.


�


206



Hooton N, Lee-Smith D, Nasinyama G and Romney D Learning lessons from the Kampala urban agriculture policy process International Livestock Research Institute and Overseas Development Institute, Pro-Poor Policy Change Project. Available via: http://www.pppppc.org/. 2006 UK ODI




Empirical


What actors, events and influences affected urban agricultural policy change in Kampala?

Study design

RAPID Outcome Assessment (ROA), including episode study, Outcome Mapping, and case study within a Context-Evidence Links framework.

Theory

Analysis

Findings/key points

This evaluation further developed the ROA methodology (from Leksmono et al 2006) to strengthen the episode study component (starting from a change in policy), and using a workshop to verify case study/episode study components. The evaluators used triangulation of interviews, documentary analysis and workshop to develop a timeline of activities that contributed to changed behaviour, a description of policy development, and a description of key actors and changes in their behaviour (p37 for visual map). The authors found that credibility and communication of research played a key role: p44 also


207

explores the role of partnerships/networks and notes the role of ‘champions’ of policy change (influential actors).


� �


208



Hovland I Making a difference: M&E of policy research Working Paper 281. Available via: http://www.odi.org.uk/publications/. 2007 UK Overseas Development Institute




Descriptive


How to monitor and evaluate the broader aims and impacts of policy research (from programmes or institutions).

Study design

Theory

Analysis

Identifies five performance areas: 1. Strategy and direction; 2. Management; 3. Outputs; 4. Uptake; and 5. Outcomes and impacts.

Findings/key points

The report distinguishes between instrumental and conceptual impacts. Page 14 highlights Davies (2005) modular matrix which maps desired contribution of research project, and a second matrix which maps the degree to which each output is reaching target audiences. The report also describes the use of impact logs, new areas for citation analysis (i.e. grey literature) and user surveys (pp26-7) to evaluate research uptake (outputs). In terms of evaluating outcomes and impacts, highlights forwards looking approaches: Outcome Mapping (Earl et al, 2001) (p27); Rapid Outcome Assessment (Leksmono et al, 2006) (p29) as a 13-step learning methodology (steps 1-11 in a workshop); and Most Significant Change (Davies and


209

Dart, 2005) (p32). Backwards looking approach are: Innovation Histories (Douthwaite and Ashby, 2005) (p33) (actor by actor matrix, case studies); and Episode Studies (RAPID, p34). The author suggests that best practice monitoring and evaluation incorporates methodologies/monitoring from the outset of a project, and an appropriate combination of internal/external assessment. The author also suggests that case study analysis (information obtained through document review, staff workshop/interviews, external stakeholders) is the most appropriate methodology.


�


210



International Food Policy Research Institute Impact evaluation: assessing the impact of policy-oriented social science research Summary report of a 2002 IFPRI symposium workshop IFPRI http://www.ifpri.org/pubs/ib/ib5.pdf. 2002 USA IFPRI


Science


Descriptive


What are the issues and considerations in evaluating the impact of social science research?

Study design Theory Analysis Findings/key points

The report addresses five key ‘themes’ of issues in impact evaluation. These are: 1. Scale, attribution and time horizon Evaluation of a project, programme, institution or body of knowledge. Moving up the scale, it becomes more difficult to attribute policy responses to individual actors/pieces of research. Similarly, in an era of partnerships, attribution is challenging. The focus on project-level evaluation implies that short term impacts receive more attention than longer term impacts, which are a feature of strategic policy research. 2. Supply-side vs demand-side Ideally, impact assessment would start from a policy decision and work backwards, but because of the need for attribution, most studies start from the research project and track forward. 3. Importance of surprise


211

Unanticipated research results have a greater impact, but this does not equate to confirmatory research having no impact. 4. Choice of indicators Considers the impact of policy (i.e. changes in economic welfare), suggested methods include bibliometrics, although observes that where research is confirmatory, it is even harder to develop indicators. Interviews with policy makers and historical retrospective narrative are especially useful when starting with a demand-side approach. Pros/cons of case study, survey and peer review (costly) include objectivity/subjectivity and credibility. 5. Ex ante and ex post, and time lags All useful considerations. The report also summarises mechanisms to enhance research impact (communication, supply/demand side approaches, objectivity and quality). ‘Much remains [to be known] about evaluating the impact of policy-oriented social science research’ (p4). The report concludes that IFPRI should take the lead in developing a consortium to improve interdisciplinary methods of assessing impact.


�


212



Innvaer S, Vist G, Trommald M and Oxman A Health policy-makers’ perceptions of their use of evidence: a systematic review Journal of Health Services Research and Policy 2002 7(4) pp239-244 Norway National Institute of Public Health


Health


Empirical


A summary of evidence regarding policy makers’ reported use of evidence in policy decisions.

Study design

Interviews with policy makers, literature review (identified through database searches and hand searching of journals).

Theory

The ‘two communities’ thesis.

Analysis

Findings/key points

The most commonly reported facilitators of research utilisation are personal contact, timeliness and research summaries. However, personal communications might also promote selective/inappropriate use of research evidence. Weiss and Weiss (1996) argue that both decision makers and social scientists behave as though research genuinely contributes to policy. Common wisdom about improving the use of evidence is not hard to find, although empirical evidence is. The authors note that previous empirical studies were informed by face-to-face interviews, telephone interviews and postal questionnaires. ‘Given the diversity of contexts in which health policy decision making occurs, it is not possible to obtain a generalisable sample’ (p241). The


213

degree to which evidence is used directly (instrumental) or selectively (enlightenment) may vary in relation to the types of decision makers, the types of policy question, and issues of adoption/implementation, decision/action (p242). Enlightenment is more difficult to assess than direct use. ‘Future research should combine interviews with document analysis, focus on commissioned research and clearly define what is meant by ‘use’ of research.’ (p243).


� (evaluation of utilisation) �


214



Jensen F S and Koch N E Twenty-five years of forest recreation research in Denmark and its influence on forest policy Scandinavian Journal of Forest Research 2007 19(4) pp93-102 Denmark Danish Centre for Forest Landscape and Planning


Environment


Empirical

Research question/Aim

Correlates information about forest user preferences with changes in forestry policies and tries to elucidate linkages.

Study design

Survey, postal questionnaire

Theory

Analysis

Findings

The methodology to assess the impact of forestry research on environmental policy compared survey data of forest users across three decades, and linked trends with introduction/amendments to new policies. However, whilst the survey of users raises many policy questions in itself, there is no explanation of presumed linkages between research policy in the past. The authors conclude that past basic research has informed policy discussions and the aim of policy through an enlightenment role.


�


215



216



Johnston R Research impact quantification Scientometrics 1995 34(3) pp415-426 Australia University of Sydney




Descriptive


A review of methods and management of R&D evaluation.

Study design

Theory

Analysis

Findings

The paper discusses the following methodologies in relation to R& D evaluation (in general): citation analysis; patent-based measures; cost-benefit analysis (economic impacts, and impacts on company performance); and social rates of return. The authors note a general trend away from linear research impact assumptions, and instead observe the use of research/knowledge at every stage from identifying further knowledge needs to addressing specific policy questions. The paper recommends an interactive research user/producer relationship, and highlights the essential importance of visibility, transparency and accountability.



217


�


218



Jones S A, Fischhoff B and Lach, D Evaluating the science-policy interface for climate change research Climatic Change 1999 43(3) pp581-599 USA Rose-Hulman Institute of Technology, Carnegie Mellon University, Oregon State University


Environment


Empirical and descriptive.


Proposes a methodology to assess the ability of research results to affect decision making (not just in policy but also management) and uses a case study of salmon farming.

Study design

Interviews with policy makers and scientists, formal analyses of the links between research and the decisions described in interviews. Ex-ante case study (but method can be applied retrospectively).

Theory

Methods drawn from organisational theory, impact assessment and decision theory.

Analysis

Findings

Model proposes two to four questions within the descriptive categories of relevance, compatibility, accessibility, receptivity. Evaluating the utility of research requires considering its technical content within a specific institutional context. There is ‘No substitute for interviewing individuals within the policy arena’. The authors present the findings in a decision model (p592) to identify barriers to the effective creation and utilisation of research within a policy arena. However, the method does not


219

evaluate the impact of this research on policy.


�


220



Kalpazidou-Schmidt E RTD evaluation and policy in the European Research Area Evidence & Policy 2006 2(2) pp185-209 Denmark University of Aarhus




Conceptual and empirical.


Exploration of Research and Technology Development (RTD) evaluation and its use in research and technology policy.

Study design

Comparative study of 22 European countries conducted in 2002 using a questionnaire answered by agents of RTD activities and policy making. The survey included questions on the evaluation landscape, and the concepts and methods used in different countries.

Theory

Analysis

Findings

The survey of RTD evaluations at a national level found that peer review was the most common method used, followed by survey descriptions and bibliometric analyses. Finland is an exception, where users’ evaluation and capacity evaluation are used. Other methods included review of strategies and management, impact evaluations, societal impact analyses, and comparison with similar programmes in other countries. These different methods were, according to the interviewees, developing rapidly. The author stresses that ‘It is important to draw attention to the necessity of using multiple methods and achieving a balance of quantitative and qualitative methodology…especially


221

when providing information for policy making in relation to large investments’ (p198). Respondents felt that there was a need for improvement in methods used at a macro level, methods used to assess societal impacts, and relevance analyses.


� Although it’s an account of methods for evaluating the impact of evaluations of R&D on R&D policy and practice.


222



Kilpatrick H E Some useful methods for measuring the benefits of social science research Impact Assessment Discussion Paper 5 http://www.ifpri.org/impact/iadp05.pdf. 1998 USA International Food Policy Research Institute


Science


Descriptive


A description of methods for evaluating the wider benefits of social science research.

Study design

Literature review

Theory

Analysis

Discusses pros/cons of bibliometrics, peer review measures, case studies, user evaluations, cost-benefit analyses, social rates of return (production function modelling), operations research modelling, and simulation (operations research and computational economics).

Findings

The author finds that no single method completely meets theoretical needs, and the use of mixed methods (triangulation) may produce better quality evaluation findings. The author advises that replication of original research be conducted as part of the evaluation process (to ensure quality and reliability). Surveys are more costly than focus groups, so the author suggests that they are best reserved for evaluation of programmes (p7).



223


� � �


224



Kivimaa, P, Mela, H, and Furman, E Approaches and practices in mid-term and ex-post evaluation of environmental research programmes Scientific Knowledge for Environmental Protection (SKEP) Work Package 3 http://www.skep-era.net/site/files/ WP3%20final_eval.v1.2.pdf 28th June 2007 Finland Finnish Environmental Institute (SYKE) funded by EU Sixth Framework Programme


Environmental Research


Descriptive and empirical


The work package concentrates on programme evaluation and draws together good practice in the management and evaluation of environment-related research funding programmes.

Study design

Literature review, documentary analysis of documents from 18 research funding organizations in ten EU member countries, two deeper case studies of evaluation activities, an expert workshop and further contacts with experts by e-mail and telephone

Theory

Analysis

Findings/key points

The evaluations identified were most commonly programme evaluations, (focusing on a single funding programme –either mid term or ex-post), with others using either a thematic, or a research field based or a knowledge systems approach (the latter looking at, for example human capital and networks). They focused on


225

processes, effectiveness, cost effectiveness, immediate, intermediate side effects and final outcomes. An evaluation may include a combination of different focuses. In terms of cost effectiveness, they comment that the costs of research programmes are straightforward to calculate but the results and outcomes are much more difficult to judge in terms of monetary value. Evaluations are categorised in two ways in the paper: on the basis of the evaluator or the techniques used to do the evaluation. List of techniques includes those seen elsewhere including: bibliometrics, surveys, cost benefit analysis, case studies, historical tracing, benchmarking. At a workshop on previous experience of evaluating EU programmes, participants stressed the importance of the composition of evaluation panels (ensuring that they are representative of different countries and organizations) and suggested that modified peer panels combining scientific and other experts would be useful when policy level outcomes need to be evaluated. The authors argue that environmental policy requires inter- and transdisciplinary, environmental policy involves a wide range of stakeholders


�


226



Kostoff R N Assessing research impact: US government retrospective and quantitative approaches Research Evaluation 1994 21(1) pp13-22 USA Office of Naval Research




Descriptive


Assesses the strengths and weaknesses of semi-quantitative impact assessment methods.

Study design

Theory

Analysis

Findings/key points

‘The greater the variety of measures used to evaluate research impact, the greater is the likelihood of converging to an accurate understanding of the knowledge produced by research’ (Irvine and Martin, 1984). Semi-quantitative methods are widely used by research-sponsoring organisations, but they are considered less credible than quantitative methods by the academic community. Project Hindsight (1960s US Defense Department retrospective study to identify key events and contributing factors that influenced outcomes). TRACES (National Science Foundation – Technology in Retrospect and Critical Events in Science) backwards study from the point of publication/citation to identify contributory non-mission oriented factors for up to nine years previously. A key theme emerging from


227

the methods was analysis of the context of publication/citation (i.e. source, time/date). The author observes that no published retrospective methods account for the cumulative indirect and direct impacts of basic research (p15). ‘The degree of success or impact is more difficult to measure’ (p18). He also warns of a tendency to exaggerate benefits/ attribution when evaluating the contribution of a particular piece of research.


� � �


228



Kostoff R N Research requirements for research impact assessment Research Policy 1995 24(6) pp869-882 USA Office of Naval Research




Descriptive


Examines the pros and cons of different research impact assessment methodologies.

Study design

Theory

Analysis

Findings/key points

As policy-oriented assessment has not been part of the past research process, its implementation must be both gradual and flexible (Brown, 1992). The costs of research reviews are not negligible – total real costs dominate direct costs. The major cost of any review is staff time. Peer review is efficient resource allocation and is a credible predictor of research impact, although query over reliability. However, there is no link between peer review and downstream impacts. Quantitative approaches are relative inexpensive but because of multiple interpretations, these should be coupled with peer review. The author proposes a database which allows tracking of the evolution of products of research and would facilitate peer review, qualitative and quantitative predictive models (p879).

Paper includes


229


� �


230



Kostoff R N Federal research impact assessment: axioms, approaches, applications Scientometrics 1995 34(2) pp163-206 USA Office of Naval Research




Conceptual


A description of qualitative, quantitative and retrospective methods of research evaluation.

Study design

Theory

Analysis

Findings/key points

The methodologies described refer to assessing the impact of research on society, but many of the lessons are relevant. The author recommends that assessments weight different impacts according to what is most important to an organisation (p169). ‘Because of cost and subjectivity, other methods to complement or replace peer review...are actively being pursued’ (p170). The report also recommends that R&D evaluation be included as an integral component of research, not a subsequent add-on. The report examines the pros and cons of peer review, bibliometrics and econometric analysis of the impact of research. It also highlights network approaches as ‘invaluable’ in tracing the indirect impacts of research (p199).



231


� �


232



Kuhlmann S German government department’s experience of RT&D programme evaluation and methodology Scientometrics 1995 34(3) pp461-471 Germany Fraunhofer Institute for Systems and Innovation Research




Conceptual


Assesses current practice and recent trends in German government RTD evaluation.

Study design

Theory

Analysis

Findings

Parameters for evaluation are: • Planning (how, when, why, how

often?) • Selection of evaluators (who?) • Content, scope, depth and extent of

analysis • What methods and indicators? • Presentation and use of results?

Publication? Policy recommendations? Evaluations can facilitate but not replace responsible decision making. Pages 465-466 also discusses horizontal evaluation (used increasingly), and uses case studies to illustrate specific contexts of evaluation. The report notes that user-oriented evaluation reduces potential conflicts.



233


�


234



Referenced


Kuruvilla S, Mays N, Pleasant A and Walt G Describing the impact of health research: a Research Impact Framework BMC Health Services Research http://www.biomedcentral.com/content/pdf/1472-6963-6-134.pdf. 2006 6(134) UK London School of Hygiene and Tropical Medicine


Health


Descriptive


Proposes a framework to evaluate the impact of health research (in general).

Study design

Semi-structured interviews and literature review.

Theory

Draws on: the HERG Payback Model (Buxton and Hanney, 1996); a framework to analyse Health Research Systems (Pang et al, 2003); a ‘knowledge transfer’ approach (Lavis et al, 2003); a model of the path from evidence to clinical application (Hayes, 1999); and economic approaches (McClellan and Heidenich, 1999).

Analysis

Findings

The Research Impact Framework (RIF) has four descriptive categories: research-related impacts; policy impacts; service impacts; and societal impacts. When describing impacts within these categories, important considerations include accountability, whether the impact is positive or negative (and for whom), and what individual/institutional biases may operate. The RIF assesses policy impact by: 1. Level of policy-making; 2. Type of policy; 3. Nature of policy impact; 4. Policy networks; and 5. Political capital (pp9-10). It is not evaluative in itself and does not prioritise impacts or propose


235

causal pathways. However, the categories enable individual researchers to identify and describe relevant impacts without specialised training.


�


236



Kuruvilla S, Mays N and Walt G Describing the impact of health services and policy research Journal of Health Services Research and Policy 2007, Vol 12 (Suppl 1) 23-31 UK London School of Hygiene and Tropical Medicine


Health


Empirical


Develop and application of a methodological approach to identifying and describing research impact

Study design

Primary analysis of seven research projects and secondary analysis of four research projects (drawing on reports that included impact analysis). Projects were purposively selected for variety. Principle investigators were interviewed by a researcher using a semi-structured guide organized around the RIF categories (see 84). Additional information was collected from project documents including reports, published papers and correspondence.

Theory

Application of the Research Impact Framework (RIF)

Analysis

Focused on analysis of the RIF in terms of its validity, feasibility and acceptability. Each individual project was treated as a case. Data from across the cases was analysed thematically.

Findings/key points

The approach generated verifiable and comparable narratives of a wide range of health service and policy impacts. The RIF provided a structured ‘methodological’ approach to organising


237

accounts of research impact. The RIFs seemed to provide similar information to that generated by more intensive approaches, although the ‘scope and level of analysis may be different.’(p30) Using a standardised approach across cases facilitated comparison. The authors argue that the RIF has potential as a practical tool to encourage researchers to think about the wider range of potential impacts of their work in a systematic way.


�


238



Lavis J N, Ross S E, Hurley J E et al Examining the role of health services research in public policymaking Milbank Quarterly 2002 80(1) pp125-154 Canada McMaster University


Health


Empirical


What are the conceptual, methodological and practical issues that limit the use of health services research in developing public policy?

Study design

Eight case studies of policy (backwards looking), telephone interviews (to gauge scope of project and identify key figures), interviews with involved policy makers and researchers, documentary analysis.

Theory

Analysis

Interview material coded by whether research was used for agenda setting/policy formulation, and whether research impacted on all policy or only partially.

Findings/key points

‘Discrete policy decisions may not cover the many roles of research in policymaking’ (p130), ‘Instrumental use is acting on research in specific and direct ways, such as solving a particular problem at hand. Conceptual use refers to a more general and indirect form of enlightenment. Symbolic use is using research to justify a position or action that has already been taken for other reasons’ (p130). The study recommends that policy typology (Lowi, 1964) be taken into consideration when examining impact on policy. More than half of the case studies are embedded in broader


239

policy trajectories (i.e. if starting with policy, difficult to pinpoint starting point). The report proposes three methods to identify explicit use of research (p133): asking policy makers directly involved in process; asking researchers whose research may have been used; reviewing documents that may have been used. However, it is often difficult to access all data (internal documents). This report used telephone interviews to ensure that appropriate stakeholders would be interviewed. Even where research was not used (five of eight policies), the policies were still particularly well informed. The evaluation also found that where citable research was a major influence in policy making, policy makers were in direct contact with researchers (p140).


� �


240



Lavis J N, Ross S E, McLeod C and Gildiner A Measuring the impact of health research Journal of Health Services Research and Policy 2003 8(3) pp165-170 Canada McMaster University


Health


Conceptual


Proposes measures to assess the impact of health research.

Study design

Theory

Analysis

Findings/key points

Impact measures should be based on an evolving understanding of knowledge transfer, utilisation and cultural shifts. The report finds that measuring the impact on decision making is the most appropriate level of impact assessment (i.e. not changes to health status). It is important to understand who is the target audience, what kind of decision (regulatory, distributory) is to be taken, and what the context is (competing interests etc). The authors suggest documentary analysis, surveys and semi-structured interviews to obtain information, and recommend the use of case studies where the focus is on demonstrating long-term impacts. Page 167 lists impact measures according to whether the active role is played by researcher, decision maker or both, and proposes methodologies for each. It is important to move beyond whether


241

research is used to how, and beyond process measures to outcomes.


� �


242



Leksmono C, Young J, Hooton N, Muriuki H and Romney D Informal traders lock horns with the formal milk industry: the role of research in pro-poor dairy policy shift in Kenya Working Paper 266 http://www.odi.org.uk/publications/. 2006 UK Overseas Development Institute




Empirical


Case study analysis of R&D project (Smallholder Dairy Project: 3 phases, 1997-1999, 1999-2000, 2000-2005) which aimed to influence policy makers.

Study design

Case study analysis using RAPID Outcome Assessment (ROA, including episode studies of specific policy changes, case study analysis of specific research projects, and outcome mapping approaches). Information was collected through literature review and documentary analysis; workshops with key staff, partners and actors; stakeholder interviews to triangulate workshop outputs; field visits; and a debriefing session to discuss initial findings.

Theory

Analysis

The information was presented in five overlapping categories: description of key actors and key organisations; general description of evolution of sub-sector over last century; description of evolution of policies affecting sub-sector; description of what the project did and why; and description of changes of behaviour of key actors which contributed to policy change, and reasons


243

for these. Findings/key points

The ROA uses a combined methodological approach, as different methods tend to over-emphasise particular elements (i.e. case studies over-emphasise research, episode studies over-emphasise political influences). ‘An enormous range of different factors influence most policy processes, waxing and waning and in different combinations over time.’ (p3). The original strategy to increase policy influence was mapped on a log frame (p22), although this did not appear to structure the impact analysis. Although changes in national policy are still in legislative process, changes in the attitudes and behaviour of key actors can be observed at national and local levels. The evaluation identifies changes in written and implemented policy, and tracks change over time. It highlights the importance of communication, collaboration and influential individuals, and timing. The authors conclude that ROA is an effective methodology to map cause and effect of key events, and the contribution of key actors. It goes some way towards mapping attribution (p50).


� � �


244



Internet search


Lewis J M A network approach for researching partnerships in health Australia and New Zealand Health Policy http://www.anzhealthpolicy.com/content/2/1/22. 2005 2(22) Australia University of Melbourne


Health


Empirical


Explores theories and methods for examining partnerships in health. An attempt to evaluate how new governance could improve outcomes through agency coordination.

Study design

Uses surveys and interviews. Telephone interviews used to generate names of key stakeholders.

Theory

Structure and agency (Giddens, Jessop) (p2).

Analysis

Combines network mapping and analysis with narratives.

Findings/key points

The author argues that network analysis can reflect the reality of policy making processes and complex interactions (partnerships, alliances, collaborations, networks). Social networks are analysed as a set of connections, with descriptive narratives about those connections (use and value). Key actors were questioned about their actions/links within the previous six months. The visual ‘map’ of networks can also demonstrate links for multiple purposes (i.e. work, policy advice, strategic information). Network analysis can be used to map the progress of networks/ relationships within and between networks. Mapping ties can highlight weaknesses and strengths. The author reflects that network analysis can indicate linkages/changes in linkages, but cannot indicate whether outcomes


245

improve as a consequence. Paper includes An evaluation of the impact of research on policy A reflective piece about evaluation of the impact of research on policy A description of method(s) for evaluating impact of research on policy Evaluation of effectiveness/value for money of methods A review of methods for evaluating impact more generally Others of relevance (give reason)

� � An empirical study which reflects on network analysis as a methodology.


246



Lindquist E A Discerning policy influence: framework for a strategic evaluation of IDRC-supported research University of Victoria, School of Public Administration http://www.idrc.ca/uploads/user-S/10359907080discerning_policy.pdf 2001 Canada University of Victoria




Conceptual


To develop a conceptual framework to guide evaluation of IDRC-sponsored projects (i.e. in the South).

Study design

Theory

Knowledge utilisation e.g. Weiss (1980); intermediary institutions (e.g. Heclo, 1978); advocacy coalition framework (Sabatier, 1998); policy transfer; policy windows (Kingdon, 1995).

Analysis

Findings/key points

It is difficult to assess the role and impact of research on public policy because of the intrinsic nature of research, and because the goal is to achieve influence in dynamic processes with a multiplicity of actors (p1). The report suggests that assessing policy influence is typically about assessing intermediate influences such as expanding policy capacities, broadening policy horizons and affecting policy regimes (p23). It distinguishes between three types of information: research; data; and analysis. The three stage framework is: 1. Description of policy problem; 2. Description of


247

intention and scope of project; amd 3. Description of project cycle, key outputs and events, and policy influence. The author reflects that this framework: establishes the contributions and intentions of research before attempting to assess policy influence: accounts for the policy making context and multiplicity of actors; identifies a sufficiently long time-frame; and recognises the values and struggle over ideas and policy matters in the commissioning, interpretation and utilisation of research (p5).


� �


248



Link A N Economic performance measures for evaluating government-sponsored research Scientometrics 1996 36(3) pp325-342 USA University of North Carolina




Conceptual


A critique of the usefulness of economic methodologies to evaluate government-sponsored research.

Study design

Theory

Analysis

Findings/key points

Selection of an evaluation method should only take place after the research and its objectives are fully understood (p329). Evaluation is primarily for the purpose of accountability (hence Government Performance Results Act). Evaluation is part art (tailoring techniques and methods to the problem and data available), part science. The author notes the role of judgement in applying different methods and research designs that are only partly standardised. Evaluation itself is classified as research and needs to be quality controlled and replicable. The author reflects that single criterion evaluations (p331) are more cost-effective than multiple criteria evaluations, but the latter allow for trade-offs. The time lag for assessing the impacts of basic research is much greater than for applied research.


249


�


250



Luukkonen T The difficulties in assessing the impact of EU Framework Programmes Research Policy 1998 27(6) pp599-610 Finland VTT Group for Technology Studies




Conceptual


A summary of the Framework evaluation findings, and an analysis of elements that make impact assessments difficult.

Study design

Theory

Analysis

Findings/key points

The EU Framework evaluations have had difficulties in assessing competitiveness, which is a major objective of the research programmes. Reasons include the nature of the objectives, methodological difficulty in measuring attainment (of objectives), attribution and additionality (especially at project level), and quantifying results.

Paper includes; An evaluation of the impact of research on policy A reflective piece about evaluation of the impact of research on policy A description of method(s) for evaluating impact of research on policy Evaluation of effectiveness/value for money of methods

�


251



252



Luukonen-Gronow T Scientific research evaluation: a review of methods and various contexts of their application R&D Evaluation 1987 17(3) pp207-221 Finland Academy of Finland




Conceptual


Pros and cons of peer review, interview and questionnaire, quantitative methods and case studies.

Study design

Theory

Analysis

Findings/key points

Traditional evaluation methods are reliant on peer review. The author then groups evaluation methods for discussion: 1. peer review and variants; 2. interview and questionnaire method; 3. quantitative methods; and 4. case studies, case histories. The author notes issues in evaluation to be timing (ex-ante, ongoing or ex-post) and bias (can include non-experts in peer review to counter this, for example). Internal criteria of assessment can include quality of research and the contribution to advancing knowledge. External criteria (circumstantial) include economic growth and policy impact.



253


�


254



Luukkonen-Gronow T The impact of evaluation data on policy determination In: The evaluation of scientific research [conference 6-8 June 1988] CIBA Foundation 1989 Finland The Academy of Finland




Conceptual


What are the uses and impacts of evaluations of scientific research?

Study design

Theory

Instrumental, conceptual and persuasive use (Leviton and Hughes, 1981).

Analysis

Findings/key points

Utilisation and impact have nearly identical meanings: both relate to research influence on policy, but utilisation is the process whereas impact is the outcome. The author proposes bibliometrics to evaluate instrumental use of research and peer review for conceptual use. However, anecdote is the only methodology offered to assess persuasive use. It is difficult to prove that a decision is taken directly because of a piece of research as ‘most impacts are piecemeal’ (p238). Utilisation depends upon both credibility and communication. Evaluations must stress the importance of quality and productivity.

Paper includes An evaluation of the impact of research on policy A reflective piece about evaluation of the


255

impact of research on policy A description of method(s) for evaluating impact of research on policy Evaluation of effectiveness/value for money of methods A review of methods for evaluating impact more generally Others of relevance (give reason)

�


256



Mackinnon J and Reinikka R How research can assist policy: the case of economic reforms in Uganda World Bank Research Observer 2002 17(2) pp267-292 USA World Bank




Empirical


Explores the roles and influences of research in different economic reforms.

Study design

Case study of macroeconomic policy (note that most research considered was officially sponsored and donor-financed). No details given.

Theory

Analysis

Findings/key points

The evaluation found that research brought issues to the attention of policy-makers, but often after decisions had been made. Research determined the impact of explicit/implicit tariff price changes. The authors reflect that survey data enable trends to be identified. The report highlights the significance of research findings (cost-benefits between different policy options). However, the value of many research projects was open to interpretation. The authors imply that it is only given a set of assumptions about what is feasible and desirable that a research finding has policy implications. They conclude that research has had a powerful impact on policy and specifically on the climate of opinion. Research usually has the greatest impact on policy when it is embedded in a social process.


257


�


258



Marburger J Wanted: better benchmarks Science 2005 308(5725) p1087 USA Office of Science and Technology Policy, Executive Office of the President of the United States


Science


Conceptual


Comments on the emerging ‘science of science policy’ which the author hopes will improve understanding and measurement of the effectiveness of science policy.

Study design

Theory

Analysis

Findings

There is a need to develop models of the research-policy process that can capture the complex impacts of interdisciplinary, innovatory research. Marburger’s editorial cites the finding of the National Research Council Committee on National Statistics that: ‘the structure of…data collection is tied to models of R&D performance that are increasingly unrepresentative of the whole of the R&D enterprise.’


�


259



260



McNie E C Reconciling the supply of scientific information with user demands: an analysis of the problem and review of the literature Environmental Science and Policy 2007 10(1) pp17-38 USA University of Boulder, Colorado


Environment


Descriptive


How can scientific research be more useful to environmental decision makers?

Study design

Theory

Analysis

Findings/key points

This paper identifies an imbalance between the funding of basic scientific research and research which improves decision making by expanding alternatives, clarifying choice and enabling decision makers to achieve desired outcomes. Information must be credible, legitimate and salient. The report discusses ‘boundary management’ and cultural/scientific issues. Adaptive management is a learning technique, based on the notion that policies are experiments and outcomes are opportunities for learning. Adaptive management (Lee, 1999) compares outcomes of policy decisions against predictions. National Research Ccouncil (2005, p30) internal criteria are necessary for evaluating policy decision making, rather than solely relying on the outcomes of those decisions. Shaw (1997, p5)


261

suggests criteria for evaluation of research impact to include consistency of the decision against available information, whether all relevant information is considered, whether the science is understood and well interpreted and whether risks are adequately addressed. The report proposes ‘ex-post satisficing’ as a user-oriented methodology to evaluate how useful scientific information was (Deelstra et al, 2003).


� � (section 4.4)


262


263



Meagher L, Lyle C and Nutley S Flows of knowledge, expertise and influence: a method for assessing policy and practice impacts from social science research Submitted to Research Evaluation 21 January 2008 UK Technology Development Group and the University of Edinburgh




Empirical


Explore methodological issues raised by a study commissioned by the ESRC to investigate new approaches to assessing the policy and practice impacts of psychology research

Study design

Used a multi-method approach comprising case studies, surveys, focus groups, document analysis (including end of award reports) and semi-structured interviews. Bibliometric methods were used for three of the five case studies. In addition to bibliometric databases, Google was searched for grey literature. Looked at 134 responsive mode awards made by ESRC over three cohorts (3 year grants ending in 1998, 2001 and 2004).

Theory

The study was informed by the ‘linkage and exchange’ conceptual framework developed by the Canadian Health Services Research Foundation. This framework was used to develop a framework of core questions.

Analysis

The conceptual framework was used to structure the analysis

Findings/key points

Multiple methods very valuable for triangulating findings. Conceptual


264

framework facilitated the synthesis of findings across the methods. The e-mail survey (53% response rate) and end of award analysis also seemed to work well and generated useful data, with the interviews providing a more nuanced response and the case studies exploring the nature of impacts and good practice. The bibliometric databases generated very little material that didn’t come up through the Google search. However, the search worked better with researchers with unique names (and were more limited with researchers with common names). The authors suggest that the timing of the study did not allow for full impact (but steps in the direction of impact at most). Also the variable way in which impact was addressed in end of award reports made analysis difficult.


� � �


265



Internet search


Merkx F, van der Weijden I, Oostveen A-M, van der Besselaar P and Spaapen J Evaluation of research in context: quick scan of an emerging field Rathenau Instituut and EriC (Evaluating Research in Context) www.rathenau.nl/showpageBreed.asp?steID=1&ID=2963. June 2007 The Netherlands Rathenau Instituut and ERiC




Descriptive


An overview of research evaluation literature.

Study design

Theory

Analysis

Findings/key points

Research evaluation is context dependent (i.e. it should reflect specific goals/activities) which poses specific questions about appropriate criteria, performance indicators and evaluation procedures. Societal impact is defined as the implementation of research output by professionals (in the policy field). Issues in designing evaluation include whether evaluation is to be formative or summative, ex ante or ex post, stakeholder-oriented. This review finds that a key benefit of the HERG payback model is that in addition to traditional measures of research output, it also highlights the wider benefits of research (including economic). In the social sciences and humanities, bibliometrics alone are an inappropriate indicator of research quality and impact (as only 30% of research is published in journals). Research councils in Australia, Canada, the UK and the Netherlands are in the


266

process of developing alternative evaluative methods (p26). The CHASS model for research quality and impact assessment incorporates quality (peer review), impact (panel review and user evaluations) and capability (the capacity to contribute to future goals of research – panel review). The AHRC Evaluation Framework evaluates the research landscape, and distinguishes between metrics of activity, impact assessment and assessment of quality/excellence.


� �


267



Michelson E S Approaches to research and development performance assessment in the United States: an analysis of recent evaluation trends. Science and Public Policy 2006 33(8) pp546-560 USA Woodrow Wilson International Center for Scholars




Empirical


What are the benefits and challenges of bibliometric analysis as an R&D evaluation technique? An account of US R&D evaluation methodologies.

Study design

Documentary analysis of journals, workshop proceedings, committee reports, agency websites (National Institutes of Health, National Science Foundation, National Institute of Standards and Technology, Advanced Technology Program), conference presentations and government documents.

Theory

Analysis

Broad overview of roles of different government agencies in evaluation. Specific analysis of methodologies of NSF, NIH, NIST Advanced Technology Program and Department of Energy Office of Science.

Findings

Increasingly, bibliometric analysis as an evaluation methodology is being combined with qualitative methods to create more appropriate hybrid methodologies. There is some fragmentation of methodologies used by different agencies/different levels of government (difficult to eliminate


268

macro/micro friction). Weingart (2005, p123) notes that bibliometrics is becoming so selective it can only represent part of the communication and research process, and evaluators may actually cause ‘perverse effects’ by over-relying on it. Concludes with a call for new sophisticated analytical techniques to better capture the range of qualitative and quantitative variables (p557).


�


269



Milner S J, Bailey C, Deans J and Pettigrew D Integrated impact assessment in the UK: use, efficacy and future development Environmental Impact Assessment Review 2005 25(1) pp47-61 UK Lancaster University


Health


Empirical


Map of Integrated Impact Assessment (IIA) activity in the UK.

Study design

350 organisations contacted by e-mail; from these 77 key interviewees were contacted (via telephone).

Theory

Analysis

Findings

Most IIA activity is at the strategic level of policy decision-making and implementation. The findings of this report suggest that there is a growing receptiveness to integrating different forms of assessment into a single assessment process. The authors suggest that assessments need to be evaluated in terms of their impact on inequalities. Core values of IIA are: the need for cross-sectoral understanding of impacts; the need to use IIA in a cyclical or iterative way to evaluate as plans develop rather than artificially distinguish between prospective, retrospective and concurrent; and the need for ongoing quality control in the assessment process. IIA toolkits need to be simple, user-friendly and transferable, as well as capable of inclusive ownership (i.e.


270

stakeholder involvement) Paper includes An evaluation of the impact of research on policy A reflective piece about evaluation of the impact of research on policy A description of method(s) for evaluating impact of research on policy Evaluation of effectiveness/value for money of methods A review of methods for evaluating impact more generally Others of relevance (give reason)

� Reflective of factors to consider in developing an evaluation framework.


271


ID reference number 100 Method of identification (e.g. database) Personal recommendation Details of publication Author Title Source (journal/ conference etc) Year/volume/pages Country of origin Institutional affiliation

Molas-Gallart J and Tang P Policy and practice impacts of ESRC funded research: Case study of the ESRC Centre for Business Research Report to the ESRC March 2007 UK INGENIO and SPRU, University of Sussex




Empirical


To identify policy and practice impacts of the ESRC-funded Centre for Business Research (CBR). The study also aimed to contribute to the development of evaluation methods.

Study design

Case study using documentary research and a programme of structured telephone interviews with researchers, potential users and beneficiaries.

Theory

Developed a conceptual framework distinguishing content (research outputs, skills etc gained by staff), processes (by which content achieved non academic impact) and context (within which the processes took place).

Analysis

Findings

The authors stress the value of having a conceptual framework and believe that their conceptual framework is applicable to other ESRC investments. A non-linear approach is needed to capture complex interactions. Contextual elements need to be integrated into analytical frameworks used to describe the interactions between research and policy. However, this is likely to require new research designs. At what level should evaluation be


272

conducted – projects, programmes or policies? Researchers rarely conceptualise their work on a project basis. The report discusses what counts as ‘impact’. Suggests that in this area (business research) influence is likely to be indirect, i.e. a contribution to discussions in policy circles that input into complex policy formulation processes. The report suggests that this sort of impact can be defined as the use of research as a source of evidence in policy discussions.


� � �


273



Molas-Gallart J, Tang P and Morrow S Assessing the non-academic impact of grant-funded socio-economic research: results from a pilot study Research Evaluation 2000 9(3) pp171-182 UK Univesity of Sussex, Science Policy Research Unit


Health


Conceptual, empirical

Research question

What are the difficulties in assessing the non-academic impact of grant-funded research?

Study design

Surveys and interviews with researchers and potential beneficiaries (two rounds).

Theory

Analysis

Pilot studies of potential assessment techniques. One case study of tracing post-research non-academic impact.

Findings

The outputs of R&D are not distributed along a normal distribution curve and thus the representativeness of sampling techniques is questionable (p172). Additionality can be distinguished if the methodology includes a technique to map the ‘baseline’ prior to the research project (p172). The report notes the importance of evaluating both indirect and direct impact (i.e. research as a problem-solving tool and/or influencing judgement). The authors suggest that development of a conceptual framework for impact assessment is underpinned by: 1. types of outputs; 2. channels through which diffusion takes place (i.e. networks); and 3. forms through which non-academic actors make use of research outputs (forms of impact). Mapping networks of


274

researchers and relevant users/ beneficiaries after end of projects can identify existing/potential applications of research results. It is important to involve both researchers and research users in any assessment of non-academic impact, and to adopt a qualitative approach. The results of an impact assessment are likely to differ dependent on the timing of the study. This study was 8-10 yrs after research activity – recommends that one to two years provides optimum trade-off between a long-term perspective and tracing individuals.


� � �


275



National Audit Office Getting the evidence: using research in policy making Stationery Office. Available via: http://www.nao.org.uk/publications/. 2003 HC 586-I UK NAO




Descriptive


How does government commission research, and how well is that research used to improve service delivery and develop policies?

Study design

Review of funding policies; comparative literature review; face-to-face interviews with ten civil servants and stakeholders; case studies.

Theory

Analysis

Findings

The recommendations of the strategy to improve the use of research focus on reviewed commissioning processes, clearly stated aims and objectives for using research, early involvement of research users and early dissemination. ‘Research impacts are often not quantifiable and it is difficult to attribute a policy impact to a particular research result’ (p21). Page 22 places interpretation of benchmark indicators within peer review. Page 23 (model of inputs, processes and outcomes) defines outcomes of procurement of research to be (short term) changes in policy, improvements in service delivery, increased awareness; and (long term) progress towards departmental objectives.


276


� Understanding of departmental approaches to the use of research.


277



National Audit Office An international review on governments’ research procurement strategies. A paper in support of Getting the evidence: Using research in policymaking Stationery Office. Available via: http://www.nao.org.uk/publications/. 2003 HC 586-II UK NAO




Descriptive


How is R&D commissioned, managed and used in Canada, Finland, Germany, the Netherlands and the United States?

Study design

Theory

Analysis

Findings

This report finds that although evaluation of research quality is widely practised, there is lack of models or practices whereby retrospective evaluation encompasses research relevance and value for money in linking research and policy. However, it is increasingly an ex-ante consideration. All five countries do have models of evaluation but these evaluate the returns to investment in R&D rather than its impact on policy (although the report recognises that this is a central element). There is an emphasis on accountability and learning. The authors recognise that research that is essential for long-term development can bring little short-term return to investment. The development of effective methods to evaluate and report


278

performance requires participation of the science community.


�


279



Referenced


NHS Service Delivery Organisation The impact of the NHS Service Delivery Organisation Research and Development Programme 2001-2006 NCCSDO http://www.sdo.lshtm.ac.uk/files/adhoc/impact-study-full.pdf. 2007 UK National Co-ordinating Centre for the NIHR Service Delivery and Organisation Programme (NCCSDO)


Health


Empirical


Assessment of the impact of the first five years of the SDO programme

Study design

A documentary and literature review, a bibliographic analysis of 23 projects representing all the programmes themes and detailed case studies of 11 of the projects.

Theory

Analysis

Findings

For this evaluation, analysis of primary outputs (report downloads, conference presentations, peer reviewed journal papers), secondary outputs (citations in policy and professional guidance and the mass media) and case studies (documentary analysis, telephone interviews with principal investigators and users of research) is guided by an impact analysis framework. Conclusions are presented in terms of impact on service delivery, policy, practice, research and capacity building (descriptive categories of the HERG payback model). Concern is raised about the timing of evaluation, given the time lag involved in achieving impact. Case studies provided in-


280

depth and focused examples, but challenges included dealing with time lags and handling a large volume of data, linearity and the use of retrospective methods that rely on the recall of stakeholders.


� �


281



Internet search


National Research Council Prospective evaluation of applied energy research and development at DOE (Phase Two) National Academies Press http://books.nap.edu/catalog.php?record_id=11806. 2007 USA National Research Council


Science


Empirical


What can outcome indicators of public expenditure be when measured against environmental and security benefits?

Study design

Applies the ‘benefits evaluation methodology’ to six R&D case studies. Expert panel evaluation. Models were outcome-specific and predicted costs and benefits using decision tree framework

Theory

Analysis

Findings

This report measures the benefits of research in terms of output, i.e. physical quantities of emissions compared with an estimated scenario without the research. Measures the policy benefits in terms of when/whether the benefits of a programme would be realised across domains and at different levels of policy (national/local). One drawback of using a review panel is potential bias which may affect the quality of evaluation. The report recommends that should certain skills be missing/over-represented, the panel be reformed accordingly.

Paper includes; An evaluation of the impact of research on policy A reflective piece about

� �


282

evaluation of the impact of research on policy A description of method(s) for evaluating impact of research on policy Evaluation of effectiveness/value for money of methods A review of methods for evaluating impact more generally Others of relevance (give reason)


283



Nederhof A Bibliometric monitoring of research performance in the social sciences and the humanities: a review Scientometrics 2005 66(1) pp81-100 The Netherlands Leiden University




Descriptive


How can extended bibliometric monitoring demonstrate research impact?

Study design

Theory

Analysis

Findings/key points

Early bibliometric analyses focus on citation impact of journals, but the trend towards more sophisticated analysis focuses on the impact of individual publications. Citation analyses do need to account for the intended audiences of publication. ‘A broader range of publications and indicators is needed in many social sciences and humanities’ (p92). For non-journal material, citation peaks tends to be relatively late (suggests a five year lag time) (p93). The author suggests three ways of extended bibliometric methods: 1. Using weighting (but this does not monitor impact); 2. Standard citation analysis; 3. Citation analysis including non-ISI publications. ‘Bibliometric monitoring studies are not perfect, and neither is peer review. Ideally, combining both most likely produces an optimal result’ (p96).

Paper includes


284


�


285


ID reference number 107 Method of identification (e.g. database) Internet Details of publication Author Title Source (journal/ conference etc) Year/volume/pages Country of origin Institutional affiliation

Norton G and Alwang J Policy for plenty: measuring the benefits of policy-oriented social science research Impact Assessment Discussion Paper 6 http://www.ifpri.org/impact/iadp06.pdf. 1998 USA Intenational Food Policy Research Institute


Environment


Conceptual, empirical


What methods are appropriate for ex-post and ex-ante evaluations of policy research programmes?

Study design

Two case studies (deforestation and pesticide policy).

Theory

Economic surplus, Bayesian decision theory, econometrics.

Analysis

Description of quantitative methodologies, and two case studies.

Findings

Proposes triangulation framework for quantitative analysis, combining data and formulae for economic surplus to calculate internal rates of return for specific policy research. The uncertainty of parameters of measurement requires carefully structured questions for interview. Impact evaluations need to account for policy change and additionality. Evaluations need also to allow for the political costs of making a policy decision.


� �


286



287



Internet search


Nutley S, Percy-Smith J and Solesbury W Models of research impact: a cross-sector review of literature and practice Learning and Skills Research Centre http://www.lsrc.ac.uk/publications/index.asp. 2000 UK Learning and Skills Research Centre




Conceptual


How can the impact of research produced by an organisation be increased?

Study design

Review of research literature and comparable practices in the education, social care, criminal justice and health care sectors.

Theory

Analysis

Identifies an approach for maximising research impact across four domains: interaction of research production and research use; research production; research use; and their organisational contexts.

Findings

This paper is reflective about issues in bridging the research-policy gap, and the process of research impact (specific practices to maximise impact). It notes that not all impacts are tangible. The authors also observe that the most effective impact analyses form part of research planning. There are no methodological reflections about how to quantify these impacts, although the approach to analysis offers some useful considerations.


�


288



289



Internet search


Office of Science and Innovation Science review of the Department for Environment, Food and Rural Affairs Office of Science and IUnnovation http://www.berr.gov.uk/files/file35751.pdf. 2006 UK Office of Science and Innovation


Environment


Descriptive


A review of the current use of science within Defra.

Study design

Informed by documentary analysis, external consultation, stakeholder interviews, case studies and peer review.

Theory

Analysis

Findings/key points

The purposes of the review are: learning; quality assurance; identification of areas of good practice within the UK and beyond, and areas for improvement. Defra has approximately 30 direct science advisory bodies, and is in contact with up to 100 delivery bodies. ‘Defra needs to ensure that it consistently identifies, engages with and makes best use of external scientific expertise and advice’ (pviii) including in-house and external expertise. Table 1 outlines the Defra science utilisation rationale: 1. To develop a clear, overall science strategy; 2. Horizon scan; 3. Review and harness existing science and identify gaps and opportunities for future research; 4. Commission and manage new science; 5. Ensure the quality and relevance of the science Defra sponsors and carries out; 6. Use science and


290

scientific advice; 7. Publish results and debate findings openly; 8. Share, transfer and manage knowledge; 9. Implement Guidelines 2005; and 10. Use, maintain and develop scientific expertise (including capacity and capacity building).


� This is a strategic document, but the mixed-method approach for contextualising perceptions and priorities is of particular interest.


291



Orosz, E The impact of social science research on health policy Social Science & Medicine 1994 39(9) pp1287-1293 Hungary Eötvös Loránd University


Health


Conceptual


What socio-economic and political circumstances influence the research-policy relationship? What are the main characteristics of research that influences policy? What impact can social sciences have on the context of health policies?

Study design

Theory

Analysis

Distinguishes between policy content, policy process and policy context.

Findings/key points

This report argues that it is not enough just to consider research and policy: it is also important to consider the media, different interest groups and the general public. Similarly, it is not enough to examine the impact on national policy: international (WHO) and local policies are equally important. The author identifies the following issues to consider when developing an evaluation strategy: timing; communication; methodological difficulties (i.e. cost is easier to quantify than benefits); and political sensitivity towards particular results.

Paper includes An evaluation of the impact of research on policy A reflective piece about evaluation of the

�


292

impact of research on policy A description of method(s) for evaluating impact of research on policy Evaluation of effectiveness/value for money of methods A review of methods for evaluating impact more generally Others of relevance (give reason)


293



Paarlberg R L External impact assessment of IFPRI’s 2020 Vision for Food, Agriculture and the Environment Initiative Impact Assessment Discussion Paper 10 http://www.ifpri.org/impact/iadp10.pdf. June 1999 USA International Food Policy Research Institute


Environment


Empirical


What has been the impact of the 2020 Vision initiative (intended outcome: policy change), on researchers and educators, international policy leaders, and developing country policy leaders?

Study design

Records of conference attendees and activities, surveys of attendees, staff testimonies, documentary analysis and literature review.

Theory

Analysis

Findings/key points

The author argues that impacts were difficult to assess given that target audiences were large, diverse and geographically dispersed. The assessment was in two parts (for each audience): whether the initiative impacted policy thinking; and whether this catalysed any policy actions. Impacts are rated highly significant, significant, noticeable or not noticeable. Of the three target audiences (researchers and educators, international policy leaders, developing country policy leaders), the first may be the easiest to reach, but the third is more important in the long run. This report classifies


294

impacts as ‘reach only’, ‘reach and catalyse consensus only’, and ‘both consensus and action’ (i.e. materials reaching audience). The author reflects that whilst ‘reach’ (consumed document enough to have an opinion) is fairly simple to attribute, ‘reaching consensus’ is more difficult (documented change is rare, explicit references missing from internal memos etc – interviews/ anecdotes may be incomplete), and ‘action’ is the most methodologically difficult to assess for attribution (other contributing factors). The findings are mapped on a matrix (audience on x-axis, impact on y), which the author argues helps to distinguish between easy and difficult goals (p8) and identifies where impacts are greatest (p47). Page 17 – in influencing policy debate, international institutions, NGOs, donor governments and developing country governments are considered. Page 38 – it is impossible to report on impacts on developed country government (even if these are desirable) as this involves the longest time lag, and data not yet available. Similarly, network activities (p41) are difficult to ‘evaluate with confidence’ early on in the initiative.


� �


295



Pang T, Sadana R, Bhutta Z, Hyder A and Simon J Knowledge for better health: a conceptual framework and foundation for health research systems Bulletin of the World Health Organisation 2003 81(11) pp815-820 Switzerland World Health Organisation


Health


Conceptual


Proposes a conceptual framework to describe the boundaries, components, goals and functions of health research systems.

Study design

Informed by a literature review and international consultation.

Theory

Analysis

Findings/key points

Some approaches to gauging the impact of research on practice include bibliometric analysis (in guidelines), and tracking the impact on public attention, public awareness and understanding. Any analyses and eventual strategies to strengthen research systems should address all functions and goals rather than the narrow focus on research outputs. The authors note that impact analysis is largely for the purpose of accountability. The report argues that a conceptual framework enables benchmarking, identification of best practices and institutional learning. Analysis of health research systems can improve understanding of how research contributes to gains in health and health


296

equity (long term impacts). Paper includes An evaluation of the impact of research on policy A reflective piece about evaluation of the impact of research on policy A description of method(s) for evaluating impact of research on policy Evaluation of effectiveness/value for money of methods A review of methods for evaluating impact more generally Others of relevance (give reason)

�


297



Park D A proposal for measuring the benefits of policy-oriented social science research Impact Assessment Discussion Paper 3 http://www.ifpri.org/impact/iadp03.pdf. 1998 USA International Food Policy Research Institute




Conceptual


Develops an economic approach to quantifying the benefits of policy-oriented social science research.

Study design

Theory

Analysis

This report proposes a theoretical model to quantify the value of knowledge according to the quality of individual policies that can be associated with particular research (does not propose how to identify policies), in particular calculating public sector efficiency gains and economic growth in terms of output, profit and cost. The paper acknowledges the shortcomings of an empirical approach and observes that qualitative evaluation will be necessary to judge non-efficiency objectives of policy.



298


�


299


ID reference number 114 Method of identification (e.g. database) Recommended Details of publication Author Title Source (journal/conference etc) Year/volume/pages Country of origin Institutional affiliation

Peckham S, Willmott M, Allen P, Anderson S and Goodwin N Assessing the impact of the NHS Service Delivery Organisation Research Programme Submitted to Evidence & Policy (under consideration 2007 UK London School of Hygiene and Tropical Medicine, Service Delivery Organisation


Health


Empirical


What has been the impact of the SDO-funded research programme?

Study design

Impact assessment using case studies and the HERG payback model.

Theory

Analysis

Organised around the five ‘payback domains’: service delivery; policy; practice, research and capacity building. Also reflects on the value of the model in evaluating impact.

Findings

Application of the HERG payback model provided a useful framework for reflecting on challenges involved in assessing the impact of research programmes. In particular, there were issues of timing (when to do the assessment and how to capture the complex interactions occurring between knowledge producers and potential users).


� �


300



301



Internet search

Details of publication Author Title Source (journal/conference etc) Year/volume/ pages Country of origin Institutional affiliation

Percy-Smith J, Speller V and Nutley S Evidence informed Policy and Practice: a review of approaches used in Scotland NHS Health Scotland www.healthscotland.com/documents/1318.aspx. 2006 Scotland NHS Health Scotland


Health


Empirical


What was the context, objectives, approaches of evidence informed policy and practice (EIPP)? How and why did programme priorities develop? What is the value for outputs of the five EIPP programmes? What is the quality of review methods? How effective were these outputs in influencing policy and practice?

Study design

Literature review, documentary analysis, preliminary seminar, 22 stakeholder interviews, survey of network participants, feedback seminar/workshop.

Theory

Analysis

Analysis at three levels: programme; institution; practitioner.

Findings/key points

In most cases, interviewees were only able to comment on very specific cases, not overall activities and outputs. The report also found that the highest policy impacts were observed where issues were already high on the policy agenda, and where there was local engagement (for local policy impact). The paper also reflects upon the difficulty of discerning the impact of research on policy, as to a large extent the programmes were set up to be EIPP. Thus, the question is to what extent specific research projects had impact. In one case (Heart Health), impact was linked to networks. The report distinguishes between local and national policy impacts (these do not necessarily occur


302

together). Credibility and utility, a close fit to policy developments, active dissemination and priority, implementation advice and good practice were critical to impact. Emphasises that influencing/impacting on policy is a process. ‘The importance of producing succinct but credible evidence briefings is paramount’ (p60).


� �


303



Referenced


Pestieau C Evaluating policy research Research Paper W/22 http://www.cprn.org/documents/24336_en.pdf. 2003 Canada Canadian Policy Research Networks


Policy research


Conceptual


How can research undertaken to influence public policy be evaluated? How can the ‘value added’ by research be measured?

Study design

Theory

Analysis

Findings

This report examines supply/demand of policy research, and suggests that evaluation needs to go beyond measuring outputs (quantitative) to outcomes (qualitative), and relate these to research objectives. Evaluation is a tool not an end in itself (p8). Evaluations need to consider the timescale and intended impact within that timescale. ‘To choose the most appropriate approach, stakeholders…should define what they want to find out and how the information will be used’ (p9). The report highlights the use of Results-based Management and Accountability Framework in Canada (audit/ monitoring tool), Outcome Mapping, and Performance Indicators.



304


�


305



Pielke R A Usable information for policy: an appraisal of the US Global Change Research Program Policy Sciences 1995 28(1) pp39-77 USA National Center for Atmospheric Research


Environment


Empirical


An empirical evaluation of the US Global Change research programme and why it was deemed to not produce ‘usable’ policy research.

Study design

Used documentary analysis (including speech transcripts) and interviews.

Theory

Analysis

Logic framework to evaluate programme performance within its legal mandates, and in relation to successes and failures.

Findings/key points

Evaluation within this report distinguishes between the practice of policy making (politics, negotiation, compromise) and the science of policy (invention and evaluation of alternatives). The authors reflect upon ongoing evaluation, outputs, events (i.e. meetings) and assess the contribution of the US programme given its mandate to produce policy-relevant ‘usable’ research. Examines the factors contributing to utilisation (and thus non-utilisation) of research (i.e. communications, behaviours). The authors acknowledge that even where policy makers have access to ‘good’ (subjective) information, policy decisions are neither obvious nor easy to reach. ‘Program performance is


306

often as much a function of securing promises which are enforceable, as it is of measuring program output’ (p71).


� (an empirical evaluation of a programme whose mandate it was to influence policy)


307



Internet search


PREST (Policy Research in Engineering, Science and Technology) Smart innovation: a practical guide to evaluating innovation programmes. Manchester University, PREST. Available via: www.mbs.ac.uk/Research/engineeringpolicy/index.aspx. 2006 UK Manchester University


Science


Descriptive


This is a guide aimed at building an evaluation culture around innovation (rather than research) programmes. It looks at reasons for doing evaluations, design and management, and methods.

Study design

Literature reviews, a survey, interviews and case studies were used to inform the development of the guide.

Theory

Analysis

Findings/key points

The authors argue that the task of evaluating innovation is much more challenging than evaluating R&D programmes (although it argues that R&D can take even longer to have visible payoffs). Research programmes aimed at the promotion of research (and development) activities, generally act in a fairly straightforward manner through the provision of financial support either for the activities themselves, the facilities and instrumentation required for research, or for research personnel or their mobility. The anticipated outcomes of research programmes tend to be quantifiable and easily characterised (publications, qualifications, trained personnel, etc. – interesting to note these are ‘outputs’ in other evaluative literature). The report also outlines ROAME as an evaluative approach to assess research programmes.


308


�


309



Recommended


Research Council Economic Impact Group Increasing the economic impact of research councils: advice to the Director General of Science and Innovation, DTI from the Research Council Economic Impact Group Department of Trade and Industry http://www.berr.gov.uk/files/file32802.pdf. 2006 UK Research Councils UK




Descriptive


How can the research councils increase the economic impact of research?

Study design

Theory

Analysis

Findings

Research councils are increasingly emphasising knowledge transfer and the economic impact of research. This strategy document is three-fold, covering: leadership of each council; influencing the behaviour of research producers and means of dissemination; and engaging government and business in developing ‘research missions’. It notes the importance of measuring outcomes (but that it is easier to measure outputs) and recommends developing a one-off measure of impact as a baseline for future evaluations. Appendix C maps the strategy on a matrix to illustrate the areas where research impacts in different policy domains (the only attempt to evaluate impact distinguishes between a ‘small’ and


310

‘major’ contribution). Paper includes An evaluation of the impact of research on policy A reflective piece about evaluation of the impact of research on policy A description of method(s) for evaluating impact of research on policy Evaluation of effectiveness/value for money of methods A review of methods for evaluating impact more generally Others of relevance (give reason)

�


311


ID reference number 120 Method of identification (e.g. database) Recommended Details of publication Author Title Source (journal/conference etc) Year/volume/pages Country of origin Institutional affiliation

Research Councils UK Increasing the economic impact of the research councils Research Councils UK. Available via: http://www.rcuk.ac.uk 2007 UK Research Councils UK




Descriptive


Study design

Theory

Analysis

Findings

UK Funding Councils are developing a successor to the Research Assessment Exercise (peer review to assess quality). The strategy emphasises the importance of collaborative research, knowledge transfer and other activities to increase economic impact. The strategy is considering how best to develop peer review to assess potential economic impacts. In future, RCUK also intends to commission user surveys to assess satisfaction and increase dialogue.



312


� strategy for assessing impact


313


ID reference number 121 Method of identification (e.g. database) BLDS Catalogue search Details of publication Author Title Source (journal/conference etc) Year/volume/pages Country of origin Institutional affiliation

Rider Smith D with Sutherland A Institutionalising IMPACT Orientation: building a performance managing approach that enhances the impact orientation of research organisations University of Greenwich, Natural Resources Institute. Available via: http://www.parcinfo.org/ 2002 UK University of Greenwich




Empirical


Case study analysis using the Balanced Scorecard management technique to evaluate and enhance impact orientation of research organisations.

Study design

Three case studies using the Balanced Scorecard employed field visit, semi-structured interviews, workshops and stakeholder mapping.

Theory

Analysis

Findings/key points

The Balanced Scorecard measure was developed in response to a recognised need to focus on changes over which programmes have direct influence, rather than longer term impacts over which they have less direct influence. The scorecard draws upon Results Based Management, which evaluates for the purposes of performance and accountability, and focuses on attributable results through the chain of ‘resources, process, outputs, client reach, outcomes and impact’ (World Bank, 1999). The ten-step tool is primarily for organisation/programme performance measurement across four


314

perspectives: employee learning; internal business; financial; and client (p30). The paper reflects that the scorecard enables a ‘thoughtful consideration and partial reconfiguration of each case study organisation’, and for performance indicators to reflect specific organisational goals and objectives. The framework adds value to existing performance measures as it clarifies current capacity, issues, opportunities and threats. Factors determining the impact of research on policy have been well documented, but less so at the meso- and micro- levels. (p6) One reason why impact assessments have lesser impact is that their purpose is primarily to validate previous findings rather than inform future strategies. The report suggests that indicators of organisational uptake can provide reliable proxies for impact. Balanced indicators will provide a more realistic assessment of progress and development. Assessments of research impact are criticised for three main reasons (CGIAR, 1997): attribution and the degree of success; a focus on end user benefits rather than a broader focus on other benefits of research that contribute (i.e. capacity building of organisation); and tensions between learning and accountability (information is rarely useful for both).


� �


315



Rigby J Assessing impacts of social science research Paper presented at Social Science Research Impact: Mapping and Measurement, International Social Science Comparisons Workshop, Berlin-Brandenburgische Akademie der Wissenschaften, Berlin, 14-15 March 2002 http://www.stage-research.net/SPSG/paris-issc/BerlinRigby.html 2002 UK PREST, University of Manchester




Conceptual


Concepts and issues in assessing the value of social science research in academic circles, societal development and for government policy.

Study design

Theory

Analysis

Findings/key points

Attempts to conceptualise impact in other disciplines have focused on performance indicators to assess the interaction between social science knowledge and application. Assessments of the impact of research can be either research- or user-focused but, as this report reflects, the use of indicators is complicated when uptake is not linear. ‘To measure and map social science research impacts successfully within single and multidisciplinary contexts, a focus is needed upon each of three logically separable stages of


316

research activity: input stage; research process stage; and output stage’ (p7), i.e. when research is used, how it is used, how widely it is disseminated and by whom/with what benefit.


�


317


ID reference number 123 Method of identification (e.g. database) Referenced Details of publication Author Title Source (journal/ conference etc) Year/volume/pages/ Country of origin Institutional affiliation

Riggirozzi M P and Tuplin T The influence of research on policy: MIMAP Philippines International Development Research Centre, Evaluation Unit http://www.idrc.ca/en/ev-57585-201-1-DO_TOPIC.html. January 2004 Canada IDRC




Empirical


To develop a ‘plausible hypothesis’ about the policy impact of MIMAP (research project to develop new instruments to measure the impact of macro-economic policies on citizens).

Study design

Informed by a previous IDRC paper (Parikj, 2002), project publications and in-depth interviews with selected actors.

Theory

Analysis

Almost half of paper describes the context of the project, the objectives and the socio-economic situation in the Philippines. Analysis focuses on the process of knowledge transfer (e.g. p14).

Findings/key points

The MIMAP project research sought to impact directly on policy makers (problem solving). The analysis reports that this research was utilised, noting that in particular it enhanced the understanding and capacities of policy advisers, and improved national policy discussions. However, information was also utilised at local and municipal decision making levels (p11). Knowledge transfer was important in broadening the policy approach and was found to be a direct consequence. ‘Any assessment of


318

policy impact of research has to account for the many pathways through which research affects policies’: immediate acceptance; raising new questions; policy debate; consensus building; staff development (p11). The project also had an indirect impact on planning, budgeting and programme design through capacity building among researchers and policy makers (p12). Analysis differentiates between direct and indirect influences by categorising impacts into: broadening policy capacities; broadening policy horizons; and changing policies and regimes. The report also identifies impacts beyond the impact on policy (p13) but relates these to the objectives of the research programme (i.e. involvement of local citizens). The authors reflect upon limitations of the impact on policy, as research failed to transfer models to policy making bodies. Page 15 highlights that not only did research enhance understanding, but it also gave credibility to policy formulation. Factors affecting policy influence were identified as: funding; supply and demand; dissemination; technology; collaboration; ownership; policy windows; key people (i.e. personal influence); and institutions.


� �


319



Ryan J G Assessing the impact of rice policy changes in Viet Nam and the contribution of policy research Impact Assessment Discussion Paper 8 http://www.ifpri.org/impact/iadp08.pdf. 1999 USA International Food Policy Research Institute


Environment


Empirical


Ex-post assessment of the perceived influence of the IFPRI research project. Proposes a framework to measure the economic impact of policy changes.

Study design

Interviews with IFPRI partners and stakeholders

Theory

Analysis

Describes how the research was conceived, the socioeconomic and policy environment in Viet Nam at the time, the research outputs (under two headings: information and policy options) and the ensuing recommendations and policy responses. This report was published approximately five years after first IFPRI contact with Viet Nam.

Findings/key points

The evaluation found that the research used was high quality, independent, rigorous and timely: these factors inevitably contributed to influence. Policy decisions were also made earlier than would have been the case without IFPRI research. The author reflects the importance of noting sensitivities in interviews, which can cloud attribution. Analysis distinguishes between lasting and short-term impact (p21). The value of


320

impact is quantified in terms of time/ money saved etc (i.e. the consequences of policy change). When conducting ex-ante policy impact evaluation, it is necessary to account for the directions and speed of policy change.


� �


321



Ryan J G Synthesis report of workshop on assessing the impact of policy-oriented social science research in Scheveningen, the Netherlands, November 12-13 2001 Impact Assessment Discussion Paper 15 http://www.ifpri.org/impact/iadp15.pdf. 2002 USA International Food Policy Research Institute


Environment



How can the economic impact of policy-oriented research be measured? How can the effectiveness of this research in policy making be improved?

Study design

Summaries of conference speeches and workshop discussion groups.

Theory

Analysis

Findings/key points

When conducting an evaluation of the impact of research on policy, evaluators need to consider the context (conducive policy environment, policy-maker/researcher collaboration), timing and dissemination. Evaluation should be integral to the research process from the outset. The economic value of research is likely to be higher if it does not confirm conventional wisdom, but uptake by policy makers less likely. Impact assessments need to consider the scope, time horizon, supply/demand approaches, importance of surprise, attribution, choice of indicators and case studies, and time lags. Research should not compromise on quality and objectivity in the quest for


322

impact. Researchers should consider whether the impacts are indeed desirable, or quantifiable.


�


323



Ryan J G and Garrett J L The impact of economic policy research: lessons on attribution and evaluation from IFPRI Impact Assessment Discussion Paper 20 http://www.ifpri.org/impact/iadp20.pdf. October 2003 USA International Food Policy Research Institute


Environment


Conceptual


Review of research impact assessments at IFPRI, and how evaluation can be used to enhance institutional learning.

Study design

Review of five IFPRI case studies (ex-post evaluations) illustrate lesson-learning.

Theory

Analysis

Findings/key points

‘Methodologies for impact assessment of social science research are not well developed’ (p1). Quantitative methods are particularly useful to assess historical trends in rates of return and compare different contexts, whereas qualitative evaluations describe the processes by which research outputs influence policy formulation. A 1997 IFPRI workshop concluded that at this stage of methodological developments, case studies were more appropriate for drawing conclusions about impact and the means to achieve it, rather than quantitative measures. Ideally, impact assessment would be conducted backwards from the demand-side at the point where a policy initiative occurs.


324

Evaluation needs to consider the scale; time lags/discontinuities; demand-side vs supply-side; surprise; attribution; choice of indicators; sampling; and ex-ante and ex-post assessments. ‘All assessments require databases of outputs, outcomes/influences, and policy responses to enable the evaluator to verify them, track their influence and measure their impact’ (p7).


�


325



Internet search


Sayce K and Norrish P Perceptions and practice. an anthology of impact assessment experiences Technical Centre for Agricultural and Rural Cooperation www.anancy.net/uploads/file_en/impact%20assessment.pdf. 2006 The Netherlands Technical Centre for Agricultural and Rural Cooperation




Empirical


Collection of ‘stories’ about organisational impact assessment perspectives, told from a key stakeholder perspective. What is the purpose of evaluation? Attribution? Who is involved? How are indicators selected and used? What is the scale and scope of assessments? When should assessment be conducted?

Study design

Workshops, 61 in-depth interviews, literature review (and reflects upon the benefits of using ‘Most Significant Change’ p161), website- and documentary analysis, online forum.

Theory

Analysis

Findings/key points

All ‘findings’ are interviewee perspectives. The report notes a trend away from quantitative impact assessments towards qualitative assessments. The approach suggests that plausible ‘stories’ are the closest evidence to ‘proof’. The author reflects that evaluations should be theory-led (p17). DfID is proposing network perspective for monitoring and evaluation, linked to Outcome Mapping’s boundary partners (p18, p22) – i.e. identifying key actors and links between them (indirect impact – the three year timescale of the programme is insufficient to measure direct impact on policy). Interviews and workshops are very useful tools but


326

‘each project is different, and for each you need a different combination of tools. You can’t just pick up a toolkit or blueprint and apply it’ (p22). ‘The process determines the impact’, ‘no individual agency can claim to cause impact, only to make a contribution to impact’ (p28). The report also considers issues in evaluation, such as timescale, scope, attribution, although there are no examples of how these are addressed within impact assessments. Methodologies used by the Tearfund include interviews, focus groups and field visits (p49). Overall, the emphasis is on participatory tools of data collection (also including surveys and questionnaires) to gauge the perceptions of end users (in the case of international development, often local villagers as well as key actors in the project), and there is only one reference to reference-counting as a measure of output. Largely, the programmes were concerned with assessing the processes within the project, and the broader impact of the project, including its impact on policy. Page 146 concludes that there is a shift towards more flexible, process-based approaches that take greater account of community social dynamics. Learning comes across as the major issue in most stories (p148), as well as context and communication. The use of indicators is widespread.


� �


327



Internet search


Science Advisory Council CSA response to SAC recommendations on the End to End Review of Science into Policy in Defra (SAC (07) 02) Defra. Available via: www.defra.gov.uk/corporate/publications/default.htm. 2007 UK Defra


Science


Descriptive


Response to SAC recommendations.

Study design

Theory

Analysis

Findings

This response accepts the recommendations of the review in principle. R&D Strategic Assessment Annex B outlines the approach to valuing research and development according to a) the scale and outcomes being addressed and b) the nature, scale and criticality of the research to influencing those outcomes. The methodology also values quantitatively the costs and benefits, and seeks to assess other contributory factors qualitatively.

Paper includes An evaluation of the impact of research on policy A reflective piece about evaluation of the impact of research on policy A description of method(s) for evaluating impact of


328

research on policy Evaluation of effectiveness/value for money of methods A review of methods for evaluating impact more generally Others of relevance (give reason)

� Annex B is useful reflection on evaluation methodologies within Defra.


329



Science and Technology Committee Research council support for knowledge transfer: government response to the Committee’s third report of session 2005-06: fifth special report of session 2005-06 HC 1653. Available via: http://www.publications.parliament.uk/. 2006 England House of Commons Science and Technology Committee


Science


Strategy


Response to recommendations.

Study design

Theory

Analysis

Findings

The response highlights the role of communication, secondments and collaboration to further knowledge transfer between academia, policy makers and industry. The Committee is currently examining the role of impact evaluations – at present, most are conducted retrospectively. Government does not distinguish between basic and applied research – thus, should the evaluation of research do so?



330


� Useful for understanding of the government’s approach to evaluation.


331



Scott A The dissemination of the results of environmental research Environmental Issues Series No 15. Available via: http://reports.eea.europa.eu/. 2000 UK SPRU (Science and Technology Policy Research), University of Sussex, for European Environment Agency


Environment


Conceptual


What are the issues in implementing the EEA’s strategy for dissemination of policy-relevant research?

Study design

Literature reviews and interviews.

Theory

Analysis

Findings

This report is concerned with bridging the policy-research gap, and makes recommendations to improve dissemination of research findings. Dissemination can be formal/informal, internal/external. In terms of evaluation, retrospective, quantitative and qualitative approaches are proposed. The author recommends moving away from the short term consideration of knowledge use, towards evaluation of the longer term uptake of ideas. Although not specifically an empirical assessment of the impact of research, it is useful to observe the methodology used to discern the issues in bridging the policy-research gap.



332


�


333



Referenced


Scott A, Steyn G, Guena A, Brusoni S and Steinmueller E The economic returns to basic research and the benefits of university-industry relationships: a literature review and update of findings SPRU-Science and Technology Policy Research www.sussex.ac.uk/spru/documents/econreturnsost.pdf. 2001 UK SPRU-Science and Technology Policy Research, University of Sussex


Science


Conceptual


Literature review for Office of Science and Technology on measuring the economic returns to academic research and university-industry partnerships (to identify the various benefits of research to innovation, and the channels through which these benefits are realised).

Study design

Theory

Analysis

Findings/key points

Previous attempts to quantify the returns to research are based on a linear model of impact. By definition, calculations of economic returns must include all costs and benefits (direct and indirect), but the authors observe methodological difficulties in quantifying these, and especially difficulties in quantifying with time lags. The first ‘channel’ of impact is the transmission of ideas (research outputs) which can be measured in publications or patents. A second channel is that of contacts for stronger relationships and effective communication.

Paper includes An evaluation of the impact


334

of research on policy A reflective piece about evaluation of the impact of research on policy A description of method(s) for evaluating impact of research on policy Evaluation of effectiveness/value for money of methods A review of methods for evaluating impact more generally Others of relevance (give reason)

�


335



Smith V H Measuring the benefits of social science research Impact Assessment Discussion Paper 2 http://www.ifpri.org/impact/iadp02.pdf. 1998 USA International Food Policy Research Institute




Conceptual


What are the benefits of social science research? How should these benefits be estimated?

Study design

Theory

Analysis

Findings

This report examines the broader impacts of social science research on firms, households etc. However, the methodological considerations offer useful observations. The report finds that it is difficult to assess the benefits of research using modelling or econometric analyses, in particular where time-relevant information is missing. It suggests that case studies offer one possible alternative, yet these analyses raise questions about selectivity and representation, bias and attribution. The author also reflects that although the diffusion of knowledge takes time, its value and relevance diminishes with time.



336


�


337


ID reference number 133 Method of identification (e.g. database) Referenced Details of publication Author Title Source (journal/conference etc) Year/volume/pages/ Country of origin Institutional affiliation

Soper B and Hanney S Lessons from the evaluation of the UK’s NHS R&D Implementation Methods Programme Implementation Science 2007 2(7). Available via: http://www.implementationscience.com/. UK Health Economics Research Group, Brunel University


Health


Empirical

Research question

An evaluation of the quality of R&D outputs, lessons about communication and commissioning, and the benefits from the projects.

Study design

Triangulation of documentary analysis, 25 semi-structured interviews, questionnaires (to 36 project leaders followed by phone calls/emails; 535 potential users), citation analysis; desk analysis.

Theory

Analysis

Case study analysis and HERG Payback model.

Findings

This evaluation study examined the impact of 22 implementation research projects on further research, teaching and clinical practice. The authors note that the evaluation methods used prompted wide debate among research users and producers, but that it was still difficult to evaluate the programme. Evaluations need to consider the timing, who conducts the evaluation (i.e. researchers better placed to assess quality, users to assess relevance), and the process of dissemination. The report also found that one of the difficulties facing the projects


338

was lack of research capacity in the field. Paper includes An evaluation of the impact of research on policy A reflective piece about evaluation of the impact of research on policy A description of method(s) for evaluating impact of research on policy Evaluation of effectiveness/value for money of methods A review of methods for evaluating impact more generally Others of relevance (give reason)

� �


339



BLDS search


Spilsbury M J and Kaimowitz D The influence of research and publications on conventional wisdom and policies affecting forests Unasylva 2000 51(203) pp3-10. www.fao.org/docrep/x8080e/x8080e02.htm#P0_0. Indonesia Center for International Forestry Research, Jakarta


Environment


Empirical


What is the impact of policy research on policy narratives, and how can these influence policy outcomes?

Study design

Email survey of experts to name influential publications, and their opinion of the influence.

Theory

Analysis

Findings

This analysis does not attribute policy impact to specific publications, nor does it evaluate how impact was brought about. Instead it focuses on whether publications were influential, and uses expert opinion to identify such publications. The findings of this report highlight that for the most part, policy research is most influential in shaping ‘conventional wisdom’ about particular policies, and is likely to be filtered to policy makers via policy narratives and debate in international policy, academic and funding circles (this process takes years). The authors reflect that surveys are subjective findings, and it is not possible to separate argument and issues. The credibility of research is as important to impact as the findings being ‘right’. The authors argue that researchers must respond not only to the ‘research product’ but also to the ‘research market’, i.e. policy


340

impact is most likely when it supports policy-makers’ aims, is timely and accessible.


� (whether it had impact, not how) �


341



Referenced


Thiele G, Devaux A, Velasco C and Manrique K Horizontal evaluation: stimulating social learning among peers ILAC Brief 13 http://www.cgiar-ilac.org/downloads/Briefs/ILAC_Brief13.pdf. November 2006 USA Consultative Group on International Agricultural Research, Institutional Learning and Change




Descriptive


A description of horizontal evaluation and the advantages/disadvantages of its use.

Study design

Theory

Analysis

Findings/key points

Horizontal evaluation is a methodology to evaluate research programmes. It combines self-assessment and external peer review, via a three-day workshop of ‘local participants’, field visits and group discussions. The workshop is conducted as followed: Day 1. Introducing the methodology; Day 2. field visits; Day 3. comparative analysis and closure. The paper suggests that by sharing in the development of the methodology and indicators with which to further evaluate the programme, learning, participation and knowledge exchange are encouraged.



342


�


343



Tress B, Tress G and Fry G Integrative studies on rural landscapes: policy expectations and research practice Landscape and Urban Planning 2005 70(1-2) pp177-191 The Netherlands, Norway Alterra Green World Research Landscape Centre, and The Agricultural University of Norway


Environment


Empirical


What do funding agencies expect from integrative research projects, how do researchers make them operational and what are their outputs?

Study design

Open-ended interviews with researchers, project leaders, PhD students and representatives of funding bodies, complemented by literature review.

Theory

Analysis

Findings

The emphasis of this report is on understanding policy expectations and research practice of integrative studies. It finds that most research projects are funded because of their problem solving capacity. However, improved communication between researchers and funders regarding policy expectations are still required. The 6th European Framework Programme has increased the expectation that research is more directly relevant to policy. Success criteria for interdisciplinary studies include not only the quantity and quality of scientific results, but also the quality and quantity of communication. Research outputs can be characterised as scientific publications,


344

non-scientific publications and management tools. This report highlights the need to distinguish between research for policy and research for practice when evaluating impacts.


�


345



Trostle J, Bronfman M and Langer A How do researchers influence decision-makers? Case studies of Mexican policies Health Policy and Planning; 14 (2): 103-114 1999 Mexico


Health


Empirical


Study of the relationship between health research and health policy in four health programmes

Study design

Four case studies of health programmes (AIDS, Cholera, family planning and immunization) in Mexico. Local experts in each topic area conducted 67 interviews with researchers and officials and reviewed relevant documents. The case studies used a common set of interview guidelines and all but two were taped and transcribed. The results of the case studies were reviewed with additional researchers and government officials.

Theory

Drew on Walt and Gilson (1994) importance of looking at policy in terms of actors, processes and context as well as content.

Analysis

Aimed to reconstruct the processes of research use, to characterize the processes and to identify barriers and facilitators to knowledge transfer.

Findings/key points

The case study approach allowed the researchers to compare across the four programmes and to characterize the process of linking health research and


346

decision-making. Difficult to compare the results with other studies as there are very few other empirical studies and ‘ideal parameters for comparison do not exist’ p112.


�


347



Referenced


UK Evaluation Forum Medical research: assessing the benefits to society Academy of Medical Sciences et al http://www.wellcome.ac.uk/assets/wtx031817.pdf. 2006 UK Academy of Medical Sciences, Medical Research Council, Wellcome Trust


Health


Conceptual


What methodologies are currently used to assess the socioeconomic impacts of medical research? What works?

Study design

Theory

Analysis

Examines bibliometrics, case study analysis, peer review, survey/ consultations, and economic methods.

Findings/key points

This report finds that there is no single ‘best practice’ method of evaluation – different methods will be used by different organisations for different purposes at different times. Quantitative methods demonstrate measures of efficiency and trends over time, whereas qualitative methods inform understanding about how particular research has an impact on policy/practice. This study recommends the use of a framework for evaluation, broadly: 1. inputs; 2. process; 3. outputs,;4. outcomes (p30).



348


�


349



Database


Walter I, Nutley S and Davies H Research impact : a cross sector review – literature review Research Unit for Research Utilisation. Available via: http://www.ruru.ac.uk/discussion_papers.html. 2003 UK University of St Andrews, Research Unit for Research Utilisation




Conceptual and descriptive


Exploring models of research impact (for a wider project commissioned by the Learning and Skills Development Agency – see Ref 108)

Study design

A cross-sector literature review exploring models of research impact. The search strategy identified 5,800 references, 341 of which were relevant to the review question and included in the review. Of these, 186 were empirical papers.

Theory

Analysis

Findings/key points

This report summarises the literature on conceptual frameworks relevant to impact. It explores practices (i.e. dissemination) designed to promote impact, examines the effectiveness of the different practices and the theoretical frameworks in which they are located. However, it does not specifically consider how effectiveness is best assessed (one of the three aims of the report).



350


� Consistently finds theoretical frameworks within which to conduct evaluation. A useful discussion of mechanisms to enhance research impact.


351



Reference


Walter I, Nutley S and Davies H Assessing research impact: report of seminar, 15-16 January 2004 Research Unit for Research Utilisation. Available via: http://www.ruru.ac.uk/discussion_papers.html. 2004 UK St Andrews University, Research Unit for Research Utilisation




Descriptive


Account of a seminar looking at the assessment of research impact.

Study design

Theory

Analysis

Findings/key points

This report highlights many key issues in conducting an evaluation of the broader impact of research. It finds that it is as important to understand dynamic social processes involved in using research as it is to measure its outcomes – as such, approaches to assessment need to capture complexity. Indeed, it may not even be possible to trace many impacts. Other issues in conducting an evaluation include the timing, what to measure, what methodologies to use, and the broader context of research, utilisation and evalution. ‘Assessing outcomes for end users is important, but to what extent can we disentangle the impacts of complex, multifaceted and contingent interventions?’ (p8).

Paper includes An evaluation of the impact of


352

research on policy A reflective piece about evaluation of the impact of research on policy A description of method(s) for evaluating impact of research on policy Evaluation of effectiveness/value for money of methods A review of methods for evaluating impact more generally Others of relevance (give reason)

�


353



Warburton D Evaluation of Your Health, Your Care, Your Say: an independent report commissioned by the Department of Health Department of Health. Available via: http://www.dh.gov.uk. July 2006 UK Department of Health


Health


Empirical


To what extent did the YHYCYS consultation methodology and delivery meet objectives?

Study design

110 of the 1240 (total) attendees at deliberative events on YHYCYS were asked for feedback on their satisfaction with the use of their inputs in the subsequent White Paper. The methodology assessed participants’ perceptions of the influence on the process, supplemented by documentary analysis. Evaluation was overseen by a steering group.

Theory

purpose + process + context = outcome (Pawson and Tilley, 1997)

Analysis

Findings/key points

For this report, evidence is taken to be consultation responses and participants’ inputs at four regional deliberative events. The costs of the consultation were widely acceptable to participants, and largely deemed to be money well spent. Interviews (conducted after the YHYCYS White Paper was published) enabled participants to reflect on the consultation process as a whole. The report examined the purpose (learning and capacity


354

building) and the context (i.e. pressure for timing) of the consultation, and mapped the process on a timeline to identify strengths, weaknesses and barriers. The authors reflect that this analysis enabled both direct and indirect benefits to be identified (i.e. improved policy process, as well as generated awareness).


� �


355



Weiss C H Studying research utilization Paper prepared for the International Development Research Centre meeting, Ottawa, March 24-25 http://www.idrc.ca/en/ev-66465-201-1-DO_TOPIC.html. 2003 USA Harvard University




Descriptive


How does academic and non-academic research get used in policy?

Study design

Interviews with federal, state and local decision makers.

Theory

Analysis

Findings/key points

Research can have an impact the whole way from inception through to policy decision making. A single study is unlikely to (and should not) have direct or apparent influence. Increasingly, studies of research impact have shifted away from examining the influence on individuals in policy making positions, to the policy making system as a whole. The context of decision making, and the nature of the topic, is important in gauging how research is used. The author reflects upon the IDRC study, and suggests that the central framework can facilitate comparable qualitative analyses, but as it is retrospective it relies on memory and individual judgements.



356


�


357


ID reference number 142 Method of identification (e.g. database) British Library search Details of publication Author Title Source (journal/conference etc) Year/volume/pages Country of origin Institutional affiliation

Williams D and Rank AD Measuring the economic benefits of research and development: the current state of the art Research Evaluation 1998 7(1) pp17-30 Canada ARA Consulting Group Inc




Conceptual


A review of methodologies to measure the socio-economic impacts of R&D.

Study design

Theory

Analysis

Findings/key points

The report finds that the most commonly used approach to assessing socio-economic impact is cost-benefit analysis of selected case studies. The authors reflect that case studies ‘probably the best method for applied R&D’. The report identifies a number of key methodologies for impact assessment: modified peer review (i.e. commentary on impact); user/ client surveys; benefit-cost methods (more appropriate for innovation impact than policy); case studies and histories; and partial indicators/integrated partial indicators. Econometric methods and bibliometrics were found to be not useful for measuring impact beyond the research community: however, patent analysis could be a useful indicator in some circumstances. The authors note the importance of observing both direct (from known use of research) and indirect (unknown use) impacts (p20), and


358

measuring them separately (p21). ‘In order for the direct benefits of research projects to be capable of estimation in this way it is necessary that, for each project, the analyst is able to identify and believably estimate:

• The use of research results and the user groups;

• The potential economic benefits from applying the research results, and the link between using the research results and these benefits;

• The market for whatever is produced as a result of using the research results;

• The time pattern for market penetration; and

• The value of the economic benefits in dollar terms, ideally based on direct market prices.’ (p22)

Attribution should be distinguished at three levels (p26): the extent to which research output is attributable to a particular research producer; the extent to which the outputs could have been attained without this particular research (i.e. could another research producer have the same expertise) (incrementality); and the extent to which apparent impacts are attributable to using these specific outputs. The authors recommend reporting both lower and upper boundaries of estimate. ‘The benefits associated with R&D activities are equal to the sum of the value each of the end-users of the R&D outputs obtains’ (p28). In conclusion, the report suggests that the focus of impact evaluation should be two-fold: the direct benefits of known uses of research; and benefits of the known uses of competencies developed during R&D activities.



359


� � �


360



Referenced


Wooding S, Hanney S, Buxton M and Grant J The returns from arthritis research: a report prepared by RAND Europe for the Arthritis Research Campaign. Volume 1: approach, analysis and recommendations RAND Europe http://www.rand.org/pubs/monographs/MG251/. 2004 UK RAND Europe and Brunel University


Health


Empirical


An evaluation of the long-term outcomes of 16 ARC research grants (early 1990s). Presents a conceptual framework to evaluate the relationship between inputs, process, outputs and outcomes.

Study design

Documentary analysis, literature review, semi-structured interviews, bibliometrics (outputs and knowledge flows), cross-case analysis of case studies, peer review. 16 case studies were selected from 56 using a matrix of representativeness.

Theory

Analysis

HERG Payback model. Two approaches to cross-case analysis: 1. qualitative assessment based on discussion within the team of key observations made by each member of the team; 2. novel method of scoring against the five payback categories.

Findings/key points

This evaluation report was designed specifically for the purpose of accountability and informing future funding priorities. The evaluation develops the HERG Payback model (knowledge production, research targeting and capacity building, informing policy and product development, health and health sector benefits, wider economic benefit – sub-categories


361

indicate assessment criteria, p10). The authors recommend that all research outputs be considered in evaluation (not just peer reviewed journal papers). OST (2003) uses bibliometrics for PSA target metrics. The authors reflect that at best, bibliometrics can be used to generate hypotheses rather than conclusive evidence, as they underestimate impact. Citation analysis can be applied to assess impact within the research community. Policy impact can occur at many levels (organisational, local, national) and include education and training, product development, inspection etc. The case study approach is recommended where the emphasis is on demonstrating the long term benefits. Evaluation team members scored each category of payback on a scale 1-9, followed by group discussion and the opportunity to re-score, and the median score was used for analysis. This converts qualitative data into quantitative measures, but issues of variable quality (subjective), systematic bias and different levels of understanding remain (p29). The authors reflect upon the payback model and raise concerns about the generalisability of case studies, whether a largely linear framework (i.e. logic model input-output-outcome) would be more suitable, issues of attribution, and how to select a suitable timeframe (a compromise between allowing outcomes to fruition and ensuring that records are detailed) (p63). However, the authors conclude that many of the diverse paybacks would not have been observed without a case study approach. Regular logging of impact over time would enable a greater detail of impact analysis.

Paper includes; An evaluation of the impact of research on policy A reflective piece about evaluation of the impact of research on policy A description of method(s) for evaluating impact of research on policy Evaluation of effectiveness/value for money of methods

� �


362



363



Personal recommendation


Wooding S, Nason E, Klautzer L, Rubin J, Hanney S and Grant J Policy and practice impacts of research funded by the Economic and Social Research Council: a case study of the Future of Work programme, approach and analysis RAND Europe. Available via: http://www.rand.org/pubs/technical_reports/ 2007 UK RAND Europe, Brunel University




Empirical


Evaluation of the impact of the Future of Work programme, and reflection on the applicability of the HERG Payback model to the social sciences.

Study design

Data were collected using document review, key informant interviews, an on-line survey of all Principal Investigators, and four case studies to analyse the pathways to impact in more detail. Literature review for context.

Theory

Analysis

Health Economics Research Group (HERG) Payback Model. Data collected were analysed in a one day workshop.

Findings/key points

This analysis used the HERG Payback model as an analytical conceptual framework, and the authors reflect that it was useful for evaluating the wider impacts of social science (originally developed for health services research). The classification of impacts had to be generalised, but the logic model could be used in its existing form. The researchers found that some impacts were inaccessible to evaluation, because of political sensitivity (with


364

participants feeling unwilling to discuss them) or anonymity guarantees (couldn’t assess the impact of participation in the research as anonymity was guaranteed). Payback captured some elements of knowledge creep and interactions between different outputs (as it allowed for context to be explored). However, ‘the incremental nature of policy remains a difficulty in assessing impact at the project level’. Timing was identified as a critical factor: research on impacts can take place before impacts occur or after they have come and gone. The authors suggest that researchers are asked to record early impacts and look for impacts 5-10 years later (perhaps in two years for ‘hot’ topics which are likely to have earlier impacts). Participants in the study suggested that structured interviews would be more useful than written surveys (the on-line survey ended up being very time consuming for participants). This hybrid methodology has been used by the authors elsewhere to overcome problems of low survey response rates (Wooding et al, 2000). In future there is a need to go beyond ‘zero order approximation ‘(the counting of impacts, valuing them all equally). Also there is a need for studies focusing on getting research through the door rather than ‘to’ the door of policy makers (seminars, reports etc).


� �


365



World Bank Monitoring and evaluation: some tools, methods and approaches World Bank. Available via http://www.worldbank.org/ieg/. 2004 USA World Bank




Conceptual


Assessment of current knowledge-sharing practices.

Study design

Theory

Analysis

Findings/key points

This report focuses on the processes of knowledge sharing rather than the starting and end points. The report highlights the importance of evaluation reflecting the context – the relationships between source and recipient, form and location of knowledge, recipient’s learning predisposition, source’s knowledge sharing capability and the broader environment. In the discussion, tacit and explicit knowledge are distinguished (p20). Highly tacit knowledge is embedded with meanings and values and is thus more difficult to transfer, and in the same way that a ‘lack of assessment of the knowledge’s form could lead to inappropriate sharing activities being undertaken’ (p35), it would be useful for evaluators to note the form of research. Credibility (Arrow, 1971), receptivity, and intent (Hamel, 1991) (p30) all have implications for


366

knowledge-transfer. Paper includes An evaluation of the impact of research on policy A reflective piece about evaluation of the impact of research on policy A description of method(s) for evaluating impact of research on policy Evaluation of effectiveness/value for money of methods A review of methods for evaluating impact more generally Others of relevance (give reason)

�


367



Zilberman D and Heiman A The value of economic research Impact Assessment Discussion Paper 7 http://www.ifpri.org/impact/iadp07.pdf. January 1999 USA International Food Policy Research Institute


Science


Conceptual


A reflective report about how to quantify the benefits of research (information, technological change and improved policy).

Study design

Theory

Analysis

Findings/key points

The impact of research depends upon dissemination and the capacity of users to take advantage of findings. Thus, investment in dissemination and improving understanding are important means of increasing the impact of research. The authors suggest that a case study approach would limit the cost of an extensive impact analysis of each research project. The report indicates that the most important contribution of impact assessments of research on policy may be preventative (i.e. screening wasteful policy proposals) (p4), which would not be apparent in policy implementation but could be reflected in prior policy discussions. The latter part of the paper develops an economic formula to assess the net benefit of research that led to innovations, which considers time


368

horizon, attribution, contributing factors – but at the same time acknowledges the empirical challenges of obtaining actual estimates. ‘The cost, speed and accuracy of research evaluation are significantly improved if documentation procedures are established and followed as part of the research effort’ (p13). Attribution can be difficult to discern between multiple actors in the policy process. The authors suggest estimating attribution through testimonies and evidence obtained from interviews with non-researchers and media coverage. Obtaining policy makers’ opinions may be the ideal way of understanding which research was most influential, but obtaining this data is not always feasible. Attributing policy change needs to consider conceptual and applied research, as well as political actors (p16). ‘One of the most difficult obstacles in preparing an assessment of the benefits of economic research is a lack of evidence’ (p20).


� � � �


369

Appendix B: Review Methodology

Scope of the review

The review aimed to answer the following question:


In order to do this, it focused on English language publications covering a period of

twenty years (1987-2007) and sought to identify:

• Evaluations of the impact of research on policy

• Reflective pieces about evaluation of the impact of research on policy

• Descriptions of method (s) for evaluating impact of research on policy

• Evaluations of effectiveness/ value for money of methods

• Reviews of methods for evaluating impact more generally

The review used a number of systematic review methods, including identifying a

question, developing a well defined search strategy, using pre-defined select criteria

and completing data extraction sheets for each study (Boaz et al 2001). The main

advantage to this approach is that it seeks to be as transparent as possible and

therefore open to scrutiny.

Search strategy

The search was conducted in three parts.

1. Database searches

Ten key databases were searched by an information scientist: INSPEC; Engineering

Index; Web of Knowledge (science and technology and social science sections);

International Bibliography of the Social Sciences (IBSS); World Political Science

Abstracts; British Library Direct (covering the last 5 years of the INSIDE database);

EMBASE (Excerpta Medica); Pollution Abstracts; Geobase; and public affairs,

science and technology and economic databases produced in Australia under the

corporate title Informit (2004 onwards).

The following combinations of key words were used:


370

• research AND impact AND policy AND (waste OR environment OR

pollution)

• research impact AND policy AND (waste OR environment OR pollution)

• Research AND impact AND policy AND evaluation AND (waste OR

environment OR pollution)

• research impact AND policy AND evaluation AND (waste OR environment

OR pollution)

From titles, 142 papers were identified as relevant, 27 of which were selected for

inclusion based on a closer examination of the abstract (or full paper where an

abstract was not available). A further search was conducted of three library

catalogues: the British Library; SPRU – Science and Technology Policy Research;

and the British Library for Development Studies (BLDS). This identified 47 papers

based on titles, 29 of which were selected for inclusion based on the abstract.

2. Web searches

This involved a more creative search of key organizations, including UK government

departments, Select Committees, evaluation societies and research councils. The

websites of the following 30 organizations were searched:

AHRC (Arts and Humanities Research Council), Australian Research Council,

BBSRC (Biotechnology and Biological Sciences Research Council), Canadian Health

Services Research Foundation, Canadian Environmental Assessment Research

Council, CGIAR (Consultative Group on International Agriculture Research), UK

Department of Health, DEFRA (Department for Environment, Food and Rural

Affairs), EPSRC (Engineering and Physcial Sciences Research Council), ERFF

(Environment Research Funders Forum), European Environment Agency, ESRC

(Economic and Social Research Council), GAO (Government Accountability Office),

IDRC (International Development Research Centre), IFPRI, (International Food

Policy Research Institute) ODI, National Academies Press, NSERC (Natural Sciences

and Engineering Research Council of Canada), NSF (National Science Foundation),

NHS SDO (NHS Service Delivery Organisation), NRC (National Research Council),


371

ODI (Overseas Development Institute) PREST (Centre for Science and Technology

Policy and Management Research), REPP (Research Evaluation and Policy Project),

RURU (Research Unit for Research Utilisation), SCIE (Social Care Institute for

Excellence), SPRU (Science and Technology Policy Research), Technopolis, UKES

(UK Evaluation Society) and the World Bank.

The database and web search were conducted in September 2007.

3. Citation tracking and expert contacts

This involved following up references and contacts emerging from documents

identified through the initial stages of the search, and proved to be a particularly

fruitful source of relevant material. Handsearching key journals in the field (Evidence

and Policy and Evaluation Research) also identified some additional papers. The team

were also contacted to see if they were aware of any additional sources.

A final part of the search involved writing to six key individuals identified through the

review and discussions with the team, for their feedback and advice on any missing

references: Fred Carden (International Development Research Centre), Luke

Georghiou (Centre for Science and Technology Policy and Management Research

PREST); Steve Hanney (Health Economics Research Group, Brunel University);

Sandra Nutley (Research Unit for Research Utilization, University of Edinburgh);

John Holmes (Department of Earth Sciences, Oxford University) and John Young

(RAPID programme, Overseas Development Institute). Comments and suggestions

additional papers were received from five of the six. The consultation was

particularly useful in identifying new research: Nine of the ten additional papers were

either published in 2007 or are due to be published in 2008. All the new papers can be

identified by an ‘a, b, c or d’ after the number (for example 26a).

In total, 351 papers were identified through the different search strategies, of which

156 met the selection criteria for inclusion in the review.


372

Selection criteria

There is a wide ranging literature exploring the relationship between research and

policy but this review focuses on identifying methods for evaluating the impact of

research on policy. The main inclusion criteria was the fitness of the paper or report

with the purpose of the review. To this end, papers were included that addressed one

of the following:

• An evaluation of the impact of research on policy

• A reflective piece about evaluation of the impact of research on policy

• A description of method (s) for evaluating impact of research on policy

• Evaluation of effectiveness/ value for money of methods

• A review of methods for evaluating impact more generally

Particular effort was made to identify papers with specific reference or relevance to

environmental, waste and sustainable consumption and production policy. The review

was restricted to English language publications covering a period of twenty years

(1987-2007)

Data extraction

Study details were recorded in Endnote (reference management software) and a data

extraction sheet was completed for each study identified for inclusion. A subset of

papers (5%) was read and discussed by all three members of the review team in order

to improve consistency between reviewers and to ensure that the data extraction

sheets covered all the critical fields. Following this process, a number of additional

fields were added to the data extraction sheet, including details of where the paper

was identified, and its disciplinary focus.

Synthesis

The synthesis was organized around the questions identified in the original research

specification. The original questions identified in the original specification were:

• How do people evaluate the impact of research programmes (different

methodological approaches) in different policy contexts and countries?


373

• Are there any specific issues for evaluating the impact of research on

environmental policy?

• Which methods are most effective?

• What are the pros and cons to different approaches?

• Which offer value for money?


374

Appendix C: Data tables Field Total Extraction sheets International development 26 01, 02, 03, 21, 25, 26, 26a, 26b, 26c, 26d, 30,

39, 42, 43, 44, 49, 50, 71, 72, 87, 89, 94, 121, 123, 127, 135

Utilisation literature 13 13, 14, 40, 53, 61, 102, 103, 108, 119, 120, 138, 139, 141

Evaluation literature 38 07, 08, 09, 10, 20, 22, 29, 35, 37, 38, 41, 48, 54, 55, 56, 60, 62, 76, 78, 80, 81, 82, 83, 90, 91, 92, 93, 97, 98, 100, 106, 113, 116, 122, 131, 132, 142, 144

Health 35 04, 05, 06, 11, 12, 19, 23, 34, 36, 45, 52, 57, 58, 63, 64, 65, 65a, 68, 74, 84, 84a, 85, 86, 88, 101, 104, 110, 112, 114, 115, 133, 136a, 137, 140, 143

Policy research 04 16, 24, 31, 146 Science 14 15, 18, 28, 33, 59, 69, 73, 79, 95, 105, 118,

128, 129, 145 Environment 26 17, 27, 32, 46, 47, 51, 66, 67, 70, 70a, 75, 77,

79a, 96, 96a, 99, 107, 109, 111, 117, 124, 125, 126, 130, 134, 136

Sustainable Development 0 Waste 0 Sustainable Production/ Consumption

0


375

Country of origin

Total Extraction sheets

Australia 8 06, 11, 13, 22, 29, 60, 76, 88 Belgium 2 41, 46 Brazil 1 05 Canada 20 02, 03, 19, 25, 26, 26a, 26b, 26c, 26d, 42, 43, 44, 59, 61,

85, 86, 89, 116, 123, 142 Denmark 2 75, 78 Finland 4 79a, 91, 92, 93 Germany 2 57, 83 Hungary 1 110 Indonesia 1 134 Mexico 1 136a Norway 2 20, 74 Spain 1 48 Switzerland 3 58, 68, 112 The Netherlands 9 07, 12, 17, 34, 40, 97, 106, 127, 136 UK 61 04, 08, 09, 10, 14, 15, 16, 23, 24, 30, 31, 36, 37, 38, 39,

45, 47, 49, 50, 55, 56, 62, 63, 64, 65, 65a, 69, 70, 70a, 71, 72, 84, 84a, 87, 96a, 99, 100, 101, 102, 103, 104, 108, 109, 114, 115, 118, 119, 120, 121, 122, 128, 129, 130, 131, 133, 137, 138, 139, 140, 143, 144

USA 38 01, 18, 21, 27, 28, 32, 33, 35, 51, 52, 53, 54, 66, 67, 73, 77, 79, 80, 81, 82, 90, 94, 95, 96, 98, 105, 107, 111, 113, 117, 124, 125, 126, 132, 135, 141, 145, 146


376

Paper includes: Total Extraction sheets

An evaluation of the impact of research on policy

58 (42)

02, 03, 06, 12, 13, 14, 16, 17, 20, 21, 23, 24, 26a, 26b, 26c, 26d, 30, 31, 36, 40, 44, 45, 47, 50, 51, 54, 58, 61, 64, 65a, 67, 68, 71, 74, 75, 77, 84a, 85, 87, 88, 94, 96a, 100, 101, 104, 107, 111, 114, 115, 117, 121, 123, 124, 127, 133, 134, 136a, 140, 143, 144

A reflective piece about evaluation of the impact of research on policy

74 02, 03, 05, 06, 10, 11, 12, 20, 21, 25, 26, 27, 28, 31, 32, 35, 36, 38, 41, 45, 46, 54, 58, 59, 61, 62, 64, 65, 66, 70a, 71, 73, 74, 80, 83, 85, 86, 87, 89, 90, 91, 93, 95, 96, 96a, 100, 101, 103, 104, 105, 108, 110, 111, 114, 115, 119, 121, 122, 123, 124, 125, 126, 130, 131, 133, 134, 136, 139, 140, 141, 143, 144, 145, 146

A description of method(s) for evaluating impact of research on policy

42 01, 10, 11, 15, 18, 26b, 26c, 26d 29, 34, 38, 40, 46, 55, 57, 60, 64, 65a, 72, 78, 79, 79a, 80, 81, 84, 86, 87, 89, 92, 96, 96a, 97, 98, 100, 101, 106, 107, 112, 116, 127, 142, 146

Evaluation of effectiveness/value for money of methods

13 06, 09, 11, 15, 29, 33, 39, 63, 79, 81, 82, 142, 146

A review of methods for evaluating impact more generally

24 07, 09, 22, 29, 37, 39, 42, 43, 46, 48, 52, 56, 76, 79, 80, 82, 93, 97, 113, 132, 135, 137, 142, 146

Others of relevance (give reason) 15 04, 08, 19, 49, 53, 69, 70, 88, 99, 102, 109, 120, 128, 129, 138


377

Method of identification Total Extraction sheets Internet 45 02, 03, 09, 10, 18, 20, 22, 28, 30, 32, 33, 42, 43,

44, 50, 51, 53, 54, 59, 61, 64, 70, 71, 79, 83, 88, 89, 91, 97, 105, 106, 107, 108, 109, 111, 113, 115, 118, 124, 126, 127, 128, 132, 141, 146

Database/library search 27 01, 06, 07, 12, 13, 17, 35, 37, 47, 48, 49, 62, 67, 75, 76, 81, 90, 92, 93, 94, 99, 121, 130, 134, 136, 138, 142

Personal recommendation 25 14, 15, 16, 21, 26a, 26b, 26c, 26d, 65a, 70a, 72, 73, 78, 79a, 84a, 85, 96, 96a, 100, 114, 119, 120, 136a, 140, 144

Referenced 55 05, 11, 19, 23, 24, 25, 26, 27, 29, 31, 34, 36, 38, 39, 40, 45, 46, 52, 55, 56, 57, 58, 60, 63, 65, 68, 69, 74, 77, 80, 82, 84, 86, 87, 95, 98, 101, 102, 103, 104, 110, 112, 116, 117, 122, 123, 125, 129, 131, 133, 135, 137, 139, 143, 145

Journal search 04 04, 08, 41, 66


378

DATABASES Identified Selected (Inc Overlap)

Engineering Index 1 1 Engineering Village (1)5 33 13 2 Engineering Village (2) 16 1 Geobase 12 4 2 International Bibliography of Social Sciences

10 1 1

Ovid6 4 1 1 Pollution Abstracts 12 4 Web of Science 12 2 2 LIBRARY CATALOGUES BL 11 1 4 BLDS 7 7 SPRU 29 21 4 Serendipity (papers found through searches for other clients)

3 1

150 57 16

5 Engineering Village is a platform through which various databases, including Inspec, can be accessed 6 OVID is a host that includes a range of databases including EMBASE and Environmental Science Database


379

Appendix D Table 1: Approaches and methods for gathering data7

Description Pros Cons Semi-structured interviews: Total 57 02, 06, 11, 13, 14, 16, 20, 21, 24, 25, 28, 30, 31, 36, 45, 46, 47, 50, 53, 54, 55, 56, 57, 58, 59, 61, 64, 65, 68, 70a, 71, 73, 74, 77, 84a, 85, 86, 87, 88, 92, 93, 96a, 101, 109, 111, 115, 117, 121, 123, 124, 127, 133, 136a, 140, 143, 144, 146

A flexible interview around a framework of themes, with pre-identified key actors.

• Flexible structure enables interviewees to respond in own terms, and interviewer to respond as part of a two way conversation

• Framework ensures comparability of interviews

• Seen as most appropriate when unravelling diverse layers and subtle nuances

• Issues of bias and attribution • Training for interviewer is

necessary (to avoid pre-empting answers or not probing enough)

• Time-intensive (collection and analysis)

• Interviewees may themselves be unaware of indirect influences of research

• Time recall

Case study analysis: Total 56 03, 06, 10, 11, 12, 14, 15, 16, 17, 18, 21, 23, 25, 26, 26a, 26b, 26c, 26d8, 28, 29, 30, 38, 40, 45, 47, 51, 52, 53, 65, 65a, 67, 72, 73, 79a, 85, 86, 87, 92, 93, 96a, 101, 104, 105, 107, 109, 114, 121, 125, 126, 132, 133, 136a, 137, 142, 143, 144, 146

An empirical approach that explores in-depth a project/programme, describing and explaining how and why developments of interest have occurred.

• Can be descriptive and explanatory

• Can demonstrate pathways from research to impact

• Potential to combine sources and methods

• Difficulty in selecting cases • Issues of bias and attribution • Often over-emphasises

importance of research – ‘supply perspective’

• Time-intensive to ensure rigour

7 Bold signifies studies that use the methods, non bold signifies studies that discuss the methods 8 There are multiple references to one particular study conducted by IDRC (26, 26a, 26b, 26c, 26d).


380

(triangulation) • Explores context

• Difficult to apply a common framework across case studies

Documentary analysis: Total 45 02, 03, 09, 11, 14, 16, 20, 21, 30, 41, 44, 45, 46, 47, 50, 52, 53, 54, 57, 58, 64, 65, 65a, 68, 71, 74, 79a, 84a, 85, 86, 87, 96a, 100, 104, 109, 111, 112, 115, 117, 123, 127, 133, 140, 143, 144

Exploration and interpretation of existing documents. Can elicit quantitative or qualitative findings. Often used in conjunction with other methods.

• Can be applied to a wide range of sources (including policy statements, technical reports, minutes, speeches)

• Provides contextual understanding

• Potentially cost-effective

• Relies upon the quality of existing records and access to these

• No single methodology for analysis

Bibliometrics/citation analysis: Total 37 09, 11, 15, 18, 21, 23, 28, 29, 30, 34, 36, 41, 52, 56, 61, 64, 65, 72, 73, 76, 78, 79, 80, 81, 82, 97, 96a, 98, 104, 106, 127, 131, 133, 134, 137, 143, 144

Method for quantifying the impact of research by counting the number of outputs and citations, and analysing citation data.

• Suitable for repeated analyses and comparisons

• Measures original research, not the programmes resulting from it

• Identifies research deemed to be important by subsequent research

• Traditionally considers academic citations, but extended analysis includes grey literature

• Measures outputs not outcomes

• Not comparable between disciplines

• Quantity of output may not reflect quality; risks distorting importance of select publications

• Limited role where publication is not the goal of research

• Many uncertainties, and thus can only be used as a partial indicator

• Time lag between publication and citation

Peer/panel review: Total 32 Widely used advisory • ‘Experts’ confer • Reflects upon rather than


381

09, 11, 15, 18, 28, 29, 37, 38, 41, 48, 50, 55, 56, 60, 73, 78, 79, 80, 81, 82, 83, 92, 93, 97, 102, 105, 109, 120, 135, 137, 142, 143

process of expert scrutiny of projects and programmes. Traditionally used to enhance or maintain quality of science, on the basis that experts in a particular field can reach a consensus.

status, credibility, and acceptability on findings

• Can offer range of constructive feedback to guide process

• Can be conducted at any time during the process of impact analysis

• Relatively cost-effective

• Flexible

measures impact • Time-consuming method,

particularly for the experts involved

• Issues of objectivity and variability (including the quality of experts)

• Impractical to evaluate a broad area (number of peers involved)

• Criticised as eliciting acceptability rather than validity of findings

• Qualitative findings only • Requires comprehensive

information for reviewers Surveys: Total 30 06, 09, 11, 15, 16, 18, 21, 23, 28, 31, 55, 56, 64, 65a, 73, 74, 75, 78, 86, 92, 93, 96a, 101, 111, 115, 133, 134, 137, 142, 144

A pre-formatted series of questions asked of multiple actors, generating both quantitative and qualitative data.

• Can identify outputs/outcomes associated with particular research

• Cost-effective means of providing overview from range of actors

• Wider range of stakeholders than possible with interviewing

• Can identify aspects to focus on in interviews

• Relies upon access to respondents

• Reflects the bias of those surveyed and those who respond

• Unresponsive to unforeseen issues

• May require follow-up interviews to fully understand the results


382

Workshop, focus group: Total 20 03, 18, 19, 24, 25, 28, 29, 42, 43, 49, 53, 58, 71, 79a, 87, 96a, 115, 121, 127, 135

An organised discussion with a group of individuals. The groups can involve a range of different stakeholders.

• Can be conducted at any time during the process of impact analysis

• Less expensive than surveys

• Exploratory in-depth insights

• Can reach a consensus that individual responses may not

• Risk of sample bias • Selective memory of

participants • Unsuitable for competitive or

sensitive topics

Literature review : Total 18 12, 14, 16, 25, 29, 30, 31, 45, 65, 74, 79a, 87, 104, 111, 115, 127, 143, 144

Synthesis of existing research relevant to the study. In the context of impact evaluation, usually used with other methods.

• Useful initial research to define the scope of an impact study/ look at evaluation methods

• Cost-effective

• Depends upon ability to identify and access existing research

Field visit: Total 13 09, 16, 42, 43, 45, 47, 48, 50, 71, 87, 121, 127, 135

Primary research method where the research team visits in person the site of activity. Often used in the international development field to evaluate the broader impacts of a research programme. Can include observing meetings.

• Can ensure information is up-to-date

• Direct observation of activities

• Time-intensive and costly (requires planning and ex-post evaluation if it is to be beneficial)


383

User evaluations: Total 8 40, 72, 78, 79, 96, 97, 120, 140

Participatory method for assessing stakeholder (either users of research or producers of research) satisfaction. May involve interviews, e-mail or telephone survey.

• Looks at both research and research utilisation from a stakeholder perspective

• In-depth understanding of utilisation processes

• Risk that stakeholders have a vested interest in expressing satisfaction

• Can be both time-intensive and costly

Telephone interviews: Total 8 06, 36, 74, 79a, 85, 88, 100, 104

Usually semi-structured interview, often used as preliminary means of identifying key stakeholders.

• Can elicit open-ended information early on in research process

• Enables greater sampling dispersion

• Cost- and time-effective

• Less able to develop rapport with unknown interviewee (restricts line of questioning)

• Interview length is limited • Cannot use visual

prompts/sources

Historical tracing: Total 5 15, 18, 45, 73, 80

Tracing backwards from an outcome to identify contributing factors, using a range of (usually qualitative) data collection tools

• Policy-oriented approach to tracking events and processes

• Can be used to explain ‘how’, ‘what’ and ‘why’

• Relies upon the quality of, and access to, existing documentation

• Difficult to attribute causality • Does not account for indirect

impacts Patents/new technologies: Total 6 11, 18, 34, 41, 76, 82

Where research may have patentable outcomes, this approach gathers data about the number and nature of patents.

• Useful to identify linkages between research and specific outcomes

• Difficult to compare between disciplines

• Time lag after publication • Most commonly used to

evaluate the wider impact of research on industry


384

Network mapping and analysis: Total 4 05, 48, 82, 88

Analysis of the structure of relationships and the consequences for actors’ decisions on actions. Mapping can identify multiple links (structure) and surveys/ interviews can explore how they are used and valued (agency). Can be examined from the perspective of a single actor or total network. (formalised survey, complemented by interviews, workshops).

• Reflects complex interactions of the realities of policy making

• Can identify linkages but cannot identify impacts/ outcomes as a consequence

Positive utilisation narratives: Total 4 01, 39, 72, 127

A participatory method to identify stakeholder accounts of impact, and the use of secondary analysis to understand who contributed to change and how.

• Identifies unexpected change

• Reflects organisational values

• Quick and cost-effective method of evaluation

• Self-evaluation and learning

• Anecdotal accounts and within-organisation evaluation prone to bias

• Does not identify negative impacts or the non-use of research

Impact Log: Total 3 11, 38, 72

A means of logging real-time direct impacts and uptake of research

• Records impacts over time

• Can be used to

• Cannot be conducted retrospectively • Much of the information is


385

(i.e. from informal feedback, field observations).

construct actor matrices

anecdotal and subjective • Does not measure indirect

impact Tracing post-research activity: Total 101

Follows the impact of research according to channels of diffusion through networks and post-research activity of researchers.

• Broadens the scope of research ‘user’

• Seeks to understand the dynamics of research flow, and interactions between key actors

• Difficulty in tracking activities

• Anecdotal


386

Frameworks for structuring and interpreting data

Description Pros Cons Economic analysis (i.e. cost-benefit analysis): Total 18 15, 28, 48, 51, 76, 79, 80, 82, 90, 105, 107, 113, 124, 128, 132. 137, 142, 146

Statistical analysis of quantified inputs, outputs and outcomes. Often used to evaluate effectiveness of research programmes.

• Enables comparison of programmes on the basis of pre-established criteria

• Discounted rates of return can account for time lags and weight factors according to importance

• Easily understood

• Difficulties in quantifying outcomes, social welfare and externalities

• Subjective interpretation of attribution and what to ‘cost’

• Resource-intensive method

HERG payback model: Total 12 11, 23, 29, 38, 65a, 84, 97, 104, 114, 133, 143, 144

A framework for recording the wider impacts of research across five domains: services; policy; practice; research; and capacity-building. Incorporates qualitative and quantitative data collection.

• Common framework enables comparative analysis of projects

• Explores and explains impact • Mixed-method to counteract bias

• Assumes outputs have equal impact • Does not measure non-utilisation • Multi-dimensional categories risk

double-counting • Does not fully explain/account for

complex research-policy interface • Costly and time-intensive method


387

(breadth of information) Outcome mapping: Total 8 42, 43, 52, 71, 72, 87, 116, 1279

Participatory planning tool to identify and monitor progress towards intended outcomes (defined as changes of behaviour, relationships, activities or actions of boundary organisations). Main purpose of learning and accountability.

• Assesses both process and outcome • Assesses contribution (not attribution) • Enables stakeholder dialogue and

consensus building

• Does not evaluate value for money • Observed behavioural change posits a

link between cause and effect which may be impossible to demonstrate

• Costly and time-intensive • Captures elements of policy

implementation rather than focusing on ‘paper’ policy

Benchmarking: Total 5 15, 46, 48, 112, 116

Baseline indicators against which to monitor progress towards objectives. Most useful methodology for ongoing monitoring.

• Measures specific pre-defined outcomes

• Objectives not necessarily measurable • Does not attribute causality • Policy has multiple-objectives; no

single aggregate indicator appropriate • Can provide incentives to achieve

Social analysis: Total 4 15, 48, 76, 79

A methodology to value the benefits of research according to quality of life/social rates of return relative to research inputs.

• Enables comparison of programmes on the basis of pre-established criteria

• Discounted rates of return can account for time lags and weight factors according to importance

• Not all outcomes are quantifiable – requires subjective judgement

• Measures impact of policy rather than impact on policy

Episode studies: Total 4 26c, 71, 72, 87

A four-dimensional framework (political context, use of evidence, links and relationships, external factors) to identify factors, key actors and events that have contributed to a policy change. Mixed methods include interviews

• Both direct and indirect observations • Assesses relative importance of

influences • Process of working backwards can

capture complex policy processes

• Over-emphasises political factors and under-emphasises the role of research

• Risk of actors ‘re-writing’ history

9 26, 26a, 26b, 26c, 26d were coded based on the written reports so they are not listed under outcome mapping. However, in a personal communication, Fred Carden from IDRC explained that outcome mapping had influenced the study design for the evaluations


388

and documentary analysis. Simulation: Total 3 41, 48, 79

Ex-ante probability-based Operations Research Modelling approach to predicting the impact of policy.

• Can reproduce variation and predict impact of multiple objectives

• Can help choose between different options

• Complexity of model determines cost • Limited by the extent of

understanding of the interaction between research and policy

• Hypothetical, limited real world data RAPID outcome assessment: Total 3 71, 72, 87

A participatory mixed-methods methodology to map visually the impact of a programme (including research, events and actors) on policy and the policy environment. Informed by episode studies, case study analysis and outcome mapping.

• Maps cause and effect of key events • Assesses contribution of key actors • Mixed methods to counteract bias

• Costly and time-intensive • Does not capture the economic

impacts of a programme

ROAMEF : Total 3 37, 56, 118

ROAMEF framework (Rationale, Objectives, Appraisal, Monitoring, Evaluation and Feedback) is recommended in the Treasury’s Green Book on evaluation; reflects key stages of the policy process.

• Puts evaluation within a wider framework describing the policy process

• Does not specify methods for conducting evaluation

Logical Framework Approach: Total 2 49, 117

A planning and management tool with four key dimensions: goals, purpose; outputs; and activities. Data are presented in a matrix and mapped against indicators of achievement, means of verification, and risks and assumptions.

• Evaluation is embedded in the planning process

• Does not integrate contextual analysis • Log frame does not represent reality • Does not assess causality/attribution

Results-Based Framework to monitor outcomes • Focuses on attribution by guidance • No prescribed methodology


389

Management and Accountability Framework: Total 2 116, 121

through the process of activity. Results chain: resources; process; outputs; client reach; outcomes; and impact

through the ‘results chain’ • Helps set target and track progress

• Linear model – not reality • Useful for evaluating programme

impacts, not necessarily impact on policy

Research Impact Framework (RIF) : Total 2 84, 84a

An ex-ante four-dimensional (research, policy, services, society) framework for overview of potential (sometimes overlapping) research impact areas.

• Maps potential impacts (positive and negative) which can be used to clarify dimensions of use

• Standardised structure that can be used to compare projects and programmes

• Simple and low cost approach compared to some of the more detailed frameworks

• Descriptive categories – analysis remains subjective

Balanced Scorecard: Total 1 121

A ten-step framework to evaluate research programmes/ organisations across four perspectives: employee learning; internal business; financial; and client.

• Focuses on direct attribution of a programme as it tracks immediate impacts

• Performance indicators reflect specific goals and objectives

• Learning, accountability and strategy

• Short term impact – less useful for assessing attribution of longer term outcomes

• Useful for evaluating programme impacts, not necessarily impact on policy

Correlation matrix : Total 1 20

A matrix mapping the Pearson’s correlation co-efficient between different utilisations of research

• Quantifies the relationship between conceptual, political and instrumental utilisation of a particular piece of research

• Tool for interpretation • Does not attribute causality


390

Appendix E: RAPID Outcome Assessment Where does it come from? The RAPID Outcome Assessment (ROA) was developed by the Overseas Development Institute Research and Policy in Development (RAPID) programme. It is a visual mapping tool combining elements from Outcome Mapping (changes in behaviour), episode studies (tracking back from policy changes) and case study analysis (tracking forwards from research). ROA is designed to assess the contribution of a project’s actions and research on a particular change in policy or policy environment, both during and after a project is completed.

Does the approach have a conceptual/ analytical framework? The method involves triangulation and integration of data collected using three different methods including classical case study (what was done, what happened), behavioural changes among key actors during the life of the policy process (based on the IDRC Outcome Mapping approach) and an examination of the factors contributing to that change interpreted within the ODI’s Context-Evidence-Links framework (left), which considers the process

and influences from the perspectives of political context, role and use of evidence, role of linkages, and effect of the external environment. What methods are used? Preliminary stages of data collection include documentary analysis and key actor interviews. The primary methodology however, is a participatory workshop to identify change and causes of change. Complementary methods including literature review, documentary analysis, interviews and field visits are used to triangulate and clarify workshop findings before and after the event. The last stage of data collection is a de-briefing session with the original stakeholders to discuss initial findings and explore expectations. The data are used to develop a timeline of policy change, activities that contributed to changes in behaviour, and the changes in behaviour itself. Example of the application of RAPID Outcome Assessment The first application of the ROA focused on the Smallholder Dairy Project in Kenya. The programme objectives included a strategy to influence pro-poor policy change based on research findings. The study identified changes in the attitudes and behaviour of key actors at national and local level that could be attributed to the project, even though national policy changes were still in legislative process. What can be learnt from this approach? This is a multi-method approach that tracks forwards from research and backwards from policy. It purports to address the issue of attribution by mapping cause and effect of key events and the contribution of key actors, and by drawing links through participatory evaluative techniques. ROA tracks change and impact at both local and national level. It is important to note the importance of identifying key actors, as the findings are dependent on their contribution and perceptions of change.


391

How do I find out more? Leksmono C et al (2006) Informal traders lock horns with the formal milk industry: the role of research in pro-poor dairy policy shift in Kenya, Working Paper 266. London: Overseas Development Institute.


392

Appendix F: HERG payback model Where does it come from? The Health Services Research Group (HERG) at Brunel University has developed a model to evaluate the ‘payback’ from research. Originally developed to examine the impact of health services research, it has recently been applied to other areas, such as basic and biomedical research, and social science research. Does the approach have a conceptual/analytical framework? The HERG payback model categorises the types of ‘payback’ (benefits from research) in five domains: knowledge production, research targeting and capacity building, informing policy and product development, health (and health sector) benefits and broader economic benefits. Figure 1 is the conceptual framework of how payback might occur.

Figure 1: The Research Sequence, the Knowledge Reservoir and the Assessment of Research Impacts What methods are used? Both quantitative (e.g. bibliometric analysis including ‘grey’ literature) and qualitative methods (case studies, documentary analysis, interviews with key stakeholders, questionnaires) have been used to gather information. Example of the application of the HERG payback model: The HERG model was used to assess the impact of the Arthritis Research Campaign, demonstrating which research was, or was not, used. The findings of the study can be used as an evidence base for future research funding policy. What can be learnt from this approach? The payback model maps different potential impacts of research, of which impact on policy is only one. It tends to employ a mixed methods approach and attempts to quantify the impact of research. Using a common structure between case studies enables a comparative analysis of impacts. How do I find out more? Hanney S R, Grant J, Wooding S and Buxton M J (2004) Proposed methods for reviewing the outcomes of health research: the impact of funding by the UK's ‘Arthritis Research Campaign’. Health Research Policy and Systems 2(4).

Assessing the impact of research on policy: A review of the literature for a project ... ·...

Documents

Transcript of Assessing the impact of research on policy: A review of the literature for a project ... ·...