The Impact of Research on Development Policy and Practice:

This much we know.A Literature Review and the Implications for ICT4D

Dr. Roger HarrisVisiting Professor

Institute of Social Informatics and Technological Innovationroger.harris@rogharris.orghttp://www.rogharris.org/

@HarrisrwhRoger

INTRODUCTION

Higher Education Minister Datuk Seri Mohamed Khaled

Nordin

“Universities should live up to their roles

within the local community while

making an impact in the international

arena.” What does he mean by impact?

Universities must make an impact

What is the impact of research?

Academic Research

Research ImpactsAcademic Impact

TrainingPublications Citations

Socio-economic Impact

Capacity building

Socio-economic benefits

Public Policy

Research Impacts

Academic Research

Research Impacts

Academic Impact

TrainingPublications Citations

Socio-economic Impact

Capacity building

Socio-economic benefits

Public Policy

My question is, are we making any impact?

Achieving either type of impact does not automatically lead to achieving

the other.

There are considerable differences between the processes that lead to

each.

High quality research is always the starting point for both.

Non-academic impacts require more attention than academic.

Two communities

Researchers

• puzzle why there is resistance to policy change despite clear and convincing evidence;

• generally do not see policy engagement as part of their role;

• face strong organizational disincentives that dissuade them from engaging in outreach beyond the scientific community;

• face structural barriers to engaging in knowledge translation activities.

Practitioners and Policy-makers

• lament that researchers do not speak with one voice;

• bemoan the inability of researchers to make their findings accessible and digestible in time for policy decisions;

• lack confidence in their researchers;• regard research-based evidence as a

minor factor when policies for development are formulated and practices shaped.

Researchers, practitioners, and policy makers live in

parallel universes…

…with different values, languages, time-frames,

reward systems and professional ties

Two ‘impacts’

Researchers’ impact

• usually reflected by the impact factor of the journal in which the research report is published;

• impact factor is a measure of the average number of citations that have been made to recently published papers;

• frequently used as a proxy for the relative importance of a journal within its field.

Policy and practitioners’

impact

• for research to have any impact, the results must inform and shape policies and programmes, and be adopted into practice;

• use or consideration of research findings for actual outcomes of social change;

• emphasizes the need for economic and societal impacts.

The implications of two ‘impacts’

Academic ‘impact’

misrepresents what impact is

Research institutions discourage impact-

oriented research by prioritizing the

number and frequency of

publications in peer-reviewed

For policy research, academic peer

review and citation counting are

regarded as too limited for effective

evaluation

The activities that make up the work of knowledge transfer

pose risks to an academic career as they are not widely

accepted as legitimate forms of

scholarship.

Academic performance indicators have given rise to an identity representation of academics as performers;• Researchers are

assessed on the basis of their ‘hits’ instead of on the substance of their work.

• Performativity stifles innovation; promotes superficiality and conformity

THEMES IN THE LITERATURE

Intent

Researchers need to have the intent to

influence policy and/or practice

Intent to influence must be expressly

included among the research objectives

Communication

The most cited factor; repeatedly

stressed

Should start early in the research;

designed into the plan

Communication for development

(C4D) is a sub-field of international

development

Crucial for researchers to communicate effectively in a language that policymakers understand

Evidence requires ‘translating’ the

technical language to make it

comprehensible for policy-makers

Plain language communication

with the public is not widely

accepted as a legitimate form of

scholarship

Information and Communication Technologies

ICTs are blurring the line dividing academia and

professional and amateur writers; e.g.,

op-ed writers, bloggers

Participatory communication processes in development use

ICTs for rapid, multi-sited, multimedia and participant-

driven production and communication of research, as

it unfolds

The ‘echo-chamber’ effect of social media;

@odi_development’s 50 biggest followers have a combined reach of 4.3

million people

Many UK academics are reluctant to adopt web 2.0 tools for their work.

There is a lack of institutional incentives for using web 2.0 tools or for publishing online

UK researchers are discouraged from publishing online by the policy of having international

peer-reviewed journal citations, rather than online

citations, count towards academic promotion

Important to understand the online behaviour of the target

audiences

Open sharing and joint activity is at odds with the nature of the research process, where

the tradition is for solo teams of researchers to prepare their

findings privately

IntermediariesCultural differences,

systemic inadequacies in incentive structures

and the apparent weaknesses in

academic communication skills

and processes, imply a need for

intermediaries

Knowledge intermediary organisations synthesise,

interpret and communicate research, enable access and

make information edible, create demand for information, enable

marginalised voices, create alternative framings, connect

spheres of action, enable accountability, informing,

linking, matchmaking, facilitate collaboration and build sustainable institutions

Developing countries often lack

intermediary institutions that carry

research to policy - mechanisms of policy influence are missing

Policy Entrepreneurs

Critical skills ; being able to;

understand politics

identify key

players

synthesise research by simple

compelling stories

network effectively

build programmes

that bring all these factors

together

A policy entrepreneur is an individual who invests time and resources to

advance a position or policy

It is a role for researchers wishing to influence policy.

Networks

Although research may not have direct

influence on specific policies, the production

of research may still exert a powerful indirect influence

through introducing new terms and shaping

the policy discourse

Research findings and concepts

circulate and are gradually filtered through various policy networks

Various types of networks, such as

policy streams, policy

communities, epistemic

communities, think tank networks and advocacy coalitions

Internet networks facilitate the role

of policy entrepreneur

that is played by intermediary organisations

National, regional and global

networks are playing an

increasing role in development

policy

Incentives

Incentives to produce outputs

that reach a broader section of society are so low

that if engaged in at all, this occurs as an after-thought once

results are published

Incentives are needed for

researchers to engage effectively

with users of their research

Incentives are needed for

officials to engage more closely with

researchers

A determining factor for whether research-based and other forms of evidence are likely to be adopted by policy makers and practitioners

Research is more likely to contribute to policy if

the evidence fits within the political and institutional

limits and pressures of policy makers, and if it

resonates with their assumptions

Researchers must know and understand the key stakeholders in the policymaking process, and understand the way in which the door can be opened to politicians and public interest

Knowledge producers

need to be more self-aware of the political nature of their

engagement in policy processes

Political Context

Demand

Understand the demand for research among policy-makers and

practitioners, by, for example, mapping the existing information-

demand and information-use environment.

The effectiveness and impact of research will be driven by

continuous interactions between supply-drivers and

demand-drivers

Researchers are advised to participate in activities that

would stimulate demand for their outputs,

We know more about how to improve the supply of

evidence than we do about how to improve the

demand for it

Models of knowledge transfer

Science-push or

knowledge-driven

A unidirectional and logical flow of information

from researchers to policy makers.

Demand-pull or problem-

solving

Commissioning information from

researchers by policy makers with

the intent of addressing a well-

defined policy problem.

Interactive

A reciprocal and mutual activity

involving researchers and users in the development,

conduct, interpretation, and

application of research and

research-based knowledge.

EngagementNeed for closer relationships

between researchers and research users, especially

policy-makers. Public engagement processes draw on

a range of methods and approaches

Individuals are moving from simply accessing or consuming

the content and services offered by an online platform to becoming more involved in recommending or promoting it

and actively co-creating the content.

Not all researchers see policy engagement as part of their

role; they accord it a low priority. It is more suited to

those who see themselves as issue advocates

Fewer than 5% of academics regard engagement with the

media as an outlet for scientific findings as having any

consequence for measuring scientific performance at their

institutions.

IMPLICATIONS

Implications for ICT4D research

Identifying the particular contribution of ICTs to

specific development goals has proven to be extremely

difficult

Whilst the contribution in terms of technology diffusion and use –

especially of mobile phones – is easy to detect, the focus has only

recently shifted towards the question of development impact

The poor quality of ICT impact assessment to date derives from its

lack of conceptual foundations. ICT4D researchers need to produce high-quality research if they wish to

influence policy and practice.

Few researchers in ICT4D are drawn from the development studies

discipline, resulting in the use of an impoverished understanding of

development . There is little evidence of any impact of ICT4D research on

development policy or practice.

LESSONS

Lessons for research and academic institutions

Research & academic

institutions

Restructure institutional

incentives to take into account actual

impact.

Create incentives to invest in

dissemination and an expanded range of research

products.

Raise awareness and encourage social change

agents, knowledge brokers and linkage

mechanisms.

In hiring, balance consideration of

publication record with capabilities such as

originality, creativity, commitment, depth of

field experience and impact orientation.

Lessons for researchers

Interact with stakeholders to ensure relevance of research

questions and outputs.

Identify uptake pathways as part of

project design.

Design projects to meet end users’ needs and

aspirations.

Share and publish experiences of how research

results have been ‘translated’ or used for a non-scientific audience.

Lessons for journal editors and publishing organizations

Challenge researchers to propose ways to evaluate

the real impact of their work.

Provide incentives to researchers to publish practitioner-oriented

results of relevance to civil society.

Break the language barrier by publishing ‘mirror’

papers; translations of the complete paper into the language of where the

research was undertaken.

Recognize that sustainable change is a long-term process. Support longer term project time frames (4–10 years) in which sufficient dialogue occurs at the initiation of projects.

Expand proposal requirements to

include the sharing of relevant research

results in an accessible format to appropriate

audiences.

Ensure proposals designate sufficient funds for translation, printing, mailing costs and communication.

Acknowledge that originality often occurs at the fringes. Identify and support small

but innovative, locally driven initiatives.

Lessons for Donors

Examples

Research to Action

Caters for the strategic and

practical needs of people trying to improve the

uptake of development research; in

particular those funded by DFID

Science into Policy

Publication of the UK’s National Environment

Research Council, which helps scientists to;

recognise the relevance of science to policy-makers;

identify available opportunities, routes and best practice to influence

policy-making; and communicate science in

an appropriate and accessible way, to the right policy-makers,

showing how it fits their policy needs.

ResearchImpact

Connects university research with research users

across Canada to ensure that research helps to

inform decision-making. Knowledge Mobilization

Units work to match researchers with key policy-

makers in government, health, and social service agencies to ensure that

academic research is employed by policy-makers and community groups to develop more effective, efficient, and responsive public policies and social

programs.

Conclusions

ICT4D research has been assessed overall as lacking

Little if any evidence of any impact on practice

and policy-making

Need major cultural and

institutional shifts

Some of the changes that are necessary can be initiated

relatively easily by promoting the transitions that ICT4D researchers

will have to make in order to increase the relevance of their

work to wider audiences

Framework for the impact of research on policy and practice

Policy and

practice

IntentComm-

unication

ICTs

Interm-ediaries

Policy Entrep-reneursNetworks

Incentives

Political Context

Demand

Engage-ment

References• Adolph, B., Jones, S.H., and Proctor, F., Learning Lessons on Research Communication and Uptake, Triple Line Consulting Ltd for DFID, 2010. • Brown, C., Are Southern Academics Virtually connected? GDNet, August 2012. http://depot.gdnet.org/cms/files//GDNet_study_of_adoption_of_web_2_tools_v2.pdf accessed 28 March 2013• Carden, F., Knowledge To Policy: Making The Most Of Development Research, SAGE Publications Inc/IDRC, 2009. • Court, J., and Maxwell, S., Policy Entrepreneurship for Poverty Reduction: Bridging Research and Policy in International Development, Journal of International Development, vol. 17, pp 713–725, 2005. • Court, J. and Young, J., Bridging Research and Policy in International Development: An Analytical and Practical Framework, Development in Practice, Volume 16, Number 1, February 2006. • Datta, A., Deliberation, Dialogue and Debate: Why Researchers need to Engage with others to Address Complex Issues, IDS Bulletin, Volume 43, Number 5, September 2012. • de Vibe, M., Hovland I., and Young, J., Bridging Research and Policy: An Annotated Bibliography, ODI Working Paper 174, Sep 2002. • DFID, Social Media Engagement; A Report of the Activities on the R4D Project, • http://r4d.dfid.gov.uk/pdf/outputs/Communication/R4D%20Social%20Media%20Engagement%20Report_HR.pdf, accessed 28 March 2013.• Fisher E. and Holland, J.D., Social Development as Knowledge Building: Research as a Sphere of Policy Influence, Journal of International Development, vol. 15, pp 911–924, 2003. • Gendron, Y., Constituting the Academic Performer: The Spectre of Superficiality and Stagnation in Academia, European Accounting Review, Vol.17 No.2, pp 97-127, 2008. • Gregory, N.F., Should Aid Donors Support Economic and Social Research? Journal of International Development, Vol. 4, No. 2, pp 233-242, 1992. • Greijn, H., Linking Research-Based Evidence to Policy and Practice, Research, Policy and Practice. Capacity, Issue 35, December 2008. • Harvey, B., Lewin, T. and Fisher, C., Is Development Research Communication Coming of Age? IDS Bulletin, Volume 43, Number 5, September 2012. • Heeks, R., Do Information and Communication Technologies (ICTs) Contribute to Development? Journal of International Development, vol. 22, pp 625–640, 2010. • Hovland, I., Communication of Research for Poverty Reduction: A Literature Review. Overseas Development Institute, Working Paper 227, October 2003. • Hovland, I., Making a difference: M&E of policy research, Overseas Development Institute, July 2007. • Jacobson, N., Butterill, D. and Goering, P, Organizational Factors that Influence University-Based Researchers’ Engagement in Knowledge Transfer Activities, Science Communication, Vol. 25 No. 3, pp 246-259, March 2004.

Jones, H., Jones, N., Shaxson, L. and Walker, D., Knowledge, Policy and Power in International Development: A Practical Framework for Improving Policy, ODI Background Note, January 2013. • Kleine, D., ICT4WHAT? Using the Choice Framework to Operationalise the Capability Approach to Development, Journal of International Development, Volume 22, Issue 5, pp 674–692, 2010. • Langou, G.D., Developing Capacities for Policy Influence, Research, Policy and Practice, Capacity, Issue 35, December 2008. • Lewin, T. and Patterson, Z., Approaches to Development Research Communication. IDS Bulletin, Volume 43, Number 5, September 2012. • Masset, E., Mulmi, R. and Sumner, A., Does Research Reduce Poverty? Assessing the Welfare Impacts of Policy-Oriented Research in Agriculture, Institute of Development Studies, the University of Sussex Brighton, UK, March

2011. • Newman, K., Fisher, C. and Shaxson, L., Stimulating Demand for Research Evidence: What Role for Capacity-building? IDS Bulletin, Volume 43, Number 5, September 2012. • O’Neil, M., What Determines the Influence that Research has on Policy-Making? Journal of International Development, vol. 17, pp 761–764, 2005. • OECD, Opportunities, Challenges and Good Practices in International Research Cooperation between Developed and Developing Countries, April 2011. • Ryan, J.G and Garrett, J.L., The Impact of Economic Policy Research: Lessons on Attribution and Evaluation for IFPRI, International Food Policy Research Institute, Oct 2003. • Saxena, N.C., Bridging Research and Policy in India, Journal of International Development, vol. 17, pp 737–746, 2005. • Sen, K., Rates of Return to Research: A Literature Review and Critique, DFID, 2005. • Shanley, P. and López, C., Out of the Loop: Why Research Rarely Reaches Policy Makers and the Public and What can be Done, BIOTROPICA 41(5), pp 535–544, 2009. • Shaxson, L., Improving the Impact of Development Research through Better Research Communications and Uptake, Report of the AusAID, DFID and UKCDS Funded Workshop: London, November 29th and 30th 2010, • Stone, D. RAPID Knowledge: Bridging Research and Policy at the Overseas Development Institute, Public Administration and Development, v29, pp 303–315, 2009. • Sumner, A., Ishmael-Perkins, N. and Lindstrom, J., Making Science of Influencing: Assessing the Impact of Development Research, IDS Working Paper 335, Sep 2009. • Taylor, M., Bridging Research and Policy: A UK Perspective, Journal of International Development, vol. 17, pp 747–757, 2005. • Weiss, C. Research for Policy’s Sake: The Enlightenment Function of Social Research, Research, Policy Analysis, 3:4 p.531, Fall, 1977. • Wheeler, J., Creating Spaces for Engagement: Understanding Research and Social Change, Development Research Centre on Citizenship, Participation and Accountability, 2007. • Williams, G., The Disciplining Effects of Impact Evaluation Practices: Negotiating the Pressures of Impact Within an ESRC–DFID Project, Transactions of the Institute of British Geographers, No. 37 pp 489–495, 2012. • Young, J., Research, Policy and Practice: Why Developing Countries are Different, Journal of International Development, vol. 17 pp 727–734, 2005. • Young, J., Impact of Research on Policy and Practice, Research, Policy and Practice, Capacity, Issue 35, December 2008.