Mapping the impact of MSF research: an evaluation framework and preliminary results

Adam Kamenetzky1, Louise Bishop1, Sarah Venis1, Rafael Van den Bergh2 1Médecins Sans Frontières (MSF), London, UK; 2MSF, Brussels, Belgium

Background

“MSF’s research should improve patients’

lives and the effective delivery of medical

interventions. Does it?”

MSF’s research is rooted in its experience in

delivering humanitarian medical aid in resource-

limited and conflict settings.

In the past 20 years, MSF has published over 1,000

articles in peer-reviewed journals.1

Yet the contribution of MSF research to real-world

outcomes (e.g. programme and policy change) and

impacts (e.g. improvements in patient care) has not

been systematically analysed across the organisation.

Recently, MSF has defined a series of indicators

to assess the impact of its operational research.2

And UK charitable funders have joined together

to develop a practical framework with which

to evaluate medical research impact.3

We conducted a pilot study with the aim of

determining how best to capture and evaluate

the long-term impact of MSF research. Here

we present preliminary findings.

Methods

We used a mixed methods approach.

Quantitative information was sought via

a survey, using a series of pre-defined impact

indicators (A). Interviews with researchers

explored perceptions of their experiences,

and drew out underlying narratives (B).

A. Survey (indicators described by Zachariah et al.)2

B. Interviews (topic guide informed by focus group)

Results

Almost all respondents to the survey indicated

their work had been disseminated. Most also

reported contribution to policy change.

Few indicated their research had directly

improved MSF programmes or patient

outcomes.

Iterative analysis of interview transcripts

is currently underway.

Preliminary findings suggest:

• The importance and relevance of research to

patient care is not well understood in many MSF

project settings.

• There is uncertainty about what constitutes “good”

research, and the value of collecting data in a

consistent way to improve MSF’s operations.

• Documenting the real-world outcomes of research

as they happen is crucial for realistic and thorough

impact reporting and evaluation.

Conclusions

The workload associated with delivering the

questionnaire survey method was not felt to

be sustainable.

The range of research methods and topics

within MSF necessitates a more nuanced

approach.

Above: stuck in the mud? Innovation in how MSF shares research information within

and outside of the organisation could prevent valuable data becoming internally

siloed, or lost when field staff move on (photo w/ permission from Jen Jennings).

References 1 Bishop, L et al. Trends in MSF research publications. Poster: MSF UK

Scientific Day. 10 May 2013

2 Zachariah R et al. Is operational research delivering the goods? The

journey to success in low-income countries. Lancet Infect Dis. 2012

May;12(5):415-21

3 Association of Medical Research Charities: Tracking the impact of

charity research funding. 13 February 2013, available at: bit.ly/10Oz8Xv

Above: efforts to map the impact of MSF’s research must take into account the

complex network of associations that link different areas of its humanitarian work.

0

10

20

30

40

50

60

70

80

90

100

Met one or more criteria

for dissemination

Contributed to national /

international policy

Improved performance of

MSF programmes

Reduced patient

morbidity / mortality

% of projects responding to survey (n=27) reporting achievements against indicators

Left: snapshot of categories of questions

forming the basis of a standardised set of

medical research impact evaluation criteria,

currently being trialled in MSF via a pilot

of Researchfish – a system adopted by a

majority of UK medical research funders.

Attempts to map the long-

term impact of MSF’s

research must use

a comprehensive series

of indicators.

Both quantitative (metric)

and qualitative (narrative)

information is needed to

build a complete picture

of MSF’s research

achievements.

Efforts are underway

to evaluate whether a

standardised series

of outcome measures

can be used to document

MSF’s research impact

prospectively across

a cohort of MSF

researchers.

Advocacy

Impact? Publication

Operations

Policy

*approved by MSF/

national ethical review

boards in 2009

Dissemination

• Publication

• Interviews/workshops

• Presentations/media

Policy/practice

• Rapid advice/circular

• Guidelines

• Training materials

n=31*

projects principal

investigators

emailed survey;

self-reporting

purposively selected;

*at end April 2013

n=10*

interviewees

principal

investigators

telephoned;

semi-structured

interview

interviews transcribed;

themes identified through

an iterative and

deductive process