Collection and analysis of research outputs and impactsDr Ian Viney

Medical Research Council

• Provide evidence for communicating the outcomes and impacts of medical research (economic, societal and academic)

• Quantitative and qualitative information used to develop/strengthen strategic decisions (what worked, what didn’t)

• Understand better how research leads to impact (how can we maximise the opportunities for impact?)

MRC Evaluation Programme

What evidence?

• Tracing the interactions between the producers and users of research, establishing evidenced links across as many steps as possible

• Publications may have a minor impact in transferring knowledge to industry, people exchanges and problem solving (collaboration) are more important

• Similarly focussing on hard commercialisation metrics (spin outs, patents etc.) miss a large range of activities (e.g. product development, policy influences)

• Need a holistic approach to output, challenge is to capture these “softer” processes effectively

• Failures (absence of output, closure/re-direction of projects) also have to be factored in

• Advantage to “micro-economic” approaches is that casual links are established, disadvantage is that these links are rarely “generalisable”

How to gather this evidence?

• Has to be gathered from researchers, either by employer or funder – there are currently no repositories of this data

• In the UK there has been no concerted effort to collect structured and comprehensive information about outputs and outcomes

• Need to keep administrative burden as low as possible

• Developed an online system to do this, in collaboration with researchers and research organisations in 2008

• In 2011 MRC licensed the approach to a commercial provider (Researchfish Ltd.) to provide/further develop the service

Reporting new products/interventions in development

Categorisation for products and interventions based on NCI

translational research flowcharts and US DoD technology readiness levels

Simple flow charts, unambiguous categories, allow the data to be structured and reported consistently

Automated “lookups” suggest categories to researchers – no keying in of organisation/funder names etc.

Progress to date?

• MRC dataset covers outputs realised 2006 – 2012, from 98% of our portfolio (£2.5bn expenditure across 4,500 awards, 3,500 researchers)

• “Federated” Researchfish approach has been joined by 20 funding organisations across the charitable and public sectors – these organisations invest £1.8bn each year in medical research. 6,500 researchers are now using the system and this is rapidly growing

• A further 50 UK organisations are in the process of joining, this is likely to increase the investment tracked to £3bn per year (almost all medical research funding in the UK)

• We have used the data to easily compile case studies, and to set these in the context of the progress, productivity and quality of the whole portfolio

• Researchfish now exchanges data with Europe PubMed Central, enhancing information about the links between funding and papers

Wellcome Trust Project Grants 2006Developed Professor Edward Watkins research independence.

NIHR South-West CLAHRC (2007)Provision of institutional programmatic support to support the further translation of this work into a viable treatment.Total value of Peninsula CLAHRC £10m

Basic Research

Applied Research

HealthcarePractice & Policy

1st translational gap 2nd translational gap

MRC Experimental Medicine Grant (2007)“Cognitive training as a facilitated self-help intervention for depression”Allowed for growth of research team and shift in focus from experimental research to evaluation of therapy via RCT.£464k

MRC Brain Sciences Trial Platform Grant (2005)“Preventing depression relapse in NHS practice using Mindfulness-Based Cognitive Therapy" £237k

Mood Disorders Centre (University of Exeter): Cognitive therapy for depression

OutcomesProof-of-principle evidence that training depressed individuals to think about their problems in a more concrete way reduces their symptoms of depression. Pilot work in a multiple case series has demonstrated in patients with major depression that is safe, tolerable, and acceptable. The final stages of the current MRC grant have tested out the efficacy of the intervention in a small scale controlled trial (n =121). The results of the main outcome analysis indicate that the cognitive training guided self-help treatment adds significant clinical benefit in terms of reducing depression relative to treatment-as-usual alone, and that it is of equivalent efficacy to a relaxation based guided self-help treatment in reducing depression, but superior in changing depressogenic thinking such as rumination. The project has therefore developed a novel treatment derived from experimental findings that is easily accessible, cost-effective and able to be disseminated, and could be made widely available (for example via Improving Access to Psychological Treatments IAPT initiative). This treatment material has been copyrighted. The report of the main outcome of the trial has now been prepared as a paper and submitted to a high impact peer-reviewed journal for publication.Collaborations with researchers in the Netherlands (2009) has extended this work to a new population (adolescents at risk) and to a prevention treatment (rather than an acute treatment).The work has been cited in the NICE guideline update on depression (2009)

Wellcome Trust Capital Grant to build the new Sir Henry Wellcome Mood Disorders Centre at Exeter (due to open in 2011)£3.6m

Health Technology Assessment (2010)“Preventing depressive relapse in NHS Practice through mindfulness-based cognitive therapy (MBCT)”£2.1m

Mood Discorders CentreFounded 2003 by Professor Edward Watkins and Professor Willem Kuyken

DiscoveryClinical trial

New treatment

High impact paper

Policy influence

New collaboration

Intellectual property

Overall, the total cost of health services for depression in England in 2007 was estimated to be £1.7 billion whilst lost employment increased this total to £7.5 billion. By 2020 it is expected that depression will

be the second leading cause of disability in the world, and by 2026 the condition is projected to cost the UK health service £3 billion (£12.2 billion cost to the UK with lost employment).

Pipeline analysis

One of 40 therapeutic behavioural interventions in development10 of which are in early “clinical assessment”3 of which have relevance to treating depression

Prospective tracking provides information on the speed of translation into practice

MRC use of Researchfish data

• spending review submissions to Government (2010, 2013), medical research treated positively

• published and returned data each year to Universities and MRC Units, some of which are ingesting this into their research information systems

• used to adjust/make the case for new funding initiatives (e.g. a further £20m per year investment in experimental medicine - funding underway)

• 2012 quinquennial review of all three MRC Institutes included bibliometric data, providing an international comparison with other research organisations.

• The UK Government has provided additional science funding (e.g. £90m of the £180m biomedical catalyst fund announced in 2011) referring directly to the MRC data.

• MRC has monitored progress against the objectives set out in its Strategic Plan (Research Changes Lives) using outcomes compiled from Researchfish data. These convey good progress since 2009, http://www.rclprogress.mrc.ac.uk/

Challenges

• Capturing information on outcomes in an efficient way, minimise the burden on researchers (providing a system that supports easy data entry – engage experts)

• Maintaining the support of the research community (good communications, regular feedback, build on success), allow re-use of the data e.g. in CVs and grant applications

• Extend the approach to other disciplines (STFC – particle physics community now using Researchfish)

• Extend the approach to acknowledge and involve other research team members (“research teams” functionality now live). Collaborators now enter output once and share it. Outputs are uniquely identified.

Where next?

• We aimed straight for the information we most wanted – not workforce statistics, but what our research investment was delivering

• Project “Snowball” in the UK has a critical mass of leading Universities sharing data on research performance (research income, staffing etc.) aiming to drive quality and efficiency (www.snowballmetrics.com/)

• Lots of scope for improving the technology – e.g. links with clinical trials, patent, and other datasets are being developed

• There are no geographical limits on the use of Researchfish, a single portal for capturing output information means the data is comparable across funders (multi-language versions could be developed, exchange rates already addressed)

• CERIF is gaining support as an open European standard for the exchange of information about research, some Researchfish outputs are mapped into CERIF – this may assist in the sharing of data between separate systems

European Research Area with a joined up view of scientific progress, productivity and quality?

• UK – 1 member state

• £26bn R&D investment each year, £8.1bn health relevant research, £3.3bn medical research supported by public/charitable sectors

• More than 150 funding organisations, similar number of research performing organisations

• MRC funded groups alone leverage £300m from outside the UK each year , collaborate with 95 countries, and report 1000 or so new interactions every year

• Strong argument for a single approach to defining outputs, and if possible a single system for collection of this data - rather than resource-intensive mapping of data from one system to another, even via a standard such as CERIF

Thank you

• Email – evaluation@headoffice.mrc.ac.uk

• Web - http://www.mrc.ac.uk/Achievementsimpact/Outputsoutcomes/e-Val/index.htm

• Twitter – @MRCeVal