Assessing Research Impact

Dr Ian McMahon

Definition for the Research Excellence Framework [UK] An effect on, change or benefit to the economy, society, culture,

public policy or services, health, the environment or quality of life, beyond academia

Includes an effect, change or benefit to: The activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding

Of an audience, beneficiary, community, constituency, organisation or individuals

In any geographic location whether locally, regionally, nationally or internationally

Excludes impacts on research or the advancement of academic knowledge within HE; and impacts on teaching or other activities within the submitting HEI

What is Impact?

Economic Social Public Policy and services Health Cultural Quality of Life Environment

Types of Impact

Advocacy

Accountability to taxpayers

Funding allocation

Why Assess Impact?

Demonstrate the importance of university research

increasing productivity and wellbeing engaging effectively with current and future

national and global challenges.   focus on showcasing or measuring the societal

benefits from research,

Advocacy

Provide assurance to Government and general public of the return from their investment in research  

Increasingly important in a tight fiscal government environment where returns on investment in research will need to be demonstrated in terms of environmental, economic and social impact.

Accountability to taxpayers

UK Research Evaluation Fund

Australia possibility of funding some research block grants

Funding Allocation

Concerns

Measuring impact is not easy Attribution or contribution Time lags between research and impact Research user engagement Transaction costs of impact assessment That some impact is negative That all research must have impact That impact is only economic Demonstrating impact in the arts and humanities

That impact cannot be ‘measured’ That the requirement to demonstrate it distorts

academic behaviour That it takes time for impact to happen That the expectation of impact is a threat to

academic freedom That impact will become an industry That measures will become targets

Concerns

Research Quality Framework preparation in Australia 2005 - 2007

Excellence in Research for Australia 2010, 2012 Impact and UK REF – pilot in 2010, full REF in 2014 RAND Corporation: Capturing Research Impacts –

A review of international practice. (for HEFCE) OECD projects – 5% funding to demonstrate

impacts Economic Impact of Research & Commercialisation

at Leiden University.

Impact Context

Research Excellence Framework (REF) 2014 requires impact case studies (1 per every 10 academics submitted) and an impact strategy for each unit of assessment submitted. Impact judged on significance and reach. http://www.ref.ac.uk

REF will be used at allocate UK Quality-related research block grant funding.

Research Councils UK (including Arts and Humanities Research Council) requires applicants to submit a ‘pathways to impact’ statement

UK RESEARCH AND IMPACT

Australian Government 2011 Focusing Australia’s Publicly Funded Research Review recommended a feasibility study on possible approaches for assessing the wider benefits arising from publicly funded research.

2013 New Government with greater emphasis on research with immediate benefit

Australian Context

Options for university-based research impact assessment To provide evidence for public investment in research To increase incentives for universities/end-user engagement improve understanding of pathways between research and

innovation to inform policy.

Findings: feasible to introduce two new research impact mechanisms1. University assessment based on case studies and metric

data, 2. integrating data from government, research agencies and

universities.

AUSTRALIAN GOVERNMENT FEASIBILITY STUDY

Research skills for an innovative future

Smarter Manufacturing for a smarter Australia

Review of Collaborations between the Public and Private Sectors

Review of Higher Education Access and Outcomes for Aboriginal and Torres Strait Islander People

The Place of Science in Policy Development in the Public Service

AUSTRALIAN REPORTS

Excellence in Innovation for Australia The Conversation: public engagement metrics IP Australia: patent-based metrics University of Melbourne: STAR metrics Higher Education Funding Council of England:

Research Excellence Framework Research Councils UK: pathways to impact. UNSW: income from Technology Transfer Cooperative Research Centres: Impact Study CSIRO: Impact 2020

RELATED STUDIES

Research impact is the demonstrable contribution that research makes to the economy, society, culture, national security, public policy or services, health, the environment, or quality of life, beyond contributions to academia.

Impact Measurement and operational principles. Respect the diversity in research

disciplines/sectors in demonstrating research impact.

Australian Research Council Definitions and Principles

There are compelling stories that need to be told of research impact arising from research at Australian universities.

• EIA arose out of a commitment shared by the ATN and the Go8 – in the national interest - to produce research that is both excellent in and of itself and also benefits the broader Australian community.

• The excellence of research produced by Australian universities has been assessed by the Excellence in Research for Australia (ERA)

• EIA seen as being potentially complementary to ERA in assessing the impact of research

ATN – Go8 Excellence in Innovation for Australia (EIA) Trial

A trial assessing the impact of university research by case studies

Conducted over 2nd half of 2012 – finished yesterday 12 universities, 162 case studies across Defence, Economic

Development, Society, and Environment 7 assessment panels – 70% non-academic membership Main conclusions:

Can assess impact across a broad range of disciplines using case studies

Universities produce high-impact research

The EIA in a nutshell

RQF preparation in Australia 2005 - 2007 ERA: Trial 2009, v1.0 2010, v2.0 2012 Recognise work in UK REF – pilot in 2010, full REF

in 2014 RAND Corporation: Capturing Research Impacts –

A review of international practice. (for HEFCE) OECD projects – 5% funding to demonstrate

impacts DIISRTE feasibility study into assessment of impact ATN/Go8 Symposium in November, 2011

Context of the Trial

12 universities ATN: UniSA, Curtin, UTS, RMIT, QUT Go8: UQ, UNSW, Melbourne, UWA Newcastle, UTas, CDU

VCs, DVCs-R, Research Directors + many more Advisory Board: Phil Clark, Robin Batterham, Patricia Kelly,

Laurie Hammond, Craig Roy, David Sweeney + uni representatives

75 assessment panellists – 53 non-academics including from Rio Tinto, Chevron, IBM, Microsoft, Salvation Army, CSIRO, Deloitte, KPMG, …

Australian Government – supporting assessment panel process

Lots of researchers

Who was involved

1. That the development of the EIA should look to leverage the work performed in developing the UK REF where appropriate

2. That the EIA examine retrospective impact rather than prospective impact

3. That there be no explicit excellence threshold for submissions, but the impact should show a link to underpinning research

4. That the EIA take a verifiable approach to evidence in submissions

5. That impact will be primarily assessed by means of case-studies for all disciplines.

6. That research underpinning a submitted impact should be attributed to the university (ies) at which the research was generated

General principles of the EIA

Four broad sectors, Defence, Economic Development, Society, Environment

Time period – impacts from 1st January 2007, underpinning research from 1st January, 1992

Unit of evaluation – the impact itself, not discipline, academic unit or individual

A maximum of five case studies for each of the four Sectors (i.e. 20 in total)

A minimum of two case studies in three of the four Sectors (i.e. 6 in total)

Guidelines for the Trial

Case Studies on prescribed templates ≤ 15 pages Information sought on

Context Details of the impact Underpinning research and research publications –

submissions to draw the link to research quality Income and other relevant metrics Validation material available People involved in research and impact

The mechanics of submissions

• Each case study rated A-E based on reach & significance of impact• A = outstanding impact• B = very considerable impact• C = considerable impact• D = recognised impact• E = limited impact

• 7 assessment panels based on broad SEO code categorieso 1 x Defenceo 2 x Economic Developmento 2 x Societyo 2 x Environment

The assessment of submissions

Submissions

EIA outcomes

1. There are compelling stories that need to be told of research impact arising from research at Australian universities.

2. Using a case study methodology with SEO codes to assess research impact is applicable as a way forward to a national assessment of research impact – with provisos

3. The use of expert Panels for the impact assessment process, with significant input from representative external panel members, is applicable for a national impact assessment process.

4. The range of impact ratings was wider than anticipated.5. Many case studies put forward cases based on prospective

impact rather than demonstrated impact

The 9 Key Learnings of the EIA

The 9 Key Learnings of the EIA

6. The link between impact and good quality research was able to be identified.

7. It was appropriate to assess impact using the industry-based grey literature.

8. Panel Chairs all commented on the time involved assessing case studies

9. Universities reported challenges in putting together case studies.

• High quality research carried out in Australian universities has had enormous benefits for the health, security, prosperity, cultural and environmental wellbeing of Australia, the region and the world.

• Approximately 87% of the case-study assessments demonstrated at least ‘considerable’ impact.

• The case-study method adopted in the Trial enables these benefits to be communicated to governments, business, and the community.

• The combination of end-user assessment informed by expert advice has made it possible to verify that the underpinning research has contributed to the outcomes.

What the EIA has demonstrated

1. It is possible to assess research impact across a broad range of disciplines.

2. The case study approach can provide a compelling narrative of the impact of research.

3. Research impact could be assessed against an outcomes based system of classification such as the ABS SEO codes – with provisos

4. Expert Panels comprising a majority of end-user stakeholders are able to assess research impact.

5. Development of an impact component of any broader research assessment exercise would require further consideration of the number of case studies to be submitted.

Conclusions of the EIA

Further information including final report

http://www.atn.edu.au/eia/index.htmwww.go8.edu.au/eia

• Review of Trial documentation• A critical review of 10 case studies• An assessment of Panel scoring• Surveys of ATN universities

Findings• A success but could be improved• Consider “contribution” as an assessment criteria• A need to create a culture of impact• Scale up issues• Amount of money needed for national assessment• Liked concise documentation, SEO Codes• Engage with non-Trial universities

Review of EIA BY RAND (Europe)

Review of EIA BY RAND (Europe)

Positive outcomes from Trial (researchers’ perspective)

• Understanding the impact/effect of their research• Ability to adapt material for other purposes• Appreciating need to collect ongoing evidence• Consolidating networks through re-establishing contacts with clients and end-users

discussion paper on assessing the wider benefits arising

from university-based research released June 2013. The National Research Investment Plan actions to

maximise public benefits that arise from publicly funded

research Case studies vs metrics Some universities starting Impact stocktakes New Government Would it result in extra funding or help maintain existing

funding?

Future of Impact Assessment in Australia

Ian McMahon

Director, Research Group of Eight

ian.mcmahon@go8.edu.au

Thank You