Impact workshop

Impact workshop

Phil HannafordVP Research and Knowledge Exchange

www.abdn.ac.uk

Agenda• Introduction• Overview• HEFC Case Study Template• Impact case studies- examples of best

practice• Next steps

Overview

Purpose of the REF

The REF replaces the RAE as the UK-wide framework for assessing research in all disciplines. Its purpose is:

• To inform research funding allocations by the four UK HE funding bodies (approximately £2 billion per year)

• Provide accountability for public funding of research and demonstrate its benefits

• To provide benchmarks and reputational yardsticks

Key changes since the 2008 RAE • Inclusion of assessment of impact

• Fewer UOAs/panels, operating more consistently

• Strengthened equality and diversity measures

• Revised eligibility criteria for staff

• Addition of (limited) use of citation data in some UOAs

• Removal of ‘esteem’ as a distinct element

• Revised approach to ‘environment’ and data collection

• Increased ‘user’ input on panels; and an integrated role for additional assessors

• Publication of overall quality profiles in 1% steps

The assessment framework: Overview

Overall quality

Outputs

Maximum of 4 outputs per researcher

Impact

Impact template and case studies

Environment

Environment data and template

65% 20% 15%

Timetable

2011•Panels appointed (Feb)

•Guidance on submissions (Jul)

•Draft panel criteria for consultation (Jul)

•Close of consultation (5 Oct)

2012•Final panel criteria and

methods (Jan)

•HEIs submit codes of practice (final deadline Jul)

•Requests for multiple submissions (final deadline Dec)

•Survey of submission intentions complete (Dec)

2013•Launch REF

submissions system (Jan)

•Recruit additional assessors

•Staff census date (31 Oct)

•Submissions deadline (29 Nov)

2014•Panels assess

submissions

•Publish outcomes (Dec)

www.abdn.ac.uk

Impact- the big unknown

Impact: Submissions

Impact template (REF3a)

• Sets out the submitted unit’s general approach to enabling impact from its research

• One template per submission – with a page limit depending on the number of staff submitted

• Covers the period 1 Jan 2008 to 31 Jul 2013

• Contributes 20% to the impact sub-profile

Case studies (REF3b)

• Specific examples of impacts already achieved, that were underpinned by the submitted unit’s excellent research conducted between 1 Jan 1993 to 31 Dec 2013

• Number required depends on the number of staff submitted (2 up to 14.99 FTE, plus 1 for every extra 10 FTEs (or part thereof))

• Impacts during 1 Jan 2008 to 31 Jul 2013; underpinned by research since 1 Jan 1993

• Contributes 80% to the impact sub-profile

Impact: Definition for the REF (1)• An effect on, change or benefit to the economy,

society, culture, public policy or services, health, the environment or quality of life, beyond academia

• Impact includes an effect, change or benefit to:- The activity, attitude, awareness, behaviour,

capacity, opportunity, performance, policy, practice, process or understanding

- Of an audience, beneficiary, community, constituency, organisation or individuals

- In any geographic location whether locally, regionally, nationally or internationally

Impact: Definition for the REF (2)• Impact includes reduction or prevention of harm, risk,

cost or other negative effects

• It excludes impacts on research or the advancement of academic knowledge within HE; and impacts on teaching or other activities within the submitting HEI

• Other impacts within the HE sector, including teaching or students, are included where they extend significantly beyond the submitting HEI

www.abdn.ac.uk

Impacts: Definition for the REF

Impact: CriteriaThe criteria for assessing impacts are reach and significance*

Four star Outstanding impacts in terms of their reach and significance

Three star Very considerable impacts in terms of their reach and significance

Two star Considerable impacts in terms of their reach and significance

One star Recognised but modest impacts in terms of their reach and significance

UnclassifiedThe impact is of little or no reach and significance; or the impact was not eligible; or the impact was not underpinned by excellent research produced by the submitted unit

* Each main panel provides a descriptive account of the criteria

www.abdn.ac.uk

Reach – How widely felt it was

Significance – How much difference it made to beneficiaries

Impact: Template (REF3a)• The unit’s approach to enabling impact from its research:

- Context for the approach- The unit’s approach during 2008-2013- Strategy and plans for supporting impact- Relationship to the submitted case studies

• Provides additional information and context for the case studies, and can take account of particular circumstances that may have constrained a unit’s selection of case studies

• To be assessed in terms of the extent to which the unit’s approach is conducive to achieving impact of ‘reach and significance’

Impact: Case studies (REF3b) • In each case study, the impact described must:

- Meet the REF definition of impact

- Have occurred between 1 Jan 2008 and 31 July 2013 (can be at any stage of maturity)

- Be underpinned by excellent research (of at least 2* quality) produced by the submitting unit between 1 January 1993 to 31 December 2013

• Submitted case studies need not be representative of activity across the unit: pick the strongest examples

NB. Its not about the esteem or influence of an individual or unit

Impact: Case studies (REF3b)Each case study is limited to 4 pages and must:

- Describe the underpinning research produced by the submitting unit

- Reference one or more key outputs and provide evidence of the quality of the research

- Explain how the research made a ‘material and distinct’ contribution to the impact (there are many ways in which this may have taken place)

- Explain and provide evidence of the nature and extent of the impact: Who/what was affected? How were they affected? When?

- Provide independent sources that could be used to verify claims about the impact (on a sample audit basis)

www.abdn.ac.uk

• Impacts may be at any stage of development or maturity

• Impacts stay with the institution (unlike publications)- so cannot ‘buy-in’ impacts

• Can be shared with other insitutions (each has to show their disticntice contribution to the imapct)

• Impacts must have taken place during the assessment period (not future or potential impacts)

• Impacts or benefits arising from engaging the public with the submitted unit’s research will be eligible (but not dissemination activity unless there is evidence of its benefits)

• Impacts arising from public engagement must show that that the engagement activity was at least in part based on the submitted unit’s research AND drew materially and distinctly upon it

Impact: other key points

www.abdn.ac.uk

• Clinical Medicine (17 case studies) + Earth Systems (4 case studies)

• Writing team• Steering Group (SG) of senior academics chaired by VP R&E• Process was:

− Trawl for stories− Interviewed researchers

Glasgow’s pilot experience

www.abdn.ac.uk

Labour intensive for the staff

involved

− collection and collation of the material we need to submit

− the iterative nature of the drafting process


www.abdn.ac.uk

External supportive evidence• Challenging to engage external

contacts:

- who?

- ‘Sorry they left a few months ago...’

- S – t – r – e – t – c – h – i – n – g Goodwill!


www.abdn.ac.uk

External supportive evidence• Challenging to engage external

contacts:

- who?

- ‘Sorry they left a few months ago...’

- S – t – r – e – t – c – h – i – n – g Goodwill!


NB. We need to think when to engage with external supporters

www.abdn.ac.uk


Possible types of evidence• Testimonials from named individuals

• Press coverage

• Guidelines/Documents/Reports

• Training materials

• Details of conference/invitations to speak

• Links to relevant background information

• Public engagement – speak to the ‘organiser’ of the event

• Details of grants

• Publications – highlighted where peer-reviewe

www.abdn.ac.uk

Glasgow’s pilot experienceAvoid subjectivity

• Striking the balance

• Once upon a time...

www.abdn.ac.uk

Glasgow’s pilot experienceReading group

• External reading team from user community

• Clinical medicine UoA

• Users and academics

• Extremely helpful

• Internal reading group

www.abdn.ac.uk


Selecting our case studies• Pipeline – extra stories

• Look into the past

• Consider ‘reach’ and ‘significance’

• Pilot panel reports

• Refer to panel criteria

• Furthest along the pathway

• Believable

• Choose your strongest

www.abdn.ac.uk


Our results• Very pleased with the overall result – there were no surprises

Managed to avoid:• Generalised, vague claims• Excessive publication lists or web references • Lack of coherence• Claiming potential impact• Lack of necessary information

www.abdn.ac.uk


Top four tips...

• Watch the template limits

• Make it easy for the reader

• So what?

• Start with the impact

www.abdn.ac.uk

Glasgow’s preparations

• Mini-REF - best examples of impact

• UoA “Pipelines” created (Colleges/R&E)

• Prioritising pipelines (UoA Champions/R&E)

• Developing case studies with academics

• Horizon scanning

www.abdn.ac.uk

Useful resources

• HEFCE REF2014 sitewww.hefce.ac.uk/research/ref/

• Pilot panel reports and best practice examples www.hefce.ac.uk/research/ref/impact/

• HEFCE REF FAQswww.hefce.ac.uk/research/ref/faq/

• Public Engagement – NCCPE Materialswww.publicengagement.ac.uk/how-we-help/event-reports/ref-workshop

• Your colleagues (in Aberdeen and outside)

www.abdn.ac.uk

Read the guidance !!!!!

• Assessment framework and guidance on submissions

• Panel criteria and working methods

Case study template (REF 3b)

Examples of good practice

Preparing an impact case study

www.abdn.ac.uk

Preparing an impact case study

Suggested questions to help clarify ‘impact’:

•What user groups outwith academia did you work with?

•What was the purpose of the ‘interaction’?

•What has been the effect on the users/audience?

•Did it change something for them?

•How did they benefit from the ‘interaction’?

•How did your contribution effect the impact/benefit?

Impact workshop

Documents

Transcript of Impact workshop