Dr Steven Hill, Head of Research, Higher Education Funding Council for England
-
Upload
openforumevents -
Category
Science
-
view
272 -
download
0
Transcript of Dr Steven Hill, Head of Research, Higher Education Funding Council for England
Understanding the impact of the Research Excellence Framework
Steven Hill
Head of Research Policy
Science Summit 2014
03 December 2014
• Introduction to the Research Excellence Framework (REF)
• Results publication and analysis
• Analysis of REF submissions
• Evaluation of the REF
• Future REF
Summary
• Introduction to the Research Excellence Framework (REF)
• Results publication and analysis
• Analysis of REF submissions
• Evaluation of the REF
• Future REF
Summary
Overall process
• Universities makes submissions
• Submissions in 36 discipline-based Units of
Assessment
• Universities select which eligible staff to include in
submissions
• Submissions are assessed by sub-panels, one for
each Unit of Assessment
• Sub-panels are grouped into 4 main panels,
responsible for consistency
• Results provided as quality profiles
Overview:
The assessment framework
Overall quality
Outputs
Maximum of 4 outputs per researcher
Impact
Impact template and case studies
Environment
Environment data and template
65% 20% 15%
Overview:
• Introduction to the Research Excellence Framework (REF)
• Results publication and analysis
• Analysis of REF submissions
• Evaluation of the REF
• Future REF
Summary
• 18 December 2014: Results day!
• Overall quality profiles for each HEI/UoA
• And sub-profiles for outputs, impact and environment
• Analysis of results and process
• [Eligible staff numbers from HESA]
• Early January 2015
• Publication of submissions
• February 2015
• Panel and sub-panel overview reports
• REF Manager’s report
The REF results and other outcomes
• Introduction to the Research Excellence Framework (REF)
• Results publication and analysis
• Analysis of REF submissions
• Evaluation of the REF
• Future REF
Summary
Analysis of REF Impact case studies
6,975
• Work commissioned from Digital Science and Kings College London
• Two objectives:
• Publish case studies as a searchable, text-minable database
• High level synthetic analysis of case studies – what do they tell us about the impact of research?
• Published at end of March 2015
• Crucial evidence for spending review
• Database will lead to further detailed analysis
Analysis of REF Impact case studies
• What are the features of Units of Assessment that have scored very highly in the REF?
• Funding? Talent management? Strategic focus? Multi- and inter-disciplinary research?
• Does this vary by HEI type? Discipline?
• Commissioning shortly
• Publication in mid 2015
Factors leading to high performance
• Introduction to the Research Excellence Framework (REF)
• Results publication and analysis
• Analysis of REF submissions
• Evaluation of the REF
• Future REF
Summary
• Work commissioned from RAND Europe
• Phase 1 – evaluation of preparation of submissions
• Challenges, benefits and costs to HEIs
• Sample of 21 HEIs (35% of case studies)
• Phase 2 – evaluation of assessment process
• Strengths and weaknesses of the impact assessment process
• Panel members: survey, focus groups, interviews
• Publication late March 2015
Evaluation of impact element
• Work commissioned from Technopolis
• Estimate of costs to sector of the REF
• Specific focus on ‘individual staff circumstances’
• Benefits of REF to institutions
• Comparison to equivalent study following RAE2008
• [Impact element covered in impact evaluation]
• Publication late March 2015
Review of costs and benefits
• Written feedback sought in early 2014
• Responses from more than 100 HEIs
• Headline messages:
• Benchmarking, driver of quality
• Some aspects burdensome for institutions
• Positive effects of new staff circumstances measures
• Many positive views about impact (but issues with the detail)
• Publication of full analysis late March 2015
Feedback from participating HEIs
• Conferences and focus groups November 2014/January 2015
• User panel members
• Academic panel members (x2)
• Covering all aspects of REF (including contribution to impact evaluation and metrics review)
• Publication late March 2015
Feedback from panel members
• Introduction to the Research Excellence Framework (REF)
• Results publication and analysis
• Analysis of REF submissions
• Evaluation of the REF
• Future REF
Summary
• Independent panel chaired by Professor James Wilsdon
• Call for evidence – summary published
• 57% ‘sceptical’ about use of metrics
• Literature review
• Feedback from REF panel members
• Workshops
• ‘In metrics we trust?’
• ‘Metrics for all?’
• Metrics in the Arts and Humanities
• Emerging findings March 2015, publication Summer 2015
Review of metrics and research assessment
• Request from minister
• Informal consultation with
• Overseas stakeholders
• UK stakeholders (roundtable event)
• Open survey
• Advice to minister in next few weeks
International REF
• Most evaluation evidence complete March 2015
• Policy development/informal consultations Summer 2015
• Proposals and formal consultation Autumn 2015
Future REF – tentative timetable
• Introduction to the Research Excellence Framework (REF)
• Results publication and analysis
• Analysis of REF submissions
• Evaluation of the REF
• Future REF
Summary