RQA/RAE/REF: A critical chronology of UK exercises and some comparisons with approaches elsewhere

Ian McNayProfessor Emeritus, Higher Education and Management

University of Greenwich23 October 2012

i.mcnay@gre.ac.uk

Structure of content

• Chronology 1986 to 2014

• Comparisons with elsewhere

• Criteria and critique

Chronology of the UK RAE

1986 [evaluative/productive, baseline]First exercise to make assessment, and allocation of funds more transparent – UGC universities only. Part of the Thatcher manta of ‘value for money’ and the distrust of professionals, particulalrly their self-regulatory autonomy. Small impact on funding. Opposition on the basis of poor methodology

1989 [evaluative/productive, scaled]Second exercise. Still UGC universities only. Formal peer review panels. Publications as evidence. Criticism, particularly over poor information and disorganised process. First major reduction in funding for some, and abandoning of myth that all academics do research for 30-40% of their time.

Continued

1992 [evaluative/qualitative]All HEIs included [at very late notice for the ‘newcomers’] after abolition of the binary line. No extra funds provided, so some [9%] transferred to the newcomers. Not popular amongst the ‘establishment’.5 point scale; 2783 submissions; 15% [423] grade 1 [unfunded]; 13% [350] grade 5. Funding linked to grade on linear scaleI did a major evaluation of impact, funded by HEFCE. Published 1997 – very quietly, since they did not like the findings, and without any researcher interpretation.

Continued

1996 [evaluative competitive/selective]7 point scale; bottom three not funded.2894 submissions; 8% [236] grade 1; 403 [14%] grade 5; 170[6%] grade 5* [first use of this to discriminate at the top: compare A* at A level]Concept of ‘research active’ academic used to decide who to include [or not] in submissions. Transfer market develops for ‘stars’ [a bad investment]. A lot of game playing – see Lisa Lucas’s work.

Continued

2001 [evaluative/strategic]Research ‘concentration’ as an objective. Grade: funding ratio adjusted to favour higher grades. Grade 3a, and, later, grade 4, unfunded. So improvement of lower/unfunded units not rewarded.2598 submissions [considerable ‘churning’ continues]. 18 at grade 1, 140 at 2, 278 at 3b, 499 at 3a = 36% of units, 20.5%of staff. 4* - 26% units; 25% staff; 5 – 28/36%; 5* 11/18%. So, bigger units got higher grades.Criticism continued and low ‘enterprise’ research income of higher rated units noted; correlation of panel representation with higher grades; exclusion of staff from modern universities from areas of major activity – business, education. Criticism grows louder over lower recognition of practice related work and professional based research.

Continued

2008 ‘excellence wherever it is demonstrated’Shift to a unit ‘profile’ of excellence to reveal high quality work across a wide range of units with previous low average grades. 150/159 institutions had recordable levels of ‘world-leading’ work [grade4*]. More pressure from the elitist group and proposals for ‘critical mass’ as a criterion linked to ‘research culture’ and exclusion of the ‘plebs’ from project bidding to research councils and funding of PhD students.2363 submissions: 17% at 4*; 37% at 3*; 33% at 2*; 11% at 1*; 2% unclassified. Note the decline in number of submissions with re-structuring and institutions’ internal selection.Some initial funding for 2* work, cut after one year and balance of funding between 3* and 4* adjusted. Initial funding shift to modern universities re-balanced.

Grade inflation

• 1992 Grade 5 15% [units]• 1996 Grade 5 14%, Grade 5* 6% [units]• 2001 Grade 5 28%, Grade 5* 11% [units]• 2001 Grade 5 36%, Grade 5* 18% [staff]• 2008 [output, on a different scale]• Grade 4* 17%, 3* 37* [those two equating roughly to 5/5*]

2014…the new REF

The battle continues within developing plans for a Research Excellence Framework.

‘Impact’ will be a criterion. Now integrated into main quality rating, having initially been an endorsement. Will count for 20% of rating [down from 25% after pressure from the elite, who came out worse than modern universities in pilot exercises to test approaches to evaluation and evidence].Political pressure continues to designate ‘teaching only’ universities, excluded from the exercise

From informing [funding] policy to influencing researcher practice

2001 The introduction of user representatives on RAE panels

2004 Parliamentary Select Committee criticism of lack of recognition of professionally relevant and practice based research [based on an ESRC report]

2006 The Warry Report Increasing the economic impact of Research Councils, said councils should examine– Their leadership of the knowledge transfer agenda– Their role in influencing knowledge transfer behaviour of universities and RC

institutes– Increasing their engagement with user organisations

From informing [funding] policy to influencing researcher practice

2009 HEFCE second consultation on REF, p8:‘By continuing the practice adopted in the RAE of assessing bodies of activity at the level of coherent groupings, we will be able to use the REF to encourage desirable behaviours at three levels:- the behaviour of individual researchers within a submitted unit- research units, as the level at which research activity is primarily managed- whole HEIs – the level at which the block grant is allocated’

[emphasis added]

First principles

• What do you think is/are the purpose/s of Research Quality Assessment?

• What evidence of quality would you look for?

• What process of assessment would you use?

A [quick] comparative survey

Purpose:to inform funding [UK, NZ]; to enhance quality [Aus]; to promote strategic ‘fit’ [Aus, HK]; to ‘reward and retain’ [Mex]; to enhance public image [NZ]; to promote impact on policy, practice and people’s lives [Aus]

Level:Individual [Mexico, Spain, NZ]; Unit [UK, HK]; Team [Netherlands]; Programme [USA]; Institution [China, HK,Norway, Ireland]

Continued

ProcessPeer review [ UK, Netherlands, NZ]; metrics [Flanders]; matched funding [Aus]; hybrid [USA]; reputation [USA]

Linkagesto funding [not HK, Netherlands]; to teaching/learning [HK, USA-NSF, Netherlands]; to impact and application [UK, from 2014, Aus, briefly]

Compulsory NZ

Continued

Compulsory NZ

Confidential All, but…

Public reporting not HK, Pakistan

Graded [UK, NZ, Aus] or threshold [HK, Spain, Mexico]

Principles governing the RAE [HEFCE, 1999ff]

• Clarity of documentation• Consistency across academic areas• Continuity between exercises as far as possible• Credibility to those being assessed• Efficiency in the use of public funds• Neutrality over research approaches• Parity across different forms of output• Transparency

• From your experience, how would you score against those 8?

Criteria: UK 2001

5* Quality that equates to attainable levels of international excellence in more than half the research activity submitted and attainable levels of national excellence in [all] the remainder5 Quality that equates to attainable levels of international excellence in up to half of the research activity submitted and to attainable levels of national excellence in virtually all of the remainder4 Quality that equates to attainable levels of national excellence in virtually all of the research activity submitted showing some evidence of international excellence

Continued

3a Quality that equates to attainable levels of national excellence in over two-thirds of the research activity submitted, possibly showing some evidence of international excellence3b Quality that equates to attainable levels of national excellence in more than half of the research activity submitted2 Quality that equates to attainable levels of national excellence in up to half of the research activity submitted1 Quality that equates to attainable levels of national excellence in none, or virtually none, of the research activity submitted.

‘National Excellence’ in 2001

History of Art‘Making an intellectually substantial contribution to new

knowledge and understanding and/or original thought’.

History‘…Highly competent work within existing paradigms which does

not necessarily alter interpretations’.

International excellence

• Economics and econometrics, 2001 exercise‘International excellence will be interpreted within a wide international context. This will be defined by reference to high quality research activity, where it has been identified, around the world’.

Select Committee 2004‘The RAE should recognise that excellent research may not be internationally significant but it may transform the fortunes of a local business or the provision of public services. We recommend that quality criteria concentrate more on the impact of research rather than the place where it has been published’

Government response 2004‘In determining the grade quality descriptors, panels will take account of the fact that research of the highest quality may not always have an international dimension’.

International excellence 2008

Panel J4*...a primary reference point of the field3*...a major reference point that substantially advances knowledge and understanding of the

field2* a reference point that advances knowledge and understanding of the field1* output that makes a contribution to knowledge or understandingPanel K4* ...outputs at the forefront of research of international quality ...generating...new methods,

practices, theoretical frameworks, understandings...a highly significant contribution to its area

3* ...of high quality and which match the standards of internationally peer-reviewed research...a significant contribution to its area

2* ...which match the standards of international peer-reviewed research...a recognised contribution to its area

1* ...which match the standards of peer-reviewed research...a limited contribution to its area

Questions raised

Can you assess the level of ‘international recognition’ without citation data?

Compare the two 4* statements on intrinsic quality and external reputation: are they equitable?

Did things change/develop for 2014?

REF 2014 Grade criteria

• Panel A does not give grade criteria: ‘The sub-panels will look for evidence of the quality of the output in terms of its originality, significance and rigour, and will apply the generic definitions of the starred quality levels’,

• Panel B, for 4* will expect to see evidence of, or potential for, some of the following types of characteristics [do they overlap with impact assessment?]:

agenda settingleading or at the forefront of the research areagreat novelty in developing new thinking, new techniques or novel

resultsmajor influence on a research theme or fielddeveloping new paradigms or fundamental new concepts for Rmajor changes in policy or practicemajor influence on processes production and managementmajor influence on user engagement

REF 2014 grade criteria

• Panel C’s ‘types of characteristics’ list is:– Outstandingly novel in developing concepts techniques or outcomes– A primary or essential point of reference in its field or sub0-field– Major influence on the intellectual agenda of a research theme or field– Application of exceptionally rigorous research design and techniques of investigation and

analysis, and the highest standards of intellectual precision– Instantiating an exceptionally significant, multi-user data set or research resource

Panel D looks for the following across and possibly beyond its area/ field:- A primary or essential point of reference- Of profound influence- Instrumental in developing new thinking, practices, paradigms, policies or audiences- A major expansion of the range and depth of research and its application- Outstandingly novel, innovative and/or creative

REF 2014 citation data use

• Panel A: ‘…all sub-panels … will make use of citation data, where available and appropriate, as an indicator of academic significance to inform the assessment of output quality’

• Panel D: ‘The sub-panels…will neither receive nor make use of any citation or bibliometric data to inform their judgements…[They will not privilege any journal or conference rankings/lists, the perceived standing of the publisher or the medium of publication, or where the research output is published.

• Panel B: ‘Sub-panels 7, 8, 9, and 11 acknowledge that citation data are widely used and consider that it is well understood in the disciplines covered in their UOAs. They will make use of citation data…Subpanels 10, 12, 13, 14 and 15 recognise that the uneven and often sparse coverage of citation data in their disciplines would not provide fair and robust additional data to inform the assessment of output quality. They therefore will not receive nor make use of citation data or any other form of bibliometric analysis, including journal impact factors.’

• Panel C aligns with D except for Economics, which goes with A.

Evaluation study 1

• McNay, 1997, funded by HEFCE

– Document search – policy and academic– 4 focus groups of senior staff, setting an agenda for...– 30 institutional case studies on strategic responses– 2 questionnaires within 15 institutions: one to heads of

units [c150 returns], one to academic staff [c400]– Interviews with other stakeholders and an open invitation

to submit views.

Evaluation study 1 : main findings

These differed across previous binary line divide.- The pre-92 university responses stressed status issues and better

management: clearer strategies, support and resources focused on the best with transparent funding and more accountability.

- For ex-PCFC staff the gains were of resources, even if marginal, the emergence of research as a legitimate, valued activity, the recognition of achievement and development of embryonic research career routes.

- One contrast was on T+R impact. 73% of HoD respondents in traditional universities saw the impact on teaching as negative and more so among the higher graded; 56% in modern universities and colleges saw it as positive. The figures for positive research impact were 67% and 81%.

Evaluation study 1: more findings

Nearly all HoDs agreed that ‘our research is now more effectively organised’ - 77/95% - and ‘the quality of our research is higher than 5 years ago – 72/95%. Interestingly, fewer staff agreed about the improved quality of their own published work – 64%; and about research being better managed and supported – 56/72%. Only 34% agreed that ‘the RAE has improved the quality of research conducted in HE.Only 32% agreed that the RAE has made me feel more valued, and that gain came at a cost – 70% said that research work encroached more on their private/personal time and 65% said it had increased their stress levels. A later McNay study [2009] found that stress levels were highest in middle ranked units and lower among the ‘confident’ and the ‘carefree’.

Evaluation study 2

• Oancea, 2010, funded by BERA/UCET

- focus on Education- structured sample of 30 departments [out of 82 submitting in 2008]- electronic survey of staff: 131 responses [20%]- interviews [31] within 9 other departments: staff at all levels, including administrators and some at university level. 28 f2f, 3 by phone.

Evaluation 2: findings

• Key issue was ‘positioning’ within the discipline and the institution, with impact on staffing, structure and strategic direction.– ‘The RAE [passim, not just the 2008 exercise] was seen as having brought to the surface

underlying tensions in academic work in education, as well as structural mismatches between existent career paths in education departments and existent rewards and resource allocation structures’.

Positive impacts for individuals included1. Being supported to work in a stronger research environment2. More, and more focused publications, because of pressure3. Better career prospects and achievements4. Enhanced motivation, confidence and identity

Evaluation 2: more findings

• Negative impacts included– 1. Work climate: excessive pressure, unreasonable workloads,

competition, not collaboration, rivalry not collegiality, conformity not creativity, short-termism, game playing

– 2. Lower commitment to teaching, pastoral work, student support, supervision, curriculum development, CPD, user engagement, administration, scholarship

– 3. Staff morale– 4. Quality, focus and breadth of research and of publications– 5. Career development

Evaluation 2: degrees of agreement on key statements

The RAE [% agree/disagree]- Was based on fair judgements of all types of education research 28/30- Stimulated better quality education research 26/35- Adequately supported ed. res. in all countries of the UK 8/24- Led to fair distribution of research funds among institutions 14/46- Treated fairly all categories of education researchers 22/41- Led to improvements in teaching in the submitting institutions 4/53- Enabled positive work climates in education departments 8/59- Enabled stronger impact of education research 16/35

Continuing issues

• Control of the research agenda – the Haldane Principle• Retrospective judgement vs future development: QA vs QE• Historic haloes• Efficiency of concentration of funds• Research as a discrete [isolated?] function• Star-struck delusions and ‘outsourcing’• National interests vs international league tables• Peer review vs citation indices• Effect on teaching and other ‘academic citizen’ activities

Conclusion

On a model plotting ends and means, successive exercises have had somewhat occluded ends, changing in a chronological progression from a self-referential closed community approach [with inherent tribal conflicts] to a clearly economic model where the ends are set outside the research community.For the first, the means to the end was seen as peer review [with some arguments about balance across qualitative/quantitative methods, and ambiguity and vagueness about criteria and grade statements]; for the second, outcomes through impact become more important than outputs, with emphasis on ‘useful knowledge’ and application beyond the academy.