Excellence in Research for Australia 2015Associate Law Deans National Meeting

The University of Western Australia November 2015

Sarah HowardDirector, Research EvaluationAustralian Research Council

Introduction

• ERA 2015—where are we up to?• ERA 2015 National Report• ERA 2015 methodology• Peer review in ERA• ERA 2015 outcomes and data

ERA 2015—where are we up to?

• Currently finalising evaluation• Preparing State of Australian

University Research: ERA National Report 2015-16

ERA 2015 National Report

WHAT?• State of Australian University Research: Volume 1

ERA National Report 2015–16- Includes additional analysis of 10 year

longitudinal dataset (2003–2013)- ERA 2010, ERA 2012, ERA 2015 comparisons

• PLUS—additional volumes to come… applied research, gender, open access

WHEN?• Early December….

ERA 2015 methodology

ERA Documentation

• Range of information about ERA development and ERA 2015 processes

• ERA 2015 documentation includes:- Submission Guidelines- Discipline Matrix- Notifications and FAQs- 2015 REC members

• Background information includes:- National Reports- ERA development documentation

• www.arc.gov.au > ERA

Useful Terminology to start …

• System to Evaluate Excellence of Research (SEER)—ERA IT system

• Research Evaluation Committee (REC)—committee of experts undertaking evaluations

• ANZSRC—Australia and New Zealand Standard Research Classification

• Field of Research (FoR)—ANZSRC research classification (e.g. 2101 Archaeology)

• Unit of Evaluation (UoE)—ERA unit of assessment for an institution by FoR (e.g. ANU 2101)

• Dashboard—suite of ERA indicators

ERA 2015

The ERA objectives are unchanged:1. Establish an evaluation framework that gives government,

industry, business and the wider community assurance of the excellence of research conducted in Australian higher education institutions.

2. Provide a national stock-take of discipline level areas of research strength and areas where there is opportunity for development in Australian higher education institutions.

3. Identify excellence across the full spectrum of research performance.

4. Identify emerging research areas and opportunities for further development and.

5. Allow for comparisons of research in Australia, nationally and internationally, for all discipline areas.

ERA 2015 Reference Periods

Staff census date: 31 March 2014

Data type Reference period YearsResearch outputs 1 Jan 2008–31 Dec 2013 6

Research income 1 Jan 2011–31 Dec 2013 3

Applied measures 1 Jan 2011–31 Dec 2013 3

Esteem measures 1 Jan 2011–31 Dec 2013 3

  Discipline Matrix ERA 2015 Peer Review (30% of outputs to be nominated by output type)

HERDC Research Income

(Categories 1-4)

Esteem Measures

FoRCode FoR Title BooksBook

ChaptersJournal articles

Conference Publications

Non-traditional outputs by type

Editor Prestigious Works of

Reference

Membership of learned academy

Category 1 research

fellowships Other NTRO

Research Report for External Body

18 Law and Legal Studies Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes

1801 Law Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes

1802 Maori Law Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes

1899Other Law and Legal Studies Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes

ERA 2015 Discipline Matrix

Evaluation Overview

Note - There are no weightings

Research Evaluation Committees

Citation Analysis or Peer Review

Volume and Activity Research Income

Applied Measures Esteem

UoE rating

by FoR

Access to Repositories for Peer Review Disciplines

Peer Reviewers

Stages of EvaluationStage 1REC Members evaluate assigned material and record preliminary evaluations in SEER

Stage 2AREC Members evaluate assigned material and record preliminary evaluations in SEER

Stage 2BREC Members are given access to the preliminary evaluation outcomes for 2 digit UoEs from other REC members and Peer Reviewers’ evaluations for moderation

Stage 2CREC Members are given view only to access to moderated 2 & 4 digit evaluations to prepare for Stage 3 meeting

Stage 3REC Members meet to finalise all UoEs

Stage 1 ERA Peer Reviewers evaluate assigned material and record evaluations in SEER

REC

Mem

ber

Peer

Rev

iew

er

ERA 2015 REC Member and Peer Reviewer Recruitment

Research Evaluation Committee (REC) Membership• 8 REC Chairs announced in November—to ensure ERA

evaluations are of the highest standard• Full list of RECs members can be found on the ARC website

Peer Reviewers• The call for peer reviewer nominations closed on

26 November 2014• ~ 1300 peer reviewers—even better than 2012! • ~ 1100 of these were assigned material for evaluation• ~ 1000 peer reviewers submitted reports

Recruitment of REC members

• Universities invited to nominate REC members in mid–2014

• Maximum of 50 nominations per university—10% of nominations from other countries

• Nominations against published criteria:- research excellence as well as a sound understanding of the

importance of research- broad discipline expertise (interdisciplinary experience is

desirable)- professional and academic standing- interpersonal and team skills

• Database of 700 researchers with broad range of expertise

• Nominees not selected were invited to participate as peer reviewers (in the ERA peer review disciplines)

Recruitment of Peer Reviewers

• Peer review is conducted by REC members in concert with external peer reviewers who are appointed to - extend the disciplinary expertise of the RECs and to - share the workload

• Peer reviewers for ERA 2015 were required for the following 2 digit FoRs: 08, 12–16, and 18–22 (as well as 0101, 1005 and 1006)

• All universities were invited to nominate peer reviewers (national and international)

• Peer reviewers are nominated by six-digit expertise to facilitate assignment

Peer Review in ERA 2015—Overview

Universities nominate a ‘representative’ sample 30% of total outputs for each assessable unit (e.g. ANU1801)

The sample is lodged in a digital repository and made available to ERA reviewers through SEER

REC members conduct peer review of the nominated sample

REC members also assign from a pool of peer reviewers to extend the expertise available and assist with workload

Peer reviewers use a structured report form The final rating = all reviewer assessments +

dashboard information

Peer Reviewer Assignment

• Every assessable unit is assigned three REC members who conduct peer review

• Every assessable unit is also assigned a minimum of two peer reviewers

• Assignment to peer reviewers is based on:- target maximum of 50 outputs for each reviewer- target maximum of 6 units for each reviewer- no reviewer should have only one assigned unit*- no unit has only one peer reviewer- notionally enough reviewers are assigned to mean all outputs in

the unit will be read by at least one reviewer

*This means that peer reviewers have a target of no more than 25 outputs in any one unit. In a unit with 200 outputs a minimum of 8 reviewers have to be assigned.

Peer Review Criteria

Criterion—Approach

• What contextual information should the RECs know about the common practice in the discipline and the sample of outputs reviewed?

• Are the methodologies clearly reported?

• Are the methodologies appropriate for the research questions or not? If not, what are limitations/effects?

• Are the venues/outlets appropriate for reporting the research? Again, there might be reference to common practice for the discipline.

• What is the quality of the venues/outlets?Criterion—Contribution

• Contribution of the group of outputs reviewed to the field and/or practice – nationally and/or internationally

• Originality and degree of innovation in the research approach or research questions

• Theoretical and conceptual rigour

• Depth of discussion and analysis

• Contribution to the further development of knowledge and understanding

ERA 2015 Peer Reviewer Feedback

Survey of Peer Reviewers

• Peer reviewers invited to complete an online survey after they had finished their assessments

• 660 of approximately 1300 reviewers returned completed survey

• Peer reviewers were also invited to submit comment on any aspect of their engagement in ERA 2015

• Most additional comment was already captured in the structured questionnaire and so there were no additional common themes

Were the instruction were clear?

Disagree

Neutral

Agree

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

Was the assigned workload manageable?

Disagree

Neutral

Agree

0% 10% 20% 30% 40% 50% 60% 70%

Main issues raised by Peer Reviewers

• Workload

• Allow supplementary metrics (e.g. Google Scholar, journal rankings etc.)

• Universities provide contextual/research statement for each 30% sample

• Standardise user-interface for university repositories

• Recognition of the contribution of peer reviewers (e.g. NHMRC example)

• Feedback—especially for ‘first time’ reviewers

Can we talk about ERA 2015 results?

Not yet….

ERA 2015—some feedback

• Inclusion of text books—to be included these must meet the definition of research—we saw a number of these

• Watch out for predatory publishers which don’t peer review manuscripts—we saw a number of these

• Make sure the coding of outputs reflects the content of the output—know your ANZSRC codes

• Watch out for ‘recycled’ content—a revision or subsequent version must include substantial new research content

Know your ANZSRC codes

18 Law and Legal Studies1801 Law 1802 Maori Law 1899 Other Law and Legal StudiesExclusions:a) Criminology, including policing and correctional

theory, is included in Group 1602 Criminology. b) Legal ethics and human rights and justice issues are

included in Group 2201 Applied Ethics. c) History and philosophy of law and justice is included

in Group 2202 History and Philosophy of Specific Fields.

Know your ANZSRC codes1801 LAW

This group covers law.It includes:

• legal institutions;

• legal theory and practice; and

• litigation, adjudication and dispute resolution.

This group has twenty-seven fields:

180101 Aboriginal and Torres Strait Islander Law 180102 Access to Justice 180103 Administrative Law 180104 Civil Law and Procedure 180105 Commercial and Contract Law 180106 Comparative Law 180107 Conflict of Laws (Private International Law) 180108 Constitutional Law 180109 Corporations and Associations Law 180110 Criminal Law and Procedure 180111 Environmental and Natural Resources Law 180112 Equity and Trusts Law 180113 Family Law 180114 Human Rights Law

180115 Intellectual Property Law 180116 International Law (excl. International Trade Law) 180117 International Trade Law 180118 Labour Law 180119 Law and Society 180120 Legal Institutions (incl. Courts and Justice Systems) 180121 Legal Practice, Lawyering and the Legal Profession 180122 Legal Theory, Jurisprudence and Legal Interpretation 180123 Litigation, Adjudication and Dispute Resolution 180124 Property Law (excl. Intellectual Property Law) 180125 Taxation Law 180126 Tort Law 180199 Law not elsewhere classified

Exclusions:

a) Criminology, including policing and correctional theory, is included in Group 1602 Criminology.

b) Legal ethics and human rights and justice issues are included in Group 2201 Applied Ethics.

c) History and philosophy of law and justice is included in Group 2202 History and Philosophy of Specific Fields.

Know your ANZSRC codes

1802 MAORI LAW

This group covers Maori Law.

This group has five fields:180201 Nga Tikanga Maori (Maori Customary Law) 180202 Te Maori Whakahaere Rauemi (Maori Resource Law)) 180203 Te Tiriti o Waitangi (The Treaty of Waitangi) 180204 Te Ture Whenua (Maori Land Law) 180299 Maori Law not elsewhere classified

Know your ANZSRC codes

1899 OTHER LAW AND LEGAL STUDIES *DOES YOUR OUTPUT REALLY BELONG HERE?

This group covers law and legal studies not elsewhere classified.

This group has one field:

189999 Law and Legal Studies not elsewhere classified

Exclusions:a) Criminology, including policing and correctional theory, is included

in Group 1602 Criminology. b) Legal ethics and human rights and justice issues are included in

Group 2201 Applied Ethics. c) History and philosophy of law and justice is included in

Group 2202 History and Philosophy of Specific Fields.

ERA-HERDC Alignment Update

ERA-HERDC Alignment

Background• PhillipsKPA Review of Reporting Requirements for Universities• Department of Education and the ARC working together• A single research data collection• Part of Government’s Reducing Red Tape agenda

Aim• A more efficient streamlined process for the collection of data to inform

RBG and evaluate excellence

Potential Efficiencies • University admin staff and IT systems only need to cater for one data set• Reduces opportunities for error and misunderstanding—enhancing

integrity, transparency and utility of data• Auditing requirements in relation to the data will be streamlined

ERA-HERDC Alignment cont.

• Consultation paper on options for alignment (8 December–13 February)

• Sector working group (3 July–21 September)• Draft specifications completed• Next steps – sector consultation on draft

specifications, when…?

Thank you