Prof Max KingPVC [Research and Research Training]

Research Excellence on Trial

April, 2009

Excellence in Research for Australia [ERA]

• Australian Government Initiative• Hon. Senator Kim Carr, Minister for

Innovation, Industry, Science and Research

• Developed by Australian Research Council in consultation with the National Health and Medical Research Council

• Supersedes RQF

Aim of ERA

• In replacing RQF, ERA is designed to– Reduce the burden on researchers in

providing data by streamlining the requirements

– Move from ‘story telling’ to ‘objective measures’

Objectives of ERA

• Establish an evaluation framework• Provide a national stock-take of

research strengths and opportunities for development

• Identify excellence in research performance

• Identify emerging research areas• Allow for national and international

comparisons

Objectives Full ESG Text

Australia and New Zealand Standard Research Classification [ANZSRC]

• More inclusive than RFCD; Revised 2008

• 3 Fields of Research [FoR] ‘Tiers’ – 02 Physical Sciences [2-digit]

> 0203 Classical Physics [4-digit]– 020303 Fluid Physics [6-digit]

Actual Level of Evaluation

2-digit Field of Research [FoR] DisciplinesMathematical Sciences 01

Physical Sciences 02

Chemical Sciences 03

Earth Sciences 04

Environmental Sciences 05

Biological Sciences 06

Agricultural and Veterinary Sciences 07

Information and Computing Sciences 08

Engineering 09

Technology 10

Medical and Health Sciences 11

Built Environment and Design 12

Education 13

Economics 14

Commerce, Management, Tourism and Services

15

Studies in Human Society 16

Psychology and Cognitive Sciences 17

Law and Legal Studies 18

Studies in Creative Arts and Writing 19

Language, Communication and Culture

20

History and Archaeology 21

Philosophy and Religious Studies 22

PCE Cluster RED HCA Cluster BLUE

22 Disciplines in Eight Clusters

Physical, Chemical, and Earth Sciences [2009 Trial] PCE

Humanities and Creative Arts [2009 Trial] HCA

Engineering and Environmental Sciences EE

Social, Behavioural, and Economic Sciences SBE

Mathematical, Information, and Computing Sciences MIC

Biological and Biotechnological Sciences BB

Biomedical and Clinical Health Sciences BCH

Public and Allied Health Sciences PAH

Trial Then Real ERA

• 2009 Trial ERA– Submit PCE Cluster by 3 July, 2009– Submit HCA Cluster by 21 August, 2009– ARC to evaluate and publish results by

Nov/Dec 2009

• Potential revisions to guidelines and other key documents [Journal Lists/Rankings]

• Early 2010 [possibly March] all eight Clusters to be submitted for ‘Real ERA’ with

The 2010 Real

• Published Monash research profile/reputation

• Potential to influence research funding• Ranking of Monash [in some form]• Impact on HDR places• Student and staff choices• Indicator of strengths• Shapes focus of future research efforts• Identifies areas for potential improvement

ARC Evaluation Approach

• “Each REC may include internationally-recognised members with expertise in research evaluation and broad discipline expertise.

• For the PCE cluster, quantitative information will provide the primary source for the provision of indicators in the evaluation process.

• The ERA evaluation process will use those quantitative indicators … to inform expert review undertaken by RECs.

• ERA indicators will be presented to RECs as both profile and trend data as allowable by the type of indicator, and will be informed by discipline benchmarks where applicable.” [ESG]

Inputs to ARC Evaluation

• RECs informed by combinations of indicators from three categories

– Research Quality:> ranked outlets, citation analysis, research income,

peer review– Volume and Activity:

> research outputs, research income– Application:

> research commercialisation and other applied measures

• Choices will be made depending on the nature of the research output within each cluster [e.g. Peer Review for Works of Art not Citation Analysis]

The Evaluation Guidelines

Evaluation Guidelines to be published ‘shortly’

?

Use of 2009 Evaluation Information

• ARC reports by Disciplines [4-digit FoR Codes]– Not by individual researcher– Not by Academic Organisational Unit [AOU]

• National Ranking/Rating– Publicly released

• Institutional Ranking/Rating [more resolution]– Released to Institution only

• DIISR owned HERDC processes currently continue to inform research block grant allocation [cf. recent press release]

Trimmed Copy of Speech

Internal Monash Analysis and Reporting

• “Institutions may devise their own reporting codes that link components of their cluster submission to particular Institutional Units within the institution, such as academic organisation units, research centres, centres of excellence or departments.

• Institutions may submit up to two Institutional Units with each research output or other research item.” [ESG]

• Coding in this way will provide powerful analytical tools and reporting mediums to inform Academic Organisational Units and other Admin groups

Interdisciplinary and Multidisciplinary Research• “As ERA is a discipline-based research evaluation exercise,

interdisciplinary and multidisciplinary research will be disaggregated based on its discipline components.

• The ARC will have the capacity to identify profiles of interdisciplinary research based on the FoR codes submitted with research outputs having more than one FoR code.

• However, this will not form part of the evaluation process.

• … institutions will be able to internally recompile interdisciplinary and multidisciplinary research by submitting up to two Institutional Unit code(s) and/or up to two research themes for each research item.” [ESG]

Definition of Research under ERA

• ‘Creative work undertaken on a systematic basis in order to increase the stock of knowledge, including knowledge of humanity, culture, and society, and the use of this stock of knowledge to devise applications’– HERDC Specifications

Unit of Evaluation and Reporting

Eligible Researchers for 2009 ERA Trial

• Affiliated with Monash at 31 March, 2008

• Full Time, Part Time, Casual, Visiting, Exchange, Seconded, Conjoint, Clinical, Adjunct, or Honorary

• Designated Function as either ‘Research Only’, ‘Research and Teaching’, or ‘Other’

• Affiliation must be evidenced by Employment Contract, or the existence of a Monash-specific Byline in the research output

Discipline Evaluation and Reporting

Eligibility of Research Outputs [PCE/HCA Trial Clusters] 2009 Trial

• Meet definition of Research

• Introduced to public domain during reference period: 1 Jan, 2002 to 31 Dec, 2007

• Have one or more eligible researchers

• Be of appropriate type– Books, Book Chapters, Journal Articles, Conference

Publications

Research Outputs: Books

• Likely Eligibility– Critical scholarly texts– New interpretations of historical events– New ideas or perspectives based on established

research findings

• Unlikely Eligibility– Textbooks– Anthologies– Edited Books– Revisions or New Editions

Research Outputs: Book Chapters

• Likely Eligibility– Scholarly Introduction of Chapter Length– Critical Scholarly Chapter– Critical Review of Current Research

• Unlikely Eligibility– Chapters in Textbooks, Entries in Reference Books– Anthologies– Revisions of Chapters– Forewords, Brief Introductions, Brief Editorials– Appendices

Research Outputs: Journal Articles

• If in the ARC defined Cluster Journal Lists– Has ISSN

– Is considered peer-reviewed by ARC-ERA

• Rare cases with ISBN [not ISSN]– Is identified as a Journal [not a book]

– ISSN cited in extract from ISI or Scopus

– Evidence that the Journal is ‘Refereed’ as per Ulrich

… ARC Cluster Journal Lists

• Iterative consultation has resulted in an ARC-approved list of journals suitable for inclusion in ERA

• Uniquely identified by ISSN [International Standard Serial Number]

• Individually Ranked as A* [Top 5%], A [Next 15%], B [Next 30%], and C [Remaining 50%]

– These rankings are used in determining research quality in the ERA evaluation process

– Articles not in Journals in these lists are not submitted with ERA

Research Outputs: Conference Publications• Peer-reviewed

• Published in full

• Presented at conferences, workshops, or seminars of national or international significance

• Unlikely eligibility for papers included in volumes only handed out to conference participants

Research Outputs: HCA-Specific

• Original Creative Works

• Live Performance of Creative Works

• Recorded/Rendered Creative Works

• Curated or Produced Substantial Public Exhibitions and Events

Discipline Evaluation and Reporting

Research Income [Trial 2009]

• Reference period 1 Jan, 2005 to 31 Dec, 2007 [for 2009 Trial]

• Australian Competitive Grants [HERDC]

• Other Public Sector Research Income [HERDC]

• Industry and Other Research Income [HERDC] disaggregated:– Australian– International A– International B

• CRC Research Income [HERDC]

Discipline Evaluation and Reporting

Applied Measures [Trial 2009]

• Reference Period 1 Jan, 2005 to 31 Dec, 2007 [for 2009 Trial]

• Patents

• Registered Designs

• Research Commercialisation Income

Applied Measures: Patents

• Became enforceable within Reference Period– Australian Standard [not Innovation]

– United States Patent and Trademark Office [USPTO]

– European Patent Office [EPO]

– Japan Patent Office [JPO]

– Other International

– Triadic [corresponding patents at USTPO, EPO, and JPO]

Applied Measures: Registered Designs

• Certified within Reference Period

• “… a right granted for new and distinctive designs.”

• In this context design refers to features which, when applied to a product, render it unique in appearance and may include shape, pattern, or ornamentation

• Meets the definition of Research

Applied Measures: Commercialisation Income• Received within Reference Period

• Licenses, Options, Assignments [LOAs]

• Earned by Monash, its subsidiaries and spin-off companies

• Appears on audited financial statements

• The ARC has explicitly excluded consultancy work that has no LOA research income

PCE Journal Article Volume Hurdle

If the number of indexed journal articles for a 4-digit FoR is less than 50, and

If there are 50 or more articles at the 2-digit level Then those articles are included in 2-digit FoR analyses

If the number of indexed journal articles submitted for a 2-digit FoR is less than 50 that 2-digit FoR will not be evaluated or reported at

Institutional level

All submitted eligible articles will be included in national evaluation and reporting

PCE Journal Articles and Scopus EIDs

• PCE journal articles submitted to ARC must– Be included in the ARC Ranked Journal List

– Have an Electronic ID [from Scopus] attached

• The closing date for the RQT to obtain EIDs is 29 May

• The deadline for inclusion of PCE articles in RM is 15 May, 2009

HCA Peer Review [2009 Trial]• PCE and HCA outputs include Books, Chapters, Journals, and

Proceedings; lead to measures of volume and activity via Citation Analyses, Rankings etc

• For specialised works in the Creative Arts and related fields [HCA], such measures are inappropriate

• Peer Review will be undertaken with each piece nominated by Monash, to allow review of 20% of the total outputs submitted [20% of each 4-digit FoR within a cluster]

• Outputs in the Creative Arts category, and in the 20% for-Peer Review group, are to be accompanied by a statement [250 words] identifying the ‘research component of the output’ [stored in an institutional repository]

• National and international experts shall evaluate these outputs and recommend measures to ERA

HCA Repository [2009 Trial]

• Outputs nominated as part of the 20% for Peer Review must be made available to ARC in an institutional repository

– Provisions are made for commercially sensitive and outputs that cannot be digitised

• MU Library Information Resources Division [Andrew Harrison] is managing this repository

2-Digit Discipline Background Statements• ARC-ERA invites submission of a Background Statement for

each 2-digit FoR within a Cluster

• To inform the RECs about context, background, and events within the reference period

• Most of the 2-digit FoRs span multiple AOUs

• Cluster Sub-Committees of the ERA Strategy Committee have panels [representatives from 4-digit fields within AOUs] that are collaborating with the Research Quality Team to create these Statements

• 10,000 character limit [approx. 3–4 A4 pages]

Reference Period & Submission Timing

Estimated by adding 12 Months

Trial ERA 2009 Real ERA 2010

Mon, 31-Mar-08 Tue, 31-Mar-09

Start Tue, 01-Jan-02 Wed, 01-Jan-03End Mon, 31-Dec-07 Tue, 30-Dec-08

Start Sat, 01-Jan-05 Sun, 01-Jan-06End Mon, 31-Dec-07 Tue, 30-Dec-08

Start Sat, 01-Jan-05 Sun, 01-Jan-06End Mon, 31-Dec-07 Tue, 30-Dec-08

PCE Cluster Fri, 03-Jul-09HCA Cluster Fri, 21-Aug-09All Clusters Mar-2010

Submission Deadlines

Applied MeasuresReference

Period

Eligible ResearchersCensus Date

Research OutputsReference

Period

Research IncomeReference

Period

Imperatives

• Data collection is a University-wide effort

• For the 2009 submission– All Eligible Researchers– All PCE and HCA outputs, income, and applied

measures

• For the 2010 submission all data for all clusters

• Vice Chancellor Certification regarding completeness and accuracy

Evaluation Reporting

• ERA 2009 [PCE and HCA]– Only national level discipline outcomes published

nationally– University level discipline outcomes provided

individually to Universities

• ERA 2010 [All Clusters]– National and University level outcomes published

nationally– University level discipline outcomes provided to

Universities

What Now?

• Researchers should visit the ERAMIS application to validate FoRs, to choose National Priority and Monash Themes, and to check their publication listing

• The RQT is collaborating with Research Admin personnel to gather and code FoRs to research outputs and other measures

The Big Picture

Research activities

Resources and

facilities

People with knowledge and abilities

Individual, social, and

institutional priorities

Problems

Prior research

Funding

A multitude

of research

outputs of varying

types

Evaluation using metrics based onRankingsCountsFormulaeand [where appropriate]Undefined peer review

ERA

Recent Announcement Confirms

Monash Rating by

4-digit FoR

Information

• This presentation shall be made available at http://intranet.monash.edu.au/research/quality/framework.html

• The full ERA Submission Guidelines and other documents are available at

– www.arc.gov.au– http://intranet.monash.edu.au/research/quality/framework.html

• ADRs and Research Coordinators within AOUs

• RQT

The RQT

• Portfolio of PVC [R&RT]– Established the Research Quality Team

– Constituted the ERA Strategy Committee [ERASC]

> Formed Cluster Sub-Committees

– Liaises with Associate Deans of Research and ERA Admin Coordinators

– Supported by> Integrated Administrative Services> Research Information Systems

Thank You

• ERAMIS Overview

Sen Carr’s Announcement [Trimmed]• “Perhaps the most important thing we expect from our universities is

an unwavering commitment to excellence. Australia simply isn’t big enough or rich enough to be good at everything.

• We will further that objective by progressively addressing the gap in funding for the indirect costs of university research, subject to the capacity of future budgets.

• But in return, universities have to become more accountable. They must give us a better idea of what research really costs.

• They must meet specific performance targets, which will be developed in consultation with the sector.

• And they must submit to the discipline of international comparison and competition.

• That’s why Excellence in Research for Australia is so important.• It will guarantee research standards and performance by measuring

our achievements against the best in the world.• It will ultimately be used to plan future investments in research

and research training."

Return to Presentation

Objectives Full ESG Text

1. Establish an evaluation framework that gives government, industry, business and the wider community assurance of the excellence of research conducted in Australia’s institutions;

2. Provide a national stocktake of discipline-level areas of research strength and areas where there is opportunity for development in Australia’s higher education institutions;

3. Identify excellence across the full spectrum of research performance;

4. Identify emerging research areas and opportunities for further development; and

5. Allow for comparisons of Australia’s research nationally and internationally for all discipline areas.

Return to Presentation

Physical, Chemistry, and Earth Sciences Cluster [PCE]PHYSICAL SCIENCES 02

ASTRONOMICAL AND SPACE SCIENCES 0201

ATOMIC, MOLECULAR, NUCLEAR, PARTICLE AND PLASMA PHYSICS

0202

CLASSICAL PHYSICS 0203

CONDENSED MATTER PHYSICS 0204

OPTICAL PHYSICS 0205

QUANTUM PHYSICS 0206

OTHER PHYSICAL SCIENCES 0299

CHEMICAL SCIENCES 03

ANALYTICAL CHEMISTRY 0301

INORGANIC CHEMISTRY 0302

MACROMOLECULAR AND MATERIALS CHEMISTRY

0303

MEDICINAL AND BIOMOLECULAR CHEMISTRY

0304

ORGANIC CHEMISTRY 0305

PHYSICAL CHEMISTRY (INCL. STRUCTURAL) 0306

THEORETICAL AND COMPUTATIONAL CHEMISTRY

0307

OTHER CHEMICAL SCIENCES 0399

EARTH SCIENCES 04

ATMOSPHERIC SCIENCES 0401

GEOCHEMISTRY 0402

GEOLOGY 0403

GEOPHYSICS 0404

OCEANOGRAPHY 0405

PHYSICAL GEOGRAPHY AND ENVIRONMENTAL GEOSCIENCE

0406

OTHER EARTH SCIENCES 0499

Return to Presentation

Humanities and Creative Arts Cluster [HCA ]

Return to Presentation

BUILT ENVIRONMENT AND DESIGN 12

ARCHITECTURE 1201

DESIGN PRACTICE AND MANAGEMENT 1203

URBAN AND REGIONAL PLANNING 1205

OTHER BUILT ENVIRONMENT AND DESIGN 1299

LAW AND LEGAL STUDIES 18

LAW 1801

MAORI LAW 1802

OTHER LAW AND LEGAL STUDIES 1899

STUDIES IN CREATIVE ARTS AND WRITING 19

ART THEORY AND CRITICISM 1901

FILM, TELEVISION AND DIGITAL MEDIA 1902

JOURNALISM AND PROFESSIONAL WRITING 1903

PERFORMING ARTS AND CREATIVE WRITING 1904

VISUAL ARTS AND CRAFTS 1905

OTHER STUDIES IN CREATIVE ARTS AND WRITING

1999

LANGUAGE, COMMUNICATION AND CULTURE 20

COMMUNICATION AND MEDIA STUDIES 2001

CULTURAL STUDIES 2002

LANGUAGE STUDIES 2003

LINGUISTICS 2004

LITERARY STUDIES 2005

OTHER LANGUAGE, COMMUNICATION AND CULTURE

2099

HISTORY AND ARCHAEOLOGY 21

ARCHAEOLOGY 2101

CURATORIAL AND RELATED STUDIES 2102

HISTORICAL STUDIES 2103

OTHER HISTORY AND ARCHAEOLOGY 2199

PHILOSOPHY AND RELIGIOUS STUDIES 22

APPLIED ETHICS 2201

HISTORY AND PHILOSOPHY OF SPECIFIC FIELDS

2202

PHILOSOPHY 2203

RELIGION AND RELIGIOUS STUDIES 2204

OTHER PHILOSOPHY AND RELIGIOUS STUDIES 2299

Engineering and Environmental Sciences Cluster [EE]ENVIRONMENTAL SCIENCES 05

ECOLOGICAL APPLICATIONS 0501

ENVIRONMENTAL SCIENCE AND MANAGEMENT

0502

SOIL SCIENCES 0503

OTHER ENVIRONMENTAL SCIENCES 0599

ENGINEERING 09

AEROSPACE ENGINEERING 0901

AUTOMOTIVE ENGINEERING 0902

BIOMEDICAL ENGINEERING 0903

CHEMICAL ENGINEERING 0904

CIVIL ENGINEERING 0905

ELECTRICAL AND ELECTRONIC ENGINEERING

0906

ENVIRONMENTAL ENGINEERING 0907

FOOD SCIENCES 0908

GEOMATIC ENGINEERING 0909

MANUFACTURING ENGINEERING 0910

MARITIME ENGINEERING 0911

MATERIALS ENGINEERING 0912

MECHANICAL ENGINEERING 0913

RESOURCES ENGINEERING AND EXTRACTIVE METALLURGY

0914

INTERDISCIPLINARY ENGINEERING 0915

OTHER ENGINEERING 0999

BUILT ENVIRONMENT AND DESIGN 12

BUILDING 1202

ENGINEERING DESIGN 1204

Return to Presentation

Social, Behavioural, and Economic Sciences Cluster [SBE]

EDUCATION 13 TRANSPORTATION AND FREIGHT SERVICES 1507

EDUCATION SYSTEMS 1301 OTHER COMMERCIE, MANAGEMENT, TOURISM AND SERVICES

1599

CURRICULUM AND PEDAGOGY 1302 STUDIES IN HUMAN SOCIETY 16

SPECIALIST STUDIES IN EDUCATION 1303 ANTHROPOLOGY 1601

OTHER EDUCATION 1399 CRIMINOLOGY 1602

ECONOMICS 14 DEMOGRAPHY 1603

ECONOMIC THEORY 1401 HUMAN GEOGRAPHY 1604

APPLIED ECONOMICS 1402 POLICY AND ADMINISTRATION 1605

ECONOMETRICS 1403 POLITICAL SCIENCE 1606

OTHER ECONOMICS 1499 SOCIAL WORK 1607

COMMERCE, MANAGEMENT, TOURISM AND SERVICES

15 SOCIOLOGY 1608

ACCOUNTING, AUDITING AND ACCOUNTABILITY

1501 OTHER STUDIES IN HUMAN SOCIETY 1699

BANKING FINANCE AND INVESTMENT 1502 PSYCHOLOGY AND COGNITIVE SCIENCES 17

BUSINESS AND MANAGEMENT 1503 PSYCHOLOGY 1701

COMMERCIAL SERVICES 1504 COGNITIVE SCIENCE 1702

MARKETING 1505 OTHER PSYCHOLOGY AND COGNITIVE SCIENCES

1799

TOURISM 1506 Return

Mathematical, Information, and Computing Sciences Cluster [MIC]

MATHEMATICAL SCIENCES 01

PURE MATHEMATICS 0101

APPLIED MATHEMATICS 0102

NUMERICAL AND COMPUTATIONAL MATHEMATICS

0103

STATISTICS 0104

MATHEMATICAL PHYSICS 0105

OTHER MATHEMATICAL SCIENCES 0199

INFORMATION AND COMPUTING SCIENCES

08

ARTIFICIAL INTELLIGENCE AND IMAGE PROCESSING

0801

COMPUTATION THEORY AND MATHEMATICS

0802

COMPUTER SOFTWARE 0803

DATA FORMAT 0804

DISTRIBUTED COMPUTING 0805

INFORMATION SYSTEMS 0806

LIBRARY AND INFORMATION STUDIES 0807

OTHER INFORMATION AND COMPUTING SCIENCES

0899

TECHNOLOGY 10

COMMUNICATIONS TECHNOLOGIES 1005

COMPUTER HARDWARE 1006

NANOTECHNOLOGY 1007

OTHER TECHNOLOGY 1099

Return to Presentation

Biological and Biotechnological SciencesCluster [BB]

BIOLOGICAL SCIENCES 06

BIOCHEMISTRY AND CELL BIOLOGY 0601

ECOLOGY 0602

EVOLUTIONARY BIOLOGY 0603

GENETICS 0604

MICROBIOLOGY 0605

PHYSIOLOGY 0606

PLANT BIOLOGY 0607

ZOOLOGY 0608

OTHER BIOLOGICAL SCIENCES 0699

AGRICULTURAL AND VETERINARY SCIENCES

07

AGRICULTURE, LAND AND FARM MANAGEMENT

0701

BIOLOGICAL SCIENCES 06

BIOCHEMISTRY AND CELL BIOLOGY 0601

Return to Presentation

ANIMAL PRODUCTION 0702

CROP AND PASTURE PRODUCTION 0703

FISHERIES SCIENCES 0704

FORESTRY SCIENCES 0705

HORTICULTURAL PRODUCTION 0706

VETERINARY SCIENCES 0707

OTHER AGRICULTURAL AND VETERINARY SCIENCES

0799

TECHNOLOGY 10

AGRICULTURAL BIOTECHNOLOGY 1001

ENVIRONMENTAL BIOTECHNOLOGY 1002

INDUSTRIAL BIOTECHNOLOGY 1003

MEDICAL BIOTECHNOLOGY 1004

ANIMAL PRODUCTION 0702

CROP AND PASTURE PRODUCTION 0703

Biomedical and Clinical Health SciencesCluster [BB]

Return to Presentation

MEDICAL AND HEALTH SCIENCES 11

MEDICAL BIOCHEMISTRY AND METABOLOMICS 1101

CARDIOVASCULAR MEDICINE AND HAEMATOLOGY 1102

CLINICAL SCIENCES 1103

DENTISTRY 1105

IMMUNOLOGY 1107

MEDICAL MICROBIOLOGY 1108

NEUROSCIENCES 1109

ONCOLOGY AND CARCINOGENESIS 1112

OPHTHALMOLOGY AND OPTOMETRY 1113

PAEDIATRICS AND REPRODUCTIVE MEDICINE 1114

PHARMACOLOGY AND PHARMACEUTICAL SCIENCES 1115

MEDICAL PHYSIOLOGY 1116

Public and Allied Health SciencesCluster [BB]

Return to Presentation

MEDICAL AND HEALTH SCIENCES 11

COMPLEMENTARY AND ALTERNATIVE MEDICINE 1104

HUMAN MOVEMENT AND SPORTS SCIENCE 1106

NURSING 1110

NUTRITION AND DIETETICS 1111

PUBLIC HEALTH AND HEALTH SERVICES 1117

OTHER MEDICAL AND HEALTH SCIENCES 1199