FOKUS Research evaluation in Sweden - Startsida ... and Engineering Sciences Medicine Natural...

FOKUS –Research evaluation in Sweden

19 December 2014

HEI research income 2013

• Resurser för kvalitet (SOU 2007:81), Dan

Brändström

• Performance based resource funding was

introduced in 2009

• Prestationsbaserad resurstilldelning för

universitet och högskolor (2011, Anders

Flodström)

• Research and Innovation Bill 2012

Research evaluation in Sweden – FOKUS

BackgroundPerformance based resource funding in Sweden

The Government Commission (14 March 2013):

The Government commissions the Swedish Research

Council to investigate and submit – in consultation with

Forte, Formas and Vinnova – a proposed model for resource

allocation to universities and university colleges involving

peer review of the quality and relevance of research

• Should enable resource allocation that rewards quality and performance in research

• Should comprise both assessment of research quality and assessment of the societal relevance and impact of the research

• May provide a good platform for long-term planning at higher-education institutions

• Should be done bearing in mind the preconditions for the respective research area

• All research should be regularly assessed in subject-area based evaluations in a cycle of four to six years

• May include indicators

• To be submitted by 31 December 2014Government decision I:8 U2013/1700/F, 14 March 2013


InformativeEfficient

Guiding principles of

the Swedish Research Council

• Improve the quality of research

• All data should be subject to panel assessments

• Understandable – as simple as possible

• Transparent

• Fair

• Include all research areas

• Broad interpretation of societal impact

• Balance between two principles:

Working process

• Mapping of performance based research funding

systems Nov 2012–May 2013 (published at

www.vr.se)

• Continous dialogue and collection of stakeholders’

views within and outside academia

• Advisory boards

• Discussions with relevant government agencies

• Meetings and open hearings with HEIs (May 2014)


All research is to be evaluated in a cohesive manner every sixth year(initially more frequently).


Step 1: The evaluation model – Elements of

evaluation and suggested weighting

Units of assessment

• Based on existing classification and statistics

• Consideration given to multidisciplinary research

NL T M S HK

A B C D E F B H I J K L M N O P Q R S T U V W X

5 fields ofscience

24 research areas

O


Units of assessment


A Mathematics

B Physics

C Chemistry

D Geosciences and environmental sciences incl. climate research

E Biology

O Agricultural sciences

F Chemical engineering, material engineering, nanotechnology and mechanical engineering

G Civil and environmental engineering

H Computer science, signals and systems and electrical and electronic engineering

I Biotechnology and medical technology

J Basic medical sciences I

K Basic medical sciences II

L Clinical medicine I

M Clinical medicine II

N Health sciences

P Psychology

Q Economics (incl. social and economic geography, business and industrial economy)

R Educational science

S Sociology, anthropolgy, tehnology, cultural studies and gender studies

T Political science and law

U History and archaeology

V Language, literature and aesthetics

W Philosophy, ethics and religious studies

X Artistic research

HK

Field of Science

Humanities and

Artistic Research

Social Sciences

Technical and

Engineering

Sciences

Medicine

Natural Sciences

and Agricultural

Sciences

Research Area

NL

T

M

S

Collection of data

Data

Responsible organisation (IT-system)

Economy

The Swedish Higher EducationAuthority, UKÄ and StatisticsSweden, SCB

Economy - external fundingSwedish Research Council, VR

(SweCRIS)

Staff UKÄ/SCB

Doctoral students UKÄ

Students UKÄ

Exams UKÄ

PublicationsNational Library of Sweden, KB

(SwePub)

Citation analysis VR (WoS data)

Case studies Implementing agency (Prisma)

Own descriptions Implementing agency (Prisma)

Main principles: • Reduce the work load for individual researchers

• Use as much existing data as possible

Panel structure


Panel chair meetings

Research area panels Main panels

approx. 400 panel membersapprox. 500 externalreviewers

FOKUS – Research evaluation in SwedenPurpose: To improve the quality of research and ensure that high quality research is of

benefit to society

Backgroundinformation(not to be graded)

70 % Scientific/

artistic quality

15 % Quality

enhancingfactors

15 % Impactoutside

academia

Background information to

the panels

• Description of:

– research profile (eg. research focus, interdisciplinaryresearch)

– vision and strategy of HEI

– organisation, administration, management, recruitment

– infrastructure (access to)

• Quantitative data:

– research funding

– teaching and research staff

– doctoral education

– publication profile and publication volume


Scientific/artistic quality

• Weight: 70 %, based on the quality of research production per research area.

• Criteria: novelty and originality, significance and rigour

• The panels’ assessments should include research excellence as well as overall quality.

• When grades are set, the unit’s productivity should be taken into consideration, i.e. research publicationvolume in relation to the volume of staff and financialresources. If two units are of equally high quality, resource efficiency should be rewarded.


Scientific/artistic qualityData to panels


• Research statement

• To evaluate excellence:

- peer review (i.e. review of outputs) of research

outputs in all areas, done by the panels

• Alternatives to evaluate overall quality:

- external peer review, done by external reviewers

- citation analysis

• Special mechanisms to describe and assessmultidisciplinary research

Scientific/artistic researchData to panels

Sample of research production over a six year period

Overall quality: Around 50 % of

the remaining research production

is nominated by HEIs to represent

the overall production – only

research areas without citation

analysis.

Around half of the total research

production is nominated within

these areas.

Excellence: 5 % of

research production

nominated by HEIs and

reviewed by panel

members – all

research areas

A unit’s total research output Panel members use publication

profiles to get an overall picture of

the total research production and

the nominated output.

A share of the nominated output

(around 40 %) is then selected by

panel members for review by

external reviewers.

Description

- Research profile, i.e.

multidisciplinary

research

- Vision and strategy

- Organisation,

administration,

management,

recruitment

- Infrastructure

Research statement

- Research focus, ie.

multidisciplinarity

- Potential

- Other

Quantitative data

- Research funding

- Teaching and

research staff

- Doctoral education

- Publication profile and

publication volume

Nominated sample

(excellence)

5 % of unit’s research

production reviewed by

panel members

Citation analysis or

nominated sample (ca

50 %), (”overall

quality”)

reviewed by panels

(citation analysis) or

external reviewers

(nominated sample)

Scientific/artistic qualityBackground

information (not to be graded)

70 %

Pro

du

cti

vit

y:

(re

se

arc

h p

ub

lication

vo

lum

ein

rela

tio

n to

the

vo

lum

eo

fsta

ffa

nd

fin

ancia

lre

so

urc

es).

Assessment criteria

• Novelty and

originality

• Significance

• Rigour

Results from panel

• Grade profile

• Explanatory

statements

Summary: research area panels

– data and assessment

Quality enhancing factors


• Weight: 15 %, assessment at the level of Field of

Science

• Criteria: potential for renewal and sustainability

• Quality enhancing factors:

– Doctoral education and early career researchers

– Collaboration and mobility within academia (nationally

and internationally)

– Collaboration, partnerships and mobility outside

academia (nationally and internationally)

– Integration of research and education

– Gender equality

• Quantitative data

• Short description with comments in relation to the

HEI’s overarching strategies

Quality enhancing factorsData to panels


Impact outside academia

• Definition: an application of research in order to

achieve social, economic, environmental and/or

cultural benefits

• Weight: 15 %, assessment at the level of Field of

Science

• Criteria: reach and significance


Impact outside academiaData to panels


FOKUS – Forskningskvalitetsutvärdering i SverigeÖvergripande syfte: Kvalitetsdrivande och verka för att forskningen ska bidra till

samhällets utveckling

Bakgrunds-

information

(betygssätts

ej)

70 %

Vetenskaplig

kvalitet

15 % Kvalitets-

utvecklandefaktorer

15 %

Genomslag

utanför

akademin

• Case studies, eg. 2 per at least 10 and maximum 50

full time equivalents

• Description of HEI strategy and resources for

working with dissemination and the promotion of use

outside academia, existing support functions

regarding dissemination of research results,

collaboration between researchers, community,

business interests etc.

Quantitative data


and early career

researchers

- Collaboration and

mobility within

academia (int+nat)

- Collaboration,

partnerships and

mobility outside

academia (int+nat)

- Integration of

research and

education

- Gender equality

Case studies

Examples of activities:

- Dialogue and

dissemination of

results

- Collaboration

- New products and

processes

- Application

- Documented

impact

Description

- Results

- Strategies

- Documentation ofprocesses

Description

- Strategies and

resources for the

dissemination of

results and the

promotion of use

outside academia

Impact outside academiaQuality enhancing factorsBackground

information (not to be graded)

15 % 15 %

As

se

ss

me

nt

cri

teri

a

Po

ten

tia

lfo

r re

ne

wa

la

nd

su

sta

ina

bili

ty

As

se

ss

me

nt

cri

teri

a

Rea

ch

an

d s

ign

ific

an

ce

Re

su

lts

fro

m p

an

el:

gra

de

sa

nd

exp

lan

ato

rysta

tem

en

t

Re

su

lts

fro

m p

an

el:

gra

de

sa

nd

exp

lan

ato

rysta

tem

en

t

Description

- Research profile, ie.

multidisciplinary

research

- Vision and strategy

- Organisation,

administration,

management,

recruitment

- Infrastructure

Quantitative data

- Research funding

- Teaching and

research staff


- Publication profile

and publication

volume

Summary: main panels

– data and assessment

Results of the evaluation at a HEI


– Scientific/artistic quality: grade profiles for each research area

– Quality enhancing factors and impact outside academia: single overall

grades for each field of research– Explanatory statements are also submitted by the panels

The focus of the evaluation


Step 2: Translation of evaluation results

to resource allocationPoints of departure for resource allocation

• The performance based proportion of the block

grant for research and doctoral education remains at

20 percent.

• A proposed weighting of 70:15:15.

• No reallocation if all units receive the same grade.

• Possible reallocations between HEIs should neither

be too large, nor negligible.

• An almost linear weighting of the grades is proposed(6,4,3,2,0).



Principles for the calculation of

resource allocation

• Resource allocation is calculated according to two

measures: grade and a combined volume measure.

• Combined volume measure: the HEI’s share of the

total block grant, and each research area’s share of

the given HEI’s research staff.

• A ”distributional credit” is calculated by multiplying

the grade with the combined volume measure.

• The HEI is free to allocate funds internally (i.e. no

change compared to today).

Preliminary costs

Organisation Description Costs

Responsibleorganisation(s)

Panels, staff, collection of data etc.

ca 75 million SEK

HEIs Coordination, data reporting, nomination of outputs etc.

ca 95 million SEK

Total for one evaluation(every 6th year)

ca 170 million SEK*

*This corresponds to approx. 0.2 percent of the total block grant (or approx. 1 percent of the performance based share of the block grant).

Organisation Description Costs

Responsibleorganisation(s)

Data systems, quality assurance ofdata, administration,communication etc.

ca 20 million SEK

HEIs Data systems, routines for reporting etc.

ca 45 million SEK

Total start-up costs ca 65 million SEK

Start-up costs

Costs for one evaluation

Consequences: strengths and possibilities

• A driver of quality

• An inclusive model – all research areas

• A possibility to identify ”pockets of excellence”,

future potential and impact

• National overviews and benchmarking, possibility to

identify research strengths and trends

• Facilitate strategic decisions at various levels

• Improved reporting of statistics, improved follow-ups

and improved basis for research policy analysis.


Consequences: weaknesses and challenges

• Challenge to evaluate multidisciplinary research and

new research areas

• Higher costs (but greater advantages) compared to

the current resource allocation model

• May be conservative?

• Risk that education is given lower priority by HEIs?

• Case studies as a way of assessing impact outsideacademia is a less tried method


Timeline

Timeline cont.

Organisation

• Internationally it is most common that an

independent body carries out the research

evaluation (eg. UK: HEFCE, IT: ANVUR, FR:

AERES)

• Australia is the exception – Australian Research

Council

• The proposal from the Swedish Research Council is

designed to be organisationally neutral, but the

council has the necessary expertise to undertake

the administration of FOKUS

• Two advisory boards should be created to monitor

the evaluation process and its implementation


In conclusion, the Swedish Research

Council proposes that:

• FOKUS should be circulated for comment during 2015

• the Swedish Research Council and other relevant government agencies should be commissioned during 2015 to prepare pilot studies, and special funding should be provided for these pilots

• pilot studies should be carried out in 2016/2017 followed by analysis and adjustments

• the possibility to coordinate FOKUS with other modelsshould be investigated (eg. evaluation of clinical medicalresearch, ALF, and Vinnova’s model on collaboration)

• The first round of evaluation should be carried out in 2017/2018

• FOKUS should be implemented with a second round ofevaluation in 2021/2022, thereafter evaluations every sixthyear.


FOKUS Research evaluation in Sweden - Startsida ... and Engineering Sciences Medicine Natural...

Documents

Transcript of FOKUS Research evaluation in Sweden - Startsida ... and Engineering Sciences Medicine Natural...