Bureau du F.R.S. – FNRS 23 juin 2009

Possible role of a research funding agency

The role of evaluation and ranking of Universities in the quality culture

Prof. Véronique Halloin; General Secretary of F.R.S.-FNRSDr. Pascal Perrin; head of research evaluation department

The quality of a University depends on:Academic/Faculty performanceResearch performanceQuality of educationLibrarySupports (administration, IT, infrastructure, …)Campus cultureQuality of life…

Ranking: need for a multidimensional framework (see Ph Vincke;

see report of the EC Expert Group on Assessment of University-based Research)

addressing users’ needs and interests (students, scientists, governments, universities, industries)

1. Research performance in ranking of universities

1. Research performance in ranking of universities

3

The present contribution of research performances:

Shangai: 80% researchNobel prizes: 20% (by staff)Highly cited staff in 21 disciplines: 20%Articles published: 20% (Nature, Science) + 20% (citation indexes)

THES: <= 40% researchAcademic peer review : 40%/2Citation per Faculty: 20%

CHE: no weights; 4 fields: biology, chemistry, mathematics, & physics

Size indicator: ouput volume in citation indexesPerception indicators: citationsNumber of often-cited staff & Nobel prize winners at the universityEurope indicator: # projects in Marie Curie

SHS poorly considered

1. Research performance in ranking of universities

4

IndicatorIndicators used for Research Ranking System (Country)

Overall grants (money amount) SlovakiaGrants per faculty (money amount) Austria, Germany, ItalyGrants per faculty (absolute numbers) ItalyResearch projects funded by EU ItalyParticipation in int’l research programmes PolandNo. of publications SwedenPublications per researcher Germany, Slovakia, SwitzerlandCitations per faculty UKCitations per publication Germany, Slovakia, SwitzerlandNo. of int’l publications Poland% articles cited within 1st two years after publication SwedenNo. of publications with 5+ citations Slovakia% articles belonging to top 5% most cited articles (HiCi)

Sweden

No. of patents (absolute number) GermanyPatents per faculty GermanyRatio of pg research students UKResearch quality Germany, UKReputation for research Austria, Germany

Ellen Hazelkorn; Dean of the Graduate Research School, Dublin Institute of Technology, Ireland; International Symposium on University Rankings; University of Leiden, 6-7 February 2009

1. Research performance in ranking of universities

2.The possible role of a research funding agency

3. Challenges of bibliometrics

4. Research and science mapping

5. Ranking of research actors

6. Pending questions

The role of evaluation and ranking of Universities in the

quality culture

CAF - 26.06.09 - V3

6

2. The possible role of a research funding agency

Funding evaluation is needed for allocation of money Research evaluation might be outsourced (if bibliometrics)but used to

describe the global landscape of research to be funded Research policy is fed by funding evaluation and research evaluation

A research funding agency is a university stakeholder: allocation of money & research policy making

UNIVERSITIES UNIVERSITIES

UNIVERSITIESPOLITICS, CITIZENS

2. The possible role of a research funding agency

INPUT

OUTCOME

OUTPUT

Data bases, Researchers, Institutional repositories

Indicator – metricsGraphs - maps

FRS

Methodology:bibliometrics

Universities: monitoringRanking makers: indicators; possible counterweight?Funding agency: research policy

research evaluationindicators(economy& society)

bibliometrics(research

community)

studies(retrospective & prospective)

academic research landscape atinternational, national & community levels

FNRS

Aggegation to be avoided!!Multi-dimensional space crashed into linearity, information is definitely lost!!

1. Research performance in ranking of universities

2. The possible role of a research funding agency

3.Challenges of bibliometrics

4. Research and science mapping

5. Ranking of research actors

6. Pending questions

8

The role of evaluation and ranking of Universities in the

quality culture

3. Challenges of bibliometrics

DATA: many problems linked to the collection of data

FIELDS – DISCIPLINES: various classifications, not

satisfactory

INDICATORS: which ones are valid/needed to quantify

research performance?

MAPPING: to assess potential rather than past

performances (metrics)

workshop

3. Challenges of bibliometrics

DATAVarious databases: Web of Science (ISI); Scopus; Google scholar

Overlap/coverage: strong dependency with the scientific field (Exemple: for management, only less than 20% of

publications are published in the ISI listed journals ; Harzing & van der Wal, Ethics in science and environmental policy, 2008)

Do not include productions such as art pieces or books, and limited open access publications (Scopus, Google) or conference proceedings

institutional repositories to capture ouputs!

Choice linked to what measurement is needed for(Google Scholar not suited to macro-level

studies)

Realibility of data is a precondition for all ranking exercices, the best methodology and soundest mathematical approach cannot correct what data collection might have distorsed…

3. Challenges of bibliometrics

DATA: problems of identification

Spelling variances of ULB on WoS (1st encoder)

2001 – 2008: # publications?First step: 20.545 publicationso ULB, univ, univ libre, free univo IRIBHM, IBMM, ECARES, IIHE, erasme,

Brugmann, St Pierre, Bordet

Second step: - 3.850 “false” publicationso Vrije or dutch or flemish or vlaams or UZ or AZ or ziekenhuiso St Luc, St Louis, catholique, …..

Re-coding errors (last encoder)

o Univ. Zimbabwe, Dept Thorc.Surg., Leuven,Belgium (Ziekenhuis)

o Free Univ. Brussels, Inst Math 1, D-14195 Berlin, Germany (Freie Universität Berlin)

Belgium, BrusselsBruxellesAnderlecht, 1070…

Except co-publications

16.695

3. Challenges

MAPPING: a field of bibliometrics in very active development (to

detect clusters, new disciplines, emergence…)

scientific fields mapped along time evidence of change in networks each block in a column represents a field (ordered by size) the height of the block reflects citation flow through the field apparition of a new scientific field, neuroscience, from the convergence of

neurology, psychology, and a part of molecular & cell biology

Map of science2007 - 2008Los Alamos

on-line behaviour of scientists accessing different scientific journals (on-line access of a paper recorded by the servers of scholar web portals)objective: to visualize the

links between disciplines

circles = individual journals

colours = disciplines lines = scientists

navigation between publications

unexpected relations revealed: connection between ecology and architecture

prominent and central position for HSS (while maps based on citations favor natural sciences)

1. Research performance in ranking of universities

2. The possible role of a research funding agency

3. Challenges of bibliometrics

4.Pending questions

14

The role of evaluation and ranking of Universities in the

quality culture

6. Pending questions

Research evaluation is usefull

But

Needs to be carefully handled

And

Raises several questionsImpact of evaluation on research activity, strategy (researchers, universities)Impact of evaluation on research funding (disciplines?)Risk to miss emergence of teams, themes (bibliometrics assess the past)Choice of methodology (peer review? bottom-up?)

How to feed the research part of the (probably unavoidable) rankings with indicators developed as a support of research policyPossible role of networks (Eurohorcs, Unica, …)