Comparing scientific performance across disciplines: Methodological and conceptual challengesLudo WaltmanCentre for Science and Technology Studies, Leiden University

7th International Conference on Information Technologies and Information SocietyNovember 5, 2015

2

THE ranking

3

Shanghai ranking

4

Website University of Ljubljana

5

Impact factors

6

Centre for Science and Technology Studies (CWTS)• Research center at Leiden

University focusing on science and technology studies

• History of more than 25 years in bibliometric and scientometric research

7

Key challenges in measuring scientific performance1. Defining scientific performance2. Designing sensible metrics3. Correcting for field differences4. Quantifying productivity5. Assessing validity of metrics6. Balancing simplicity, accuracy, and robustness to

gaming

What are Slovenia’s research strengths?

8

Challenge 1: Defining scientific performance

9

Indicators of scientific performance• Publications:

– Total– Per faculty– Per student– Interdisciplinary– International collaboration– Nature and Science

• Citations:– Total– Per publication– Per faculty– Highly cited researchers

• Reputation survey• Others:

– Nobel Prizes/Field Medals– PhDs awarded– PhDs awarded per faculty– Post-doc positions– Research income

10

What do we mean by scientific performance?Size-dependent concept of scientific performance:• Overall contribution of a research unit to science• Total number of ‘performance points’ (e.g.,

publications, citations, expert recommendations, awards)

Size-independent concept of scientific performance:• Contribution of a research unit to science relative to

available resources• Number of ‘performance points’ divided by

available resources (e.g., number of faculty, research budget)

11

Indicators of scientific performance• Publications:

– Total– Per faculty– Per student– Interdisciplinary– International collaboration– Nature and Science

• Citations:– Total– Per publication– Per faculty– Highly cited researchers

• Reputation survey• Others:

– Nobel Prizes/Field Medals– PhDs awarded– PhDs awarded per faculty– Post-doc positions– Research income

Size-dependent indicators Size-independent indicators

12

Mixing up different concepts of scientific performance• Shanghai, THE, QS, and US News use composite

indicators• These composite indicators combine size-

dependent and size-independent indicators

It is unclear which concept of scientific performance is measured by Shanghai, THE, QS,

and US News

13

Challenge 2: Designing sensible metrics

Definition of the h-index

14

A scientist has index h if h of his papers have at least h citations each and the other papers have

at most h citations each

15

Arbitrariness of the h-index

Consistency requirement

16

If two scientists achieve the same performance improvement, their ranking relative to each other

should remain unchanged

17

Inconsistency of the h-index

18

Newly proposed metric: Relative Citation Ratio

This metric has the peculiar property

that receiving additional citations

may cause the metric to decrease

rather than increase

19

Challenge 3: Correcting for field differences

20

Differences in citation density between fields

0 1 2 3 4 5 6 7 8 9 100

5

10

15

20

25

30

35

publication age in years

aver

age

num

ber

of c

itatio

ns p

er p

ublic

atio

n

biochemistry & molecular biologycardiac & cardiovascular systemschemistry, analyticalsurgeryeconomicsphysics, appliedmathematics

21

Differences in citation density between fields

22

Field-normalized indicators

Elsevier SciVal

THE ranking

23

Differences in citation density within fields (clinical neurology)

24

About 4000 fields of science in the CWTS Leiden Ranking

Social sciences and

humanities

Biomedical and health sciences

Life and earth sciences

Physical sciences

and engineering

Mathematics and computer science

25

CWTS Leiden Ranking

26

CWTS Leiden Ranking

27

SNIP: Source normalized impact per paper

28

Differences in citation density within fields (clinical neurology)

29

Differences in reference density within fields (clinical neurology)

30

Citation density vs. reference density (clinical neurology)

Citation density Reference density

31

SNIP: Source normalized impact per paper• Impact per paper (IPP) of journal X in 2013:

• Source normalized impact per paper (SNIP) of journal X in 2013:

C11CIPP P P

i

PR1

SNIP

C

1 i i

Number of publications in journal X in 2010–2012

Number of citations in 2013 to publications in journal X in 2010–2012

Number of references in citing publication i to publications in 2010–2012

Be careful in using SNIP in the humanities!

32

Challenge 4: Quantifying productivity

33

Quantifying productivity

• Same resources as Univ. B• P = 1000• P(top 10%) = 200• PP(top 10%) = 20%

• Same resources as Univ. A• P = 2000• P(top 10%) = 300• PP(top 10%) = 15%

Univ. A Univ. B

• Taking into account that both universities have the same resources, it is clear that university B has performed better

• However, according to the PP(top 10%) indicator, university A has performed better

34

Challenge 5: Assessing validity of metrics

35

Correlating metrics with peer review

Correlating journal impact metrics with Norwegian expert

judgments

Correlating citation metrics with F1000 recommendations (post publication peer review)

36

Correlating metrics with peer review

Correlating various metrics with UK REF expert judgments

37

Difficulties of using peer review to validate metrics• Biases in peer review, e.g.:

– Gender– Interdisciplinarity– Conservatism

• Influence of metrics on peer judgment– Did REF expert panels truly make their decisions without taking

into account any metrics?

• Limited precision of peer review outcomes doesn’t allow accurate comparisons of metrics:– Norwegian expert judgments: 3 classes– F1000 recommendations: 3 classes– UK REF expert judgments: 5 classes

38

Challenge 6: Balancing simplicity, accuracy, and robustnessto gaming

39

40

Suspicious editorials: Time trend

• At least 50 citations to publications in own journal in past 2 years

• At least 75% journal self citations

41

Suspicious editorials: Effect on IF

42

Coercive citing

“you cite Leukemia (once in 42 references). Consequently, we kindly ask you to add

references of articles published in Leukemia to your present article”

43

Balancing simplicity, accuracy, and robustness to gaming

Simplicity

Accuracy

Robustness to gaming

IF vs. SNIP?

Exclude citations from

editorials?

Exclude journal self-citations?

44

How to deal with these challenges?

45

46

Leiden Manifesto

1. Quantitative evaluation should support qualitative, expert assessment

2. Measure performance against the research missions of the institution, group or researcher

3. Protect excellence in locally relevant research4. Keep data collection and analytical processes

open, transparent and simple5. Allow those evaluated to verify data and analysis

47

Leiden Manifesto

6. Account for variation by field in publication and citation practices

7. Base assessment of individual researchers on a qualitative judgement of their portfolio

8. Avoid misplaced concreteness and false precision9. Recognize the systemic effects of assessment

and indicators10.Scrutinize indicators regularly and update them

48

Slovenia’s research strengths

49

Top 5 Slovenian research strengths

Social sciences and

humanities

Biomedical and health sciences

Life and earth sciences

Physical sciences

and engineering

Mathematics and computer science

50

Top 5 Slovenian research strengths

Keywords:Journals:

Institute:

acrylic acid; ethylene terephthalate; polypropylenesurface & coatings technology; j. of applied polymer science; plasma processes & polymersjozef stefan institute

Keywords:Journals:Institute:

cholesteric liquid crystal; holographic polymer; nanoparticlemolecular crystals & liquid crystals; physical review e; liquid crystalsjozef stefan institute

Keywords:Journals:Institute:

evolution; dilemma game; altruismj. of econ. behavior & organization; games & econ. behavior; j. of theoretical biologyuniv. of maribor

Keywords:Journals:Institute:

acidic medium; mild steel corrosion; hydrochloric acid solutioncorrosion science; int. j. of electrochemical science; corrosionjozef stefan institute

Keywords:Journals:Institute:

wood density; wood property; treeeur. j. of wood & wood products; holzforschung; forest products journaljozef stefan institute; nat. inst. of biology; univ. of ljubljana

51

Thank you for your attention!