How to design a ranking system: Criteria and opportunities for a comparison
-
Upload
nees-jan-van-eck -
Category
Science
-
view
372 -
download
2
Transcript of How to design a ranking system: Criteria and opportunities for a comparison
![Page 1: How to design a ranking system: Criteria and opportunities for a comparison](https://reader034.fdocuments.in/reader034/viewer/2022051503/5a6d1aef7f8b9a1b428b4d5b/html5/thumbnails/1.jpg)
How to design a ranking system:
Criteria and opportunities for a comparison
Nees Jan van Eck
Centre for Science and Technology Studies (CWTS), Leiden University
Workshop: “University rankings: A challenge of a foe?”
University of Milan, Milan, Italy, May 29, 2017
![Page 2: How to design a ranking system: Criteria and opportunities for a comparison](https://reader034.fdocuments.in/reader034/viewer/2022051503/5a6d1aef7f8b9a1b428b4d5b/html5/thumbnails/2.jpg)
Outline
• CWTS Leiden Ranking
• Responsible use of university rankings
1
![Page 3: How to design a ranking system: Criteria and opportunities for a comparison](https://reader034.fdocuments.in/reader034/viewer/2022051503/5a6d1aef7f8b9a1b428b4d5b/html5/thumbnails/3.jpg)
CWTS Leiden
Ranking
2
![Page 4: How to design a ranking system: Criteria and opportunities for a comparison](https://reader034.fdocuments.in/reader034/viewer/2022051503/5a6d1aef7f8b9a1b428b4d5b/html5/thumbnails/4.jpg)
CWTS Leiden Ranking
• Provides bibliometric indicators of:
– Scientific impact
– Scientific collaboration
• Calculated based on Clarivate Analytics Web of
Science data
3
![Page 5: How to design a ranking system: Criteria and opportunities for a comparison](https://reader034.fdocuments.in/reader034/viewer/2022051503/5a6d1aef7f8b9a1b428b4d5b/html5/thumbnails/5.jpg)
Selection of universities
• All universities worldwide with ≥1000 Web of
Science publications in period 2012–2015
• 902 universities from 54 countries
4
![Page 6: How to design a ranking system: Criteria and opportunities for a comparison](https://reader034.fdocuments.in/reader034/viewer/2022051503/5a6d1aef7f8b9a1b428b4d5b/html5/thumbnails/6.jpg)
Differences with other university
rankings
• No composite indicators
• Exclusive focus on scientific performance
• No survey data or data provided by universities
• Advanced bibliometric methodology
5
![Page 7: How to design a ranking system: Criteria and opportunities for a comparison](https://reader034.fdocuments.in/reader034/viewer/2022051503/5a6d1aef7f8b9a1b428b4d5b/html5/thumbnails/7.jpg)
6
![Page 8: How to design a ranking system: Criteria and opportunities for a comparison](https://reader034.fdocuments.in/reader034/viewer/2022051503/5a6d1aef7f8b9a1b428b4d5b/html5/thumbnails/8.jpg)
7
![Page 9: How to design a ranking system: Criteria and opportunities for a comparison](https://reader034.fdocuments.in/reader034/viewer/2022051503/5a6d1aef7f8b9a1b428b4d5b/html5/thumbnails/9.jpg)
8
![Page 10: How to design a ranking system: Criteria and opportunities for a comparison](https://reader034.fdocuments.in/reader034/viewer/2022051503/5a6d1aef7f8b9a1b428b4d5b/html5/thumbnails/10.jpg)
9
![Page 11: How to design a ranking system: Criteria and opportunities for a comparison](https://reader034.fdocuments.in/reader034/viewer/2022051503/5a6d1aef7f8b9a1b428b4d5b/html5/thumbnails/11.jpg)
10
![Page 12: How to design a ranking system: Criteria and opportunities for a comparison](https://reader034.fdocuments.in/reader034/viewer/2022051503/5a6d1aef7f8b9a1b428b4d5b/html5/thumbnails/12.jpg)
11
![Page 13: How to design a ranking system: Criteria and opportunities for a comparison](https://reader034.fdocuments.in/reader034/viewer/2022051503/5a6d1aef7f8b9a1b428b4d5b/html5/thumbnails/13.jpg)
12
![Page 14: How to design a ranking system: Criteria and opportunities for a comparison](https://reader034.fdocuments.in/reader034/viewer/2022051503/5a6d1aef7f8b9a1b428b4d5b/html5/thumbnails/14.jpg)
Responsible
use of
university
rankings
13
![Page 15: How to design a ranking system: Criteria and opportunities for a comparison](https://reader034.fdocuments.in/reader034/viewer/2022051503/5a6d1aef7f8b9a1b428b4d5b/html5/thumbnails/15.jpg)
Responsible use of university
rankings
• Ten principles for responsible use of rankings:
– Design of rankings
– Interpretation of rankings
– Use of rankings
• Covers university rankings in general, not only the
Leiden Ranking
• Partly builds on Leiden Manifesto
14
![Page 16: How to design a ranking system: Criteria and opportunities for a comparison](https://reader034.fdocuments.in/reader034/viewer/2022051503/5a6d1aef7f8b9a1b428b4d5b/html5/thumbnails/16.jpg)
Design of rankings
1. A generic concept of university performance
should not be used
2. A clear distinction should be made between size-
dependent and size-independent indicators
3. Universities should be defined in a consistent way
4. University rankings should be sufficiently
transparent
15
![Page 17: How to design a ranking system: Criteria and opportunities for a comparison](https://reader034.fdocuments.in/reader034/viewer/2022051503/5a6d1aef7f8b9a1b428b4d5b/html5/thumbnails/17.jpg)
Do not use a generic concept of
university performance
16
Composite
indicator
![Page 18: How to design a ranking system: Criteria and opportunities for a comparison](https://reader034.fdocuments.in/reader034/viewer/2022051503/5a6d1aef7f8b9a1b428b4d5b/html5/thumbnails/18.jpg)
Interpretation of rankings
5. Comparisons between universities should be made
keeping in mind differences between universities
6. Uncertainty in university rankings should be
acknowledged
7. An exclusive focus on ranks of universities should
be avoided; values of underlying indicators should
be taken into account
17
![Page 19: How to design a ranking system: Criteria and opportunities for a comparison](https://reader034.fdocuments.in/reader034/viewer/2022051503/5a6d1aef7f8b9a1b428b4d5b/html5/thumbnails/19.jpg)
Take into account values of indicators
18
Rockefeller University
Rank 1 (PPtop 10%
= 28.2%)
Queen Mary University London
Rank 50 (PPtop 10%
= 14.8%)
University of Bari Aldo Moro
Rank 550 (PPtop 10%
= 8.0%)
Difference in PPtop 10%
is two times larger for universities
at ranks 1 and 50 than for universities at ranks 50 and 500
![Page 20: How to design a ranking system: Criteria and opportunities for a comparison](https://reader034.fdocuments.in/reader034/viewer/2022051503/5a6d1aef7f8b9a1b428b4d5b/html5/thumbnails/20.jpg)
Use of rankings
8. Dimensions of university performance not covered
by university rankings should not be overlooked
9. Performance criteria relevant at university level
should not automatically be assumed to have same
relevance at department or research group level
10.University rankings should be handled cautiously,
but they should not be dismissed as being
completely useless
19
![Page 21: How to design a ranking system: Criteria and opportunities for a comparison](https://reader034.fdocuments.in/reader034/viewer/2022051503/5a6d1aef7f8b9a1b428b4d5b/html5/thumbnails/21.jpg)
Use of rankings for promotion
20
![Page 22: How to design a ranking system: Criteria and opportunities for a comparison](https://reader034.fdocuments.in/reader034/viewer/2022051503/5a6d1aef7f8b9a1b428b4d5b/html5/thumbnails/22.jpg)
21
![Page 23: How to design a ranking system: Criteria and opportunities for a comparison](https://reader034.fdocuments.in/reader034/viewer/2022051503/5a6d1aef7f8b9a1b428b4d5b/html5/thumbnails/23.jpg)
More information
• www.leidenranking.com
– Leiden Ranking 2017 statistics
– Methodological information
– Contact form
• www.cwts.nl/blog
– Principles for responsible use of university rankings
22
![Page 24: How to design a ranking system: Criteria and opportunities for a comparison](https://reader034.fdocuments.in/reader034/viewer/2022051503/5a6d1aef7f8b9a1b428b4d5b/html5/thumbnails/24.jpg)
Thank you for your attention!
23