Mapping Diversity –
The U-Multirank Approach to Rankings
Gero FederkeilWorkshop Universidade Nova de Lisboa,
29th June 2012
Presentation
Universidade Nova de Lisboa, 2012/06/29 2
Portuguese Universities in Global RankingsPortuguese Universities in Global Rankings
A (Short) Critique of Existing Global RankingsA (Short) Critique of Existing Global Rankings
An Alternative Approach: U-MultirankAn Alternative Approach: U-Multirank
Portuguese Universities in Global
Rankings
Universidade Nova de Lisboa, 2012/06/29 3
Shanghai Ranking QS Ranking THE Ranking
401- 500 U Lisboa 394 U Coimbra 301-350 U Aveiro
301-400 U Porto 401-450 U Nova de Lisboa 351-400 U Porto
401-450 U Porto
501-550 U Catolica Portuguesa
Portuguese universities are not well represent edin international rankings
Hence those rankings are not a good instrument to look at the Portuguese HE system
Presentation
Universidade Nova de Lisboa, 2012/06/29 4
Portuguese Universities in Global RankingsPortuguese Universities in Global Rankings
A (Short) Critique of Existing Global RankingsA (Short) Critique of Existing Global Rankings
An Alternative Approach: U-MultirankAn Alternative Approach: U-Multirank
The coverage of Global Rankings
Universidade Nova de Lisboa, 2012/06/29 5
Existing global rankings cover only a small minority of all universities
EUA: Global University Rankings and their Impact. Bruxelles, 2011.
The Indicators
Universidade Nova de Lisboa, 2012/06/29 6
Existing global rankings are rankings of internationally oriented research universities only
Shanghai Jiaotong Ranking QS Indicator Weight Indicator Weight
SCI publications 20 % Reputation among scholars 40 %
Publications Science & Nature 20 % Reputation among employers 10 %
Highly cited authors 20 % Citations 20 %
Nobel Prizes & Field Medals 20 % Student-staff-ratio 20 %
Alumni with NobelPrizes 10 % International students 10 %
Size 10 % International staff 10 %
The methodology: Ranking orthodoxy
Universidade Nova de Lisboa, 2012/06/29 7
Ranking of whole institutions
Ranking of whole institutions
Composite overall score
Composite overall score
League table approach
League table approach
Most users are interested in information about “their” field”
Institutional rankings give misleading averages across fields/units
Most users are interested in information about “their” field”
Institutional rankings give misleading averages across fields/units
• Composite indicators blur profiles and strengths & weaknesses
There are neither theoretical nor empirical arguments for assigning specific pre-defined weights to single indicators
• Composite indicators blur profiles and strengths & weaknesses
There are neither theoretical nor empirical arguments for assigning specific pre-defined weights to single indicators
• Small differences in the scores of indicators lead to big differences in league table positions
Give false impression of exactness (“Number 123 is better than number 127”)
• Small differences in the scores of indicators lead to big differences in league table positions
Give false impression of exactness (“Number 123 is better than number 127”)
Conclusions
Universidade Nova de Lisboa, 2012/06/29 8
1. Existing global rankings cover only a small minority of universities
2. Due to their indicators they only measure research and cover internationally oriented research universities only
3. They devaluate other profiles and missions (teaching, LLL, regional engagement)
4. They are a threat to diversity in higher education
Presentation
Universidade Nova de Lisboa, 2012/06/29 9
Portuguese Universities in Global RankingsPortuguese Universities in Global Rankings
A Critique of Existing Global RankingsA Critique of Existing Global Rankings
An Alternative Approach: U-MultirankAn Alternative Approach: U-Multirank
The Project
Universidade Nova de Lisboa, 2012/06/29 10
• Commissioned by the European Commission• 2-year feasibility project, 2009 – June 2011• Report now available:
http://ec.europa.eu/education/higher-education/doc/multirank_en.pdf
• Ján Figel, the former European Commissioner for Education, Training, Culture and Youth:“- to allow stakeholders to make informed choices; - to help institutions to position themselves and improve their performance”
• Two phases:o Design of new instrumento Testing the feasibility of new instrument
Specification of the Project
Universidade Nova de Lisboa, 2012/06/29 11
• Five dimensions:o Teaching & learningo Researcho Knowledge transfero International orientationo Regional engagement
• Development of a list of indicators to be tested in pilot project
• Development of data collection tools and processes (question-naires, definitions, FAQs, communication + feedback processes)
• Done by CHERPA consortium: CHE, CHEPS, CWTS, INCENTIM; OST
Specification of the project
Universidade Nova de Lisboa, 2012/06/29 12
• Two levels:• Institution (FIR)• Fields (FBR)
• Global sample of higher education and research institutions: • 159 (target: 150), 2/3 Europe, • 109 completed institutional questionnaires
• Two pilot fields: • Business studies• Engineering (electrical and mechanical)
The Basic Methodology
Universidade Nova de Lisboa, 2012/06/29 13
Multi-dimensional ranking
Multi-dimensional ranking
Multi-level ranking
Multi-level ranking
Grouping approachGrouping approach
• Different levels of analysis are relevant for different users
Field-based and institutional rankings
• Different levels of analysis are relevant for different users
Field-based and institutional rankings
• Rank groups instead of league tables provide more meaningful and valid information
• Rank groups instead of league tables provide more meaningful and valid information
There is no single objective ranking Each ranking reflects the ideas and preferences
of those doing them The decision about the relevance of indicators
should be left to the user
There is no single objective ranking Each ranking reflects the ideas and preferences
of those doing them The decision about the relevance of indicators
should be left to the user
++
++
++
Mapping Diversity
Universidade Nova de Lisboa, 2012/06/29 14
• A basic aim of U-Multirank is to make visible the diversity of Higher Education Institutions
• and to show excellence beyond research excellence
HOW ?
Universidade Nova de Lisboa, 2012/06/29
15
Starting point: Diversity of higher education institutions in Europe & the world
Starting point: Diversity of higher education institutions in Europe & the world
Identifying comparable institutions that can be
compared in one ranking
Identifying comparable institutions that can be
compared in one ranking
Description of horizontal diversity
Types/profiles
Description of horizontal diversity
Types/profiles
Assessment of vertical diversity
Performance
Assessment of vertical diversity
Performance
Complementary instruments of transparencyComplementary instruments of transparency
+
Mapping and Ranking
Universidade Nova de Lisboa, 2012/06/2916
1. Step: Mapping: Selection of a comparable set of universities based on institutional profiles
1. Step: Mapping: Selection of a comparable set of universities based on institutional profiles
Teaching and learning
Teaching and learning
Research involvementResearch
involvement
Knowledge exchange
Knowledge exchange
Regional engagement
Regional engagement
International orientation
International orientation
Student profileStudent profile
Example:
• Comprehensive, teaching oriented institution• Mainly undergraduate education• Low research orientation• Low international orientation• Regionally embedded (e.g. recruiting)
2. Step: Rankingof subset of comparable institutions with similar profiles
2. Step: Rankingof subset of comparable institutions with similar profiles
Multi-Dimensional Ranking for Subset of
Comparable Institutions
Universidade Nova de Lisboa, 2012/06/29 17
No composite
indicator!No composite
indicator!
No number 1 !
No number 1 !
User-driven, Personalised Ranking
Universidade Nova de Lisboa, 2012/06/29 18
A Personalised Ranking …
Universidade Nova de Lisboa, 2012/06/29 19
… Helping to make an informed choice
based on invidual preferences
… Helping to make an informed choice
based on invidual preferences
Results of the feasibility study
Universidade Nova de Lisboa, 2012/06/29 20
• Generally the concept, indicators and instruments of U-Multirank are feasible both on the institutional and the field level
• There are some problems concerning indicators, mainly on• Issues of employability• Knowledge transfer, and,• Regional engagment
• There has to be some refinement of concepts and indicators
Outlook : U-Multirank II
Universidade Nova de Lisboa, 2012/06/29 21
• New call for tender launched by the European Commission• CHE applied with partners (CHEPS, CWTS, …)• Decision at the end of July (?)• Start of the project on 1 October (?)
• 2 * 2 years• Implementation of the concept: First publication of ranking at
the end of 2013• Min. 500 institutions, institutional and 4 fields• Annual extension of number of institutions and fields• Development of a business model for a sustainable system
But after all, there still might be some
limits to ranking…
22„You‘re kidding! You count publications?“
Muito obrigado!
For more information:
www.u-multirank.eu
Mapping Diversity –
The U-Multirank Approach to Rankings
Gero Federkeil, CHE Centre for Higher Education
Workshop Universidade Nova de Lisboa,
29th June 2012
Top Related