Sustainability in University Rankings · Sustainability in University Rankings 21 October 2013,...
Transcript of Sustainability in University Rankings · Sustainability in University Rankings 21 October 2013,...
Sustainability in University Rankings
21 October 2013, Venice (Italy)
Rankings and Sustainability
Stefano PALEARI
University of Bergamo, Rector and CRUI, President
Results are from the research project: “Le università nel nuovo secolo: nuovi modelli per una società in trasformazione”
Team Higher Education UnBG-CRUI
Stefano Paleari, Venice, 21/10/2013 Rankings and sustainability
• Sustainability issues are to be contextualized
– Why do we use measures (rankings) in Higher Education?
– Sustainability in Higher Education
– Measures (Rankings) and Sustainability
Our approach
2
Stefano Paleari, Venice, 21/10/2013 Rankings and sustainability
• Efficient allocation of limited resources
• Accountability towards funders and society
– Universities are (also) financed by the state
• Transparency of results
• Main World University Rankings:
– Academic Ranking of World Universities (ARWU) - 2003
– Quacquarelli Symonds (QS) - 2004
– Times Higher Education Ranking (THE) – 2010
Why do we need measures for HE?
3
Stefano Paleari, Venice, 21/10/2013 Rankings and sustainability
ARWU
4
Criteria Indicator Weight
Quality of EducationAlumni of an institution winning Nobel Prizes and
Fields Medals10%
Staff of an institution winning Nobel Prizes and
Fields Medals20%
Highly cited researchers in 21 broad subject
categories20%
Papers published in Nature and Science* 20%
Papers indexed in Science Citation Index-expanded
and Social Science Citation Index20%
Per Capita Performance Per capita academic performance of an institution 10%
Total 100%
Quality of Faculty
Research Output
Stefano Paleari, Venice, 21/10/2013 Rankings and sustainability
QS
5
Criteria Indicator Weight
Teaching Faculty Student Ratio 20%
Academic Peer Review from global survey 40%
Recruiter Review from global survey 10%
Research Citations per Faculty from Sciverse Scopus 20%
Percentage of intanational students 5%
Percentage of intanational staff 5%
Total 100%
Reputation
International Orientation
Stefano Paleari, Venice, 21/10/2013 Rankings and sustainability
THE
6
Criteria Indicator Weight 2011
Reputational survey - teaching 15% 15% =
PhD awards per academic 6% 6% =
Undergraduates admitted per academic 4,5% 4,5% =
Income per academic 2,25% 2,25% =
PhD awards/ bachelor's awards 2,25% 2,25% =
Citations ResearchCitations impact (normalised avarage citations per
paper) from Scopus32,5% 30% ↓
Reputational survey - research 19,5% 18% ↓
Research income (scaled) 5,25% 6% ↑
Papers per academic and research staff 4,5% 6% ↑
Ratio of international to domestic staff 0,75% 2,5% ↑
Ratio of international to domestic students 3% 2,5% ↓
Proportion of internationally co-authored research
papers 2% 2,5% ↑
Industry Income - innovationResearch income from industry (per academic
staff)2,5% 2,5% =
Total 100% 100%
Weight 2012
Research - volume, income and
reputation
International Outlook - staff,
students and research
Teaching - the learning
environment
Stefano Paleari, Venice, 21/10/2013 Rankings and sustainability
• Methodological issues
– Subjective measures
• i.e.: Subjectivity of weights and indicators
– Biased measures
• Rankings favour hard science and publications in English language
– Reputation vs. Contemporaneous performance measures
• Advantage for big universities
• Advantage for historical universities (Nobel prize winners)
– Measures affected by exogenous factors
• Total income of institutions
Worldwide University Ranking
7
Stefano Paleari, Venice, 21/10/2013 Rankings and sustainability
World University Ranking Correlation
• Spearman coefficient:
• High level of correlation among top-100 ranked institutions
(Sample: 137 universities listed in all the three rankings)
8
ARWU 2011 THE 2011 QS 2011
ARWU 2011 1,000
THE 2011 0,684 1,000
QS 2011 0,421 0,614 1,000
Spearman correlation Matrix in year 2011 and among different rankings and among the same ranking in 2011 and 2012
di: difference among the position of the i university
n: number of universities listed
ARWU 0,994
THE 0,957
QS 0,989
2011-2012 Correlation
Stefano Paleari, Venice, 21/10/2013 Rankings and sustainability
• Top 10 universities in Times Higher Education Ranking 2012
9
“Best” University according to THE 1/2
Rank Università Students Faculty Student / Faculty
1 California Institute of Technology 2.175 425 5
2 University of Oxford 20.466 4.962 4
2 Stanford University 15.666 1.910 8
4 Harvard University 21.229 2.242 9
5 Massachusetts Institute of Technology (MIT) 10.384 1.009 10
6 Princeton University 7.567 1.172 6
7 University of Cambridge 18.396 2.957 6
8 Imperial College London 15.215 3.594 4
9 University of California, Berkeley 36.142 2.393 15
10 University of Chicago 15.438 2.168 7
Average students number "top 10" 16.268
Average students per faculty "top 10" 7
Avarage students per faculty in Italy 30
Top 10 universities in Times Higher Education Ranking 2012
Stefano Paleari, Venice, 21/10/2013 Rankings and sustainability
• Top university by country in THE 2012
“Best” University according to THE 2/2
10
Istituzione Studenti DocentiStudent to
faculty ratio
% studenti
internazionali
% graduate
students
California Institute of Technology 2.130 425 5,0 25,0% 55,4%
University of Oxford 18.703 4.962 3,8 29,2% 35,8%
ETH Zürich 14.116 1.150 12,3 33,1% 50,2%
University of Toronto 71.100 10.650 6,7 10,3% 18,8%
University of Tokyo 27.821 4.719 5,9 7,6% 49,5%
University of Melbourne 34.046 3.595 9,5 28,5% 27,3%
National University of Singapore 30.244 5.252 5,8 36,3% 21,0%
University of Hong Kong 15.863 3.046 5,2 30,0% 41,7%
Karolinska Institute 7.051 1.371 5,1 7,1% 51,6%
Peking University 38.228 2.900 13,2 5,6% 51,0%
Ludwig-Maximilians-Universität München 49.180 3.576 13,8 14,1% 14,0%
Pohang University of Science and Technology 3.217 429 7,5 1,6% 48,0%
Katholieke Universiteit Leuven 32.756 6.781 4,8 15,5% 47,2%
École Normale Supérieure 2.700 800 3,4 11,1% 77,8%
Utrecht University 30.344 2.904 10,4 6,3% 12,9%
25.167 3.504 7,5 17,4% 40,1%
Stefano Paleari, Venice, 21/10/2013 Rankings and sustainability
• Leading position of US universities: 17 out of 20, and 57 of
the top 100 in the ARWU 2011/2012
• Considering the top 500 universities, Europe with 202
overtakes America with 182 top-ranked institutions
• In Europe, with a predominant focus on broad-based and
regionally distributed support for a fairly large number of
universities, resources were more evenly distributed
• Key issue: What is Excellence?
– E.g. the University of Göttingen does rank very highly in
rankings, but it is not successful in the German Initiative on
Excellence
Europe vs. US
11
Stefano Paleari, Venice, 21/10/2013 Rankings and sustainability
• Use of a sole indicator to evaluate multi-output organisation
– They do not consider emerging aspects as sustainability and
diversity of institutions
• They measure only measurable facts
– Measure only what is measurable, not all what matters
– Describe the whole only through a part
• In sports…
• F1: who determines the result? the driver, the car, the team…
• Does one component really represent the whole?
Why do current rankings fail in measuring sustainability?
12
Stefano Paleari, Venice, 21/10/2013 Rankings and sustainability
• If universities are to challenge, rankings should refer to
universities in the same “league”,
• By contrast they do not discriminate:
– Different budget
– Different recruitment procedures
• Human capital is the most important productive factor into a
university!
– Different tuition fees
• Once again… in sports…
– NBA: the same salary cap for all the teams (with fines for those
not complying)
Do rankings improve (a fair) university competition?
13
Stefano Paleari, Venice, 21/10/2013 Rankings and sustainability 14
Funding of European HE
Source EUA’s Public Funding Observatory (June 2012)
• Many countries reduced
public funding to their
tertiary education
systems from the
beginning of the crisis
• Reduction of funding
mainly in the countries
in the south and east of
Europe
Trends in Public Funding to Higher Education Europe over the period 2008-2012
Stefano Paleari, Venice, 21/10/2013 Rankings and sustainability
* Including EU Funds
Funding per citizen (2008-2012)
15
Fonte Elaborazione propria dati EUA’s Public Funding Observatory e Word Bank Statistics
CountryFunding 2012 -
(mln €)
Population 2011
('000)
Funding per
citizen
Change 2008-
2012
Change 2008-2012
Inflation-adjusted
Norway 3.621 4.953 731 22,0% 21,0%
Sweden 6.235 9.449 660 22,0% 21,0%
Germany 24.900 81.798 304 23,0% 20,0%
France 19.800 65.434 303 8,8% 6,4%
Iceland 87 319 273 13,0% 7,2%
Ireland 1.236 4.576 270 -20,0% -21,0%
Austria 2.169 8.424 257 15,0% 13,0%
Netherland 3.232 16.693 194 10,0% 7,5%
Spain* 7.258 46.175 157 -9,5% -11,0%
UK 9.815 62.744 156 -10,0% -13,0%
Italy 6.633 60.724 109 -12,0% -14,0%
Croatia 369 4.403 84 5,3% 1,8%
Slovakia 447 5.398 83 2,1% -1,5%
Poland* 3.015 38.534 78 12,0% 8,6%
Czech Reupublic 802 10.496 76 -14,0% -17,0%
Lithuania 189 3.030 62 -19,0% -22,0%
Portugal 602 10.557 57 -1,5% -4,1%
Hungury 542 9.972 54 -20,0% -24,0%
Greece 200 11.300 18 -25,0% -25,0%
Belgium - French Community 585 n.a. 0 19,0% 16,0%
Stefano Paleari, Venice, 21/10/2013 Rankings and sustainability
Funding development: Italy
16
Fonte EUA’s Public Funding Observatory
Stefano Paleari, Venice, 21/10/2013 Rankings and sustainability
Funding development: other countries
17
Fonte EUA’s Public Funding Observatory
Stefano Paleari, Venice, 21/10/2013 Rankings and sustainability
• If rankings have to increase efficiency:
– They need to be an instrument, they should not be the objective!
– Universities should be empowered the instruments to correct
their performances
• i.e.: more autonomy in recruitment
Do rankings help universities to work more efficiently?
18
Stefano Paleari, Venice, 21/10/2013 Rankings and sustainability
• To do more with less?
– Sharp declining of academic staff
• Permanent academic staff went back to the level of 2001
– Dramatic declining of non-academic staff
– Increase in Student to Faculty Ratio
Italy: Consequence of funding drop
19
Source: Reprocessing data from MIUR e CINECA database
State universities 2012 2011 2010 2009 2008 Δ Δ %
Bachelor & Master students n.a. 1.625.787 1.665.060 1.684.726 1.699.038 -73.251 -4,3%
Tenure Academic staff 52.456 53.901 55.199 58.307 60.254 -7.798 -12,9%
Full Professors 13.841 14.532 15.169 17.174 18.218 -4.377 -24,0%
Associate Professors 15.435 15.884 16.229 16.858 17.547 -2.112 -12,0%
Tenure Researchers 23.180 23.485 23.801 24.275 24.489 -1.309 -5,3%
Non-tenured researchers 1.770 1.049 732 408 304 1.466 482,2%
Non-academic staff 55.810 57.459 58.966 61.873 69.916 -14.106 -20,2%
Studenti to Faculty Ratio n.a 30 30 29 28
Stefano Paleari, Venice, 21/10/2013 Rankings and sustainability
• Advantages
– Better transparency of performance evaluation
– Measure excellence in different missions and levels
– User-oriented
– Evaluation not only at institutional level, but also at cource level
• Drawbacks:
– Still developing (first release in February 2014)
– Costly for institution to collect and provide information and data
Something new: the U-Multiranking
20
Stefano Paleari, Venice, 21/10/2013 Rankings and sustainability
• Efficiency oriented
– Including at least one measure which considers both inputs
(staff, funding) and outputs
• Importance of network
– What does really matter? To be the first, or to add value to the
system?
– Importance of diversity
• Promote best practices, rather then the use of synthetic (and
biased) rankings
How can we improve rankings?
21
Stefano Paleari, Venice, 21/10/2013 Rankings and sustainability
• Set the «ultimate goal» for the HE system, and set rankings
accordingly
– Is the HE system designed to train the top students, or to
improve the quality of the system as a whole?
– Is the HE system designed to mitigate the tendency to inequality
in the social system?
How can we improve the HE system as a whole?
22
Stefano Paleari, Venice, 21/10/2013 Rankings and sustainability
• Sustainability is…
– Efficiency
– The role of the university in the society
– Competing in the same field (classifying universities)
– Competing with the same rules
– Setting long-term goals
Conclusions
23