Rankings and Strategic Decision-Making: Pursuing Quality in a Global

World

Professor Ellen HazelkornDirector, Higher Education Policy Research Unit (HEPRU),

Education Consultant, BH Associates, (Ireland).Global University Rankings and Their Impact, Belgrade

7 November 2017

Are Rankings the Appropriate Instrument for Achieving Excellence?

Policy and Institutional Choices

Use Rankings Carefully & Strategically

1. Are Rankings the Appropriate Instrument for Achieving Excellence?

Objectives: Development of Education, 2020

1. Increasing the quality of the process and outcomes of education to the maximum attainable level - arising from scientific knowledge on education and respectable educational practice;

2. Increasing coverage of population of the Republic of Serbia on all educational levels, from preschool education to lifelong learning;

3. Achieving and maintaining the relevance of education, particularly the one that is fully or partially funded by the public sources, by aligning the educational system structure with immediate and developmental needs of individuals and economic, social, cultural, media, research, educational, public, administrative and other systems;

4. Increasing the efficiency of use of all the education resources, i.e. completion rate within the stipulated period, with minimum extended duration and reduced dropout rates.

Objectives & Performance Indicators

• By 2020, formulate/fully implement specific policies, actions and measures to improve international competitiveness and recognition of the Serbian higher education,…to improve position of universities on credible international rankings or in the region (p130);

• All universities organise scientific research; … continuously strengthen the research capacity and excellence of universities (p156)

– Adopted strategies and action plans of science and research;

– Development of reports on international and national rankings of universities;

• Develop criteria/mechanisms for systematic monitoring, evaluation and promotion of results of scientific research, doctoral students and all persons engaged in education/research process (p159)

– An act on the classification and ranking of higher education institutions

http://erasmusplus.rs/wp-content/uploads/2015/03/Strategy-for-Education-Development-in-Serbia-2020.pdf

Rankings-led Strategy

Governments are increasingly influenced by rankings.

• Restructuring of national systems and priorities;

⎻ Excellence Initiatives used to restructure HE/research systems andinstitutions to create “world-class” or flagship universities

• Changes in research practice: language, publication, orientation, basic/applied, etc.

– Emphasis on research vs. teaching; postgraduate vs. undergraduate –with implications for the academic profession;

• Classify universities & evaluation performance;

• Criteria for collaboration, scholarships and immigration Laws;

Beware Rankings-led Strategy (1)

1. Obsession with individual HEI (global) performance confuses elite research universities with the “system”:

– Rankings-led strategy based on philosophy of picking winners, and giving them the spoils:

• Wrongly assumes the “system” is the aggregate of individual institutional performance;

• Skews our understanding of student cohort:

– Gauging performance by looking only at the top-20 or the ‘winners’ creates perverse benchmarks.

Beware Rankings-led Strategy(2)

3. Prestige and reputation become dominant drivers of “system” leading tosteep(er) hierarchy – rather than pursuance of equity and diversity –

– Measure resource-intensity, and reward sustained concentration and selectivity in a few elite universities;

– Drives isomorphism and undermines mission diversity;

– Amplify benefits and prestige of elite universities and their graduates,

4. Concentrating resources will undermine society/national objectives:

– Drives growing social and educational stratification;

– Widens privilege gap, affecting HEIs/their students, threatening cities and regions where reside, exaggerating long-standing inequality issues;

– No evidence more concentrated national systems generate higher citation impact.

Beware Rankings-led Strategy (3)

5. Rankings affect/reorient research priorities and practices:

─ Emphasis on global reputation is undermining nationally/regionally relevant activity and outcomes;

– Fails to capture activity across the full research-innovation eco-system;

6. Privilege global reputation over societal responsibility

⎻ Undermine teaching mission and service to society;

⎻ Big implications for research which has regional/national significance and value.

7. Financial costs can be very high – and threaten other policy goals.

Rankings Produce Perverse Outcomes

• There is no such thing as an objective ranking

• Because:

- The evidence is never self-evident

- Measurements are rarely direct but consist of indicators

- Weightings of indicators reflect value-judgements on priorities

• Assessment methods must therefore be related to function and purpose

2. Policy and Institutional Choices

World-class-Universities Or A World-class System?

To move towards a mass knowledge society (where progressdepends on the “wisdom of the many”) or towards an eliteknowledge society (where progress depends on the cutting-edgeknowledge of the chosen few)?

To improve the capacity and quality of the whole system – orreward the achievements of elite flagship institutions?

What is a World-class University?

• No convincing definition of “world-class university” despite the fact that the term is widely used.

– Almost everyone agrees that top research performance andprominence in the rankings are key characteristics.

– However, quality of student learning, internationalization, regionalengagement, societal impact, entrepreneurship, educationalrelevance, etc. are also important – but rarely discussed.

– No one talks about “world-class teaching universities” or “world-classvocational colleges”— research and the rankings dominate this fuzzyterm.

Characteristics of a W-C University

Excellence Initiatives

• Excellence strategies in 30+ countries: France, Germany, Russia, Spain, China, South Korea, Taiwan, Malaysia, Finland, India, Japan, Singapore, Sri Lanka, South Africa and Latvia, etc.

• Characteristics:

– Emphasis on concentrating resources in handful of universities;

– Actions: special government subvention, preferential research investment, recruitment strategies to attract high-achieving domestic and especially international students and faculty, etc.

– Some governments have devised classification systems or national rankings to measure which institutions are at the top;

– Focus almost exclusively on improving research performance and to some extent linking research to the economy;

– Little, if any, attention is paid to teaching quality or to student achievement;

– Success measured in terms of climbing rankings ladder.

Building a World-class System

• “At the end of the day, world-class systems are not those that can boast the largest number of highly ranked universities. They are, instead, those that manage to develop and sustain a wide range of good quality and well articulated tertiary education institutions with distinctive missions, able to meet collectively the great variety of individual, community and national needs that characterize dynamic economies and healthy societies.” (Salmi,

2014)

• “For society, it means that we must look at the tertiary education system not simply as a mechanism for churning out a handful of elites and perpetuating social inequality…; to the contrary, the system must be capable…of empowering and equipping the largest possible number of individuals with the fullest set of tools she or he will need to become well-rounded participants in our social democracy and fully-functioning economic units in that society.” (Lisbon Council, 2008)

World Class University Model: Hierarchical/Reputational Differentiation

World Class System Model: Horizontal/Field or Mission Specialisation

Gavin Moodie, correspondence 7 June 2009

Choices Matter

• Being part of global science is vital. No country can afford to be on the wrong side of history.

• Nations are right to develop the human capital and knowledge infrastructure necessary to compete successfully in the global economy.

• Are there alternative strategies which can benefit the whole system and society?

• Can being smart with limited resources can pay off?

Research/Science Strategy

If an objective is to boost or improve scientific-scholarly output and impact, there are various actions to consider taking:

• Develop a national science/research strategy;

• Identify national/institutional priorities;

• Build Collaborative National Research Clusters and Critical Mass by emphasizing collaboration,

• Create National Graduate Schools

• Leverage Global Potential of Regional Specialization (aka EU Smart Specialization Strategy

Improve Learning Outcomes Strategy

• If the objective is to improve educational outcomes for individuals and society, there are various actions to consider taking:

• Build regional comprehensive universities;

• Encourage collaboration to pool resources to strengthen the expertise and quality of education programmes and support services,

• Use funding strategically to reinforce national objectives;

• Encourage stronger links between vocational, further and higher education;

• Widen access and broaden educational opportunities,

• Take a ‘whole of education’ perspective to ensure the educational system from cradle to career is well aligned.

Different Policy Choices: Ireland

• National Strategy for Higher Education to 2030

• Innovation 2020 (replaces Strategy for Science, Technology & Innovation 2015-2020)

• National Performance Framework

• National Prioritisation Exercise w/selective prioritisation/challenge-centric research

• Irish Research Council – supporting excellence/frontier research across all disciplines, w/ emphasis on ECR

• Strategic dialogue – focus on aligning HEI with national objectives

• Quality Assurance via Peer Review and International Benchmarking

• Emphasis on collaboration and shared services;

• Emphasis on equity and access;

• Regional Skills Fora & Regional Clusters

• Link between FE/HE and secondary

3. Use Rankings Carefully & Strategically

Alternative Ways to Measure & Compare (1)

• Profiling

– U-Map (EU)

– Institutional Profiles (Ireland, Norway, Australia)

• Multi-dimensional Rankings/Banding

– U-Multirank (EU)

– CHE-HochschulRanking (Germany)

• System-level Rankings

– Universitas 21 (Australia)

– OECD System Benchmarking

• Benchmarking

– Compare performance to competitor or to “good practices

Alternative Ways to Measure & Compare (2)

• Measuring Value to Community, Value-for-Money

– Washington Monthly (US)

– HE Innovate (EU)

– E 3 M – Societal Engagement Indicators (EU)

– Education Scorecard (US Government)

– Carnegie Engagement Classiciation (US)

• Assessment of Learning Outcomes

– Survey of Student Engagement (US + Canada, Australia, China, South Africa, New Zealand, Ireland)

– Collegiate Learning Assessment (US)

Align Metrics and Indicators to National Objectives

• What are you trying to achieve?

– Metrics must be fit for purpose

• Common purposes include:

– Allocation of resources

– Improvement of education outcomes

– Improvement of research performance

– Driver of mission differentiation

– Widen access and participation – socio-economic, gender, other under-represented, etc.

– Attraction of talent

– Promotion of innovation

– Societal Impact

– Engagement with business, Etc.

Some Simple Steps

1. Ensure data submitted is accurate: faculty involved in teaching/research are properly counted, items reflected in "institutional income" are appropriate; research income;

2. Ensure common attribution to be used by everyone for all publications, e.g. ensure the same university name and address;

3. Conduct search of publications to i) check journals included in WoS, ii) university listed correctly in co-authored articles

4. Greater communication about university and its research to boost reputational impact;

5. Establish an IR office to ensure professional competence, and accurate and strategic data collection, monitoring and analysis.

Use Rankings Carefully & Strategically

• What is your institution trying to achieve?

– What are your institution’s profile, mission and goals?

– How much, if any, attention should your institution pay to rankings?

• How is your institution trying to do it?

– What are your institution’s goals? How is progress measured?

– Are rankings the most appropriate measurement?

• How does your institution know its strategy is working?

– How does your institution assess its performance?

– What is your institutional research capacity? How can this be enhanced?

• How does the institution change in order to improve?

– Would other transparency instruments be more useful for benchmarking and helping improve performance?

Dos and Don’ts

Don’t

• Change your institution’s mission to conform with rankings;

• Use rankings to inform policy or resource allocation decisions;

• Direct resources to a few and neglect the needs of the university;

• Manipulate public information and data in order to rise in the rankings.

Do:

• Ensure your university has a coherent strategy/mission;

• Use rankings only as part of an overall quality assurance, assessment or benchmarking system and never as a stand-alone evaluation tool;

• Be accountable and provide good quality public information about learning outcomes, impact and benefit to students and society;

• Engage in an information campaign to broaden media and public understanding of the limitations of rankings.

Summary

• Performance and quality of HE is vital to success in the global economy.

• Cross-country comparisons are both common and essential.

• Rankings appear to provide a simple and useful way to measure and compare quality and performance.

BUT:

• Big lessons of global rankings is the extent to which higher education (policy) has become vulnerable to an agenda set by others;

• Using rankings distorts public policy;

• Threatens national sovereignty and institutional autonomy!

Contact Information:

Higher Education Policy

Research Unit (HEPRU),

Dublin Institute of Technology,

Ireland

BH Associates – Education

Consultants, Ireland

ellen.hazelkorn@dit.ie

info@bhassociates.eu

https://www.dit.ie/hepru

https://www.bhassociates.eu

Serbia in Rankings

Serbia in Global University Rankings

• University of Belgrade features in most major rankings (ARWU, QS, World University Rankings, Leiden, but not THE);

• University of Novi Sad features in Leiden and QS Eastern Europe and Central Asia (QS EECA);

• U-Multi-Rank: Blegrade, Educons, Kragujevac, Novi Sad

2018 2017 2016 2015

ARWUUniversity of

Belgraden/a 201-300 201-300 301-400

QS WorldUniversity of

Belgrade801-1000 701+ 701+ 701+

QS EECA

University of Belgrade

54 66 66 -

University of Novi Sad

161-170 121-130 131-140 -

THE BRICs and Emerging Economies

University of Belgrade

n/a 196 150 -

THE BRICs and Emerging Economies Ranking

• Three “types” of country included:

1. Advanced Emerging: Brazil, Czech Republic, Greece, Hungary, Malaysia, Mexico, Poland, South Africa, Taiwan, Thailand, Turkey

2. Secondary Emerging: Chile, China, Colombia, Egypt, India, Indonesia, Pakistan, Peru, Philippines, Qatar, Russia, UAE

3. Frontier: Bahrain, Bangladesh, Botswana, Bulgaria, Cote d'Ivoire, Croatia, Cyprus, Estonia, Ghana, Jordan, Kenya, Latvia, Lithuania, Macedonia, Malta, Mauritius, Morocco, Nigeria, Oman, Palestine, Romania, Serbia, Slovakia, Sri Lanka, Tunisia, Vietnam

Has a greater focus on Teaching than the main, global ranking, and less reliance on reputation surveys.

2017 2016 2015

University of Belgrade 196 150 -

Comparison Countries 1

• Number of universities: Serbia 17; Bulgaria 41; Belarus 23. Note, this only includes full universities, not polytechnics etc. [Source: studyinserbia.rs, studyinbulgaria.com,

studyinbelarus.com.ng]

• GDP per capita (2016): Serbia 5,376.261 USD; Bulgaria 7,368.516 USD; Belarus 5,142.893 USD. [Source: IMF World Economic Outlook Database]

• Population (2017): Serbia 8,790,574; Bulgaria 7,084,571; Belarus 9,468,338. [Source United Nations World Population Prospects 2017 revision]

• Number of universities in main relevant rankings:

ARWUQS

WURQS EECA THE WUR

THE BRICs & EE

Leiden U-Multirank

Belarus 0 2 2 1 0 0 5

Bulgaria 0 1 3 1 1 0 11

Serbia 1 1 2 0 1 1 4

Comparison Countries 2

• Number of universities: Croatia 10; Serbia 17; Slovenia 4. Note, this only includes full universities, not polytechnics etc. [Source: studyincroatia.hr, studyinserbia.rs,

studyinslovenia.si]

• GDP per capita (2016): Croatia 12,095.483 USD; Serbia 5,376.261 USD; Slovenia 21,320.160 USD. [Source: IMF World Economic Outlook Database]

• Population (2017): Croatia 4,189,353; Serbia 8,790,574; Slovenia 2,079,976. [Source United Nations World Population Prospects 2017 revision]

• Number of universities in main relevant rankings:

ARWU QS WUR QS EECA THE WURTHE BRICs &

EELeiden U-Multirank

Croatia 0 1 2 2 1 1 4

Serbia 1 1 2 0 1 1 4

Slovenia 1 2 4 2 2 1 4

Implications

• DATA: More/all Serbian universities should endeavour to supply data to U-Multirank.

• RESEARCH: Consider supports for academics to publish through English for inclusion in the “core journals” featured in databases such as Scopus, Web of Science.

• BIBLIOMETRICS: Engage with Elsevier and Thomson Reuters to ensure that relevant Serbian journals are indexed in these databases.

• INTERNATIONAL OUTLOOK: Internationalisation is a feature of the major rankings, and an improvement in the ratios of international to local staff and students can be improved through targeted recruitment.

• REPUTATION: University of Belgrade does well in ARWU, but less so in QSWUR and doesn’t appear in THE WUR at all. Both THE and QS rely on reputation surveys, so this count against Serbian HEIs.