USING WORLD UNIVERSITY RANKING SYSTEMS TO INFORM AND GUIDE STRATEGIC POLICY MAKING. A CASE STUDY OF A CANADIAN RESEARCH UNIVERSITY

AC21 International FORUM 2010Shanghai Jiao Tong University

Roland ProulxUniversité de Montréal

- Canada

PURPOSE PURPOSE OF THE PRESENTATION THPRESENTATION

The intent of this paper is to illustrate, in the format of a casestudy, how the international rankings have initiated andinformed a strategic thinking and planning process at the University of Montreal.

The methodology used:• Set the strategic position of the University from a global perspective :

assessing the “overall scores” of the rankings as a general referenceand addressing, beyond the “overall scores”, the various individual

performance indicators;• Apply, according to the intent of the organization, the various scores

and indicators to a formal strategic decision making process.

Outline of the presentation

The Findings

Ranking of University of Montreal according to the “overall scores”

Ranking of University of Montreal according to selectively chosen research indicators

Diagnosis and strategies

THE FINDINGS

FINDING # 1UNIVERSITY’S WORLD POSITION ACCORDING THE FINAL SCORES (2008)

RANKINGS World North America Europe Canada TAIWAN 94 (62)* 63 (45) 24 (12) 5 (4) THES 91 38 35 5 ARWU 101 59 35 6

LEIDEN (crown indicator) 125 87 38 5

WEB (2009) 63 54 9 5 RESEARCH INFOSOURCE na na na 6

The University of Montreal is clearly ranked among the top 100 world universities, though on the border line on most of the rankings. On the North American scene, the University does not make the top 50, though it does rank among the top 50 European universities. Among the 15 Canadian research universities with medical schools, the rank of the University of Montreal fluctuates from position 4 to 6.

FINDING # 2 Ranking of University of Montreal according to selectively chosen research indicators

• For the purpose of a more targeted and more limited analysis of the University of Montreal as a world class research intensive university, the University have decided to evaluate and position its performance mostly against the various research indicators, thus considering the indicators per se without the biases of the selection of criteria, the score calculating and sorting, adopting a customizing process for planning purposes (the CHE model).

• Marginson (2008) has rightly argued that “Each indicator has distinct meanings for measures of value for the k-economy, and for policies of innovation and performance improvement”

The research performance indicators used

performance indicators in research according to 3 sub-categories

33%24%

43% publications (7)citations (9)impact (5)

Université de Montréal: its position according to the ranks of the indicators: world, canada, north america, europe

criteria source indicators weight world Canada N.America Europe

productivity ARWU Score on SCI (2007) 20% 63 5 38 14

  LEIDEN Number of publications (2003-2007) N/A 117 5 56 43

  TAIWAN Number of articles (1997-2007) 10% 84 6 57 36

  TAIWAN Current articles (2007) 10% 67 5 45 23

 Excellence THES Citations/Faculty (2002-2007) 20% 104 4 48 37

& intensity LEIDEN Citations / publication (2003-2007) N/A 104 5 69 35

  ARWU Score on HiCi 20% 194= 12= 113 62

  TAIWAN HiCi papers (1997-2007) 15% 92 6 37 29

  TAIWAN Hi-Index (2006-2007) 20% 76 4 30 19

  TAIWAN Hi-Impact journal articles (2007) 15% 77 5 24 22

  ARWU Score on N&S (2003-2007) 20% 170= 6= 87 66

impact  TAIWAN 11 year citations (1997-2007) 10% 93 6 70 34

  LEIDEN Number of citations (2003-2007) N/A 110 5 63 40

  TAIWAN number of citations (2006-2007) 10% 76 4 56 21

  TAIWAN Average # of citations (1997-2007) 10% 79 8 58 64

  LEIDEN CPP/FCSm (2003-2007) N/A 125 5 87 38

Summarizing the posisitions

The various indicators related to research productivity and impact tend to give a good mark to the University of Montreal (with the exception of the Leiden Ranking 2008), being ranked in the top 100

But the excellence/intensity indicators do not compete as well…

On most indicators, the University performs well on the European scene, being among the top 50, but is unable to make the top 50 North American Universities and reach the first positions among the Canadian research universities

As a whole, the most indicators are borderline…

Diagnosis and Strategies

The reputation of University of Montreal

• The THES "Peer review" indicator classifies the University of Montreal to the 58th rank among the top 100 universities worldwide, to the 25th rank among the North American universities and to the 14th rank among the European universities.

Diagnosis Doubts may be expressed whether the specific indicators do reflect and

support such a reputation and classification, at least for how long! The research performance of the University of Montreal remains fragile and precarious and may not substantiate and support its reputation.

The productivity indicators are on the border line of the top 100 world-wide and of the top 50 North American

The excellence/intensity and impact indicators have serious problems on the world and North American scenes

the Institution have not succeeded to improve its position among the Canadian research universities.

As a whole, we notice a lost of performance when shifting from productivity indicators to intensity and impact indicators

the University’s future strategic direction and plan related to research are at stake

Formulating Strategic Goals

• The University of Montreal has no choice: it must consolidate and improve significantly its performance in scientific publications and citations if the Institution aims at maintaining its enviable international reputation.

• The University must then take action to position its research performance clearly under the threshold of the top 100 world universities

(preferably under the top 75) the top 50 North American universities (preferably under the top 40); the University of Montreal must also aim at being clearly ranked

among the 3 first Canadian research universities.

Crafting Strategies

To confirm the international stature of the University, 2 sets of well targeted strategies must be implemented.

Strategy #1

The first set would aim at recovering the numerous publications and citations not counted in or left aside by the major databases: Thomson Reuters, Scopus and Google Scholar.

The problems of language Absence of institutional signature ISI Highly Cited Researchers missing

Strategy # 2

A second set of strategies to be crafted, beyond the potential recovery that should yield the bibliometric evaluation of the databases, must deal with the increase in the number, quality and impact of the publications attributed to the University of Montreal.

Mapping the scenarios

• Three Canadian research universities (Toronto, UBC and McGill) are clearly ranked among the top 100 world universities by all ranking systems and two others are listed among the top 100 according to some systems. • Being easily benchmarked with full access to reliable data ((publications numbers, citations numbers, citations per publications and field normalized average impact), the first five universities will serve here as the main reference for the various mapping scenarios.

Scientific publications, citations per faculty and field nornalized impact (2003-2007) Top six Canadian research universities

PUBLICATIONS CITATIONS FT FACULTY PUB/FAC CIT/FAC CIT/PAPER CCP/FCSm

TORONTO 31 170 251 802 2 400 13,0 105 8,1 1,45 UBC 18 679 139 266 2 190 8,5 64 7,5 1,51 McGILL 17 241 131 902 1 590 10,8 83 7,7 1,41 ALBERTA 14 224 85 414 1 602 8,9 53 6,0 1,18 MONTREAL 12 840 84 304 1 887 6,8 45 6,6 1,26 McMASTER 9 428 66 754 1 194 7,9 56 7,1 1,41

Sources: Thomson Reuters reported by University of Toronto, Research Infosource and Leiden ranking.

Implementing the scenario

To align the research performance of the University of Montreal from its present world positions to the 3d and 4th quartiles of the top 100 world universities and to position itself among the 3 first Canadian research universities –in other words to live up to its reputation (THES) and visibility (WEB),

The average number of publications and citations should massively be increased;

Accordingly the average normalized field impact (CPP/FCSm) of the papers published by the Faculty members, the ratio of 1, 26 should be increased: a special care should then be given to publishing in high impact fields and journals.

Concluding remarks

• The author of this paper has argued that the world university rankings can reasonably inform an effective decision making process, if such a process goes beyond the overall scores to take full advantage of the individual indicators, being selectively chosen according to the strategic intent of the institution and linked to a formal strategic environmental thinking exercise leading to operational changes.

We have then learned that

In encapsulating the rankings with a single composite index and focusing mostly on world “top” universities, the league tables have somehow hidden and concealed the very broad information used for their production.

We must restore the intrinsic value and individual indicators: "Each indicator has distinct meanings for measures of value for the k-economy, and of policies for innovation and performance improvement" (Marginson 2008: 17).

In this sense, the individual indicators provided by the rankings draw the contours of a worldwide genuine public information system capable of producing a customized ranking “à la carte” of universities (Bourdin Report 2007-2008).