How to increase one's ranking “efficiently”?
- Decision Pathway Sensitivity Analysis
Fang XuKent Business School, UKC
Centre for the Evaluation of Research Performance (CERP)
3rd Workshop on IWIPM, 1st July, 2010, UKC
IntroductionIntroduction Input data sensitivity is an important issue for any
ranking methods. Results reliabilityInfluence of decision making
Normally we wish that a ranking system has a good stability with respect the data perturbation so to have stable ranking results.
If “iceberg tip” type of data (like Nobel Prize numbers, top awards, top journals etc.) are given heavy weights in a ranking system, the results often are quite sensitive to the data perturbation – one extra award may just change the rankings so much.
An example of CAS International An example of CAS International Institution Ranking ResearchInstitution Ranking Research
Great change of scores can be found in the results due to data perturbation.
Final Results Change Rate
Institutes Data Perturbation (+/-) 5%
Institute 1 64.86%
Institute 2 56.02%
Institute 3 19.78%
Institute 4 22.41%
Institute 5 36.05%
Institute 6 10.16%
Institute 7 13.54%
Institute 8 17.63%
……. …….
One of the possible remedy…One of the possible remedy…
To use data from different part of the “iceberg” – from the base to middle to tip – important is the shape of the iceberg, not necessary the most visible part – tip.
E.g. in CAS study, indicators aimed to describe the middle part of the “iceberg” have been applied, such as top 1% publications, top 10% publications etc.
More details have been introduced by Prof. Steve Liu.
On the other hand…On the other hand… This sensitivity does raise another issue:
it may be possible to increase one’s rank rapidly by invest on these sensitive areas;
E.g. recruit a small number of certain type of personnel ( Nobel prize winners etc)
Influence to decision makers
In this talk, we show some initial analysis in this regard.
Decision pathway analysisDecision pathway analysis To simulate such decision scenarios and analyze
how one’s rank change;
From the comparison and analysis of the simulation results, it is possible to find out “efficient” way to increasing one’s rank for an organization.
We will illustrate our idea using Academic Ranking for World Universities (ARWU) produced by Shanghai Jiaotong University and World Famous University (WFU) produced by Wuhan University.
MethodologyMethodology Data recovery
Raw data is necessary for simulation analysis;
Raw data can be recovered via a sample (e.g. Harvard University) using the methods stated;
However, Răzvan Florian show that accurate raw data is hardly recovered in some ranking schemes.
(Răzvan Florian, Ad Astra 5, 2006,
www.ad-astra.ro/journal)
Several scenarios are consideredSingle indicator perturbationDual indicators perturbationChain influence of multi-indicators perturbation (Decision
Pathway sensitivity analysis)
One more Award winner
Two more Award
winners
Two more HICI
researchers
One more HICI
researcherChang
e in Rank?
One more N&S Pubs
Two more N&S Pubs
Criteria and weights of ARWU Criteria and weights of ARWU rankingranking
……..Criteria Indicator Code
Weight
Quality of Education
Alumni of an institution winning Nobel Prizes and Fields Medals
Alumni 10%
Quality of Faculty
Staff of an institution winning Nobel Prizes and Fields Medals
Award 20%
Highly cited researchers in 21 broad subject categories
HiCi 20%
Research Output
Papers published in Nature and Science N&S 20%
Papers indexed in Science Citation Index-expanded and Social Science Citation Index
PUB 20%
Per Capita Performance
Per capita academic performance of an institution
PCP 10%
Total 100%
For institutions specialized in humanities and social sciences such as London School of Economics, N&S is not considered, and the weight of N&S is relocated to other indicators.
Data sensitivity of ARWU rankingData sensitivity of ARWU ranking --- --- Single indicator Single indicator perturbationperturbation
Influence of sample universities’ ranks in the case of single indicator perturbation in an interval.
Change Rate
HICI PUB
A Universit
y
B Universit
y
A Universit
y
B Universit
y
2.50% 2 2 11 7
5.00% 7 5 16 14
7.50% 14 7 25 22
10.00% 16 8 30 26
12.50% 28 13 36 32
Data sensitivity of ARWU rankingData sensitivity of ARWU ranking --- --- multi-multi-indicators indicators perturbationperturbation Influence of A university’s ranks in the case of multi
indicators perturbation in an interval.
Change In Final Scores
HICI
2.50% 5.00% 7.50% 10.00% 12.50%
PUB
2.50% 13 15 18 25 34
5.00% 16 23 25 33 40
7.50% 25 28 34 36 51
10.00% 33 34 38 48 59
12.50% 36 42 48 56 65
Data sensitivity of ARWU rankingData sensitivity of ARWU ranking------ Chain Influence of multi indicators Chain Influence of multi indicators perturbationperturbationScenarios
Decision Pathway DescriptionChange in Rank
AUniversity
B University
Scenario 0 0 Award, 0 HICI winner, 0 N&S Pubs - -
Scenario 1 0 Award,1 HICI winner 11 5
Scenario 2 0 Award, 1 HICI winner, 2 N&S Pubs (SSCI) 15 8
Scenario 3 1 Award & HICI winner 70 45
Scenario 4 1 Award & HICI winner, 2 N&S Pubs (SSCI) 72 60
Scenario 5 1 Award & HICI winner, 6 N&S Pubs (SSCI) 83 82
Scenario 6 2 Award & HICI winners 118 82
Scenario 7 2 Award & HICI winners, 8 N&S Pubs (SSCI) 130 110
Scenario 82 Award & HICI winners, 16 N&S (SSCI), 20 other SSCI Pubs 144 127
Criteria of World Famous University Criteria of World Famous University (WFU)(WFU)
Description Indicators Weights
Research productivity
Total publications 0.3
Total citations 0.2
Research impact
Highly cited papers 0.2
Number of subjects in the rank 0.05
Research innovation
patents 0.05
Hot papers 0.1Research development Percentage of highly cited papers 0.1
Data sensitivity of WFU rankingData sensitivity of WFU ranking------ Chain Influence of multi indicators Chain Influence of multi indicators perturbationperturbation
Scenarios
Decision Pathway DescriptionChange in Rank of C University
Scenario 11 more Highly Cited paper 11
Scenario 21 more Highly Cited paper, 1 more Hot paper 48
Scenario 31 more Highly Cited paper, 1 more Hot paper & 1 more patents 48
Scenario 4 2 more Highly Cited papers 66
Scenario 52 more Highly Cited papers, 2 more Hot papers 97
Scenario 62 more Highly Cited papers, 2 more Hot papers & 2 more patents 98
Conclusions & LimitationsConclusions & Limitations
Data sensitivity has been estimated of two ranking schemes, significant changes have been detected in the case of single indicator perturbation, and dual indicators perturbation.
Particularly, more attention has been paid to decision pathway sensitivity analysis, where three universities have been selected for an illustration.
Unavoidable errors exist due to difficulties in data recovery in some ranking schemes.
Thanks to the useful discussions with AWRU and WFU groups!
Thanks for your patience!