How the top 500 ARWU can provide a misleading rank
2
Letter to the Editor How the Top 500 ARWU Can Provide a Misleading Rank Dear Sir, Every year, academia anxiously awaits the 15 th of August, when the official Academic Ranking of World Universities (ARWU) list makes its appearance. This is particularly the case for those universities that want to break into the Top 500 list for the first time. University rankings are very appealing, but the very simplicity of use can be highly mis- leading in that most rankings are based on a simple formula that aggregates subjectively chosen indicators (Paruolo, Saisana, & Saltelli, 2013; Saisana, D’Hombres, & Saltelli, 2011). Nonetheless, official rankings are quite often used as an indicator of a university’s reputation and performance (Jovanovic, Jeremic, Savic, Bulajic, & Martic, 2012; Docampo, 2013). Accordingly, officials from many univer- sities have been committed to determining which of the ARWU’s six indicators will most likely bring about a much desired Top 500 position. Since the Nobel and Field medal indicators (“Alumni” and “Awards”) may prove far from attainable, as well as the number of highly-cited researchers (HiCi) and number of articles published in Nature and Science (N&S), attention has swiftly turned to the number of articles indexed in the Science Citation Index Expanded and the Social Sciences Citation Index (SCIe and SSCI). Thus, universities—such as The University of Belgrade, hereafter referred to as UB—have introduced much stricter criteria for university professorships (especially concerning the number of papers published in SCIe and SSCI). It would seem that the UB’s results have been immediate: after having 1,650 papers published in 2008 and 1,978 in 2009, 2012 saw a peak result of 3,491 papers published. This resulted in an ARWU Top 301–400 position. With all due respect to the results accomplished, one can debate whether the total number of published papers (without incorporating aspects of quality, such as the impact factor [IF] of the journal) is a proper representation of a university’s scientific output. We gathered data on the UB’s papers published from 2008 to 2012 and the journals (see Table 1) in which the greatest improvement has been noted (the greatest differences in the number of published papers between the years 2008 and 2012) are herein shown. As can be seen, Physical Letters B saw the greatest improvement, with 74 papers published in 2012, in stark contrast to 2008 when only six papers were published. This impressive result is praiseworthy, especially when taking into account the high IF of the journal (4.569). In addition, this conclusion can apply to many other top physics journals (Journal of High Energy Physics and Physical Review Letters, among others). However, the larger number of published papers in 2012 was also the consequence of papers published in the journals Meta- lurgia International, Technics Technologies Education Management, and HealthMed (all of which have been removed from the Thomson Reuters SCIe list). In addition, the increase in the number of papers published in Serbian journals indexed in the Thomson Reuters SCIe and SSCI list also significantly contributed to the 2012 scientific per- formance of the UB. In point of fact, among the 582-paper difference for the 14 selected journals, 294 originated from journals removed from the Thomson Reuters SCIe and SSCI list or were journals published in Serbia (accounting for more than 50%). We next looked at several other universities that also placed in the ARWU Top 301–400. The universities selected were those that had a high score for the indicator PUB (the number of articles indexed in the SCIe and SSCI). The key findings are presented in Table 2 concerning the top five journals in which the greatest improvement has been noted (the greatest difference in the number of published papers between the years 2008 and 2012) for each of the universi- ties observed. As can be seen, Sao Paulo State University and the Federal University of Rio de Janeiro have a similar number of published papers in SCIe and SSCI. However, if the jour- nals that saw the greatest increase in the number of pub- lished papers for specified universities are examined further, one can easily conclude that the Federal University of Rio de Janeiro published their work in world-leading journals that are accorded a high IF. On the other hand, Sao Paulo State University channeled their scientific contribution to journals with a lower IF than their Rio de Janeiro peers. Very similar findings are obtained when comparing Sichuan and Shan- dong Universities. When taking all findings into account, it may be con- cluded that one could easily be distracted by a University’s Top 500 ARWU position and not consider which journals are the underlying reasons for such a ranking. Consequently, it is recommended that the Top 500 universities of the ARWU be more carefully examined for their scientific per- formance and that more attention be paid to a University’s publishing in world-class journals. In addition, the ARWU methodological framework could be updated to include aspects of journal quality (such as IF) rather than merely quantity (total number of published papers). © 2014 ASIS&T JOURNAL OF THE ASSOCIATION FOR INFORMATION SCIENCE AND TECHNOLOGY, ••(••):••–••, 2014
Transcript of How the top 500 ARWU can provide a misleading rank
Letter to the Editor
How the Top 500 ARWU Can Provide aMisleading Rank
Dear Sir,Every year, academia anxiously awaits the 15th of August,
when the official Academic Ranking of World Universities(ARWU) list makes its appearance. This is particularly thecase for those universities that want to break into the Top500 list for the first time. University rankings are veryappealing, but the very simplicity of use can be highly mis-leading in that most rankings are based on a simple formulathat aggregates subjectively chosen indicators (Paruolo,Saisana, & Saltelli, 2013; Saisana, D’Hombres, & Saltelli,2011). Nonetheless, official rankings are quite often used asan indicator of a university’s reputation and performance(Jovanovic, Jeremic, Savic, Bulajic, & Martic, 2012;Docampo, 2013). Accordingly, officials from many univer-sities have been committed to determining which of theARWU’s six indicators will most likely bring about a muchdesired Top 500 position. Since the Nobel and Field medalindicators (“Alumni” and “Awards”) may prove far fromattainable, as well as the number of highly-cited researchers(HiCi) and number of articles published in Nature andScience (N&S), attention has swiftly turned to the number ofarticles indexed in the Science Citation Index Expanded andthe Social Sciences Citation Index (SCIe and SSCI). Thus,universities—such as The University of Belgrade, hereafterreferred to as UB—have introduced much stricter criteria foruniversity professorships (especially concerning the numberof papers published in SCIe and SSCI). It would seem thatthe UB’s results have been immediate: after having 1,650papers published in 2008 and 1,978 in 2009, 2012 saw apeak result of 3,491 papers published. This resulted in anARWU Top 301–400 position.
With all due respect to the results accomplished, one candebate whether the total number of published papers(without incorporating aspects of quality, such as the impactfactor [IF] of the journal) is a proper representation of auniversity’s scientific output. We gathered data on the UB’spapers published from 2008 to 2012 and the journals (seeTable 1) in which the greatest improvement has been noted(the greatest differences in the number of published papersbetween the years 2008 and 2012) are herein shown.
As can be seen, Physical Letters B saw the greatestimprovement, with 74 papers published in 2012, in starkcontrast to 2008 when only six papers were published.
This impressive result is praiseworthy, especially whentaking into account the high IF of the journal (4.569). Inaddition, this conclusion can apply to many other topphysics journals (Journal of High Energy Physics andPhysical Review Letters, among others). However, thelarger number of published papers in 2012 was also theconsequence of papers published in the journals Meta-lurgia International, Technics Technologies EducationManagement, and HealthMed (all of which have beenremoved from the Thomson Reuters SCIe list). In addition,the increase in the number of papers published in Serbianjournals indexed in the Thomson Reuters SCIe and SSCIlist also significantly contributed to the 2012 scientific per-formance of the UB. In point of fact, among the 582-paperdifference for the 14 selected journals, 294 originated fromjournals removed from the Thomson Reuters SCIe andSSCI list or were journals published in Serbia (accountingfor more than 50%).
We next looked at several other universities that alsoplaced in the ARWU Top 301–400. The universities selectedwere those that had a high score for the indicator PUB (thenumber of articles indexed in the SCIe and SSCI). The keyfindings are presented in Table 2 concerning the top fivejournals in which the greatest improvement has been noted(the greatest difference in the number of published papersbetween the years 2008 and 2012) for each of the universi-ties observed.
As can be seen, Sao Paulo State University and theFederal University of Rio de Janeiro have a similar numberof published papers in SCIe and SSCI. However, if the jour-nals that saw the greatest increase in the number of pub-lished papers for specified universities are examined further,one can easily conclude that the Federal University of Rio deJaneiro published their work in world-leading journals thatare accorded a high IF. On the other hand, Sao Paulo StateUniversity channeled their scientific contribution to journalswith a lower IF than their Rio de Janeiro peers. Very similarfindings are obtained when comparing Sichuan and Shan-dong Universities.
When taking all findings into account, it may be con-cluded that one could easily be distracted by a University’sTop 500 ARWU position and not consider which journalsare the underlying reasons for such a ranking. Consequently,it is recommended that the Top 500 universities of theARWU be more carefully examined for their scientific per-formance and that more attention be paid to a University’spublishing in world-class journals. In addition, the ARWUmethodological framework could be updated to includeaspects of journal quality (such as IF) rather than merelyquantity (total number of published papers).© 2014 ASIS&T
JOURNAL OF THE ASSOCIATION FOR INFORMATION SCIENCE AND TECHNOLOGY, ••(••):••–••, 2014
References
Docampo, D. (2013). Reproducibility of the Shanghai academic ranking ofworld universities. Scientometrics, 94(2), 567–587.
Jovanovic, M., Jeremic, V., Savic, G., Bulajic, M., & Martic, M. (2012).How does the normalization of data affects the ARWU ranking? Scien-tometrics, 93(2), 319–327.
Paruolo, P., Saisana, M., & Saltelli, A. (2013). Ratings and rankings:Voodoo or science? Journal of the Royal Statistical Society Series A(Statistics in Society), 176(3), 609–634.
Saisana, M., D’Hombres, B., & Saltelli, A. (2011). Rickety numbers: Vola-tility of university rankings and policy implications. Research Policy,40(1), 165–177.
Nikola ZornicFaculty of Organizational SciencesDepartment of ManagementUniversity of BelgradeBelgrade, SerbiaE-mail: [email protected]
Aleksandar MarkovicFaculty of Organizational SciencesDepartment of ManagementUniversity of BelgradeBelgrade, SerbiaE-mail: [email protected]
Veljko JeremicFaculty of Organizational SciencesDepartment of StatisticsUniversity of BelgradeBelgrade, SerbiaE-mail: [email protected]
Published online in Wiley Online Library(wileyonlinelibrary.com).DOI: 10.1002/asi.23207
TABLE 1. The papers of UB staff published in journals indicating the greatest difference in the number of published papers from 2008 to 2012.
Source 2008 2012 2012–2008 IF 2012 Published in
PHYS LETT B 6 74 68 4.569METAL INT / 63 63 0.134*J HIGH ENERGY PHYS / 62 62 5.618VOJNOSANIT PREGL / 56 56 0.210 SerbiaPHYS SCRIPTA / 51 51 1.032TECH TECHNOL EDUC MA / 42 42 0.414*HEALTHMED / 39 39 /*EUR PHYS J C 4 43 39 5.247PHYS REV LETT 5 44 39 7.943PHYS REV D 5 34 29 4.691HEM IND 25 50 25 0.463 SerbiaSRP ARK CELOK LEK 42 67 25 0.228 SerbiaNUCL TECHNOL RADIAT 7 30 23 1.000 SerbiaTHERM SCI 4 25 21 0.838 SerbiaΣ 98 680 582
Note. *Dropped from SCIe list.
TABLE 2. Overview of the top five journals indicating the greatest difference in the number of published papers from 2008 to 2012 for each of theuniversities observed.
Sichuan Univ. Shandong Univ. Sao Paulo State Univ. Fed. Univ. Rio de Jan.
Journal IF 2012 Journal IF 2012 Journal IF 2012 Journal IF 2012
PLOS ONE 3.730 PLOS ONE 3.730 SEMIN-CIENC AGRAR 0.301 PLOS ONE 3.730HEPATO-GASTROENTEROL 0.774 CRYSTENG-COMM 3.879 J HIGH ENERGY PHYS 5.618 PHYS LETT B 4.569ASIAN PAC J CANCER P 1.271 J MATER CHEM 6.108 MICROSC RES TECHNIQ 1.593 PHYS REV LETT 7.943J APPL POLYM SCI 1.395 PHYS REV D 4.691 MYCOSES 1.278 PHYS REV D 4.691INT J NANOMED 3.463 PHYS LETT B 4.569 J CRANIOFAC SURG 0.686 EUR PHYS J C 5.247
2 JOURNAL OF THE ASSOCIATION FOR INFORMATION SCIENCE AND TECHNOLOGY—•• 2014DOI: 10.1002/asi
