On the Limits of Bibliometrics
-
Upload
scielo-scientific-electronic-library-online -
Category
Technology
-
view
588 -
download
2
description
Transcript of On the Limits of Bibliometrics
Faculté des arts et des sciences SciELO15 anos
1/15
Faculté des arts et des sciencesÉcole de bibliothéconomie et des sciences de l'information
On the Limits of Bibliometrics
Vincent LarivièreCanada Research Chair on the transformations
of scholarly communicationUniversité de Montréal
SciELO 15 anos
24 Outubro 2013
Faculté des arts et des sciences SciELO15 anos
2/15
Outline of the talk
- Technical limits
- Coverage
- Limits of indicators
- Impact Factor and prediction of citations
- Citation window
- Adverse consequences
Faculté des arts et des sciences SciELO15 anos
3/15Technical limits
- No database is covers all of the published literature.
- Particularly true in the social sciences and humanities.
- Local journals are not indexed by major databases and hence, local topics are not well covered
- No books were indexed (until recently).
Faculté des arts et des sciences SciELO15 anos
4/15
Percentage of references made to journal articles in the natural and medical sciences (Larivière et al. 2006)
55%
60%
65%
70%
75%
80%
85%
90%
95%
1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000
Year
Average SCI Biomedical Research Clinical MedicinePhysics Biology Earth and Space SciencesChemistry Psychology MathematicsEngineering
Faculté des arts et des sciences SciELO15 anos
5/15Percentage of references made to journal articles in the social sciences and humanities (Larivière et al. 2006)
Faculté des arts et des sciences SciELO15 anos
6/15Proportion of references made to conference proceedings in the natural and medical sciences (Lisée
et al. 2008)
0% 5% 10% 15% 20%
Applied Physics
Metals & Metallurgy
General Engineering
Optics
Chemical Engineering
Library & Information Science
Human Factors
Acoustics
Mechanical Engineering
Industrial Engineering
Aerospace Technology
Miscellaneous Engineering & Technology
Nuclear Technology
Civil Engineering
Electrical Engineering & Electronics
Computers
Faculté des arts et des sciences SciELO15 anos
7/15Coverage of journals as a function of the
language of publication (Archambault et al. 2006)
NSE SSH
Thomson ISI Ulrich Difference Thomson ISI Ulrich Difference
English 89% 78% 13% 90% 75% 20%
Czech 0.04% 0.3% -85% 0.2% 0.2% 8%
Russian 0.5% 0.9% -48% 0.3% 0.4% -24%
French 3.3% 3.4% -3% 3.2% 4.4% -26%
Multiple languages 0.2% 0.2% -14% 0.3% 0.5% -45%
Dutch 2.2% 2.2% 0% 1.3% 2.6% -48%
German 3.2% 3.9% -18% 3.0% 5.8% -50%
Japanese 0.4% 1.7% -74% 0.2% 0.6% -64%
Swedish - 0.1% -100% 0.1% 0.4% -69%
Spanish 0.6% 2.6% -75% 0.9% 3.0% -69%
Italian 0.1% 0.8% -83% 0.2% 1.1% -80%
Danish 0.04% 0.1% -50% 0.1% 0.3% -83%
Portuguese 0.1% 0.7% -85% 0.1% 1.0% -86%
Chinese 0.3% 2.4% -88% 0.04% 1.2% -96%
Polish 0.05% 0.7% -92% - 0.9% -100%
Arabic - 0.1% -100% - 0.3% -100%
Turkish 0.01% 0.2% -95% - 0.1% -100%
Other 0.3% 1.6% -80% 0.3% 2.4% -87%
Language
Faculté des arts et des sciences SciELO15 anos
8/15
Median age of cited literature(Archambault et Larivière, 2010)
0
2
4
6
8
10
12
14
16
1980 1985 1990 1995 2000
Âge
méd
ian
de la
litt
érat
ure
cité
e
Année
Arts ethumanités
Sciencessociales
Sciencesnaturelleset génie
Médecine
Faculté des arts et des sciences SciELO15 anos
9/15
How to correct (some of) these limits
- Creation of book databases
- Book Citation Index (but it still has an anglo-american bias!)
- Triangulate data sources
- Scopus vs. Thomson vs. Google Scholar
- Creation of local databases
- SciELO in Brazil
- Érudit au Québec
Faculté des arts et des sciences SciELO15 anos
10/15
Impact factor and its limits
Cannot be compared across disciplines
Skewness of citation distributions
Assymetry between numerator and denominator
Short citation window – especially for SSH
Includes journal self-citations
Is less and less correlated with citations received by papers
Faculté des arts et des sciences SciELO15 anos
11/15
Different citation practices: mean citation rates of papers after 2 and 10 years, 1900-2005
0
1
2
3
4
5
6
7
1900 1920 1940 1960 1980 2000
Mean - 2 year citation window
MED
NSE
SS
0
2
4
6
8
10
12
14
16
18
20
22
24
1900 1920 1940 1960 1980 2000
Mean - 10 year citation window
MED
NSE
SS
Faculté des arts et des sciences SciELO15 anos
12/15
Weakening relationship between Impact Factor and citations (Lozano et al. 2012)
y = 0.0011x - 1.9418r2 = 0.4666
y = 0.003x - 5.7759r2 = 0.579
y = -0.0049x + 9.9863r2 = 0.4788
0,00
0,05
0,10
0,15
0,20
0,25
0,30
0,35
1900 1920 1940 1960 1980 2000 2020
Annu
al c
oeffi
cient
of d
eter
min
atio
n (r2
)
Faculté des arts et des sciences SciELO15 anos
13/15
Decline of top journals (Larivière et al. 2013)
Faculté des arts et des sciences SciELO15 anos
14/15
Adverse effect
- Change of research topics
Type of journal Brazil in abstractBrazilian Journals 31%USA Journals 15%Other Journals 22%
Faculté des arts et des sciences SciELO15 anos
15/15
Archambault, É., Vignola-Gagné, É., Côté., Larivière, V. and Y. Gingras (2006). Benchmarking scientific output in the social sciences and humanities: The limits of existing databases, Scientometrics, 68(3): 329-342.
Larivière, V., Archambault, É, Gingras, Y. and É. Vignola-Gagné (2006). The place of serials in referencing practices: Comparing natural sciences and engineering with social sciences and humanities, Journal of the American Society for Information Science and Technology, 57(8): 997-1004.
Larivière, V., Archambault, É., Gingras, Y. (2008) Long-term variations in the aging of scientific literature: from exponential growth to steady-state science (1900-2004). Journal of the American Society for Information Science and Technology, 59(2): 288–296.
Larivière, V., Gingras, Y. and É. Archambault (2006). Canadian Collaboration Networks: A Comparative Analysis of the Natural Sciences, Social Sciences and the Humanities, Scientometrics, 68(3): 519-533.
Larivière, V. , Lozano, G.A., Gingras, Y. (2013). Are elite journals declining? Forthcoming in the Journal of the American Society for Information Science and Technology.
Lozano, G.A., Larivière, V., Gingras, Y. (2012) The weakening relationship between the Impact Factor and papers’ citations in the digital age. Journal of the American Society for Information Science and Technology. 63(11): 2140-2145.
Lisée, C., Larivière, V., Archambault, É. (2008) Conference proceedings as a source of scientific information: A bibliometric analysis. Journal of the American Society for Information Science and Technology, 59(11): 1776–1784.
Bibliography