Ranking

Journals

Using

Altmetrics

© Imperial College LondonPage 1

15th International Society of Scientometrics and Informetrics

Conference, Istanbul

30th June 2015

Tim Evans

Centre for

Complexity Science

Work done with:

Tamar Loach

© Imperial College LondonPage 2

Ranking Journals Using Altmetrics

• Traditional Journal Metrics

• The Altmetric Data

• Journal Ranking from Altmetric Data

• Results

• Visualisation

• Conclusion

Paper Citation Network OR Journal Citation Network

© Imperial College LondonPage 3

Paper

Journal A Journal B

A

B

3

2

3

2

Journal or Paper Citation Network?

• Measure for each paper then aggregate

over papers in journal

– e.g. Impact Factor

• Aggregate papers in journal, then analyse

citations between journals e.g.

– e.g. EigenFactor

Order makes a difference when distributions

are fat-tailed and methods are non-linear.

© Imperial College LondonPage 4

Traditional Journal Metrics Used Here

We used

• Impact Factor 2013

– Number of citations in 2013 to articles

published in journal in 2011 & 2012,

divided by number of articles published in

‘11 and ‘12.

• EigenFactor 2013 [Bergstrom 2007]

– PageRank on Journal citation network

© Imperial College LondonPage 5

Note both based on the same

Web of Science database

Traditional Journal Metrics

• Aim to measure attention received by a

journal

• Based on citations

• Simple Citation Counts per paper

• Count citations weighted by the

importance of source (self-consistent,

PageRank style)

© Imperial College LondonPage 6

© Imperial College LondonPage 7

• Traditional Journal Metrics

• The Altmetric Data

• Journal Ranking from Altmetric Data

• Results

• Visualisation

• Conclusion

Altmetric Data

• Data from altmetric.com

• 20 months ending June 2013

• Sources include

– Facebook

– Twitter

– BBC

– Blogs

– …

© Imperial College LondonPage 8

Information in our altmetric.com data

• An account can be one or many people

• One person may contribute to several accounts

© Imperial College LondonPage 9

A1

P1

J1 Journal identifier

Paper identifier

Author (account and source)

© Imperial College LondonPage 10

• Traditional Journal Metrics

• The Altmetric Data

• Journal Ranking from Altmetric Data

• Results

• Visualisation

• Conclusion

Simple Altmetric Counts

Rate Journals by counting altmetric mentions

Example: CA = Combined altmetric.com

Counts number of mentions of articles

published in a journal weighted by source

– Blog or News item worth more than Twitter mention

© Imperial College LondonPage 11

>,

Pairwise Comparison by each Author

• No statement

about a journal if

author does not

read its articles

© Imperial College LondonPage 12

A1

P1

J1

P2 P3

J2

P4

J3

Authors Define their own Field

Pairwise comparison

by each Author

means comparisons

are within their field

© Imperial College LondonPage 13

A1

P1

J1

P2 P3

J2

P4

J3 J4

J5

Count Mentions

© Imperial College LondonPage 14

A1

J1 J2 J3

2

1

1

0

0

Jaj=

j

a-th column of matrix J

counts mentions by author a

a=A1,

j=J1,J2,J3,J4,J5

J4

J5

Combine Counts using Author Comparison

© Imperial College LondonPage 15

Score for journal j

when compared to

journal l

number of mentions

of journal j by

authors who mention

BOTH j and l,

divided by number of

such authors

𝑆𝑗𝑙 =1

𝐴𝑗𝑙

𝑎∈𝐴𝑗𝑙

𝑆𝑎𝑗

Sjl =

Sjl =

, Ajl = set of authors who

mention papers in

both journal j and l

1

j l

32

Combine Counts using Author Comparison: Example

© Imperial College LondonPage 16

• Author 1 reads journal j

twice but journal l once.

• Author 2 reads both

journals once.

• Author 3 does not read

journal j.

𝑆𝑗𝑙 =1

𝐴𝑗𝑙

𝑎∈𝐴𝑗𝑙

𝐽𝑎𝑗

a=2 a=3

a=1

j l

Sjl = ½(2+1) = 1.5

Slj = ½(1+1) = 1.0

Combine Counts using Author Comparison: Example

© Imperial College LondonPage 17

• Author 1 reads journal j

twice but journal l once.

• Author 2 reads both

journals once.

• Author 3 does not read

journal j.

𝑆𝑗𝑙 =1

𝐴𝑗𝑙

𝑎∈𝐴𝑗𝑙

𝐽𝑎𝑗

a=2 a=3

a=1

j l

Sjl = ½(2+1) = 1.5

Slj = ½(1+1) = 1.0

Combine Counts using Author Comparison: Other

© Imperial College LondonPage 18𝑃𝑗𝑙 =

1

𝐴𝑗𝑙

𝑎∈𝐴𝑗𝑙

Θ 𝐽𝑎𝑗 − 𝐽𝑎𝑙

a=2 a=3

a=1

j l

Pjl = ½(1+0.5) =0.75

Plj = ½(0+1) =0.25

Many, many alternatives[Langville & Meyer, 2012]

e.g. 1 point for a ‘win’,

0 for a ‘loss’,

½ for a ‘draw’

Produce Journal Ranking and Rating

Once you have a journal-journal

score matrix, Sjl, Pjl, etc etc

apply use centrality measures

• PageRank

• HITS

• PSR = Points Spread Rating

• etc, etc …

© Imperial College LondonPage 19

[Langville & Meyer, 2012]

© Imperial College LondonPage 20

• Traditional Journal Metrics

• The Altmetric Data

• Journal Ranking from Altmetric Data

• Results

• Visualisation

• Conclusion

Overall Comparisons

• Basic Multivariate Statistics for overview of

behaviour of different ratings

– Hierarchical Clustering

– Principle Component Analysis

• Look at top forty or so journals by eye

© Imperial College LondonPage 21

© Imperial College LondonPage 22

EF=EigenFactor

IF=Impact

Factor

CA=Combined

altmetric.com

Most rating

measures

similar

except

PSR

(Points

Spread

Rating)

One source, several rating variations

[Spearman Correlation Matrix]

Different Rating Methods

Most

sources

reasonably

similar

but they do

show

interesting

variations

© Imperial College LondonPage 23

[Spearman Correlation Matrix]

Different Sources

Blogs

© Imperial College LondonPage 24

EF=EigenFactor

IF=Impact

Factor

[Spearman Correlation Matrix]

Blogs & Other Sources

Blogs are

closest to

traditional

standard

rankings:

Impact

Factor

&

Eigen-

Factor

Blogs

© Imperial College LondonPage 25

EF=EigenFactor

IF=Impact

Factor

[Spearman Correlation Matrix]

CA = Combined altmetric.com

Blogs

CA(combined

altmetric.com)

weighted

altmetric

count

balances

source

differences

nicely,

much

closer to

IF & EF

CA=Combined

altmetric.com

© Imperial College LondonPage 26

EF=EigenFactor

IF=Impact

Factor

[Spearman Correlation Matrix]

Simple Alt Media Counts

CA=Combined

altmetric.com

Simple

counts of

alt. media

are often

similar to IF

& EF:

CA (2),

Blind Count

(1),

Paper

Count (7),

Social

Media

Count (14)

BC (1) &

SMC (14)

Paper

Count (7)

© Imperial College LondonPage 27

Q, H, news Q, H,Blogs S,PR,blogs IF 20131 Nature Nature Nature N.E.J.Medicine2 PNAS PNAS Science Rev.Mod.Phys.3 PLoS ONE Science PNAS Chemical Rev.4 Science PLoS ONE PLoS ONE Nat.Rev.Genetics

5 N.E.J.Medicine N.E.J.Medicine N.E.J.Medicine The Lancet6 B.M.J. B.M.J. B.M.J. Nature7 Nature Comm.s The Lancet The Lancet Nature Rev.Mol.Cell Bio.

8 JAMA JAMA JAMA Ann.Rev.Immunology

9 The Lancet Proc.R.Soc.B: Bio.Sci PLoS Biology Nature Materials

10 Pediatrics Current Biology Current BiologyNature Genetics11 Psychological Sci. Pediatrics PLoS Medicine Nat.Rev.Cancer12 Current Biology Psychological Sci. Proc.R.Soc.B: Bio.Sci Adv. in Physics13 Sci. Reports J.Neuroscience Psychological Sci. Nat.Rev.Immunology

14 Cell Am.J.Clinical Nutrition Pediatrics Nat.Rev.Drug Discovery

15 Nature Medicine PLoS Medicine Cell Nat.Biotechnology

© Imperial College LondonPage 28

• Traditional Journal Metrics

• The Altmetric Data

• Journal Ranking from Altmetric Data

• Results

• Visualisation

• Conclusion

Cube Box Space

• Each journal has D

rankings

• Connect

from journal j to

journal l

iff

𝑗𝑟 < 𝑙𝑟 ∀ 𝑟 = 1,2,⋯ , 𝐷

© Imperial College LondonPage 29

D=2

[Bollobás & Brightwell 1991]

Ranking 1

Rankin

g 2

Journal Top Trumps: Nature vs. PRL

© Imperial College LondonPage 30

Nature

Nature is a weekly, international, interdisciplinary journal of science, published by MacMillan Publishing.

Physical Review

Letters

The American Physical Society publishes Physical Review Letters with short article

on fundamental and interdisciplinary physics research.

Connect as Nature higher than PRL on all three ratings

Journal Top Trumps: Nature vs. NEJM

© Imperial College LondonPage 31

The New England

Journal of Medicine

The mission of NEJM has been to bring physicians the best research at

the intersection of biomedical science and clinical practice.

Nature

Nature is a weekly, international, interdisciplinary journal of science, published by MacMillan Publishing.

No connection as Nature beats (loses) NEJM on EF & CA (IF)

Box Space Representation of

Journal Rankings

© Imperial College LondonPage 32

• Node = Journal

• Directed Edge

if journal has

better IF, EF

& CA

• Height

=

minimum

longest

path

distance

to root

• Top 1000

journals

used

• Top 40

shown

wors

e

Hasse diagram technique[Brüggeman et al. 1994]

© Imperial College LondonPage 33

• Traditional Journal Metrics

• The Altmetric Data

• Journal Ranking from Altmetric Data

• Results

• Visualisation

• Conclusion

Conclusions

• Shown how to extract journal ranking from

mentions of papers in alternative media

sources

• Shown that most ranking schemes are

reasonable

– But not Point Ranking Scheme

• Some interesting differences

– To understand why need some bibliometric input

• Which scheme is best?

– Axiomatic, Social science criteria?

© Imperial College LondonPage 34

Tim Evans, Imperial College London

http://netplexity.org

Work done with: Tamar Loach

Thanks

Work done in

collaboration with Tamar Loach

• Euan Adie (altmetric.com)

– provided and explained the altmetric data

• Daniel Hook (Digital Science)

– provided financial support and useful discussions.

• Jonathan Adams (Digital Science)

– for useful discussions

© Imperial College LondonPage 35

© Imperial College LondonPage 36

Bibliography

• Archambault, É., & Lariviére, V. (2009). History of the journal impact factor: Contingencies and consequences. Scientometrics, 79(3), 635–649.

• Bergstrom, C. (2007). Eigenfactor: Measuring the value and prestige of scholarly journals. College and Research Libraries News, 68, 314-316.

• Bornmann, L. (2014). Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics. Journal of

Informetrics, 8, 895-903.

• Brin, S. & Page, L. (1998). The anatomy of a large-scale hypertextual Web search engine. Computer networks and ISDN systems, 30, 107-117.

• Bruggemann, R.; Halfon, E.; Welzl, G.; Voigt, K. & Steinberg, C. Applying the Concept of Partially Ordered Sets on the Ranking of Near-Shore

Sediments by a Battery of Tests J. Chem. Inf. Model., American Chemical Society (ACS), 2001, 41, 918-925.

• Bruggemann, R., M¨unzer, B. and Halfon, E., An algebraic/graphical tool to compare ecosystems with respect to their pollution – the German

river “Elbe” as an example - I: Hasse-diagrams, Chemosphere, 28 (1994) 863–872.

• Garfield, E. (1955). Citation Indexes for Science: A New Dimension in Documentation through Association of Ideas, Science, 122 (3159), 108-

111.

• Garfield, E. (1990). Journal citation studies 52. The multifaceted structure of crystallography research. Part 1. Core journals, high impact

papers, and current research fronts. Essays of an Information Scientist: Journalology, KeyWords Plus, and other Essays, 13, 327-336

• Kleinberg, J. M. (1999). Authoritative sources in a hyperlinked environment. JACM, 46, 604-632.

• Langville, A. N. & Meyer, C. D. (2012) Who’s #1?: The science of rating and ranking, Princeton: Princeton University Press.

• Clough, J. R.; Gollings, J.; Loach, T. V. & Evans, T. S., Transitive reduction of citation networks, J.Complex Networks, 2015, 3, 189-203

[10.1093/comnet/cnu039 arXiv1310.8224]

• Clough, J.R. & Evans, T.S. What is the dimension of citation space? 2014 arXiv:1408.1274

• Evans, T.S., Complex Networks, Contemporary Physics, 2004, 45, 455-474 [10.1080/00107510412331283531 arXiv:cond-mat/0405123]

• Evans, T.S., Clique Graphs and Overlapping Communities, J.Stat.Mech, 2010, P12037 [10.1088/1742-5468/2010/12/P12037 arXiv:1009.0638]

• Evans, T.S. & Lambiotte, R., Line Graphs, Link Partitions and Overlapping Communities, Phys.Rev.E, 2009, 80, 016105

[10.1103/PhysRevE.80.016105 arXiv:0903.2181]

• Evans, T.S. & Lambiotte, R., Line Graphs of Weighted Networks for Overlapping Communities, Eur.Phys.J. B 2010, 77, 265–272

[10.1140/epjb/e2010-00261-8 arXiv:0912.4389]

• Expert, P.; Evans, T. S.; Blondel, V. D. & Lambiotte, R. Uncovering space-independent communities in spatial networks, PNAS, 2011, 108,

7663-7668 10.1073/pnas.1018962108 [10.1073/pnas.1018962108 arXiv:1012.3409]

• Garfield, E.; Sher, I. H. & Torpie, R. J., The use of citation data in writing the history of science, DTIC Document, 1964.

• Holme, P. & Saramäki, J., Temporal Networks, Physics Reports, 2012, 519, 97-125.

• Hummon, N. P. & Dereian, P., Connectivity in a citation network: The development of DNA theory, Social Networks, 1989, 11, 39-63.

© Imperial College LondonPage 37

Tim Evans

Centre for Complexity Science

http://netplexity.org