Empirical analyses of scientific papers and researchers on Twitter: Results of two studies

Stefanie Haustein, Timothy D. Bowman, Kim Holmberg, Vincent Larivière, Isabella Peters, Cassidy R. Sugimoto, & Mike Thelwall

Background •  when Garfield created SCI, sociologists of science

analyzed meaning of publications and citations (Merton, Zuckerman, Cole & Cole, etc.)

•  sociological research •  What is it to publish a paper? •  What are the reasons to cite?

•  empirical bibliometric research •  disciplinary differences in publication

and citation behavior •  delay and obsolescence patterns

Background •  empirical studies helped sociologists to understand

structure and norms of science •  for bibliometricians, studies provided a theoretical

framework and legitimation to use citation analysis in research evaluation

•  knowledge about disciplinary differences and obsolescence patterns helped to normalize statistics and create more appropriate indicators

Background •  recently social-media metrics have become

important in the scholarly world •  suggestions to complement (or even replace)

citation analysis by so-called ”altmetrics“ •  broader audience (not just citing authors) •  more timely

•  however, similar to bibliometrics in the 1960s, little is known about the actual meaning of various social-media counts

Research questions •  What is the relationship between social-media and

citation counts? •  How do various social-media metrics differ? •  Why are papers tweeted, bookmarked, liked…? •  Who tweets (bookmarks, likes…) scientific papers? •  How do these aspects differ across scientific disciplines?

Two case studies on Twitter •  large-scale analysis of tweets of biomedical papers •  in-depth analysis of astrophysicists on Twitter

Aim of the study •  large-scale analysis of tweets of biomedical papers

•  Twitter coverage •  Twitter citation rates (tweets per paper) •  correlation with citations

•  discovering differences between: •  documents •  journals •  disciplines & specialties

!  providing empirical framework to understand the extent to which biomedical journal articles are tweeted

Study I: Tweeting biomedicine

Haustein, S., Peters, I., Sugimoto, C.R., Thelwall, M., & Larivière, V. (in press). Tweeting biomedicine: an analysis of tweets and citations in the biomedical literature. Journal of the American Society for Information Science and Technology, http://arxiv.org/abs/1308.1838.

Data sets & methods •  1.4 million PubMed papers covered by WoS

•  publication years: 2010-2012 •  document types: articles & reviews •  matching of WoS and PubMed

•  tweet counts collected by Altmetric.com •  collection based on PMID, DOI, URL •  matching WoS via PMID

•  journal-based matching of NSF classification •  tweets per article, Twitter coverage and correlation

with citations for: •  journals •  NSF disciplines and specialties

Study I: Tweeting biomedicine

Data sets & methods: framework

Study I: Tweeting biomedicine

Data sets & methods: correlations

Study I: Tweeting biomedicine

PY=2010 PY=2011 PY=2012

Results: documents Study I: Tweeting biomedicine

Publication year

Twitter coverage

Papers (T≥1)

Spearman's ρ Mean Median Maximum

T2010 2.4% 13,763 .104** 2.1 1 237 C2010 18.3 7 3,922

T2011 10.9% 63,801 .183** 2.8 1 963 C2011 5.7 2 2,300

T2012 20.4% 57,365 .110** 2.3 1 477 C2012 1.3 0 234

T2010-2012 9.4% 134,929 .114** 2.5 1 963 C2010-2012 5.1 1 3,922

•  Twitter coverage is quite low but increasing •  correlation between tweets and citations is very low

Results: documents Study I: Tweeting biomedicine

Article Journal C T

Hess et al. (2011). Gain of chromosome band 7q11 in papillary thyroid carcinomas of young patients is associated with exposure to low-dose irradiation PNAS 9 963

Yasunari et al. (2011). Cesium-137 deposition and contamination of Japanese soils due to the Fukushima nuclear accident PNAS 30 639

Sparrow et al. (2011). Google Effects on Memory: Cognitive Consequences of Having Information at Our Fingertips Science 11 558

Onuma et al. (2011). Rebirth of a Dead Belousov–Zhabotinsky Oscillator Journal of Physical Chemistry A -- 549

Silverberg (2012). Whey protein precipitating moderate to severe acne flares in 5 teenaged athletes Cutis -- 477

Wen et al. (2011). Minimum amount of physical activity for reduced mortality and extended life expectancy: a prospective cohort study Lancet 51 419

Kramer (2011). Penile Fracture Seems More Likely During Sex Under Stressful Situations Journal of Sexual Medicine -- 392

Newman & Feldman (2011). Copyright and Open Access at the Bedside New England Journal of Medicine 3 332

Reaves et al. (2012). Absence of Detectable Arsenate in DNA from Arsenate-Grown GFAJ-1 Cells Science 5 323

Bravo et al. (2011). Ingestion of Lactobacillus strain regulates emotional behavior and central GABA receptor expression in a mouse via the vagus nerve PNAS 31 297

Top 10 tweeted documents: catastrophe & topical / web & social media / curious story scientific discovery / health implication / scholarly community

Results: journals • 97.7% of 3,812

journals at least tweeted once

• two-thirds of journals have coverage below 20% and Twitter citation rate < 2.0

• high Twitter citation rates often caused by few papers

• high coverage and Twitter citation rates for general journals

Study I: Tweeting biomedicine

Results: disciplines Study I: Tweeting biomedicine

Results: specialties Study I: Tweeting biomedicine

• specialties differ in terms of coverage, Twitter citation rate and correlations with citations

• 47 of 61 specialties show low positive, 3 negative and 13 no correlation

bubb

le s

ize

= Tw

itter

cita

tion

rate

Aim of the study •  in-depth analysis of astrophysicists on Twitter

•  number of tweets, followers, retweets •  characteristics of tweets: RTs, @messages,

#hashtags, URLs •  comparison with scientific output

•  publications •  citations

•  comparison of tweet and publication content !  provide evidence in how far astrophysicists on Twitter

use Twitter for scholarly communiation

Study II: Astrophysicists on Twitter

Haustein, S., Bowman, T.D., Holmberg, K., Larivière, V., & Peters, I., (submitted). Astrophysicists on Twitter: An in-depth analysis of tweeting and scientific publication behavior. Aslib Proceedings.

Data sets & methods •  37 astrophysicists on Twitter identified by

Holmberg & Thelwall (2013) •  web searches to identify person behind account •  publications in WoS journals

•  publication years: 2008-2012 •  author disambiguation

•  Twitter account information •  68,232 of 289,368 tweets downloaded and analyzed:

•  number of RTs per tweet •  % of tweets that are RTs •  % of tweets containing #hashtags, @usernames, URLs

Study II: Astrophysicists on Twitter

Holmberg, K., & Thelwall, M. (2013). Disciplinary differences in Twitter scholarly communication. In: Proceedings of ISSI 2013 – 14th International Conference of the International Society for Scientometrics and Informetrics, Vienna, Austria (Vol. 1, pp. 567-582).

Data sets & methods •  grouping astrophysicists according to tweeting and

publication behavior •  analyzing differences of tweeting characteristics

between user groups

Study II: Astrophysicists on Twitter

Selected astrophysicists (N=37)!

tweet rarely (0.0-0.1 tweets per day)!

tweet occasionally (0.1-0.9)!

tweet regularly (1.2-2.9)!

tweet frequently (3.7-58.2)!

total (publishing activity)!

do not publish (0 publications 2008-2012)! --! --! 1! 5! 6!publish occasionally (1-9)! 4! 3! 4! 2! 13!publish regularly (14-37)! --! 5! 5! 3! 13!publish frequently (46-112)! 1! 3! 1! --! 5!total (tweeting activity)! 5! 11! 11! 10! 37!

Data sets & methods •  comparison of tweet and publication content

•  extraction of noun phrases from tweets and abstracts •  limited to 18 most frequently publishing astrophysicists

to ensure certain number of abstracts •  analyzing overlap of character strings •  calculating similarity with cosine per person and overall

Study II: Astrophysicists on Twitter

Selected astrophysicists (N=37)!

tweet rarely (0.0-0.1 tweets per day)!

tweet occasionally (0.1-0.9)!

tweet regularly (1.2-2.9)!

tweet frequently (3.7-58.2)!

total (publishing activity)!

publish regularly (14-37)! --! 5! 5! 3! 13!publish frequently (46-112)! 1! 3! 1! --! 5!total (tweeting activity)! 1! 8! 6! 3! 18!

Results: correlations •  comparison of Twitter and publication activity and impact

•  publications and tweets per day: ρ=−0.339* •  citation rate and tweets per day: ρ=−0.457** •  citation rate and RT rate: ρ=0.077

Study II: Astrophysicists on Twitter

Results: characteristics Study II: Astrophysicists on Twitter

Mean share of tweets containing at least one user name or URL per person per group

Results: content similarity Study II: Astrophysicists on Twitter

•  overall similarity between abstracts and tweets is low •  cosine=0.081 •  4.1% of 50,854 tweet NPs in abstracts •  16.0% of 12,970 abstract NPs in tweets

•  Twitter coverage among most frequent abstract terms is high, although this differs between users •  97,1% of 104 most frequent noun phrases on Twitter

Conclusions •  Twitter coverage of biomedical papers is low but increasing •  number of tweets per paper varies between journals,

disciplines, specialties and from year to year !  tweet counts need to be normalized accordingly

•  correlations between tweet and citation counts are low (biomedical papers) or even moderately negative (astrophysicists) !  tweets cannot replace citations as measures of

scientific impact !  challenge is to differentiate between high tweet counts

because of value (to scientists and/or the general public) and curiosity

Outlook •  user surveys and qualitative research to investigate who is

using scholarly content on social media and why •  empirical large-scale studies on other metrics

Haustein, S., Peters, I., Sugimoto, C.R., Thelwall, M., & Larivière, V. (in press). Tweeting biomedicine: an analysis of tweets and citations in the biomedical literature. Journal of the American Society for Information Science and Technology.

Haustein, S., Bowman, T.D., Holmberg, K., Larivière, V., & Peters, I., (submitted). Astrophysicists on Twitter: An in-depth analysis of tweeting and scientific publication behavior. Aslib Proceedings.

Stefanie Haustein

Thank you for your attention! Questions?

stefanie.haustein@umontreal.ca @stefhaustein