Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics...
Transcript of Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics...
![Page 1: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/1.jpg)
MINDY THUNA - [email protected] CUNNINGHAM - [email protected]
RESEARCHER IMPACT & RESPONSIBLE USE OF METRICS
October 3, 2019
Material in support of a verbal presentation and not intended as a stand alone document.
![Page 2: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/2.jpg)
Objectives
•At the end of this session, participants will be able to:
• Define what scholarly metrics are• Name commonly used scholarly metrics• Identify strengths and weaknesses of using different scholarly metrics • Use metrics responsibly
![Page 3: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/3.jpg)
Research Impact Defined
“Impact is usually demonstrated by pointing to a record of the active consultation, consideration, citation, discussion, referencing or use of a piece of research.”
http://blogs.lse.ac.uk/impactofsocialsciences/introduction/
![Page 4: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/4.jpg)
Impact Metrics are…
…calculations/algorithms that quantify the impact of research or scholarly activity.
![Page 5: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/5.jpg)
Basic Premise
Number of citations
Impact
Reach and significance
![Page 6: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/6.jpg)
Examples of Metrics
Images from Metrics Toolkit: http://www.metrics-toolkit.org/
![Page 7: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/7.jpg)
Responsibleuse of Metrics
• San Francisco Declaration on Research Assessment: DORA (2012)https://sfdora.org/
• Leiden Manifesto (2015)http://www.leidenmanifesto.org/
• Metric Tide (2015)https://responsiblemetrics.org/the-
metric-tide/
![Page 8: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/8.jpg)
Three main providers
•Web of Science
•Scopus
•Google Scholar
![Page 9: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/9.jpg)
Web of Science
• Does not cover books, chapters, dissertations, etc.
• Limited coverage of non-English material
Scopus
• Does not cover chapters, dissertations, etc.
• Covers many low-impact journals and trade publications
Google Scholar• Extent of coverage is
cryptic• Time coverage is
inconsistent
Weaknesses
![Page 10: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/10.jpg)
Three Levels
Journal Level
Article Level
Author Level
![Page 11: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/11.jpg)
Journal Level Metrics
Web of Science
/JCR
Impact Factor (2 & 5 year) Cited half-life Eigenfactor®
Score
Article Influence®
Score
Scopus CiteScoreSCImagoJournal
Rank (SJR)
Source Normalized Impact per
Paper(SNIP)
GoogleScholar h5-index h5-median
![Page 12: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/12.jpg)
Journal Impact Factor
Average number of times articles published in a two year (or five year) period have been cited
A = total number of times ALL articles published in 2 (or 5) year period were cited in WofSindexed journals during the next year
B = total number of "citable items" (usually articles, reviews, proceedings or notes; not editorials and letters-to-the-editor) published in 2 (or 5) year period
Impact factor = A/B
Example: New England Journal of Medicine (2017 JCR)
![Page 13: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/13.jpg)
Activity
1. Using the JCR, select "Browse by Category”
2. Select your category of research interest
3. Make a note of the top three journals sorted by JIF
![Page 14: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/14.jpg)
Journal Impact Factor Debate
• Often used as a surrogate measure for quality of articles, researchers, etc., (only evaluates journals in conjunction with other criteria)
• Vary across disciplines
• e.g., variants in time-to-publication
• Review journals have higher impact
• fewer articles per journal, cited more
• Calculations easily skewed/gamed
• Data not transparent nor available to public
• Includes retracted articles
• Mostly English language journals
• Not all journals indexed by Web of Science
• Linked to higher APCs
• Not a predictive measure
• Mean, not a median value; distribution can be skewed
![Page 15: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/15.jpg)
CiteScore
• average number of citations received in a calendar year by all items published in that journal in the preceding three years
• Does not include articles in press or abstracts
• Three year publication window• Includes all document types in both
the numerator and denominator
![Page 16: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/16.jpg)
CiteScore
A = number of citations received in 2017 by any of the content published in the 3 previous years, 2014, 2015 and 2016
B = ALL documents indexed in Scopus in 3 previous years (original research articles, reviews, conference papers, letters, editorials, errata…)
![Page 17: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/17.jpg)
CiteScore
![Page 18: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/18.jpg)
JIFvs. CiteScore
Journal Impact Factor
CiteScore
Data Web of Science ScopusSources Approx. 11,000 Approx. 22,000
Document Types “Citable items” –articles & reviews
All document types in denominator
Free/subscription-based
Subscription to Journal Citation
Reports
Freely available but data requires
subscription to Scopus
Timeframe 2 years or 5 years 3 yearsRemove self-
citations?Yes No
JIF JIF – self cites
5 Yr JIF CiteScore
Nature 40.137 39.533 43.769 13.33
Harvard Business Review
4.374 4.374 4.933 1.53
![Page 19: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/19.jpg)
Activity
Take the top three journals by JIF from the first activity and enter the title or ISSN into the compare journals box in Scopus to compare their CiteScorevalues.
![Page 20: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/20.jpg)
Top 5 Library Science Journals
By Impact Factor (2018)
CATEGORY: INFORMATION SCIENCE &
LIBRARY SCIENCE
CATEGORY: LIBRARY AND
INFORMATION SCIENCES
By CiteScore (2018)
![Page 21: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/21.jpg)
Article Level Metrics
• # of citations• Highly cited • Hot papers
Traditional metrics
• Viewed/Usage • Cited• Discussed
Altmetrics
![Page 22: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/22.jpg)
ArticleLevelMetrics
… are emerging or newer data sources, e.g. Wikipedia, Mendeley, Twitter, Facebook, blogs.… report the impact of a wider range of
research outputs, e.g. presentation slides, data sets, articles, code. … represent diverse, often complex
engagements with research outputs.
Altmetrics
![Page 23: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/23.jpg)
ArticleLevelMetrics
![Page 24: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/24.jpg)
ArticleLevelMetrics
![Page 25: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/25.jpg)
ArticleLevelMetrics
![Page 26: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/26.jpg)
ArticleLevelMetrics: Limitations
• Citations do not decline (e.g., dormant researcher, death)
• Time frame – some new tools cannot search old mentions, tweets, etc.
• Doesn’t distinguish positive vs. negative attention• Can these tools be easily manipulated to raise
significance of an article? E.g. gaming?• North American, Western European, and English-
language titles dominate• Limited data for books, book chapters, dissertations,
working papers, reports, and conference papers• Different coverage among disciplines• Citing errors, e.g. inconsistency in use of initials and
in spelling non-English names
![Page 27: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/27.jpg)
Activity
1. Look up a topic in Web of Science or Scopus
2. Sort by Times Cited
3. What do you notice?
![Page 28: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/28.jpg)
Author Level Metrics
Web of Science h-index Total # cites
Scopus h-index Total # cites
Google Scholar h-index i10-index
![Page 29: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/29.jpg)
The number of articles N in a group that have N or more citations. For example, an h-index of 15 means that there are 15 articles that have15 or more citations.
“gives an estimate of the importance, significance and broad impact of a scientist’s cumulative research contributions”
Hirsh, J. E. (2005). An index to quantify an individual's scientific research output. Proceedings of the National Academy of Science, 102(46), 16569-16572. Published online before print November 7, 2005, doi: 10.1073/pnas.0507655102
h index
![Page 30: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/30.jpg)
h index is NOTperfect
Paper 1 Paper 2 Paper 3 Paper 4 Paper 5
Author A 3 cites 6 cites 100 cites 4 cites 1 cite
Author B 3 cites 6 cites 100 cites
Author C 400 cites 150 cites 3 cites 6 cites
h index = 3
![Page 31: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/31.jpg)
• Balance of quality & quantity of research output
• More accurate than total # papers or # citations
• Order of authors• Multi-author papers• Status of journals• Self-citations• Retracted articles• Excludes non-article outputs
h index
Pros Cons
![Page 32: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/32.jpg)
Activity
Compare h index values for an iSchoolfaculty member using:
• Google Scholar User Profile• Scopus• Web of Science
![Page 33: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/33.jpg)
ResponsibleMetrics Guidelines
• Quantitative evaluation should support & not replace qualitative, expert assessment
• Keep data collection and analytical processes open and transparent, so those being evaluated can test and verify the results
• One size does not fit all: different fields/disciplines have different publication & citation practices
• Inappropriate indicators create perverse incentives• Encourage responsible authorship practices &
provision of information about specific contributions of each author.
• Cite primary literature in favour of reviews in order to give credit to author(s) who first reported a finding.
• Value all research outputs: articles, data, policy, code, etc.
• Suite of indicators is preferred – reduces gaming
![Page 34: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/34.jpg)
Research impact in context
Abbott A. Do metrics really matter? Nature. 2010;465:860-862.
![Page 35: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/35.jpg)
Research impact in context
Abbott A. Do metrics really matter? Nature. 2010;465:860-862.
![Page 36: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/36.jpg)
Research impact in context
• Common use of name• Author identifier (e.g. ORCID)• Research outputs• Distribution (Figshare, F1000, OSF,
Slideshare)• Collaboration• Make research open & accessible• Responsible authorship practices• Understand the context and limitations of
metrics
Steps for researchers
![Page 37: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/37.jpg)
Research impact beyond quantitative
• Becker Model (Medical)• Rand/AAMC Report: 100 metrics to
Assess and Communicate the Value of Biomedical Research (Medical)
• Approaches to Assessing Impacts in the Humanities and Social Sciences (Humanities & Social Sciences)
• UofT President’s Research Impact Award (all disciplines)
Frameworks of research impact
![Page 38: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/38.jpg)
ImpactOutcomesIndicators
EconomyScholarshipKnowledge
Society & Culture,
Quality of Life
Practice & Policy,
Legislation
Inventions, Innovation
Professional Practices,
Clinical Implementat
ion
Public engagement, Community involvement
EnvironmentHealth
Education
![Page 39: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/39.jpg)
Final thoughts
Any thoughts or ideas or observations on research impact that you would like to share?
![Page 40: Researcher impact & responsible use of metrics · weaknesses of using different scholarly metrics • Use metrics responsibly. Research Impact D. efined “Impact is usually demonstrated](https://reader033.fdocuments.in/reader033/viewer/2022060500/5f1a110ff5cca8325757649f/html5/thumbnails/40.jpg)
Questions?
Mindy Thuna, M.Sc., MIStHead, Engineering & Computer Science [email protected]
Heather Cunningham, M.Sc., MLISAssistant Director for Research & Innovation ServicesGerstein Science Information [email protected]
Thank you!