Using bibliometrics
Transcript of Using bibliometrics
USING BIBLIOMETRICSA GUIDE TO EVALUATING RESEARCH PERFORMANCE USING CITATION DATA
Philip PurnellLiverpool, March 2011
DO WE NEED MORE METRICS?1962
METRICS ARE HERE TO STAY!2010
• Context and limitations
• Data
• Examples
4
Context and limitations
HOW IS RESEARCH EVALUATED?
• Peer Evaluation– Reputation
• Productivity– Number of researchers– Publication output
• Grant awards– Research funding awarded
• Prestigious awards – Nobel Prizes
• Innovation– Industry income and patents
• Teaching– Academic Reputation Survey,
higher degrees
• International Mix– National / International staff
and students
• Citation analysis– Normalised for volume and
subject area
7
UNIQUE CHALLENGES IN HUMANITIESCITEDNESS
8
UNIQUE CHALLENGES IN HUMANITIESCITATION IMPACT
Impact = CitationsPapers
The Data
A BRIEF HISTORY OF THE CITATION INDEX
• Concept first developed by Dr Eugene Garfield – Science, 1955
• The Science Citation Index (1963)– SCI print (1960’s)– On-line with SciSearch in the 1970’s – CD-ROM in the 1980’s– Web interface (1997) Web of Science
• Content enhanced:– Science Citation Index – Expanded (SCI-E)– Social Sciences Citation Index (SSCI)– Arts & Humanities Citation Index (AHCI)– Conference Proceedings Citation Index (CPCI)
WHY NOT INDEX ALL JOURNALS?
0
20
40
60
80
100
120
0 1000 2000 3000 4000 5000 6000
# of journals
% o
f dat
abas
e
Articles Citations
40% of the journals:
• 80% of the publications
• 92% of cited papers
4% of the journals:
• 30% of the publications
• 51% of cited papers
HOW DO WE DECIDE WHICH JOURNALS TO INDEX?• Approx. 2.500 journals evaluated annually
– 10-12% accepted
• Thomson Reuters editors– Information professionals – Librarians – Experts in the literature of their subject area
Web of Science
Journals under evaluation
Journal ‘quality’
THOMSON REUTERSJOURNAL SELECTION POLICY
• Publishing Standards– Peer review, Editorial conventions
• Editorial content– Addition to knowledge in specific subject field
• Diversity– International, regional influence of authors, editors, advisors
• Citation analysis– Editors and authors’ prior work
Region # Journals from Region in Web of ScienceEurope 5,573 49%
North America 4,251 38%Asia-Pacific 965 9%
Latin America 272 2%Middle East/Africa 200 1%
Language # Journals in Web of ScienceEnglish 9114 81% Other 2147 19%
GLOBAL RESEARCH REPRESENTATIONWEB OF SCIENCE COVERAGE
WEB OF SCIENCESUMMARY• World’s largest citation index
– >49 millon papers (75% with citations)– Approx. 800 million citations
• Multidisciplinary– 252 categories covering Science, Social Sciences, Arts & Humanities
• 110 years consistent coverage– Publishers, societies, open access and electronic journals
• Content derived from – 12,000 scholarly journals– 120,000 conferences
• Updated weekly• Exclusively hosted on ISI Web of Knowledge platform
16
GLOBAL RESEARCH COMMUNITYUSING WEB OF SCIENCE
Asia-Pacific
353 institutions 26 countries
Europe,Middle
East andAfrica
2,500+ institutions 50 countries
244 institutions 12 countries
LatinAmerica
760 institutions2 countries
NorthAmerica
Russia 237 institutions
4.000 Research institutions 91 countries
GOVERNMENTS AND INSTITUTIONS USING TR DATA FOR EVALUATION (INCL.)•United Kingdom: King’s College London; HEFCE
•Germany: IFQ, Max Planck Society, DKFZ, MDCUS
•Netherlands: NWO & KNAW
•France: Min. de la Recherche, OST - Paris, CNRS
•Russia: Russian Foundation for Basic Research (RFBR)
•European Union: EC’s DGXII(Research Directorate)
•US: National Science Foundation (since 1974)
•Canada: NSERC, FRSQ (Quebec), Alberta Research Council
•Australian Academy of Science, gov’t lab CSIRO
•Japan: Ministry of Education, Ministry of Economy, Trade & Industry
•People’s Republic of China: Chinese Academy of Science
•University rankings agencies (THE, Shanghai, etc.)17
Authoritative data from the world’s leading provider of research evaluation data
Strict selection policy applying consistent criteria over the last 50 years
This has created a large set of journals containing comparable papers and citations
One consistent editorial policy
Unique set of multi-disciplinary comparable data
CONSISTENCY IS THE KEY TO VALIDITY
Examples
PRODUCTIVITY
20
PRODUCTIVITY – INSTITUTIONS
21
PRODUCTIVITY BY UNIVERSITY DEPARTMENT
INFLUENCE
Source: Thomson Reuters U.S. and Canadian University Science Indicators
Citations to North American papers
HOW COMPETITIVE IS OUR UNIVERSITY?
24
……
…
HOW DOES OUR UNIVERSITY COMPARE LOCALLY AND GLOBALLY?
25
H-INDEX
27
EFFICIENCY H-INDEX• Distribution based indicator
– Attempt to reflect productivity and influence– Does not control for time period, field or doc. type
Thomson Reuters InCites
SELF-CITATIONS
29
EFFICIENCYJOURNAL IMPACT FACTOR
2009 Impact Factor
200920082007
Source paper – published in 2009
Cited reference – published in 2007 or 2008
Citations
All Previous Years
2006 2010
CALCULATING 2009 JOURNAL IMPACT FACTOR HEALTH ECONOMICS
Citations in 2009To items published in 2008 = 156
To items published in 2007 = 210
Sum = 366
Number of itemsPublished in 2008 = 97
Published in 2007 = 85
Sum = 182
366
182
= 2.011
EFFICIENCYJOURNAL IMPACT FACTOR
32
INTRODUCTION TO THE IMPACT FACTOR NATURAL FLUCTUATIONS OVER TIME
CITATION BEHAVIOUR VARIES BETWEEN SUBJECT CATEGORIES
EFFICIENCY5-YEAR IMPACT FACTOR
Citation rates to journals in the ‘Acoustics’ category are
maintained thus increasing the 5-year Impact Factor relative to
its 2-Year counterpart
EFFICIENCY5-YEAR IMPACT FACTOR
The opposite picture in ‘Genetics and Heredity’ demonstrates a
shorter citation lag
USING THE IMPACT FACTOREVALUATING JOURNALS
• Appropriate use– To evaluate journals
• Misuse– Evaluation of individual articles– Evaluation of institution or researcher
38
USING THE IMPACT FACTOR MISUSE: EVALUATING INDIVIDUAL PAPERS
30% of articles in Food Policy were not cited at all
Source: Thomson Reuters Journal Citation Reports
Journal Impact Factor = 2,011
BENCHMARK YOUR PAPERS AGAINST GLOBAL AVERAGES – IS THIS A HIGHLY CITED PAPER?
39
Hematology articles from this year have been cited 18,83 times
This article is ranked in the 12,92nd percentile in its field by citations
Articles published in ‘Blood’ from 2004 have been cited 34,30 times
This paper has received 40/34,30=1,17 times the expected citations for this journal
This paper has received 40/18,83=2,12 times the expected citations for this subject category
WHICH AUTHORS’ PAPERS HAVE PERFORMED BEST IN THEIR FIELD?
40
Source: Thomson Reuters InCites
Computer Science
Medicine
WHICH ARE OUR CENTRES OF EXCELLENCE?
41
Dept. Medicine Dept. Mechanical Engineering
PERCENTILE INDICATORS
• Paper percentile– Determine percentage of papers at each level of citation
IN WHICH JOURNALS HAS OUR CHEMISTRY DEPT. PUBLISHED?
HOW HAVE THOSE PAPERS PERFORMED?
44
HOW MANY CITATIONS HAS THE CHEMISTRY DEPT. RECEIVED?
45
IN WHICH JOURNALS ARE THOSE CITATIONS PUBLISHED?
46
LIMITATIONS TO CITATION ANALYSIS
• Albert Einstein, modest paper and citation count
• « What nobler employment,or more valuable to the state,than that of the man who instructs the rising generation? »Marcus Tullius Cicero 106B.C. – 43B.C. H-Index still going up !
• High co-authorships
• Author placement
48
USING BIBLIOMETRICSA GUIDE TO EVALUATING RESEARCH PERFORMANCE USING CITATION DATA
Philip PurnellLiverpool, March 2011