About use and misuse of impact factor and other journal metrics Dr Berenika M. Webster Strategic Business Manager

23 January 2009, Sydney

2

Outline

• INTRODUCTION

• IMPACT FACTOR – WHAT IT CAN CANNOT TELL YOU ABOUT A JOURNAL

• WHAT OTHER TOOLS AND INDICATORS ARE THERE?

• CONCLUSIONS

3

How to measure impact/influence of a publication?

• Measures of volume– Counts of …

• Measures of impact– Citations

• Measures of influence – Combine volume and impact

• Weighted, relative and normalised

4

Journal citation information has been used successfully by…

• Publishers

• Librarians

• Authors

• Research policy and funding agencies

• Information scientists

• Industry

5

Sources of bibliometric indicators of impact/influence of a publication

• Journal Citation Reports– Impact Factor (2 and 5-year)

– Measures of obsolescence (cited half-life) and currency (citing half-life and immediacy index)

– Eigenfactor

• Web of Science indexing and visualisation

• Citation reports (standard and custom)

• Indicators – journal, university, national

6

Journal inclusion: factors considered

• Publishing standards

• Editorial content

• International diversity

• Citation data analysis

• Self citation rates

7

Impact Factor – some headlines• Worshiping false idols: The impact factor dilemma

• The power of the unrelenting impact factor - Is it a force for good or harm?

• Impact factor, impact, and smoke and mirrors

• Impact factor: Is it dragging science off course?

• The dreaded impact factor is back to haunt us!

• The impact factor or the fairy tale of impartiality

• The magic of the impact factor: unmasking of a phenomenon

• Is the "impact factor" killing the German language?

• The tyranny of the impact factor

• Nightmare impact factor

8

Impact Factor – what it does not measure

• The Journal Impact Factor was not designed for or intended to be used as a measure or proxy for the performance of individual papers or researchers. (Seglen, 1997)

• The Impact Factor does not measure the impact or influence of a journal, but of an average item published in that journal. (Leydesdorff , 2008)

• Specific IF requirements (say, n papers in journals with IF > 1) ignore field differences

• Summing up or combining IFs only make matters worse. As computer scientists say: GIGO

9

Impact Factor – where it can be helpful• Global view of internationally influential journals, a vetted

corpus

• Simple-to-understand calculation

• Does not reward age or size/frequency of output

• Gives insight to recent performance in field

• Citation counts matched to journal titles to collect as many citations as possible

• Long history of journal impact factors allows for time-series studies

• Generally reasonable results (journals recognised as influential occupy top ranks)

• Widely available and used over 30+ years (market success implies some utility for users)

10

Scenario 1: Journal editor asks…

• I want to know how my journal “stacks up” against competition.

• I want to know who are the most influential authors in my journal’s field in Australia.

• I want to know which institutions in Australia publish in the field covered by my journal. And which institution’s research has the biggest impact.

11

Scenario 2: Librarian asks…

• To rationalise my journal collection, I need to know in which journals staff at my university publish and which journals cite their publications.

• I want to identify the most influential journals in the field of Nanoscience and Nanotechnology in the last 10-20 years. And important emerging journals.

12

Scenario 3: A researcher asks …

• I am looking for a post-doc position in the leading institution in the field of Nanotechnology. Who publishes in my field and whose publications have the greatest impact?

• I am looking for a research group conducting research in Nanotechnology for possible collaboration on my next project.

13

How does my journal stack up against the competition?

• Basic statistics from JCR (field delineation, influence, currency and obsolescence)

• Basic statistics in WoS (including h-index)

• Journal Performance Indicators data (benchmarking against field and world baselines)

• Essential Science Indicators (10-year perspective)

14

Nature Materials in JCR

15

Impact Factor (IF)Five Year Impact Factor (VIF)

Numerator: citations to all items

Denominator: citatable items only (arts and revs)

16

Journal self-citations

– displays the contribution of journal self-citations to Impact Factor calculations

– helps to identify journals with a narrow focus

Journal Citation Reports 2007

17

Nature Materials over years

Journal Citation Reports 2007

18

Category ranking and quartile distribution

19

Cited / citing journalsNature Materials

Journal Citation Reports 2007

I F Cited journals

Number of references in Nature

Materials

6.944 PHYS REV LETT 485 3.172 PHYS REV B 435 

26.372 SCIENCE 345 28.751 NATURE 344 3.596 APPL PHYS LETT 271 

19.782 NAT MATER 271 7.885 J AM CHEM SOC 165 2.171 J APPL PHYS 131 9.627 NANO LETT 104 8.191 ADV MATER 102 4.086 J PHYS CHEM B 79 4.411 MACROMOLECULES 77 

10.031 ANGEW CHEM INT EDIT 75 3.044 J CHEM PHYS 75 4.009 LANGMUIR 66 9.598 P NATL ACAD SCI USA 63 4.883 CHEM MATER 40 1.247 JPN J APPL PHYS 38 2.483 PHYS REV E 36 

All Journals (N=359) 5814

I F Citing journals

Number of citations to Nature

Materials

3.596 APPL PHYS LETT 1010 3.172 PHYS REV B 650 

J PHYS CHEM C 460 8.191 ADV MATER 416 2.171 J APPL PHYS 394 7.885 J AM CHEM SOC 309 9.627 NANO LETT 303 4.009 LANGMUIR 297 3.31 NANOTECHNOLOGY 297 6.944 PHYS REV LETT 289 

19.782 NAT MATER 271 4.883 CHEM MATER 243 

10.031 ANGEW CHEM INT EDIT 214 1.886 J PHYS-CONDENS MAT 189 7.496 ADV FUNCT MATER 181 4.339 J MATER CHEM 164 4.411 MACROMOLECULES 159 3.044 J CHEM PHYS 144 

All Journals (n=584) 13606

20

Journal metrics: WoS analytics

21

Journal Performance Indicators – 5-year view

Citations per paper for subject category and Nature Neuroscience

0

2

4

6

8

10

12

14

16

18

20

22

24

26

28

30

32

94-98 95-99 96-00 97-01 98-02 99-03 00-04 01-05 02-06 03-07

Cit

atio

ns

per

pap

er

Avg cit/pap for subject category Avg cit/pap for Nature Neuroscience

22

Journal Performance Indicators – 5-year view

Percentage of cited papers for subject category and Nature Neuroscience

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

94-98 95-99 96-00 97-01 98-02 99-03 00-04 01-05 02-06 03-07

% of Field Cited % of Nature Neuroscience Cited

23

Essential Science Indicators – 10 year view

24

Which regions conduct nanotechnology research?

0

5,000

10,000

15,000

20,000

25,000

30,000

35,000

40,000

45,000

0

5,000

10,000

15,000

20,000

25,000

30,000

35,000

40,000

45,000

5 Year Period

ASIA PACIFIC-Nanoscience & Nanotechnology

EU-27-Nanoscience & Nanotechnology

LATIN AMERICA-Nanoscience & Nanotechnology

MIDDLE EAST-Nanoscience & Nanotechnology

USA-Nanoscience & Nanotechnology

WORLD-Nanoscience & Nanotechnology

National Science Indicators, 2007

25

Which countries in AP publish nanotechnology papers?

National Science Indicators, 2007

0

1,000

2,000

3,000

4,000

5,000

6,000

0

1,000

2,000

3,000

4,000

5,000

6,000

5 Year Period

AUSTRALIA-Nanoscience & Nanotechnology

CHINA-Nanoscience & Nanotechnology

INDIA-Nanoscience & Nanotechnology

JAPAN-Nanoscience & Nanotechnology

SINGAPORE-Nanoscience & Nanotechnology

SOUTH KOREA-Nanoscience & Nanotechnology

26

Which institutions in Australia conduct research in my area?

0 50 100 150 200 250 300 350 400 450

UNIV QUEENSLAND

UNIV NEW S WALES

MONASH UNIV

UNIV SYDNEY

UNIV MELBOURNE

UNIV WOLLONGONG

AUSTRALIAN NATL UNIV

UNIV WESTERN AUSTRALIA

DEAKIN UNIV

ROYAL MELBOURNE INST TECHNOL

SWINBURNE UNIV TECHNOL

QUEENSLAND UNIV TECHNOL

UNIV TECHNOL SYDNEY

GRIFFITH UNIV

UNIV ADELAIDE

UNIV NEWCASTLE

LA TROBE UNIV

UNIV S AUSTRALIA

FLINDERS UNIV S AUSTRALIA

CURTIN UNIV TECHNOL

MACQUARIE UNIV

JAMES COOK UNIV N QUEENSLAND

Citations Papers

Nanotechnology papers and citation in Australian institutions, 2003-2007

University Science Indicators, 2007

27

and whose publications have the biggest impact…

Institution ImpactUNIV MELBOURNE 6.98FLINDERS UNIV S AUSTRALIA 6.14UNIV NEWCASTLE 5.44DEAKIN UNIV 4.45UNIV ADELAIDE 4.36ROYAL MELBOURNE INST TECHNOL 3.46UNIV QUEENSLAND 3.37GROUP OF 8 3.34MACQUARIE UNIV 3.2UNIV NEW S WALES 3.1GRIFFITH UNIV 3.08MONASH UNIV 3.08UNIV TECHNOL SYDNEY 2.93UNIV WESTERN AUSTRALIA 2.82NON GROUP OF 8 2.74SWINBURNE UNIV TECHNOL 2.5UNIV SYDNEY 2.36LA TROBE UNIV 1.88AUSTRALIAN NATL UNIV 1.68UNIV WOLLONGONG 1.64QUEENSLAND UNIV TECHNOL 0.88UNIV S AUSTRALIA 0.88CURTIN UNIV TECHNOL 0.4JAMES COOK UNIV N QUEENSLAND 0.33

University Science Indicators, 2007

28

Who are the most influential authors in my field in Australia?

National Citation Reports, 2007

29

in more detail…

National Citation Reports, 2007

30

Which journals do my university researchers cite and which journals cite them?

Citation Reports, custom data

31

Is my library subscribing to the right journals?

0

1

2

3

4

5

6

7

8

1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007

NANO LETT

NAT NANOTEC

PLASMONICS

LAB CHIP

SMALL

BIOSENS BIO

MICROP M M

BIOMED MICR

NANOTECHNOL

NANO TODAY

IEEE NANO

CURR NANOSC

J MICROM M

J NANOPAR R

SCR MATER

IEEE T NANO

MICROFL NAN

J NANOSCI N

MAT SCI E A

PHOTON NANO

REV ADV MAT

J VAC SCI B

MICROSYST T

FULLER NANO

PRECIS ENG

PHYSICA E

MICROEL ENG

SYNTH REACT

J COMPUT TH

J PHYS CH C

INT J NANOT

J EXP NANOS

MICROELEC J

NANOMED-N B

Journal Performance Indicators, 2007

32

• Other bibliometric journal performance measures– IF variants

– h-index and its variants

– Based on other data sources

– Related to national performance

• Ranking lists– Organisation-based

– National

– Regional

• Publisher data– Subscriptions

– Downloads

33

Conclusions

• “My measure is better than your measure”

• But different measures attempt to answer different questions, emphasize different aspects and nuances of a phenomenon

• The fallacy of the demand for single-number metrics that are all-revealing

• A lack of comprehensive, validated standards by which to compare different metrics or to show which combination is superior to others