What’s all the h about? A Summary of Performance Metrics for Academics and Journals
58
description
The attraction of metrics is irresistible. The emergence of the quantified self as an emergent phenomenon to improve performance, health, and daily functioning has dramatically reshaped cultural perceptions of sharing and the relative good. The science of publica- tions is not free from this movement with a proliferation of metrics associated with both individuals and their scientific products. A dominant index, h, is central to the discussion associated with singular point estimate metrics versus the ineffable quality of the science we produce. The tension between quantity and quality or the modern redux of metrics versus quality is an excellent starting point in examination of the role metrics can play in improving or impeding scientific discovery. This dichotomy is of course fallacious but useful as means to test ideas associated with assigning merit to peer-reviewed publications. A brief overview of metrics is pro- vided in this talk including h with a strong emphasis on the theory of merit for the current dissemination pipeline in science.
Transcript of What’s all the h about? A Summary of Performance Metrics for Academics and Journals
context quantities merit elite
metrics timeline h analysis
significancecontrasts options application
data are the currency of science quantities
data are the currency of science
+
quantities
the greater good = feedback, filtering, change in behaviors.
quality versus quantity
volume of scientific literature immense
volume of scientific literature immense
constellation of ideas
merit
performancetime
loss
filter
rank
sort
group
legacy
quality
impact forms
+ -
unfortunately, much of the filtering not based on reading but numbers & tendencies
merit
critical assumption
0
5
10
15
20
25
30
-5 -4 -3 -2 -1 0 1 2 3 4 5
Cite
s/yr
/pap
er a
nd IF
Effect size
cites/yr/paper
IF
reject accept0
0.1
0.2
0.3
0.4
0.5
0.6E
ffect
siz
e
Support hypotheses
0
1
2
3
4 5 6 7
LOG
cita
tions
per
pub
licat
ion
LOG total funding
Why is this merit concept important?
Fig. a)
Number of Articles
Num
ber o
f HC
0
5
10
15
20
25
100 200 300 400
Fig. b)
Number of Citations
Num
ber o
f HC
0
5
10
15
20
25
30
0 5000 10000 15000 20000
Fig. c)
Proportion Citations to Most Cited Article
Num
ber o
f HC
0
10
20
30
40
50
60
70
0.0 0.1 0.2 0.3 0.4 0.5
Fig. d)
Number of Journals
Num
ber o
f HC
0
10
20
30
40
0 20 40 60 80
Box 1
elite
Fig. a)
log(Articles in Nature or Science)
Num
ber o
f HC
0
20
40
60
80
0 1 2 3 4 5
Fig. b)
Proportion Articles in Nature and Science
Prop
ortio
n C
itatio
ns
0.0
0.2
0.4
0.6
0.8
●●●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●● ●●
●●
●●
●
● ●●●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●●
●●
●
●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●●● ●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●● ●
●
●
●
●●
●
●
●●
●
●
● ●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●●
0.1 0.2 0.3 0.4 0.5 0.6
Fig. c)
Proportion Articles in Favorite Journal
Num
ber o
f HC
0
10
20
30
0.2 0.4 0.6 0.8
Fig. d)
Proportion Articles in Favorite Journal
Prop
ortio
n C
itatio
ns
0.2
0.4
0.6
0.8
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8
Box 2
Fig. a)
Proportion Single Authored Articles
Num
ber o
f HC
0
20
40
60
80
100
120
0.0 0.2 0.4 0.6
Fig. b)
Proportion Single Authored Articles
Prop
ortio
n C
itatio
ns
0.0
0.2
0.4
0.6
0.8
●
●
●●
●
●● ●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
● ●
●
●
●
●
●
●
●●
● ●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●●
●●
●
●
●
●
●
●
●
●
●
● ●
●
●●
●●
●●
●
●●
●
●●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●●●
●
●
●
●
●●●●
●
●
●
●
●
● ●●
●●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
0.0 0.1 0.2 0.3 0.4 0.5 0.6
Fig. c)
Proportion First Authored Articles
Num
ber o
f HC
0
5
10
15
20
25
30
0.0 0.2 0.4 0.6 0.8
Fig. d)
Proportion First Authored Articles
Prop
ortio
n C
itatio
ns
0.0
0.2
0.4
0.6
0.8
1.0
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
0.0 0.2 0.4 0.6 0.8
Box 3
the metric elite misses not only diversity of people but ideas
# publications# citationsh-index g-index w-index a-index r-index e-index m-index hg-index q2-index i10-index
20052006
2011
1905
mcpp
timeline
1902
JIF
Why isn’t impact factor or citations sufficient?
h
h relies on citations to papers not journalsnot skewed by singletons
not influenced by large body of uncited papersminimizes politics of publication
useful for similar stage comparisonsapplied to any group
counts citations regardless of whydoes not account for variations in average numbers of pubs
ignores number & position of authorslimited by total number so juniors disadvantagedincreasing h at high levels difficult so compression
data looks backwards not forward
+-
analysis
psychology
contrasts
economics
finance
marketing
quality quantity
h solution
however at larger scales, N still important
overvalued scientists do publish significantly more
Matthew effect
m
options
i10
h derivations such as h-coreor g index
the solution is composite.
still numbers, just a different box
volume of scientific literature immense application
volume of scientific literature immense
curation connections
application
application
journals individuals
departmentscollections
curation
crowdsource
crowdsource
reputation economynot based on citation capital
journals/collections
editorssubject editors
refereesreaders
writers
dissemination pyramid
curation by connection - big data
not by merit or citations - by relationships
individuals ideas data place utility
science
merit
ideas
metrics
metrics
filters
publications
however, mini-manuscripts, figshare, slideshare, pre-print servers, and data publications are transforming the publication process in science and providing new opportunities for discovery.
metrics that illuminate & provide insight will be critical
ultimately, publications in all forms are extensions of learning
citations to datasets/figures in talk
Costas, R. and Borodons, M. 2007. The h-index: Advantages, limitations and its relation with other bibliometric indicators at the micro level. - The Journal of Informetrics 1: 193-203.
Harzing, A. W. and van der Wal, R. 2008. Comparing the Google Scholar h-index with the ISI Journal Impact Factor. - Resarch in International Management Products & Services for Academics Report.: 1-25.
Lortie, C. J., Aarssen, L. W., Budden, A. E., Koricheva, J., Leimu, R. and Tregenza, T. 2007. Publication bias and merit in ecology. - Oikos 116: 1247-1253.
Lortie, C. J., Aarssen, L. W., Parker, J. N. and Allesina, S. 2012. Good news for the people who love bad news: an analysis of the funding of the top 1% most highly cited ecologists. - Oikos 121: 1005-1008.
Lortie, C. J., Aarssen, L. W., Budden, A. E. and Leimu, R. 2012. Do citations and impact factors relate to the real numbers in publications? A case study of citation rates, impact, and effects sizes in ecology and evolutionary biology. - Scientometrics DOI: 10.1007/s11192-012-0822-6.
Marnett, A. 2013. H-Index: What It Is and How to Find Yours. - Benchfly blog.
Priem, J., Piwowar, H. and Hemminger, B. M. 2012. Altemtrics in the wild: using scoial media to explore scholarly impact. - arXiv 1203.4745v1.
Wardle, D. A. 2010. Do ‘Faculty of 1000’ (F1000) ratings of ecological publications serve as reasonable predictors of their future impact? . - Ideas in Ecology and Evolution 3: 11-15.
