Publizieren für den Index? - ft- · PDF filegoogle scholar Web page ... with Wikipedia...

Friedemann Mattern

F. Mattern, ETH Zurich, 2013 1

Publizieren für den Index? Was bibliometrische Evaluationswerkzeuge nicht ermöglichen und trotzdem anrichten

Friedemann Mattern ETH Zürich

We all publish –for many good reasons

Roles of publication

Making ones research available to others Keep a permanent record of research Feedback from community to researchers Claiming credit for the research Influence future research Reputation to authors …

+ please the citation index

Friedemann Mattern


Many of us have a personal google scholar Web page

For example, 50% of all CS faculty at ETH Zurich More popular among the younger scientists

Use “eth zurich computer science” or a similar query:

Within a few milliseconds you get the hitlist

“Evaluate” whole institutions…

Friedemann Mattern


Ω

A

Or use plain old google

Friedemann Mattern


23% of scholars have public Google Scholar profiles, 84% have web pages, and 16% are on Twitter. [Anna Samoilenko and Taha Yasseri, 2013]

Friedemann Mattern


Tools like google scholar…(…“Publish or Perish“, „Web of Science“,…)

Are useful and versatile tools for scientists Get immediate access to a paper (also preprints etc.) Find related papers (“where is this paper cited”?) Learn about recent papers by an author

But they also reveal much about the “impact” How many papers? How many citations? Bibliometric values Productivity and impact charts

The “h-index”

…a constructive definition:

Friedemann Mattern


0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

Num

ber

of c

itatio

ns

Histogram of all your papers sorted by decreasing citation counts

The “h-index”

0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

f(x) = x

Determine intersection point with f(x)=x diagonal

Friedemann Mattern


0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

f(x) = x

h-index cuts off contributionof highly cited papers

h-index cuts off long tail

0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

The h-index is this single number

h-Index

Friedemann Mattern


ICT progress as a game changer

Science Citation Index

Online access to publication databases

Instant rating tools Everywhere, always,

free, easy to use Evaluations with our finger-

tips instead of our intellect

Big Data

Ubiquitous networking

Cloudservices

Questions

Impact on the publication culture? Use of such tools for evaluation purposes? Measure popularity impact importance quality?

?

Bibliometry of CS compared to other fields? Ranking of Departments etc.?

Friedemann Mattern


Bibliometry has become popular

Politics and the public want to have simple indicators transparency (in a globalized and complex world)

More “objective” than peer review?

Bibliographic databases

ISI Thomson Reuters Traditionally only few conference proceedings and books Emphasis on natural and life sciences Technical sciences are under-represented

Scopus Owned by Elsevier, covers 19500 titles, including 16500 peer-

reviewed journals, mostly articles published after 1995

Google scholar uses its own „database“ It scans the Web for articles

[+ some others]

The choice of the underlying database greatly influences the bibliometric indicators

Friedemann Mattern


Computer science in the bibliographic databases

Why is this relevant?

CS gives a weak impression relative to other fields CS has a particular publication culture De facto standards in bibliometric evaluation (ISI, Scopus)

make many of our papers invisible In the eyes of the classical sciences, we cannot be taken serious

Friedemann Mattern

F.Ma. 43

ISI-Coverage is very different for different disciplines

Physics, chemistry,biology: ~60%

CS: 14%

Analysis of all publications from ETH Zurich:

Friedemann Mattern


Friedemann Mattern

F.Ma. 44

Does ISI cover at least all relevant publications?

[3] Nat. J. Obscurity

[26] Detailed statistics in TR 314

[7] Personal communication

ISI

Non-ISI

Never catch 100%

Relevant = cited How many [non] ISI papers do ISI papers cite?

%?

Friedemann Mattern

F.Ma. 45

ISI internal coverage

Biology:

90%

CS:

40%

ISI misses more than 50% of all publications considered relevant by the CS-community better in theoretical CS, worse in practical CS

Adding proceedingsfrom ACM, IEEE-CS,and LNCS yields 51%Henk F. Moed and Martijn S. Visser: Developing BibliometricIndicators of Research Performance in Computer Science. CWTS, 2007

Bas

ed o

n th

e 20

03 E

TH

Zur

ich

publ

icat

ion

pool

ISI

Friedemann Mattern


Friedemann Mattern

F.Ma. 72

Different disciplines

M.

Am

in, M

. M

abe:

Impa

ct fa

ctor

: use

and

abu

se. P

ersp

ectiv

es

in P

ublis

hing

, N

o. 1

, Oct

ober

200

0, p

p. 1

-6

Average citations per article

Friedemann Mattern

F.Ma. 74

100 researchers from the top 5% from 6disciplines

Global rank by h-index

Loca

l ran

k

Local rank 40 in Biochemistrymeans „top 18%“ globally

Local rank 40 in Mathematicsmeans „least 20%“ globally

Jasleen Kaur, Filippo Radicchi, Filippo Menczer: Universality of scholarly impact metrics, 2013

Different disciplines: Ranking by h-index

Friedemann Mattern


Friedemann Mattern

F.Ma. 75

How to deal with Bioinformaticswithin CS?

Global rank by h-index

Loca

l ran

k

Local rank 40 in Biochemistrymeans „top 18%“ globally

Local rank 40 in Mathematicsmeans „least 20 %“ globally

Different disciplines: Ranking by h-index

Discipline 1

Discipline 2

Friedemann Mattern

F.Ma. 78

Survey papers versus research papers

Years after publication

Cita

tions

M.

Am

in, M

. M

abe:

Impa

ct fa

ctor

: use

and

abu

se. P

ersp

ectiv

es

in P

ublis

hing

, N

o. 1

, Oct

ober

200

0, p

p. 1

-6

Survey

Research paper

Friedemann Mattern


Friedemann Mattern

F.Ma. 81

Do citations measure quality?

If not, what else? Why do we care about the h-index? Does popularity at least correlate with “quality”?

An example – two of my own (co-authored) papers:

The derivation of distributed termination detection algorithms from garbage collection schemes, ACM TOPLAS 15(1), 1-35, 1993 cited by 87 (4.4 per year)

The design space of wireless sensor networks, Wireless Communications, IEEE, 11(6), 54-61, 2004 cited by 1225 (122 per year)

Factor of 28

This solves an important problem

This mentions many problems

Friedemann Mattern

F.Ma. 82

Do citations measure quality?

If not, what else? Why do we care about the h-index? Does popularity at least correlate with “quality”?

An example – two of my own (co-authored) papers:

The derivation of distributed termination detection algorithms from garbage collection schemes, ACM TOPLAS 15(1), 1-35, 1993 cited by 87 (4.4 per year)

The design space of wireless sensor networks, Wireless Communications, IEEE, 11(6), 54-61, 2004

This mentions many problems

Friedemann Mattern


Abourt 200 citations per year – is that much?

Friedemann Mattern


Why is Einstein‘s paper, which gave rise to the theory of relativity, not cited much more often than our sensor network design space paper?

Friedemann Mattern


Friedemann Mattern

F.Ma. 94

How I became Einstein‘s co-author

Friedemann Mattern

F.Ma. 95

How I became Einstein‘s co-author

Zur Evaluation der Informatik mittels bibliometrischer.. - Einstein.. (Correct)…Analyse Nicht alles was zhlt, kann gezhlt werden, und nicht alles was gezhlt werden kann, zhlt! Albert Einstein, Friedemann Mattern, ETH Zurich, Switzerland

Friedemann Mattern


Friedemann Mattern

F.Ma. 96

The „Einstein & Mattern“ paper

22

Einstein

Friedemann Mattern


Nicht alles was zählt, kann gezählt werden, und nicht alles

was gezählt werden kann, zählt!

Albert Einstein

Friedemann Mattern


Friedemann Mattern

F.Ma. 113

What do bibliometricindicators measure?

Correlations… with probability being a Turing award winner? with third-party money indicators? with Wikipedia notability?

Year Recipient # Cit. Rank

1984 Niklaus Wirth 946 1245 1985 Richard M. Karp 4951 24 1986 John Hopcroft 4542 34 1986 Robert Tarjan 6525 7 1987 John Cocke 1074 1017 1988 Ivan Sutherland 663 2152 1989 William (Velvel) Kahan 413 3973 1990 Fernando J. Corbato’ 34 ∞1991 Robin Milner 7900 4 1992 Butler W. Lampson 1643 471 1993 Juris Hartmanis 742 1817 1993 Richard E. Stearns 380 4434 1994 Edward Feigenbaum 363 4684 1994 Raj Reddy 270 6703 1995 Manuel Blum 1704 442 1996 Amir Pnueli 5212 19 1997 Douglas Engelbart 113 ∞1998 James Gray 3945 50 1999 Frederick P. Brooks, Jr. 908 1332 2000 Andrew Chi-Chih Yao 2019 304 2001 Ole-Johan Dahl 505 3094 2001 Kristen Nygaard 498 3161 2002 Ronald L. Rivest 6930 5 2002 Adi Shamir 3492 76 2002 Leonard M. Adleman 1746 418

Citation ranking of theTuring award recipientsaccording to Citeseer

Dror G. Feitelson and Uri Yovel: Predictive Ranking of Computer Scientists Using CiteSeer Data, May 2003

Esteem of the com-munity does not corre-late with # of citations

Friedemann Mattern


Friedemann Mattern

F.Ma. 115

CS Departments atGerman Universities

Third party money vs. citations

Corr.coeff. = 0.23

Consequences if twosensible performancemeasures are onlyweakly correlated?

Third party money/scholar vs. citations/faculty for whole

See: Bernhard Nebel: Ranking? Informatik-Spektrum 4:24, pp. 234-249, Aug. 2001

Citations

3rd party money

Friedemann Mattern

F.Ma. 116

The h-index and Wikipedia notability

Friedemann Mattern


Friedemann Mattern

F.Ma. 117

Wikipedia notability guidelines for academics http://en.wikipedia.org/wiki/Wikipedia:Notability_%28academics%29

“1. The person’s research has made significant impact in their scholarly discipline [...]

The most typical way […] is to show that the academic has been an author of highly cited academic work.”

Is appearance in Wikipedia correlated with citation metrics?

Analysis of 400 Wikipedia articles on academics from Biology, Computer Science, Physics, Psychology [Anna Samoilenko and Taha Yasseri, 2013]

Friedemann Mattern

F.Ma. 118

h-index distribution of Wikipedia-featured academics

Only few have high h-values

Friedemann Mattern


Friedemann Mattern

F.Ma. 119


Similar for other disciplines

Friedemann Mattern

F.Ma. 120


[Anna Samoilenko and Taha Yasseri, 2013]

“The majority of the re-searchers’ biographies on Wikipedia do not meet the primary notability criterion”

Correlation Wikipedia / h-index? metrics: existence, length, number

of edits, number of incoming links,…

“We found no statistically significant correlation”

Only few have high h-values

Friedemann Mattern


Friedemann Mattern

F.Ma. 121

Correlations?

With probability being a Turing award winner? With third-party money indicators? With Wikipedia notability?

With “being a good scientist”

Friedemann Mattern

F.Ma. 122

The subtle influence of instant rating tools

Instant rating tools are becoming more and more popular Search committee members often check the h-index of

the persons in question The CVs of applicants now often mention the h-index Some scientists even advertise their h-index on the front

page of their web site

The h-index (and other bibl. indicators) gains importance People adapt

Friedemann Mattern


www.google.com/trends shows how often a particular search term is entered relative to the total Google search volume

Popularity increase

“h-index” relative to “bibliometric”

Appointments Committee

Expert's evaluation of applicants for Professorship in

most important criteria, in general terms, must be the impact of the scientific output of the candidates. This has traditionally been measured using the number of publications

A fairly recent, useful measure for evaluating impact is the so-called h-index. A scholarhas an index of h if he or she has published h papers each of which has been cited by others

The following table lists the number of citations of the most influential publication(with most citations) from each applicant, as well as the h-index of each applicant. Thenumbers are based on a Google Scholar search on

Although the differences are not huge, the group of top candidates emerges clearly:

Candidate Citations for top paper h-index29 851 642 8

„Publish or perish“Be cited

Friedemann Mattern


Friedemann Mattern

F.Ma. 126

“Be cited or perish” –how will authors adapt?

Publish even more “The average number of authors on research papers, which now

stands at 4.5, has doubled since 1980” (Nature, 18 Oct 2012)

Publish differently Advertize papers (and venues)

“Cite my paper in this way, just copy & paste this:…” “Our authors see extremely wide access to their work” “You might also be interested in that paper” Advertizing one’s own publications in blogs and social media …

Friedemann Mattern

F.Ma. 127

How to increase your bibliometric values

Write your name on papers by your PhD students Ignore your publisher’s copyright: put your paper online Work in a popular area so that many others can cite you Write survey papers, not research papers Never change your established research area Avoid innovative and new (but risky) projects Chose catchy titles for your papers Emphasize quantity instead of quality Do not lose valuable time, avoid events like this one Concentrate on paper production, not good teaching Heavily cite your own (and your friend’s) papers Never publish more than a single “Least Publishable Unit” Cannibalize your old papers: refurbish and republish them

Friedemann Mattern


Friedemann Mattern

F.Ma. 129

Mighty tools

Computer Science brings mighty tools to the people Usable by non-experts

Google scholar + iPad + WiFi=

instant rating tool

Friedemann Mattern

F.Ma. 130

Computer power to the people!

People's Computer Company, Menlo Park, CA, early 70ies

Friedemann Mattern


Friedemann Mattern

F.Ma. 133


40 years later

Bildquelle.: “Forschung und Lehre”

Friedemann Mattern


Friedemann Mattern

F.Ma. 135

Ω

A


Friedemann Mattern

F.Ma. 136


CameraHigh Resolution

Display

Internet

Google Glass

Friedemann Mattern


The Future of Instant Rating

Friedemann Mattern


Friedemann Mattern



This is not a publication.The material is intendedfor one-time educationaluse only. Pictures shouldnot be copied or redistrib-uted. No violations of copy-rights are intended.– Friedeman Mattern

Publizieren für den Index? Was bibliometrische Evaluationswerkzeuge nicht ermöglichen und trotzdem anrichten

Friedemann Mattern ETH Zürich

Publizieren für den Index? - ft- · PDF filegoogle scholar Web page ... with Wikipedia...

Documents

Transcript of Publizieren für den Index? - ft- · PDF filegoogle scholar Web page ... with Wikipedia...