Indicators of Innovative Research (Klavans, Boyack, Small, Sorensen, Ioannidis)

Physics Chemistry Engineering Biology Disease Medicine Brain Health SocialComputer HumanitiesEarth

Indicators of Innovative Research2014 Science & Technology Indicator Conference

Leiden UniversityThe Netherlands

Sept 3, 2014

Richard Klavans (presenter)Kevin W. BoyackHenry SmallAaron A. SorensenJohn P.A. Iaonnidis

SciTech Strategies Berwyn, PA.SciTech Strategies, Albuquerque, NMSciTech Strategies, Bala Cynwd, PA.

Temple University School of Medicine, Philadelphia, PA.Stanford University School of Medicine, Stanford, CA.

2


“What gets us into trouble is not what we don't know. It's what we know for sure that just ain't so.”

Mark Twain

Motivation

3


Motivation

Ioannidis et al [Nature, 2013] assumes that highly cited papers are more innovative

4


Motivation

We hope that the ‘distance’ betweena citing paper & its references will correlate with innovativeness


5


Motivation

Uzzi et al. [Science, 2013]argue that one needs a combination oftypical & atypicalknowledge relationships in order to be innovative. He also assumes thatinnovative papers arehighly cited.

We hope that the ‘distance’ betweena citing paper & its references will correlate with innovativeness


6


Is this so?

Invited 400 biomedical researchers to participate in a survey» Invitation came from Ioannidis (Stanford University)» Researchers had over 25,000 citations and H index > 75» Participation rate was 30.8% (123 out of 400)

Researchers were asked to rate their top 10 (most cited) papers» Papers had to be published between 2005-2008 (& in the Scopus database)» Ratings were along six dimensions of impact

Survey results will be compared with bibliometric indicators» Traditional Indicators» Indicators suggested by K&B [2013]» Indicators suggested by Uzzi et al. [2013]

7


Survey Design

8


Survey Analysis

Assigned Papers to Impact Categories

9


Bibliometric Data*

Common Sense 1. Review Paper?2. # of References3. Team Science (# of Authors)

K&B [2013]4. Distance between citing paper and cited references

USMJ [2013] 5. Typical Knowledge Relationships6. Atypical Knowledge Relationships

* Scopus Data

10


Finding #1:

Most High Impact Papers represent ‘Normal Science’

Type of Impact # of Papers % of Sample

Progress 335.4 28.8%

Synthesis 262.9 21.3%

Broader Interest 220.0 17.8%

Innovation 195.9 15.9%

Surprise 99.3 8.0%

Difficulty 41.6 3.4%

Unassigned 58.0 4.7%

68.9%

27.3%

11


Finding #2:

Traditional indicators work (but ‘team science’ is associated with the less innovative

papers!) Type of Impact % Review # Ref #Authors

Progress 12.5 51.7 17.8

Synthesis 26.1 71.0 16.2

Broader Interest 17.3 59.8 15.5

Innovation 6.3 42.9 12.3

Surprise 4.5 41.7 11.4

Difficulty 12.7 51.1 9.8

Unassigned 12.1 55.0 14.9

F-stat 13.8 9.63 1.90

Probability .0000 .0000 .0773

12


Finding #3:

‘Distance’ does not identify innovative papers

Type of Impact Distance

Progress 2.05

Synthesis 2.01

Broader Interest 2.13

Innovation 2.05

Surprise 2.04

Difficulty 1.98

Unassigned 2.04

F-Stat .42

Probability .867

13


Finding #4:

Only atypical knowledge relationships works

Type of Impact Atypical Typical

Progress .491 .511

Synthesis .506 .480

Broader Interest .574 .475

Innovation .636 .467

Surprise .608 .500

Difficulty .620 .451

Unassigned .672 .500

F-stat 4.61 .41

Probability .0001 .875

14


Finding #5

And the Overall Explanatory Value is very low (R2~.033)

Regression Equation Dependent Innovation or Surprise (0,1 variable) Variable:

t P>|t| Independent Review Paper (0,1) -5.14 .000 Variables: References Log(# ref) -2.08 .037

Team Log(# authors) -1.81 .071Atypical (0,1) 3.87 .000

15


highly cited papers are not necessarily innovative.

Implications

‘distance’ didn’t work.

Atypical knowledge relationships works….

but why?

16


Implications

And we still don’t know how to identify innovative papers

17


Thank-you for your attention!

Indicators of Innovative Research (Klavans, Boyack, Small, Sorensen, Ioannidis)

Science

Transcript of Indicators of Innovative Research (Klavans, Boyack, Small, Sorensen, Ioannidis)