A New Measure of Knowledge Diffusion

Stephen Carley; Alan Porter

Georgia Tech

Our Interests

• Interest in research interdisciplinarity Possible key to more creative research Possible means to solve complex scholarly and

societal problems • Interest in research knowledge diffusion

Trans-disciplinary knowledge enrichment Translation of research to clinical practice Fostering science-based innovation

Measures• Have devised metrics:

Integration score [how diverse are the research knowledge resources upon which an article (or body of research) draws?]

Specialization score[how diverse are the venues in which a body of research (e.g., the works of one author in one time period) appears?]

• Now adding -- Diffusion score [how diverse are the fields that cite an article (or body of research)?]

Heuristics of diversity

(Stirling, 1998; 2007)(Rafols and Meyer,

2009)

Diversity:Attribute of a system whose elements may be apportioned into categories

Characteristics: Variety: Number of distinctive categoriesBalance: Evenness of the distribution Disparity: Degree to which the categories

are different.

Variety

Balance Disparity

Herfindahl (concentration): i pi2

[** Shannon & Herfindahldo not includeDisparity]

Shannon (Entropy): i pi ln pi

Dissimilarity: i di

Generalised Diversity (Stirling) ij(ij) (pipj)a (dij)b

Integration• Integration Defined:

equivalently:

• Integration Range: 0 → 1• Integration Benchmarking: ~0.42 for much of modern

scientific research (increasing moderately)

where i = row, j = column, f = frequency, SC = Cited Web of Science Subject Category;cosine similarity of SC cross-citation (based on one year of Web of Science journal cross-citation)

ji

jiij ppsI,

1

Changes in Average Integration Scores over Time

Porter, A.L., and Rafols, I. "Is science becoming more interdisciplinary? Measuring and mapping six research fields over time." Scientometrics. 81.3 (2009): 719-745.

Diffusion Scoring Example

Paper 1 Paper 2

Cited by: Biology Genetics Math Music Forestry

• In the example above Paper 2 will likely have a higher diffusion score than Paper 1 because there is greater heterogeneity among its citing subject categories [variety & disparity; not clear on balance].

Diffusion Scoring• Diffusion Defined:

where:pi = proportion of citing references corresponding to the SC i in a given paper sij = cosine measure of similarity between SCs i & j for cross-citation in WOS

• Diffusion Range: 0 → 1• Diffusion Benchmarking: ~0.42 for much of

modern scientific research [but somewhat higher for papers “out there” longer]

Mean Diffusion scores forall benchmark years combined

Modest increase over time since publication

Diffusion Score Behavior• Diffusion score behavior is more challenging to study than

Integration:• Calculating Diffusion score requires additional “citation

searching” in Web of Science to retrieve the citing papers• A paper’s Diffusion score is not fixed; over time it will

accrue more citations• Self-citation could affect Diffusion score

• We examined – for 5 of our 6 target Subject Categories -- minimal effect

• For Math – removing self-cites increased scores for 1995 publications by 0.01-.02, until recent years [Since Math 1995 annual Diffusion scores are ~0.22, this is ~5-10% increase]

Self-Cites (1)

Self-Cites (2)

Diffusion Scores: Total vs. Annual

• Annual Diffusion may be of interest• e.g., to pursue Chen’s hypothesis that broad +

rapid diffusion may signal really impactful papers• To characterize research knowledge diffusion

patterns• Calculation issue

• We use a threshold of at least 3 Citing SCs for a paper to get a Diffusion Score[could be 1 cite by a paper whose journal is associated with 3 SCs (rare)]

• For Total score calculation this is less a factor than for Annual scores

For most of the 1995 benchmarks, Diffusion scores increase steadily with time. Mathematics is an outlier.

Mean Annual Diffusion Scores for 6 Subject Categories

Mean NeuroScience Diffusion scores for all 4 benchmark years

Diffusion Patterns

• Can we categorize research knowledge diffusion patterns? A first try here:• Steady growth• Staying power• Late-bloomer• Robust growth

• Next 4 slides illustrate for those 1995 publications in the 6 SCs that fit these patterns

Steady Growth Typology

Data Source: 1995 benchmarks

Staying Power Typology

Data Source: 1995 benchmarks

Late-Bloomer Typology

Powerhouse articles (N’s are small) tend to have high Diffusion scores

Data Source: 1995 benchmarks

Robust Growth Typology

Data Source: 1995 benchmarks

• Papers are citing more references over time -- does this exert upward pressure on Diffusion scores?

• What happens to the speed of diffusion over time? - Will electronic open source publishing boost Diffusion?

• How differently does research published in various Subject Categories diffuse?

• Chen: papers which are cited quickly and diffusely are more likely to be blockbusters

• Tool for tracking individual researchers or research groups’ influence

Diffusion Issues

Taking a Look at the Research of Robert Nerem

• Institute Professor Emeritus and Director, Georgia Tech/Emory Center for Regenerative Medicine

• 240 Web of Science (WOS) publications, in• 76 Journals, over• 47 Years, and more than• 8,000 Citations

Diffusion Scores v. Publication Age for Dr. Nerem’s Research

Increasing Diffusion score from his early papers (high Age) to more recent ones

For more on Integration see:

www.idr.gatech.edu Porter, Alan, Alex Cohen, David Roessner and Marty

Perreault. "Measuring researcher interdisciplinarity." Scientometrics. 72.1 (2007): 117-147.

Porter, Alan, David Roessner, and Anne Heberger. "How interdisciplinary is a given body of research?" Research Evaluation. 17.4 (2008): 273-282.

Porter, Alan, and Ismael Rafols. "Is science becoming more interdisciplinary? Measuring and mapping six research fields over time." Scientometrics. 81.3 (2009): 719-745.

For more on Diffusion see: Chen, Chaomei, and Diana Hicks. "Tracing knowledge

diffusion." Scientometrics. 59.2 (2004): 199-211. Chen, C., Chen, Y., Horowitz, M., Hou, H., Liu, Z., &

Pellegrino, D. (2009). Towards an explanatory and computational theory of scientific discovery. Journal of Informetrics, 3(3), 191-209

Yu, G., Wang, M.Y., Yu, D.R. “Characterizing knowledge diffusion of Nanoscience & Nanotechnology by citation analysis.” Scientometrics, 84.1 (2010): 81-97.

Carley, S. and Porter, A.L. (2011) "A Forward Diversity Index." Scientometrics, forthcoming.

Thank you!

Mean Biotech Diffusion scores for all 4 benchmark years

Mean EE Diffusion scores for all 4 benchmark years

Mean Math Diffusion scores for all 4 benchmark years

Mean Med-R&E Diffusion scores for all 4 benchmark years

Mean Phy-AMC Diffusion scores for all 4 benchmark years