OFFICE OF PORTFOLIO ANALYSIS

September 5 2012

The Office of Portfolio Analysis (OPA)

OPA was established in 2011 to provide multiple services

Scientific portfolio analysishellip

o of all Common Fund initiatives

o in response to requests by senior leadership

Coordinate portfolio analysis activities across NIH

Train NIH staff to promote the effective use of analytical tools

o Regularly scheduled courses

o Ad hoc consultations

Improve portfolio analysis at NIH (see next slide)

Office of Portfolio Analysis (OPA) DPCPSI OD 2

Improve Portfolio Analysis at NIH

Solve NIH-wide limitations in cross-referencing databases

Build new computational tools that address specific NIH needs

Model NIH output and health impact accurately

o Advanced bibliometrics

Track progress in the new field of ldquoscience of sciencerdquo and find

useful methods and synergies among the parallel efforts inhellip

o Academia

o Other government agencies

o Private sector

Adopt new methods that meet NIH needs and provide training

opportunities for portfolio analysis stakeholders at NIH

Office of Portfolio Analysis (OPA) DPCPSI OD 3

Scientific Portfolio Analysis at NIH

Data-driven approaches to program development

What is the best way to accelerate scientific progress

An example

OPA analysis of 38 Metabolomics Centers in the US

most of which receive funding from NIH (~$70M in 2010)

Office of Portfolio Analysis (OPA) DPCPSI OD 4

$225M Global Investment in Metabolomics in FY10

bull Netherlands Metabolomics Center $67M

bull BBSRC UK Plant and Microbial Metabolomics $104M

bull Canadian Human Metabolome Database $81M

Office of Portfolio Analysis (OPA) DPCPSI OD 5

Europe

Japan

FY09 Metabolomics Co-authorship Networks

USA

Office of Portfolio Analysis (OPA) DPCPSI OD 6

Scientific Portfolio Analysis at NIH

What is the best way to accelerate scientific progress

OPA analysis of 38 Metabolomics Centers in the US most of which receive funding from NIH (~$70M in 2010)

These Centers have overlapping goals yet operated in isolation

with little coordination or collaboration

Portfolio analysis shaped a new Metabolomics RFA aimed at

improving coordination and leveraging existing resources

Office of Portfolio Analysis (OPA) DPCPSI OD 7

o httpdpcpsinihgovportfolio_analysis

Coordination of Portfolio Analysis Efforts at NIH

Portfolio Analysis Workshop (February 6 Natcher Conf Ctr)

Portfolio Analysis Symposium (July 23-24 Natcher Conf Ctr)

Build a computer lab to tailor existing and new computational tools to

NIH needs and to train NIH staff in their use

Centralized web-based repository to disseminate computational tools

Standing trans-NIH Working Group

Office of Portfolio Analysis (OPA) DPCPSI OD 8

The Office of Portfolio Analysis (OPA)

OPA was established in 2011 to provide multiple services

Scientific portfolio analysishellip

o of all Common Fund initiatives

o in response to requests by senior leadership

Coordinate portfolio analysis activities across NIH

Train NIH staff to promote the effective use of analytical tools

o Regularly scheduled courses

o Ad hoc consultations

Improve portfolio analysis at NIH (see next slide)

Office of Portfolio Analysis (OPA) DPCPSI OD 2

Improve Portfolio Analysis at NIH

Solve NIH-wide limitations in cross-referencing databases

Build new computational tools that address specific NIH needs

Model NIH output and health impact accurately

o Advanced bibliometrics

Track progress in the new field of ldquoscience of sciencerdquo and find

useful methods and synergies among the parallel efforts inhellip

o Academia

o Other government agencies

o Private sector

Adopt new methods that meet NIH needs and provide training

opportunities for portfolio analysis stakeholders at NIH

Office of Portfolio Analysis (OPA) DPCPSI OD 3

Scientific Portfolio Analysis at NIH

Data-driven approaches to program development

What is the best way to accelerate scientific progress

An example

OPA analysis of 38 Metabolomics Centers in the US

most of which receive funding from NIH (~$70M in 2010)

Office of Portfolio Analysis (OPA) DPCPSI OD 4

$225M Global Investment in Metabolomics in FY10

bull Netherlands Metabolomics Center $67M

bull BBSRC UK Plant and Microbial Metabolomics $104M

bull Canadian Human Metabolome Database $81M

Office of Portfolio Analysis (OPA) DPCPSI OD 5

Europe

Japan

FY09 Metabolomics Co-authorship Networks

USA

Office of Portfolio Analysis (OPA) DPCPSI OD 6

Scientific Portfolio Analysis at NIH

What is the best way to accelerate scientific progress

OPA analysis of 38 Metabolomics Centers in the US most of which receive funding from NIH (~$70M in 2010)

These Centers have overlapping goals yet operated in isolation

with little coordination or collaboration

Portfolio analysis shaped a new Metabolomics RFA aimed at

improving coordination and leveraging existing resources

Office of Portfolio Analysis (OPA) DPCPSI OD 7

o httpdpcpsinihgovportfolio_analysis

Coordination of Portfolio Analysis Efforts at NIH

Portfolio Analysis Workshop (February 6 Natcher Conf Ctr)

Portfolio Analysis Symposium (July 23-24 Natcher Conf Ctr)

Build a computer lab to tailor existing and new computational tools to

NIH needs and to train NIH staff in their use

Centralized web-based repository to disseminate computational tools

Standing trans-NIH Working Group

Office of Portfolio Analysis (OPA) DPCPSI OD 8

Improve Portfolio Analysis at NIH

Solve NIH-wide limitations in cross-referencing databases

Build new computational tools that address specific NIH needs

Model NIH output and health impact accurately

o Advanced bibliometrics

Track progress in the new field of ldquoscience of sciencerdquo and find

useful methods and synergies among the parallel efforts inhellip

o Academia

o Other government agencies

o Private sector

Adopt new methods that meet NIH needs and provide training

opportunities for portfolio analysis stakeholders at NIH

Office of Portfolio Analysis (OPA) DPCPSI OD 3

Scientific Portfolio Analysis at NIH

Data-driven approaches to program development

What is the best way to accelerate scientific progress

An example

OPA analysis of 38 Metabolomics Centers in the US

most of which receive funding from NIH (~$70M in 2010)

Office of Portfolio Analysis (OPA) DPCPSI OD 4

$225M Global Investment in Metabolomics in FY10

bull Netherlands Metabolomics Center $67M

bull BBSRC UK Plant and Microbial Metabolomics $104M

bull Canadian Human Metabolome Database $81M

Office of Portfolio Analysis (OPA) DPCPSI OD 5

Europe

Japan

FY09 Metabolomics Co-authorship Networks

USA

Office of Portfolio Analysis (OPA) DPCPSI OD 6

Scientific Portfolio Analysis at NIH

What is the best way to accelerate scientific progress

OPA analysis of 38 Metabolomics Centers in the US most of which receive funding from NIH (~$70M in 2010)

These Centers have overlapping goals yet operated in isolation

with little coordination or collaboration

Portfolio analysis shaped a new Metabolomics RFA aimed at

improving coordination and leveraging existing resources

Office of Portfolio Analysis (OPA) DPCPSI OD 7

o httpdpcpsinihgovportfolio_analysis

Coordination of Portfolio Analysis Efforts at NIH

Portfolio Analysis Workshop (February 6 Natcher Conf Ctr)

Portfolio Analysis Symposium (July 23-24 Natcher Conf Ctr)

Build a computer lab to tailor existing and new computational tools to

NIH needs and to train NIH staff in their use

Centralized web-based repository to disseminate computational tools

Standing trans-NIH Working Group

Office of Portfolio Analysis (OPA) DPCPSI OD 8

Scientific Portfolio Analysis at NIH

Data-driven approaches to program development

What is the best way to accelerate scientific progress

An example

OPA analysis of 38 Metabolomics Centers in the US

most of which receive funding from NIH (~$70M in 2010)

Office of Portfolio Analysis (OPA) DPCPSI OD 4

$225M Global Investment in Metabolomics in FY10

bull Netherlands Metabolomics Center $67M

bull BBSRC UK Plant and Microbial Metabolomics $104M

bull Canadian Human Metabolome Database $81M

Office of Portfolio Analysis (OPA) DPCPSI OD 5

Europe

Japan

FY09 Metabolomics Co-authorship Networks

USA

Office of Portfolio Analysis (OPA) DPCPSI OD 6

Scientific Portfolio Analysis at NIH

What is the best way to accelerate scientific progress

OPA analysis of 38 Metabolomics Centers in the US most of which receive funding from NIH (~$70M in 2010)

These Centers have overlapping goals yet operated in isolation

with little coordination or collaboration

Portfolio analysis shaped a new Metabolomics RFA aimed at

improving coordination and leveraging existing resources

Office of Portfolio Analysis (OPA) DPCPSI OD 7

o httpdpcpsinihgovportfolio_analysis

Coordination of Portfolio Analysis Efforts at NIH

Portfolio Analysis Workshop (February 6 Natcher Conf Ctr)

Portfolio Analysis Symposium (July 23-24 Natcher Conf Ctr)

Build a computer lab to tailor existing and new computational tools to

NIH needs and to train NIH staff in their use

Centralized web-based repository to disseminate computational tools

Standing trans-NIH Working Group

Office of Portfolio Analysis (OPA) DPCPSI OD 8

$225M Global Investment in Metabolomics in FY10

bull Netherlands Metabolomics Center $67M

bull BBSRC UK Plant and Microbial Metabolomics $104M

bull Canadian Human Metabolome Database $81M

Office of Portfolio Analysis (OPA) DPCPSI OD 5

Europe

Japan

FY09 Metabolomics Co-authorship Networks

USA

Office of Portfolio Analysis (OPA) DPCPSI OD 6

Scientific Portfolio Analysis at NIH

What is the best way to accelerate scientific progress

OPA analysis of 38 Metabolomics Centers in the US most of which receive funding from NIH (~$70M in 2010)

These Centers have overlapping goals yet operated in isolation

with little coordination or collaboration

Portfolio analysis shaped a new Metabolomics RFA aimed at

improving coordination and leveraging existing resources

Office of Portfolio Analysis (OPA) DPCPSI OD 7

o httpdpcpsinihgovportfolio_analysis

Coordination of Portfolio Analysis Efforts at NIH

Portfolio Analysis Workshop (February 6 Natcher Conf Ctr)

Portfolio Analysis Symposium (July 23-24 Natcher Conf Ctr)

Build a computer lab to tailor existing and new computational tools to

NIH needs and to train NIH staff in their use

Centralized web-based repository to disseminate computational tools

Standing trans-NIH Working Group

Office of Portfolio Analysis (OPA) DPCPSI OD 8

Europe

Japan

FY09 Metabolomics Co-authorship Networks

USA

Office of Portfolio Analysis (OPA) DPCPSI OD 6

Scientific Portfolio Analysis at NIH

What is the best way to accelerate scientific progress

OPA analysis of 38 Metabolomics Centers in the US most of which receive funding from NIH (~$70M in 2010)

These Centers have overlapping goals yet operated in isolation

with little coordination or collaboration

Portfolio analysis shaped a new Metabolomics RFA aimed at

improving coordination and leveraging existing resources

Office of Portfolio Analysis (OPA) DPCPSI OD 7

o httpdpcpsinihgovportfolio_analysis

Coordination of Portfolio Analysis Efforts at NIH

Portfolio Analysis Workshop (February 6 Natcher Conf Ctr)

Portfolio Analysis Symposium (July 23-24 Natcher Conf Ctr)

Build a computer lab to tailor existing and new computational tools to

NIH needs and to train NIH staff in their use

Centralized web-based repository to disseminate computational tools

Standing trans-NIH Working Group

Office of Portfolio Analysis (OPA) DPCPSI OD 8

Scientific Portfolio Analysis at NIH

What is the best way to accelerate scientific progress

OPA analysis of 38 Metabolomics Centers in the US most of which receive funding from NIH (~$70M in 2010)

These Centers have overlapping goals yet operated in isolation

with little coordination or collaboration

Portfolio analysis shaped a new Metabolomics RFA aimed at

improving coordination and leveraging existing resources

Office of Portfolio Analysis (OPA) DPCPSI OD 7

o httpdpcpsinihgovportfolio_analysis

Coordination of Portfolio Analysis Efforts at NIH

Portfolio Analysis Workshop (February 6 Natcher Conf Ctr)

Portfolio Analysis Symposium (July 23-24 Natcher Conf Ctr)

Build a computer lab to tailor existing and new computational tools to

NIH needs and to train NIH staff in their use

Centralized web-based repository to disseminate computational tools

Standing trans-NIH Working Group

Office of Portfolio Analysis (OPA) DPCPSI OD 8

o httpdpcpsinihgovportfolio_analysis

Coordination of Portfolio Analysis Efforts at NIH

Portfolio Analysis Workshop (February 6 Natcher Conf Ctr)

Portfolio Analysis Symposium (July 23-24 Natcher Conf Ctr)

Build a computer lab to tailor existing and new computational tools to

NIH needs and to train NIH staff in their use

Centralized web-based repository to disseminate computational tools

Standing trans-NIH Working Group

Office of Portfolio Analysis (OPA) DPCPSI OD 8

Goals of the Portfolio Analysis Workshop Feb 5-6 2012

Discuss perceived needs in portfolio analysis

Use needs assessment to plan the Portfolio Analysis Symposium

(July 23-24 Natcher Conf Ctr)

Cover topics of broad interest to NIH decision-makers including

o Strategic planning

o Uses of portfolio analysis

o Overlap in NIH portfolios

o Measuring impact

o New portfolio analysis tools

o Identification of emerging areas

Office of Portfolio Analysis (OPA) DPCPSI OD 9

90

80

70

60

50

40

30

20

10

0

Outcomes of the Portfolio Analysis Workshop

o Full registration within 48 hrs of the announcement

o gt500 participants ndash approved for ESA training credit

Office of Portfolio Analysis (OPA) DPCPSI OD 10

Outcomes of the Portfolio Analysis Workshop

Survey results

Highest priority Topics included

o Measuring impact 47

o Gaps and overlap in NIH portfolios 30

o Identification of emerging areas 18

o Categorizing portfolios 5

What needs should OPA try to address

o Build better tools easier to use tools 65

o Provide training and support 50

o Develop targeted case studies 15

Office of Portfolio Analysis (OPA) DPCPSI OD 11

Goals of the Portfolio Analysis Symposium July 23-24 2012

Bring outside experts from academia government and the

private sector to discuss and demo state-of-the-art

approaches in scientific portfolio analysis

Choose those with expertise in areas identified as critical and

of greatest interest to portfolio analysis stakeholders at NIH

o Measuring Impact

o Identifying Gaps and Impact

o Identifying Emerging Areas

Facilitate the development of collaborations that address NIH

needs in scientific portfolio analysis

Office of Portfolio Analysis (OPA) DPCPSI OD 12

Outcomes of the Portfolio Analysis Symposium

Links to the Symposium agenda and archived videocast are

available on the OPA web site

httpdpcpsinihgovopaindexaspx

Outside experts presented new tools and approaches in each

topic one of these tools (Sci2) was particularly well received

by Symposium attendees and is being added to QVR

The Symposium resulted in several collaborations Examples

include OPA initiatives in building tools for co-author network

analysis analysis of patent development and tracking of

patent licenses

Office of Portfolio Analysis (OPA) DPCPSI OD 13

90

80

70

60

50

40

30

20

10

0

Outcomes of the Portfolio Analysis Workshop

o Full registration within 48 hrs of the announcement

o gt500 participants ndash approved for ESA training credit

Office of Portfolio Analysis (OPA) DPCPSI OD 10

Outcomes of the Portfolio Analysis Workshop

Survey results

Highest priority Topics included

o Measuring impact 47

o Gaps and overlap in NIH portfolios 30

o Identification of emerging areas 18

o Categorizing portfolios 5

What needs should OPA try to address

o Build better tools easier to use tools 65

o Provide training and support 50

o Develop targeted case studies 15

Office of Portfolio Analysis (OPA) DPCPSI OD 11

Goals of the Portfolio Analysis Symposium July 23-24 2012

Bring outside experts from academia government and the

private sector to discuss and demo state-of-the-art

approaches in scientific portfolio analysis

Choose those with expertise in areas identified as critical and

of greatest interest to portfolio analysis stakeholders at NIH

o Measuring Impact

o Identifying Gaps and Impact

o Identifying Emerging Areas

Facilitate the development of collaborations that address NIH

needs in scientific portfolio analysis

Office of Portfolio Analysis (OPA) DPCPSI OD 12

Outcomes of the Portfolio Analysis Symposium

Links to the Symposium agenda and archived videocast are

available on the OPA web site

httpdpcpsinihgovopaindexaspx

Outside experts presented new tools and approaches in each

topic one of these tools (Sci2) was particularly well received

by Symposium attendees and is being added to QVR

The Symposium resulted in several collaborations Examples

include OPA initiatives in building tools for co-author network

analysis analysis of patent development and tracking of

patent licenses

Office of Portfolio Analysis (OPA) DPCPSI OD 13

Outcomes of the Portfolio Analysis Workshop

Survey results

Highest priority Topics included

o Measuring impact 47

o Gaps and overlap in NIH portfolios 30

o Identification of emerging areas 18

o Categorizing portfolios 5

What needs should OPA try to address

o Build better tools easier to use tools 65

o Provide training and support 50

o Develop targeted case studies 15

Office of Portfolio Analysis (OPA) DPCPSI OD 11

Goals of the Portfolio Analysis Symposium July 23-24 2012

Bring outside experts from academia government and the

private sector to discuss and demo state-of-the-art

approaches in scientific portfolio analysis

Choose those with expertise in areas identified as critical and

of greatest interest to portfolio analysis stakeholders at NIH

o Measuring Impact

o Identifying Gaps and Impact

o Identifying Emerging Areas

Facilitate the development of collaborations that address NIH

needs in scientific portfolio analysis

Office of Portfolio Analysis (OPA) DPCPSI OD 12

Outcomes of the Portfolio Analysis Symposium

Links to the Symposium agenda and archived videocast are

available on the OPA web site

httpdpcpsinihgovopaindexaspx

Outside experts presented new tools and approaches in each

topic one of these tools (Sci2) was particularly well received

by Symposium attendees and is being added to QVR

The Symposium resulted in several collaborations Examples

include OPA initiatives in building tools for co-author network

analysis analysis of patent development and tracking of

patent licenses

Office of Portfolio Analysis (OPA) DPCPSI OD 13

Goals of the Portfolio Analysis Symposium July 23-24 2012

Bring outside experts from academia government and the

private sector to discuss and demo state-of-the-art

approaches in scientific portfolio analysis

Choose those with expertise in areas identified as critical and

of greatest interest to portfolio analysis stakeholders at NIH

o Measuring Impact

o Identifying Gaps and Impact

o Identifying Emerging Areas

Facilitate the development of collaborations that address NIH

needs in scientific portfolio analysis

Office of Portfolio Analysis (OPA) DPCPSI OD 12

Outcomes of the Portfolio Analysis Symposium

Links to the Symposium agenda and archived videocast are

available on the OPA web site

httpdpcpsinihgovopaindexaspx

Outside experts presented new tools and approaches in each

topic one of these tools (Sci2) was particularly well received

by Symposium attendees and is being added to QVR

The Symposium resulted in several collaborations Examples

include OPA initiatives in building tools for co-author network

analysis analysis of patent development and tracking of

patent licenses

Office of Portfolio Analysis (OPA) DPCPSI OD 13

Outcomes of the Portfolio Analysis Symposium

Links to the Symposium agenda and archived videocast are

available on the OPA web site

httpdpcpsinihgovopaindexaspx

Outside experts presented new tools and approaches in each

topic one of these tools (Sci2) was particularly well received

by Symposium attendees and is being added to QVR

The Symposium resulted in several collaborations Examples

include OPA initiatives in building tools for co-author network

analysis analysis of patent development and tracking of

patent licenses

Office of Portfolio Analysis (OPA) DPCPSI OD 13

The Office of Portfolio Analysis (OPA)

OPA was established in 2011 to provide multiple services

Scientific portfolio analysishellip

o of all Common Fund initiatives

o in response to requests by senior leadership

Coordinate portfolio analysis activities across NIH

Train NIH staff to promote the effective use of analytical tools

o Regularly scheduled courses

o Ad hoc consultations

Improve portfolio analysis at NIH

Office of Portfolio Analysis (OPA) DPCPSI OD 14

Build Computational Tools and Train NIH Staff

Office of Portfolio Analysis (OPA) DPCPSI OD

OPA

TOOLS

LAB

12 laptops

Two workstations Building 1 Room B301

Smartboard

Linux server

Videoconferencing capabilities

15

OPA Consultations with NIH Staff

IC Contact Activity

NCCAM John Williamson Consult

NCI John Hewes Consult amp Collaboration

NCI Tanya Agurs-Collins Consult

NHLBI Marc Charette Consult amp Collaboration

NIAMS Faye Chen Consult

NIAMS Anita Linde Consult

NIDDK Lisa Spain Consult

NIGMS Michelle Hamlet Consult

NIGMS Ward Smith Consult

NIGMS Kelley Smith Consult amp Training

NIMH Yancy Bodenstein Consult amp Collaboration

NIMHD Ligia Artiles Consult amp Training

NLM Alan Vanbierlet Consult ampTraining

Office of Portfolio Analysis (OPA) DPCPSI OD 16

Improve Portfolio Analysis at NIH

Database management

Modeling

NIH output Tool

development

Office of Portfolio Analysis (OPA) DPCPSI OD 17

Improve Portfolio Analysis at NIH

DATABASE MANAGEMENT

Our current ability to track awards output and health impact is limited

Funding of clinical trials

o Inadequate linkage between ClinicalTrialsgov and IMPAC II data

o NCI and CIT are collaborating to address this problem

Patents and licensing of intellectual property

o No database exists for NIH-funded patents and university licensing

o OPA is addressing this in collaboration w NIH tech transfer officers

Output of awards

o Inadequate linkage between NIH awards and literaturecitation data

o OPA is developing a next-gen disambiguation tool

18Office of Portfolio Analysis (OPA) DPCPSI OD

Build Computational Tools and Train NIH Staff

Office of Portfolio Analysis (OPA) DPCPSI OD

OPA

TOOLS

LAB

12 laptops

Two workstations Building 1 Room B301

Smartboard

Linux server

Videoconferencing capabilities

15

OPA Consultations with NIH Staff

IC Contact Activity

NCCAM John Williamson Consult

NCI John Hewes Consult amp Collaboration

NCI Tanya Agurs-Collins Consult

NHLBI Marc Charette Consult amp Collaboration

NIAMS Faye Chen Consult

NIAMS Anita Linde Consult

NIDDK Lisa Spain Consult

NIGMS Michelle Hamlet Consult

NIGMS Ward Smith Consult

NIGMS Kelley Smith Consult amp Training

NIMH Yancy Bodenstein Consult amp Collaboration

NIMHD Ligia Artiles Consult amp Training

NLM Alan Vanbierlet Consult ampTraining

Office of Portfolio Analysis (OPA) DPCPSI OD 16

Improve Portfolio Analysis at NIH

Database management

Modeling

NIH output Tool

development

Office of Portfolio Analysis (OPA) DPCPSI OD 17

Improve Portfolio Analysis at NIH

DATABASE MANAGEMENT

Our current ability to track awards output and health impact is limited

Funding of clinical trials

o Inadequate linkage between ClinicalTrialsgov and IMPAC II data

o NCI and CIT are collaborating to address this problem

Patents and licensing of intellectual property

o No database exists for NIH-funded patents and university licensing

o OPA is addressing this in collaboration w NIH tech transfer officers

Output of awards

o Inadequate linkage between NIH awards and literaturecitation data

o OPA is developing a next-gen disambiguation tool

18Office of Portfolio Analysis (OPA) DPCPSI OD

OPA Consultations with NIH Staff

IC Contact Activity

NCCAM John Williamson Consult

NCI John Hewes Consult amp Collaboration

NCI Tanya Agurs-Collins Consult

NHLBI Marc Charette Consult amp Collaboration

NIAMS Faye Chen Consult

NIAMS Anita Linde Consult

NIDDK Lisa Spain Consult

NIGMS Michelle Hamlet Consult

NIGMS Ward Smith Consult

NIGMS Kelley Smith Consult amp Training

NIMH Yancy Bodenstein Consult amp Collaboration

NIMHD Ligia Artiles Consult amp Training

NLM Alan Vanbierlet Consult ampTraining

Office of Portfolio Analysis (OPA) DPCPSI OD 16

Improve Portfolio Analysis at NIH

Database management

Modeling

NIH output Tool

development

Office of Portfolio Analysis (OPA) DPCPSI OD 17

Improve Portfolio Analysis at NIH

DATABASE MANAGEMENT

Our current ability to track awards output and health impact is limited

Funding of clinical trials

o Inadequate linkage between ClinicalTrialsgov and IMPAC II data

o NCI and CIT are collaborating to address this problem

Patents and licensing of intellectual property

o No database exists for NIH-funded patents and university licensing

o OPA is addressing this in collaboration w NIH tech transfer officers

Output of awards

o Inadequate linkage between NIH awards and literaturecitation data

o OPA is developing a next-gen disambiguation tool

18Office of Portfolio Analysis (OPA) DPCPSI OD

Improve Portfolio Analysis at NIH

Database management

Modeling

NIH output Tool

development

Office of Portfolio Analysis (OPA) DPCPSI OD 17

Improve Portfolio Analysis at NIH

DATABASE MANAGEMENT

Our current ability to track awards output and health impact is limited

Funding of clinical trials

o Inadequate linkage between ClinicalTrialsgov and IMPAC II data

o NCI and CIT are collaborating to address this problem

Patents and licensing of intellectual property

o No database exists for NIH-funded patents and university licensing

o OPA is addressing this in collaboration w NIH tech transfer officers

Output of awards

o Inadequate linkage between NIH awards and literaturecitation data

o OPA is developing a next-gen disambiguation tool

18Office of Portfolio Analysis (OPA) DPCPSI OD

Improve Portfolio Analysis at NIH

DATABASE MANAGEMENT

Our current ability to track awards output and health impact is limited

Funding of clinical trials

o Inadequate linkage between ClinicalTrialsgov and IMPAC II data

o NCI and CIT are collaborating to address this problem

Patents and licensing of intellectual property

o No database exists for NIH-funded patents and university licensing

o OPA is addressing this in collaboration w NIH tech transfer officers

Output of awards

o Inadequate linkage between NIH awards and literaturecitation data

o OPA is developing a next-gen disambiguation tool

18Office of Portfolio Analysis (OPA) DPCPSI OD

Improve Portfolio Analysis at NIH

TOOL DEVELOPMENT

New tools are needed to provide NIH staff grant applicants et al

with a current and accurate picture of NIH investments

In collaboration with Calvin Johnsonrsquos group at CIT OPA has

developed several new tools

Examples include the Hierarchical Clustering amp Classifier tools

o Comparative Effectiveness Research (CER)

o NIGMS Technology Development

o Disruptive Proteomics Common Fund Analysis

o Similarity Matrix of Standing Study Sections (SRGs)

bull Characterize the assignment of applications to SRGs

bull Evaluate how SRGs are presented to PIs

Office of Portfolio Analysis (OPA) DPCPSI OD 19

Cancer Genetics Study Section vs All 165 Study Sections 2011 3 cycles

Study Section Code Match Score

Cancer Genetics CG 1000

Cancer Molecular Pathobiology

Tumor Cell Biology

Molecular Oncogenesis

Tumor Progression and Metastasis

Basic Mechanisms of Cancer Therapeutics

Developmental Therapeutics

Genetics of Health and Disease

Tumor Microenvironment

Genomics Computational Biology and Technology

Cancer Biomarkers

Molecular Genetics B

ChemoDietary Prevention

Epidemiology of Cancer

Radiation Therapeutics and Biology

Drug Discovery and Molecular Pharmacology

Molecular Neurogenetics

Cancer Etiology

Molecular Genetics C

Molecular Genetics A

CAMP

TCB

MONC

TPM

BMCT

DT

GHD

TME

GCAT

CBSS

MGB

CDP

EPIC

RTB

DMP

MNG

CE

MGC

MGA

0509

0452

0445

0405

0397

0394

0394

0393

0365

0364

0362

0360

0353

0346

0339

0331

0326

0321

0288

+

Office of Portfolio Analysis (OPA) DPCPSI OD 20

Neural Basis of Psychopathology Addictions and Sleep Disorders Study Section vs All 165 Study Sections

2011 3 cycles

Study Section Code Match Score

Neural Basis of Psychopathology Addictions and Sleep Disorders NPAS 1000

Adult Psychopathology and Disorders of Aging

Pathophysiological Basis of Mental Disorders and Addictions

Cognition and Perception

Biobehavioral Regulation Learning and Ethology

Cognitive Neuroscience

Clinical Neuroscience and Neurodegeneration

Child Psychopathology and Developmental Disabilities

Neurobiology of Motivated Behavior

Molecular Neuropharmacology and Signaling

Developmental Brain Disorders

Neurotoxicology and Alcohol

Genetics of Health and Disease

APDA

PMDA

CP

BRLE

COG

CNN

CPDD

NMB

MNPS

DBD

NAL

GHD

0528

0493

0398

0378

0368

0363

0353

0341

0339

0326

0294

0189

+

Office of Portfolio Analysis (OPA) DPCPSI OD 21

Improve Portfolio Analysis at NIH

Database management

Modeling

NIH output Tool

development

Office of Portfolio Analysis (OPA) DPCPSI OD 22

MODELING NIH OUTPUT

INPUT $$

OUTPUT pubs patents etc

HEALTH amp ECONOMIC

IMPACT

Office of Portfolio Analysis (OPA) DPCPSI OD 23

MODELING NIH OUTPUT Advanced Bibliometrics

INPUT $$

R01

OUTPUT pubs patents etc

HEALTH amp ECONOMIC

IMPACT

Office of Portfolio Analysis (OPA) DPCPSI OD 24

Citation Profile for 12 Selected Biomed Journals

Four journals from each 1200

category

High impact factor gt25

Medium IF 13-17

Low IF 1-6

All papers in those 12 journals

in a single year 2007 Tota

l of

citations

800

400

0 0 1000 2000 3000 4000 5000 6000

All citations of those papers

since 2007 Rank order of 2007 papers (least to most cited)

Office of Portfolio Analysis (OPA) DPCPSI OD 25

Averages Apply Only to Gaussian Distributions P

erc

en

tag

e

Adapted from

MEJ Newman

Contemp Physics

46322-351 (2005)

Heights of males Speeds of cars

10000000107500E+065

Po

pu

lation

(m

illio

ns)

400E+064 Size of cities Size of cities P

op

ula

tion

1000000106

Power law100000105

300E+063 in California in California

f(x) = Cx-10000104

2200E+06 1000103

100102 (with a tail) 1100E+06

1010

0 1010

0 100 200 300 400 500 1 10 100 000E+00

Rank order (smallest to largest)

26Office of Portfolio Analysis (OPA) DPCPSI OD

1

Power Law Distribution for a High IF Journal

HIGH IMPACT FACTOR JOURNAL 2007

Office of Portfolio Analysis (OPA) DPCPSI OD 27

Power Law Distribution for a Medium IF Journal

MEDIUM IMPACT FACTOR JOURNAL 2007

Office of Portfolio Analysis (OPA) DPCPSI OD 28

Power Law Distribution for a Low IF Journal

LOW IMPACT FACTOR JOURNAL 2007

Office of Portfolio Analysis (OPA) DPCPSI OD 29

Modeling NIH output Advanced Bibliometrics

INPUT $$

R01

OUTPUT pubs patents etc

For NIH R01 awardees who published in those 12 journals in 2007

Most of the projects have been resubmitted (T2 applications)

What is the relationship if any between prior output and review outcomes

Office of Portfolio Analysis (OPA) DPCPSI OD 30

Journal IF Affects R01 Fate More Than of Citations

Effect of Impact Factor

Effect of Citations

Rela

tive

pro

babili

ty o

f success (

)

+10

+

20

+

30

+

40

R01 T2 renewal applicants who published in selected 2007 journals (N = 1631)

Baseline +20 +40 +60 +80 +100

Journal Impact Factor Number of Citations

Office of Portfolio Analysis (OPA) DPCPSI OD 31

Acknowledgments

THE OPA TEAM

Geetha Senthil PhD Science Analyst

Paula Fearon PhD Science Analyst

Jean Yuan PhD Science Analyst

Carole Christian PhD Health Science Administrator

Ehsan Haque MS Data Analyst

Chuck Lynch PhD IT Specialist

Kevin Small PhD Software Developer

COLLABORATORS

Calvin Johnson PhD Chief High Performance Computing amp Informatics CIT

Katy Borner PhD Victor H Yngve Professor of Information Science

Indiana University

Office of Portfolio Analysis (OPA) DPCPSI OD 32

Cit

ati

on

s

The Citation Pattern is Established Soon After Publication

Papers from 1997 in the 12 journals analyzed HI25

20

15

10

5

0

Years after date of publication

MED

LOW

(average of four journals in each group)

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Office of Portfolio Analysis (OPA) DPCPSI OD 33

Cell Science NEJM Nature Genes Dev

J Exp Med

J Clin Inv JBC Lancet Mol Pharm

Genetics FEBS Lett

The Citation Pattern is Established Soon After Publication

45 Papers from 1997 in the 12 journals analyzed

40 1998

199935

2000

200130 2002

2003

2004

25

20 2005

2006 15

2007

200810

2009

20105 2011

0 JOURNAL 1 2 3 4 1 2 3 4 1 2 3 4

Cit

atio

ns

IMPACT FACTOR HIGH MEDIUM LOW

Office of Portfolio Analysis (OPA) DPCPSI OD 34

0

50

100

150

200

Cit

ati

on

s

200

MEDIUM IF JOURNALS

150

100

50

0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 1 2 3 4 5 6 7

Years after date of publication

8 9 10 11 12 13 14

HIGH IF JOURNALS

The Citation Pattern is Established Soon After Publication

Office of Portfolio Analysis (OPA) DPCPSI OD 35

Office of Portfolio Analysis (OPA) DPCPSI OD

Cancer Genetics Study Section vs All 165 Study Sections 2011 3 cycles

Study Section Code Match Score

Cancer Genetics CG 1000

Cancer Molecular Pathobiology

Tumor Cell Biology

Molecular Oncogenesis

Tumor Progression and Metastasis

Basic Mechanisms of Cancer Therapeutics

Developmental Therapeutics

Genetics of Health and Disease

Tumor Microenvironment

Genomics Computational Biology and Technology

Cancer Biomarkers

Molecular Genetics B

ChemoDietary Prevention

Epidemiology of Cancer

Radiation Therapeutics and Biology

Drug Discovery and Molecular Pharmacology

Molecular Neurogenetics

Cancer Etiology

Molecular Genetics C

Molecular Genetics A

CAMP

TCB

MONC

TPM

BMCT

DT

GHD

TME

GCAT

CBSS

MGB

CDP

EPIC

RTB

DMP

MNG

CE

MGC

MGA

0509

0452

0445

0405

0397

0394

0394

0393

0365

0364

0362

0360

0353

0346

0339

0331

0326

0321

0288

+

Office of Portfolio Analysis (OPA) DPCPSI OD 20

Neural Basis of Psychopathology Addictions and Sleep Disorders Study Section vs All 165 Study Sections

2011 3 cycles

Study Section Code Match Score

Neural Basis of Psychopathology Addictions and Sleep Disorders NPAS 1000

Adult Psychopathology and Disorders of Aging

Pathophysiological Basis of Mental Disorders and Addictions

Cognition and Perception

Biobehavioral Regulation Learning and Ethology

Cognitive Neuroscience

Clinical Neuroscience and Neurodegeneration

Child Psychopathology and Developmental Disabilities

Neurobiology of Motivated Behavior

Molecular Neuropharmacology and Signaling

Developmental Brain Disorders

Neurotoxicology and Alcohol

Genetics of Health and Disease

APDA

PMDA

CP

BRLE

COG

CNN

CPDD

NMB

MNPS

DBD

NAL

GHD

0528

0493

0398

0378

0368

0363

0353

0341

0339

0326

0294

0189

+

Office of Portfolio Analysis (OPA) DPCPSI OD 21

Improve Portfolio Analysis at NIH

Database management

Modeling

NIH output Tool

development

Office of Portfolio Analysis (OPA) DPCPSI OD 22

MODELING NIH OUTPUT

INPUT $$

OUTPUT pubs patents etc

HEALTH amp ECONOMIC

IMPACT

Office of Portfolio Analysis (OPA) DPCPSI OD 23

MODELING NIH OUTPUT Advanced Bibliometrics

INPUT $$

R01

OUTPUT pubs patents etc

HEALTH amp ECONOMIC

IMPACT

Office of Portfolio Analysis (OPA) DPCPSI OD 24

Citation Profile for 12 Selected Biomed Journals

Four journals from each 1200

category

High impact factor gt25

Medium IF 13-17

Low IF 1-6

All papers in those 12 journals

in a single year 2007 Tota

l of

citations

800

400

0 0 1000 2000 3000 4000 5000 6000

All citations of those papers

since 2007 Rank order of 2007 papers (least to most cited)

Office of Portfolio Analysis (OPA) DPCPSI OD 25

Averages Apply Only to Gaussian Distributions P

erc

en

tag

e

Adapted from

MEJ Newman

Contemp Physics

46322-351 (2005)

Heights of males Speeds of cars

10000000107500E+065

Po

pu

lation

(m

illio

ns)

400E+064 Size of cities Size of cities P

op

ula

tion

1000000106

Power law100000105

300E+063 in California in California

f(x) = Cx-10000104

2200E+06 1000103

100102 (with a tail) 1100E+06

1010

0 1010

0 100 200 300 400 500 1 10 100 000E+00

Rank order (smallest to largest)

26Office of Portfolio Analysis (OPA) DPCPSI OD

1

Power Law Distribution for a High IF Journal

HIGH IMPACT FACTOR JOURNAL 2007

Office of Portfolio Analysis (OPA) DPCPSI OD 27

Power Law Distribution for a Medium IF Journal

MEDIUM IMPACT FACTOR JOURNAL 2007

Office of Portfolio Analysis (OPA) DPCPSI OD 28

Power Law Distribution for a Low IF Journal

LOW IMPACT FACTOR JOURNAL 2007

Office of Portfolio Analysis (OPA) DPCPSI OD 29

Modeling NIH output Advanced Bibliometrics

INPUT $$

R01

OUTPUT pubs patents etc

For NIH R01 awardees who published in those 12 journals in 2007

Most of the projects have been resubmitted (T2 applications)

What is the relationship if any between prior output and review outcomes

Office of Portfolio Analysis (OPA) DPCPSI OD 30

Journal IF Affects R01 Fate More Than of Citations

Effect of Impact Factor

Effect of Citations

Rela

tive

pro

babili

ty o

f success (

)

+10

+

20

+

30

+

40

R01 T2 renewal applicants who published in selected 2007 journals (N = 1631)

Baseline +20 +40 +60 +80 +100

Journal Impact Factor Number of Citations

Office of Portfolio Analysis (OPA) DPCPSI OD 31

Acknowledgments

THE OPA TEAM

Geetha Senthil PhD Science Analyst

Paula Fearon PhD Science Analyst

Jean Yuan PhD Science Analyst

Carole Christian PhD Health Science Administrator

Ehsan Haque MS Data Analyst

Chuck Lynch PhD IT Specialist

Kevin Small PhD Software Developer

COLLABORATORS

Calvin Johnson PhD Chief High Performance Computing amp Informatics CIT

Katy Borner PhD Victor H Yngve Professor of Information Science

Indiana University

Office of Portfolio Analysis (OPA) DPCPSI OD 32

Cit

ati

on

s

The Citation Pattern is Established Soon After Publication

Papers from 1997 in the 12 journals analyzed HI25

20

15

10

5

0

Years after date of publication

MED

LOW

(average of four journals in each group)

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Office of Portfolio Analysis (OPA) DPCPSI OD 33

Cell Science NEJM Nature Genes Dev

J Exp Med

J Clin Inv JBC Lancet Mol Pharm

Genetics FEBS Lett

The Citation Pattern is Established Soon After Publication

45 Papers from 1997 in the 12 journals analyzed

40 1998

199935

2000

200130 2002

2003

2004

25

20 2005

2006 15

2007

200810

2009

20105 2011

0 JOURNAL 1 2 3 4 1 2 3 4 1 2 3 4

Cit

atio

ns

IMPACT FACTOR HIGH MEDIUM LOW

Office of Portfolio Analysis (OPA) DPCPSI OD 34

0

50

100

150

200

Cit

ati

on

s

200

MEDIUM IF JOURNALS

150

100

50

0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 1 2 3 4 5 6 7

Years after date of publication

8 9 10 11 12 13 14

HIGH IF JOURNALS

The Citation Pattern is Established Soon After Publication

Office of Portfolio Analysis (OPA) DPCPSI OD 35

Office of Portfolio Analysis (OPA) DPCPSI OD

Neural Basis of Psychopathology Addictions and Sleep Disorders Study Section vs All 165 Study Sections

2011 3 cycles

Study Section Code Match Score

Neural Basis of Psychopathology Addictions and Sleep Disorders NPAS 1000

Adult Psychopathology and Disorders of Aging

Pathophysiological Basis of Mental Disorders and Addictions

Cognition and Perception

Biobehavioral Regulation Learning and Ethology

Cognitive Neuroscience

Clinical Neuroscience and Neurodegeneration

Child Psychopathology and Developmental Disabilities

Neurobiology of Motivated Behavior

Molecular Neuropharmacology and Signaling

Developmental Brain Disorders

Neurotoxicology and Alcohol

Genetics of Health and Disease

APDA

PMDA

CP

BRLE

COG

CNN

CPDD

NMB

MNPS

DBD

NAL

GHD

0528

0493

0398

0378

0368

0363

0353

0341

0339

0326

0294

0189

+

Office of Portfolio Analysis (OPA) DPCPSI OD 21

Improve Portfolio Analysis at NIH

Database management

Modeling

NIH output Tool

development

Office of Portfolio Analysis (OPA) DPCPSI OD 22

MODELING NIH OUTPUT

INPUT $$

OUTPUT pubs patents etc

HEALTH amp ECONOMIC

IMPACT

Office of Portfolio Analysis (OPA) DPCPSI OD 23

MODELING NIH OUTPUT Advanced Bibliometrics

INPUT $$

R01

OUTPUT pubs patents etc

HEALTH amp ECONOMIC

IMPACT

Office of Portfolio Analysis (OPA) DPCPSI OD 24

Citation Profile for 12 Selected Biomed Journals

Four journals from each 1200

category

High impact factor gt25

Medium IF 13-17

Low IF 1-6

All papers in those 12 journals

in a single year 2007 Tota

l of

citations

800

400

0 0 1000 2000 3000 4000 5000 6000

All citations of those papers

since 2007 Rank order of 2007 papers (least to most cited)

Office of Portfolio Analysis (OPA) DPCPSI OD 25

Averages Apply Only to Gaussian Distributions P

erc

en

tag

e

Adapted from

MEJ Newman

Contemp Physics

46322-351 (2005)

Heights of males Speeds of cars

10000000107500E+065

Po

pu

lation

(m

illio

ns)

400E+064 Size of cities Size of cities P

op

ula

tion

1000000106

Power law100000105

300E+063 in California in California

f(x) = Cx-10000104

2200E+06 1000103

100102 (with a tail) 1100E+06

1010

0 1010

0 100 200 300 400 500 1 10 100 000E+00

Rank order (smallest to largest)

26Office of Portfolio Analysis (OPA) DPCPSI OD

1

Power Law Distribution for a High IF Journal

HIGH IMPACT FACTOR JOURNAL 2007

Office of Portfolio Analysis (OPA) DPCPSI OD 27

Power Law Distribution for a Medium IF Journal

MEDIUM IMPACT FACTOR JOURNAL 2007

Office of Portfolio Analysis (OPA) DPCPSI OD 28

Power Law Distribution for a Low IF Journal

LOW IMPACT FACTOR JOURNAL 2007

Office of Portfolio Analysis (OPA) DPCPSI OD 29

Modeling NIH output Advanced Bibliometrics

INPUT $$

R01

OUTPUT pubs patents etc

For NIH R01 awardees who published in those 12 journals in 2007

Most of the projects have been resubmitted (T2 applications)

What is the relationship if any between prior output and review outcomes

Office of Portfolio Analysis (OPA) DPCPSI OD 30

Journal IF Affects R01 Fate More Than of Citations

Effect of Impact Factor

Effect of Citations

Rela

tive

pro

babili

ty o

f success (

)

+10

+

20

+

30

+

40

R01 T2 renewal applicants who published in selected 2007 journals (N = 1631)

Baseline +20 +40 +60 +80 +100

Journal Impact Factor Number of Citations

Office of Portfolio Analysis (OPA) DPCPSI OD 31

Acknowledgments

THE OPA TEAM

Geetha Senthil PhD Science Analyst

Paula Fearon PhD Science Analyst

Jean Yuan PhD Science Analyst

Carole Christian PhD Health Science Administrator

Ehsan Haque MS Data Analyst

Chuck Lynch PhD IT Specialist

Kevin Small PhD Software Developer

COLLABORATORS

Calvin Johnson PhD Chief High Performance Computing amp Informatics CIT

Katy Borner PhD Victor H Yngve Professor of Information Science

Indiana University

Office of Portfolio Analysis (OPA) DPCPSI OD 32

Cit

ati

on

s

The Citation Pattern is Established Soon After Publication

Papers from 1997 in the 12 journals analyzed HI25

20

15

10

5

0

Years after date of publication

MED

LOW

(average of four journals in each group)

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Office of Portfolio Analysis (OPA) DPCPSI OD 33

Cell Science NEJM Nature Genes Dev

J Exp Med

J Clin Inv JBC Lancet Mol Pharm

Genetics FEBS Lett

The Citation Pattern is Established Soon After Publication

45 Papers from 1997 in the 12 journals analyzed

40 1998

199935

2000

200130 2002

2003

2004

25

20 2005

2006 15

2007

200810

2009

20105 2011

0 JOURNAL 1 2 3 4 1 2 3 4 1 2 3 4

Cit

atio

ns

IMPACT FACTOR HIGH MEDIUM LOW

Office of Portfolio Analysis (OPA) DPCPSI OD 34

0

50

100

150

200

Cit

ati

on

s

200

MEDIUM IF JOURNALS

150

100

50

0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 1 2 3 4 5 6 7

Years after date of publication

8 9 10 11 12 13 14

HIGH IF JOURNALS

The Citation Pattern is Established Soon After Publication

Office of Portfolio Analysis (OPA) DPCPSI OD 35

Office of Portfolio Analysis (OPA) DPCPSI OD

Improve Portfolio Analysis at NIH

Database management

Modeling

NIH output Tool

development

Office of Portfolio Analysis (OPA) DPCPSI OD 22

MODELING NIH OUTPUT

INPUT $$

OUTPUT pubs patents etc

HEALTH amp ECONOMIC

IMPACT

Office of Portfolio Analysis (OPA) DPCPSI OD 23

MODELING NIH OUTPUT Advanced Bibliometrics

INPUT $$

R01

OUTPUT pubs patents etc

HEALTH amp ECONOMIC

IMPACT

Office of Portfolio Analysis (OPA) DPCPSI OD 24

Citation Profile for 12 Selected Biomed Journals

Four journals from each 1200

category

High impact factor gt25

Medium IF 13-17

Low IF 1-6

All papers in those 12 journals

in a single year 2007 Tota

l of

citations

800

400

0 0 1000 2000 3000 4000 5000 6000

All citations of those papers

since 2007 Rank order of 2007 papers (least to most cited)

Office of Portfolio Analysis (OPA) DPCPSI OD 25

Averages Apply Only to Gaussian Distributions P

erc

en

tag

e

Adapted from

MEJ Newman

Contemp Physics

46322-351 (2005)

Heights of males Speeds of cars

10000000107500E+065

Po

pu

lation

(m

illio

ns)

400E+064 Size of cities Size of cities P

op

ula

tion

1000000106

Power law100000105

300E+063 in California in California

f(x) = Cx-10000104

2200E+06 1000103

100102 (with a tail) 1100E+06

1010

0 1010

0 100 200 300 400 500 1 10 100 000E+00

Rank order (smallest to largest)

26Office of Portfolio Analysis (OPA) DPCPSI OD

1

Power Law Distribution for a High IF Journal

HIGH IMPACT FACTOR JOURNAL 2007

Office of Portfolio Analysis (OPA) DPCPSI OD 27

Power Law Distribution for a Medium IF Journal

MEDIUM IMPACT FACTOR JOURNAL 2007

Office of Portfolio Analysis (OPA) DPCPSI OD 28

Power Law Distribution for a Low IF Journal

LOW IMPACT FACTOR JOURNAL 2007

Office of Portfolio Analysis (OPA) DPCPSI OD 29

Modeling NIH output Advanced Bibliometrics

INPUT $$

R01

OUTPUT pubs patents etc

For NIH R01 awardees who published in those 12 journals in 2007

Most of the projects have been resubmitted (T2 applications)

What is the relationship if any between prior output and review outcomes

Office of Portfolio Analysis (OPA) DPCPSI OD 30

Journal IF Affects R01 Fate More Than of Citations

Effect of Impact Factor

Effect of Citations

Rela

tive

pro

babili

ty o

f success (

)

+10

+

20

+

30

+

40

R01 T2 renewal applicants who published in selected 2007 journals (N = 1631)

Baseline +20 +40 +60 +80 +100

Journal Impact Factor Number of Citations

Office of Portfolio Analysis (OPA) DPCPSI OD 31

Acknowledgments

THE OPA TEAM

Geetha Senthil PhD Science Analyst

Paula Fearon PhD Science Analyst

Jean Yuan PhD Science Analyst

Carole Christian PhD Health Science Administrator

Ehsan Haque MS Data Analyst

Chuck Lynch PhD IT Specialist

Kevin Small PhD Software Developer

COLLABORATORS

Calvin Johnson PhD Chief High Performance Computing amp Informatics CIT

Katy Borner PhD Victor H Yngve Professor of Information Science

Indiana University

Office of Portfolio Analysis (OPA) DPCPSI OD 32

Cit

ati

on

s

The Citation Pattern is Established Soon After Publication

Papers from 1997 in the 12 journals analyzed HI25

20

15

10

5

0

Years after date of publication

MED

LOW

(average of four journals in each group)

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Office of Portfolio Analysis (OPA) DPCPSI OD 33

Cell Science NEJM Nature Genes Dev

J Exp Med

J Clin Inv JBC Lancet Mol Pharm

Genetics FEBS Lett

The Citation Pattern is Established Soon After Publication

45 Papers from 1997 in the 12 journals analyzed

40 1998

199935

2000

200130 2002

2003

2004

25

20 2005

2006 15

2007

200810

2009

20105 2011

0 JOURNAL 1 2 3 4 1 2 3 4 1 2 3 4

Cit

atio

ns

IMPACT FACTOR HIGH MEDIUM LOW

Office of Portfolio Analysis (OPA) DPCPSI OD 34

0

50

100

150

200

Cit

ati

on

s

200

MEDIUM IF JOURNALS

150

100

50

0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 1 2 3 4 5 6 7

Years after date of publication

8 9 10 11 12 13 14

HIGH IF JOURNALS

The Citation Pattern is Established Soon After Publication

Office of Portfolio Analysis (OPA) DPCPSI OD 35

Office of Portfolio Analysis (OPA) DPCPSI OD

MODELING NIH OUTPUT

INPUT $$

OUTPUT pubs patents etc

HEALTH amp ECONOMIC

IMPACT

Office of Portfolio Analysis (OPA) DPCPSI OD 23

MODELING NIH OUTPUT Advanced Bibliometrics

INPUT $$

R01

OUTPUT pubs patents etc

HEALTH amp ECONOMIC

IMPACT

Office of Portfolio Analysis (OPA) DPCPSI OD 24

Citation Profile for 12 Selected Biomed Journals

Four journals from each 1200

category

High impact factor gt25

Medium IF 13-17

Low IF 1-6

All papers in those 12 journals

in a single year 2007 Tota

l of

citations

800

400

0 0 1000 2000 3000 4000 5000 6000

All citations of those papers

since 2007 Rank order of 2007 papers (least to most cited)

Office of Portfolio Analysis (OPA) DPCPSI OD 25

Averages Apply Only to Gaussian Distributions P

erc

en

tag

e

Adapted from

MEJ Newman

Contemp Physics

46322-351 (2005)

Heights of males Speeds of cars

10000000107500E+065

Po

pu

lation

(m

illio

ns)

400E+064 Size of cities Size of cities P

op

ula

tion

1000000106

Power law100000105

300E+063 in California in California

f(x) = Cx-10000104

2200E+06 1000103

100102 (with a tail) 1100E+06

1010

0 1010

0 100 200 300 400 500 1 10 100 000E+00

Rank order (smallest to largest)

26Office of Portfolio Analysis (OPA) DPCPSI OD

1

Power Law Distribution for a High IF Journal

HIGH IMPACT FACTOR JOURNAL 2007

Office of Portfolio Analysis (OPA) DPCPSI OD 27

Power Law Distribution for a Medium IF Journal

MEDIUM IMPACT FACTOR JOURNAL 2007

Office of Portfolio Analysis (OPA) DPCPSI OD 28

Power Law Distribution for a Low IF Journal

LOW IMPACT FACTOR JOURNAL 2007

Office of Portfolio Analysis (OPA) DPCPSI OD 29

Modeling NIH output Advanced Bibliometrics

INPUT $$

R01

OUTPUT pubs patents etc

For NIH R01 awardees who published in those 12 journals in 2007

Most of the projects have been resubmitted (T2 applications)

What is the relationship if any between prior output and review outcomes

Office of Portfolio Analysis (OPA) DPCPSI OD 30

Journal IF Affects R01 Fate More Than of Citations

Effect of Impact Factor

Effect of Citations

Rela

tive

pro

babili

ty o

f success (

)

+10

+

20

+

30

+

40

R01 T2 renewal applicants who published in selected 2007 journals (N = 1631)

Baseline +20 +40 +60 +80 +100

Journal Impact Factor Number of Citations

Office of Portfolio Analysis (OPA) DPCPSI OD 31

Acknowledgments

THE OPA TEAM

Geetha Senthil PhD Science Analyst

Paula Fearon PhD Science Analyst

Jean Yuan PhD Science Analyst

Carole Christian PhD Health Science Administrator

Ehsan Haque MS Data Analyst

Chuck Lynch PhD IT Specialist

Kevin Small PhD Software Developer

COLLABORATORS

Calvin Johnson PhD Chief High Performance Computing amp Informatics CIT

Katy Borner PhD Victor H Yngve Professor of Information Science

Indiana University

Office of Portfolio Analysis (OPA) DPCPSI OD 32

Cit

ati

on

s

The Citation Pattern is Established Soon After Publication

Papers from 1997 in the 12 journals analyzed HI25

20

15

10

5

0

Years after date of publication

MED

LOW

(average of four journals in each group)

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Office of Portfolio Analysis (OPA) DPCPSI OD 33

Cell Science NEJM Nature Genes Dev

J Exp Med

J Clin Inv JBC Lancet Mol Pharm

Genetics FEBS Lett

The Citation Pattern is Established Soon After Publication

45 Papers from 1997 in the 12 journals analyzed

40 1998

199935

2000

200130 2002

2003

2004

25

20 2005

2006 15

2007

200810

2009

20105 2011

0 JOURNAL 1 2 3 4 1 2 3 4 1 2 3 4

Cit

atio

ns

IMPACT FACTOR HIGH MEDIUM LOW

Office of Portfolio Analysis (OPA) DPCPSI OD 34

0

50

100

150

200

Cit

ati

on

s

200

MEDIUM IF JOURNALS

150

100

50

0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 1 2 3 4 5 6 7

Years after date of publication

8 9 10 11 12 13 14

HIGH IF JOURNALS

The Citation Pattern is Established Soon After Publication

Office of Portfolio Analysis (OPA) DPCPSI OD 35

Office of Portfolio Analysis (OPA) DPCPSI OD

MODELING NIH OUTPUT Advanced Bibliometrics

INPUT $$

R01

OUTPUT pubs patents etc

HEALTH amp ECONOMIC

IMPACT

Office of Portfolio Analysis (OPA) DPCPSI OD 24

Citation Profile for 12 Selected Biomed Journals

Four journals from each 1200

category

High impact factor gt25

Medium IF 13-17

Low IF 1-6

All papers in those 12 journals

in a single year 2007 Tota

l of

citations

800

400

0 0 1000 2000 3000 4000 5000 6000

All citations of those papers

since 2007 Rank order of 2007 papers (least to most cited)

Office of Portfolio Analysis (OPA) DPCPSI OD 25

Averages Apply Only to Gaussian Distributions P

erc

en

tag

e

Adapted from

MEJ Newman

Contemp Physics

46322-351 (2005)

Heights of males Speeds of cars

10000000107500E+065

Po

pu

lation

(m

illio

ns)

400E+064 Size of cities Size of cities P

op

ula

tion

1000000106

Power law100000105

300E+063 in California in California

f(x) = Cx-10000104

2200E+06 1000103

100102 (with a tail) 1100E+06

1010

0 1010

0 100 200 300 400 500 1 10 100 000E+00

Rank order (smallest to largest)

26Office of Portfolio Analysis (OPA) DPCPSI OD

1

Power Law Distribution for a High IF Journal

HIGH IMPACT FACTOR JOURNAL 2007

Office of Portfolio Analysis (OPA) DPCPSI OD 27

Power Law Distribution for a Medium IF Journal

MEDIUM IMPACT FACTOR JOURNAL 2007

Office of Portfolio Analysis (OPA) DPCPSI OD 28

Power Law Distribution for a Low IF Journal

LOW IMPACT FACTOR JOURNAL 2007

Office of Portfolio Analysis (OPA) DPCPSI OD 29

Modeling NIH output Advanced Bibliometrics

INPUT $$

R01

OUTPUT pubs patents etc

For NIH R01 awardees who published in those 12 journals in 2007

Most of the projects have been resubmitted (T2 applications)

What is the relationship if any between prior output and review outcomes

Office of Portfolio Analysis (OPA) DPCPSI OD 30

Journal IF Affects R01 Fate More Than of Citations

Effect of Impact Factor

Effect of Citations

Rela

tive

pro

babili

ty o

f success (

)

+10

+

20

+

30

+

40

R01 T2 renewal applicants who published in selected 2007 journals (N = 1631)

Baseline +20 +40 +60 +80 +100

Journal Impact Factor Number of Citations

Office of Portfolio Analysis (OPA) DPCPSI OD 31

Acknowledgments

THE OPA TEAM

Geetha Senthil PhD Science Analyst

Paula Fearon PhD Science Analyst

Jean Yuan PhD Science Analyst

Carole Christian PhD Health Science Administrator

Ehsan Haque MS Data Analyst

Chuck Lynch PhD IT Specialist

Kevin Small PhD Software Developer

COLLABORATORS

Calvin Johnson PhD Chief High Performance Computing amp Informatics CIT

Katy Borner PhD Victor H Yngve Professor of Information Science

Indiana University

Office of Portfolio Analysis (OPA) DPCPSI OD 32

Cit

ati

on

s

The Citation Pattern is Established Soon After Publication

Papers from 1997 in the 12 journals analyzed HI25

20

15

10

5

0

Years after date of publication

MED

LOW

(average of four journals in each group)

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Office of Portfolio Analysis (OPA) DPCPSI OD 33

Cell Science NEJM Nature Genes Dev

J Exp Med

J Clin Inv JBC Lancet Mol Pharm

Genetics FEBS Lett

The Citation Pattern is Established Soon After Publication

45 Papers from 1997 in the 12 journals analyzed

40 1998

199935

2000

200130 2002

2003

2004

25

20 2005

2006 15

2007

200810

2009

20105 2011

0 JOURNAL 1 2 3 4 1 2 3 4 1 2 3 4

Cit

atio

ns

IMPACT FACTOR HIGH MEDIUM LOW

Office of Portfolio Analysis (OPA) DPCPSI OD 34

0

50

100

150

200

Cit

ati

on

s

200

MEDIUM IF JOURNALS

150

100

50

0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 1 2 3 4 5 6 7

Years after date of publication

8 9 10 11 12 13 14

HIGH IF JOURNALS

The Citation Pattern is Established Soon After Publication

Office of Portfolio Analysis (OPA) DPCPSI OD 35

Office of Portfolio Analysis (OPA) DPCPSI OD

Citation Profile for 12 Selected Biomed Journals

Four journals from each 1200

category

High impact factor gt25

Medium IF 13-17

Low IF 1-6

All papers in those 12 journals

in a single year 2007 Tota

l of

citations

800

400

0 0 1000 2000 3000 4000 5000 6000

All citations of those papers

since 2007 Rank order of 2007 papers (least to most cited)

Office of Portfolio Analysis (OPA) DPCPSI OD 25

Averages Apply Only to Gaussian Distributions P

erc

en

tag

e

Adapted from

MEJ Newman

Contemp Physics

46322-351 (2005)

Heights of males Speeds of cars

10000000107500E+065

Po

pu

lation

(m

illio

ns)

400E+064 Size of cities Size of cities P

op

ula

tion

1000000106

Power law100000105

300E+063 in California in California

f(x) = Cx-10000104

2200E+06 1000103

100102 (with a tail) 1100E+06

1010

0 1010

0 100 200 300 400 500 1 10 100 000E+00

Rank order (smallest to largest)

26Office of Portfolio Analysis (OPA) DPCPSI OD

1

Power Law Distribution for a High IF Journal

HIGH IMPACT FACTOR JOURNAL 2007

Office of Portfolio Analysis (OPA) DPCPSI OD 27

Power Law Distribution for a Medium IF Journal

MEDIUM IMPACT FACTOR JOURNAL 2007

Office of Portfolio Analysis (OPA) DPCPSI OD 28

Power Law Distribution for a Low IF Journal

LOW IMPACT FACTOR JOURNAL 2007

Office of Portfolio Analysis (OPA) DPCPSI OD 29

Modeling NIH output Advanced Bibliometrics

INPUT $$

R01

OUTPUT pubs patents etc

For NIH R01 awardees who published in those 12 journals in 2007

Most of the projects have been resubmitted (T2 applications)

What is the relationship if any between prior output and review outcomes

Office of Portfolio Analysis (OPA) DPCPSI OD 30

Journal IF Affects R01 Fate More Than of Citations

Effect of Impact Factor

Effect of Citations

Rela

tive

pro

babili

ty o

f success (

)

+10

+

20

+

30

+

40

R01 T2 renewal applicants who published in selected 2007 journals (N = 1631)

Baseline +20 +40 +60 +80 +100

Journal Impact Factor Number of Citations

Office of Portfolio Analysis (OPA) DPCPSI OD 31

Acknowledgments

THE OPA TEAM

Geetha Senthil PhD Science Analyst

Paula Fearon PhD Science Analyst

Jean Yuan PhD Science Analyst

Carole Christian PhD Health Science Administrator

Ehsan Haque MS Data Analyst

Chuck Lynch PhD IT Specialist

Kevin Small PhD Software Developer

COLLABORATORS

Calvin Johnson PhD Chief High Performance Computing amp Informatics CIT

Katy Borner PhD Victor H Yngve Professor of Information Science

Indiana University

Office of Portfolio Analysis (OPA) DPCPSI OD 32

Cit

ati

on

s

The Citation Pattern is Established Soon After Publication

Papers from 1997 in the 12 journals analyzed HI25

20

15

10

5

0

Years after date of publication

MED

LOW

(average of four journals in each group)

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Office of Portfolio Analysis (OPA) DPCPSI OD 33

Cell Science NEJM Nature Genes Dev

J Exp Med

J Clin Inv JBC Lancet Mol Pharm

Genetics FEBS Lett

The Citation Pattern is Established Soon After Publication

45 Papers from 1997 in the 12 journals analyzed

40 1998

199935

2000

200130 2002

2003

2004

25

20 2005

2006 15

2007

200810

2009

20105 2011

0 JOURNAL 1 2 3 4 1 2 3 4 1 2 3 4

Cit

atio

ns

IMPACT FACTOR HIGH MEDIUM LOW

Office of Portfolio Analysis (OPA) DPCPSI OD 34

0

50

100

150

200

Cit

ati

on

s

200

MEDIUM IF JOURNALS

150

100

50

0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 1 2 3 4 5 6 7

Years after date of publication

8 9 10 11 12 13 14

HIGH IF JOURNALS

The Citation Pattern is Established Soon After Publication

Office of Portfolio Analysis (OPA) DPCPSI OD 35

Office of Portfolio Analysis (OPA) DPCPSI OD

Averages Apply Only to Gaussian Distributions P

erc

en

tag

e

Adapted from

MEJ Newman

Contemp Physics

46322-351 (2005)

Heights of males Speeds of cars

10000000107500E+065

Po

pu

lation

(m

illio

ns)

400E+064 Size of cities Size of cities P

op

ula

tion

1000000106

Power law100000105

300E+063 in California in California

f(x) = Cx-10000104

2200E+06 1000103

100102 (with a tail) 1100E+06

1010

0 1010

0 100 200 300 400 500 1 10 100 000E+00

Rank order (smallest to largest)

26Office of Portfolio Analysis (OPA) DPCPSI OD

1

Power Law Distribution for a High IF Journal

HIGH IMPACT FACTOR JOURNAL 2007

Office of Portfolio Analysis (OPA) DPCPSI OD 27

Power Law Distribution for a Medium IF Journal

MEDIUM IMPACT FACTOR JOURNAL 2007

Office of Portfolio Analysis (OPA) DPCPSI OD 28

Power Law Distribution for a Low IF Journal

LOW IMPACT FACTOR JOURNAL 2007

Office of Portfolio Analysis (OPA) DPCPSI OD 29

Modeling NIH output Advanced Bibliometrics

INPUT $$

R01

OUTPUT pubs patents etc

For NIH R01 awardees who published in those 12 journals in 2007

Most of the projects have been resubmitted (T2 applications)

What is the relationship if any between prior output and review outcomes

Office of Portfolio Analysis (OPA) DPCPSI OD 30

Journal IF Affects R01 Fate More Than of Citations

Effect of Impact Factor

Effect of Citations

Rela

tive

pro

babili

ty o

f success (

)

+10

+

20

+

30

+

40

R01 T2 renewal applicants who published in selected 2007 journals (N = 1631)

Baseline +20 +40 +60 +80 +100

Journal Impact Factor Number of Citations

Office of Portfolio Analysis (OPA) DPCPSI OD 31

Acknowledgments

THE OPA TEAM

Geetha Senthil PhD Science Analyst

Paula Fearon PhD Science Analyst

Jean Yuan PhD Science Analyst

Carole Christian PhD Health Science Administrator

Ehsan Haque MS Data Analyst

Chuck Lynch PhD IT Specialist

Kevin Small PhD Software Developer

COLLABORATORS

Calvin Johnson PhD Chief High Performance Computing amp Informatics CIT

Katy Borner PhD Victor H Yngve Professor of Information Science

Indiana University

Office of Portfolio Analysis (OPA) DPCPSI OD 32

Cit

ati

on

s

The Citation Pattern is Established Soon After Publication

Papers from 1997 in the 12 journals analyzed HI25

20

15

10

5

0

Years after date of publication

MED

LOW

(average of four journals in each group)

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Office of Portfolio Analysis (OPA) DPCPSI OD 33

Cell Science NEJM Nature Genes Dev

J Exp Med

J Clin Inv JBC Lancet Mol Pharm

Genetics FEBS Lett

The Citation Pattern is Established Soon After Publication

45 Papers from 1997 in the 12 journals analyzed

40 1998

199935

2000

200130 2002

2003

2004

25

20 2005

2006 15

2007

200810

2009

20105 2011

0 JOURNAL 1 2 3 4 1 2 3 4 1 2 3 4

Cit

atio

ns

IMPACT FACTOR HIGH MEDIUM LOW

Office of Portfolio Analysis (OPA) DPCPSI OD 34

0

50

100

150

200

Cit

ati

on

s

200

MEDIUM IF JOURNALS

150

100

50

0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 1 2 3 4 5 6 7

Years after date of publication

8 9 10 11 12 13 14

HIGH IF JOURNALS

The Citation Pattern is Established Soon After Publication

Office of Portfolio Analysis (OPA) DPCPSI OD 35

Office of Portfolio Analysis (OPA) DPCPSI OD

Power Law Distribution for a High IF Journal

HIGH IMPACT FACTOR JOURNAL 2007

Office of Portfolio Analysis (OPA) DPCPSI OD 27

Power Law Distribution for a Medium IF Journal

MEDIUM IMPACT FACTOR JOURNAL 2007

Office of Portfolio Analysis (OPA) DPCPSI OD 28

Power Law Distribution for a Low IF Journal

LOW IMPACT FACTOR JOURNAL 2007

Office of Portfolio Analysis (OPA) DPCPSI OD 29

Modeling NIH output Advanced Bibliometrics

INPUT $$

R01

OUTPUT pubs patents etc

For NIH R01 awardees who published in those 12 journals in 2007

Most of the projects have been resubmitted (T2 applications)

What is the relationship if any between prior output and review outcomes

Office of Portfolio Analysis (OPA) DPCPSI OD 30

Journal IF Affects R01 Fate More Than of Citations

Effect of Impact Factor

Effect of Citations

Rela

tive

pro

babili

ty o

f success (

)

+10

+

20

+

30

+

40

R01 T2 renewal applicants who published in selected 2007 journals (N = 1631)

Baseline +20 +40 +60 +80 +100

Journal Impact Factor Number of Citations

Office of Portfolio Analysis (OPA) DPCPSI OD 31

Acknowledgments

THE OPA TEAM

Geetha Senthil PhD Science Analyst

Paula Fearon PhD Science Analyst

Jean Yuan PhD Science Analyst

Carole Christian PhD Health Science Administrator

Ehsan Haque MS Data Analyst

Chuck Lynch PhD IT Specialist

Kevin Small PhD Software Developer

COLLABORATORS

Calvin Johnson PhD Chief High Performance Computing amp Informatics CIT

Katy Borner PhD Victor H Yngve Professor of Information Science

Indiana University

Office of Portfolio Analysis (OPA) DPCPSI OD 32

Cit

ati

on

s

The Citation Pattern is Established Soon After Publication

Papers from 1997 in the 12 journals analyzed HI25

20

15

10

5

0

Years after date of publication

MED

LOW

(average of four journals in each group)

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Office of Portfolio Analysis (OPA) DPCPSI OD 33

Cell Science NEJM Nature Genes Dev

J Exp Med

J Clin Inv JBC Lancet Mol Pharm

Genetics FEBS Lett

The Citation Pattern is Established Soon After Publication

45 Papers from 1997 in the 12 journals analyzed

40 1998

199935

2000

200130 2002

2003

2004

25

20 2005

2006 15

2007

200810

2009

20105 2011

0 JOURNAL 1 2 3 4 1 2 3 4 1 2 3 4

Cit

atio

ns

IMPACT FACTOR HIGH MEDIUM LOW

Office of Portfolio Analysis (OPA) DPCPSI OD 34

0

50

100

150

200

Cit

ati

on

s

200

MEDIUM IF JOURNALS

150

100

50

0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 1 2 3 4 5 6 7

Years after date of publication

8 9 10 11 12 13 14

HIGH IF JOURNALS

The Citation Pattern is Established Soon After Publication

Office of Portfolio Analysis (OPA) DPCPSI OD 35

Office of Portfolio Analysis (OPA) DPCPSI OD

Power Law Distribution for a Medium IF Journal

MEDIUM IMPACT FACTOR JOURNAL 2007

Office of Portfolio Analysis (OPA) DPCPSI OD 28

Power Law Distribution for a Low IF Journal

LOW IMPACT FACTOR JOURNAL 2007

Office of Portfolio Analysis (OPA) DPCPSI OD 29

Modeling NIH output Advanced Bibliometrics

INPUT $$

R01

OUTPUT pubs patents etc

For NIH R01 awardees who published in those 12 journals in 2007

Most of the projects have been resubmitted (T2 applications)

What is the relationship if any between prior output and review outcomes

Office of Portfolio Analysis (OPA) DPCPSI OD 30

Journal IF Affects R01 Fate More Than of Citations

Effect of Impact Factor

Effect of Citations

Rela

tive

pro

babili

ty o

f success (

)

+10

+

20

+

30

+

40

R01 T2 renewal applicants who published in selected 2007 journals (N = 1631)

Baseline +20 +40 +60 +80 +100

Journal Impact Factor Number of Citations

Office of Portfolio Analysis (OPA) DPCPSI OD 31

Acknowledgments

THE OPA TEAM

Geetha Senthil PhD Science Analyst

Paula Fearon PhD Science Analyst

Jean Yuan PhD Science Analyst

Carole Christian PhD Health Science Administrator

Ehsan Haque MS Data Analyst

Chuck Lynch PhD IT Specialist

Kevin Small PhD Software Developer

COLLABORATORS

Calvin Johnson PhD Chief High Performance Computing amp Informatics CIT

Katy Borner PhD Victor H Yngve Professor of Information Science

Indiana University

Office of Portfolio Analysis (OPA) DPCPSI OD 32

Cit

ati

on

s

The Citation Pattern is Established Soon After Publication

Papers from 1997 in the 12 journals analyzed HI25

20

15

10

5

0

Years after date of publication

MED

LOW

(average of four journals in each group)

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Office of Portfolio Analysis (OPA) DPCPSI OD 33

Cell Science NEJM Nature Genes Dev

J Exp Med

J Clin Inv JBC Lancet Mol Pharm

Genetics FEBS Lett

The Citation Pattern is Established Soon After Publication

45 Papers from 1997 in the 12 journals analyzed

40 1998

199935

2000

200130 2002

2003

2004

25

20 2005

2006 15

2007

200810

2009

20105 2011

0 JOURNAL 1 2 3 4 1 2 3 4 1 2 3 4

Cit

atio

ns

IMPACT FACTOR HIGH MEDIUM LOW

Office of Portfolio Analysis (OPA) DPCPSI OD 34

0

50

100

150

200

Cit

ati

on

s

200

MEDIUM IF JOURNALS

150

100

50

0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 1 2 3 4 5 6 7

Years after date of publication

8 9 10 11 12 13 14

HIGH IF JOURNALS

The Citation Pattern is Established Soon After Publication

Office of Portfolio Analysis (OPA) DPCPSI OD 35

Office of Portfolio Analysis (OPA) DPCPSI OD

Power Law Distribution for a Low IF Journal

LOW IMPACT FACTOR JOURNAL 2007

Office of Portfolio Analysis (OPA) DPCPSI OD 29

Modeling NIH output Advanced Bibliometrics

INPUT $$

R01

OUTPUT pubs patents etc

For NIH R01 awardees who published in those 12 journals in 2007

Most of the projects have been resubmitted (T2 applications)

What is the relationship if any between prior output and review outcomes

Office of Portfolio Analysis (OPA) DPCPSI OD 30

Journal IF Affects R01 Fate More Than of Citations

Effect of Impact Factor

Effect of Citations

Rela

tive

pro

babili

ty o

f success (

)

+10

+

20

+

30

+

40

R01 T2 renewal applicants who published in selected 2007 journals (N = 1631)

Baseline +20 +40 +60 +80 +100

Journal Impact Factor Number of Citations

Office of Portfolio Analysis (OPA) DPCPSI OD 31

Acknowledgments

THE OPA TEAM

Geetha Senthil PhD Science Analyst

Paula Fearon PhD Science Analyst

Jean Yuan PhD Science Analyst

Carole Christian PhD Health Science Administrator

Ehsan Haque MS Data Analyst

Chuck Lynch PhD IT Specialist

Kevin Small PhD Software Developer

COLLABORATORS

Calvin Johnson PhD Chief High Performance Computing amp Informatics CIT

Katy Borner PhD Victor H Yngve Professor of Information Science

Indiana University

Office of Portfolio Analysis (OPA) DPCPSI OD 32

Cit

ati

on

s

The Citation Pattern is Established Soon After Publication

Papers from 1997 in the 12 journals analyzed HI25

20

15

10

5

0

Years after date of publication

MED

LOW

(average of four journals in each group)

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Office of Portfolio Analysis (OPA) DPCPSI OD 33

Cell Science NEJM Nature Genes Dev

J Exp Med

J Clin Inv JBC Lancet Mol Pharm

Genetics FEBS Lett

The Citation Pattern is Established Soon After Publication

45 Papers from 1997 in the 12 journals analyzed

40 1998

199935

2000

200130 2002

2003

2004

25

20 2005

2006 15

2007

200810

2009

20105 2011

0 JOURNAL 1 2 3 4 1 2 3 4 1 2 3 4

Cit

atio

ns

IMPACT FACTOR HIGH MEDIUM LOW

Office of Portfolio Analysis (OPA) DPCPSI OD 34

0

50

100

150

200

Cit

ati

on

s

200

MEDIUM IF JOURNALS

150

100

50

0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 1 2 3 4 5 6 7

Years after date of publication

8 9 10 11 12 13 14

HIGH IF JOURNALS

The Citation Pattern is Established Soon After Publication

Office of Portfolio Analysis (OPA) DPCPSI OD 35

Office of Portfolio Analysis (OPA) DPCPSI OD

Modeling NIH output Advanced Bibliometrics

INPUT $$

R01

OUTPUT pubs patents etc

For NIH R01 awardees who published in those 12 journals in 2007

Most of the projects have been resubmitted (T2 applications)

What is the relationship if any between prior output and review outcomes

Office of Portfolio Analysis (OPA) DPCPSI OD 30

Journal IF Affects R01 Fate More Than of Citations

Effect of Impact Factor

Effect of Citations

Rela

tive

pro

babili

ty o

f success (

)

+10

+

20

+

30

+

40

R01 T2 renewal applicants who published in selected 2007 journals (N = 1631)

Baseline +20 +40 +60 +80 +100

Journal Impact Factor Number of Citations

Office of Portfolio Analysis (OPA) DPCPSI OD 31

Acknowledgments

THE OPA TEAM

Geetha Senthil PhD Science Analyst

Paula Fearon PhD Science Analyst

Jean Yuan PhD Science Analyst

Carole Christian PhD Health Science Administrator

Ehsan Haque MS Data Analyst

Chuck Lynch PhD IT Specialist

Kevin Small PhD Software Developer

COLLABORATORS

Calvin Johnson PhD Chief High Performance Computing amp Informatics CIT

Katy Borner PhD Victor H Yngve Professor of Information Science

Indiana University

Office of Portfolio Analysis (OPA) DPCPSI OD 32

Cit

ati

on

s

The Citation Pattern is Established Soon After Publication

Papers from 1997 in the 12 journals analyzed HI25

20

15

10

5

0

Years after date of publication

MED

LOW

(average of four journals in each group)

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Office of Portfolio Analysis (OPA) DPCPSI OD 33

Cell Science NEJM Nature Genes Dev

J Exp Med

J Clin Inv JBC Lancet Mol Pharm

Genetics FEBS Lett

The Citation Pattern is Established Soon After Publication

45 Papers from 1997 in the 12 journals analyzed

40 1998

199935

2000

200130 2002

2003

2004

25

20 2005

2006 15

2007

200810

2009

20105 2011

0 JOURNAL 1 2 3 4 1 2 3 4 1 2 3 4

Cit

atio

ns

IMPACT FACTOR HIGH MEDIUM LOW

Office of Portfolio Analysis (OPA) DPCPSI OD 34

0

50

100

150

200

Cit

ati

on

s

200

MEDIUM IF JOURNALS

150

100

50

0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 1 2 3 4 5 6 7

Years after date of publication

8 9 10 11 12 13 14

HIGH IF JOURNALS

The Citation Pattern is Established Soon After Publication

Office of Portfolio Analysis (OPA) DPCPSI OD 35

Office of Portfolio Analysis (OPA) DPCPSI OD

Journal IF Affects R01 Fate More Than of Citations

Effect of Impact Factor

Effect of Citations

Rela

tive

pro

babili

ty o

f success (

)

+10

+

20

+

30

+

40

R01 T2 renewal applicants who published in selected 2007 journals (N = 1631)

Baseline +20 +40 +60 +80 +100

Journal Impact Factor Number of Citations

Office of Portfolio Analysis (OPA) DPCPSI OD 31

Acknowledgments

THE OPA TEAM

Geetha Senthil PhD Science Analyst

Paula Fearon PhD Science Analyst

Jean Yuan PhD Science Analyst

Carole Christian PhD Health Science Administrator

Ehsan Haque MS Data Analyst

Chuck Lynch PhD IT Specialist

Kevin Small PhD Software Developer

COLLABORATORS

Calvin Johnson PhD Chief High Performance Computing amp Informatics CIT

Katy Borner PhD Victor H Yngve Professor of Information Science

Indiana University

Office of Portfolio Analysis (OPA) DPCPSI OD 32

Cit

ati

on

s

The Citation Pattern is Established Soon After Publication

Papers from 1997 in the 12 journals analyzed HI25

20

15

10

5

0

Years after date of publication

MED

LOW

(average of four journals in each group)

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Office of Portfolio Analysis (OPA) DPCPSI OD 33

Cell Science NEJM Nature Genes Dev

J Exp Med

J Clin Inv JBC Lancet Mol Pharm

Genetics FEBS Lett

The Citation Pattern is Established Soon After Publication

45 Papers from 1997 in the 12 journals analyzed

40 1998

199935

2000

200130 2002

2003

2004

25

20 2005

2006 15

2007

200810

2009

20105 2011

0 JOURNAL 1 2 3 4 1 2 3 4 1 2 3 4

Cit

atio

ns

IMPACT FACTOR HIGH MEDIUM LOW

Office of Portfolio Analysis (OPA) DPCPSI OD 34

0

50

100

150

200

Cit

ati

on

s

200

MEDIUM IF JOURNALS

150

100

50

0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 1 2 3 4 5 6 7

Years after date of publication

8 9 10 11 12 13 14

HIGH IF JOURNALS

The Citation Pattern is Established Soon After Publication

Office of Portfolio Analysis (OPA) DPCPSI OD 35

Office of Portfolio Analysis (OPA) DPCPSI OD

Acknowledgments

THE OPA TEAM

Geetha Senthil PhD Science Analyst

Paula Fearon PhD Science Analyst

Jean Yuan PhD Science Analyst

Carole Christian PhD Health Science Administrator

Ehsan Haque MS Data Analyst

Chuck Lynch PhD IT Specialist

Kevin Small PhD Software Developer

COLLABORATORS

Calvin Johnson PhD Chief High Performance Computing amp Informatics CIT

Katy Borner PhD Victor H Yngve Professor of Information Science

Indiana University

Office of Portfolio Analysis (OPA) DPCPSI OD 32

Cit

ati

on

s

The Citation Pattern is Established Soon After Publication

Papers from 1997 in the 12 journals analyzed HI25

20

15

10

5

0

Years after date of publication

MED

LOW

(average of four journals in each group)

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Office of Portfolio Analysis (OPA) DPCPSI OD 33

Cell Science NEJM Nature Genes Dev

J Exp Med

J Clin Inv JBC Lancet Mol Pharm

Genetics FEBS Lett

The Citation Pattern is Established Soon After Publication

45 Papers from 1997 in the 12 journals analyzed

40 1998

199935

2000

200130 2002

2003

2004

25

20 2005

2006 15

2007

200810

2009

20105 2011

0 JOURNAL 1 2 3 4 1 2 3 4 1 2 3 4

Cit

atio

ns

IMPACT FACTOR HIGH MEDIUM LOW

Office of Portfolio Analysis (OPA) DPCPSI OD 34

0

50

100

150

200

Cit

ati

on

s

200

MEDIUM IF JOURNALS

150

100

50

0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 1 2 3 4 5 6 7

Years after date of publication

8 9 10 11 12 13 14

HIGH IF JOURNALS

The Citation Pattern is Established Soon After Publication

Office of Portfolio Analysis (OPA) DPCPSI OD 35

Office of Portfolio Analysis (OPA) DPCPSI OD

Cit

ati

on

s

The Citation Pattern is Established Soon After Publication

Papers from 1997 in the 12 journals analyzed HI25

20

15

10

5

0

Years after date of publication

MED

LOW

(average of four journals in each group)

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Office of Portfolio Analysis (OPA) DPCPSI OD 33

Cell Science NEJM Nature Genes Dev

J Exp Med

J Clin Inv JBC Lancet Mol Pharm

Genetics FEBS Lett

The Citation Pattern is Established Soon After Publication

45 Papers from 1997 in the 12 journals analyzed

40 1998

199935

2000

200130 2002

2003

2004

25

20 2005

2006 15

2007

200810

2009

20105 2011

0 JOURNAL 1 2 3 4 1 2 3 4 1 2 3 4

Cit

atio

ns

IMPACT FACTOR HIGH MEDIUM LOW

Office of Portfolio Analysis (OPA) DPCPSI OD 34

0

50

100

150

200

Cit

ati

on

s

200

MEDIUM IF JOURNALS

150

100

50

0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 1 2 3 4 5 6 7

Years after date of publication

8 9 10 11 12 13 14

HIGH IF JOURNALS

The Citation Pattern is Established Soon After Publication

Office of Portfolio Analysis (OPA) DPCPSI OD 35

Office of Portfolio Analysis (OPA) DPCPSI OD

Cell Science NEJM Nature Genes Dev

J Exp Med

J Clin Inv JBC Lancet Mol Pharm

Genetics FEBS Lett

The Citation Pattern is Established Soon After Publication

45 Papers from 1997 in the 12 journals analyzed

40 1998

199935

2000

200130 2002

2003

2004

25

20 2005

2006 15

2007

200810

2009

20105 2011

0 JOURNAL 1 2 3 4 1 2 3 4 1 2 3 4

Cit

atio

ns

IMPACT FACTOR HIGH MEDIUM LOW

Office of Portfolio Analysis (OPA) DPCPSI OD 34

0

50

100

150

200

Cit

ati

on

s

200

MEDIUM IF JOURNALS

150

100

50

0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 1 2 3 4 5 6 7

Years after date of publication

8 9 10 11 12 13 14

HIGH IF JOURNALS

The Citation Pattern is Established Soon After Publication

Office of Portfolio Analysis (OPA) DPCPSI OD 35

Office of Portfolio Analysis (OPA) DPCPSI OD

0

50

100

150

200

Cit

ati

on

s

200

MEDIUM IF JOURNALS

150

100

50

0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 1 2 3 4 5 6 7

Years after date of publication

8 9 10 11 12 13 14

HIGH IF JOURNALS

The Citation Pattern is Established Soon After Publication

Office of Portfolio Analysis (OPA) DPCPSI OD 35

Office of Portfolio Analysis (OPA) DPCPSI OD

Office of Portfolio Analysis (OPA) DPCPSI OD