Improving Peer Review

CSR Initiatives

toni scarpa scarpat@csr.nih.gov

301-435-1109

PRAC

Nov. 3, 2008

National Institutes of Health

U.S. Department of Health and Human Services

New Data and the Drivers for Change

CSR’s Efforts to Enhance Peer Review

The NIH Director’s Peer Review Initiatives

Improving Peer Review

New Data and the Drivers for Change

CSR Peer Review: 2008

• 77,000 applications received

• 56,000 applications reviewed

• 16,000 reviewers

• 240 Scientific Review Officers

• 1,800 review meetings

$13.

7

$15.

6

$17.

8 $20.

5 $23.

3 $27.

1

$28.

0

$28.

6

$28.

6

$29.

1

$29.

5

$29.

5

$0

$5

$10

$15

$20

$25

$30

1998

1999

2000

2001

2002

2003

2004

200520

0620

0720

0820

09

1st Driver: The NIH Budget

Doubling

0

20000

40000

60000

80000

1998 2000 2002 2004 2006 2008

2nd Driver: Number of Applications Submitted

Historical Growth

SBIR Applications

7000

6000

5000

4000

3000

2000

1000

01999 2000 2001 2002 2003 2004 2005 2006 2007

INST

CSR

R01 Applications Reviewed at CSR vs. Other ICs

0

5000

10000

15000

20000

25000

30000

35000

2001 2002 2003 2004 2005 2006 2007 2008

IC

CSR

R21 Applications Reviewed at CSR vs. Other ICs

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

2001 2002 2003 2004 2005 2006 2007 2008

IC

CSR

0

2

4

6

8

10

12

3rd Driver: Reviewer’s Load

Applications Per Reviewer

October Council Rounds

4th Driver: CSR Budget

0

10

20

30

40

50

60

70

2004 2005 2006 2007 2008 2009

CSR Reviewer Cost Constant $

$ M

illion

s

4th Driver: CSR Budget

0

10

20

30

40

50

60

70

2004 2005 2006 2007 2008 2009

CSR Reviewer Cost Constant $

$ M

illion

s

Annual Savings in Reviewers’ Expenses Budget

• Non-refundable tickets with one possible change

$15 million• 3,000 fewer reviewers

$3 million• 15% reviews using electronic platforms

$5 million • One meeting a year on the West Coast

$1.8 million

Next Savings?

• Replace shipping of CDs with online downloading

Cost of 18,000 CDs $750,000

Cost of shipping $200,000

Handling $50,000

• 5% more reviews using electronic platform

SREA budget $1,500,000

CSR budget -$100,000

• More meetings in Baltimore

Savings/100 meetings $500,000

• Review of fellowship applications

CSR’s Efforts to Enhance Peer Review

1. Reorganizing CSR’s Review Groups

2. Recruiting CSR Staff

3. Shortening the Review Cycle

4. Holding Out-of-Town Meetings

5. Recruiting the Best Reviewers

6. Performing a Second Cycle of IRG Reviews

7. Revising Study Section Guidelines

8. Training Chairs

9. Managing Conflicts of Interest

10. Advancing Additional Review Platforms and Processes

CSR’s Efforts to Enhance Peer Review

CSR’s Efforts to Enhance Peer Review

1. Reorganizing CSR’s Review Groups

1. CSR Organization of Peer Review

Divisions Scientific Review OfficersIntegrated Review Groups

1. CSR Reorganization

Rene

Etcheberrigaray

Translational and Clinical Sciences

Cardiovascular and Respiratory Sciences

Surgical Sciences, Biomedical

Imaging and Bioengineering

Musculoskeletal, Oral And Skin Sciences

Oncology 2 – Translational Clinical

Vascular and Hematology

Physiological and Pathological Sciences

Endocrinology, Metabolism, Nutrition & Reproductive Sciences

Immunology

Infectious Diseases& Microbiology

Digestive, Kidney &Urological Systems

Neuroscience, Development and Aging

Brain Disorders &Clinical Neuroscience

Molecular, Cellular &Developmental Neuroscience

Integrative, Functional & Cognitive Neuroscience

Emerging Technologies &Training in Neuroscience

Biology of Development &

Biobehavioral & Behavioral Processes

Risk, Prevention& Health Behaviors

Epidemiology & Population Sciences

Healthcare Delivery & Methodologies

AIDS &Related Research

AIDS, Behavioral and Population Sciences

Basic and Integrative Biological Sciences

Biological Chemistry & Macromolecular

Biophysics

Bioengineering Sciences& Technologies

Genes, Genomes &Genetics

Oncology 1 – Basic Translational

Cell Biology

Interdisciplinary Molecular & Training

Don

Schneider

Update on CSR Realignments

Don Schneider

Update on CSR Realignments

Don Schneider

1. CSR Organization of Peer Review

CSR’s Efforts to Enhance Peer Review

1. Reorganizing CSR’s Review Groups

2. Recruiting CSR Staff

2. Recruiting CSR Staff

3 Division Directors

6 Integrated Review Chiefs

20 Scientific Review Officers

CSR’s Efforts to Enhance Peer Review

1. Reorganizing CSR’s Review Groups

2. Recruiting CSR Staff

3. Shortening the Review Cycle

3. Shortening the Review Cycle

The Goal

• To provide applicants a review and score within 3 months of application submission. This will permit resubmission of applications (when doable and desirable) 4 months earlier than in the past.

RO1 A1 Resubmission Within 4 Months of Original Application

0

100

200

300

400

500

600

2006 2007 2008

New PI

Estab.PI

CSR’s Efforts to Enhance Peer Review

1. Reorganizing CSR’s Review Groups

2. Recruiting CSR Staff

3. Shortening the Review Cycle

4. Holding Out-of-Town-Meetings

4. Holding Out-of-Town Meetings

• To facilitate participation from colleagues on the West Coast, one meeting out of three each year will be held in the metro areas of:o Los Angeleso San Franciscoo Seattleo Chicago

Benefits• Great satisfaction• Significant savings• Ranking of hotels

5. Near-Term Solutions for Recruiting and Retaining the Best Reviewers

Move a meeting a year to the West Coast

Develop a national registry of volunteer reviewers Searchable database with 3,500 reviewers

Provide tangible rewards for reviewers No submission deadlines for chartered members of study sections (effective February 2008). 1,574 chartered members used flexible deadlines during the last 6 months

Provide flexible time for reviewers Choice of 3 times/year for 4 years or2 times/year for 6 years

CSR’s Efforts to Enhance Peer Review

1. Reorganizing CSR’s Review Groups

2. Recruiting CSR Staff

3. Shortening the Review Cycle

4. Out-of-Town Meetings

5. Recruiting the Best Reviewers

6. Performing a Second Cycle of IRG Reviews

6. Second Cycle of IRG ReviewsScheduled 2008-5 IRGs Scheduled 2009-12 IRGs (Continued)

Brain Disorders and Clinical Neuroscience (BDCN) and Molecular, Cellular and Developmental Neuroscience (MDCN)

Bioengineering Sciences and Technologies (BST) andSurgical Sciences, Biomedical Imaging and Bioengineering (SBIB)

Biology of Development and Aging (BDA), Emerging Technologies and Training in Neuroscience (ETTN) andIntegrative, Functional and Cognitive Neuroscience (IFCN)

AIDS and Related Research (AARR) andEndocrinology, Metabolism, Nutrition and Reproductive Sciences (EMNR)

Scheduled 2009-12 IRGs Scheduled 2010-8 IRGs

Biobehavioral and Behavioral Processes (BBB) andRisk, Prevention and Health Behavior (RPHB)

Vascular and Hematology (VH) andCardiovascular and Respiratory Sciences (CVR)

Healthcare Delivery and Methodologies (HDM) andEpidemiology and Population Sciences (EPS)

Musculoskeletal, Oral and Skin Sciences (MOSS) and Digestive, Kidney and Urological Systems (DKUS)

Immunology (IMM) and Infectious Diseases and Microbiology (IDM)

Genes, Genomes and Genetics (GGG) andBiological, Chemical and Macromolecular Biophysics (BCMB)

Oncology 1—Basic and Translational (OBT) andOncology 2—Translational Clinical (OTC)

Interdisciplinary Molecular Sciences and Training (IMST) and Cell Biology (CB)

6. Performing a Second Cycle of IRG Reviews

• All pertinent study section chairs were invited• Advice on study section and IRG• Revision of study sections description/outline• Training of the chair

6. Performing a Second Cycle of IRG Reviews

Second Cycle of IRG Reviews

Don Schneider

CSR’s Efforts to Enhance Peer Review

1. Reorganizing CSR’s Review Groups

2. Recruiting CSR Staff

3. Shortening the Review Cycle

4. Holding Out-of-Town Meetings

5. Recruiting the Best Reviewers

6. Performing a Second Cycle of IRG Reviews

7. Revising Study Section Guidelines

7. Revising Study Section Outlines

• Cellular Signaling and Regulatory Systems

• [Roster]

• The Cellular Signaling and Regulatory Systems (CSRS) study section reviews applications that focus on the initiation and execution of programs that control cellular homeostasis and physiology. A distinguishing characteristic of these applications is an emphasis on signaling networks and the coordination of processes related to cell proliferation, survival, and growth.

• Cell cycle regulation, mitosis, meiosis, checkpoint controls and regulation by ubiquitination• Proteolytic mechanisms associated with cell cycle, senescence and death• Programmed cell death and apoptosis, particularly their regulation in the context of stress,

growth, and transformation.  • Proliferation and growth control by the nucleus; signaling pathways regulating transcription• Integrative cell physiology, e.g., stress, clocks, cellular modeling; cell differentiation and

transformation• Basic studies of cytokine signaling• Application of state-of-the-art technologies such as imaging and computational modeling of

cellular signaling networks

• Study sections with most closely related areas of similar science listed in rank order are:

• Molecular and Integrative Signal Transduction• Intercellular Interactions• Membrane Biology and Protein Processing• Molecular Genetics A• Molecular Genetics B

CSR’s Efforts to Enhance Peer Review

1. Reorganizing CSR’s Review Groups

2. Recruiting CSR Staff

3. Shortening the Review Cycle

4. Holding Out-of-Town Meetings

5. Recruiting the Best Reviewers

6. Performing a Second Cycle of IRG Reviews

7. Revising Study Section Guidelines

8. Training Chairs

8. Training Study Section Chairs

Scope and Schedule • 120 new chairs every year, more if “permanent” SEPs

chairs are included• 5 meetings in January 2009 for chairs recently appointed• 5 meetings in July 2009 for chairs appointed next June

Training Program• Share data • Explain the new changes and the significance• Share the best practice (ours and theirs)• Answer questions and address concerns• Make Chairs more effective stakeholders

CSR’s Efforts to Enhance Peer Review

1. Reorganizing CSR’s Review Groups

2. Recruiting CSR Staff

3. Shortening the Review Cycle

4. Holding Out-of-Town Meetings

5. Recruiting the Best Reviewers

6. Performing a Second Cycle of IRG Reviews

7. Revised Study Section Guidelines

8. Training of Chairs

9. Managing Conflicts of Interest

9. Managing Conflicts of Interest

Report of Conflict of Interest Committee

Noni Byrnes

CSR’s Efforts to Enhance Peer Review

1. Reorganizing CSR’s Review Groups

2. Recruiting CSR Staff

3. Shortening the Review Cycle

4. Holding Out-of-Town Meetings

5. Recruiting Reviewers

6. Performing a Second Cycle of IRG Review

7. Revising Study Section Guidelines

8. Training of Chairs

9. Managing Conflicts of interest

10. Advancing Additional Review Platforms and Processes

10. Advancing Additional Review Platforms and Process

• Video (VED)

• Internet (AED)

• Editorial Board

• Order of Discussion

10. Advancing Additional Review Platforms and Processes

Editorial Board Reviewo Heavy triage based on innovation and potential science

transformation by a small study section of distinguished, broad-science reviewers (the editors)

o Specific science reviewed by appropriate reviewers (the editorial board)

o Final ranking by the editors

10. Advancing Additional Review Platforms and Process

New Transformative R01

Betsy Wilder

10. Advancing Additional Review Platforms and Process

Electronic Discussion Survey

Andrea Kopstein

NIH Enhancement of Peer Review

• The Charge from Dr. Zerhouni:

““Fund the best science, by the best scientists, Fund the best science, by the best scientists,

with the least administrative burden…”with the least administrative burden…”

http://enhancing-peer-review.nih.gov

Two advisory committees to the NIH Director

The NIH Director’s Peer Review Initiatives

The NIH Director’s Peer Review Initiatives

Corporate NIH Peer Review Enhancement: Implementation

Raynard Kington

This is CSR

September 2008

Recognizing Outstanding Reviewers

• Going above and beyond the call of duty

• Having a significant impact on the quality of peer review

• Educating and inspiring other reviewers

• Setting the highest standard of peer review excellence

The First Marcy Speer Award, 2007

Marcy Speer

2008 Marcy Speer Outstanding CSR Reviewer Award

Dr. David J. Sahn, M.D. Oregon Health & Science University

The 2nd Marcy Speer Award, 2008