1

CSR’s Mission and Function

and

What’s New in Peer Review

Martha M. Faraday, Ph.D.

Scientific Review OfficerDivision of AIDS, Behavioral & Population Sciences

Risk Prevention & Health Behavior IRGPsychosocial Risk & Disease Prevention Study Section

Date : April 22, 2009

National Institutes of HealthU.S. Department of Health and Human Services

National Institutes of HealthNational Institutes of Health

National Instituteon Alcohol Abuseand Alcoholism

National Instituteon Alcohol Abuseand Alcoholism

National Instituteof Arthritis andMusculoskeletal

and Skin Diseases

National Instituteof Arthritis andMusculoskeletal

and Skin Diseases

National CancerInstitute

National CancerInstitute

National Instituteon Drug Abuse

National Instituteon Drug Abuse

National Instituteof Environmental Health Sciences

National Instituteof Environmental Health Sciences

National Instituteon Aging

National Instituteon Aging

National Instituteof Child Health

and HumanDevelopment

National Instituteof Child Health

and HumanDevelopment

National Institute onDeafness and Other

CommunicationDisorders

National Institute onDeafness and Other

CommunicationDisorders

National EyeInstitute

National EyeInstitute

National HumanGenome Research

Institute

National HumanGenome Research

Institute

National Instituteof Mental Health

National Instituteof Mental Health

National Instituteof NeurologicalDisorders and

Stroke

National Instituteof NeurologicalDisorders and

Stroke

National Instituteof General

Medical Sciences

National Instituteof General

Medical Sciences

National Instituteof Nursing Research

National Instituteof Nursing Research

National Libraryof Medicine

National Libraryof Medicine

Center for Center for Scientific ReviewScientific Review

National Centerfor Complementary

and AlternativeMedicine

National Centerfor Complementary

and AlternativeMedicine

National Instituteof Allergy and

Infectious Diseases

National Instituteof Allergy and

Infectious Diseases

FogartyInternational

Center

FogartyInternational

Center

National Centerfor ResearchResources

National Centerfor ResearchResources

Clinical Center

Clinical Center

National Center on Minority Health andHealth Disparities

National Center on Minority Health andHealth Disparities

National Institute of Biomedical Imagingand Bioengineering

National Institute of Biomedical Imagingand Bioengineering

Office of the DirectorOffice of the Director

Center for InformationTechnology

Center for InformationTechnology

National Heart,Lung, and Blood

Institute

National Heart,Lung, and Blood

Institute

National Instituteof Dental andCraniofacial

Research

National Instituteof Dental andCraniofacial

Research

National Instituteof Diabetes andDigestive and

Kidney Diseases

National Instituteof Diabetes andDigestive and

Kidney Diseases

CSR Mission StatementCSR Mission Statement

To see that NIH grant applications receive

fair, independent, expert, and timely reviews -

free from inappropriate influences - so the

Institutes and Centers within the NIH can fund

the most promising research.

CSR Peer Review: 2008 StatisticsCSR Peer Review: 2008 Statistics

• 77,000 applications received77,000 applications received

• 56,000 applications reviewed 56,000 applications reviewed

• 16,000 reviewers16,000 reviewers

• 240 Scientific Review Officers240 Scientific Review Officers

• 1,600 review meetings1,600 review meetings

Scientific Review ProcessScientific Review Process

Dual Review System for Grant ApplicationsDual Review System for Grant Applications

Second Level of ReviewSecond Level of Review NIH Institute/Center CouncilNIH Institute/Center Council

First Level of Review : First Level of Review : CSR/Institute ReviewCSR/Institute Review

Scientific Review Group (SRG)Scientific Review Group (SRG)

(Study Section)(Study Section)

Translational and Translational and

Clinical SciencesClinical Sciences

Cardiovascular and Cardiovascular and

Respiratory Sciences Respiratory Sciences

Surgical Sciences, Surgical Sciences,

Biomedical Biomedical

Imaging and Imaging and

Bioengineering Bioengineering

Musculoskeletal, Oral Musculoskeletal, Oral

And Skin Sciences And Skin Sciences

Oncology 2 – Oncology 2 –

Translational Clinical Translational Clinical

Vascular and Vascular and

HematologyHematology

Physiological and Physiological and

Pathological Sciences Pathological Sciences

Endocrinology, Endocrinology,

Metabolism, Metabolism,

Nutrition &Nutrition &

Reproductive SciencesReproductive Sciences

ImmunologyImmunology

Infectious DiseasesInfectious Diseases

& Microbiology& Microbiology

Digestive, Kidney &Digestive, Kidney &

Urological Systems Urological Systems

Neuroscience, Development Neuroscience, Development

and Agingand Aging

Brain Disorders &Brain Disorders &

Clinical NeuroscienceClinical Neuroscience

Molecular, Cellular &Molecular, Cellular &Developmental NeurosciencDevelopmental Neuroscience

Integrative, Functional & Integrative, Functional &

Cognitive NeuroscienceCognitive Neuroscience

Emerging Technologies &Emerging Technologies &

Training in NeuroscienceTraining in Neuroscience

Biology of Development Biology of Development

& Aging& Aging

Biobehavioral &Biobehavioral &

Behavioral ProcessesBehavioral Processes

Risk, Prevention& Risk, Prevention&

Health Behaviors Health Behaviors

Population Sciences & Population Sciences &

EpidemiologyEpidemiology

Healthcare Delivery Healthcare Delivery

& Methodologies& Methodologies

AIDS &AIDS &

Related ResearchRelated Research

AIDS, Behavioral AIDS, Behavioral

and Population Sciencesand Population SciencesBasic and Integrative Basic and Integrative

Biological SciencesBiological Sciences

Biological Chemistry & Biological Chemistry &

Macromolecular Macromolecular

Biophysics Biophysics

Bioengineering SciencesBioengineering Sciences

& Technologies& Technologies

Genes, Genomes Genes, Genomes

& Genetics & Genetics

Oncology 1 – Basic Oncology 1 – Basic

TranslationalTranslational

Cell BiologyCell Biology

Interdisciplinary Interdisciplinary

Molecular Molecular

& Training& Training

CSR Review DivisionsCSR Review Divisions

• Standing Study Sections when the subject matter of the application matches the referral guidelines for the study section

• Special Emphasis Panels (SEPs) when the subject matter does not fit into any study section, or when assignment of an application to the most appropriate study section would create a conflict of interest. Also used for special mechanisms (e.g., fellowships, SBIRs, AREAs)

Within an IRG, applications are assignedfor review to

Assignment to CSR Review Groups

• Read the instructions

• Never assume that reviewers “will know what you mean”

• Refer to the literature thoroughly

• State rationale of proposed investigation

• Include well-designed tables and figures

• Present an organized, lucid write-up

• Remember to address human subjects, vertebrate animals, potential biohazards; these could affect your score

• Obtain pre-review from faculty at your institution

NIH Grant Writing Tips: http://grants.nih.gov/grants/grant_tips.htm

When Preparing Your ApplicationWhen Preparing Your Application

Directing Your Application to a Specific Study Section

• Peruse CSR Study Section Guidelines* to Identify a Possible Home for Your Application

http://csr.nih.gov/

* Recently revised; alternative study sections listed in approximate order of degree of overlap

• Submit a Cover Letter

• About CSRAbout CSR

• News and ReportsNews and Reports

• Peer Review Meetings

• Resources for Applicants Resources for Applicants

CSR Web Site: http://www.csr.nih.gov

Role of Scientific Review Officer (SRO)Role of Scientific Review Officer (SRO)

• Performs administrative and technical review of applications to ensure completeness and accuracy

• Selects reviewers based on broad input

• Manages study section meetings

• Prepares summary statements

• Provides any requested information about study section recommendations to Institutes/Centers and National Advisory Councils/Boards

Designated Federal official with overall responsibility for the review process

WHOM DO I CONTACT?WHOM DO I CONTACT?

Before review, contact the Scientific Review Officer in CSR

After review, contact your Program Officer in the NIH funding institute or center

Pre-Meeting Review ProcessPre-Meeting Review Process

• Appropriate reviewers recruited by SRO; minimum of 3 “interactive” reviewers per application

• Conflicts of interest identified

• Applications made available to reviewers ~6 weeks prior to meeting

• Critiques and preliminary scores posted by assigned reviewers on NIH web site at least 2-3 days prior to meeting

• Critiques and preliminary scores (excluding conflicts) available to review group prior to meeting

Where Do We Find Reviewers?Where Do We Find Reviewers?

• National Registry for Society-Recommended Reviewers

• Successful applicants

• Word of mouthRecommendations from study section membersRecommendation from NIH IC staff

• CRISP (crisp.cit.nih.gov)

• PubMed

• Scientific Conferences

Traditional* Review Meeting Process

• Upper half applications discussed:Reviewers are guided by specific review criteria Protections for Humans, Vertebrate Animals, Environment (Biohazard) may affect final scoreAssigned reviewers recommend scores for each application in upper half; all members not in conflict vote their conscience (outlier score policy pertains)Other considerations not affecting final score are discussed (e.g., budget, foreign applicants, resource sharing plans)

• Lower half applications not discussed, not assigned an overall score

* Aspects of this process will change in May, 2009

http://enhancing-peer-review.nih.gov

Post Meeting Review Process

• Scores are provided to investigators within 3 working days

• Summary Statements for discussed and scored applications include Resume & Summary of Discussion, (largely unedited) critiques, and other recommendations (e.g., Budget)

• Summary Statements for lower half (Not Discussed) applications receive (largely unedited) critiques and review criteria scores but no overall impact scores

• All Summary Statements are made available within 30 days of meeting (10 days for new investigators’ R01s)

What’s New in Peer Review?What’s New in Peer Review?

Enhancing Peer ReviewEnhancing Peer Review

““Fund the best science, by the best scientists, Fund the best science, by the best scientists, with the least administrative burden…” with the least administrative burden…”

Elias Zerhouni, MD, Former Director, NIHElias Zerhouni, MD, Former Director, NIH

2008: The Year of Peer Review2008: The Year of Peer Review

RecommendationsRecommendations

Amended Applications:Amended Applications:

To speed the funding of meritorious science and To speed the funding of meritorious science and minimize reviewer burden:minimize reviewer burden:

•As of January 25, 2009, all original new applications (i.e., never submitted) and competing renewal applications will be permitted only a single amendment (A1)single amendment (A1).

What’s New in Peer ReviewWhat’s New in Peer Review

• New Investigators/Early Stage InvestigatorsNew Investigators/Early Stage Investigators

• Enhanced Review CriteriaEnhanced Review Criteria

• Template-Based CritiquesTemplate-Based Critiques

• Scoring Scale (9 point scale)Scoring Scale (9 point scale)

Criterion ScoringCriterion Scoring

Overall Impact ScoreOverall Impact Score

21

New Investigators/Early Stage InvestigatorsNew Investigators/Early Stage Investigators

• New Investigator (NI): New Investigator (NI):

PD/PI who has not yet competed successfully for a substantial NIH research grant

o For multiple PD/PIs-all PD/PIs must meet requirements for NI status

• Early Stage Investigator (ESI):Early Stage Investigator (ESI):

PD/PI who qualifies as a New Investigator AND is within 10 years of completing the terminal research degree or is within 10 years of completing medical residency (or equivalent)

• Applies only to R01 applicationsApplies only to R01 applications• New Investigators/Early Stage Investigators will be clustered New Investigators/Early Stage Investigators will be clustered

together for reviewtogether for review

22

Enhanced Review CriteriaEnhanced Review Criteria

• Overall Impact: Overall Impact: Assessment of the likelihood for the project to exert a sustained, powerful influence on the exert a sustained, powerful influence on the research field(s) involvedresearch field(s) involved

• New Core Criteria Order:New Core Criteria Order:Significance

Investigator(s)

Innovation

Approach

Environmento Review criteria enhanced and expandedReview criteria enhanced and expanded

23

CritiquesCritiques

Goal: To improve the quality of the critiques and to Goal: To improve the quality of the critiques and to focus reviewer attention on the review criteria:focus reviewer attention on the review criteria:

•Provide clear, concise, and explicit information•Aid in identifying the strengths and weaknesses of each criterion

Template-Based CritiquesTemplate-Based Critiques

Template-Based CritiquesTemplate-Based Critiques

• Critique template contains a total of 18 boxesCritique template contains a total of 18 boxesReviewers should provide text for only those criteria that Reviewers should provide text for only those criteria that are applicable.are applicable.

25

1. Significance1. Significance 6. Resubmission 13. Overall Impact13. Overall Impact

2. Investigator(s)2. Investigator(s) 7. Renewal 14. Budget and Period of Support

3. Innovation3. Innovation 8. Revision 15. Select Agents

4. Approach4. Approach 9. Protection of Human Subjects

16. Applications from Foreign Organization

5. Environment5. Environment 10. Inclusion of Women, Minorities, and Children

17. Resource Sharing Plan

11 Vertebrate Animals 18. Additional 18. Additional Comments to Comments to ApplicantApplicant

12. Biohazards

Template-Based CritiquesTemplate-Based Critiques

• Goal: is to write evaluative statements and to Goal: is to write evaluative statements and to discourage summarizing the applicationdiscourage summarizing the application

Comments should be in the form of bullet points or if necessary short narratives

Do not record scores Do not record scores on the critique template

The entire template is uploaded to IAR to become part of the summary statement.

26

1. SignificanceSignificance  Please limit text to ¼ pagePlease limit text to ¼ page

Strengths Weaknesses

Goal: To improve the transparency of the scoring Goal: To improve the transparency of the scoring process:process:

• Score applications on five review criteria using five review criteria using a scale of 1-9a scale of 1-9.

• Preliminary overall impact score using 1-9 overall impact score using 1-9 scalescale.

Should not be the average of the criterion scores.

Not DiscussedNot Discussed applications will receive initial criterion scores from the three assigned reviewers

Scoring – 9 Point ScaleScoring – 9 Point Scale

Scoring DescriptionsScoring Descriptions

Impact Score Descriptor Strengths/Weaknesses

High Impact

1 Exceptional

 Weaknesses

2 Outstanding

3 Excellent

Moderate Impact

4 Very Good

5 Good

6 Satisfactory

Low Impact

7 Fair

8 Marginal

9 Poor

29

ClusteringClustering

• NI/ESI R01 NI/ESI R01 applications will be clustered together in review.

ESI applications will not be separately ESI applications will not be separately clustered within the NI\ESI group.clustered within the NI\ESI group.o NI/ESI applications will be identified for

reviewers so there can be appropriate review in context of career stage.

o Expectations of preliminary data and publication track record less than for established investigators.

Order of ReviewOrder of Review

Goal: Discuss applications in order of average Goal: Discuss applications in order of average preliminary score.preliminary score.

Why:Why:•Concern - variation of scores during different times of the meeting.

One recommendation was to recalibrate scores at One recommendation was to recalibrate scores at the end of the meeting .the end of the meeting .

Solution:Solution:•Recalibrate “dynamically” throughout meeting.

Order of ReviewOrder of Review

• For calibration purposes: For calibration purposes: Begin meeting by discussing the best scored application (any activity code)

o NI/ESI R01s clustered beginning of meetingNI/ESI R01s clustered beginning of meetingo All other activity codes clustered if feasible (if at

least 10 discussed (may include R03, R15, and R21s as a group that can be clustered)

Order of ReviewOrder of Review

SummarySummaryDiscussion order is based on the average of the impact scores from assigned reviewers

Final scores of discussed applications may differ from preliminary scores as re-calibration happens dynamically

Discuss ~ 50-60% of applications

SRO will then ask if there are any other applications that panel wishes to discuss

The remaining applications will not be discussed(applications receive criterion scores only)

o Same after review of ~60% of SBIR applications

Not DiscussedNot Discussed

Final ScoresFinal Scores

• Discussed applications will receive an overall score from each eligible (i.e., without conflicts of interest) panel member and these scores will be averaged to one decimal place, and multiplied by 10. The 81 possible priority scores will thus range from 10-90.

• Percentiles will be reported in whole numbers.

•Summary statement will be shortershorter and more focused.

•DiscussedDiscussed applications will also have a summarysummary of of the panel’s discussiondiscussion at the meeting.

•ALL applications will be scoredALL applications will be scored.

Not discussed Not discussed applications will receive criterion scorescriterion scores only.

Summary StatementsSummary Statements

Recruiting the Best ReviewersRecruiting the Best Reviewers

Move a meeting a year to the West Coast

Additional review platforms

Develop a national registry of volunteer reviewers Searchable database with 4,000 reviewers

Provide tangible rewards for reviewers No submission deadlines for chartered members of study sections (effective February 2008). 1574 chartered members used flexible deadlines during the last 6 months

Provide flexible time for reviewers Choice of 3 times/year for 4 years or2 times/year for 6 years

THANK YOU!THANK YOU!

This is CSRThis is CSR