1 Electronic Records Management and Preservation Denis Plude June 26, 2006.
THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy...
-
Upload
savannah-mccabe -
Category
Documents
-
view
219 -
download
1
Transcript of THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy...
![Page 1: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/1.jpg)
1
THE NIH PEER REVIEW PROCESS
Sally A. Amero, Ph.D. Dana Plude, Ph.D.NIH Review Policy Officer Biobehavioral and Behavioral Processes IRGNational Institutes of Health NIH Center for Scientific Review
NIH Regional Seminars 2013
![Page 2: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/2.jpg)
2
Your Research Project
• Do you have a – Research interest– New idea– Reason to think the experiments would work, and others
would care about the results?• Do you have– Time (at least 20%)– Resources (a lab)– Qualifications– Motivation?
• Considering applying for an NIH grant
![Page 3: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/3.jpg)
3
NIH Peer Review
• Cornerstone of the NIH extramural mission• Standard of excellence worldwide• Partnership between NIH and the scientific
community• Per year:
~ 80,000 applications~ 18,000 reviewers
![Page 4: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/4.jpg)
4
Review Process
• Receipt and referral – Center for Scientific Review
• Initial peer review – “Study Sections” • Second level peer review – Advisory Councils
or Boards
Receipt and Referral
Initial Peer Review
National Advisory Councils
![Page 5: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/5.jpg)
5
NIH Grants Process - Overview
Application NIH Center for Scientific Review (CSR)Assignments made
Initial peer review Funding considerations Study section Institutes or Centers (ICs)
IC or CSR Duals possible Scientific Review Officer Program Officer
Second level of review Funding decisions
Council or Board (IC) IC Director
Award!
![Page 6: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/6.jpg)
6
Receipt and Referral
• Key decisions– Format compliance– Timeliness– Assignment to study section for initial peer review– Assignment to IC(s) for funding consideration
• Initial peer review• (CSR or IC)
Study Section
IC(s)
CSR
Council
![Page 7: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/7.jpg)
7
Locus of Review
• CSR Review– Most R01’s, F’s and SBIR’s– Some Program Announcements– Some Requests for Applications (RFAs)
• Institute/Center Review– IC-specific features– P’s, T’s, K’s– Most RFAs
• Initial peer review• (CSR or IC)Study
Section
![Page 8: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/8.jpg)
8
Requesting a Study Section
• The locus of review (CSR/IC) is usually stated in the FOA.
• The Study Section assignment is available in the PD/PI’s Commons account.
• Descriptions of study sections in the Center for Scientific Review (CSR) are posted online: http://public.csr.nih.gov/StudySections/IntegratedReviewGroups/Pages/default.aspx
![Page 9: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/9.jpg)
9
Requesting a Study Section
•Rosters are available on NIH websites– http://era.nih.gov/roster/index.cfm– http://www.csr.nih.gov/committees/rosterindex.asp
• Permanent membership is available anytime• Membership for a given meeting is posted 30
days before the meeting– Subject to change– Some CSR rosters are posted in aggregate
![Page 10: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/10.jpg)
10
Requesting a Study Section
•Cover letter of application – Application title– FOA # and title– Request:
Particular Study Section(s) or Integrated Review Group(s) Particular IC for funding consideration
– Disciplines involved, if multidisciplinary• Not all requests can be honored
![Page 11: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/11.jpg)
11
Conflict of Interest• Bases for Conflict of Interest (COI) – Financial - Professional– Employment - Study Section membership– Personal - Other interests
• Appearance of COI• Depending on nature of COI, individual with a COI– must be excluded from serving on the Study Section, or – must be recused from discussion and scoring of
application.
![Page 12: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/12.jpg)
12
NIH Scoring System
• Reviewers give numerical scores – 1 (exceptional) to 9 (poor) – Integers
• Used for:– Final impact scores– Individual criterion scores
1
–
high impact
9
– low impact
![Page 13: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/13.jpg)
13
Score DescriptorsImpact Score Descriptor
High Impact1 Exceptional
2 Outstanding
3 Excellent
Moderate Impact
4 Very Good
5 Good
6 Satisfactory
Low Impact7 Fair
8 Marginal
9 Poor
![Page 14: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/14.jpg)
14
Final Impact Scores
• Voted by all eligible (w/o COI) SRG members• Voted by private ballot at the meeting• Calculated by:– Averaging all reviewers’ votes– Multiplying by 10
• Range from 10 through 90• Percentiled for some mechanisms
10 – Highest Impact
90 – Lowest Impact
![Page 15: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/15.jpg)
15
Criterion Scores
• Minimum of five scored criteria • Given by assigned reviewers as part of their
critiques• Generally not discussed at the meeting• Reported on the summary statement
1
–
high impact
9
– low impact
![Page 16: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/16.jpg)
16
Not Discussed Applications
• Allows discussion of more meritorious applications– Less meritorious applications are tabled– Designated Not Discussed (ND)
• Requires full concurrence of the entire SRG• Summary statements contain:
– Reviewer critiques– Criterion scores
1
ND
![Page 17: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/17.jpg)
17
Scientific Review Officer
• Identifies and recruits reviewers• Assigns reviewers to individual applications• Manages conflicts of interest• Arranges and presides at review meetings• Prepares summary statements – official
written outcome of initial peer review
![Page 18: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/18.jpg)
18
Reviewers
• Expertise• Stature in field• Mature judgment• Impartiality• Ability to work well in a group• Managed conflicts of interest• Balanced representation• Availability
![Page 19: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/19.jpg)
19
Types of Reviewers• Regular reviewers– Participate in committee discussions – Contribute preliminary impact scores, criterion
scores, written critiques, final impact scores• “Mail” reviewers– Contribute preliminary impact scores, criterion
scores, written critiques– Do not participate in committee discussion– Cannot submit final impact scores
![Page 20: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/20.jpg)
20
Confidentiality
• All confidential materials, discussions, documents are deleted, retrieved, or destroyed.
• All questions must be referred to the SRO. • Applicants: Do not contact reviewers directly!
![Page 21: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/21.jpg)
21
Overall Impact• Overall consideration for all NIH applications• Defined differently for different types of
applications– Research grant applications: Likelihood for the
project to exert a sustained, powerful influence on the research field(s) involved
– See “Review Criteria at a Glance” (http://grants.nih.gov/grants/peer/reviewer_guidelines.htm)
![Page 22: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/22.jpg)
22
Scored Review Criteria
• Receive individual, numerical scores from the assigned reviewers. • For research grant applications:– Significance - Approach– Investigator(s) - Environment– Innovation
![Page 23: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/23.jpg)
23
Additional Review Criteria
• Are considered in determining the impact score, as applicable for the project proposed
• For research grant applications: – Protections for Human Subjects– Inclusion of Women, Minorities, and Children – Vertebrate Animals – Resubmission, Renewal, and Revision Applications– Biohazards
![Page 24: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/24.jpg)
24
Additional Review Considerations
• Are not considered in determining impact score• Generate reviewer comments for Program
Officials to consider• For research grant applications:
– Applications from Foreign Organizations– Select Agent Research– Resource Sharing Plans– Budget and Period of Support
![Page 25: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/25.jpg)
25
Study Sections
• Make recommendations on:– Scientific and technical merit– Impact
Impact scores Criterion scores Written critiques
– Other review considerations
![Page 26: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/26.jpg)
26
Reviewer Assignments
• For each application: – ≥ Three qualified reviewers are assigned for in-depth
assessment– Assignments are made by the SRO
Expertise of the reviewer Suggestions from the PI on expertise – not names! Suggestions from Program staff and Study Section members Managing conflicts of interest Balancing workload
• Assignments are confidential
![Page 27: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/27.jpg)
27
Before the Meeting• Reviewers – Examine assignments (~ six weeks in advance)– Often participate in an SRO orientation teleconference– Sign Conflict of Interest and Confidentiality certifications– Read applications, prepare written critiques – Enter preliminary scores and critiques into secure website– Read and consider critiques and preliminary scores from other Study Section members
![Page 28: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/28.jpg)
28
Critique Templates
Links to definitionsof reviewcriteria
![Page 29: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/29.jpg)
29
Study Section Agenda
• Introductions and policy review• In some meetings, streamlining occurs first • Cluster where feasible: – New Investigator (NI) applications– Clinical applications
• Discuss each remaining application – Assigned reviewers lead off – Chairperson summarizes main points – Members score after its discussion– Members discuss other considerations
![Page 30: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/30.jpg)
30
Discussion Format
• Members with conflicts excused • Initial levels of enthusiasm stated (assigned reviewers)• Primary reviewer - explains project, strengths,
weaknesses• Other assigned reviewers and discussants follow• Open discussion (full panel)• Levels of enthusiasm re-stated (assigned reviewers)• Chair summarizes main points from discussion• All Study Section members vote – private ballot• Other review considerations discussed (budget)
![Page 31: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/31.jpg)
31
After the Review
• eRA Commons (http://era.nih.gov/commons/index.cfm)
– Final Impact Score is available in 3 days. – Summary statement is available in 4 – 8 weeks.
• Available to:– PD/PIs– NIH officials– Advisory Council members
• NIH Program Officer = Point of Contact
![Page 32: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/32.jpg)
32
Summary Statement
• First page– NIH Program Officer (upper left corner)– Final Impact Score or other designation– Percentile (if applicable)– Codes (human subjects, vertebrate animals,
inclusion) – Budget request
• A favorable score does not guarantee funding!
![Page 33: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/33.jpg)
33
Summary Statement - continued
• Subsequent Pages– Description (provided by applicant)– Resumé and Summary of Discussion (if discussed)– Reviewer critiques – essentially unedited– Administrative Notes– Meeting roster
![Page 34: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/34.jpg)
34
After the Review
• If the outcome is favorable, congratulations!• If the outcome is unfavorable, consider your
options:– Revise and resubmit your application– Appeal the review outcome
![Page 35: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/35.jpg)
35
Appeals of initial peer review
• Acceptable reasons (NOT-OD-11-064)– Evidence of bias– Conflict of interest, as specified in regulation (42 CFR 52h.5)– Lack of appropriate expertise within the SRG – Factual error(s) that could have altered the outcome of the review substantially.
• Differences of scientific opinion cannot be appealed
![Page 36: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/36.jpg)
36
National Advisory Councils
• Broad and diverse membership– Basic /research scientists– Clinician scientists– “Public” members
• Nominated by Institutes; approved by HHS (or the President in a few cases)
• Awards cannot be made without Council approval
• Council procedures vary across IC’s
![Page 37: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/37.jpg)
37
National Advisory Councils
• Advise IC Director about– Research priority areas– Diverse policy issues– Concept Clearance for future initiatives– Funding priorities
• Recommend applications for funding– Expedited awards– En bloc concurrence
![Page 38: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/38.jpg)
38
National Advisory Councils
• Consider unresolved appeals and grievances• Council options
– Support the Study Section review– Support the appeal, recommend re-review• Application could be deferred for next round• Application cannot modified or updated
– Results of re-review cannot be appealed further• Council cannot overturn the review or impact
score
![Page 39: THE NIH PEER REVIEW PROCESS Sally A. Amero, Ph.D.Dana Plude, Ph.D. NIH Review Policy OfficerBiobehavioral and Behavioral Processes IRG National Institutes.](https://reader030.fdocuments.in/reader030/viewer/2022032701/56649c8f5503460f94948bbb/html5/thumbnails/39.jpg)
39
Additional Information
• Office of Extramural Research Peer Review Process
http://grants.nih.gov/grants/peer_review_process.htm
• Peer Review Policies & Practices http://grants.nih.gov/grants/peer/peer.htm
• Center for Scientific Review http://cms.csr.nih.gov/AboutCSR/Welcome+to+CSR/