PEER REVIEW AT NIH

B. Duane Price, Ph.D.

Senior Scientific Review Officer

Immunology Review Branch

National Institute of Allergy and Infectious Diseases (NIAID)

National Institutes of Health (NIH)

15th Annual Upstate New York Immunology Conference

October 22, 2012

Presentation Outline

I. Overview

II. First Level of Review - Scientific Review Groups

A. Peer Review Roles and Meeting Overview

B. Peer Review Criteria and Considerations

C. Scoring

D. Summary Statement

E. Appeals

V. Second Level of Review - Advisory Council/Board

IV. Post-Review

V. Resources

2 http://grants.nih.gov/grants/peer_review_process.htm

I. Overview – Fair and Equitable

3

I. Overview - Timely

4

I. Overview – Free from Bias

5

I. Overview

NIH policy is intended to ensure that grant

applications submitted to the NIH are evaluated

on the basis of a process that is fair, equitable,

timely, and free of bias.

6

I. Overview

The NIH dual peer review system is mandated by

statute in accordance with section 492 of the

Public Health Service Act and federal regulations

governing "Scientific Peer Review of Research

Grant Applications and Research and

Development Contract Projects" (42 CFR Part

52h).

7

Scientific Review Group (SRG)

National Advisory Council

I. Overview (continued)

8

I. Overview (continued)

9

The first level of review is

carried out by a Scientific

Review Group (SRG)

composed primarily of

non-federal scientists who

have expertise in relevant

scientific disciplines and

current research areas.

I. Overview (continued)

10

The second level of review is

performed by Institute and Center

(IC) National Advisory Councils or

Boards. Councils are composed

of both scientific and public

representatives chosen for their

expertise, interest, or activity in

matters related to health and

disease.

I. Overview (continued)

11

Only applications that are favorably recommended

by both the SRG and the Advisory Council may be

recommended for funding.

http://public.csr.nih.gov/Pages/default.aspx

II. First Level of Review – SRG at the Center for Scientific Review

12

http://www.niaid.nih.gov/Pages/default.aspx

II. First Level of Review – SRG at the Funding Institute/Center

13

http://grants.nih.gov/grants/guide/pa-files/PA-11-260.html

II. First Level of Review – Locus of Review Specified in the FOA

14

https://www.federalregister.gov/

II. First Level of Review – All Reviews Announced in the Federal Register

15

II. First Level of Review – Scientific Review Groups

16

Initial peer review meetings are administered by either the

Center for Scientific Review (CSR) or another NIH IC.

The focus of review is specified in the Funding Opportunity

Announcement (FOA).

Peer review meetings are announced in the Federal

Register.

The meetings are closed to the public, although some

meetings may have an open session; the Federal Register

provides the details of each meeting.

II. A. Peer Review Roles

17

Scientific Review Officer (SRO)

Chairperson

Reviewers

Other NIH Staff

II A. Peer Review Roles - SRO

18

II A. Peer Review Roles - SRO

Scientific Review Officer:

Each SRG is led by a Scientific Review Officer (SRO).

The SRO is an extramural staff scientist and the

Designated Federal Official responsible for ensuring that

each application receives an objective and fair initial peer

review, and that all applicable laws, regulations, and

policies are followed.

19

II A. Peer Review Roles - SRO (cont.)

SROs:

Analyze the content and check for completeness.

Document and manage conflicts of interest.

Recruit qualified reviewers.

Assign applications to reviewers.

Attend and oversee administrative and regulatory aspects

of peer review meetings.

Prepare summary statements.

20

II A. SRO manages Conflict of Interest

21

II A. SRO manages Conflict of Interest

http://grants.nih.gov/grants/guide/notice-files/NOT-OD-11-120.html

Conflict of Interest (COI):

COI in scientific peer review exists when a reviewer has an

interest in an application that is likely to bias his or her

evaluation of it. A reviewer who has a real or apparent COI

with an application, as defined in 42 CFR Part 52h, may not

participate in its review unless a deviation is granted by the

NIH Deputy Director for Extramural Research.

22

II A. SRO manages the appearance of Conflict of Interest

http://grants.nih.gov/grants/guide/notice-files/NOT-OD-11-120.html

Conflict of Interest (COI) - Apparent:

The appearance of COI occurs when a reviewer or close

relative or professional associate of the reviewer has a

financial or other interest in an application that is known to

the reviewer or the government official managing the review,

and this circumstance would cause a reasonable person to

question the reviewer's impartiality if he or she were to

participate in the review.

23

II A. SRO recruits and assigns reviewers

24

II A. SRO recruits and assigns reviewers

25

SROs recruit qualified reviewers based on scientific and

technical qualifications and other considerations, including:

► Authority in their scientific field

► Dedication to high quality, fair, and objective reviews

► Ability to work collegially in a group setting

► Experience in research grant review

► Balanced representation

II A. Peer Review Roles - Chairperson

26

II A. Peer Review Roles - Chairperson

Chair:

Serves as moderator of the discussion of scientific and

technical merit of the applications under review.

Is also a peer reviewer for the meeting.

27

II A. Peer Review Roles - Reviewers

28

II A. Peer Review Roles - Reviewers

Reviewers:

Declare Conflicts of Interest.

Receive access to the grant applications approximately six

weeks prior to the peer review meeting.

Prepare a written critique for each application assigned.

Assign a numerical score to each scored review criterion.

Make recommendations concerning the scientific and

technical merit of applications under review.

Make recommendations concerning additional review criteria,

including protections for human subjects.

Make recommendations concerning budget requests.

29

II A. Peer Review Roles – Other NIH Staff

30

Program Officer (PO)

Grants Management Specialist (GMS)

Support Staff

II A. Peer Review Roles – Other NIH Staff

Other NIH Staff

Federal officials who have need-to-know or pertinent

related responsibilities are permitted to attend closed

review meetings.

NIH IC or other federal staff members wishing to attend an

SRG meeting must have advance approval from the

responsible SRO. These individuals may provide

programmatic or grants management input at the SRO's

discretion.

31

II A. Peer Review Meeting Overview

32

II A. Peer Review Meeting Overview

Peer Review Meeting Procedures:

Applications are reviewed based on established review

criteria.

Assigned reviewers summarize their prepared critiques for

the group.

An open discussion follows.

Final scoring of overall impact scores is conducted by

private ballot.

33

II B. Peer Review Criteria and Considerations

Review Criteria for Grants and Cooperative Agreements:

Overall Impact

► Scored Review Criteria

► Additional Review Criteria

Additional Review Considerations

34

II B. Peer Review Criteria and Considerations

35

Scored Review Criteria:

Significance

Investigator(s)

Innovation

Approach

Environment

II B. Peer Review Criteria and Considerations

Additional Review Criteria:

Protections for Human Subjects

Inclusion of Women, Minorities, and Children

Vertebrate Animals

Biohazards

Resubmission

Renewal

Revision

36

II B. Peer Review Criteria and Considerations

Additional Review Considerations:

Applications from Foreign Organizations

Select Agent

Resource Sharing Plans

Budget and Period Support

37

The scoring system utilizes a 9-point rating scale

(1 = exceptional; 9 = poor).

Score Descriptor Additional Guidance on Strengths/Weaknesses

1 Exceptional Exceptionally strong with essentially no weaknesses

2 Outstanding Extremely strong with negligible weaknesses

3 Excellent Very strong with only some minor weaknesses

4 Etc…

38

II C. Scoring

II. C. Peer Review Scoring Chart

Impact Score Descriptor Additional Guidance on

Strengths/Weaknesses

High

1 Exceptional Exceptionally strong with essentially no

weaknesses

2 Outstanding Extremely strong with negligible

weaknesses

3 Excellent Very strong with only some minor

weaknesses

Moderate

4 Very Good Strong but with numerous minor

weaknesses

5 Good Strong but with at least one moderate

weakness

6 Satisfactory Some strengths but also some

moderate weaknesses

Low

7 Fair Some strengths but with at least one

major weakness

8 Marginal A few strengths and a few major

weaknesses

9 Poor Very few strengths and numerous major

weaknesses Minor: An easily addressable weakness that does not substantially lessen the impact of the project

Moderate: A weakness that lessens the impact of the project

Major: A weakness that severely limits the impact of the project

Non-numeric score options: NR = Not Recommended for Further Consideration

DF = Deferred, AB – Abstention, CF = Conflict, NP = Not Present, ND = Not Discussed

http://enhancing-peer-review.nih.gov/scoring&reviewchanges.html

40

II C. Scoring (cont.)

Before the SRG meeting, each reviewer and discussant

assigned to an application gives a separate score for each

of the five review criteria (i.e., Significance, Investigator(s),

Innovation, Approach, and Environment).

For all applications, even those not discussed by the full

committee, the individual scores of the assigned reviewers

and discussant(s) for these criteria are reported to the

applicant.

41

III C. Scoring (cont.)

In addition, each reviewer and discussant assigned to an

application gives a preliminary overall impact score for that

application.

The preliminary scores are used to determine which

applications will be discussed in full.

42

II C. Scoring (cont.)

For each application that is discussed at the meeting, a

final impact score is given by each eligible committee

member (without COI) including the assigned reviewers.

Each member's score reflects his/her evaluation of the

overall impact that the project is likely to have on the

research field(s) involved, rather than being a calculation

of the reviewer's scores for each criterion.

Following the discussion, all reviewers rate the overall impact from 1 to 9.

Scores are averaged and rounded to create the overall impact score.

Example: final overall impact score calculation

(1+1+1+2+2+2+3+3+3)/9 * 10 = 20

43

II C. Scoring (cont.)

44

II C. Scoring (cont.)

The final overall impact scores range from 10

(high impact) through 90 (low impact).

The final overall impact score is reported on the

summary statement.

Numerical impact scores are not reported for

applications that are not discussed (ND), which

may be reported as ++ on the face page of the

summary statement.

45

II C. Scoring - Percentile

SRG #1

SRG #3

SRG #2

SRG #4 SRG #5

1.Percentiles are determined by matching an application's overall impact

score against a table of relative rankings containing all scores of

applications assigned to a study section during the three last review

cycles.

NIH calculates percentiles using the following formula:

Percentile = 100/Number of Applications x (relative rank minus 0.5)

(The 0.5 percent is a standard mathematical procedure used for rounding.)

2.These numbers are then rounded up, e.g., a percentile of 10.1

becomes 11, to create a whole number percentile ranging from 1 to 99.

NIH includes not discussed applications in the percentile calculation.

http://www.niaid.nih.gov/researchfunding/grant/strategy/pages/7payline.aspx#c

46

II C. Scoring - Percentile

Percentile Rank:

The percentile rank is based on a ranking of the impact

scores assigned by a peer review committee.

The percentile rank is normally calculated by ordering the

impact score of a particular application against the impact

scores of all applications reviewed in the current and the

preceding two review rounds.

An application that was ranked in the 5th percentile is

considered more meritorious than 95% of the applications

reviewed by that committee.

47

II C. Scoring – Percentile (cont.)

Summary Statement (SS):

The Summary Statement is a combination of the reviewers'

written comments and the SRO's summary of the members'

discussion during the study section meeting.

It includes the recommendations of the study section, a

recommended budget, and administrative notes of special

considerations.

48

II D. Summary Statement

http://www.niaid.nih.gov/researchfunding/grant/Documents/ParrishSS.pdf

II D. Summary Statement (cont.)

49

II D. Summary Statement - Face Page

II D. Summary Statement – At the top

II D. Summary Statement – individual critiques

II D. Summary Statement – Additional Review Criteria

II D. Summary Statement

– Additional Review Criteria

55

II D. Appeals

An appeal… describes a flaw in the review

process for a particular application. It must

display concurrence of the Authorized

Organization Representative (AOR).

56

II D. Appeals

The letter 1) describes a flaw(s) in the review process…, 2)

explains the reasons for the appeal, and 3) is based on one or

more of the following issues…:

Evidence of bias on the part of one or more peer reviewers

COI, on the part of one or more non-federal peer reviewers

Lack of appropriate expertise within the SRG

Factual error(s) made by one or more reviewers that could

have altered the outcome of review substantially

57

II D. Appeals

II D. Appeals (cont.)

The Council may concur:

with the appeal, and recommend that the application be

re-reviewed.

with the SRG’s recommendation and deny the appeal.

58

The Advisory Council/Board of the potential awarding IC

performs the second level of review. Advisory Councils/Boards

are composed of scientists from the extramural research

community and public representatives (NIH Federal Advisory

Committee Information). Members are chosen by the respective

IC and are approved by the Department of Health and Human

Services. For certain committees, members are appointed by the

President of the United States.

59

III. Second Level of Review - Advisory Council

60

III. Recommendation Process

NIH program staff members examine applications, their

overall impact scores, percentile rankings and their summary

statements and consider these against the IC's needs.

Program staff provide a grant-funding plan to the Advisory

Board/Council.

The Advisory Board/Council also considers the IC’s goals and

needs and advises the IC director.

The IC director makes final funding decisions based on staff

and Advisory Council/Board advice.

61

III. Recommendation Process

Fundable Score - Next Steps?

Some of the ICs publish paylines as part of their

funding strategies to guide applicants on their

likelihood of receiving funding. Application scores can

only be compared against the payline for the fiscal

year when the application will be considered for

funding, which is not necessarily the year when it was

submitted. There may be a delay of several months to

determine paylines at the beginning of fiscal years.

62

IV. Post-Review - Funded

http://grants.nih.gov/grants/next_steps.htm

Fundable Score - Next Steps?

If the application is assigned to an IC that does not announce

a payline, the Program Officer listed at the top of the summary

statement may be able to provide guidance on the likelihood of

funding. After the Advisory Council meeting, if an application

results in an award, the applicant will be working closely with

the program officer of the funding Institute or Center on

scientific and programmatic matters and a Grants

Management Officer on budgetary or administrative issues.

The Grants Management Specialist will contact the applicant

to collect information needed to prepare the award.

63

IV. Post-Review - Funded

http://grants.nih.gov/grants/next_steps.htm

Not Funded - Next Steps?

The NIH receives thousands of applications for each application

receipt round. Funding on the first attempt is difficult, but not

impossible. If an application does not result in funding, there

may be an opportunity to respond to the reviewers’ comments

and resubmit the application. Applicants just receiving their

summary statements, should contact the NIH Program Officer

listed at the top of the summary statement to discuss the

review. Also consult the Next Steps page (listed below) for

additional information.

64 http://grants.nih.gov/grants/next_steps.htm

IV. Post-Review – Not Funded

V. Resources

NIH Office of Extramural Research

► http://grants.nih.gov/grants/peer_review_process.htm#Overv

iew

NIH Center for Scientific Review

► http://public.csr.nih.gov/ApplicantResources/Pages/default.a

spx

NIH NIAID Peer Review

► http://www.niaid.nih.gov/researchfunding/grant/pages/review.

aspx

65

15th Annual Upstate New York Immunology Conference: Peer Review at NIH

Thank you!

For More Information

B. Duane Price, Ph.D.

Senior Scientific Review Officer

Immunology Review Branch

Scientific Review Program

National Institute of Allergy and Infectious Diseases

National Institute of Health

Department of Health and Human Services

6700-B Rockledge Drive

Rm. 3139, MSC 7616

Bethesda, MD 20892-7616

Phone:(301) 451-2592 Fax: (301) 480-2408

pricebd@niaid.nih.gov

15th Annual Upstate New York Immunology Conference: Peer Review at NIH

Mock Peer Review

Great Science

CSR assigns applications

to study section (review)

& NIH Institute (funding)

3- 4 reviewers assigned to read and write critiques for each application

Summary Statement and

Priority Score

transmitted to applicant

(Commons) and NIH

extramural staff

Researcher writes and Institution submits application to NIH

Peer Review

Study section composed of 20-30 reviewers – review/discuss applications

*

Mock Peer Review

71