Mary Ann Guadagno, PhD Senior Scientific Review Officer CSR Office of the Director Review Issues –...
-
Upload
leonard-mccarthy -
Category
Documents
-
view
227 -
download
0
Transcript of Mary Ann Guadagno, PhD Senior Scientific Review Officer CSR Office of the Director Review Issues –...
Mary Ann Guadagno, PhD
Senior Scientific Review Officer CSR Office of the Director
Review Issues – CSR Surveys
• Peer Review is Fundamental to the NIH Mission
• CSR Goal – to ensure that grant applications submitted to NIH are evaluated on the basis of a process that is fair, equitable, timely, and free of bias.
• CSR Review Issues – Are current CSR best practices optimal for achieving the mission?
CSR Peer Review Issues
• Receives all NIH applications from Applicants
• Refers them to NIH Institutes/Centers and to scientific review groups
• SROs manage the peer review process
• Reviewers review grant applications for scientific merit
• NIH Program Officers make funding recommendations
CSR is the Focal Point for Initial Review at NIH
Are Current CSR Best Practices Optimal for Achieving the Mission?
• CSR Leadership Program Surveys• CSR New Chair Training SurveysOther Studies
• Focus Groups – SROs, POs, Reviewers, Applicants• Personal Interviews – NIH Institute and Center Directors
Evaluation of the Peer Review Process from Stakeholders
• Pilot 1 - January 2015Quick Feedback
for CSR from Program Officers
• Pilot 1 - February/March 2014• Pilot 2 - May/June 2014• Pilot 3 - September/October 2014• Pilot 4 - March/April 2015
Quick Feedback for CSR from
Reviewers
Evaluating the CSR Peer Review Process
To assess the utility of asking Reviewers and NIH Program Officers about their CSR study section meeting experience (*queried both Reviewers and POs):
–Program Officer Experience–*Quality of Prioritization–*Collective Expertise–*Assignment of Applications to Reviewers–*Quality of Discussion
Objectives
–Alternative Meeting Formats–Meeting Issues–Overall Quality of Review–General Comments
Objectives continued …
REVIEWER
SATISFACTION WITH SELECT ELEMENTS OF PEER REVIEW
• S1 - The Panel was able to prioritize applications according to their impact/scientific merit.
• S2 – The roster of reviewers was an appropriate assembly of scientific expertise for the set of applications in the meeting.
• S3 – Assignment of applications to reviewers made appropriate use of their broad expertise.
• S4 – The nature of the scientific discussions supported the ability of the panel to evaluate the applications being reviewed.
• General Comments – In addition to the answers you provided in this questionnaire, please add any other comments in the text box below.
Four Agreement Statements and Comments
NIH PROGRAM OFFICER
SATISFACTION WITH SELECT ELEMENTS OF PEER REVIEW
–If Program Officer did not observe or listen-into a CSR Study Section - 2 text boxes:
• Why not?• What could CSR do to help facilitate your ability to follow
our SRGs?
–If Not Applicable (N/A) – not responsible for applications reviewed that round.
• End of interview. Thank you.
Skip Patterns on Question 1
• S1 - The panels were able to appropriately score my programs’ applications.
• S2 – The rosters of reviewers represent strong scientific expertise for the set of applications in the meetings.
• S3 – Assignments of applications to reviewers made appropriate use of their broad expertise.
• S4 – The scientific discussions supported my ability to make programmatic decisions.
• S5– I am comfortable having my own programs’ applications reviewed using:–Video Assisted Meeting (VAM) format–Internet Assisted Meeting (IAM) format
Program Officer Agreement Statements and Comments
• Of the meetings you followed, how many CSR study sections were…?–Face-to-face meetings in person–Face-to-face meetings by telephone–Internet Assisted Meetings (IAM)–Video Assisted Meetings (VAM)–Teleconference
• How many years have you been an NIH Program Officer?
• Meeting Issues – In the meetings you followed, did you have any concerns about the following issues? (please check all that apply)
–Internet connectivity–Telephone–Microphone–Real Time Meeting Status Tool–Travel to the meeting was inconvenient
• Overall Quality of Review – In your experience, the best Scientific Review Groups are (list up to 3 – either CSR or IC)?
• General Comments – In addition to the answers you provided in this questionnaire, please add any other comments in the text box below.
Additional Questions and Comments
Face to Face by Teleconference
Face to Face in Person
Teleconference
IAM
VAM
0 100 200 300 400 500 600 700 800 900 1000
Program Officer Survey: Meeting Attendance by Format
January 2015 Council
1 2 3 4+
Number of Meetings
n= 720 (50%)
n= 245 (17%)
n= 243 (17%)
n= 192 (13%)
n= 29 (2%)
Meeting Format
Overall NIH response rate = 38% (n=916): total number of meetings attended = 1,429.
VAM not attended n= 254
VAM attended n= 20
IAM not attended n= 175
IAM attended n= 122
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%Program Officer Survey: Satisfaction with VAM and IAM Formats
1 2 3 4 5 6 7Likert type scale where 1=very strongly agree to 7=very strongly disagree. Overall NIH response rate = 38% (n=916).
REVIEWERS VS. PROGRAM OFFICERS
Quality
of P
riorit
izatio
n
Colle
ctive
Exp
ertis
e
Assig
nmen
ts
Quality
of D
iscus
sion
Revie
w of P
rogr
am A
pps
usin
g VAM
Revie
w of P
rogr
am A
pps
usin
g IA
M0%
20%
40%
60%
80%
100%
87.6% 89.1% 87.1% 87.5%
86.7% 88.7% 88.3% 87.1%
81.7%86.5% 83.7% 83.1%
44% 47% 49% 45%
22% 20%
Pilot1, n=248 Pilot2, n=256 Pilot3, n=2,990 PO Survey, n=308
Per
cent
Str
ongl
y A
gree
/Agr
eeOverall CSR Quick Feedback Favorable Responses
SATISFACTION VARIES ACROSS NIH INSTITUTES AND CENTERS
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Program Officer Survey: Response Rates
Overall NIH response rate = 38% (n=916).
1
2
3
4
5
6
7
Program Officer Survey: IC Rank by Total of Q1-Q4 Averages
Likert type scale where 1=very strongly agree to 7=very strongly disagree. Overall NIH response rate = 38% (n=916). (n=916
MEETING CHALLENGES REPORTED BY PROGRAM OFFICERS
0%
10%
20%
30%
40%
Program Officer Survey: Meeting Challenges January 2015 Council
n= 209 (35%)
n= 170 (29%)
n= 103 (17%)
n= 76 (13%)
n= 46 (6%)
Overall NIH response rate = 38% (n=916)
CONTENT ANALYSIS
CHALLENGES FROM GENERAL COMMENTS
• Positive - SROs
• Neutral - RTMS Tool + Quality of Peer Review
• Challenges• Microphone Issues• Telephone Issues• Roster Quality• IAM Meeting Format• Scoring• Meeting Logistics
Themes from General Comments
• Identification of areas where CSR is meeting or exceeding customer expectations as well as areas that present challenges, using survey research to guide us.
• Strengths and limitations of methodology –1 survey to all POs at end of council round–Response rate
• Next steps–CSR Director begins discussions with IC Directors.–Share results with CSR staff, PLC, RPC, EPMC.–Begin addressing actionable items.–Continue to survey over time to track changes.
What Do We Hope to Learn?
Suggestions or Questions?
Thank you!