Using an Internal GME House Staff Survey as An Early ......Introduction of the Stanford Housestaff...
Transcript of Using an Internal GME House Staff Survey as An Early ......Introduction of the Stanford Housestaff...
Using an Internal GME House Staff Survey as
An Early Warning System for the ACGME Resident SurveyApril 30, 2018
Nancy Piro, PhD
Sr Program Manager/Education Specialist, Stanford GME
Ann Dohn, MA
GME Director & DIO Stanford GME
❖ Nancy Piro, PhD❖ Ann Dohn, MA
Neither of the above speakers have any conflicts of interest to report.
Agenda
I. Background - Stanford GME Housestaff Survey
II. Analysis of the Construct of the Stanford GME Housestaff Survey
III. Predictions of ACGME Resident Survey Results based on the Internal GME Housestaff Survey
IV. Group Discussion
V. Stanford Program’s Scorecard System
VI. Discussion & Questions
Background: Graduate Medical Education (GME) House Staff Survey
History
Purpose
Questions & Framework
Face Validity
Delivery
Analysis & Reporting
Background: Stanford Housestaff Survey – Historical Perspective
An institutional-wide internal survey, with IRB approval, used since 2008 which is distributed via email to all housestaff officers.
GME Office delivers the survey annually in mid November to all Stanford programs both ACGME and non-ACGME.
All responses are confidential and anonymous. Programs with >3 responses receive aggregated reports.
Background: Stanford Housestaff Survey – Purpose
Initially designed to gather resident and fellow opinions on the educational quality of their training programs, and to find out about other GME-related issues i.e., how GME needed to function more effectively in meeting our trainees’ needs. “Voice of the Resident”
As the survey has evolved, we saw the potential for it to serve as an Early Warning System for programs to detect potential concerns that could be understood/clarified before the ACGME Resident Survey.
Provides programs with aggregated data for review in their Annual Program Evaluations (APEs) and the Annual Institutional Review (AIR)
Background: Stanford Housestaff Survey – Questions & Framework
Survey questions parallel the six (6) main domains from the ACGME Resident Survey: Work Hours, Educational Content, Faculty, Evaluation, Resources, Patient Safety/Team Work.
Questions also focus on the institutional areas needing improvement delineated in prior Clinical Learning Environment Review (CLER) site visit reports.
Our focus is on critical questions – Total Survey Time ~ 5-7 minutes maximum.
Additional Questions:
−Added in alternate years
−For research and program quality control purpose
−Up to 5 questions related to GME issues, such as wellbeing, mentorship, etc.
Background: Stanford Housestaff Survey – Questions
Core Quantitative Questions (1-6 Likert scale):
1. Overall, how satisfied are you with the training you have received in your current program at Stanford?
2. I would recommend my training program here at Stanford to others.
3. Faculty in my program are successful teachers.
4. Faculty in my program spend sufficient time supervising the residents or fellows in the program.
5. Faculty in my program encourage me to ask questions on a regular basis.
6. Faculty in my program encourage me to be open/honest with them.
7. I have an adequate amount of "protected time" to focus on my educational needs.
8. I think my program appropriately balances the need to fulfill service obligations to the hospital with clinical education.
9. My program is organized to meet my educational needs.
10. My overall patient load (the quantity of patients that I see) is appropriate.
11. The variety of patients I see is sufficient for meeting my educational needs.
12. In your program, have you had (will you have) the opportunity to participate in any projects related to Quality Improvement (QI)? (Yes/No)
Background: Stanford Housestaff Survey – Questions Cont’d
Core Qualitative Questions:
1. Please explain why you disagree with "I would recommend my training program here at Stanford to others".
2. What did we miss? Are there any other areas of concern or suggestions (GME Office, orientation, call rooms, security at training sites, etc.)?
Question Response Depiction in Reports for Programs
GME House Staff Question Coding Scheme (1)
GME House Staff Question Coding Scheme (2)
GME House Staff Survey Process
Questions Determined
Review by IRB
Survey Distributed
Data Analyzed
Research Begins
Reports
& Scorecards Developed & Shared
Background: Stanford Housestaff Survey – Face Validity
The Stanford GME Housestaff Survey was reviewed, revised and approved by program directors from core residency programs.
GME House Staff emails/survey cover letters
IRB Approval Letter
Introduction of the Stanford Housestaff Survey – Delivery
The GME Office sends an email with the cover letter containing a hyperlink to the survey to all Housestaff Officers.
Participants are informed that all of their responses are totally anonymous and do not include IP addresses, locations of completion, or any other identification information.
Introduction of the Stanford Housestaff Survey – Analysis & Reporting
Responses are aggregated by programs for reporting
Only programs with at least 4 responses are reported
1-6 Likert scales are designated as Red, Yellow or Green
Overall, how satisfied are you with the training you have received in your program?
Extremely Unsatisfied
Very Unsatisfied
Unsatisfied
SatisfiedVery Satisfied
Extremely Satisfied
I would recommend my training program here to others.
Disagree Strongly
Disagree Moderately
Disagree Slightly
Agree
Slightly
Agree Moderately
Agree Strongly
I have an adequate amount of "protected time" to focus on my educational needs.
I think my program appropriately balances the need to fulfill service obligations to the hospital with
clinical education.
My program is organized to meet my educational needs.
My overall patient load (the quantity of patients I see) is appropriate.
The variety of patients I see is sufficient for meeting my educational needs.
Faculty in my program are successful teachers.
Faculty in my program spend sufficient time supervising the residents or fellows in the program.
Faculty in my program encourage me to ask questions on a regular basis.
Faculty in my program encourage me to be open and honest with them.
Have you had an opportunity to participate in any projects related to Quality Improvement (QI) during
your current training at Stanford? No Yes
Introduction of the Stanford Housestaff Survey – Analysis & Reporting
Program data is compared to the institutional average for each quantitative question.
Stacked/Colored bars are utilized to visually present the data for easy interpretation.
− Solid Fill: Program Data
−Patterned Fill: Institutional Data
II. Analysis of the Construct of the Stanford GME Housestaff Survey
Hypothesized construction of the Stanford GME Housestaff Survey
1. Overall, how satisfied are you with the training you have received in your current program at Stanford?
2. I would recommend my training program here at Stanford to others.
3. Faculty in my program are successful teachers.
4. Faculty in my program spend sufficient time supervising the residents or fellows in the program.
5. Faculty in my program encourage me to ask questions on a regular basis.
6. Faculty in my program encourage me to be open/honest with them.
7. I have an adequate amount of "protected time" to focus on my educational needs.
8. I think my program appropriately balances the need to fulfill service obligations to the hospital with clinical education.
9. My program is organized to meet my educational needs.
10. My overall patient load (the quantity of patients that I see) is appropriate.
11. The variety of patients I see is sufficient for meeting my educational needs.
12. In your program, have you had (will you have) the opportunity to participate in any projects related to Quality Improvement (QI)?
Question: Does the Survey test what it was designed to test? Can the survey questions be grouped into different factors as we hypothesized above?
Overall
Faculty
Resources
Educational Content
Clinical Work
QI
Construct Analysis – Dataset
Surveys: Stanford GME Housestaff Survey (12 questions)
Years: 2014-15, 2015-16 and 2016-17
Programs: 29 (21 Residency + 8 Fellowships)
− Residency: Anesthesiology, Dermatology, Emergency medicine, Internal medicine, Neurological surgery, Neurology, Obstetrics and gynecology, Ophthalmology, Orthopaedic surgery, Otolaryngology, Pathology-anatomic and clinical, Pediatrics, Plastic Surgery – Integrated, Physical medicine and rehabilitation, Psychiatry, Child and adolescent psychiatry, Radiology-diagnostic, Radiation oncology, Surgery, Thoracic surgery – integrated, Urology
− Fellowship: Cardiovascular disease, Gastroenterology, Hematology, Nephrology, Pulmonary disease and critical care medicine, Pediatric critical care medicine, Pediatric cardiology, Pediatric hematology/oncology
Response Metrics:
−Percentage of respondents who chose “Agree Strongly/Moderately” or “Extremely/Very Satisfied” or “Yes” (All the Greens)
Construct Validity – Factor Analysis
A method to describe variability among observed and correlated variables in terms of a potentially lower number of underlying variables (called Factors).
A way to reduce number of variables
Factors GME Survey Question Factor 1 Factor 2 Factor 3 Factor 4 Factor 5
Faculty in my program are successful
teachers.0.907 0.634 0.737 0.478
Faculty in my program spend sufficient time
supervising the residents or fellows in the
program.
0.940 0.531 0.550 0.382
The variety of patients I see is sufficient for
meeting my educational needs.0.693 0.434 0.408 0.486
I would recommend my training program
here to others.0.860 0.755 0.800 0.374
I have an adequate amount of "protected
time" to focus on my educational needs.0.438 0.784 0.327 0.541
I think my program appropriately balances
the need to fulfill service obligations to the
hospital with clinical education.
0.590 0.958 0.359 0.685
My program is organized to meet my
educational needs.0.583 0.874 0.364 0.552 0.446
Overall, how satisfied are you with the
training you have received in your program?0.724 0.784 0.700 0.500
Clinical WorkMy overall patient load (the quantity of
patients I see) is appropriate.0.468 0.553 0.940 0.527
Faculty in my program encourage me to ask
questions on a regular basis.0.854 0.678 0.315 0.941 0.359
Faculty in my program encourage me to be
open and honest with them.0.735 0.668 0.432 0.933
QIIn your program, have you had the
opportunity to participate in any projects
related to Quality Improvement (QI)?
0.995
Resources
Faculty
Educational
Content
Construct Analysis – Factor Analysis Cont’d
Extraction Method: Principal Axis Factoring.
Rotation Method: Promax with Kaiser Normalization.
Construct Validity – Conclusions
The Stanford Housestaff Survey was proven to test 5 domains of program quality
Grouping of the questions are in general as we hypothesized, with 3 exceptions –
− Variety of patients factored in Faculty as opposed to Clinical Work as we originally thought
− The Overall Group (- overall, how satisfied are you and would you recommend) factored with: Educational Content and Faculty
III. Predictions of ACGME Resident Survey Results based on the Internal GME
Housestaff Survey
Stanford GME Housestaff Survey was designed to serve as an Early Warning System for programs to detect areas where there were concerns or confusion ahead of the ACGME Resident Survey.
Early Warning System
Research Question: Does the Stanford GME Housestaff Survey in fact predict/correlate with the results of the ACGME Resident Survey?
Prediction of some ACGME Survey Results– Datasets
Surveys: GME House Staff Survey (12 questions), ACGME Resident Survey (41 questions)
Years: 2014-15, 2015-16 and 2016-17
Programs: 29 (21 residency + 8 fellowships)
− Residencies: Anesthesiology, Dermatology, Emergency medicine, Internal medicine, Neurosurgery, Neurology, OB/GYN, Ophthalmology, Orthopedic surgery, Otolaryngology, Pathology-AP/CP, Pediatrics, Plastic Surgery, Physical medicine and rehabilitation, Psychiatry, Child and adolescent psychiatry, Radiology-diagnostic, Radiation oncology, Surgery, Thoracic surgery, and Urology
− Fellowships: Cardiology, Gastroenterology, Hematology, Nephrology, Pulmonary disease and Critical Care Medicine, Pediatric Critical Care medicine, Pediatric cardiology, and Peds Hem/Onc
Metrics:
−GME Housestaff Survey: Percentage of respondents who chose “Agree Strongly/Moderately” or “Extremely/Very Satisfied” or “Yes” (All the Greens)
−ACGME Resident Survey: % Program Compliance, Program Means
Prediction of some ACGME Survey Results– Overall Correlation
Overall average of % program compliance of GME Housestaff Survey positively and significantly correlated with the overall average of % program compliance in the ACGME Resident Survey
Overall, the number of % program non-compliant in the GME Housestaff Survey positively and significantly correlated with the number of % program non-compliant in the ACGME Resident Survey
GME Survey (12 Questions) ACGME Resident Survey (41 Questions)
Average of % Program Compliance of All Qs
Average of Program Mean of All Qs
Number of Non-Compliance (<80%) Number of Non-Compliance (<80%)
Average of % Program Compliance of
All Qs
Correlation
r=.349, p < .01
r=.422, p < .01
r=.434, p < .01
Prediction of the ACGME Survey – Factor Correlation
80 hours1 day free in 7In-house call every 3rd nightNight float no more than 6 nights8 hours between duty periods (differs by level of training )Continuous hours scheduled (differs by level of training )Sufficient supervisionAppropriate level of supervisionSufficient instructionFaculty and staff interested in residency educationFaculty and staff create environment of inquiryAble to access evaluationsOpportunity to evaluate faculty membersSatisfied that evaluations of faculty are confidentialOpportunity to evaluate programSatisfied that evaluations of program are confidentialSatisfied that program uses evaluations to improveSatisfied with feedback after assignmentsProvided goals and objectives for assignmentsInstructed how to manage fatigueSatisfied with opportunities for scholarly activitiesAppropriate balance for educationEducation (not) compromised by service obligationsSupervisors delegate appropriatelyProvided data about practice habitsSee patients across variety of settingsAccess to reference materialsUse electronic medical records in hospital*Use electronic medical records in ambulatory setting*Electronic medical records integrated across settings*Electronic medical records effectiveProvided a way to transition care when fatiguedSatisfied with process to deal with problems and concernsEducation (not) compromised by other traineesResidents can raise concerns without fearTell patients of respective roles of faculty and residentsCulture reinforces patient safety responsibilityParticipated in quality improvementInformation (not) lost during shift changes or patient transfersWork in interprofessional teamsEffectively work in interprofessional teams
ACGME Survey Factors & Questions
Duty Hours
Faculty
Evaluation
Educational
Content
Resources
Patient
Safety/Team
work
GME Survey Question
Faculty in my program are successful
teachers.
Faculty in my program spend sufficient time
supervising the residents or fellows in the
program.
The variety of patients I see is sufficient for
meeting my educational needs.
I would recommend my training program
here to others.
I have an adequate amount of "protected
time" to focus on my educational needs.
I think my program appropriately balances
the need to fulfill service obligations to the
hospital with clinical education.
My program is organized to meet my
educational needs.
Overall, how satisfied are you with the
training you have received in your program?
My overall patient load (the quantity of
patients I see) is appropriate.
Faculty in my program encourage me to ask
questions on a regular basis.
Faculty in my program encourage me to be
open and honest with them.
In your program, have you had the
opportunity to participate in any projects
related to Quality Improvement (QI)?
Prediction of the ACGME Survey – Factor Correlation
80 hours1 day free in 7In-house call every 3rd nightNight float no more than 6 nights8 hours between duty periods (differs by level of training )Continuous hours scheduled (differs by level of training )Sufficient supervisionAppropriate level of supervisionSufficient instructionFaculty and staff interested in residency educationFaculty and staff create environment of inquiryAble to access evaluationsOpportunity to evaluate faculty membersSatisfied that evaluations of faculty are confidentialOpportunity to evaluate programSatisfied that evaluations of program are confidentialSatisfied that program uses evaluations to improveSatisfied with feedback after assignmentsProvided goals and objectives for assignmentsInstructed how to manage fatigueSatisfied with opportunities for scholarly activitiesAppropriate balance for educationEducation (not) compromised by service obligationsSupervisors delegate appropriatelyProvided data about practice habitsSee patients across variety of settingsAccess to reference materialsUse electronic medical records in hospital*Use electronic medical records in ambulatory setting*Electronic medical records integrated across settings*Electronic medical records effectiveProvided a way to transition care when fatiguedSatisfied with process to deal with problems and concernsEducation (not) compromised by other traineesResidents can raise concerns without fearTell patients of respective roles of faculty and residentsCulture reinforces patient safety responsibilityParticipated in quality improvementInformation (not) lost during shift changes or patient transfersWork in interprofessional teamsEffectively work in interprofessional teams
ACGME Survey Factors & Questions
Duty Hours
Faculty
Evaluation
Educational
Content
Resources
Patient
Safety/Team
work
1 2 3 4 5
0.907 0.634 0.737 0.478
0.940 0.531 0.550 0.382
0.693 0.434 0.408 0.486
0.860 0.755 0.800 0.374
0.438 0.784 0.327 0.541
0.590 0.958 0.359 0.685
0.583 0.874 0.364 0.552 0.446
0.724 0.784 0.700 0.500
0.468 0.553 0.940 0.527
0.854 0.678 0.315 0.941 0.359
0.735 0.668 0.432 0.933
0.995
GME Survey Question
Faculty in my program are successful
teachers.
Faculty in my program spend sufficient time
supervising the residents or fellows in the
program.
The variety of patients I see is sufficient for
meeting my educational needs.
I would recommend my training program
here to others.
I have an adequate amount of "protected
time" to focus on my educational needs.
I think my program appropriately balances
the need to fulfill service obligations to the
hospital with clinical education.
My program is organized to meet my
educational needs.
Overall, how satisfied are you with the
training you have received in your program?
My overall patient load (the quantity of
patients I see) is appropriate.
Faculty in my program encourage me to ask
questions on a regular basis.
Faculty in my program encourage me to be
open and honest with them.
In your program, have you had the
opportunity to participate in any projects
related to Quality Improvement (QI)?
"Satisfied with process to deal with problems and concerns" &
"Residents can raise concerns without fear" % Program
Compliancer=.169, p > .1
"Satisfied with process to deal with problems and concerns" &
"Residents can raise concerns without fear" Program Mean r=.216, p < .05
Number of Non-Compliance (<80%) Number of Non-Compliance (<80%) r=.288, p < .01
Average % of Program Compliance of
Resources
Prediction of the ACGME Survey – Factor Correlation Cont’d
Faculty (% Program Compliance) r=.434, p < .01
Faculty (Program Mean) r=.469, p < .01
Number of Non-Compliance (<80%) Number of Non-Compliance (<80%) r=.395, p < .01
Average of % Program Compliance of
Faculty
Educational Content (% Program Compliance) r=.286, p < .01
Educational Content (Program Mean) r=.267, p < .05
Number of Non-Compliance (<80%) Number of Non-Compliance (<80%) r=.352, p < .01
Average of % Program Compliance of
Educational Content
"Appropriate balance for education" & "Education (not)
compromised by service obligations" (% Program Compliance)r=.438, p < .01
"Appropriate balance for education" & "Education (not)
compromised by service obligations" (Program Mean)r=.385, p < .01
Number of Non-Compliance (<80%) Number of Non-Compliance (<80%) r=.353, p < .01
% Program Compliance of
"My overall patient load is appropriate"
GME Survey (12 Questions) ACGME Resident Survey ( 41 Questions) Correlation
Average of % Program Compliance of All Qs r=.349, p < .01
Average of Program Mean of All Qs r=.422, p < .01
Number of Non-Compliance (<80%) Number of Non-Compliance (<80%) r=.434, p < .01
Average of % Program Compliance of
All Qs
"Faculty and staff create environment of inquiry" % Program r=.275, p < .05
"Faculty and staff create environment of inquiry" Program r=.340, p < .01
Number of Non-Compliance (<80%) Number of Non-Compliance (<80%) r=.220, p < .05
Average % of Porgram Compliance of
Resources
"Participated in quality improvement" % Program Compliance r=.488, p < .01
"Participated in quality improvement" Program Mean r=.485, p < .01
Number of Non-Compliance (<80%) Number of Non-Compliance (<80%) r=.225, p < .05
% of Program Compliance of
QI
Prediction of the ACGME Survey – Conclusions
GME Housestaff Survey positively and significantly correlates with the ACGME Resident Survey, in terms of:
−Overall compliance scores
−Overall program means
−Each independent factor, except one correlation as noted on the previous slide
Average % of program compliance of Resources to the average of 2 related questions in the ACGME survey
−The number of non-compliant questions
GME House Staff Survey positively and significantly correlates with
the ACGME Resident Survey… so what can we do with this data?
Early warning system for Programs and the Institution / validation of each survey
Data for evidence-based change to improve program quality
Information/statistics for DIO to allocate resources
May signal need for Program Special Review
Data for Annual Program Evaluation (APE)and the Annual Institutional Review (AIR)
Agenda
I. Introduction of the Stanford GME Housestaff Survey
II. Analysis of the Construct of the Stanford GME Housestaff Survey
III. Prediction of the ACGME Resident Surveys by the Stanford GME Housestaff Survey
IV. Group Discussion
Group Discussion
1. Customizing an internal survey that is most relevant and useful for your institution?
2. How would you advise programs to make improvements based on the survey results?
Topics
I. Introduction of the Stanford GME Housestaff Survey
II. Analysis of the Construct of the Stanford GME Housestaff Survey
III. Prediction of the ACGME Resident Surveys by the Stanford GME Housestaff Survey
IV. Group Discussion
V. Stanford Program’s Scorecard System
Scorecard System – What Is it?
A visual presentation of a program’s educational quality using both external and internal data sources (simple spreadsheet can be used)
External: ACGME Resident Survey, Number of ACGME Citations, Board Pass Rate
Internal: GME Housestaff Survey, Program Evaluation by Resident and Faculty, MedHub Work Hour Report
Illustrated in Green and Red (conditional formatting may be used)
For both the institution and individual programs
Stanford Program Scorecard Example
Trend Analysis - Stanford ProgramSHC Balanced Report Card
Key MeasuresSOURCE INT/EXT
Data
Source2017-18
Sufficient Supervision
Survey
ACGME %-
COMPLIANT
No Data
Yet
Sufficient Instruction
Survey
ACGME %-
COMPLIANT
No Data
Yet
Faculty/Staff Create
Environment of Inquiry
Survey
ACGME %-
COMPLIANT
No Data
Yet
Satisfied with Process for
Problems and Concerns
Survey
ACGME %-
COMPLIANT
No Data
YetClimate Where Residents
Can Raise Concerns
Without Fear
Survey
ACGME %-
COMPLIANT
No Data
Yet
Overall Eval of the
Program
Survey
ACGME %-
COMPLIANT
No Data
Yet
Current Number of
ACGME Citations ACGME
Board Pass Rates ABMS
Overall Satisfaction with
ProgramRESIDENT GME-Survey
100%
Program Organized to
Meet Educational NeedsGME-Survey
93%
Service Over Education GME-Survey86%
Encouraged to Ask
Questions on a Regular
Basis
GME-Survey100%
Residents Can Be Open
and Honest with FacultyGME-Survey
100%
Residents Would
Recommend ProgramGME-Survey
100%
Faculty Overall Evaluation
ProgramFACULTY
Pgm Eval
Mean
Score/10
No Data
Yet
Resident Overall Program
Evaluation RESIDENT
Pgm Eval
Mean
Score/10
No Data
Yet
>80 Violations / AYMedHub
Duty Hr Rpt
No Data
Yet
# Unreviewed Duty Hr
Periods by PD / AY
MedHub
Detailed Rpt
No Data
Yet
RESIDENT
EXT
PROGRAM
INT
PROGRAM
SHC Balanced Report Card
Key MeasuresSOURCE INT/EXT
Data
Source2013-14 2014-15 2015-16 2016-17 2017-18
Sufficient Supervision
Survey
ACGME %-
COMPLIANT
95% 93% 88%100%
Pending
Sufficient Instruction
Survey
ACGME %-
COMPLIANT
77% 84% 76%97%
Pending
Faculty/Staff Create
Environment of Inquiry
Survey
ACGME %-
COMPLIANT
70% 81% 73%89%
Pending
Satisfied with Process for
Problems and Concerns
Survey
ACGME %-
COMPLIANT
68% 77% 73%97%
Pending
Climate Where Residents
Can Raise Concerns
Without Fear
Survey
ACGME %-
COMPLIANT
66% 72% 71%94%
Pending
Overall Eval of the
Program
Survey
ACGME %-
COMPLIANT
8.5 8.6 8.6 9.5 Pending
Total Number of ACGME
Citations (new) (resolved) ACGME4 4 0 0 Pending
Board Pass Rates ABMS2009-13:
87%
2010-14:
83%
2011-15:
86%
2012-16:
82%Pending
Overall Satisfaction with
ProgramGME-Survey 82% 90% 89%
92%91%
Program Organized to
Meet Educational NeedsGME-Survey 94% 85% 74%
89%91%
Service Over Education GME-Survey 94% 80% 80% 86% 71%
Encouraged to Ask
Questions on a Regular
Basis
GME-Survey 100% 90% 80%86%
86%
Residents Can Be Open
and Honest with FacultyGME-Survey 88% NA 91%
86%86%
Residents Would
Recommend ProgramGME-Survey 94% 95% 88%
97%91%
Resident Overall Program
Evaluation
Pgm Eval
Mean
Score/10
8.4 8.4 8.6 8.4 8.5
Faculty Overall Evaluation
ProgramFACULTY
Pgm Eval
Mean
Score/10
9.1 8.9 8.1 8.5 8.6
>80 Violations / AYMedHub
Duty Hr Rpt29 14 15 24 16
# Unreviewed Work Hr
Periods by PD / AY
MedHub
Detailed Rpt0 0 0 0 9
KEY STRENGTH WEAKNESS
5- Year Trend Analysis of AAA
RESIDENT
EXT
PROGRAM
RESIDENT
INT
PROGRAM
Program Trend Analysis– Example
5-year trend analysis
80% cut-off or 0 for citations and work hours
Program Scorecard / Program Trend Analysis
SHC Balanced Report
Card Key MeasuresSOURCE
INT or
EXT
Data
Source
2010-11 2011-12 2012-13 2013-14 2014-15 2015-16 2016-17 2017-18
Sufficient
SupervisionRESIDENT EXT
Survey
ACGME %-COMPLIANT93% 85% 93% 94% 94% 88% 94% No Data
Sufficient
InstructionRESIDENT EXT
Survey
ACGME %-COMPLIANT73% 77% 80% 94% 88% 75% 94% No Data
Faculty/Staff Create
Environment of
Inquiry
RESIDENT EXTSurvey
ACGME %-COMPLIANT87% 92% 93% 81% 82% 75% 94% No Data
Satisfied with
Process for
Problems and
Concerns
RESIDENT EXTSurvey
ACGME %-COMPLIANT73% 85% 87% 56% 71% 44% 83% No Data
Climate Where
Residents Can Raise
Concerns Without
Fear
RESIDENT EXTSurvey
ACGME %-COMPLIANT80% 100% 93% 56% 88% 81% 89% No Data
Overall Eval of the
ProgramRESIDENT EXT
Survey
ACGME %-COMPLIANT100% 100% 91% 85% 90% 85% 89% No Data
Total Number of
ACGME Citations
(new) (resolved)
PROGRAM EXTACGME
0 0 0 0 0 0 0No Data
Board Pass Rates PROGRAM EXT ABMS
56% 56% 56% 56% 100% 100% No Data No Data
Overall Satisfaction
with ProgramRESIDENT INT GME-Survey
90% 100% 83% 75% 100% 100% 100% 100%Program Organized
to Meet Educational
Needs
RESIDENT INT GME-Survey
100% 100% 92% 85% 100% 55% 94% 94%
Service Over
EducationRESIDENT INT GME-Survey
100% 100% 75% 80% 91% 55% 94% 94%Encouraged to Ask
Questions on a
Regular Basis
RESIDENT INT GME-Survey
100% 100% 100% 94% 100% 100% 94% 94%Residents Can Be
Open and Honest
with Faculty
RESIDENT INT GME-Survey
100% 100% 100% 94% No Data 86% 94% 94%Residents Would
Recommend
Program
RESIDENT INT GME-Survey
100% 100% 92% 90% 91% 100% 94% 100%
Faculty Overall
Evaluation ProgramFACULTY INT
Pgm Eval
Mean Score/10
8.21 9.69 7.88 9.13 8.82 8.88 9.40 No Data
Resident Overall
Program Evaluation RESIDENT INT
Pgm Eval
Mean Score/10
8.90 9.38 8.25 8.68 7.62 8.70 No Data
> 80 Violations / AY PROGRAM INT
MedHub
Institution Duty Hr Rpt
0% 0% 0% 0% 0% 0% 0% No Data# Unreviewed Duty
Hr Periods by PD /
AY
PROGRAM INT
MedHub
DetailedRpt
0% 0% 0% 0% 0% 0% 0% No Data
KEY STRENGTH WEAKNESS
Institution-Wide Program Scorecard Rolled-Up
A B C D E
SHC Balanced Report Card
Key MeasuresSOURCE
INT or
EXT
Data
Source 2 3 6 7 9
Sufficient Supervision RESIDENT EXTSurvey
ACGME %-
COMPLIANT 92% 100% 63% 100% 86%
Sufficient Instruction RESIDENT EXTSurvey
ACGME %-
COMPLIANT 84% 100% 38% 100% 75%
Appropriate Balance for
EducationRESIDENT EXT
Survey
ACGME %-
COMPLIANT 57% 100% 63% 94% 69%
Faculty/Staff Create
Environment of InquiryRESIDENT EXT
Survey
ACGME %-
COMPLIANT 71% 100% 63% 100% 64%
Satisfied with Process for
Problems and ConcernsRESIDENT EXT
Survey
ACGME %-
COMPLIANT 75% 33% 63% 100% 75%
Climate Where Residents
Can Raise Concerns Without
Fear
RESIDENT EXTSurvey
ACGME %-
COMPLIANT 75% 67% 50% 94% 83%
Overall Eval of the Program RESIDENT EXTSurvey
ACGME %-
COMPLIANT 8.5 7.7 8.6 9.4 8.4
Total Number of ACGME
Citations PROGRAM EXT
ACGME 0 0 0 0 0
AIR / Annual Program Review Dashboard Example
41
Scorecard System – Usage
Program expansion & funding process
Monitor program quality
Monitor institutional trend
Program Expansion & Funding Process Scorecard
43
Institutional Scorecard Trend Analysis
GME SURVEY 2013-2014 2014-2015 2015-2016 2016-2017 2017-2018
Program Is
Organized to Meet
Educational Needs.
88% 92% 85% 86% 89%
Program Balances
Service with Clinical
Education.
86% 87% 84% 82% 84%
Overall Satisfaction
with Training in
Program
85% 92% 92% 93% 94%
Would Recommend
Training Program91% 94% 93% 94% 95%
Faculty Spend
Sufficient Time
Supervising Trainees
95% 96% 92% 93% 96%
Faculty Are
Successful Teachers95% 97% 93% 94% 95%
Encouraged To Ask
Questions95% 97% 91% 93% 96%
Have Not Been
Personally
Mistreated in
Training Program.
89% 89% 91% 91% 95%
Opportunity To
Participate In QI
Projects
66% 88% 76% 73% 75%
Scorecard System – Database
Our Excel database contains:
−All ACGME-accredited programs
−All data sources
GME Survey, ACGME Survey, Program Evaluation, Work Hours Reports from MedHub, etc.
−Data starting from 2012-13
Headers of the database and an example of data
Over 17,500 rows of data for over 100 ACGME programs at Stanford!
Type Year ProgramName ProgramCode Metric % Program Compliant
ACGME-ResidentSurvey 2012-13 A 111111111 Sufficient instruction 86%
ACGME Citations 2017-18 A 111111111 Current Total # of ACGME Citations 2
GME Survey 2013-14 A 111111111 Faculty are successful teachers. 67%
Program Eval by Residents 2012-13 A 111111111 Aggregated Program Evaluation by Residents 7.8
Program Eval by Faculty 2016-17 A 111111111 Aggregated Program Evaluation by Faculty 8.2
Medhub-80 Hour Violation 2014-15 A 111111111 >80 Hour Violations / AY 2
Medhub-Duty Hours Review 2015-16 A 111111111 # Unreviewed Duty Hr Periods by PD / AY 0
Scorecard System – Database (Cont’d)
Generation of scorecard is automated
−Pivot table
−Paste to a formatted spreadsheet template (scorecard)
Topics
I. Introduction of the Stanford GME Housestaff Survey
II. Analysis of the Construct of the Stanford GME Housestaff Survey
III. Prediction of the ACGME Resident Surveys by the Stanford GME Housestaff Survey
IV. Group Discussion
V. Stanford Program’s Scorecard System
VI. Discussion & Questions
Thank You’s and Questions?
Many thanks to Jie Li, PhD, Stanford GME Program Manager for her ideas, research, analyses, presentation and database development efforts.
Contact Information
Nancy Piro: [email protected]
650-723-5948
Ann Dohn: [email protected]
650-723-5948