1905

Is The ACGME Resident/Fellow Is The ACGME Resident/Fellow Survey a Valid Tool to Assess Survey a Valid Tool to Assess General Surgery Residency General Surgery Residency

Programs Compliance with Work Programs Compliance with Work Hours Regulations?Hours Regulations?Robert P. Sticca, MD, FACS

Jay M. MacGregor, MD

Randolph E. Szlabick, MD, FACS

Department of Surgery

University of North Dakota SMHS

1905

BackgroundBackground• 2003 – ACGME work hours regulations

implemented• 2004 – ACGME Residents/Fellows Survey• 2008 – Special Message from CEO

ACGME (Thomas Nasca, MD)– Standard Approach to Programs Across

All Specialties with Potential Duty Hours Violations Identified in the Resident Survey

1905

ACGME Special MessageACGME Special Message• The Monitoring Committee has accepted the Resident Survey

as a reliable, sensitive and reasonably specific tool• The Monitoring Committee has set reasonable thresholds for

categorization of a program as having Potential Duty Hours Violations (PDHV) based on the Resident Survey

• The validity of the Resident Survey has been demonstrated though significant correlations with site visit findings to warrant added scrutiny for programs meeting the thresholds of potential noncompliance found in the group targeted for follow-up.

• There is a significant correlation between resident reported violations of duty hours with deficiencies in other important areas of the learning environment (correlation 0.51, p<0.0001)

Nasca, T. www.ACGME.org

1905

ACGME Algorithm for ACGME Algorithm for Noncompliant ProgramsNoncompliant Programs

Nasca, T. www.ACGME.org

1905

UND ExperienceUND Experience• 2005 – RRC site visit – additional categorical resident

requested– 5 year approval, no deficiencies– Additional resident denied, insufficient educational

justification• 2007 – RRC site visit – additional categorical resident

requested– 5 year approval, no deficiencies– Additional resident approved

• 2009 – RRC site visit – mandatory due to residents responses on survey– 5 year approval - commendation

1905

ACGME Survey - ProblemsACGME Survey - Problems

• Answer selections vague

• High threshold for compliance

• Small program effect

• Consequences of survey punitive

• Consequences based on survey results and not actual work hours data

1905

Current StudyCurrent Study• Purpose – obtain general surgery residents views

on work hours regulations and ACGME survey• Design – 20 question survey emailed to all Gen.

Surg. Program Directors – 5/09– 9 questions – work hours

– 9 questions – ACGME survey

– 1 both, 1 demographic, comments

• Study period – 5/09 – 10/09– Multiple reminders, pleas for participation

• Internet Survey – surveymonkey.com

1905

QuestionsQuestionsTABLE 1: Questions from Surgical Residents Views on the

Accreditation Council for Graduate Medical Education (ACGME) Resident Survey.

1. How many times have you taken the ACGME Resident survey?

2. Did you have any difficulty understanding the questions as written in the ACGME Survey?

3. Did you discuss the ACGME Resident Survey with your residency Program Director or any of your faculty prior to completing the survey?

4. Were you, in any way, instructed how to answer questions on the ACGME Resident Survey?

5. Did you answer all questions on the ACGME Resident Survey truthfully?

6. Are you aware how your answers to the ACGME Resident Survey can effect your surgery program's accreditation?

7. If you indicated an ACGME duty hours requirement is "SOMETIMES" met, which of the following is true?

8. If an ACGME duty hours requirement is "SOMETIMES" met, how often are you NONCOMPLIANT with this requirement?

9. When answering the ACGME Survey, what percentage of residents in a program need to indicate a rule violation for this to be deemed significant by the ACGME?

10. Do you feel the ACGME Resident Survey accurately evaluated the hours that you work in your Residency Program?

1905

ResultsResults

• Answers collected, analyzed - 11/09

• Aggregate

• Level of training– Junior – PGY 1, 2– Senior – PGY > 2

• Statistical analysis – openepi.com– P value < 0.05, statistically significant

1905

Results - ResponsesResults - Responses

Resident Year of Training by

Post Graduate Year (PGY)

Number of

Resident ResponsesPercent of Total

PGY 1 246 25.6

PGY 2 195 20.3

PGY 3 181 18.8

PGY 4 160 16.7

PGY 5 or Greater 179 18.6

Total 961 100

965 – 13.1 % of U.S. General Surgery Residents961 completed entire survey (99.6%)

Junior Residents – 441 Senior Residents – 520

1905

Results Results Number Times Survey Number Times Survey

Taken Taken

Question Junior Residents

(% of Junior Residents)

Senior Residents(% of Senior Residents)

Total (%) P values

How many times have you taken the

ACGME Resident/Fellow

Survey?

1 time = 254 (58) 2 times = 148 (34) 3 times = 25 (5) 4 times = 9 (2) 5 times = 1 (0.1)> 5 times = 4 (0.9)

1 time = 93 (17) 2 times = 95 (18) 3 times = 164 (31) 4 times = 90 (17) 5 times = 58 (10)> 5 times = 40 (7)

1 time = 347 (36)2 times = 243 (25)3 times = 189 (19)4 times = 99 (10)5 times = 59 (6)

>5 times = 44 (4)

<0.0000001

1905

Results Results Instruction on SurveyInstruction on Survey

Question

Junior Residents(% of Junior

Residents)

Senior Residents(% of Senior Residents)

Total (%) P values

Discuss survey with PD or faculty

Yes = 156 (35)No = 285 (65)

Yes = 185 (36)No = 335 (64)

Yes = 341 (35)No = 620 (65)

0.8881

Instructed on how to answer survey

Yes = 84 (19)No = 357 (81)

Yes = 82 (16)No = 438 (84)

Yes = 166 (17)No = 795 (83) 0.2242

1905

Results – Survey Results – Survey InterpretationInterpretation

Question Junior Residents

(% of Junior Residents)Senior Residents

(% of Senior Residents)Total (%) P values

Difficulty Understanding

Questions

Yes = 61 (14)No = 380 (86)

Yes = 118 (23)No = 402 (77)

Yes = 179 (19)No =782 (81)

0.0002662

When requirement

sometimes met, how often

noncompliant

In compliance = 121 (27)

In violation = 202 (46)

Neither compliant nor in violation = 118 (27)

In compliance = 156 (30)

In violation = 236 (45)

Neither compliant nor in violation = 128 (25)

In compliance = 280 (29)

In violation = 438 (45)

Neither compliant nor in violation = 246 (26)

0.4817

Meaning of sometimes

1 - 5 times/year = 212 (48)6 - 10 times/year = 117 (27)11 - 15 times/year = 41 (9)16 - 20 times/year = 27 (6)21 -25 times/year = 11 (3)> 25 times/year = 33 (7)

1 - 5 times/year = 288 (55)6 - 10 times/year = 109 (20)11 - 15 times/year = 49 (10)16 - 20 times/year = 25 (5)21 -25 times/year = 10 (2)> 25 times/year = 39 (8)

1 - 5 times/year = 500 (52) 6 - 10 times/year = 226 (24) 11 - 15 times/year =90 (10)

16 - 20 times/year = 52 (5) 21 -25 times/year = 21(2) > 25 times/year = 72 (7)

0.2852

1905

Results Results Effect on ProgramEffect on Program

QuestionJunior Residents

(% of Junior Residents)

Senior Residents(% of Senior

Residents)

Total (%) P values

Aware of effect on program

accreditation

Yes = 337 (76)No = 103 (24)

Yes = 399 (77)No = 121 (23)

Yes = 736 (77)No = 224 (23) 0.9081

What percent necessary to be

deemed significant by ACGME

5% = 131 (30)10% = 131 (30)15% = 37 (18)20% = 68 (15)25% = 52 (12)50% = 22 (5)

5% = 161 (31)10% = 117 (23)15% = 51 (10)20% = 97 (18)25% = 43 (8)

50% = 51 (10)

5% = 292 (30)10% = 248 (26)

15% = 88 (9)20% = 165 (17)25% = 95 (10)50% = 73 (8)

0.005127

1905

ResultsResultsHonesty/AccuracyHonesty/Accuracy

Question Junior Residents

(% of Junior Residents)

Senior Residents(% of Senior Residents)

Total (%) P values

Answer questions truthfully

Yes = 382 (87)

No = 59 (13)

Yes = 446 (86)

No = 74 (14)

Yes = 828 (86)No = 133 (14)

0.8856

ACGME Survey accurately

evaluated hours that you work

Yes = 287 (65)No = 154 (35)

Yes = 319 (61)No = 201 (39)

Yes = 606 (63)No =355 (37) 0.2231

1905

ConclusionsConclusions• Evaluation mechanism in which– 52% do not understand questions– 14% did not answer truthfully– 37% do not feel is accurate

• Not best tool to evaluate work hours compliance in residency programs

• Consequences significant• Open letter to GME community 10/28/09– www.acgme.org

Nasca, T. www.ACGME.org

1905

Thank you !!Thank you !!

Program Directors

Residents