USMLE Part 1 Predicts Successful Completion of Milestones in General Surgery

Alterman DM, Jones TM, Daley BJ, Goldman MHDepartment of Surgery

Background Optimal markers of achievement and goals for

Surgical Education are being defined

Graduate Medical Education is being directed towards a competency-based education and training paradigm by the ACGME

Predictors of Goal Achievement is poorly defined

Background We currently document comprehensive resident

outcomes that may be equivalent to “Milestones”o “360” performance evaluationo ABSITE in-service examinationo ACGME core competency faculty evaluationo Academic productivity (publications, presentations, etc.)o Attrition

We sought to understand predictors of these “Milestone” markers of successful surgical education

Materials and Methods 18 year review of all General Surgery residents at a single

academic university training center (n=101)

Background and Demographic data obtained for these included extraction from ERAS application, transcripts, and interview evaluations

Outcome data was collected for the above makerso ABSITE scores o Conference attendanceo Attrition rateo Academic productivityo ACGME core competency faculty assessmentso Operative case numberso Board certification

Materials and Methods Background Data Extraction (>40 variables for each

resident)o Demographicso Personal items listed (such as marital status)o Undergraduate degree / majoro Medical School Performance (GPA, Clerkship grades, etc.)o AOA statuso USMLE STEP scoreso Prior work or research experienceo Publicationso Interview faculty score (1-5 Likert scale)o “Special Skill”

(high level of achievement / performance in non-medical field – such as concert pianist or division I athlete)

Materials and Methods Statistical analysis was performed with SPSS 19

software package

Comparison of multiple means was with ANOVA

Multiple logistic regression analysis was used to create predictive models for each outcome sought

For univariate comparison of means or percentages, t-test or z-test was used, respectively, with an alpha level of 0.05

Results USMLE Step 1 (STEP1) score is positively

correlated to ABSITE scores PGY1-5 levels (R=0.408-469, p=0.008)

Results STEP1 is positively correlated to faculty ratings of

all six ACGME core competencies across all PGY levels

Summary data for significant correlations of STEP1-3 with ACGME core competency evaluation ratings across PGY1-5 levels

A total of 150 correlations were made between each PGY level (1-5) of each of the six ACGME core competencies and STEP1-3

The number of significant correlations (all positive) are given for each comparison

Results Multivariate logistic analysis was performed to

evaluate for predictors of each discrete outcome measure

Sub-analysis was repeated for residents distributed among: o Categorical graduates and non-graduateso Non-Graduates with voluntary and involuntary attrition, o Preliminary non-designated residents matched/non-

matched

Results In multivariate logistic analysis, STEP1 and faculty

interview ranks were able to predict categorical matched residents attrition (Wald 8.4, p=0.004)

Analysis of Categorical Residents attrition:o SPECIAL SKILL was predictive of non-attrition (OR 3.5)

Results In repeated analysis among subgroups, GENDER,

GPA, AGE or ETHNICITY was not significant (in contrast to prior data)o * several of these difference were noted in univariate

analysis but none remained

Limitations Single Institution Review

The data points which we feel are equivalent to “Milestones” may not be

Some older files required some interpretation and normalization of scores

Conclusions Our current resident evaluation is “360 degrees”

and evaluates multiple domains of performance – calling into question the need for novel supplemental assessment tools

STEP1 retained strong positive predictive value among academic and clinical outcomes

The importance of STEP1 for screening and prediction of future success in Graduate Surgical Education is becoming increasingly clear

Conclusions USMLE STEP1 is a robust data point and has

documented predictive value across many specialties

o Associated with residency clinical success and board passage rates in specialties such as EM, Ortho, FP, Peds, IM, OB/GYN, GS and Anesthesiology.

o The literature and our data support a critical threshold of >215 for USMLE STEP1 evaluation

Questions?

Question of Attrition? Our overall attrition rate 23.7% is comparable to national

reported ranges, however if one considers the attrition from just categorical matched residents it would rise to 34%.

If the denominator included all categorical matched residents and preliminary residents that were matched to our system, the attrition rate would then be 30.3%.

If we were to look at attrition as a function of categorical resident’s “failure” or non-voluntary attrition; this number would be 15%.

Clearly, there are many ways to analyze and report attrition and this makes comparison of institutional reports difficult.