The Use of a Surgery Specific Written

Examination in the Selection Process of Surgical Residents

Daniel Farkas, Kamal Nagpal, Ernesto Curras,

Ajay Shah, John CosgroveDepartment of Surgery

Bronx-Lebanon Hospital CenterAlbert Einstein College of Medicine

Background

Choosing residents for a surgical residency program is difficult and time consuming

Programs are looking for many characteristics, and among them is the ability to acquire surgical knowledge Programs are required by the RRC to

have 65% of their residents pass the ABS exams on the first try

These pass rates are available online to prospective candidates

Background

Many programs use a candidates USMLE scores to help identify their ability to gain surgical knowledge Some studies have found a correlation

between USMLE scores and ABS exams passing scores

Some conflicting evidence as to whether step 1 or step 2 is the better test

In addition, the USMLE is broad based and not surgery specific

Study Purpose

The purpose of this study was to look at a different method of assessing a candidate’s ability to gain surgical knowledge

Would using a surgery specific written exam (SSWE) be a better assessment of this ability as compared with the USMLE?

SSWE

50 question multiple choice SSWE was created Questions made by a faculty member,

using standard surgical textbooks Residency candidates were given

this exam on their interview day Each exam scored out of 100% and

added to candidates folders

Study Design

Retrospective study of residents in our program between 2004 and 2012

Academic folders were reviewed and pre-residency information was collected USMLE part 1 score (USMLE-1) USMLE part 2 score (USMLE-2) SSWE score

Intra-residency data collected ABSITE score in PGY 1 through PGY 5

(ABSITE-1 to ABSITE-5)

Study Design

Residents with all three pre-residency scores were included in the study

Correlation coefficients were calculated between the pre-residency exam scores and the intra-residency exam scores

P values of less than 0.05 were considered significant

Results

53 residents had partial information 43 residents had all three pre-

residency scores and were included in the study 38 had an ABSITE-1 score 24 had an ABSITE-2 score 18 had an ABSITE-3 score 17 had an ABSITE-4 score 14 had an ABSITE-5 score

  ABSITE-1

ABSITE-2

ABSITE-3

ABSITE-4

ABSITE-5

  n = 38 n = 24 n = 18 n = 17 n = 14

USMLE-1

r = 0.327p =

0.045*

r = 0.321p = 0.126

r = 0.346p = 0.160

r = 0.165p = 0.527

r = -0.150p = 0.610

USMLE-2

r = 0.314p = 0.055

r = 0.187p = 0.381

r = 0.415p = 0.086

r = 0.245p = 0.343

r = 0.044p = 0.880

SSWE r = 0.656p <

0.001*

r = 0.275p = 0.194

r = -0.179p = 0.476

r = -0.399p = 0.113

r = -0.060p = 0.838* Statistically

significant

  ABSITE-1

ABSITE-2

ABSITE-3

ABSITE-4

ABSITE-5

ABSITE-1

  n = 21r = 0.573p = 0.007*

n = 15r = 0.850p < 0.001*

n = 12r = 0.593p = 0.042*

n = 9r = 0.870p = 0.02*

ABSITE-2

    n = 17r = 0.906p < 0.001*

n = 14r = 0.904p < 0.001*

n = 11r = 0.765p = 0.006*

ABSITE-3

      n = 15r = 0.824p < 0.001*

n = 12r = 0.682p = 0.015*

ABSITE-4

        n = 14r = 0.406p = 0.150

* Statistically significant

Results - Summary

USMLE may correlate with ABSITE-1 scores, but a SSWE has a much stronger correlation with ABSITE-1 than either step 1 or 2

ABSITE-1 scores correlate significantly with ABSITE scores in future years

Discussion

Identifying residents that will do well on their board exams is critical to the success of a surgical residency program Part of a resident’s academic

achievement is due to the educational activities within the program

However, studies have shown that residents with poor USMLE scores have higher chances of failing the boards.1,2

1. Shellito JL, Osland JS, Helmer SD, Chang FC. American Board of Surgery examinations: can we identify surgery residency applicants and residents who will pass the examinations on the first attempt? Am J Surg. 2010;199:216-22.

2. de Virgilio C, Yaghoubian A, Kaji A, et al. Predicting performance on the American Board of Surgery qualifying and certifying examinations: a multi-institutional study. Arch Surg. 2010;145:852-6.

Discussion

Corneille et al1 showed that a targeted surgery exam during residency was able to predict which residents would do well on the ABSITE

Our study showed that by giving this exam to residency candidates, we could predict which candidates were more likely to do well on the ABSITE

Corneille MG, Willis R, Stewart RM, Dent DL. Performance on brief practice examination identifies residents at risk for poor ABSITE and ABS qualifying examination performance. J Surg Educ. 2011;68:246-9.

Limitations

Correlation only significant with ABSITE-1 Possible with a bigger sample size this

would change ABSITE-1 correlated with ABSITE

scores in later years ABSITE scores have been shown to

correlate with board passing rates1

SSWE scores were absolute whereas other exams were percentile scores Makes it harder to compare between

years

1. de Virgilio C, Yaghoubian A, Kaji A, et al. Predicting performance on the American Board of Surgery qualifying and certifying examinations: a multi-institutional study. Arch Surg. 2010;145:852-6.

Limitations

SSWE only useful for surgical knowledge 5 other competencies Many other important skills in being a

successful surgical resident and a successful surgeon

Not practical for every program Many candidates are wooed by other

programs on their interview day Good potential candidates could be

turned away by getting an extra exam

Conclusions

Notwithstanding the limitations, the SSWE is a very strong predictor of who will do well on ABSITE-1

Higher ABSITE-1 scores is correlated with higher ABSITE scores in later years

Higher ABSITE scores can predict who will pass their board exams on the first chance

Final Conclusion

Residency programs looking for residents who will pass their boards on the first attempt are more likely to find these residents using a SSWE than by using the USMLE

Acknowledgements

Kamal Nagpal Ernesto Curras Ajay Shah John Cosgrove

Saundra King Nancy Rivera