The value of a commercial test-preparation course on COMLEX Level 1 Performance

Donald J. Sefcik, DO, MBA, Associate Dean, Midwestern University/CCOM

C. Pat Obi, PhD, Professor of Finance and Statistics, Purdue University

t-Test: Two-Sample for Equal VariancesReview No Review

Mean 501.56 508.47Variance 5861.06 6958.31Observations 73 68Pooled Variance 6389.95Hypothesized Mean Difference0df 139t Stat -0.5128P(T<=t) one-tail 0.3044t Critical one-tail 1.6559P(T<=t) two-tail 0.6089t Critical two-tail 1.9772

F-Test: Two-Sample for VariancesReview No Review

Mean 501.56 508.47Variance 5861.06 6958.31Observations 73 68df 72 67F 0.8423P(F<=f) one-tail 0.2372F Critical one-tail 0.6730

Correlation: Course performance and Review course COMLEX 1 Review Course

COMLEX 1 1.00000Review Course -0.04346 1.00000ICM 0.50408 -0.08630POM 0.60737 -0.02263Anatomy 0.65721 -0.11038Biochemistry 0.65338 -0.13107Microbiology 0.63646 0.00786OM1 0.50967 -0.14478OM2 0.56991 -0.08604Pathology 0.68680 -0.02139Physiology 0.70312 -0.10137Pharmacology 0.66615 0.03578Psychiatry 0.45002 -0.04508

Overview Results Conclusions

Objectives

References

Methods

The purpose of this study was to determine the relative value of participation in a commercial test-preparation course. This study was designed to determine if there was a statistically significant improvement in the average COMLEX Level 1 examination scores of osteopathic medical students that participated in a commercial COMLEX review course offered during the winter and spring of 2004.

1. McGaghie WC, Downing SM, Kubilius R. What is the Impact of Commercial Test Preparation Courses on Medical Examination Performance? Teach Learn Med. 2004;16(2):202-211.

2. Werner LS, Bull BS. The effect of three commercial coaching courses on Step One USMLE performance. Med Ed. 2003;37:527-531.

3. Thadani RA, Swanson DB, Galbrath RM. A Preliminary Analysis of Different Approaches to Preparing for the USMLE Step 1. Acad Med. 2000;75(10 suppl):S40-S42.

4. Sefcik DJ, Prozialeck WC, O’Hare TH. Characteristics of the courses that best predict COMLEX USA-Level 1 performance. JAOA. 2003;103(10):491-494.

Tests of significance revealed that there is no difference in the variability of COMLEX scores between students who participated in a commercial test-preparation course and those that did not. Further, a two-sample means test showed no statistical evidence that the average COMLEX Level 1 scores for participants in the review course were different from those of students that did not participate in the course.

Several of the students in MWU/CCOM’s Class of 2006:

• paid approximately $550 each and

• spent approximately 160 hours in weekend lectures

to participate in a commercial test-preparation review course that appeared not to have positively impacted their COMLEX Level 1 scores.

The findings of this study suggest that there is no statistical evidence that participation in a commercial test-preparation review course for the COMLEX Level 1 examination provides a positive return on investment with regard to students’ time and money.

To best position themselves for success, many osteopathic medical students expend significant resources, including time, money and emotions, preparing for their national licensing board examinations. A primary motivator is their concern that a high examination score will make them more desirable candidates for competitive residencies. This study was designed to determine whether the expenditure of these resources has a positive impact on COMLEX Level 1 examination scores.

The subjects of this study were the members of MWU/CCOM’s Class of 2006. Courses at MWU/CCOM use a numerical scale to rate student achievement; a score of 70 or greater is passing. The final grade for every course taken during the MS1 & MS2 years, as well as participation in the review course were analyzed. In preparation for their COMLEX Level 1 examination, 73 members of CCOM’s Class of 2006 participated in a well-marketed traditional board review course while 68 of their classmates prepared without the course.

To determine whether differences exist in the score characteristics of students who participated in the course versus those that did not, a 3-part statistical analysis was carried out. The initial analysis was designed to show whether the preparatory course created a more homogeneous aptitude level. To this end, an F-test of significance was used to test for difference in the variability of scores. Second, a 2-sample inference for mean scores was employed to determine whether average test scores differ between the groups. Finally, a test of correlation and regression analysis was conducted to determine whether a joint relationship exists between the COMLEX scores, review course participation and individual course performance.

Regression results (n = 141) reveled that a statistically significant relationship (p = 0.0000; R2 = 0.61; F-stat = 20.43) exists between the scores of students that participated in the review course and the specified explanatory variables jointly. Unfortunately, no individual assessment could be made regarding the individual impact of the variables due to an apparent collinearity. However, a test of correlation did reveal that the relationship between student participation in the review course and COMLEX performance was negative.