The New MCAT Exam - AAMC · PDF fileThe New MCAT Exam: Where We Are and Where We’re...

The New MCAT Exam: Where We Are

and Where We’re GoingRobert Witzburg, M.D.

Dean of Admissions, Boston University School of Medicine

Judith Byrne and Cynthia A. SearcyAAMC

CONFIDENTIAL – DO NOT DISTRIBUTE2 CONFIDENTIAL – DO NOT DISTRIBUTE2 CONFIDENTIAL – DO NOT DISTRIBUTE2

What will we talk about today?• What do we know about applicants to the class we just

admitted?• What do we know about this year’s test takers (many of whom

are in our current applicant pool)?• What are we learning about the impact, use, and validity of the

new exam?• Q&A


Today’s information appears on the Admissions Hub• Using MCAT Data in 2018 Medical

Student Selection

• Interactive Score Report • Videos• Presentations and other materials

on the new exam, new scores, and validity research

• Research articles

www.aamc.org/admissions


Just finished the third year of testing w/ the new exam

Applicants/Matriculants

Year 1Coursework

Year 2Coursework

USMLEStep 1

Clerkships

Graduate in4 Years

Graduate in5 Years

USMLE Step 2 Exams

2015 2016 2017 2019 2020 20212018

2016 2017 2018 2019 2020 2021 2022

2017 2019 2020 2021 2022 20232018

Examinees


Today, we’ll look at data for examinees, applicants, and M1s who took the new test


Year 1Coursework

Year 2Coursework

USMLEStep 1

Clerkships

Graduate in4 Years

Graduate in5 Years

USMLE Step 2-CK

USMLE Step 2 Exams

2015 2016 2017 2019 2020 20212018

2016 2017 2018 2019 2020 2021 2022

2017 2019 2020 2021 2022 20232018

Examinees


Let’s see how it’s going!


What do we know about applicants to the class we just admitted?


About 90% of 2017 applicants submitted scores from the new exam

These data include applicants who applied through AMCAS AND non-AMCAS application services

12%

88%

0%

20%

40%

60%

80%

100%

Took only the old exam Took the new exam

Applicants to the 2017 entering class

Only the new exam

Both the old and newexams


Our admissions committees put new scores in contextPercentage and Number of 2017 Applicants Accepted into at Least One Medical School, by New MCAT Total Score and Undergraduate GPA Range

New MCAT TotalGPA Total 472-485 486-489 490-493 494-497 498-501 502-505 506-509 510-513 514-517 518-528 All3.80-4.00 4% 5% 10% 21% 34% 54% 67% 76% 85% 89% 67%

2/51 5/98 23/241 111/528 353/1,040 920/1,692 1,538/2,310 1,870/2,461 1,798/2,112 1,796/2,014 8,416/12,5473.60-3.79 0% 1% 5% 15% 26% 38% 54% 68% 76% 85% 48%

0/126 3/223 23/442 129/867 366/1,430 764/1,996 1,197/2,225 1,388/2,040 987/1,301 639/755 5,496/11,4053.40-3.59 2% 1% 4% 10% 21% 29% 40% 55% 61% 71% 33%

4/183 3/293 22/552 100/955 265/1,286 479/1,635 661/1,659 694/1,260 405/660 235/329 2,868/8,8123.20-3.39 0% <1% 3% 9% 18% 24% 32% 40% 56% 58% 22%

0/226 1/306 14/527 64/749 158/884 246/,1027 279/873 255/631 176/314 63/109 1,256/5,6463.00-3.19 0% 2% 1% 7% 14% 21% 25% 35% 42% 37% 15%

0/248 4/253 4/363 36/490 74/511 109/507 100/395 98/282 43/103 21/57 489/3,2092.80-2.99 1% 1% 2% 5% 6% 17% 25% 30% 29% 42% 10%

2/247 2/181 5/201 12/238 14/233 34/203 40/160 25/84 14/49 5/12 153/1,6082.60-2.79 0% 1% 0% 2% 6% 18% 18% 17% 40% - 6%

0/157 1/117 0/142 3/132 8/126 14/79 10/57 6/36 6/15 53/8682.40-2.59 0% 2% 2% 5% 3% 13% 25% 24% - - 5%

0/118 1/57 1/48 3/59 1/40 4/30 6/24 4/17 21/3982.20-2.39 0% 0% 0% 0% 12% 33% 8% - - 5%

0/69 0/29 0/24 0/21 2/17 4/12 1/12 9/1922.00-2.19 0% 0% 0% - - - - - 0%

0/41 0/13 0/10 0/86<2.00 0% - - - - 3%

0/15 1/31All 1% 1% 4% 11% 22% 36% 50% 64% 75% 84% 42%

8/1,481 20/1,576 93/2,556 458/4,051 1,241/5,571 2,574/7,188 3,832/7,717 4,341/6,819 3,430/4,558 2,765/3,285 18,762/44,802


Percentage and Number of 2017 Applicants Accepted into at Least One Medical School, by New MCAT Total Score and Undergraduate GPA Range


2/51 5/98 23/241 111/528 353/1,040 920/1,692 1,538/2,310 1,870/2,461 1,798/2,112 1,796/2,014 8,416/12,5473.60-3.79 0% 1% 5% 15% 26% 38% 54% 68% 76% 85% 48%

0/126 3/223 23/442 129/867 366/1,430 764/1,996 1,197/2,225 1,388/2,040 987/1,301 639/755 5,496/11,4053.40-3.59 2% 1% 4% 10% 21% 29% 40% 55% 61% 71% 33%

4/183 3/293 22/552 100/955 265/1,286 479/1,635 661/1,659 694/1,260 405/660 235/329 2,868/8,8123.20-3.39 0% <1% 3% 9% 18% 24% 32% 40% 56% 58% 22%

0/226 1/306 14/527 64/749 158/884 246/,1027 279/873 255/631 176/314 63/109 1,256/5,6463.00-3.19 0% 2% 1% 7% 14% 21% 25% 35% 42% 37% 15%

0/248 4/253 4/363 36/490 74/511 109/507 100/395 98/282 43/103 21/57 489/3,2092.80-2.99 1% 1% 2% 5% 6% 17% 25% 30% 29% 42% 10%

2/247 2/181 5/201 12/238 14/233 34/203 40/160 25/84 14/49 5/12 153/1,6082.60-2.79 0% 1% 0% 2% 6% 18% 18% 17% 40% - 6%

0/157 1/117 0/142 3/132 8/126 14/79 10/57 6/36 6/15 53/8682.40-2.59 0% 2% 2% 5% 3% 13% 25% 24% - - 5%

0/118 1/57 1/48 3/59 1/40 4/30 6/24 4/17 21/3982.20-2.39 0% 0% 0% 0% 12% 33% 8% - - 5%

0/69 0/29 0/24 0/21 2/17 4/12 1/12 9/1922.00-2.19 0% 0% 0% - - - - - 0%

0/41 0/13 0/10 0/86<2.00 0% - - - - 3%

0/15 1/31All 1% 1% 4% 11% 22% 36% 50% 64% 75% 84% 42%

8/1,481 20/1,576 93/2,556 458/4,051 1,241/5,571 2,574/7,188 3,832/7,717 4,341/6,819 3,430/4,558 2,765/3,285 18,762/44,802

Some 2017 applicants w/ high UGPAs and MCATs weren’t accepted

11% of applicants with GPAs at or above 3.8 and

MCAT scores at or above 518 were

not admitted into any medical

schools




2/51 5/98 23/241 111/528 353/1,040 920/1,692 1,538/2,310 1,870/2,461 1,798/2,112 1,796/2,014 8,416/12,5473.60-3.79 0% 1% 5% 15% 26% 38% 54% 68% 76% 85% 48%

0/126 3/223 23/442 129/867 366/1,430 764/1,996 1,197/2,225 1,388/2,040 987/1,301 639/755 5,496/11,4053.40-3.59 2% 1% 4% 10% 21% 29% 40% 55% 61% 71% 33%

4/183 3/293 22/552 100/955 265/1,286 479/1,635 661/1,659 694/1,260 405/660 235/329 2,868/8,8123.20-3.39 0% <1% 3% 9% 18% 24% 32% 40% 56% 58% 22%

0/226 1/306 14/527 64/749 158/884 246/,1027 279/873 255/631 176/314 63/109 1,256/5,6463.00-3.19 0% 2% 1% 7% 14% 21% 25% 35% 42% 37% 15%

0/248 4/253 4/363 36/490 74/511 109/507 100/395 98/282 43/103 21/57 489/3,2092.80-2.99 1% 1% 2% 5% 6% 17% 25% 30% 29% 42% 10%

2/247 2/181 5/201 12/238 14/233 34/203 40/160 25/84 14/49 5/12 153/1,6082.60-2.79 0% 1% 0% 2% 6% 18% 18% 17% 40% - 6%

0/157 1/117 0/142 3/132 8/126 14/79 10/57 6/36 6/15 53/8682.40-2.59 0% 2% 2% 5% 3% 13% 25% 24% - - 5%

0/118 1/57 1/48 3/59 1/40 4/30 6/24 4/17 21/3982.20-2.39 0% 0% 0% 0% 12% 33% 8% - - 5%

0/69 0/29 0/24 0/21 2/17 4/12 1/12 9/1922.00-2.19 0% 0% 0% - - - - - 0%

0/41 0/13 0/10 0/86<2.00 0% - - - - 3%

0/15 1/31All 1% 1% 4% 11% 22% 36% 50% 64% 75% 84% 42%

8/1,481 20/1,576 93/2,556 458/4,051 1,241/5,571 2,574/7,188 3,832/7,717 4,341/6,819 3,430/4,558 2,765/3,285 18,762/44,802

Other 2017 applicants w/ modest credentials were accepted

14% of applicants with GPAs 3.00 - 3.19

and MCAT scores 498-501 were

admitted into at least one

medical school




2/51 5/98 23/241 111/528 353/1,040 920/1,692 1,538/2,310 1,870/2,461 1,798/2,112 1,796/2,014 8,416/12,5473.60-3.79 0% 1% 5% 15% 26% 38% 54% 68% 76% 85% 48%

0/126 3/223 23/442 129/867 366/1,430 764/1,996 1,197/2,225 1,388/2,040 987/1,301 639/755 5,496/11,4053.40-3.59 2% 1% 4% 10% 21% 29% 40% 55% 61% 71% 33%

4/183 3/293 22/552 100/955 265/1,286 479/1,635 661/1,659 694/1,260 405/660 235/329 2,868/8,8123.20-3.39 0% <1% 3% 9% 18% 24% 32% 40% 56% 58% 22%

0/226 1/306 14/527 64/749 158/884 246/,1027 279/873 255/631 176/314 63/109 1,256/5,6463.00-3.19 0% 2% 1% 7% 14% 21% 25% 35% 42% 37% 15%

0/248 4/253 4/363 36/490 74/511 109/507 100/395 98/282 43/103 21/57 489/3,2092.80-2.99 1% 1% 2% 5% 6% 17% 25% 30% 29% 42% 10%

2/247 2/181 5/201 12/238 14/233 34/203 40/160 25/84 14/49 5/12 153/1,6082.60-2.79 0% 1% 0% 2% 6% 18% 18% 17% 40% - 6%

0/157 1/117 0/142 3/132 8/126 14/79 10/57 6/36 6/15 53/8682.40-2.59 0% 2% 2% 5% 3% 13% 25% 24% - - 5%

0/118 1/57 1/48 3/59 1/40 4/30 6/24 4/17 21/3982.20-2.39 0% 0% 0% 0% 12% 33% 8% - - 5%

0/69 0/29 0/24 0/21 2/17 4/12 1/12 9/1922.00-2.19 0% 0% 0% - - - - - 0%

0/41 0/13 0/10 0/86<2.00 0% - - - - 3%

0/15 1/31All 1% 1% 4% 11% 22% 36% 50% 64% 75% 84% 42%

8/1,481 20/1,576 93/2,556 458/4,051 1,241/5,571 2,574/7,188 3,832/7,717 4,341/6,819 3,430/4,558 2,765/3,285 18,762/44,802

42% of 2017 applicants with new scores got one or more acceptances

42% of applicants with

new scores received one or

more acceptances


Interested in learning more about our colleagues’ use of new exam scores?

Strategies for Mission-Oriented Admissions in Light of the New MCAT Exam Monday, November 6 at 3:00 PM ET

Center 304-306

Tanisha Price-Johnson, PhD, University of Arizona-Tuscon College of MedicineTheodore Hall, MD, David Geffen School of Medicine at UCLAAaron Saguil, MD, F. Edward Herbert School of Medicine, Uniformed Services University Sunny Nakae, PhD, Loyola University Chicago Stritch School of MedicineCarol Teitz, MD, University of Washington School of Medicine


What do we know about this year’s test takers (many of whom are in your current

applicant pool)?


3% more examinees tested in 2017 than in 2016

78,410 80,997

10,000

20,000

30,000

40,000

50,000

60,000

70,000

80,000

90,000

100,000

2016 2017

Unique Examinees with Scores


Examinees tested in the same numbers and proportions as the past

Percentage of Examinees Taking the New MCAT Exam by Gender, Race/Ethnicity (N = 186,450)

46%

54%

48%

10%

11%

27%

1%

<1%

4%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Male

Female

White

Black or African American

Hispanic

Asian

American Indian or Alaska Native

Native Hawaiian or Other Pacific Islander

Other

Gender

Race/Ethnicity


Percentage of Examinees Taking the New MCAT Exam by Fee Assistance, Parental Education, Testing Condition,

and Repeater Status (N = 186,450)

Examinees tested in the same numbers and proportions as the past

6%

82%

18%

1%

76%

24%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Did not receive

Received

Not first-generation college

First-generation college

Standard

Nonstandard

Non-repeater

Repeater

Fee Assistance

ParentalEducation

Testing Condition

Repeater Status

94%

99%


How did examinees prepare for the new test?


Most examinees completed coursework in the natural, behavioral, and social sciences

87%

84%

50%

42%

83%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Biochemistry

Psychology

Sociology

Research Methods

Statistics

Biology

General Chemistry

Organic Chemistry

Physics

95%

92%

96%

94%

Coursework(N = 69,997)


Many examinees took preparation courses, used Khan tutorials, and used practice materials

41%

6%

52%

26%

75%

52%

22%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Using commercial prep courses

University-based prep courses

Khan Academy

None of the three

Practice exams

Questions packs & section banks

None of the three

Practice Materials(N = 31,723)

Preparation Courses (N = 69,525)


Khan Academy tutorials and some AAMC resources are free• The Khan Academy has over 1,100 free tutorials on exam content Free practice materials and resources are available on AAMC’s website: What’s on the MCAT Exam? Interactive Content Outline Roadmaps to MCAT Content in Biochemistry, Psychology, and

Sociology Textbooks Guide to Creating a Study Plan How I Prepared for the MCAT Exam Testimonials AAMC Pre-Med Navigator

Students-residents.aamc.org/mcatprep


There are a number of low-cost materials

Students-residents.aamc.org/mcatprep


How well did examinees score on the new test?


Interpreting Box-and-Whisker PlotsMCAT total scores for exams administered in 2015-2017

472 476 480 484 488 492 496 500 504 508 512 516 520 524 528

Overall (mean=500.5; N=239,681)

Total Score

90th Percentile10th Percentile

Median = 50th Percentile

25th Percentile 75th Percentile

501

493 508

486 514


There was a wide range of scores overall and w/in group

472 476 480 484 488 492 496 500 504 508 512 516 520 524 528

Total (mean = 500.5; N = 239,681)

Male (mean = 502.1; N = 108,262)

Female (mean = 499.2; N = 130,777)

White (mean = 502.3; N = 105,696)

Black or African American (mean = 493.5; N = 24,559)

Hispanic (mean = 495.7; N = 25,668)

Asian (mean = 502; N = 63,201)

American Indian or Alaska Native (mean = 469.9; N = 2,452)

Native Hawaiian or Other Pacific Islander (mean = 498.6; N = 690)

Total Score

Gender

Race/Ethnicity

Overall


There was a wide range of scores overall and w/in group

472 476 480 484 488 492 496 500 504 508 512 516 520 524 528

Did not receive (mean = 500.8; N = 224,656)

Received (mean = 496.4; N = 15,025)

Not first-generation college (mean = 501.8; N = 124,985)

First-generation college (mean = 496.3; N = 27,493)

Standard (mean = 500.5; N = 237,577)

Nonstandard (mean = 502; N = 2,104)

Non-repeater (mean = 502.6; N = 141,592)

Repeater - 1st attempt (mean = 496.1; N = 44,858)

Repeater - 2nd attempt (mean = 499.1; N = 44,858)

Total Score

Testing Condition

Fee Assistance

Repeater Status

Parental Education


This variation is similar to that on the old exam and other standardized tests Old MCAT exam GRE LSAT GMAT


Academic achievement is affected by multiple factors over a long period of time

Academic Achievement

and Test Scores

Student

Family Community

Prior learning opportunity and experience• K-12 schools• Undergraduate

study


Academic Achievement

and Test Scores

Student

Family Community

Prior learning opportunity and experience• K-12 schools• Undergraduate

study

Students may experience multiple barriers

Food insecurity

Less support for learning at home

Fewer role models and mentors

Fewer experienced

teachers

Dietrichson, J., Martin B., Filges, T., and Jorgensen, AMK (2017). Academic interventions for elementary and middle school students with low socioeconomic status: A systematic review and meta-analysis. Review of Educational Research. 87, 243-283.


With this in mind, what can we do about differences? Look for points of leverage to improve equity in access for all

students Monitor data on academic preparation, diversity, and

performance Expand outreach to faculty at under-resourced and minority-

serving institutions Expand outreach directly to students from sociodemographic

groups underrepresented in medicine

Example: Expanding outreach to faculty at under-resourced and minority-serving institutionsDeveloped new contacts at minority-serving and/or under-resourced colleges and universities—adding 350 contacts from:• Almost 90 HBCUs• More than 200 Hispanic-serving

institutions• More than 30 Tribal colleges and

universities

Please encourage your colleagues to add their names to our mailing list. They can contact us at [email protected]

If you’re interested in contributing to the Pre-Med Navigator, please don’t forget to sign up before you leave.31

mailto:[email protected]


Example: Expanding Outreach Directly to Students from Sociodemographic Groups Underrepresented in Medicine Reaching out to students directly through emails and

newsletters Health professions career fairs focused on minority populations

(e.g., UC Davis Pre-Health Conference, Stanford University Minority Medical Alliance Conference)

Virtual events like Health Professions Week Virtual Fair and AAMC Medical School Virtual Fair

Monitoring students’ use of the MCAT collection of the Khan Academy and other free and low-cost materials to learn if better outreach about available resources is needed


This topic is of great concern to the MCAT Validity Committee

The committee is conducting qualitative and quantitative research to understand students’ preparation strategies and barriers What is easy and difficult for examinees when they prepare for

the MCAT exam? What is easy and difficult about using the AAMC’s free and low-

cost materials to prepare for the MCAT exam? Are these barriers different or greater for those from

sociodemographic groups underrepresented in medicine? What additional resources and information do examinees and

their advisors need?


Interview Students

andAdvisors

Revise PMQ

Administer PMQ and Compare

Results for Examinees

from Different Groups

Learn what barriers

exist

Develop new

information and

resources

Results will help us improve access to information and resources


What are we learning about the impact, use, and predictive validity of the new exam?


21 medical schools are working together to evaluate the new exam


The MCAT validity research addresses multiple goals Provides evidence about the value of the new

MCAT exam in admissions decisions Answers questions about the fairness and

consequences of introducing the new MCAT exam for examinees, applicants, and medical students

Presents data to admissions officers that they can act on to improve their admissions decisions

Uses findings about the needs of aspiring physicians from underrepresented backgrounds to improve test preparation resources and outreach


The MCAT validity agenda includes three broad research topics

Academic Preparation,

Diversity, and Fairness

Admissions Decision Making

Predicting Academic

Performance


Let’s zoom in on the predictive validity research

Academic Preparation,

Diversity, and Fairness

Admissions Decision Making

Predicting Academic

Performance


How well did the new scores predict students’ academic performance in the first year of

medical school?


Nationally, 2016 entrants w/ a wide range of scores progressed to M2 w/out delay

% S

tude

nts

Prog

ress

ing

to M

2

MCAT Total Score Range

50%

80% 81%

93% 94% 96% 97% 98% 98% 99%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

472-485 486-489 490-493 494-497 498-501 502-505 506-509 510-513 514-517 518-528

% of 2016 Entrants Who Progressed to M2 Without Delay

(N=10) (N=15) (N=106) (N=309) (N=811) (N=1,360) (N=1,727) (N=1,654) (N=1,200) (N=778)

Note. Only students enrolled in regular, 4-year MD programs who took the new exam were included in this analysis.


At the validity schools, new MCAT scores predict performance across M1 courses

r Effect Size0.1 Small0.3 Medium0.5 Large

0.43

0.69

0.55

00.

20.

40.

60.

81.

0C

orre

cted

Cor

rela

tion

Overall(NSchool = 15)

Medium Effect

Correlation of MCAT Total Scores with Performance Across M1 Courses: Median and Interquartile Range

Overall, the associations of the new MCAT total scores with performance across M1 courses are medium to large


These validities compare well to those for other exams

Author(Year) Exam

Type of ExamScore

Type of Outcome

Median Validity

CoefficientUnit of

AnalysisAAMC (2017) New Exam Total Score First-Year Course Average .55 School

(Nschool=15)AAMC (2017) Old MCAT Total score First-Year Course Average .53 School

(Nschool=17)Kuncel, et al. (2007) GMAT Total score First-Year Graduate GPA .47 Meta-

analysis of multiple studies

Stilwell, et al. (2011) LSAT Total score First-Year Average Course Grade .58 School(Nschool=170)


New scores predict equally well for students from different sociodemographic backgrounds Research studied these early relationships for students grouped

by: Race/ethnicity Socioeconomic status

So far, MCAT scores neither over- nor under-predict the performance of students from underrepresented backgrounds based on two types of performance outcomes: National outcome: Progression to M2 without delay Validity school outcome: Average performance across M1

coursesWe have a lot more to learn about how students do in their second year, in their clerkships

and on their USMLE exams, and their graduation from undergraduate medical school

Each year, we’ll learn more about the new exam


Year 1Coursework

Year 2Coursework

USMLEStep 1

Clerkships

Graduate in4 Years

Graduate in5 Years

USMLE Step 2-CK

USMLE Step 2 Exams

2015 2016 2017 2019 2020 20212018

2016 2017 2018 2019 2020 2021 2022

2017 2019 2020 2021 2022 20232018

2018 2019 2020 2021 2022 2023 2024

Examinees


Questions?

The New MCAT Exam - AAMC · PDF fileThe New MCAT Exam: Where We Are and Where We’re...

Documents

Transcript of The New MCAT Exam - AAMC · PDF fileThe New MCAT Exam: Where We Are and Where We’re...