ABA/ASA JOINT COUNCIL

ON IN-TRAINING

EXAMINATIONS

2004 REPORT TO SAAC/AAPDGlenn P. Gravlee, MD

Chair, Joint Council

The Ohio State University Medical Center

Department of Anesthesiology

JOINT COUNCIL MEMBERS

• ABA– David Chestnut

– Douglas Coursin

– Glenn Gravlee

– Patricia Kapur

– Mark Rockoff (Jim DiNardo)

– Raymond Roy

– Kenneth Tuman

• ASA– Arnold Berry

– John Cooper (John Rowlingson)

– Jeffrey Gross (V.C.)

– Philip Lebowitz

– Charles Otto

– Patricia Petrozza

– Mark Rosen

50 Question Writers

Title: Junior Editors

IN-TRAINING EXAMINATION EDITORS

Steven Allen, Audree Bendo, James Di Nardo, Carter Dodge, Sylvia Dolinski, John Ebert, John Emhardt, Robert Gaiser, Eric Kitain, Bruce Kleinman, Elliott Krane, Larry Krenis, Larry Kushins, Cynthia Lien, Vinod Malhotra, Donald Martin, Roger Mecca, John Moyers, Julia Pollock, Lindo Jo Rice, Robert Sladen, Richard Stypula, Richard Teplick, Helen Westman, Thomas Wolfe

IN-TRAINING EXAMINATION

• “3 exams in 1”

– In-Training Exam

– Requalifying Exam

– ABA Written Exam• Subset of 300 questions from ITE

– Plus: Recertification Exam• Well-performing A-types from ITE form the

question pool

Demographics of In-Training Examinations

4000

5000

6000

7000

8000

2000 2001 2002 2003 2004Tot

al N

um

ber

of

Exa

min

ees

Registrants

DEMOGRAPHICS OF IN-TRAINING EXAMINATIONS - AMG + IMG

0

200

400

600

800

1000

1200

1400

1600

1999 2001 2002 2003 2004

CA-1 CA-2 CA-3

Exam Content340 Questions ~ 100 are Links

0 50 100 150 200

Disease States

Anes Procedures

Physics/Math

Anat/Biochem

Pharmacology

Physiology

Number of Questions

15%

16%

4%

8%

47%

9%

Disease States Breakdown163 Questions

0

5

10

15

20

25

30

Pain

Respi

rato

ry

Cardi

ovas

cula

rCNS

GI/L

iver

/End

o/Ren

al

Hem

atol

Peds

OB

Oth

er

Nu

mb

er o

f Q

ues

tion

s

5.7%4.6%

7.2%

4.6% 4.6% 4.3%

5.7% 4.9%5.1%

Item Difficulty (Mean Logit ± SE)

-0.8

-0.6

-0.4

-0.2

0

0.2

1999 2000 2001 2002 2003 2004

Log

it S

cale

In-training ExamAverage Scaled Score

05

10152025303540

2000 2001 2002 2003 2004

All CA3 Residents

Gaps in Knowledge – CA3s

• 8 Surprising Gaps Identified

• Report being sent by ASA office

Examination Registration/Behavior

• Government Issue Picture ID

• NO Cell Phones or Pagers

• Calculators are NOT OK

• Consequences of Cheating

• Proctors are in charge

Joint Council Finances

• Joint Oversight by ASA/ABA

• Largely sustained by Exam fees

• In the Red ($50-100K) when N was low

Joint Council Finances

• Still losing $20-50K/year with strong N: increased proctor compensation, NBME fees mainly

• Exam cost @ $85 since 1996

• Joint Council approved increase in exam fee to $100 effective for 2005 exam

Content Outline Revision

• 1996 version used for 2004 Exam

• 2003 version used for 2005 Exam

• New Format: “Organocentric”

• Both versions on ASA website

– 2004 version disappears 12/31/04– www.asahq.org/publicationsandservices/contentoutlinerev2003

FAQs/Future Issues

• Computerized Exam?

• Phase out K-types?

• Introduce R-types and question

clusters (G types) in 2005

Q. Which of the following foods grows on trees?

1. Bananas

2. Tomatoes

3. Apples

4. Freedom (French) Fries

A:1,2,3 B:1,3 C:2,4 D:4 E:1,2,3,4

Answer: B

K-type evolution

• Always on exam: approx 40% in 1970s and 1980s

• Gradual decrease to current 20%

• Elimination being considered: primary rationale for evaluating R-types

• NBME dislikes, controversial among Joint Council members

Disadvantages to K-types

• No longer on USMLE

• Take longer to answer (approx 60 vs 50 sec)

• Slightly more difficult than A-types

• Examinees dislike them

Advantages to K types

• Question performance has been good – Discriminates stronger vs weaker examinees

pretty well, ideal K-type requires greater depth of understanding

• Anesthesiology doesn’t always lend itself to single best answer

• May take 3-4 A-types to test the same information as a single well-written K-type

K-type (interim) conclusions

• See how R-types perform

• Hold the line at 15-20% for now, reassess over next two years

• Let’s base K-type fate more upon the need for knowledgeable anesthesiologists than upon question popularity among examinees– Remember that they have 3-4 years to practice

Extended Matching (R-type)

A. Acute DIC

B. Hemophilia A

C. Platelet function defect

D. Idiopathic thrombocytopenia purpura

E. von Willebrand’s disease

F. Antithrombin III deficiency

1. 70 year-old man undergoing CABG/AVR is

oozing after CPB

Answer: C

1. A 70 year-old man undergoing CABG/AVR develops a coagulopathy after CPB.

Extended Matching (R-type)

A. Acute DIC

B. Hemophilia A

C. Platelet function defect

D. Idiopathic thrombocytopenia purpura

E. von Willebrand’s disease

F. Antithrombin III deficiency

Answer: B[Continue with additional questions]

An active 30 year-old man with a history of hemarthrosesand excessive bleeding after small cuts presents for

appendectomy.

R-types and G-types

Examples soon will appear on ASA Web site

Nonstandard Examinations

• Extended time, quiet room, reader, etc.• Application processing takes more time (4

months before exam)• Process described in ABA Booklet of

Information, Section 7.01• Same process for ITE as ABA written exam• Once approved by ITE, ABA usually

rubber-stamps it (same committee)

Nonstandard Exam: Concerns

• Numbers gradually increasing• Candidates often wait for ABA exam to request

accommodation• Candidates/examinees often allow insufficient

time• Candidates/examinees presume that a previous

accommodation automatically merits an ABA/ASA or ABA accommodation

Nonstandard Exam Concerns

• Recognize that A.D.D. does not automatically qualify for extended test time

• Recognize that psychological testing must be <5 years old

• Comments from candidates and from our expert consultants at times scare the Nonstandard Exams Committee

“When a resident fails to measure up to the academic standards and/or professional behavior expected, it is incumbent upon the program director to document these failures during the Residency Review Committee-mandated semiannual review.”

Miller SH, Plastic Reconst Surg 1990

Communications

gravlee.1@osu.edu

614-293-9081

Therese O’Donnell-Leonard