Gordon Page University of British Columbia Train the Trainer Workshop October 29, 2003 Key Features...

Gordon Page University of British Columbia

Train the Trainer WorkshopOctober 29, 2003

Key Features

Hong Kong International Consortium

What are key feature problems: what do they assess?

Why use key features problems: clinical competence/expertise issues?

Why use key features problems: measurement/assessment issues?

How do you write key features problems?

Overview

What are key feature problems : what do they assess?

Why use key features problems : clinical competence/expertise issues?

Why use key features problems : measurement/assessment issues?


Overview

Sample Key Features Case A 9-month-old infant presents to the Emergency Department with a two-day history of difficulty breathing, decreased appetite and irritability. Today the infant has refused all feeds.

On initial physical examination the infant is pale with poor peripheral circulation, perioral cyanosis, intercostal and subcostal retractions. Whilst you are examining the child you observe a brief apneic spell of three seconds. Vital signs as follows: Pulse 140/minute, regular. Blood pressure 60/40 Respiratory rate 65/minute Temperature 37.9° C Pulse oximetry 88%

1. What clinical problems would you focus on in your immediate management? List up to three.

1.____________________________________2.____________________________________3.____________________________________

2. What management would you initiate at this time? Select up to three. 1. Intravenous normal saline. 2. Intravenous 2/3-1/3. 3. Bolus of normal saline. 4. IV antibiotics. 5. Intubation. 6. O2 by nasal prongs. 7. Nebulized inhalation of a bronchodilator. 8. Chest x-ray. 9. Cardiac ultrasound. 10. Cardiac catheterization. 11. Bag and mask. 12. Call for help (ICU staff, anaesthesia). 13. Lumbar puncture. 14. CT head. 15. Nasogastric tube. 16. Social Services consult.

• A paper and pencil clinical problem

• A case scenario typically followed by 2 or 3 questions

• Questions assess clinical decisions and actions (not underlying knowledge)

• Questions prepared only in relation to the critical elements ("key features") in the resolution of the problem

What is a Key Features Problem?

Key features

Relative to this problem, the clinician should: 1. Clinically recognize signs of impending respiratory

failure in an infant and associated complications including congestive cardiac failure & dehydration.

2. Realize that immediate treatment may need to

precede a complete evaluation of the patient and avoid unnecessary investigation in the acute phase.

1. What clinical problems would you focus on in your immediate management? List up to three.

1.____________________________________2.____________________________________3.____________________________________

2. What management would you initiate at this time? Select up to three. 1. Intravenous normal saline. 2. Intravenous 2/3-1/3. 3. Bolus of normal saline. 4. IV antibiotics. 5. Intubation. 6. O2 by nasal prongs. 7. Nebulized inhalation of a bronchodilator. 8. Chest x-ray. 9. Cardiac ultrasound. 10. Cardiac catheterization. 11. Bag and mask. 12. Call for help (ICU staff, anaesthesia). 13. Lumbar puncture. 14. CT head. 15. Nasogastric tube. 16. Social Services consult.

What are key feature problems : what do they assess?

Why use key features problems : clinical competence/expertise issues?

Why use key features problems : measurement/assessment issues?


Overview

Reduced: knowledge absent/disorganized

Dispersed: many individual facts -- rote lists, little understanding/integration

Elaborated: knowledge organized, integrated & structured – effective reasoning

Compiled: knowledge structured around clinical prototypes; problem recognition and

resolution through pattern recognition

Clinical Decision-Making

(Bordage, G. Academic Medicine, 1994, 1999)

Reduced “empty mind”

Dispersed “cluttered mind”

Elaborated “deductive thinking”

Compiled “recall/pattern recognition”


(Bordage, G. Academic Medicine, 1994, 1999)

• Reduced• Dispersed

• Elaborated• Compiled


• Inter-case correlation = .10 – .30

• Each case presents unique challenges (Arthritis Anemia Crohns Diabetes)

Case Specificity

Key features problems

• Assess effectiveness, not thoroughness

• Assess only key steps (usually 2 – 4) in the resolution of a problem

• Permit wide sampling of problems to address “case specificity”

What are key feature problems: what do they assess?

Why use key features problems: clinical competence/expertise issues?



Overview

Reliability – “Focussed" problems – better sampling, more accurate assessment

Content Validity – assess the most important clinical decisions within (a representative sample of) problems

Key features problems: measurement/assessment issues?

Application of Knowledge?

Mrs. Stevens has a herniated disc in her cervical spine.

What would you expect to find on your examination of Mrs. Stevens?

What symptoms is Mrs. Stevens likely to be experiencing?

Top-Down Bottom-Up

Problem Problem

Clinical ClinicalFeatures Features


Content Validity – assessing the most important clinical decisions within (a representative sample of) problems

“Bottom-up” thinking – assessing knowledge application and what clinicians do in real life!

Why use key features

(Key Features problems: measurement/assessment issues?)

Performance & Length of Option Lists

5-Option

15-Option

30 40 50 60 70 80 90 100

0

5

10

15

20

PERCENT CORRECT SCORE

PE

RC

EN

T O

F S

TU

DE

NT

S

Performance on Free Response vs Selected Response Items

50

60

70

80F

ail

Bo

rde

rlin

eF

ail

Bo

rde

rlin

eP

ass P

ass

Pe

rce

nta

ge

Sc

ore

SM

WI

Non VignetteWhat is the most likely renal abnormality in children with nephrotic syndrome and normal renal function?

(A)acute poststreptococcal glomerulonephritis (B) hemolytic-uremic syndrome*(C) minimal change nephrotic syndrome (D)nephrotic syndrome due to focal and segmental glomerulosclerosis (E) Schönlein-Henoch purpura with nephritis

Short VignetteA 2-year-old boy has a 1-week history of edema. Blood pressure is 100/60 mm Hg, and there is generalized edema and ascites. Serum concentrations are: creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows 4+ protein and no blood. What is the most likely diagnosis?

Long VignetteA 2-year-old black child developed swelling of his eyes and ankles over the past week. Blood pressure is 100/60 mm Hg, pulse 110/min, and respirations 28/min. In addition to swelling of his eyes and 2+ pitting edema of his ankles, he has abdominal distension with a positive fluid wave. Serum concentrations are: creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows 4+ protein and no blood.

Case SM, Swanson DB. Constructing Written Test Questions for the Basic and Clinical Sciences, 1996. Page 58-9.

What is the most likely renal abnormality in children with nephrotic syndrome and normal renal function?

A B *C* D E Overall P-value1 0 99 0 08 1 90 1 0 94

A 2-year-old boy has a 1-week history of edema. Blood pressure is 100/60 mm Hg, and there is generalized edema and ascites. Serum concentrations are: creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows 4+ protein and no blood. What is the most likely diagnosis?

0 0 98 2 0

5 2 82 8 1 88

A 2-year-old black child developed swelling of his eyes and ankles over the past week. Blood pressure is 100/60 mm Hg, pulse 110/min, and respirations 28/min. In addition to swelling of his eyes and 2+ pitting edema of his ankles, he has abdominal distension with a positive fluid wave. Serum concentrations are: creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows 4+ protein and no blood.

0 1 98 1 0

10 9 66 10 5 84

Case SM, Swanson DB. Constructing Written Test Questions for the Basic and Clinical Sciences, 1996. Page 58-9.


Content Validity – assessing the most important clinical decisions within (a representative sample of) problems

“Bottom-up” thinking – assessing knowledge application and what clinicians do in real life!

“Fidelity” & discriminating power – longer lists of options, free response questions, raw data in case scenarios

Why use key features

(Key features problems: measurement/assessment issues?)

Overview of the Presentation

What are key feature problems:what do they assess?

Why use key features problems:clinical competence/expertise issues?



Key Features Problems

Page, G. and Bordage, G. “Developing Key Features Problems and Examinations to Assess Clinical Decision-Making Skills”, Academic Medicine, 70:3, 1995, 194-201.

STEPS IN PREPARING KEY FEATURES CASES

1. Select a Clinical Problem (e.g., diarrhea)

2. Select a Clinical Situation

Step 2: Clinical situations

How does problem present?

Undifferentiated complaint

Single problem, typical/atypical

Multiple problem, multi-system

Urgent, life-threatening problem

Prevention, health promotion


1. Select a Clinical Problem

(e.g., diarrhea)

2. Select a Clinical Situation

(e.g., single problem, typical resentation)



2. Select a Clinical Situation (e.g., single problem, typical resentation)

3. Define the problem's Key Features

Define the Problem’s “Key Features”

• The critical or essential steps in the resolution of the problem

***********************************

• The elements or steps most likely to lead to errors

• The difficult aspects of problem identification and management

Key Features

Relative to the problem diarrhea, the clinician should: 1. Recognize dehydration and its level of severity.

2. Manage dehydration appropriately.

3. Evaluate the possible communicability of the underlying disease (family/hospital spread, possible common source).



2. Select a Clinical Situation (e.g., single problem, typical resentation)


4. Select a Case to represent the Problem

Key Features Case: Diarrhea

A 35-year-old mother of three presents to your office at 1700h

with complaints of severe, watery diarrhea. On questioning,

she indicates that she has been ill for about 24 hours. She has

had fifteen watery bowel movements in the past 24 hours, has

been nauseated, but not vomited. She works during the day as a

cook in a long-term care facility but left work to come to your

office. On her chart, your office nurse notes a resting blood

pressure of 105/50 mm Hg supine, (a pulse of 110/minute), 90/40

standing, and an oral temperature of 36.8°C. On physical

examination, you find she has dry mucous membranes and

active bowel sounds. A urinalysis (urine microscopy) was

normal, with a specific gravity of 1.030.



2. Select a Clinical Situation (e.g., single problem, typical presentation)



5. Develop Questions to test only the key features

6. Select the Format for the questions

1. What clinical problems would you focus on in your immediate management of this patient? List up to three.

1.____________________________________2.____________________________________3.____________________________________

2. How should you treat this patient at this time? Select up to three.

1. Antidiarrheal medication 2. Antiemetic medication 3. Intravenous 0.9% NaCl 4. Intravenous 2/3 to 1/3 5. Intravenous gentamicin 6. Intravenous metronidazole 7. Intravenous Ringer lactate 8. Nasogastric tube and suction 9. Nothing by mouth

10. Oral ampicillin11. Oral chloramphenicol12. Oral fluids13. Rectal tube14. Send home with close follow-up15. Surgical consultation16. Transfer to hospital

3. After management of the patient's acute condition, what additional measures, if any, would you take? Select up to four or select #11, None, if none is indicated.

1. Avoid dairy products 2. Colonoscopy 3. Enteric precautions 4. Gastroenterology consultation 5. Give immune serum globulin to patients at

long-term care facility 6. Infectious disease consultation 7. Notify Public Health Authority 8. Stool cultures 9. Strict isolation of patient

10. Temporary absence from work11. None








7. Prepare Scoring Keys

Question 1 Scoring Key

Score Keyed Responses Synonyms 1.0 Dehydration Hypovolemia

fluid loss fluid depletion

0 Listing more than three items

Question 3 Scoring Key

Score Keyed Responses

0.25 each # 3. Enteric precautions

# 8. Notify Public Health Authority

#11. Stool cultures

#13. Temporary absence from work

0 # 5. Give immune serum globulin to patients

at long-term care facility

#12. Strict isolation of patient

or

Selecting more than four items








7. Prepare Scoring Keys

• Adequate sampling across problems

• A focus on key steps in problem resolution (vs. thoroughness)

• Questions/question formats that present “high fidelity” clinical challenges “bottom-up” application of knowledge

free response questions

longer option lists

presentation of “raw” clinical data

The assessment of clinical decision-making is enhanced by:

Summary

Gordon Page University of British Columbia Train the Trainer Workshop October 29, 2003 Key Features...

Documents

Transcript of Gordon Page University of British Columbia Train the Trainer Workshop October 29, 2003 Key Features...