DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer...
-
Upload
kathleen-cummings -
Category
Documents
-
view
216 -
download
0
Transcript of DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer...
![Page 1: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/1.jpg)
DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE
School of Medicine, Flinders University
Train the Trainer WorkshopOctober 28, 2003
The sit uati on of Moun t Lof ty was fo
f rom hence and from some ot her cr
beari ngs, to be 34¡ 59 ' south and 13
east. N o l and w as vi si bl e so f ar to th
nort h as where t h e trees appeare d a
t he horizon , w hi ch show ed the coas
be v er y low, and our soundings we
f ast dec reasing.
noon t o six o'cl ock we ran th irt y t o t he nor thward, sk irt ing a sandy at the d istance of fi ve, and t he nce t mi les; the dept h was then 5 s, and we dropped the anchor upon
om of sand , mi xed wit h p ieces of o ral.
CONSTRUCTED RESPONSE ITEMS:
MEQ & SAQ
Hong Kong International Consortium
![Page 2: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/2.jpg)
![Page 3: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/3.jpg)
![Page 4: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/4.jpg)
![Page 5: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/5.jpg)
CONSTRUCTED RESPONSE ITEMS
ASSUMPTIONS
• Higher cognitive processes
• Increased reliability through wider sampling
ARE THE ASSUMPTIONS JUSTIFIED?
![Page 6: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/6.jpg)
MODIFIED ESSAY QUESTIONS: MEQs
• 1970s – 1980s
• RCGP (Hodgkin & Knox 1975)
• Newcastle Australia (Feletti 1980, Feletti and Smith 1986)
• Jefferson USA (Rabinowitz 1987, Rabinowitz and Hojat 1989)
![Page 7: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/7.jpg)
• Learning objectives, concepts, deep structures
• Clinical scenario
• 7 – 10 questions
• Sequential
(information question information question ...)
• Integrated (content, media – Rabinowitz 1985)
MODIFIED ESSAY QUESTIONS: MEQs
![Page 8: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/8.jpg)
HIGHER COGNITIVE ORDER(Bloom 1956, Buckwalter et al 1981)
1. Knowledge Level 1 Recognition Recall
2. Comprehension Level 2 ComprehensionInterpretation
3. Application
4. AnalysisLevel 3 Application
5. Synthesis
6. Evaluation
(Irwin and Bamber 1982)
MODIFIED ESSAY QUESTIONS: MEQs
![Page 9: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/9.jpg)
LEVEL 1 RECOGNITION AND RECALL
Know
• What is …..?
• List …..
• Identify …..
MODIFIED ESSAY QUESTIONS: MEQs
![Page 10: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/10.jpg)
MODIFIED ESSAY QUESTIONS: MEQs
LEVEL 2 COMPREHENSION, INTERPRETATION
Understand
• Give reasons ….
• Explain ….
• Interpret ….
![Page 11: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/11.jpg)
LEVEL 3 APPLICATION
Apply, Utilise, Judge
• How does X assist in explaining Y?
• What are the components of X?How are they related?How do they affect Y?
• How does XYZ affect treatment and management?
• Assess, judge X in terms of ….
MODIFIED ESSAY QUESTIONS: MEQs
![Page 12: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/12.jpg)
MODIFIED ESSAY QUESTIONS – VALIDITY
• CONTENT
(Blueprints, objectives, deep structures)
• CONSTRUCT
(Level 1, 2, 3 questions)
![Page 13: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/13.jpg)
MODIFIED ESSAY QUESTIONS – VALIDITY
• CONSTRUCT
Norman et al 1987Effect of educational levelMCQ and MEQ not different skillsUEQ may be a different skillSmall effect of context
Feletti 1980More Level 1 and 2 in first year
undergraduate
Irwin and Bamber 1986More Level 3 in final year undergraduate
![Page 14: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/14.jpg)
• PREDICTIVE
Irwin and Bamber 1986 Correlation with clinical examination 0.41
Rabinowitz 1987 MCQ better predictor NBME I & II
Rabinowitz and Hojat 1989 Higher correlation for MEQ than MCQ with overall clerkship ratings
MODIFIED ESSAY QUESTIONS – VALIDITY
![Page 15: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/15.jpg)
• Inter-rater reliability (model answers)
Norman et al 1987Inter-rater reliability satisfactory
• Internal consistency (cumulative errors)
Feletti 1980Alpha(60) 0.57 – 0.91
Stratford and Pierce-Fenn 1985Alpha(60) 0.39
MODIFIED ESSAY QUESTIONS – RELIABILITY
![Page 16: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/16.jpg)
• Administration
• Marking
• Higher order questions
MODIFIED ESSAY QUESTIONS – FEASIBILITY
![Page 17: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/17.jpg)
• Model PBL process
• Determine objectives
• Construct scenario
MINI-CASES: THE FLINDERS MEQ
![Page 18: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/18.jpg)
John Taylor (age 55) has been brought to the E.D. by his wife. John
woke up this morning (one hour ago) with nausea and vertigo. He
managed to walk to the bathroom but had to hold onto the furniture
and walls to steady himself. He vomited when he reached the
bathroom.
1. Give 3 hypotheses to account for John’s condition.
2. Explain the mechanisms which give rise to John’s condition.
3. What further information would you like to distinguish between hypotheses. Explain how this information will distinguish between hypotheses.
MINICASES: THE FLINDERS MEQ
![Page 19: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/19.jpg)
• Provide further information
• Cumulative error cues
• Continue scenario
MINI-CASES: THE FLINDERS MEQ
![Page 20: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/20.jpg)
John has had no significant medical problems in the last few years. He spent the previous evening at home watching TV and did not drink any alcohol or take any other substance.
….. The left pupil is smaller than the right …..
….. John’s uvula deviates to the right …..
….. Movements of the left hand are clumsy.
MINICASES: THE FLINDERS MEQ
![Page 21: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/21.jpg)
• Further questions
• Continue unfolding scenario
• Page breaks, cumulative cues
• Conclude with SAQs (Level 2/3)
MINI-CASES: THE FLINDERS MEQ
![Page 22: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/22.jpg)
• Face validity
• Construct validity
• Model answers
• Higher order questions
MINI-CASES: THE FLINDERS MEQ
![Page 23: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/23.jpg)
• Reform of RCGP MEQ– Swanson– Mullholand & McAleer
• Increase number (10 – 12 / 2 hours)– Reflect reality
• Increase independence
• Single marked – schedules
• SAQ
(Lockie et al 1990)
SHORT ANSWER QUESTIONS - SAQ
![Page 24: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/24.jpg)
Evaluation: “The ability to make a value-judgement in relation to
relevant criteria”
Bloom 1956
contextual information + Q responsecriteria
Opportunity to target highest cognitive skills
• application - analysis - synthesis - evaluation
Flinders SAQ strategies
![Page 25: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/25.jpg)
1. Provide multiple, detailed, relevant information
• Patient
• Problem
• Clinical care setting
• Your role
contextual information + Q responsecriteria
Targeting “evaluation” response
![Page 26: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/26.jpg)
• Ron Harkin is a 32 year-old father of two and self-employed
gardener. He falls heavily. He is unable to stand and is in
considerable pain. He is taken to hospital where you are the Intern
in the Emergency Department. You note swelling and that his foot
appears to be at an unusual angle to his leg.
• A 7 year-old Aboriginal girl, Alice Mundi, presents with a mild
fever and painful knee to a small rural hospital. Her mother has
brought her 50km to the hospital. You are the GP called to see
her.
Example scenario information
![Page 27: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/27.jpg)
2. Pose question requiring value-judgement• focus the response
• assume criteria are understood
• relate to what you would do
3. Extend information / probe further (optional)
contextual information + Q responsecriteria
Targeting “evaluation” response
![Page 28: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/28.jpg)
Example questions targeting “evaluation”
Rationalising the diagnosis
• What further information from the history do you require?
• On what aspects of the physical examination will you focus?
• What investigations would you undertake to establish the diagnosis?
• What are your differential diagnoses? Which do you consider most likely?
![Page 29: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/29.jpg)
Formulating an appropriate management plan
• What would be your immediate management goals?
• Which treatments would you advise?
• For what potential complications would you monitor?
• What longer-term management strategy would you advise?
• What advice would you give in relation to this new development?
Example questions targeting “evaluation”
![Page 30: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/30.jpg)
Mr Dawes, a 58 yo man, presents to the Emergency Dept with a 24h history
of vomiting and colicky abdominal pain. He has had no bowel movement for
24h. He had an appendicectomy at age 30 to remove a gangrenous
appendix. On examination you find he has a distended abdomen with no
focal tenderness. There is no palpable hernia. He has symptoms and signs
consistent with dehydration of about 10%. Fluid resuscitation is
commenced. A plain abdominal Xray shows distended loops of small bowel
with air fluid levels and normal calibre large bowel.
Q1. What is your most likely diagnosis for this patient? (2 marks)
Q2. What possible causes of this would you consider? (3 marks)
After 48h of conservative management, Mr Dawes’ condition is such that surgical intervention is warranted.
Q3. What clinical indications would suggest to you that Mr Dawes needs surgery to resolve his problem at this time?
(5 marks)
![Page 31: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/31.jpg)
Fairly reward sound responses
• Model answer (checklist)
• Writers / markers who understand:
- curriculum content
- learner stage
• Ongoing item improvement
Flinders SAQ strategies
![Page 32: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/32.jpg)
• Are the claims justified?
• Do they measure anything different?(Maquire, Skakun, Triska 1997)
• What role do they have in a balanced assessment program?
CONSTRUCTED RESPONSE ITEMS
![Page 33: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/33.jpg)
REQUIREMENTS
• Write to a blueprint
• Sample widely
• Increase level 2 and 3 questions
• Model answers
CONSTRUCTED RESPONSE ITEMS
![Page 34: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/34.jpg)
THE END
![Page 35: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/35.jpg)
You see Rebecca and Robert who are residents at a local training
centre for 27 young mentally handicapped patients. They are
accompanied by their care helper. Rebecca and Robert have
scabies.
Discuss your management
• communication
• clinical management
• health in the community
• sensitivity to the mentally handicapped
• use of extended team
• ethical issues
Deep/Surface Structures
(Lockie et al 1990)
SHORT ANSWER QUESTIONS - SAQ
![Page 36: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/36.jpg)
VALIDITY
Construct
Wright et al (1983) – Independent cognitive abilities
Predictive
Wass et al (2001) – correlate with clinical tests
RELIABILITY
Low to moderate
Des Marchais and Vu (1996)
Wakeford and Roberts (1984)
Good
Wright et al (1983) – marking schedules
SHORT ANSWER QUESTIONS - SAQ
![Page 37: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/37.jpg)
FEASIBILITY
• Marking
• Higher order questions
SHORT ANSWER QUESTIONS - SAQ
![Page 38: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/38.jpg)
• One clinical scenario – multiple questions
PAQ (Des Marchais and Vu 1996) – no sequence
SAQ (Flinders) sequence
• Level 1, 2 and 3 questions on one scenario
• Reliability
PAQ – low to moderate
• Feasibility
HYBRID APPROACHES
![Page 39: DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649eb05503460f94bb6613/html5/thumbnails/39.jpg)
• Face validity
• Construct validity
• Model answers
• Higher order questions
THE FLINDERS HYBRID MODEL