ASSESSMENT IN GP/FM
description
Transcript of ASSESSMENT IN GP/FM
ASSESSMENT IN GP/FM
Mladenka Vrcic-Keglevic, MD, PhD
Head of Family Medicine Department, Medical School, Univerity of Zagreb
Content of the session
• Definition of the terms• How to use assessment (summative,
formative)• Assessment methods (AM)• What can be assessed by different AM• Metric’s characteristics of AM• …..• Interactive session
Definition of the terms
• Evaluation: A process that attempts to systematically determine the relevance, effectiveness, and impact of activities in light of their objectives. Can be related to structure, process or outcome of (programme, person).
• Assessment: A system of evaluation of professional accomplishments in light of educational objectives, using defined criteria and usually including an attempt at measurement either by grading on a rough scale or by assigning numerical values.
AMEE – Glosary of Medical Education Terms
Well defined educational objectives - prerequsite for assessment
Example for this session:
At the end of this session participants will be able:• To named at list three differences between
summative and formative assessment• To make a list of at least three written AM• To name the most effective AM to assess clinical
skills• To describe the most effective AM to assess
attitudes
How to use assessment?
• Summative: This is usually undertaken at the end of a training programme or teaching course and determines whether the educational objectives have been successfully achieved. With summative assessment the students usually receives a grade or a mark. Exam
• Formative: This is testing that is part of developmental or ongoing teaching / learning process. It should emanate from a wish to foster learning and understanding. It should include delivery of feedback to the student.
Summative - Examination
What the exams are?
• For students – A difficult and unpleasent steeplechase to run on a way for diploma
• For teachers – A less desirable teaching activity
• For public – An important protection from un-competent doctors
Summative assessment
The reasons:• A statement of achievement - university degree
(diploma)• An entrance requirement to an educational
institution• A guide as to the wisdom of continuing with
further study• A certification of competence – public
responsibility (licence)• A determinant of programme effectiveness
Assessment methods (AM)
Written:• MCQ - Multiple Choice
Questions
• Un-Q – An-cued Questions
• Essay• MEQ – Modified Essay
Questions
• EMI - extended matching items
• PMP – Patient Management Problem
• Oral, viva• Traditional practical• OSCE – Objective
Structured Clinical Examination
• SP - Standardised Patient
• Portfolio• ACC – Assessment of
Clinical Competence
What can be assessed by different AM
• Factual knowledge
• Interpretations
• Problem-solving skills
• Ethical
• Clinical skills
• Emotional reactions
• Communication
What can be assessed?
Fact Inter P-S Ethi Skill Emo Com
Oral ++ + +/- - - - - + +
Practic. + ++ ++ + - + + - + -
MCQ ++ + - + - - - - - - - - -
Un-Q ++ + - + - - - - - - - - -
MEQ + ++ ++ + - - - - - - -
Essay ++ + - + - + - + - - - - -
OSCE + + + + - + - ++ + - + -
PMP +- ++ ++ +- - - - - - -
ACC ++ ++ ++ ++ ++ ++ ++
Metric characteristics of AM
• Validity - the degree to which a measurement instrument truly measures what it is intended to measure
• Reliability – it is an expression of the precision, consistency and reproducibility. Ideally, measurements should be the same when repeated by the same student or made by the different assessors.
• Relevance – it is a degree to which the assessment
questions and educational objectives are in concordance • Objectivity – it is a level of agreement among independent
assessors (experts) about the right answer to certain question
Metric’s characteristics?
Validity Reliability Relevent Objectivity
Oral - - - - - - - -
Practic. - - + -
MCQ ++ ++ ++ ++
Un-Q ++ ++ ++ ++
MEQ ++ ++ ++ ++
Essay - - + -
OSCE ++ ++ ++ ++
PMP/SP ++ ++ ++ ++
ACC - + - + + - +
Practicality?
• Preparation – resources (time, human…)
• Easy to perform, aplay
• NC – number of candidates
• NC/teacher – number of candidates per one teacher
Practicality
Preparation Ease to applay
NC- No of candidates
NC / teacher
Oral ++ - - - - - -
Practic. ++ + - - - -
MCQ +++ ++ ++ ++
Un-Q +++ ++ ++ ++
MEQ - ++ ++ -
Essay + + + - -
OSCE + + + +
PMP/SP + + - ++ +
ACC ++ + - - - -
MCQ – Multiple Choice Questions
• Difficulty index
• Discrimination index
• Minimum pass level
Indeks of difficultyQD = (G + P)/N
• QD – question’s difficulty• G – number of right
answers in good group• P – number of right
answers in poor group• N – number of students in
both groups
a b c d e 0 U
1
2
3
Difficulty Index
0,3 0,5 0,6 0,7
------/---------------(------------)----------/--------- recommended
--------------------------------------------------------------
acceptable
too difficult too difficult
Discrimination Index
• DI = 2 (G-P)/N• DI – discrimination Index• G – number of right
answer in “good” group• P – number of right
answer in “poor” group• N – number of students in
both group
a b c d e 0 U
1
2
3
Diskrimination Index
0.15 0.25 0.35
----------/----------/----------/---------- throw off to check good excelent
What AM to use in reality?
• There is no perfect AM• A choice depends of what is to be measured –
educational objectives (knowledge, skills, competences, attitudes)
• Careful preparation, trained assessors• Multiple methods – better then one
Specialistic exam in GP in Croatia
Should be submeted in advance:• Portfolio• 8 essays• Questionary of professional competence –
signed by trainer• Trainer’s report about trainee’s achievements• Trainee’s report – How to organise my practice,
what to invent?• Medical records (videos) of 3 patients / family
Specialistic exam in GP in Croatia
• MCQ test – 150 questions
• OSCE – 20-25 stations
• Oral – jury (3 members): 1 academic GP + 1 well recognised GP (publication) + 1 trainer
Formative assessment
The reasons:
• Information for the student about his/her achievement of educational objectives
• Repetitive – progress measurement
• Discover “week points”–teacher’s support
• Help to a teacher to correct programme
• The results should not be used in summative assessment
Formative assesssment -which AM?
• The same as for summative – progress test• Protocols for observing clinical work
(consultation, communication, p-s abilities, professional competences)
• Checklists• Diaries – what has been done• Appraising projects, audit…• Portfolio• Self and peer assessment
Formative assesssment
• Feedback
• Feedback
• Feedback
• Feedback
• Feedback
Educational objectives in this session – achieved?
At the end of this session participants will be able to:
• name at least three differences between summative and formative assessment
• make a list of at least three written AM• name the most effective AM to assess
clinical skills• describe the most effective AM to assess
attitudes