An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the...
-
date post
19-Dec-2015 -
Category
Documents
-
view
215 -
download
0
Transcript of An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the...
![Page 1: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/1.jpg)
An overview of Assessment
![Page 2: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/2.jpg)
Aim of the presentation
• Define and conceptualise assessment
• Consider the purposes of assessment
• Describe the key elements of a good assessment
• Describe two main types of assessment
![Page 3: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/3.jpg)
Definitions of assessment
• ‘… the process of documenting, often times in measurable terms, knowledge, skills, attitudes and beliefs.’
• ‘The classification of someone or something with respect to the worth’
![Page 4: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/4.jpg)
Other words that can mean “assessment”
• Appraisal
• Categorisation
• Evaluation
• Judgement and value judgement
• Adjudication
• Estimation
![Page 5: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/5.jpg)
In medical educational terms we can think of assessment as the process by which knowledge, skills and behaviours may be tested and judgments made
about competence or performance.
![Page 6: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/6.jpg)
Purposes of Assessment
![Page 7: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/7.jpg)
• Measuring competence
• Diagnosing a student’s/trainee’s problems
• Measuring improvement
• Showing the effectiveness of the curriculum/quality assurance
• Introducing curriculum change
• Identifying effective teaching
• Self-evaluation• Ranking• Motivation for teachers
and learners• Progress testing
(developmental measures of improvement)
• Deriving income
![Page 8: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/8.jpg)
The principles of assessment
![Page 9: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/9.jpg)
What should be assessed?
![Page 10: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/10.jpg)
The curriculum and assessment
The curriculum & the outcomes should define assessments
The ideal assessment fits the curriculum
![Page 11: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/11.jpg)
• In real life there is often a less than perfect match
• The objectives are not fully and transparently defined
• Students will define theirown “hidden curriculum” and may miss elements that are tested
![Page 12: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/12.jpg)
Assessing all elements of the curriculum
![Page 13: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/13.jpg)
A simple model
Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S67.
Knows
Shows how
Knows how
Does
Cognition
Behaviour
![Page 14: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/14.jpg)
Knows
Shows how
Knows how
Does
The assessment pyramid
Knowledge: Written tests
Competence: OSCEs
Performance:Workplace
![Page 15: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/15.jpg)
Knows
Shows how
Knows how
Does
The emerging assessment pyramid
Meta-cognition
![Page 16: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/16.jpg)
Constructing a “good” test
![Page 17: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/17.jpg)
U= Rx Vx Ex Cx A
U= utilityR= reliabilityV= validityE= educational impactC= costsA=acceptability
van der Vleuten (1996) The assessment of professional competence:developments,research and practical implications. Adv Health Sciences Education. 1 (1) 41-67
![Page 18: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/18.jpg)
Reliability:If this test were administered again
would the results be the same?
• REPEATABILITY/REPRODUCABILITY– Would the assessors make the same
judgements?– Would a different set of questions result in a
significantly different score?– Were there other things that may have
influenced the result?
![Page 19: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/19.jpg)
Validity: Does the test measure what it thinks it is measuring?
• Face validity– Does it look appropriate?
• Content validity– Does it only assess what you want to
assess?
• Criterion validity– Is it predictive of future performance?
![Page 20: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/20.jpg)
Educational impact
Curriculum Assessment
StudentTeacher
From: Lambert Schuwirth
![Page 21: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/21.jpg)
Issues that affect educational impact
• The content– What is taught and learned
• The format– Written, clinical
• Scheduling– When things happen
• Regulatory structure– Assessment rules & information
![Page 22: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/22.jpg)
Cost
Balancing the cost/benefit
![Page 23: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/23.jpg)
Acceptance
• Political issues and support within the faculty
• Acceptance by the students
• Perceptions of fairness
![Page 24: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/24.jpg)
Models of assessment
![Page 25: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/25.jpg)
Formative assessment
• Provides information to the candidates about their strengths and weaknesses
![Page 26: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/26.jpg)
Summative Assessment
• A measure of an end point achievement
• Even summative tests can (and should) be formative
• Must be robust and defensible
![Page 27: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/27.jpg)
Assessment guidelines to achieve a high utility index
![Page 28: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/28.jpg)
Assessment guidelines
• Establish the purpose• Define what is to be tested• Blueprint to guide selection of items• Select most appropriate test method/format• Administration and scoring• Standard setting
Newble DI, Jolly B, Wakeford R, eds. The certification and recertification of doctors: issues inthe assessment of clinical competence. Cambridge: Cambridge University Press; 1994.
![Page 29: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/29.jpg)
Using the guidelines at Sheffield• Establish the purpose
• Define what is to be tested
• Blueprint to guide selection of items
• Select most appropriate test method/format
• Administration and scoring
• Standard setting
• To graduate safe/competent junior doctors
• Outcome objectives for core clinical problems
• Assessments of knowledge & skills are blueprinted
• Range of tests used• Centralised assessment
management• Hofstee/borderline methods
![Page 30: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/30.jpg)
Strategic management of assessment
Assessment Committee
Curriculum Committee
Phase 4Phase 3Phase 2Phase 1
Administrators/academics
![Page 31: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/31.jpg)
Assessments used at Sheffield
• Formative– On-line weekly tests– Mini Cex
• Summative– Extended Matching Questions– Modified Matching Questions– Observed long cases– Objective Structured Clinical Examination– Professional behaviour– Assessments of Student Selected Components
![Page 32: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/32.jpg)
Formative assessments
• Weekly on-line tests (EMQs)– Face-to-face feedback– On-line personal results and peer
performance
![Page 33: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/33.jpg)
Formative assessments
• Mini Cex (Mini Clinical Evaluation Exercise)– USA: National Board of Medical Examiners– Performance testing in real practice– Multiple observations/multiple observers– Longitudinal assessment to assess
professional growth– Feedback is inherent in the assessment
method
![Page 34: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/34.jpg)
Mini Cex
Excellent Satisfactory Borderline Unsatisfactory
History taking
Physical examination
Communication skills
Professionalism
Overall clinical competence
![Page 35: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/35.jpg)
Summative assessments
• End of year tests of knowledge– Extended Matching Questions– Modified Essay Questions
• Tests of clinical competence– Observed long cases– OSCE
• Assessment of professional behaviour• SSCs – reports, presentations, essays,
posters, leaflets
![Page 36: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/36.jpg)
Observed long case
• 2 Observed long cases during final clinical attachments
• Successful completion entry requirement for the OSCE
• A “mastery” test – may be repeated
![Page 37: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/37.jpg)
OSCE
• Separated from knowledge test by 4 months• 12 stations
– History - 3 stations – Physical examination - 6 stations– Communication - 2 stations– X-ray interpretation – 1 station
Was: 15 stations 5 mins eachNow: 12 stations 10 mins each
• Checklist and global rating scoring
![Page 38: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/38.jpg)
Professional behaviours
• Longitudinal assessment of professional behaviours to give reliability
• Assessed against outcome objectives and Good Medical Practice (GMC 2001)
• Borderline/unsatisfactory performance triggers interview
• Consistent poor performance may lead to review by Fitness to Practise committee and possible exclusion
![Page 39: An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649d405503460f94a1a319/html5/thumbnails/39.jpg)
Student Selected ComponentsSSCs
• 25% of the overall course
• Beyond the core curriculum (depth +/- breadth)
• Various modes of assessment
• Assesses generic skills, critical analysis, ethics, research skills, clinical understanding