Psychometrics for Clinical Skills Assessment
-
Upload
inspirenetwork -
Category
Education
-
view
232 -
download
3
description
Transcript of Psychometrics for Clinical Skills Assessment
Psychometrics for Clinical Skills Assessment
Serkan Toy, PhDDirector of Evaluation and Program Development
Graduate Medical Education
Children’s Mercy Hospital – Kansas City
Outline
• What is psychometrics?– Measurement– Construct development
• Reliability
• Validity
• A few other issues to consider– Checklists vs. Global ratings
Psychometrics
• Educational & Psychological Measurement– measurement of knowledge, abilities,
attitudes, and personality traits – mainly concerned with the construction and
validation of measurement instruments (i.e. cognitive tests, surveys/questionnaires, and personality assessments.)
Measurement• Assigning numerals to observations based on
some pre-defined criteria
• Or assigning a value to one object/observation in relation to another – Intelligence - IQ– Personality - Big Five– Academic or procedural performance
Measurement
Subjective Objective
Qualitative Quantitative
Measurement
Global Analytic
Construct
Well-defined vs. Ill-definedWhat do we already know about the construct in
question?
• Epistemological Beliefs
• Self efficacy
• Academic Performance
• Procedural Competency
Defining the Construct
• Factor Analysis • Cluster Analytic Approach • Cognitive/Procedural Task Analysis
– Hierarchical Task Analysis
– Delphi Technique - expert consensus (face validity)
– Angoff Method - determining passing (cut-off) scores
Key Concepts in Assessment
• Reliability
• Validity
Reliability“The more consistent the scores are over different raters and occasions, the more reliable the assessment is thought to be” (Moskal & Leydens, 2000 as cited in Jonsson & Svingby, 2007)
• Test re-rest• Inter-rater
• Binary vs. Likert scale • Rubrics and calibration process
Jonsson, A. & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2, 130-144.
Rubrics for Assessment
• “An assessment tool that describes levels of performance on a particular task” (Jonsson & Svingby, 2007)
• Analytic, topic-specific rubrics seem to enhance reliable scoring of performance especially when accompanied by examples and/or rater training
• Example:– Objective assessment of surgical competence in
gynaecological laparoscopy: development and validation of a procedure-specific rating scale Larsen et al. (2008)
Jonsson, A. & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2, 130-144.
Larsen C.R., Soerensen J.L., Grantcharov T.P., Dalsgaard T, Schouenborg L., Ottosen C., Schroeder T.V., Ottesen B.S. (2008) Effect of virtual reality training on laparoscopic surgery: randomised controlled trial. BMJ, 14, 338, b1802.
Larsen et al. 2008Example rating scale
1 2 3 4 5
Economy of movements
Many unnecessary movements
Efficient motion but some unnecessary motion
Maximum economy of movements
Economy of time
Too long time used to perform sufficiently
Intermediate time used to perform sufficiently
Minimal time used to perform sufficiently
Errors: respect for tissue
… … …
Flow of operation
… … …
Traditionally Validity
• Three C’s of the “Trinitarian”– Content– Criterion – Construct
Conceptual Change in Validity
“Validity is not a property of the test or assessment as such, but rather of the meaning of the test scores. These scores are a function not only of the items or stimulus conditions, but also of the persons responding as well as the context of the assessment. In particular, what needs to be valid is the meaning or interpretation of the score; as well as any implications for action that this meaning entails” (p. 741).
Messick, S. (1995) Validity of psychological assessment: validation of inferences from persons’ responses and performance as scientific inquiry into score meaning. American Psychologist, 50, 741–9.
Validity
• Construct Validity– Content / Face– Convergent – Discriminant– Predictive
The question is: What can we validly conclude about a trainee who receives a score of “X” vs. that of receiving “Y”?
Construct Validity
• In competency assessment instruments validity is usually established by examining whether they distinguish between groups logically presumed to differ on competency being measured – Experienced practitioners vs. trainees or– Peer nominated superior performers vs. average
performers
Scofield, M. E., & Yoxtheimer, L. Y. (1983). Psychometric issues in theassessment of clinical competencies. Journal of Counseling Psychology.30, 413-420.
Other Issues to Consider
• Global ratings vs. Checklist scores• Required sample size for validity testing• Training the trainees• Scoring the videotaped performances
– Individual procedural vs. qualitative skills– Team performance
• Formative and summative assessment at the same time – Training (intervention) or assessment
(measurement)?
Global ratings vs. Checklist scores
• High correlation between global ratings and checklist scores
• Both seem to differentiate similarly between more experienced trainees and novices
Examples• Kim J., Neilipovitz D., Cardinal P., Chiu M. (2009) A comparison of global rating scale
and checklist scores in the validation of an evaluation tool to assess performance in the resuscitation of critically ill patients during simulated emergencies (abbreviated as ‘‘CRM simulator study IB’’). Simulation in Healthcare, 4, 6–16.
• Morgan P.J., Cleave-Hogg D., Guest C.B. (2001) A comparison of global ratings and checklist scores from an undergraduate assessment using an anesthesia simulator. Academic Medicine, 76, 1053–5.
A Comparison of Global Rating Scale and Checklist Scores in the Validation of an Evaluation Tool to Assess Performance in the
Resuscitation of Critically Ill Patients During Simulated Emergencies (Abbreviated as "CRM Simulator Study IB")
Kim, John MD, MEd, FRCPC; Neilipovitz, David MD, FRCPC; Cardinal, Pierre MD, FRCPC; Chiu, Michelle MD, FRCPC
• 32 PGY-1 & 28 PGY-3 2 simulation scenarios on Crisis Resource Management (CRM)
• Ottawa Global Rating Scale 7-point anchored ordinal scale – 5 CRM categories and overall performance score
• Ottawa CRM Checklist 12 item in 5 CRM categories- max 30 points
• 3 raters blinded to year of training rated each videotaped performance (no order specified for use of evaluation tools)
Kim et al. 2009 - Continued
Reliability: Inter-rater reliability Intraclass Correlation Coefficient (ICC)• Ottawa GRS: S1=0.59 & S2=0.61
– Subcategories showed similar ICC except “resource utilization & communication (range 0.24 to 0.38)
• Cumulative CRM checklist: S1=0.63 & S2=0.55– Again subcategories showed similar ICC except “resource utilization &
communication (again range 0.24 to 0.38)
Validity: • Content validation (face validity) Delphi process• Response process Resident orientation & rater training• Comparison of scores by PGY T test (ANOVA)
– Both the checklist and GRS overall & subcategory scores showed statistically significant differences between PGY-1 and PGY-3 (more experienced residents receiving higher scores)
– ANOVA showed similar results by each scenario as well as per rater
A Comparison of Global Ratings and Checklist Scores from an Undergraduate Assessment Using an Anesthesia Simulator
Morgan, Pamela J. MD; Cleave-Hogg, Doreen PhD; Guest, Cameron B. MD, MEd
• 140 final year medical students 15-minute faculty-facilitated sim scenario– Conducted in 2nd week of the 2-week anesthesia rotation
– Faculty followed a script and each session was videotaped
– Each student completed 1 of 6 scenarios (each with similar learning objectives)
• 25-point criterion checklist for each scenario (not performed=0; performed=1)
• 5-point global rating (clear failure=1 to superior performance=5)
• 10 faculty attended a workshop on performance protocols– Randomly assigned to a rating pair to evaluate 25 to 34 videotaped
performances
Morgan et al. 2001 - Continued
• Correlation between checklist scores and global ratings Pearson r = 0.74
• Global ratings correlated more highly with technical skills and judgment than with knowledge – Knowledge r = 0.24
– Technical skills r =0.51
– Judgment r =0.53
• Single-rater reliability (consistency) – Mean ICC for Checklist scores 0.77 (range 0.58 to 0.93)– Mean ICC for Global ratings 0.62 (range 0.40 to 0.77)
Other Issues to Consider
• Required sample size for validity testing• Training the trainees• Scoring the videotaped performances
– Individual procedural vs. qualitative skills– Team performance
• Formative and summative assessment at the same time – Training (intervention) or assessment (measurement)?
• Assessment tools to link simulated performance to actual patient outcomes