Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development...
-
Upload
julian-mcfadden -
Category
Documents
-
view
223 -
download
4
Transcript of Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development...
![Page 1: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/1.jpg)
Measuring Outcomes
Geoffrey T. MillerAssociate Director, Research and Curriculum Development
Division of Pehospital and Emergency HealthcareGordon Center for Research in Medical Education
University of Miami Miller School of Medicine
![Page 2: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/2.jpg)
Session aims
• Discuss the importance of outcomes evaluation and challenges to traditional assessments
• Discuss the importance of validity, reliability and feasibility as it relates to assessment
• Discuss types of assessments and their application in healthcare education
![Page 3: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/3.jpg)
A little terminology…
• Assessment and evaluation are often used interchangeably
• However for our purposes…– Assessment = learner outcomes– Evaluation = course/program outcomes
![Page 4: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/4.jpg)
Why is assessment important?
![Page 5: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/5.jpg)
Because… assessment:
• “Drives learning”• Allows measures of individual and programmatic
progress• Fundamental to outcomes- or competency-based
education• Assures public that providers are competent• Credentialing, privileging, licensure, board
certification – high stakes for practitioner and patient/society– All involve assessment of competence
![Page 6: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/6.jpg)
Formula for the effective use of simulation
TrainingResources
TrainedEducators
CurricularInstitutionalizationX X =
EffectiveSimulation-
basedHealthcareEducation
Issenberg, SB. The Scope of Simulation-based Healthcare Education. Simulation in Healthcare. 2006.
![Page 7: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/7.jpg)
Formula for effective outcomes measurement
DefinedOutcomes
Instruments& TrainedEvaluators
AppropriateSimulatorX X =
EffectiveOutcomes
Measurement
![Page 8: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/8.jpg)
What are some challenges totraditional methods of assessment
for healthcare providers?
![Page 9: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/9.jpg)
Challenges in traditional assessments
• Ethical issues: “using” real pts (substitutes)– Invasive procedures (patient safety)– Sensitive tasks (cultural concerns, pt modesty)– Problems using cadaveric tissue models– Animal welfare issues
![Page 10: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/10.jpg)
Challenges in traditional assessments
• Real patients for evaluation of physical exam skills• Feasibility issues for large-scale examinations• Standardized, perceived fairness issues in high-stakes
settings• Standardized patients (SPs) improve reliability, but
validity issues exist: cannot mimic many physical findings
![Page 11: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/11.jpg)
Challenges in traditional assessments
• Wide range of clinical problems, including rare and critical events
• Availability• Cost• Reliability, validity, feasibility
![Page 12: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/12.jpg)
Developing outcome measurements
![Page 13: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/13.jpg)
“Any road will get you there, when you don’t know where you are
going”
![Page 14: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/14.jpg)
Curricula development
• Analysis– Define expected outcomes
• Design • Development• Implementation • Evaluation
![Page 15: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/15.jpg)
Defining outcomes
• Learners are more likely to achieve competency and mastery of skills if the outcomes are well defined and appropriate for the level of skill training
• Define clear benchmarks for learners to achieve• Plain goals with tangible, measurable objectives• Start with the end-goal in mind and the assessment
metrics, then the content will begin to develop itself
![Page 16: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/16.jpg)
Curricula/assessment process
CurricularDevelopment
-DefineOutcomes
Teaching andLearning
+/- Refinement Assessment andEvaluation
![Page 17: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/17.jpg)
Use of assessments in healthcare simulation
InformationInformation
DemonstrationDemonstration
PracticePractice
Rosen, MA et al. Measuring Team Performance in Simulation-Based Training: Adopting Best Practices for Healthcare.Simulation in Healthcare 3:2008;33–41.
FeedbackFeedback RemediationRemediation
MeasurementMeasurement DiagnosisDiagnosis
![Page 18: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/18.jpg)
Preparing assessments
• What should be assessed?– Every aspect of curriculum considered essential
and/or has significant designated teaching time– Should be consistent with learning outcomes that
are established as the competencies students should master/perform at a given phase of study
![Page 19: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/19.jpg)
BlueprintingGlobal Global
ObjectiveObjectiveRecognize a potential terrorist incident and initiate incident operations
UM-ERT
Module Obj.2.3
Recognize and describe scene hazards and appropriate personal protective measures
FloridaFlorida
Objective(s)Objective(s)Tier 1: I (L), III (D), (F), (N), IV (J), V (A), (D), VI (B)
LearningLearning
OpportunityOpportunityLectureLecture TabletopTabletop
VideoVideo
ExerciseExerciseSkillSkill OSCEOSCE
X X X X
AssessmentPrePre
MCQMCQ
PostPost
MCQMCQSkillSkill OSCEOSCE
5, 23 6, 19, 20 X
![Page 20: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/20.jpg)
Clinical competence and performance
• “Competent performance” = requires acquisition of basic knowledge, skills & attitudes
• Competence = – Application of specific KSAs
• Performance = – “Translation of competence into action”
“Can they do it? Do they do it?”
![Page 21: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/21.jpg)
Possible outcome competencies
• Patient care• Medical knowledge• Practice-based
learning and improvement
• Interpersonal and communication skills
• Professionalism • Systems-Based
Practice
Knowledge
SkillsAttitudes
![Page 22: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/22.jpg)
Knowledge competencies
KnowledgeCognitive knowledge•(factual) Recall•Comprehension•Application•Analysis•Synthesis•Evaluation
![Page 23: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/23.jpg)
Knowledge
Skills
Skills•Communication•Physical Exam•Procedures•Informatics•Self Learning•Time Management•Problem Solving
Skill competencies
![Page 24: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/24.jpg)
Knowledge
Skills
Attitudes• Behavior• Teamwork• Professionalism• Key Personal
Qualities• MotivationAttitudes
Attitude competencies
![Page 25: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/25.jpg)
Continuous process
Knowledge
SkillsAttitudes
![Page 26: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/26.jpg)
Relating Miller’s pyramid of competence to learning and assessment
![Page 27: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/27.jpg)
Miller’s Pyramid of Competence
Miller GE. The Assessment of Clinical Skills / Competence / Performance, Academic Medicine, 65:9, S63-S67.
Does
Shows
Knows How
Knows
![Page 28: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/28.jpg)
LearningOpportunity• Reading /
Independent Study
• Lecture• Computer-
based• Colleagues /
Peers
Teaching and Learning “Knows”
Does
Shows
Knows How
Knows
![Page 29: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/29.jpg)
Assessment of “Knows”
Factual Tests
Does
Shows
Knows How
Knows
![Page 30: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/30.jpg)
The Tools of “Knows”
• Multiple Choice Questions (MCQs)
• Short Answer
• True / False
• Matching (extended)
• Constructed Response Questions
![Page 31: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/31.jpg)
Example - MCQ
Information Input (Facts)
Information Input (Facts)
Factual Output (Answers)
Factual Output (Answers)
“Wheezes are continuous, musical, whistling sounds during difficult breathing such as in asthma, croup and other respiratory disorders.”
FACT
Q. Whistling sounds associated with an asthmatic patient are called?
A. Rales B. RhonchiC. Wheezes D. Vesicular
ANSWER
Learning Opportunity
Assessment
![Page 32: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/32.jpg)
Computer-based model
Choose the best description of the patient’s finding:
A. Myoclonus
B. Partial Seizure
C. Tic
D. Fasciculations
E. Tremor
Click on picture to play video
![Page 33: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/33.jpg)
LearningOpportunity• Problem-based Ex.• Tabletop Exercises• Direct Observation• Mentors
Teaching and Learning - “Knows How”
Does
Shows
Knows How
Knows
![Page 34: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/34.jpg)
Clinical Context
Based Tests
Assessment of “Knows how”
Does
Shows
Knows How
Knows
![Page 35: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/35.jpg)
• Multiple-choice question
• Essay
• Short answer
• Oral interview
The Tools of “Knows How”
![Page 36: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/36.jpg)
Example – Clinical Context MCQ
• 64-year-old man• No past medical Hx• 1 week of intermittent
– Headache
– Double vision
• R pupil dilated
Which of the following is most likely the patients problem?
A. Migraine
B. Myasthenia gravis
C. Multiple Sclerosis
D. Ischemic Stroke
E. Cerebral aneurysm
![Page 37: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/37.jpg)
LearningOpportunity• Skill-based
Exercises•Repetitive practice
• Small Group• Role Playing
Teaching and Learning - “Shows”
Does
Shows
Knows How
Knows
![Page 38: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/38.jpg)
Assessment of “Shows”
PerformancePerformanceAssessmentAssessment
Does
Shows
Knows How
Knows
![Page 39: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/39.jpg)
• Objective Structured Clinical Examination (OSCE)
• Standardized Patient-based
The Tools of “Shows”
![Page 40: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/40.jpg)
Variables in Clinical Assessment
ClinicalAssessment
Examiner
Patient
Student
Control as many variables as possible
![Page 41: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/41.jpg)
LearningOpportunity• Experience
Teaching and Learning - “Does”
Does
Shows
Knows How
Knows
![Page 42: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/42.jpg)
Assessment of “Does”
PerformanceAssessment
Does
Shows
Knows How
Knows
![Page 43: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/43.jpg)
• Undercover / Stealth / Incognito Standardized Patient-based
• Video
• Portfolio
• Service ratings (customer satisfaction)
The Tools of “Does”
![Page 44: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/44.jpg)
Influences on clinical performance
Does
Performance
CompetenceSyste
m related Individual related
Cambridge Model for delineating performance and competence
Rethans JJ, et al. The relationship between competence and performance: implications for assessing practice performance, Medical Education, 36:901-909.
![Page 45: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/45.jpg)
Assessments types
• Choose the appropriate assessment method:– Formative– Summative– Self– Peer
![Page 46: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/46.jpg)
Assessment• Formative Assessment
– Lower stakes – One of several, over time of course or program– May be evaluative, diagnostic, or prescriptive– Often results in remediation or progression to next level
• Summative Assessment– Higher stakes – Generally final of course or program – Primary purpose is performance measurement– Often results in a “Go, No-Go” outcome
![Page 47: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/47.jpg)
Formative assessment example
![Page 48: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/48.jpg)
Assessments - self
• Encourages responsibility for the learning process, fosters skills in making judgments as to whether work is of an acceptable standard – it improves performance.
• Most forms of assessment can be adapted to a self-assessment format (MCQs, OSCEs, and short answers)
• Students must be aware of standards required for competent performance.
![Page 49: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/49.jpg)
Individual self-learning and assessment
![Page 50: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/50.jpg)
Assessments - peer
• Enables learners to hone their skills in their ability to work with others and professional insight
• Enables faculty to obtain a view of students they do not see
• An important part of peer assessment is for students to justify the marks they award to others
• Justification can also be used as a component when faculty evaluates attitudes and professionalism.
![Page 51: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/51.jpg)
Assessments - standard setting
• Should be set to determine competence• Enables certification to be documented, accountable
and defensible• Appropriately set standards for an assessment will
pass those students who are truly competent• Standards should not be two low (false positives) to pass
those who are incompetent, nor too high (false negative) to fail those who are competent.
![Page 52: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/52.jpg)
Assessments - standard setting
• Those responsible in setting standards must also have a direct role in teaching students at the level being examined and assist in providing examination material
![Page 53: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/53.jpg)
Assessments - standard setting
• Standards should be set around a core curriculum that includes the knowledge, skills and attitudes required of all students
• When setting a standard the following should be considered:– What is assessed must reflect the core curriculum– Students should be expected to reach a high standard in
the core components of the curriculum (For instance an 80-90% pass mark of for the important core and 60-80% for the less important aspects.)
– Students should be required to demonstrate mastery of the core in one phase of the curriculum before moving on to the next part of the curriculum
![Page 54: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/54.jpg)
Choosing appropriate assessment methods
• When choosing the assessment instrument, the following should be answered:– Is it valid– Is it reliable– Is it feasible
![Page 55: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/55.jpg)
Assessments - validity
• Are we measuring what we are supposed to be measuring
• Use the appropriate instrument for the knowledge, skill, or attitude you are testing
• The major types of validity should be considered (content, predictive, and face)
![Page 56: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/56.jpg)
Assessments - reliability
• Does the test consistently measure what it is supposed to be measuring
• Types of reliability:– Inter-rater (consistency over raters)– Test-retest (consistency over time)– Internal consistency (over different items/forms)
![Page 57: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/57.jpg)
Both archers are equally reliable.
Archer 1 hits bulls eye every time.
Archer 2 hits outer ring in same spot every time.
Reliability as Consistency
![Page 58: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/58.jpg)
Inter-rater Reliability
• Multiple judges code independently using the same criteria
• Reliability = raters code same observations into same classification
• Examples• Medical record reviews• Clinical skills• Oral examinations
![Page 59: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/59.jpg)
Factors Influencing Reliability
• Test length• Longer tests give more reliable scores
• Group homogeneity• The more heterogeneous the group, the higher
the reliability
• Objectivity of scoring• The more objective the scoring, the higher the
reliability
![Page 60: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/60.jpg)
Both archers are equally reliableValidity = quality of archer’s hits
Archer 1 hits bulls eye every time
Archer 2 hits outer ring in same spot every time
Validity is accuracy
![Page 61: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/61.jpg)
Reliable and Valid
Reliable, not valid
Not reliable, not valid
Not reliable, not valid
Reliability and Validity
![Page 62: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/62.jpg)
Improving reliability and validity
• Base assessment on outcome/objectives- event triggers- observable behavior- behavioral rating-assess against competence
• Define:– Low-medium-high performance– Use of rubric or rating metric– Use (video) training examples of performance– Employ quality assurance/improvement system
![Page 63: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/63.jpg)
Assessments - feasibility• Is the administration and taking of the assessment
instrument feasible in terms of time and resources• The following questions should be considered:
– How long will it take to construct the instrument?– How much time will be involved with the scoring process?– Will it be relatively easy to interpret the scores and
produce the results?– Is it practical in terms of organization?– Can quality feedback result from the instrument?– Will the instrument indicate to the students the important
elements within the course?– Will the assessment have a beneficial effect in terms of
student motivation, good study habits and positive career aspirations?
![Page 64: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/64.jpg)
Practicality
• Number of students to be assessed• Time available for the assessment • Number of staff available• Resources/equipment available• Special accommodations
![Page 65: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/65.jpg)
Assessment instruments
![Page 66: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/66.jpg)
Assessments - instruments
• Be aware of the types of assessment instruments available as well as the advantages and disadvantages of each
• It is important, if feasible, to use more than one assessment instrument and more than one assessor when looking at skills and attitudes
![Page 67: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/67.jpg)
Assessments – knowledge instruments
• Objective tests (short answer, true/false, matching, multiple choice)
• Objective Structured Clinical Evaluations (OSCEs)• Constructed response questions• Rating scales (used on clerkships)
![Page 68: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/68.jpg)
Assessments – skill instruments
• Objective tests (Simulation based)• OSCEs• Constructed response questions• Critical reading papers (interpreting literature)• Checklists• Rating Scales• Portfolios (self-evaluation, time management)
![Page 69: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/69.jpg)
Assessments – attitude instruments
• Portfolios• Essays / Modified essay questions• OSCEs• Checklists• Rating scales• Patient management problems• Short/long case assessments
![Page 70: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/70.jpg)
Assessment Metrics
• Procedural or Check List assessment• Global Rating assessment
![Page 71: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/71.jpg)
Assessment Metrics• Procedural or Check List assessment
BCLS Y N
Open Airway
Check Breathing
BCLS Y N
Open Airway(< 5 sec of LOC)
Check Breathing(< 5 sec of Airway)
BCLS Y N
Open Airway
Check Breathing
A
Rating Score +1 -1 0 *Assist
![Page 72: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/72.jpg)
Assessment Metrics• Global Rating assessment
Code Blue P F
CPR and
ACLS
Code Blue
CPR<1(low) - 5(Hi)> points
ACLS<1(low)- 5(Hi)> points
Code Blue H M
CPR
ACLS
L
Rating Score +1 0 -1
Pts.
![Page 73: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/73.jpg)
Review
• Assessment drives learning• Clearly define the desired outcome, ensure that
it can be measured• Consider the “threats” to the effectiveness of
the measurement• Feedback to individual candidates• Feedback to training programs
![Page 74: Measuring Outcomes Geoffrey T. Miller Associate Director, Research and Curriculum Development Division of Pehospital and Emergency Healthcare Gordon Center.](https://reader033.fdocuments.in/reader033/viewer/2022061306/55147d6c550346d36e8b47d7/html5/thumbnails/74.jpg)
Questions and discussion