Simulation based Assessment

Simulation based Assessment

Training the Trainer SeriesOctober, 2014

Training the Trainer SeriesOctober, 2014

Assessment

• Formative

• Summative

Formative Assessment• Assessment for

learning

Summative Assessment

• Assessment of learning

Evaluation Methodology

Bould MD, et al. Assessment of procedural skills in anesthesia. Br J of Anaesth 2009; 103:472-483

Methodology:Global Rating Scale

• Overall performance by categories

• Scores (Likert)

• Drawbacks

Global Rating Scale

Blum RH, et al. Simulation-based assessment to identify critical gaps in safe anesthesia resident performance. Anesthesiology 2014; 120:129-41

Methodology:Checklists

• Delineated actions

• Weighted, timed, cumulative...

• Adverse events

• Drawbacks

Checklists

Schwid, et al. Evaluation of anesthesia residents using mannequin-based simulation, a multi institutional study. Anesthesiology 2002; 97:1434-1444

Methodology:“Time To”

• Timing of critical actions

• Pace

• Motion analysis

Methodology:Survey Data

• Self assessments

• Psychometrics: Pre and Post Testing

Survey Data

Lammers RL, et al. Simulation-based Assessment of Paramedic Pediatric Resuscitation Skills. Prehospital Emergency Care 2009; 13:345-356

Methodology:Biometrics

• “Technometrics?”

• Mannequin physiology

Validity

• The Assessment Method

• The Simulation

Validity:“Face”

• General acceptability of the tool used for the task required

• Does this make sense?

Validity:“Content”

• Does the content match what’s to be assessed?

• Task Analysis

• The Experts

The Delphi Method

• BOGSAAT

Validity:“Construct”

• Does this reflect true performance?

• Experts

• Role of experience

• Surveys

Validity:“Construct”

• Statistical Analysis:

• Discrimination: “D Statistic”

• ANOVA

• “Concurrence”

Tool Validation

Reid J, et al. The Simulation Team Assessment Tool (STAT): Development, reliability and validation. Resuscitation 2012; 83:879-886

Validity:“Criterion”

• Or “Extrapolation”

• Similar results to other performance measures

• “Predictive”

Consistency:Internal

• Requires realism

• Reflects test quality

• Pilot Testing

Consistency:Statistical Analysis

• Do testers do well or poorly on the same items?

• Generalizability: “G statistic”

Reliability:Inter-Participant

• Set-up

• Simulator

• The Subject

Reliability:The Rater

• Inter-rater reliability (external reliability)

• Statistics: IRR, ICC

Reliability:The Rater

• Rater training

• Rater refreshing

• Scoring

Reliability:The Observation

• Real time

• Recording

• Bias

Validated Tools

Cheng A, et al. Designing and Conducting Simulation-Based Research. Pediatrics 2014; 133:1090-1101.

Take Away

• Rigor in realism, rehearsing and piloting

• Mix Checklists and GRS

• Supplement with survey, psychometric, device data, etc

• More tasks with more (recorded) views

Further SourcingSchwid, et al. Evaluation of anesthesia residents using mannequin-based simulation, a multi institutional study. Anesthesiology 2002; 97:1434-1444

Devitt, et al. The validity of performance assessment using simulation. Anesthesiology 2001; 95:36-42

Devitt, et al. Testing internal consistency and construct validity during evaluation of performance in a patient simulator. Anesth Analg 1998; 86:1160-1164

Devitt JH, Kurrek MM, Cohen MM, Fish K, Fish P, Murphy PM, Szalai JP:

Testing the raters: Inter-rater reliability of standardized anaesthesia simulator

performance. Can J Anaesth 1997; 44:924–8

Gaba DM, Howard SK, Flanagan B, Smith BE, Fish KJ, Botney R: Assessment

of clinical performance during simulated crises using both technical and behavioral ratings. Anesthesiology 1998; 89:8-18

Boulet JR, et al. summative assessment in medicine: the promise of simulation for high-stakes evaluation. Acad Emerg Med 2008; 15:1017-1024

Cheng A, et al. Designing and Conducting Simulation-Based Research. Pediatrics 2014; 133:1090-1101.

Furman GE et al. Quality Assurance Best Practices for Simulation-Based Examinations. Qual Assurance Best Practices 2014; 5:226-231

Reid J, et al. The Simulation Team Assessment Tool (STAT): Development, reliability and validation. Resuscitation 2012; 83:879-886

Siassakos D, et al. The management of a simulated emergency: Better teamwork, better performance. Resuscitation 2011; 82:203-206

Irvine S and Martin J. Bridging the gap: from simulation to clinical practice. The Clinical Teacher 2014; 11:94-98

Blum RH, et al. Simulation-based assessment to identify critical gaps in safe anesthesia resident performance. Anesthesiology 2014; 120:129-41

Bould MD, et al. Assessment of procedural skills in anesthesia. Br J of Anaesth 2009; 103:472-483

McKinley RK, et al. Checklists for assessment and certification of clinical procedural skills omit essential competencies: a systematic review. Med Educ 2008; 42:338-349

Lammers RL, et al. Simulation-based Assessment of Paramedic Pediatric Resuscitation Skills. Prehospital Emergency Care 2009; 13:345-356