Difficult to Assess Milestones

. . . An Open Collaboration In Teaching and Assessment

Saadia Akhtar, MD, FACEPProgram Director

Beth Israel Medical CenterAlbert Einstein College of Medicine

Rodney Omron, MD, FACEPAssistant Program Director

John Hopkins University

Lynn Roppolo, MD, FACEPAssociate Program Director

University of Texas SouthwesternParkland Health and Hospital System

• None to report

Conflict of Interest

Objectives• Identify the challenges and obstacles to

assessing these milestones• Create a more effective educational program

to effectively teach these milestones • Develop assessment tools to appropriately

evaluate residents in these milestones• Understand how to navigate the EM

milestones wiki site

The Problem . . .

A simple but less defined assessment tool . . . . . .

. . . are now expanded into 23 sub- competencies!!!

Most faculty are convinced that the RRC-EM Milestones are a solution

for which there is no known problem.

-anonymous

Developing a roadmap . . .

• 15-question survey, validated by 7 former PDs • Sent to CORDEM listserver in the Fall of 2013 • 62% (99/160) of EM residency programs responded • IRB approved study

Difficult Milestones Survey

Objectives (revised)

Begin with the end in mind . . .

Sub-competency 20: Problem Based Learning and Improvement (PBLI)

Sub-competency 20 (PBLI): Participates in performance improvement to optimize ED function, self-learning, and patient care.

Level 1 Level 2 Level 3 Level 4Describes basic principles of evidence-based medicine

Performs patient follow-up

Performs self-assessment to identify areas for continued self-improvement and implements learning plans Continually assesses performance by evaluating feedback and assessment Demonstrates the ability to critically appraise scientific literature and apply evidence-based medicine to improve one’s individual performance

Applies performance improvement methodologies Demonstrates evidence-based clinical practice and information retrieval mastery Participates in a process improvement plan to optimize ED practice

• 98/157 CORD attendees surveyed• 71% do not use formal critical appraisal instrument• 75% do not have established EBM curriculum • 78% do not use any extramural sources (ACP, Best

Evidence, EM Abstracts, or Annals of EM)

Carpenter CR . Incorporating evidence-based medicine into resident education: a CORD survey of faculty and resident expectations. Acad Emerg Med 2010

EBM Practices and Expectations from EM Physician Educators

Fres

no te

st

Othe

r EBM

qui

z

Jour

nal c

lub

pres

enta

tion

b...

Othe

r con

fere

nce

pres

enta

t...

Qual

ity im

prov

emen

t pro

ces..

.

Criti

cal a

ppra

isal o

f the

li...

Criti

cally

app

raise

d to

pics

...

EBM

Criti

cally

App

raise

d To

...

Vand

erbi

lt m

atrix

eva

luat

ion.

..

Trac

king

of r

esid

ent s

earc

he...

Audi

ence

resp

onse

syst

em f.

..

“Kno

wled

ge T

rans

latio

n” sh

i...

Dire

ct o

bser

vatio

n by

supe

r...

We

do n

ot cu

rrent

ly h

ave

a ...

Othe

r

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

EBM Assessment Methods (CORD survey No-

vember 2013)

Sub-competency 20: Most Common EBM Assessments Methods

Percent

Journal Club 83.3

Direct observation by supervising faculty 47.1

Critical appraisal of the literature using EBM worksheets

26.5

Quality improvement process using EBM to create a process change

23.5

Other conference presentation by EM resident 21.6

Learning Basic EBM Principles• Journals: ACP Journal Club, EM Abstracts, Annals• Books– Emergency Medicine Decision Making by S. Weingart, 2006– User’s Guide to the Medical Literature by G Guyatt, 2008– Evidence-Based Diagnosis (Cambridge Medicine), TB

Newman, 2009– EBM: How to Practice and Teach EBM, 4th Edition, SE

Straus, 2010– Evidence-Based Physical Diagnosis: Expert Consult, S

McGee, 2012– Evidence-Based Emergency Care: Diagnostic Testing and

Clinical Decision Rules by JM Pines and CR Carpenter, 2013

Learning Basic EBM Principles• Websites and podcasts– JAMAevidence: Using Evidence to Improve Care– http://www.dartmouth.edu/~library/biomed/guides/resear

ch/ebm-resources-materials.html– http://guides.mclibrary.duke.edu/ebm– https://www.med.emory.edu/EMAC/curriculum/diagnosis/

maindiagnostic.html– http://www.annemergmed.com/content/ebemresources– http://adc.bmj.com/content/90/8/845.full.pdf– http://www.cebm.net/?o=1023– http://bestbets.org/teaching/current.php– http://ktclearinghouse.ca/– http://emjclub.com

Critical Appraisal Worksheetshttp://www.cebm.net/index.aspx?o=1157

CAT Maker and Critically Appraised Topics

Flipped Classroom: Teaching and Learning More Efficiently

• Need to be information managers, not encyclopedias of knowledge.

• Need to be life long learners… to ask questions and go out and find answers for themselves.

• Need to search and filter vast quantities of information on the internet - “Free Open Access MedEd”(FOAMed) by way of blogs, podcasts, and videos, apps and mobile websites

Flipped Classroom Example

• One hour of ‘individualized learning’ at home• Provide a clinical scenario and 3-5 questions

for residents to answer in advance• Provide some electronic resources • Interactive discussion in classroom• Evaluate residents by written submission of

their findings or conference participation

Fresno Test• 2 clinical scenarios with open ended questions• Must complete 4 key steps of EBM practice • 7 short answer questions, 2 mathematical

calculations, and three fill-in-the-blank questions• The only validated, standardized, and objective

measure of EBM competence currently available• 30 minute test• 13 page rubric

Fresno Test

Knowledge Translation Shift

• Residents and an EBM faculty leader spend one shift per month in the ED answering EBM queries for all of the residents/faculty seeing patients.

Sub-competency 20: Patient Follow-Up Methods Percent

Log of patient follow up that resident does through their own review of medical record

77.5

M&M or continuous quality improvement (review of specific CASES)

65.7

Case conference presentations (not related to M&M or continuous quality improvement)

47.1

Calling back patients 23.5

Continuous quality review (review specific patient care areas)

12.7

Chart review with dedicated faculty 9.8

Sub-competency 20: Self Assessment

• 62% do a self-assessment tool or questionnaire• 61% discuss with residency leadership• 55% discuss with their advisor or mentor• 42% talk about it during clinical shifts• Most review semi-annually

Sub-competency 20: Process Improvement (PI)

• 55% have PI project in the ED• 32% do a presentation at weekly conference• 28% have resident participate in a committee• 24% do not assess this

Sub-competency 21:

Patient Safety

(System Based Based Practice 1 or SBP1)

Sub-competency 21 (SBP1): Participates in performance improvement to optimize patient SAFETY.

Level 1 Level 2 Level 3 Level 4

Adheres to standards for maintenance of a safe working environment Describes medical errors and adverse events

Routinely uses basic patient safety practices, such as time-outs and ‘calls for help’

Describes patient safety concepts Employs processes (e.g., checklists, SBAR), personnel, and technologies that optimize patient safety (SBAR= Situation – Background – Assessment – Recommendation) Appropriately uses system resources to improve both patient care and medical knowledge

Participates in an institutional process improvement plan to optimize ED practice and patient safety Leads team reflection such as code debriefings, root cause analysis, or M&M to improve ED performance Identifies situations when the breakdown in teamwork or communication may contribute to medical error

Sub-competency 21: What type of educational formats do you use to teach your residents about "Patient Safety“?

Percent

M&M or CQI conferences 97.0

Simulation 58.6

Hospital based online modules 41.4

Dedicated patient safety conferences 37.4

Mock oral boards 36.4

Small group discussions 30.3

Institute of Healthcare Improvement (IHI) Modules 4.0

Other 4.0

Sub-competency 21- Tools to assess the Patient Safety Sub-competency

185

49

9

44 4857

6 9 3

Sub-competency 21: Do you use any checklists or tools to assist with "Transitions of Care" such

as the SBAR or Team Huddle?

54%44%

2%

YesNoOther

Sub-competency 21: ToolsiPASS (EPIC)

TEAM STEPPS

Bedside Rounding

Sign-Out Tab in EPIC

Standardized HMED Transfer of Care Note

Evaluation of Handoff Form

Developed Own Transition Sheet/Template

Printed tracker board and physically rounding in ED at transition time

Check list at sign out

• After implementation of multiple medical team training programs:•Improved observed team behaviors.

•Enhanced staff attitudes toward teamwork.

•Reduced observed clinical errors.

Morey, JC, Simon, R, Jay GD, et al. Error reduction and performance improvement in the emergency departmentthrough formal teamwork training: Evaluation results of the MedTeams project. Health Serv Res. 37:1553-1581, 2002

Sub-competency 22: Systems-based Management (SBP2)

Sub-competency 22 (SBP2): Participates in strategies to improve healthcare delivery and flow. Demonstrates an awareness of and responsiveness

to the larger context and system of health care.

Level 1 Level 2 Level 3 Level 4

Describes members of ED team (e.g., nurses, technicians, and security)

Mobilizes institutional resources to assist in patient care Participates in patient satisfaction initiatives

Practices cost-effective care Demonstrates the ability to call effectively on other resources in the system to provide optimal health care

Participates in processes and logistics to improve patient flow and decrease turnaround times (e.g., rapid triage, bedside registration, Fast Tracks, bedside testing, rapid treatment units, standard protocols, and observation units) Recommends strategies by which patients’ access to care can be improved .Coordinates system resources to optimize a patient’s care for complicated medical situations

Sub-competency 22: What type of educational formats do you use to teach your residents about "Systems-based Management" to improve healthcare delivery and flow?

Percent

Didactics 69.7

Morbidity and Mortality Conference 68.7

Small group discussions 39.4

Simulation 38.4

Multi-disciplinary teaching by other hospital staff 36.4

Mock oral boards 28.3

We do not have any formal teaching on "Systems-based Management" for our residents

7.1

Online modules 5.1

Other 1.0

End of Shift

End of Rotation

Peer Evaluation

Patient Evaluation

Direct Observation Tool by Faculty

Feedback from Others

Mock Oral Boards

Simulation

M & M/CQI

None

Other

0 10 20 30 40 50 60

Sub-competency 22: Tools for Systems-based Management

www.improvediagnosis.org

Sub-competency 23: Technology (SBP3)

Sub-competency 23 (SBP3): Uses technology to accomplish and document safe healthcare delivery.

Level 1 Level 2 Level 3 Level 4

Uses the Electronic Health Record (EHR) to order tests, medications and document notes, and respond to alerts Reviews medications for patients

Ensures that medical records are complete, with attention to preventing confusion and error Effectively and ethically uses technology for patient care, medical communication and learning

Recognizes the risk of computer shortcuts and reliance upon computer information on accurate patient care and documentation

Uses decision support systems in EHR (as applicable in institution)

Sub-competency 23: Methods used to assess the use of technology for safe health care delivery

Percent

Evaluations by supervising faculty in the ED 71.6

Chart review with the resident 28.4

Conference presentation that demonstrates ability to identify and use educational resources readily available while working in the ED (e.g. PBL)

22.5

Review of billing records 17.6

No mechanism exists to assess this milestone 16.7

Identification of clinical dashboards 8.8

EM Milestones Wiki

emmilestones.pbworks.com

OR

Google “EM Milestones Wiki”

For assistance or to add to the wiki site: romron1@jhmi.edu

Summary• EBM • Safety • Technology • Systems Based Management • Wiki site– How to navigate– How to access resources– How to contribute

Thank you!

Questions• sakhtar@chpnet.org• romron1@jhmi.edu• lynn.roppolo@utsouthwestern.edu

References• Tews MC, Liu JM, Treat R. Situation-Background-Assessment-Recommendation (SBAR) and

Emergency Medicine Residents' Learning of Case Presentation Skills. J Grad Med Educ. Sep 2012; 4(3): 370–373.

• Morey, JC, Simon, R, Jay GD, et al. Error reduction and performance improvement in the emergency department through formal teamwork training: Evaluation results of the MedTeams project. Health Serv Res. 37:1553-1581, 2002

• http://teamstepps.ahrq.gov• www.improvediagnosis.org