Formative Evaluation of the Implementation of the Medical Education Metrics Standard for

Continuing Education

MedBiquitous Annual Conference 2011

May 11, 2011 Baltimore, Maryland

Presenters and Disclosures

• Francis Kwakwa, MAAssistant Director, Data Management , Radiological Society of North AmericaDoes not have an interest in selling anything to CME professionals

• Andrew Rabin, BSChief Technology Officer, CECity Does have an interest in selling a program or service to CME professionals

• Sean Hayes, PsyD Vice President of AXDEV GroupDoes have an interest in selling a program or service to CME professionals

• Valerie Smothers, MA Deputy Director, MedBiquitousDoes not have an interest in selling anything to CME professionals

2

Objectives of Today’s Session

• Increase knowledge about use of Medical Education Metrics Standard (MEMS) in CME

• Share experience of using MEMS as part of Program and Activity Reporting System

• Present formative findings collected regarding the experience of those who have implemented MEMS

• Offer case study findings as to how to assess implementation of new systems

3

MedBiquitous Metrics Working Group

Mission

To develop XML standards & Web services

requirements and descriptions for the exchange

of aggregate evaluation data and other key

metrics for health professions education.

4

MEMS: Medical Education Metrics

• Technology standard for core evaluation data

• Users– Educators want best

practices, ability to compare– Funders want to measure

reach and efficacy– Accreditors want to

measure success of activity and provider

MEMS Data

• Activity Description– What’s being evaluated

• Participant Activity Evaluation– What did participants think

• Participation Metrics– How many people participated

• Learner Demographics

Program and Activity Reporting System (PARS)

7

• Manual data entry

• Comma delimited upload

• XML upload

Formative Evaluation

• Provides immediate feedback to program participants

• Fosters reflection and self-regulation

• Provides feedback about the program itself to designers and sponsors

• Oriented to improving the focus of the program

• Based upon a dynamic model

Relationship Between Formative and Summative Evaluation

Across Life of an Innovation

Rel

a tiv

e E

mp

has

is

Program Life

Summative Evaluation

Formative Evaluation

Worthen 1987

Focus Group

• MEMS implementers using XML in association with PARS

• 4 organizations represented

• Measured – Success and satisfaction– Resources for implementation– Characteristics of implementers– Decision factors

10

Results

• Implementers felt satisfied, but limited (not many customers implementing)

• Limited effort and resources required; change management important

• Implementers all fluent in XML • ACCME adoption key; implementers

seeking to better support customers, show competitive advantage, make links between different types of data

Case Study

• Implemented MEMS for PARS upload

• Experience implementing MedBiquitous standards

• Integrated new data into user interface and workflow

• Provided user training

Lessons Learned

16

• Standards implementation can bring about efficiencies when integrated into workflow and properly supported

• Validation tools needed

• Web services needed to realize full benefits, one click data transfer

Contact Us

17

• vsmothers@jhmi.edu

• Valerie will forward questions to the working group