Alliance for CME WebinarAlliance for CME WebinarTuesday, September 25, 2007Tuesday, September 25, 2007

Wendy Turell, DrPHSenior DirectorEducational Design and Accreditation ServicesBCME

Beyond Theory: Practical Beyond Theory: Practical Tools to Tackle Educational Tools to Tackle Educational

Outcomes EvaluationOutcomes Evaluation

22

Today’s AgendaToday’s Agenda

II Measurement and CMEMeasurement and CME

IIII Educational Outcomes Educational Outcomes Evaluation: Evaluation:

MethodologyMethodology

III III Sample Approaches to Study Sample Approaches to Study DesignDesign

IVIV Questions and AnswersQuestions and Answers

33

Rationale for MeasurementRationale for Measurement

Return on Education (ROE)Return on Education (ROE)– Educational dollars should be spent on Educational dollars should be spent on

effectiveeffective education education Guide for future programmingGuide for future programming

– What formats and strategies work best?What formats and strategies work best? Protection within CME guidelinesProtection within CME guidelines

– Proof of educational effectivenessProof of educational effectiveness

44

Evaluation and ACCME Evaluation and ACCME ComplianceCompliance

Criteria 11 Criteria 11

““The provider analyzes changes in learners The provider analyzes changes in learners (competence, performance, or patient (competence, performance, or patient

outcomes) achieved as a result of the overall outcomes) achieved as a result of the overall program’s activities/educational program’s activities/educational

interventions”interventions”

–Level 1: Provisional AccreditationLevel 1: Provisional Accreditation–Level 2: Full AccreditationLevel 2: Full Accreditation–Level 3: Accreditation with CommendationLevel 3: Accreditation with Commendation

An Outcomes-Based An Outcomes-Based Educational ModelEducational Model

66

Outcomes-Based Educational Outcomes-Based Educational ModelModel

Needs Assessment

Identify Optimal Outcomes (based on educational gap)

Outcomes Assessment & Program Evaluation

Outcomes-Based Learning Objectives

Design Program for Maximum Impact

Needs Assessment

77

Educational EvaluationEducational Evaluation

Tells us whether the planned goals match Tells us whether the planned goals match what learners take from the activitywhat learners take from the activity

Judges, describes, defines, values, shapes Judges, describes, defines, values, shapes opinion, directs attentionopinion, directs attention

Determines the value or quality of Determines the value or quality of educational activities in efforts to provide educational activities in efforts to provide feedback for improvement feedback for improvement – ““Value” can be in regards to: satisfaction, Value” can be in regards to: satisfaction,

achievement, improved performance, benefits achievement, improved performance, benefits to others, return on investment, etc.to others, return on investment, etc.

Bennett NL: The voices of evaluation. J Contin Educ Health Prof 1997;17:198-206.

ACME “Evaluating CME Using Outcomes” in Evaluating Educational Outcomes, http://www.acme-assn.org, accessed 1/2006.

Green, Joseph & Eckstein, Jane (2006) A Practical Guide to Integrating an Outcomes-Based Learning Model Into Your Planning Process, Alliance for CME Almanac, 28 (1), p 1-5.

88

Outcomes Evaluation Outcomes Evaluation

More Specific Type of Educational More Specific Type of Educational EvaluationEvaluation

Evaluation that strives to Evaluation that strives to demonstrate the relative demonstrate the relative effectiveness of various approaches effectiveness of various approaches to education on learning/behavior to education on learning/behavior change/patient health outcomes (a change/patient health outcomes (a causes b)causes b)

Bennett NL: The voices of evaluation. J Contin Educ Health Prof 1997;17:198-206.

ACME “Evaluating CME Using Outcomes” in Evaluating Educational Outcomes, http://www.acme-assn.org, accessed 1/2006.

Green, Joseph & Eckstein, Jane (2006) A Practical Guide to Integrating an Outcomes-Based Learning Model Into Your Planning Process, Alliance for CME Almanac, 28 (1), p 1-5.

99

Reach for Greater HeightsReach for Greater Heights in in

MeasurementMeasurement

LEVEL 5PATIENT HEALTH

LEVEL 4PERFORMANCE

LEVEL 3LEARNING

LEVEL 2SATISFACTION

LEVEL 1PARTICIPATION

LEVEL 6POPULATION HEALTH

Moore DE. A framework for outcomes evaluation. In: Davis D. Barnes BE, Fox R, eds. The Continuing Professional Development of Physicians: From Research to Practice. Chicago: AMA Press: 2003.

1010

Levels 1,2,3Levels 1,2,3

Levels 1: ParticipationLevels 1: Participation– Not a very valid assessment of Not a very valid assessment of educationaleducational outcome outcome– ObjectiveObjective

Level 2: SatisfactionLevel 2: Satisfaction– Does not document learning Does not document learning – SubjectiveSubjective

Level 3: LearningLevel 3: Learning– Assessment of educational outcomeAssessment of educational outcome– Can be measured with survey questionnairesCan be measured with survey questionnaires– May or may not lead to behavior changeMay or may not lead to behavior change

1111

Levels 4, 5 and 6Levels 4, 5 and 6

Level 4: PerformanceLevel 4: Performance– Various ways to measureVarious ways to measure– Can document impact on practice behavior using follow-up assessmentsCan document impact on practice behavior using follow-up assessments– May not capture all new behaviors May not capture all new behaviors – Typically self-reportedTypically self-reported

Level 5: Patient HealthLevel 5: Patient Health– Objective measure; desirable outcome variableObjective measure; desirable outcome variable– Difficult to determine if change is due to intervention; also clouded by Difficult to determine if change is due to intervention; also clouded by

patient co-morbiditiespatient co-morbidities– $$$$$, patient privacy laws (HIPPA) and Institutional Review Board Barriers$$$$$, patient privacy laws (HIPPA) and Institutional Review Board Barriers

Level 6: Population HealthLevel 6: Population Health– Most desirable outcome variable!Most desirable outcome variable!– Do most CME activities deliver such a reach??Do most CME activities deliver such a reach??– Rarely measured in CME due to financial and logistical barriersRarely measured in CME due to financial and logistical barriers

Outcomes Evaluation: Outcomes Evaluation: MethodologyMethodology

1313

Sample Study PlanSample Study Plan

Study Subjects Survey Administrati

on

Survey Design

Experimental

Group

Control Group

(optional)

Case-Based Questions

Knowledge- Based Questions Pre-Test

Staggered

Post-Test

Different options to assess learning and

behavior change

1414

Validity & Statistical Validity & Statistical SignificanceSignificance

ValidityValidity- The degree of confidence one can - The degree of confidence one can have in an observed result, such as an have in an observed result, such as an improvement in knowledgeimprovement in knowledge– The degree to which the observed result can be The degree to which the observed result can be

attributed to the studied cause (ie: effectiveness of attributed to the studied cause (ie: effectiveness of the CME course) and not random error in sampling the CME course) and not random error in sampling and measurementand measurement

Statistical SignificanceStatistical Significance – Quantifies the degree – Quantifies the degree of confidence you can have in a specific resultof confidence you can have in a specific result– Ex: Statistically significant at .05 alpha level = 95% Ex: Statistically significant at .05 alpha level = 95%

chance that the result is valid and not due to chance that the result is valid and not due to chancechance

1515

Some Ways to Increase ValiditySome Ways to Increase Validity

Choose participants fairly (everyone Choose participants fairly (everyone has an equal chance of completing has an equal chance of completing surveys)surveys)

Ensure robust sample sizeEnsure robust sample size Write a clear survey (participants Write a clear survey (participants

should understand questions as you should understand questions as you understand them)understand them)

Use a control group (placebo or Use a control group (placebo or comparison subjects)comparison subjects)

1616

Pre –Tests: Pre –Tests: Walking-in-the doorWalking-in-the door knowledgeknowledge

Pre-TestsPre-Tests- Establishes baseline of - Establishes baseline of subjects’ knowledge/behavior before subjects’ knowledge/behavior before they are exposed to our educational they are exposed to our educational intervention (EI)intervention (EI)– Ex: What do registrants for our activity Ex: What do registrants for our activity

know about the medical topic before know about the medical topic before they begin the course? How are they they begin the course? How are they treating patients with this condition?treating patients with this condition?

1717

Post –Tests: Post –Tests: Walking-out-the Walking-out-the doordoor knowledge knowledge

Post-TestsPost-Tests- Collect data on participant’s - Collect data on participant’s knowledge and clinical practice patterns knowledge and clinical practice patterns regarding the specific subject matter regarding the specific subject matter after after they are exposed to the EIthey are exposed to the EI– Ex: What do CME participants know about the Ex: What do CME participants know about the

medical topic after the course? How are they medical topic after the course? How are they treating patients with this condition now?treating patients with this condition now?

Staggered Post-Tests-Staggered Post-Tests- Administer the post- Administer the post-test several weeks or months later.test several weeks or months later.– Better capture of the true retention of knowledge Better capture of the true retention of knowledge

(or behavior change) by participants(or behavior change) by participants

1818

Experimental and Control Experimental and Control GroupsGroups

Experimental Group-Experimental Group- participants who are participants who are exposed to educational activitiesexposed to educational activities– We are interested in We are interested in theirtheir learning and behavior learning and behavior

change outcomeschange outcomes

Control Group-Control Group- A “comparison” group of A “comparison” group of similar individuals who are not exposed to similar individuals who are not exposed to learning activitieslearning activities– We are interested in how the experimental We are interested in how the experimental

subjects’ outcomes differ from the control subjects’ outcomes differ from the control subjects’ outcomessubjects’ outcomes

– ““Match” on demographic variablesMatch” on demographic variables

1919

Survey Question TypesSurvey Question Types

Knowledge-BasedKnowledge-Based

Practice BasedPractice Based– Case StudiesCase Studies

2020

Sample “Practice-Based” Case Sample “Practice-Based” Case Study QuestionStudy Question

GT is a 73 year-old, non-smoking woman GT is a 73 year-old, non-smoking woman with no history of urinary complaints, with no history of urinary complaints, pelvic organ prolapse or comorbid pelvic organ prolapse or comorbid conditions. GT presents with primary conditions. GT presents with primary complaints of urgency and urinary leakage. complaints of urgency and urinary leakage. She states that she cannot travel beyond She states that she cannot travel beyond her immediate neighborhood, or attend her immediate neighborhood, or attend social events, for fear of experiencing social events, for fear of experiencing episodes of urinary incontinence. Her episodes of urinary incontinence. Her primary goal is to control her symptoms primary goal is to control her symptoms and regain a normal lifestyle.and regain a normal lifestyle.

2121

Sample Case Study Sample Case Study QuestionQuestion

What steps would you initiate in the evaluation of GT?What steps would you initiate in the evaluation of GT?a.a. Recommend initiation of a bladder diaryRecommend initiation of a bladder diaryb.b. Utilize a questionnaire with GT to help distinguish between urge and/or Utilize a questionnaire with GT to help distinguish between urge and/or

stress incontinencestress incontinencec.c. Perform a urinalysis Perform a urinalysis d.d. All of the aboveAll of the above

Which of the following steps would you take in the management of GT’s Which of the following steps would you take in the management of GT’s condition?condition?

a.a. Teach the patient to initiate pelvic floor muscle exercisesTeach the patient to initiate pelvic floor muscle exercisesb.b. Educate the patient to avoid bladder irritantsEducate the patient to avoid bladder irritantsc.c. Both A & BBoth A & Bd.d. Sacral nerve modulationSacral nerve modulation

Which of the following is LEAST likely to be a diagnosis for GT?Which of the following is LEAST likely to be a diagnosis for GT?a.a. Overactive bladderOveractive bladderb.b. Bladder outlet obstruction Bladder outlet obstruction c.c. Urinary tract infectionUrinary tract infectiond.d. Stress urinary incontinenceStress urinary incontinence

2222

Sample SizeSample Size

You don’t have to poll everyone!You don’t have to poll everyone! Effect SizeEffect Size Power CalculationPower Calculation How to Estimate without having a How to Estimate without having a

statistics degreestatistics degree Rules of Thumb: If you have <30 Rules of Thumb: If you have <30

participants, try to sample all of themparticipants, try to sample all of them

2323

IncentivesIncentives– Encourage participation Encourage participation

We want to reach our target “n”We want to reach our target “n”

– Should be “medically relevant” Should be “medically relevant”

– They They areare permitted in CME permitted in CME AMA guidelines- AMA guidelines- << $100.00 $100.00 To avoid the perception of coercion, may wish To avoid the perception of coercion, may wish

to stay lower to stay lower

– Resources:Resources: www.medicalbooks.comwww.medicalbooks.com www.medbookstore.comwww.medbookstore.com www.allheart.comwww.allheart.com http://solutions.medsite.com/medsite_rewards.asphttp://solutions.medsite.com/medsite_rewards.asp

2424

Challenges to MethodologyChallenges to Methodology

Obtaining contact information for pretest Obtaining contact information for pretest contactcontact– LiveLive– Enduring MaterialsEnduring Materials

Funding LimitationsFunding Limitations

Grantor concernsGrantor concerns– Pre-testsPre-tests– IncentivesIncentives– ObjectivityObjectivity

2525

Challenges to MethodologyChallenges to Methodology

??

2626

Sample ApproachesSample Approaches

Live Meeting (i.e.: Symposium)Live Meeting (i.e.: Symposium)– 250 attendees anticipated250 attendees anticipated– Obtained e-mails via pre-registration Obtained e-mails via pre-registration

websitewebsite Linked pre-test to this siteLinked pre-test to this site

– E-mailed post-test to pre-registrants 4 E-mailed post-test to pre-registrants 4 weeks after the live meetingweeks after the live meeting Screener question to assure they attended Screener question to assure they attended

activityactivity Match pre-test and post-test resultsMatch pre-test and post-test results

2727

Sample ApproachesSample Approaches

Journal SupplementJournal Supplement– Distribution to 7,000 physiciansDistribution to 7,000 physicians

Publisher could not provide e-mail addresses of Publisher could not provide e-mail addresses of recipientsrecipients

Could not anticipate “readers” from population of Could not anticipate “readers” from population of 7,0007,000

– Administered pre-test to control group of Administered pre-test to control group of similarly specialized physicianssimilarly specialized physicians

– Administered post-test to readers and control Administered post-test to readers and control goupgoup Recruited via mention of post-test and incentive in Recruited via mention of post-test and incentive in

the rear of printed supplementthe rear of printed supplement

2828

Sample ApproachesSample Approaches

Online CME ActivityOnline CME Activity– Challenge = no funding for incentivesChallenge = no funding for incentives– 1,000 anticipated participants1,000 anticipated participants– Present with optional (opt-out) pre-test Present with optional (opt-out) pre-test

during log-in/registrationduring log-in/registration– Present with opt-out post-test at close of Present with opt-out post-test at close of

activityactivity

2929

Sample ApproachesSample Approaches

PodcastPodcast– 500 downloads anticipated500 downloads anticipated– Obtained e-mails at point of Obtained e-mails at point of

download/registrationdownload/registration– Opt-out pre-test offered at point of Opt-out pre-test offered at point of

downloaddownload– Mention URL (easy to recall address) at Mention URL (easy to recall address) at

close of podcastclose of podcast mention incentivemention incentive

3030

Example 1Example 1 You wish to evaluate the educational You wish to evaluate the educational

outcomes of a live meeting regarding Diabetes outcomes of a live meeting regarding Diabetes that is to be held next month in your hospital. that is to be held next month in your hospital. The meeting is targeting physicians and The meeting is targeting physicians and nurses, and you anticipate a turnout of 100 nurses, and you anticipate a turnout of 100 participants. Although your funding is limited, participants. Although your funding is limited, you do have unlimited access to the hospital you do have unlimited access to the hospital information technology department should information technology department should you need computer programming assistance.you need computer programming assistance.

– How would you approach assessing outcomes How would you approach assessing outcomes for this CME Activity?for this CME Activity?

3131

Example 2Example 2 Your organization has received funding for a Your organization has received funding for a

series of 6 podcasts on the topic of Heart series of 6 podcasts on the topic of Heart Failure. Each of the 6 podcasts will feature Failure. Each of the 6 podcasts will feature different thought leader interviews regarding different thought leader interviews regarding hot topics in the therapeutic area. You are hot topics in the therapeutic area. You are not sure how many participants will listed to not sure how many participants will listed to your podcasts, since this is the first time your podcasts, since this is the first time your company has administered this type of your company has administered this type of activity. You have a budget of $10,000 for activity. You have a budget of $10,000 for “outcomes” for the entire series. “outcomes” for the entire series.

– How would you approach assessing How would you approach assessing outcomes for this CME Activity?outcomes for this CME Activity?

3232

Example 3Example 3 Your company has received funding for a Your company has received funding for a

monograph publication that is based on the monograph publication that is based on the proceedings of a live event on the topic of proceedings of a live event on the topic of Alzheimer’s Disease. The publishing Alzheimer’s Disease. The publishing company informs you that the monograph company informs you that the monograph will be distributed to 9,000 physicians and will be distributed to 9,000 physicians and allied health care professionals. You have allied health care professionals. You have $15,000 to perform an outcomes assessment $15,000 to perform an outcomes assessment of the enduring material.of the enduring material.

– How would you approach assessing How would you approach assessing outcomes for this CME Activity?outcomes for this CME Activity?

3333

Questions and AnswersQuestions and Answers

???

3434

Contact InformationContact Information

Wendy Turell, DrPHWendy Turell, DrPH

Associate Vice President, Educational Associate Vice President, Educational Design and Accreditation ServicesDesign and Accreditation Services

BCMEBCME

wturell@bcmeonline.comwturell@bcmeonline.com

www.bcmeonline.comwww.bcmeonline.com