ExtensionProgram EvaluationDale A. MooreVMTRCSchool of Veterinary MedicineUC Davis
What’s the purpose of evaluation? Improve the program Improve teaching Make a difference Scholarly pursuit Get the higher-ups off your back Meet promotion requirements
Goals of This Presentation Discuss “traditional” program evaluation
Help you focus your evaluations Learn the different levels of evaluation Help you construct evaluation questions Provide you a tool for data analysis
Discuss educational program evaluation using Learning Stage Theory
Pre-Test Take out the Pre-Test Take 5 minutes to
answer the questions Turn them in to Pat
Who will use the results? The public Your extension division Colleagues You and program planning staff … All your stakeholders
Who will use the results?
Your Audience
Do I have the resources to conduct this evaluation project? Time Money People Expertise Or – are the data already available?
Levels of Evaluation Perceptions of the program or course Competence with new skills, knowledge,
abilities or new attitudes Individual performance – a change in
behavior within their environment Industry outcomes
What are your outcomes?
Mechanisms for Evaluation Direct observation Audience response systems Surveys Telephone or face-to-face interviews Focus groups Expert or peer-review (formative evaluations) Testimonials Industry data
Audience Response System
Surveys Who is the audience? How are you going to deliver it? Are you going to randomly choose possible
responders? What percent response do you need to make it
meaningful? Pre- and Post-program delivery to address
immediate changes in knowledge/attitudes?
Pre-Test --–-- Post-Test
Pre-Test --–-- Post-Test Advantages: Get attention at beginning of the program Provide some level of anxiety so that they
“look” for the answers during the presentation Gives instructors some measure of learning
needs / areas upon which to focus Identify short-term changes in knowledge /
attitudes
Pre-Test – Post-Test Disadvantages: Takes time away from the program Need to be able to quickly “grade” and analyze
your data Need to be able to make adjustments to the
program if needed Only addresses short-term learning and not
behavior change Adults may not enjoy being graded
What kind of questions? Reaction Learning Behavior Results
?
What kind of questions/data if the outcome is behavior change?
Questions about… Satisfaction with different aspects of the
program Speaker effectiveness New knowledge or skills A change in attitude A “commitment to change” New program ideas
Question Types Open- ended questions Closed
Set of responses Likert scale Numeric responses Categorical responses
What are the advantages and disadvantages of each kind of question?
Data Analysis How do you go about analyzing responses to
different kinds of questions? Comparison of means – Student’s T-test Comparison of proportions Categorical data analysis – Chi-Square Non-parametric like the Sign test Multivariate analyses
* EpiCalc2000 – free program from Web; URL in handout
Documenting the Impacts of Continuing Education
Dale Moore and Hank Slotnick
Do you sometimes feel that…
despite…• Results of your needs assessments, and • Your best intentions to create an appealing
program
Those who need to attend the program, don’t? The minds of some who come aren’t open?
Objectives Provide the theoretical framework for
documenting change Demonstrate applications of the theory Demonstrate a new method of
representing change data
Background on the Problem Key meta-analytic findings of CME programs:
Findings consistent with adult learning: Simple provision of information is often
insufficient. Active learning appears to be more effective. Multiple exposures predispose toward success.
Most traditional CME does not changedoes not change physician behavior
Why not?
Behavioral Change Theories Prochaska & Mezirow & Slotnick
Suggest that change occurs in stages In each stage the learner has different needs To become interested in the idea of change
• the learner’s mind must be opened• the learner must feel a need to attend to
something, something has happened to unsettle them, or they have an immediate problem to solve
INTERESTED, BUT…. (contemplation)
CHOICES:DECISIONS (preparation)
ACTION
FINE TUNING & MAINTAINING CHANGE (maintenance)
UNAWARE OR UNIMPRESSED (precontemplation)
Adapted from: Prochaska et al, 1992
STAGES OF CHANGE
Mezirow: Theory of Perspective Transformation
1) Disorienting dilemma
2) Self-examination with feelings dissonance or discomfort
3) Critical re-assessment of assumptions
4) Recognition of discontent and that others have changed
5) Exploration of options for new ways of seeing/behaving
6) Planning a course of action, gathering resources
7) Acquisition of knowledge and skills
8) Provisional trying of new perspective/behavior
9) Building of competence and self-confidence in new activity
10) Integration of new perspective/behavior into one’s life
Traditional CE Efforts
Potential for mismatch between the intervention and learner readiness Over-emphasis on encouraging action
Ask them to take the leap because we say so! Presumes learners minds are open
Documenting the Outcomes A taxonomy of outcomes
Awareness of problem Knowledge/skill changes and Practice changes Improved herd outcomes
Aware
ChangeImproveSlotnick 2001
Taxonomy of Outcomes Taxonomy based on stages in
learning episodes Evaluation: Learner decides
whether to learn the solution to the problem at hand
Learning: Learner gains required skills and knowledge
Gaining experience: Learner familiarized with the skills, knowledge & problems.
Aware
ChangeImprove
Evaluation Learn
Gain Experience
Documenting Change
Hypothesis: It should be possible to document changes in prevalence of learners at different stages in addressing an educational program.
How to Document Outcomes Present clinical/practical vignettes. Ask respondents to indicate their ‘learning status’ for each
vignette/problem. Vignette example: Mary was a 58-year-old woman who had
been in good health generally. She did not exercise regularly and was about 20 pounds over her ideal weight, but her diet was relatively low in fat, she did not smoke, and she had no history of diabetes. She had been taking estrogen and progesterone hormone replacement therapy for 6 years. Her recent blood lipid panel results were as follows: (blah blah blah). She was taking hydrochlorothiazide to treat hypertension, and her mother had experienced an MI at 60 years of age. Mary was seen at the emergency department for a prolonged episode of left shoulder and upper back pain which had begun after dinner that evening.
Responses Evaluation: I’m comfortable that I can handle the
problem in the vignette; all I might be interested in is information so I can decide when next to update.”
Learning: I need to update myself in the skills and knowledge needed to address this problem.
Gaining experience: I’ve recently updated skills and knowledge. All I might be interested in is hearing about others’ experiences with similar problems.
How to Document Outcomes The AMEE (Slotnick) study
Goal: Introduce the idea of estimating prevalence for people at different stages of learning in a lecture setting (90 minutes)
Measurement: Practice vignettes were examined pre- and post, some based on issues addressed in the lecture and some not.
Analysis: Expected to see stage-to-stage movement for the first set, but not the second.
The Triangular Graph
Each side representsa response – 3 possibleresponses.
Evaluation
Learning GainingExperience
0
10
20
30
40
50
Eval Learning Gain Exp
Response
% R
espo
nse
50%
50% 50%
Eval Dominated
Learning Domin. Gain Exp.
Evaluation
Learning GainingExperience
0
10
20
30
40
50
Eval Learning Gain Exp
Response
% R
espo
nse
50%
50% 50%
Evaluation
Learning GainingExperience
0
10
20
30
40
50
Eval Learning Gain Exp
Response
% R
espo
nse
50%
50% 50%
How to Document Outcomes The AMEE study.
3 Vignettes where no changes were expected.
Evaluation
Learning GainingExperience
“The task force chair needs a list of physicians who might be called on in an emergency. She asks you to locate such a list.”
How to Document Outcomes The AMEE study.
4 Vignettes where changes were expected.
Evaluation
Learning GainingExperience
“You and two others are to put together a pencil-and-paper survey questionnaire to determine what doctors know about preventing, diagnosing, and managing anthrax and small pox.”
Dairy Veterinarian Study
Dairy Informatics Vignettes
“A 500-cow dairy client is going to expand his operation to add 500 more cows. To help support his application for a loan, the bank has asked him for information on the productivity of the dairy herd, including milk production and herd replacements. The bank would like information for the last 3 years of the dairy’s operation.”
Developed 8 practice vignettes: An example --
Moore’s Study
Evaluation
Learning Gaining Experience
Moore’s Study
Evaluation
Learn Gain Experience
Conclusions Evaluate group changes dependent on
stage of learning Documents the changes Provides for parsimonious graphical
display of change Currently working on statistical methods
Post-Test Take out the Post-
Test Take 5 minutes to
complete Turn in to Pat
What did we learn? There are different levels of program evaluation Clearly-defined program objectives lead to
easier evaluations There are a number of mechanisms of program
evaluation Learning stage theory helps us evaluate WHERE
the learner is with regard to making a change
Thank you for
your attention
References Dr. Jack Mezirow: Transformative Dimensions of Adult
Learning; Fostering Critical Reflection in Adulthood: A Guide to Transformative and Emancipatory Learning.
Prochaska, J. O. and C. C. DiClemente (1982). "Transtheoretical therapy: Toward a more integrative model of change." Psychotherapy: Theory, Research and Practice 19(3): 276-288.
Slotnick HB. How Doctors Learn: Physicians' Self-directed Learning Episodes. Acad Med 1999;74:1106-17.
Moore DA. Evaluating the stages of veterinary practitioner learning for continuing education needs assessment and program evaluation. J Vet Med Education 2003;30(1):50-56.
Top Related