The Deming Wheel From the graphic point of view, P.D.C.A. is represented by a moving circle The...
-
Upload
janis-hunter -
Category
Documents
-
view
220 -
download
2
Transcript of The Deming Wheel From the graphic point of view, P.D.C.A. is represented by a moving circle The...
The Deming Wheel
• From the graphic point of view, P.D.C.A. is represented by a moving circle
• The movement stands for dynamism and continuity of the application process.
From the graphic point of view, P.D.C.A. is represented by a moving circle called the Deming wheel.The movement stands for dynamism and continuity of the application process.P D C A CYCLE
http://www.iwolm.com/en/the-pdca-method-or-deming-wheel-for-your-improvement/11
Theoretical Frameworks Utilization Focused (Patton)
• Evaluation of effectiveness by potential users• Who might use• What will be used• Intention to use
12
Evaluation Techniques
Logic Models
“A graphic representation of a program that describes the program’s essential components and expected accomplishments and conveys the logical relationship between these components and their outcoms.” (Conrad et al. 1999)
Logic models guide evaluation
http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html
14
How the Logic Model Guides Evaluation
•Provides the program description• Aids in matching evaluation to the program• Identifies what and when to measure
• Are you interested in process and/or outcomes?
• Keeps focus on key, important information• What do we really need to know??• Where should limited evaluation resources be used?
15
OUTPUTS
Basic Logic Model
INPUTS OUTCOMES
ParticipantsActivities
16
Input
Products (teaching components)
InputsInvestment/Resources
Time
People
Money
Materials
17
Output
Activities ParticipationWhat is done Who receives
Training/teaching Counts
Products (teaching components) Description
Develop resources Satisfaction
Form collaborations/partnerships
Assessment tools and assessment
18
Outcomes
Primary Secondary Tertiary Immediate
(Short term)Behavior Results
(Long-term) Knowledge Teaching Adaptation
Skills Practice Standardization
Abilities Improved practices
Changes in attitudes
Assessments (tools and assessment)
19
When to evaluate?
• Before the experience/event/class• Mini-assessments within (quiz, exercise, etc)• Post-test only• Pre-test and Post-test• Retrospective Pretest and Post-test• Pre-test, Post-test, and follow-up• Intermediate testing
• Can be combined with pre and post test designs
20
Evaluation Design
• Experimental• Uses random assignment, control
• Quasi-experimental• Groups from “natural” characteristics
• Males to females
• Class A to Class B
• Non-experimental • Compares before and after
21
Types of Assessment
• Qualitative• Open-ended
• Quantitative• Constrained Choice
• Mixed Method• Using both quantitative and qualitative techniques
to collect data
22
Mixed Methods Evaluation
• Enhances both formative and summative evaluation
• Triangulation from combining both types of evaluation• Merging• Explanatory • Exploratory
23
Mixed Methods Evaluation - Merging
• Triangulation from combining both types of evaluation• Merging
Interpretation
Qualitative Quantitative
24
Mixed Methods Evaluation - Explanatory
• Triangulation from combining both types of evaluation
• Explanatory • Quantitative, then Qualitative, followed by interpretation
Quantitative QualitativeIdentify Issues Interpretation
25
Mixed Methods Evaluation - Exploratory
• Triangulation from combining both types of evaluation
• Exploratory
Qualitative QuantitativeDesign Measure Interpretation
26
Mixed Methods Evaluation
• Triangulation from combining both types of evaluation• Merging
• Explanatory • Quantitative, then Qualitative, followed by interpretation
• Exploratory
Interpretation
Quantitative
Qualitative
QualitativeIdentify Issues
Quantitative
Interpretation
Qualitative QuantitativeDesign Measure Interpretation
27
Selecting the Evaluation Method
• Participation records• Self-report • Achievement (knowledge) tests• Interviews• Focus groups• Direct observations • Medical record reviews• Product count
28
Example
29
Example
30
Examples
31
1.1 The percent of adults aged 65 and older in the United States is currently approximately ________ and will increase to ________ by the middle of the century.a. 10% 25% b. 12% 20% c. 12% 15% *d. 13% 20% e. 13% 25%
1.2 Persons reaching age 65 have an average life-expectancy of an additional _____ years.a. 5 b. 9 c. 10 d. 15 *e. 19
Examples:
32
Example Faculty Scholars Program Evaluation - Focus Group Questions
Group Leader (minimal input only, to encourage participation, and probe for responses): _______________________________________
Note Takers (should not speak or interact with participants, not necessary to link participants to responses except for question 1:
_________________________ ___________________________
1. Introductory questions: Pass out index cards and ask to have these two questions written. After collecting cards, go around the room and ask for answers. Note numbers of persons who find any specific topic useful, even if not the “most useful.” If a card notes that “Advanced Directives” was really useful, ask how many others also found this material pertinent. It will be interesting to all to learn how implementation will be achieved.
What did you find the most useful topic in the program this year?
How have you or how do you plan to implement what you have learned?
2. Transition questions
Are there topic areas that should be added to the program? Deleted? More or less emphasis?
3. Key questions
What are your overall thoughts about the program?
Communication: PROBE: Communication about the program? (e-mail, phone conversations, reminders, etc.)
How did you like/use: the WIKI, jump drives, hard copies? Do you have suggestions for the Newsletter guideline?
Content: PROBE: in-person sessions, self-study materials, conference?
Organization: PROBE: Smaller work teams? Group work? Self-directed projects?
How was the interaction with faculty scholars from other disciplines?
5. Ending questions
If there was one thing you would suggest . . . 33
Evaluation: Quality Improvement vs. Research
• Institution-specific• Reporting to stakeholders is not research
• Funders• Collaborators
• UAB Institutional Review Board defines research as any presentation in a local or national forum• IRB approval is needed
34
American Evaluation Association
• Founded 1986• http://www.eval.org/p/cm/ld/fid=1
35
Evaluation Strategies for
Educational Interventions
November 1, 2013 GEC Faculty Scholars Program
Patricia Sawyer, PhD