Expertise, Consumer-Oriented, and Program-Oriented Evaluation Approaches
Chapter 6: Program-Oriented Approaches
-
Upload
felicia-mcdonald -
Category
Documents
-
view
38 -
download
1
description
Transcript of Chapter 6: Program-Oriented Approaches
Chapter 6: Program-Oriented Approaches
Presentation by Jay Kerstetter, Amanda Brown, Jody Yoos, & Jane Lightner
J.K.
Orienting Questions: #1 What are the key concepts of the objectives-oriented evaluation
approach?#2 How has this approach influenced evaluation?
What is it? Objectives-oriented evaluation approach helps determine
whether some or all of the program objectives are achieved and, if so, how well they are achieved.
Evaluators may work with stakeholders to establish if program objectives are met.
Information from this approach can assist with deciding to maintain, terminate, or change approaches within the program.
J.K.
Tylerian Evaluation Approach
1. Ralph Tyler is credited with initiating this approach in the 1930’s
2. Tyler began to formulate his views on education and evaluation.
3. His approach included the following steps: Establish goals or objectives Classify the goals or objectives Define objectives in behavioral terms Find situations in which achievement of objectives can be shown Develop or select measurement techniques Collect performance data Compare performance data with behaviorally stated objectives
4. This approach was readily adoptable by evaluators and had great influence on evaluation theorists.
J.K.
Provus’s Discrepancy Evaluation Model:
1. Developed by Malcolm Provus; viewed evaluation as a continuous information management process.
2. Provus stemmed key characteristics of his proposal from Tyler.3. Provus viewed evaluation as a process.4. This process was called DEM; Discrepancy Evaluation Model, which
are broken into four developmental stages.◦ Definition ◦ Installation ◦ Process◦ Product◦ Cost-benefit analysis (optional)
The DEM was designed to facilitate the development of programs in large public school systems and later applied to state evaluations by federal bureau.
The DEM was one of the earliest approaches to evaluation and elements of it can still be found today in many evaluation. J.K.
Orienting Question #3: How is the objective-oriented evaluation approach used today?
Standards-based testingAccountability in educationPerformance monitoring systems used in many government programs
Many refinements to the system since the 1930’s
J.Y.
Ralph W. Tyler: Tylerian Evaluation Approach
Influenced the Elementary and Secondary Education Act (ESEA) of 1965, first act to require evaluation of educational programs
Started the National Assessment of Educational Progress (NAEP), only way to examine educational achievement of al 50 states.
J.Y.
Malcolm Provus: Provus’s Discrepancy Evaluation Model
Approach evaluated in Pittsburgh public schools
Viewed evaluation as a continuous information- management process designed to serve as “the watch-dog of program-management” and the “handmaiden of administration in the management of program development through sound decision making”
J.Y.
Orienting Question #4a: How are Logic Models used in evaluation?
Developed as an extension of objectives-oriented evaluation
Designed to fill in those steps between the program and its objectives
Program planners/evaluatorIdentify programo InputsoActivitiesoOutputs (immediate
program impacts)oOutcomes (long-term
objectives/goals)
J.Y.
Today Logic Models are used in program planning/evaluation
Help program staff articulate /discuss how program might achieve goals
What elements are important to evaluate at any given time
Build internal evaluation capacity or think in an evaluative way
Example Companies:United way of AmericaWK Kellogg FoundationAnnie E Casey Foundation
J.Y.
Orienting Question #4b: How are Program Theories used in evaluation?
Theory-Based evaluationIs used by evaluators to gain a better understanding of
the programBetter define the evaluation questions the study should
addressTo aid their choices of what concepts to measure and
when to measure themTo improve their interpretation of results and their
feedback to stakeholders to enhance use.
J.Y.
Steps in Theory Based Evaluation
1) Engage Relevant stakeholder2) Develop a first draft of program theory
evaluator or evaluation team3) Present draft to stakeholders for further
discussion, reaction, and input4) Conduct a plausibility check5) Communicate findings to key stakeholders6) Probe arrows for model specificity7) Finalize program impact theory
J.Y.
Theory Driven Evaluation
Work with stakeholders to identify key questions to be answered
in the evaluation and the appropriate designs and methods for
answering those questions
Emphasis is on testing the program model
Provide guidance as to what to measure and when to measure it
The selection of the evaluation questions to be addressed depend
on the stage of the program and what stakeholders hope to learn
Provides evaluator with critical information that can be used
throughout the evaluation
J.Y.
Orienting Question #5: How do theory-based evaluation and objectives approaches differ?
Objectives Approach:1. The objectives are
identified by group looking for the evaluation.
2. Purpose of activity specified and then those purposes are being acheived.
3. Individual credited-Ralph W. Tyler
Theory-Based1. The evaluator discusses
goals, basics, and objectives of program w/the stakeholders.
2. The evaluator decides how the program should work and then sees if it works that way.
3. Huey Chen and Leonard Bickman helped to develop approaches to theory based.
4. Science based and quantitative.
A.B.
Question #6: What are the strengths and limitations of program oriented
evaluation approaches?
Strengths:Objective oriented – the simplicity of the
concept and program makes it easy to understand, follow and possibly implement. ◦Face validity – the evaluator is being held
accountable for what is being asked to be evaluated. They want to see if what is currently happening or working based on their own goals and objectives for the program.
A.B.
Strengths (con’t)
Theory oriented – the chance for the evaluator to partake in dialogue with the stakeholders is a strength of the program because it helps them to expand their knowledge on the topic. ◦This then leads to the evaluator to have a clear
understanding of the program so they know how to properly evaluate the program.
A.B.
Weaknesses:
Objective oriented – the evaluator can have a single minded focus on the objectives which will cause them to overlook the complications, elements and factors contributing to that programs success or failure.a. This approach does not ask for the evaluator to gain
an understanding of the context in which the program operates this could then be what is affecting the program’s success or failure.
b. Evaluator could ignore the actual value of the objective. Since the evaluator is told what the objectives are they are not asked to evaluate whether that objective even fits the program.
A.B.
Weaknesses (con’t)
Theory oriented – this like the objective oriented can cause the evaluator to ignore certain aspects of the program that are important because the evaluator is so focused on the theory of how it runs and not how it is actually running.a. Evaluators may ignore the needs or values of
stakeholders involved with the program.b. This approach may also oversimplify the
complexity of the program making it feel easier to evaluate than it really is because not all of the surrounding factors are accounted for in the process.
A.B.
Orienting Question # 7: What is goal-free evaluation?
Rationale: “Goals should not be taken as givens.”
Developed by Scriven (1972)---believes the most important function of a goal-free evaluation is to reduce the bias that occurs from knowing program goals
Thus…increase objectivity in judging the program as a whole.
J.L.
Orienting Question #8: What does it teach us about conducting an evaluation?
Goals can act as “blinders” causing us to possibly miss the most important outcomes not related to the goals.
Goal-free evaluation was proposed to primarily indentify the unanticipated side effects that an objectives-oriented evaluation might miss
J.L.
Major Characteristics of a Goal-Free Evaluation
1. Evaluator purposely avoids
becoming aware of program goals.
2. Predetermined goals are not
permitted o narrow the focus of the
evaluation study.
3. Goal-free evaluation focuses on actual outcomes rather than intended program outcomes.
4. The goal-free evaluator has
minimal contact with the program
manager and staff.
5. Goal-free evaluation increases the likelihood that unanticipated side
effects will be noted.
J.L.
Goal Directed + Goal-Free= can work together
Internal Goal Evaluator vs. External Goal-Free Evaluator
Goal-directedHow well is program
meeting goalsProvide information to
administrator
What does the program do?
Looking at ALL the programs outcomes, intended or not
J.L.
Information taken from Program Evaluation:
Alternate Approaches and Practical Guidelines
p. 153-171