Program Evaluation: Simplifying The Program Evaluation Process
Program Evaluation In A Nutshell
description
Transcript of Program Evaluation In A Nutshell
Program Evaluation In A Nutshell
1
Jonathan Brown, M.A.
Two Types of Evaluation
• At the most basic level there are two types of evaluation:
Formative: primary purpose is to provide information to improve a program.
Summative: concerned with providing information about program adoption, continuation, or expansion.
2
Two Types of Evaluation
3
Rel
ativ
e E
mp
has
is
Program Life
Summative Evaluation
Formative Evaluation
What Are You Assessing?
The evaluation plan must consider the situation or context of the evaluation.
• The purpose of the evaluation.
• Improvement (formative evaluation).
• Accountability (summative evaluation).
• Knowledge generation.
• Hidden agendas.
4
What Are You Assessing?
The evaluation plan must consider the situation or context of the evaluation.
• The program’s structure and context.
• The stage of program development.
• Administrative and political context.
• Conceptual and organizational structure of the program.
5
What Are You Assessing?
The evaluation plan must consider the situation or context of the evaluation.
• The resources available for the evaluation.
• Personnel
• Equipment
• Facilities to support data collection, analyses, and reporting.
• Is specialised expertise needed?
6
What Are You Assessing?
Typically, evaluation involves assessing one or more of the five following areas:
1. The need for a program.
2. The program’s design.
3. The program’s implementation or process.
4. The program’s impacts or outcomes.
5. The efficiency of a program.
7
Needs Assessment
• Purpose of a social program is to alleviate a social problem.
• Needs assessments assess:
• The nature, extent, and distribution of a social problem.
• How these features will impact the design of an intervention program.
8
Needs Assessment
These assessments can be conducted when…
• Planning a new program.
• Restructuring an existing program.
• Determining if an existing program is responsive to the current needs of the target population.
9
Assessing Program Theory
• Program theory evaluation is an evaluation guided by a program’s explicit theory of how it causes intended outcomes.
• Program theory outlines the following:
•Target population
•Resources
•Activities
•Outcomes
• A program’s theory is represented by Theory of Change Models (TCMs) and Program Logic Models (PLMs).
10
Assessing Program Theory
11
Conditions before the project
• With no information about the program, it remains a mysterious black box.
• The evaluation will also not be able to attribute the final conditions to the specific aspects of the program.
Conditions after the project
The Program is a
Assessing Program Theory
• Assessing program theory can be done by applying SMART principles (Knowlton & Phillips, 2009).
• Specific – the program is clear enough to implement and evaluate.
• Measurable – indicators can be qualified and/or quantified.
• Action oriented – activities will provoke the desired change in targets.
• Realistic – the program is plausible and feasible.
• Timed – duration of activities and intended outcomes are specified.
12
Process Evaluation
• Given that a valid theory on how to intervene on a diagnosed social problem has been done, the program must be implemented well to be successful.
• Two forms:
• Process evaluation
• Program monitoring
13
Process Evaluation
• Process evaluation looks at service utilization and program organization.
Service Utilization
• The extent to which the intended targets receive services.
• Important when participation is voluntary, or participants must learn new procedures, or habits.
• Coverage
• Bias
14
Process Evaluation
• Process evaluation looks at service utilization and program organization.
Program Organization
• How well the program is in terms of managing its efforts and using its resources to accomplish its activities.
• Service delivery:
• None or not enough intervention
• Wrong intervention
• Unstandardized intervention15
Outcome Evaluation
• Also known as an impact assessment.
• Assesses the extent to which a program produces the outcomes it intends.
• But the desired changes could be caused by factors unrelated to the program.
• Interested in the changes produced by the program above and beyond any external elements.
16
Outcome Evaluation
Program outcomes
• The states or conditions of the target that changed as a
result of a program.
1. Observed characteristics of the target population or social
conditions.
2. Occur to different degrees.
17
Outcome Evaluation
Program outcomes
• Outcome level – status of an outcome at a point in time.
• Outcome change – difference between outcome levels at different points in time.
• Program effect – the unique portion of an outcome that can be attributed to only the program.
18
Outcome Evaluation
19
Out
com
e V
aria
ble
Before Program
During Program
After Program
Outcome status without program
Outcome status with program
Program effect
Post-program outcome level
Pre-program outcome level
Outcome Change
(Rossi, Lipsey, & Freeman, 2004)
Efficiency Assessment
• Due to limited resources, program accomplishments must be judged against program costs.
• Two types of efficiency assessments:
• Cost-benefit analysis
• Cost-effectiveness analysis
• Such assessments require placing a dollar amount on program activities and benefits.
20
Typical Evaluation Questions
Needs Assessment
• What is the nature and magnitude of the problem?
• What are the characteristics of the population in need?
• What are the needs of the population?
• What services are needed?
• How much service is needed and over what period of time?
• What service delivery arrangements are needed to provide services to the population?
21
Typical Evaluation Questions
Program Theory Assessment
• Is the program doing the right thing?
• How could the program better address the needs of participants?
• How should the program be organized?
• What are the best delivery systems for the services?
• Is the program specific enough to measure?
22
Typical Evaluation Questions
Process Evaluation
• How many persons are receiving services?
• Are administrative and service objectives being met?
• Are the intended targets receiving services?
• Are the necessary program functions being performed
adequately?
• Are resources being used effectively and efficiently?
23
Typical Evaluation Questions
Outcome Evaluation
• Are the goals and outcomes being achieved?
• Do the services have adverse side effects?
• Are some participants affected more by the services than others?
• Is the problem the services are intended to address made better?
24
Typical Evaluation Questions
Efficiency Assessment
• Are resources used efficiently?
• Is the cost reasonable in relation to the magnitude of the
benefits?
• Would alternative approaches yield equivalent benefits at less
cost?
25