Evaluation Research
description
Transcript of Evaluation Research
EVALUATION RESEARCH
Evaluation Research How do we begin?
What are the different types of evaluation research?
How do these different types fit together?
What purpose do they serve?
Begin with a Logic Model
Step 1 – Start with
Program
Model
Step 2 –
Identify
Resources
Step 3 –
Identify
Inputs
Step 4 Identif
y Activiti
es
Step 5 –
Identify
Outputs
Step 6 –
Identify
Outcomes
Other Common Terms Used to Identify Steps
Step 1 – Identify Program Model
Step 2 – Identify Resourc
es
Step 3 – Identify Program Events
Step 4 – Identify Resultin
g Products
Step 5 – Identify ChangesSuccess
Program Model
Teachers Grant Money
Service LearningActivities
Service Learning Projects Complet
ed
Measure Leadership Skills
Example-Service Learning Program
ExampleProgram Model
# of TeacherAmount of Money for Building Materials
Number of Service
Learning Projects
Planned by Students
Number of Projects
Completed
Perception of
Leadership Skills
Developed
Program Model
Teachers Grant Money
Service LearningActivities
Service Learning Projects Complet
ed
Leadership Skills
Another Conceptualization of the Logic Model
Resources
• Step One – Identify and count resources used to instigate program
Activities
• Step Two – Identify and count activities that took place
Outputs• Step Three – Identify and count events or products
that resulted
Short-Term
Outcomes
• Step Four – Identify and count short-term changes that took place (i.e., positive changes or accomplishments immediately after participation)
Long Term
Outcomes
• Step Five – Identify and count long-term impacts that occurred (i.e., positive changes or accomplishments at specified future date)
Service Learning Example Using This Conceptualization
Resources
• Count number of staff and amount of money used to instigate program. Count number of networks established.
Activities
• Count number of planning meetings and number of service learning activities planned by students. Count number of hours students spent in activities.
Outputs• Count number of projects completed by students. This could be
subdivided into things such as number of picnic tables built for a park etc.
Short-Term
Outcomes
• Use survey to assess positive changes or accomplishments as perceived by service learning participants. Compare to control group.
Long Term
Outcomes
• At a specified future time, use survey to assess positive changes in the lives of students (i.e., number of hours spent in volunteer work etc.) Compare to control group
Another Example Using This Conceptualization
Resources
• Count number of staff and amount of money used to instigate 4-H program. Count number of networks established.
Activities
• Number of 4-H activities that occurred, and/or number of students who participated in planning activities
Outputs
• Number of students who participated in workshops and projects. Number of educational packets distributed.
Short-Term
Outcomes
• Assessment of their sense of accomplishment using survey. Assessment of increase in knowledge using survey.
Long Term
Outcomes
• Number of 4-H students who attend college after attending educational workshops. Compare to control group.
Types of Evaluation Research In order of Use
Outcome Evaluation
Process Evaluation
Formative Evaluation
Needs Assessment
Example of Each Type of Evaluation
Example - Surveys, intensive interviews and focus groups are used to determine (1) Long-term impact on USU students and (2)Long term -
impact on high school
Outcome Evaluation
Process EvaluationAn ongoing evaluation is used to monitor the program
Example -Grantee comes from time to time to “see how things
are going”.
Formative EvaluationInformation is gathered to formulate a plan to establish a
program.
Example-Planning meetings with high school administrators
etc.
Needs AssessmentA need for a program is identified
Example –need for mentoring program with public school
students
Needs Assessment Purpose
Evaluate the need for a program/intervention/ project
Basic Steps Use secondary data (i.e., census data, community
reports etc.) and/or surveys etc. to create a profile of the community that assess existing resources and current needs.
Determine if current resources are adequately meeting needs.
Identify one or more specific needs that are not currently being met.
Example - Needs Assessment
Purpose Determine the percent of local post secondary students
that are from an underrepresented population. Compare this to the percent of students in post secondary educational institutions. Based on this and programs already available determine the need to provide additional opportunities for this group.
Basic Steps Use secondary data to determine percent of population
that is Latino. Use USU data to compute the percent of USU that is Latino. Compare differences in percentages. Note any discrepancy.
Utilize local webpages and phone books, identify programs/organizations that focus on the Latino population and meeting their needs.
Formative Evaluation Purpose
Plan (form) a program that will met the needs of the community, keeping in mind the need to develop indicators that can be used to evaluate the success of the program.
Basic Steps Locate other similar programs to use as a model
Meet with “experts” in this area and get their advice
Organize a program and apply for a grant
Form an “ad hoc” evaluation committee to evaluate program
Formative Evaluation Purpose
Plan a mentoring program where USU students mentor high school students. Identify objectives (i.e., increase high school graduation rates, increase college retention rates of mentees).
Basic Steps Identify other mentoring programs
Meet with directors of programs, professors etc. who have been/are involved in mentoring programs.
Organize your mentoring program and apply for a grant to implement program.
Select members of your committee to engage in evaluation research
Purpose To monitor the implementation of your program on an
ongoing basis. It is a type of program monitoring.
Steps Specifically describe your program
Decide what products to count Consider objectives (verb) (target)
(date)
Decide on valid way to count products Identify process indicators
Quality Assurance Identify process indicators
Process Evaluation
Purpose Monitor the mentoring program and address two questions:
(1) Is the mentoring occurring as planned, (2) How are the mentees
responding? Steps
Describe Program The mentoring program connects high school mentees with
USU mentors through a monitored blog. Decide what products to count
The number of hours that mentors spent in contact with their mentee etc.
Decide on valid way to count products Have program administrator count hours spent on website
Quality Assurance High school counselors will be contacted periodically to
determine how mentees are responding
Example-Process Evaluation
Purpose To determine how well you meet the objectives of your
program.
Steps Identify short-term and long-term outcomes
Construct indicators to measure outcomes
Decide on data collection and sampling techniques
Collect data
Analyze data
Write Report
Outcome Evaluation
Purpose To determine how well you meet the objectives of your program.
Steps Short term outcomes are establish relationship between mentors
and mentees. Long term outcomes are higher graduation and retention rates for mentees.
Mentees will be asked questions about their experience, and then about their graduation/retention
Surveys will be administered to all mentees after completion of program and then again in three years.
Surveys are administered
Using SAS, data is entered and analyzed.
Report is written
Outcome Evaluation
Quality Assurance vs. Program Outcome Evaluation
Inform Administrators and
ClientsRely on “Expert”
Opinion
Inform Administrators
Rely on Statistics
Quality Assurance
Legislative Mandate
Client Satisfaction Is Focus
Outcome Evaluation
Not Mandated
Does Not Necessarily Focus on Client
Satisfaction
Dr. Carol Albrecht Extension Assessment Specialist Phone: 979-777-2421 [email protected]