Outcome Measurement

Kaitlyn WarkProgram Evaluation OfficerThe Community Foundation

Michael MallettResearch & Evaluation Director

United Way of Greater Richmond & Petersburg

Program Evaluation: Awesome or Evil?

1. Why is evaluation scary?

2. Why is it important?

Why Evaluate?

To better serve children in Richmond Public Schools.

• Learning and improvement• Accountability

FAQsWhen will RPS use your evaluation data to make decisions on partners?

• RPS expects to see evaluation data and use those in decision making for the 2016-17 school year.

Do we have to measure grades? Is RPS only interested in grades?

• You should measure what you hope to accomplish with your participants (students, staff, etc.). For example, if your program is teaching healthy eating habits, you should measure daily vegetable intake.

What is the baseline expectation?• To measure the program or service’s outcome indicators. (You have an

evaluation plan developed and implemented).

What if I need help?• You’ll leave today with a wealth of resources both in the partners present and

this presentation with links to other resources will be emailed and on the Community Partnerships website.

Today’s Training Goals

1. Identify your program’s long term outcome

2. Identify a few potential outcome indicators for your program

3. Identify possible measurement strategies

4. Identify relevant measurement resources

Training Goal #1:

Identify your program’s long-term outcome

Outcome Definition

The measurable changes your program produces for participant:

–Knowledge–Behavior–Skills–Attitudes –Condition/Resources

Outcome Definition

Long-term outcomes–Capture enduring changes –Usually measured a year to multiple

years after the program–Usually more difficult to measure than

short-term or intermediate outcomes

Criteria for Success

Puts focus on the client

Answers the “so what?” question

Is related to your program activities – is something program can influence

Is important to your staff + outside stakeholders

Developing Long-Term Outcomes

Who? Will Do What?(an active verb)

What is the desired outcome?

Clients

Students

Teachers

Demonstrate

Improve

Obtain

Maintain

Display

Grades in schoolTeaching Skills

Social/ Emotional SkillsLeadership Skills

Health Behaviors

Long-Term Outcome Examples

“Students will demonstrate emotional intelligence and positive communication

skills.”

“Teachers will successfully employ positive behavior support interventions.”

Outcome?

“Students are satisfied with the program.”

Puts focus on the client

Answers the “so what?” question

Is related to your program activities – is something program can influence

Is important to your staff + outside stakeholders

Outcome?

“Students will increase their performance on end-of-course exams and will be prepared to advance to the next grade.”

Puts focus on the client

Answers the “so what?” question

Is related to your program activities – is something program can influence

Is important to your staff + outside stakeholders

Outcome?

“Students will attend 95% of program days.”

Puts focus on the client

Answers the “so what?” question

Is related to your program activities – is something program can influence

Is important to your staff + outside stakeholders

Developing Long-Term Outcomes

? What are you trying to achieve with your clients?

? What does success look like, for your clients?

? If you are successful, how will your clients be different after the program than before?

? What kinds of changes do you want to see?

Adapted from Patton: Utilization-focused Evaluation

Helpful Questions: Vision and Goal Setting

Developing Long-Term Outcomes

? Can your program really influence the outcome in a meaningful way?

? Is this outcome truly important to you?

? Is this outcome important and valid to others?

? Is the outcome written using language that most people will understand?

? Are there unintended consequences?

Helpful Questions: Importance

Think – Pair – Share

1. Spend 2 minutes crafting a long-term outcome for your program.

2. Share your long-term outcome with your neighbors.

3. Give/receive feedback4. Choose one long-term outcome to share

with the rest of the audience.

Training Goal #2:

Identify potential outcome indicators for your program

Define Outcome Indicators

• Specific information or data that track your progress towards your long-term outcome goals.

• They describe observable, measurable characteristics or changes that indicate achievement of an outcome.

Outcome Indicators

Observable What does the outcome LOOK like? What can you SEE that is different?

Measurable What can you count, weigh, measure?

Specific, Clear, Not Ambiguous

Client Focused (generally)

Quantitative Number and percent of clients who…

Define Outcome Indicators

Long-term Outcome: Students pursue a healthy lifestyle

Outcome indicators: # and % who report not smoking at end of

program # and % maintaining a BMI within a healthy range # and % who report getting at least two hours of

exercise each week

Researching Indicators

• Validity– Are you measuring what you think you’re

measuring?• Fidelity

– Use research as a guide– Are you implementing best practices from

evidence-based programs?

Outcome Indicator Examples

Type of Program Outcome Indicators

Tutoring Program Students will improve their academic performance and graduate high school on time.

# and % of students either improve by a letter grade or maintain a B or above

# and % of students who graduate within four years of entering high school

Professional Development for Teachers

Teachers will gain skills in classroom management and create a safe environment for learning

# and % of teachers who scored 80% or higher on the training post-test

# and % of teachers who report using scaffolded checks for understanding at least once per lesson

# and % of students who rated their mentor relationship as good or excellent

Average length of match

Mentoring Students will develop healthy relationships with adults and improve their social skills

Group Exercise

1. Spend 5 minutes:• Identifying 2 – 5 outcome indicators

for your program.

2. Share your outcome indicators with your neighbors.

3. Give/receive feedback, and fine-tune your indicators

Discussion

• What problems and issues did you run into trying to create your outcome indicators?

• What are your next steps?

Training Goal #3

Identify possible measurement strategies

Measurement & Data CollectionUse existing data sources if possible.

Possible data sources:Existing files or databases

Intake and exit records Case notes Follow-up calls and notes

Assessments or tests (validated if possible) Harvard Family Research Project OST Evaluation Database

http://www.hfrp.org/out-of-school-time/ost-database-bibliography/database

National clearinghouses/research organizations

Surveys (validated if possible) Participants Staff Teachers Family members

Measurement and Data Collection

Make a Plan! When will data be collected?

• When entering the program?

• When completing the program?

• Fixed interval after entering?

• Fixed interval after completing?

Who will collect the data? Who will analyze it? Where will the data be stored? A database? How will data quality be assured?

Where are you now?

1 • Anecdotal evidence• Client Testimonies• Success Stories

2 • Collecting mostly output data (number of participants)• Not using data in decision-making and improvement plans

3 • Occasionally monitors program performance• Occasionally monitors program indicators• No theory of change and/or logic model

4• Continuous improvement mindset• Regularly monitors program performance • Regularly monitors outcome indicators• Fully developed logic model and evaluation plan

Training Goal #4

Identify relevant measurement resources

Recommended Measures: Attendance

• United Way Pittsburgh Presentation - OST attendance vs. School Attendance

• HFRP - OST program attendance• RCGD - OST program attendance

Recommended Measures: Attendance

• Program Attendance– Intensity/Dosage : hrs per day, days per week– Duration: 8 months, school year, 2 years– Breadth: How many activities per student?

• School Attendance– # days not present at school (no need to

differentiate b/w excused, unexcused_– # and % of students missing <10 days

Recommended Measures: SEL

• http://www.hfrp.org/out-of-school-time/publications-resources/a-review-of-out-of-school-time-program-quasi-experimental-and-experimental-evaluation-results

• Rhodes, Jean E. "The critical ingredient: Caring youth staff relationships in after school ‐ ‐settings." New Directions for Youth Development 2004, no. 101 (2004): 145-161.

Recommended Measures: SEL

• http://character.org/more-resources/assessment-tools/individual/– # and % of students/participants who report

_______________• Attitudes towards school• Social Skills• Delinquency

Recommended Measures: Behavior

• Durlak, Joseph A., and Roger P. Weissberg. "The Impact of After-School Programs that Promote Personal and Social Skills." Collaborative for Academic, Social, and Emotional Learning (NJ1) (2007).

• Kremer, Kristen P., Brandy R. Maynard, Joshua R. Polanin, Michael G. Vaughn, and Christine M. Sarteschi. "Effects of after-school programs with at-risk youth on attendance and externalizing behaviors: a systematic review and meta-analysis." Journal of youth and adolescence 44, no. 3 (2014): 616-636.

• Gottfredson, Denise C., Stephanie A. Gerstenblith, David A. Soulé, Shannon C. Womer, and Shaoli Lu. "Do after school programs reduce delinquency?."Prevention Science 5, no. 4 (2004): 253-266.

• Cross, Amanda Brown, Denise C. Gottfredson, Denise M. Wilson, Melissa Rorie, and Nadine Connell. "The impact of after school programs on the routine activities of middle school ‐ ‐students: Results from a randomized, controlled trial*." Criminology & Public Policy 8, no. 2 (2009): 391-412.

Social Skills and Positive Relationships (includes sample surveys)• https://www.google.com/url?sa=t&rct=j&q=&

esrc=s&source=web&cd=9&ved=0CGQQFjAIahUKEwiihrzS-pnIAhXIGY4KHWnQA-0&url=http%3A%2F%2Fwww.uwex.edu%2Fces%2F4h%2Fevaluation%2Fdocuments%2FChildYouthOutcomeHandbook2005.pdf&usg=AFQjCNEqamaqae3NRJzOYu-M3nG_Ar9OUw&sig2=BjvFCNUHy5BRp5Xjc6Oxuw&bvm=bv.103388427,d.c2E&cad=rja

Recommended Measures: Behavior

• # and % of students who decreased detentions

• # and % of students who decreased suspensions

• Average decrease in suspensions for students with at least one suspension.

• # and % of students completing all HW assignments

Recommended Measures: Achievement

• Patricia A. Lauer, Motoko Akiba, Stephanie B. Wilkerson, Helen S. Apthorp, David Snow and Mya L. Martin-Glenn. Review of Educational Research. Vol. 76, No. 2 (Summer, 2006), pp. 275-313

Recommended Measures: Achievement

• # and % of students promoted to the next grade

• # and % of students improving by at least one letter grade or maintaining an A or B.

• # and % of students meeting targets on MAP assessments

Recommended Measures: Family Engagement

• http://www.hfrp.org/publications-resources/publications-series/issues-and-opportunities-in-out-of-school-time-evaluation/beyond-the-head-count-evaluating-family-involvement-in-out-of-school-time

• Durlak, Joseph A., Roger P. Weissberg, and Molly Pachan. "A meta-analysis of after-school programs that seek to promote personal and social skills in children and adolescents." American journal of community psychology 45, no. 3-4 (2010): 294-309.

• Durlak, Joseph A., and Roger P. Weissberg. "The Impact of After-School Programs that Promote Personal and Social Skills." Collaborative for Academic, Social, and Emotional Learning (NJ1) (2007).

• Hanlon, Thomas E., Betsy D. Simon, Kevin E. O'Grady, Steven B. Carswell, and Jason M. Callaman. "The effectiveness of an after-school program targeting urban African American youth." Education and urban society 42, no. 1 (2009): 96-118.

Recommended Measures: Family Engagement

• # and % of parents taking a leadership role within the program

• Parental expectations for children’s future educational attainment

• # and % of parents satisfied with program quality and content

• # and % of parents reporting increased involvement in child’s school as a result of program

Recommended Measures: PD

• http://www.hfrp.org/publications-resources/publications-series/issues-and-opportunities-in-out-of-school-time-evaluation/promoting-quality-through-professional-development-a-framework-for-evaluation

• # and % of teachers demonstrating changes in attitudes, behaviors, or knowledge covered by PD course content

Outcome Measurement Resources

Forum for Youth Investment – www.forumfyi.org Measuring Youth Program Outcomes– examines multiple assessment tools for youth programshttp://forumfyi.org/files/soft_skills_hard_data_single.pdf

A Shared Vision for Youth Common Outcomes and Indicators – http://www.readyby21.org/resources/report/shared-vision-youth-common-outcomes-and-indicators

Harvard Family Research Project – www.hfrp.orgPublications and Resources on Evaluation

http://www.hfrp.org/evaluation/publications-resources

Out-of-School Time (OST) Program Research and Evaluation Database – searchable database of OST programs and the evaluations they usehttp://www.hfrp.org/out-of-school-time/ost-database-bibliography/database

Logic Models for OST Programs – guide to developing a logic model http://www.hfrp.org/out-of-school-time/publications-resources/learning-from-logic-models-in-out-of-school-time

The Colorado Trust – www.coloradotrust.org Toolkit for Evaluating Positive Youth Development

http://www.coloradotrust.org/sites/default/files/ASIToolkitJun04.pdf

Outcome Measurement Resources

University of Wisconsin - Extension – http://www.uwex.edu/ces/pdande/evaluation/evaldocs.html Creating a Logic Model Online Course - http://www.uwex.edu/ces/lmcourse/#

Child Trends – www.childtrends.orgPositive Indicators Project - http://www.childtrends.org/our-research/positive-indicators/positive-indicators-project/

California Evidence-Based Clearinghouse - http://www.cebc4cw.org/assessment-tools/ Links to measurement tools

Outcome Measurement Resources

PerformWell – (Urban Institute, Child Trends, and Social Solutions) www.performwell.org Database of measurement tools and Outcomes

Child Welfare Information Gateway (USDHHS)www.ChildWelfare.gov Links to measurement toolshttps://www.childwelfare.gov/topics/systemwide/youth/outcomes/tools/

Toolfind – (United Way of Massachusetts Bay and Merrimack Valley) www.Toolfind.org Database of 36 measurement tools for youth programs