Collaborative Mentoring Webinar Series

Did it Work? Evaluating Youth Outcomes in Relationship-Based Programs

December 19th, 2013

Sarah Kremer,

Friends for Youth

Michael Garringer,

Education Northwest

Dana Gold, MP of

SWPA

Polly Roach, MP of

Minnesota

Tammy Tai,

MENTOR

Molly Brenner,

MENTOR

Meghan Ferns,

Oregon Mentors

2013 Collaborative Mentoring Webinar Series Planning Team

Good to Know…

After the webinar, all attendees receive:

Instructions for how to access PDF of presentation slides and webinar recording

Link to the Collaborative Mentoring Webinar Series webpage, where all slides, recordings, and resources are posted.

Please help us out by answering survey questions at the end of the webinar.

• All attendees muted for best sound

• Type questions and comments in the question box

• Respond to polls

• Who is with us today?

Participate in Today’s Webinar

Panelists on Today’s Webinar

Michael Garringer Resource Advisor The National

Mentoring Center at

Education Northwest

Celeste Janssen

Executive Director

Oregon Mentors

Tom Drummond

Co-Director

College Dreams

Challenges & Opportunities with Evaluating Youth Outcomes

Program Effectiveness: The Pathway to Improved Youth Outcomes

How well you run

your program

How well your

volunteers mentor

the youth

Quality of match

relation-ships

Mentee’s attitudes on certain

topics

Youth behavior

(outcomes you wish

to achieve)

Jean Rhodes’ Model of Youth Mentoring

What Makes Evaluating Relationship-Based Programs Unique?

• Multi-modal benefits create lots of opportunities for measurement and challenges in knowing your impacts

• The relationship can be tricky to measure

• The activities mentoring programs provide and the outcomes they measure aren’t necessarily aligned with policymaker goals

Three Concepts That Can Help Answer “Did It Work?”

• Theory of change

• Deeper understanding of relationships

• A focus on intermediate outcomes and measures

Theory of Change

• Similar to a logic model, but more detailed and robust – Clarifies causality toward end goal

– Challenges assumptions about the intervention

• Works backwards from the ultimate outcome – Each outcome step is connected to an

intervention component or assumption

– Illustrates the causal pathway to long-term outcomes

TOC Example

Source: Aspen Institute

TOC Example

Source: Aspen Institute

TOC Example

Source:

Aspen Institute

Example of TOC in Mentoring

• Long-term goal: HS Readiness

↓ What competencies and attitudes are needed?

↓ Activities mentors-mentees do together? Parent role?

↓ How you train your mentors to do this? Other supports?

↓ Recruiting the right mentors

↓ Bringing the right youth into the program

Resources for TOC

• Center for Theory of Change

• The Community Builder’s Approach to Theory of Change: A Practical Guide to Theory Development (Aspen Institute)

• On Logic Models Theories of Change and Evaluation

• An Introduction to Theory of Change (Harvard Family Research Project)

• Kellogg Foundation Logic Model Development Guide – Chapter 3

Understanding Mentoring Relationships

• More information about matches = Better understanding of outcomes (good and bad)

• Go beyond measures of “satisfaction” – get inside the “box” of the mentoring relationship

• Participant characteristics are a logical starting point: – Risk factors and assets

– Other supports

– Personality traits

– Personal history and past relationships

Commonly Collected Match Data

• Meeting frequency and intensity – 60% of matches meet 4 hrs. per month for one year

• Match activities (both the “what” and associated feedback) – 52% of matches engaged in 4 or more hours of service of

learning

– 60% of youth self-report an overall positive perception of the mentoring relationship

• Match duration – Average match length is 14 months

Additional Data

• Parent engagement and perceptions

• Input of teachers, other adults

• Support seeking by participants – Topics

– Frequency

– Level of staff support

– Resolution

Participant Perceptions

• Field has moved away from looking at mentors’ approaches through a “prescriptive” or “developmental” lens.

• Now emphasizing a “relational” or “instrumental” spectrum

– Pairs start at one end of this based on program model

A Framework for Understanding Interactions

• Focus – Goal-directed or relational

• Purpose – Youth (immediate, playful) or Adult (distal, progress)

• Authorship – Unilateral, reciprocal, or collaborative

• These three variables determine relationship quality and satisfaction, predictive of participant outcomes – Balance of relational and instrumental elements creates

satisfaction – Authorship (youth “voice”) is essential for receptiveness

Further Exploration of Relationships

• Internal Match Quality (relational and instrumental) – Compatibility (traits and preferences) – Competence (mentor’s approach and skills) – Satisfaction (closeness, connectedness, perceived support)

• Match structure – Purpose/goal orientation (goal setting) – Focus of activities – Authorship (one-sided or negotiated)

• External Match Quality – Program support – Parent/guardian engagement – Support network – Logistical challenges

Tools for Measuring Relationships

Youth Mentoring Survey (youth perceptions) – Harris and Nakkula

• Accurately measures match structure and quality

• Likert & short answer (grades 3-12)

• Subscales: – Fun focus

– Sharing focus

– Instrumental focus

Tools for Measuring Relationships

Match Characteristics Questionnaire (mentor perceptions) Harris and Nakkula • Also measures structure and quality • Likert and short answer • Subscales:

– Closeness – Satisfaction – Openness to Support – Academic Support Seeking – Fun, Sharing, Future Outlook, Character Development,

Education

Tools for Measuring Relationships

• YMS and MCQ can be accessed at: http://www.mentoringevaluation.com/Tools.htm

• Learn more:

– Play, Talk, Learn: Promising Practices in Youth Mentoring (New Directions for Youth Development, Summer 2010)

Tools for Measuring Relationships

Youth Survey (variation on the Youth-Mentor Relationship Questionnaire developed by P/PV)

• 19 items (youth only)

• Measures: – Youth centeredness

– Youth’s emotional engagement

– Youth dissatisfaction

• Download through the OM Toolkit or on the EdNW website

Tools for Measuring Relationships

Performance Measurement Packet: Mentoring Children of Incarcerated Parents • Two age groups: 9-12 and 13-18 • A survey for mentors • A survey for a second adult who knows the

mentee • Intermediate Outcome: Trusting mentor-mentee

relationship • End Outcome: Positive attitude toward the future

Additional Reading on Relationships

Handbook of Youth Mentoring (2nd Edition)

– Assessing Mentoring Relationships

– Mentoring Activities and Interactions

– Goals in Mentoring Relationships

– Social Networks and Mentoring

– And more…

Intermediate Outcomes

• You must make the case that your intermediate outcomes will lead to long-term changes (“needle moving”)

• Two excellent examples:

– Putting Middle Grades Students on the Graduation Path: A Policy and Practice Brief

– Out of School Time Issue Brief (United Way)

Intermediate Outcomes that Fit Mentoring

• Connectedness

• Self esteem/competence/motivation

• Future orientation and goal setting

• Peer relationships

• Adult relationships

– Support seeking

• Identity development

Where to Find Instruments

• From Soft Skills to Hard Data (Forum for Youth Investment)

• Outcome Indicators Project (Urban Institute)

• Beyond Content: Incorporating Social and Emotional Learning into the StriveTogether Framework (Strive Network) – Competencies related to academic achievement – Measures by stage on the Cradle-to-Career Continuum

• Oregon Mentors Evaluation Instrument Toolkit!

Tying It All Together

You can answer “Did it work?” by: • Showing how your program’s work feeds into

larger outcomes

• Connecting all those participant and match characteristics to outcomes – Who did the program work for (or not)? – What correlations help you improve the model? – Where can we claim some causality?

When you will need a professional’s help…

• When you want to run statistical correlations and understand relationships between many factors and outcomes.

• When you need to develop a new data collection instrument.

• When you want to establish causality in your results (experimental design, RCT or comparison group) – They can also help develop that Theory of Change!

Example from the Field: College Dreams

Tom Drummond, MSPsy

Co-Director College Dreams

www.collegedreams.org

Guiding Lights Theory of Change

Long-term goal: X ↓ What competencies and

attitudes are needed? ↓ Activities mentors-

mentees do together? Parent role?

↓ How you train your mentors to do this? Other supports?

↓ Recruiting the right mentors

↓ Bringing the right youth into the program:

Long-term goal: College Access ↓ High School Completion & Academic

Mindsets ↓ Students begin with an Academic

Motivational Interview, and meet weekly with their mentor at the YMCA or Boys & Girls Club site

↓ Academic motivational enhancement and skill training for mentors

↓ Recruit & Match volunteers with a college degree

↓ Enroll MS students with: • Academic potential (achievement test

scores of 25th to 99th percentile, no modified classes)

• At least three dropout risk factors • A Grade Point Average (GPA) between

0.00 and 2.49

Youth Outcomes in College Dreams’ Guiding Lights Mentoring Program

Grade Point Average Guiding Lights Comparison

Grade Point Average (Baseline & Current) 1.63 to 2.05 1.70 to 1.76

GPA Improvement 0.42 0.06

Percent of Students with 0.10+ GPA Gain 68% 48%

School Attendance Guiding Lights Comparison

Absence Rate (Baseline & Current) 9.2% to 6.1% 9.1% to 15.0%

Change in Absence Rate 33.6% Decrease 64.8% Increase

Improved or Perfect Attendance 72.2% 49.0%

Perfect Attendance 19.4% 10.3%

Undesirable Outcomes Guiding Lights Comparison

Any Undesirable Outcome 2.6% 14.1%

Not Attending School 0.0% 3.3%

Alternative/Remedial Programs 2.6% 10.1%

Evaluation Toolkit Domains of Interest Live Demo of Toolkit & Instruments

Evaluation Toolkit Domains of Interest Live Demo of Toolkit & Instruments

Comprehensive Surveys:

• Youth Outcome Survey

• Youth Risk Behavior Surveillance System

Connectedness:

• Hemingway Scale of Adolescent Connectedness

School and Academics:

• Education Expectations and Aspirations Scale

• Motivated Strategies for Learning Questionnaire (MSLQ)

Mentoring relationship quality:

• Match Characteristics Questionnaire (MCQ)

• Youth Mentoring Survey (YMS)

• Youth Survey: Measuring the Quality of Mentor-Youth Relationships

Personal Attitudes and Beliefs:

• Antisocial attitude scale

• Decision making skills scale

• Goal Setting Skills Scale

• Pro-social Involvement, Opportunities and Rewards

Tips for Administering Instruments

1. Plan for problems

2. Don’t hand-pick which questions to use

3. Don’t reword questions

4. Get parent permission

5. Provide a private space to administer

6. Don’t provide guidance

7. Administer the instruments in a consistent, uniform way

8. Keep the answers anonymous/ sort out any coding before administration

9. Plan for data storage and safety

Additional Resources

• Oregon Mentors Evaluation Toolkit

http://oregonmentors.org/library/evaluationtools/

• W.K. Kellogg Foundation Evaluation Handbook http://www.wkkf.org/knowledge-center/resources/2010/W-K-Kellogg-Foundation-Evaluation-Handbook.aspx

• W.K. Kellogg Foundation Logic Model Development Guide http://www.wkkf.org/knowledge-center/resources/2006/02/WK-Kellogg-Foundation-Logic-Model-Development-Guide.aspx

• Northwest Health Foundation Program Evaluation: Principles & Practices

http://nwhf.org/images/files/NWHF_Program_Eval_Handbook.pdf

Where do your Services fit within a Youth Development Continuum?

Remember…

After the webinar:

Please help us out by answering survey questions at the end of the webinar.

Everyone will get an email with information on how to

download the slides, recording, and resources on the CMWS webpage on the MENTOR website:

http://www.mentoring.org/program_resources/training_opportunities/collaborative_mentoring_webinar_series/

Access CMWS Information All Collaborative Mentoring Webinar Series information for upcoming and past

webinars is on the new CMWS webpage on the MENTOR website!

Receive Monthly CMWS News

Sign up for the newsletter!

• Information about

upcoming webinars

• Evaluation results from the

previous webinar

• Links to resources and

recordings from past

webinars

Topic:

Briefing: New Mentoring Report from MENTOR – Insights and Feedback

Registration will open next week on the MENTOR website.

Sarah Kremer,

Friends for Youth Michael Garringer,

Education Northwest

Meghan Ferns,

Oregon Mentors

Dana Gold, MP of

SWPA

Polly Roach, MP of

Minnesota

Tammy Tai,

MENTOR Molly Brenner,

MENTOR

Next Webinar: January 17, 2014 1:00-2:15PM EST/

10:00-11:15AM PST

2013 Collaborative Mentoring Webinar Series Thank you for participating today!