Learning Across Borders; A Monitoring, Evaluation, and Learning Framework for The MasterCard...

23
Learning Across Borders: A Monitoring, Evaluation, and Learning Framework for The MasterCard Foundation Scholars Program March 2014

description

The monitoring, evaluation and learning design of the MasterCard Foundation Scholars Program. Theory of Change, logic model and learning questions. Design challenges and solutions.

Transcript of Learning Across Borders; A Monitoring, Evaluation, and Learning Framework for The MasterCard...

Page 1: Learning Across Borders; A Monitoring, Evaluation, and Learning Framework for The MasterCard Foundation Scholars Program.

Learning Across Borders:

A Monitoring, Evaluation, and Learning Framework for The MasterCard Foundation Scholars Program

March 2014

Page 2: Learning Across Borders; A Monitoring, Evaluation, and Learning Framework for The MasterCard Foundation Scholars Program.

2

Overview of Presentation

The MasterCard Foundation The Scholars Program Monitoring, evaluation, and learning (MEL) framework MEL challenges, approaches, and lessons learned

Page 3: Learning Across Borders; A Monitoring, Evaluation, and Learning Framework for The MasterCard Foundation Scholars Program.

The MasterCard Foundation Scholars Program

3

Page 4: Learning Across Borders; A Monitoring, Evaluation, and Learning Framework for The MasterCard Foundation Scholars Program.

4

The MasterCard Foundation Scholars Program: Context and Programmatic Need

Developing countries have achieved a remarkable expansion in educational enrolment, however:

Enrolment in secondary and tertiary education remains low, particularly among poor and rural populations

There are concerns about access to quality and relevant education, even as demand for an educated workforce increases

Many skilled graduates migrate to higher-income countries for better remuneration and working conditions

International scholarship programs too often contribute to brain drain

Page 5: Learning Across Borders; A Monitoring, Evaluation, and Learning Framework for The MasterCard Foundation Scholars Program.

The MasterCard Foundation Scholars Program

$500 million, 10-year initiative to educate and support 15,000 young people – particularly from Africa – to act as “socially transformative leaders,” driving change and making a positive social impact in their communities.

11,000 secondary level Scholarships, 6,000 tertiary

Target population: youth who have demonstrated academic talent, social consciousness and an orientation toward leadership, and who come from highly disadvantaged socioeconomic backgrounds.

Implemented with and through a global network of 20 NGOs and secondary and tertiary education institutions, who select and directly support the Scholars.

5

Page 6: Learning Across Borders; A Monitoring, Evaluation, and Learning Framework for The MasterCard Foundation Scholars Program.

6

The MasterCard Foundation Scholars Program: Program Components

Comprehensive scholarships

Transition support

21st-century skill building

Give-back component

Alumni network

Page 7: Learning Across Borders; A Monitoring, Evaluation, and Learning Framework for The MasterCard Foundation Scholars Program.

The MasterCard Foundation Scholars Program: Partners

7

Page 8: Learning Across Borders; A Monitoring, Evaluation, and Learning Framework for The MasterCard Foundation Scholars Program.

The MasterCard Foundation Scholars Program MEL Framework

8

Page 9: Learning Across Borders; A Monitoring, Evaluation, and Learning Framework for The MasterCard Foundation Scholars Program.

Evaluation and Learning in the Scholars Program

Objectives:

Continuous program quality improvement

Evidence of outcomes, impact and best practices…

For our partners

For the Foundation’s strategic planning

For the global community of practitioners

As the basis of a proactive argument for propagating successful Program elements and the overall vision of the Program.

9

Page 10: Learning Across Borders; A Monitoring, Evaluation, and Learning Framework for The MasterCard Foundation Scholars Program.

Evaluation and Learning Overview

Program-wide evaluation: primary end-point 2016 Implementation study

Impact study

Institutional change study

Longitudinal studies of outcomes and impact Outcome study (subset of Scholar cohorts and comparison groups): continued past 5-year point

Outcome tracking (all cohorts)

Anticipated outcomes (Stay in / return to Africa, Give-back, career and leadership outcomes, citizenship outcomes)

Long-term qualitative work to understand Scholars’ roles in promoting social and economic development

Partner-initiated Research Rapid studies on emergent issues, aimed at providing actionable intelligence.

10

Page 11: Learning Across Borders; A Monitoring, Evaluation, and Learning Framework for The MasterCard Foundation Scholars Program.

11

The MEL Framework: The Process

Page 12: Learning Across Borders; A Monitoring, Evaluation, and Learning Framework for The MasterCard Foundation Scholars Program.

The MEL Framework: Theory of Change

12

Page 13: Learning Across Borders; A Monitoring, Evaluation, and Learning Framework for The MasterCard Foundation Scholars Program.

Approachand Inputs

Activities Outputs ImpactsOutcomes

Inputs

Partner expertise

To Serve Scholars

Academic and other support

Approach

Provide Scholars with a wide range of opportunities

for educational support and enrichment,

give-back, and leadership

development

At Institutions

Revise recruitment procedures

Scholars using academic and other support

services

Partners disseminate

lessons learned

Increased academic

performance

Institutionalized changes foster best practices

New cohort of leaders

supports social

transformation and economic

growth, particularly in

Africa

13

The MEL Framework: Logic Model Components

Page 14: Learning Across Borders; A Monitoring, Evaluation, and Learning Framework for The MasterCard Foundation Scholars Program.

The MEL Framework: Learning Questions, Methods, and Data Sources

Broad Learning Questions Main Design/Methods Data Sources

Implementation

How was the Scholars Program implemented? Mixed methods

Qualitative data, complemented by quantitative data

Impacts

In particular, what is the impact of the Program on educational, employment, go-back, and give-back outcomes?

Comparison group design• Random assignment• Regression discontinuity• Propensity score matching

Survey data, complemented by qualitative data

Scale-up

Did the Scholars Program become a catalyst for broader change within and beyond partner institutions?

Qualitative analysisQualitative data, complemented by quantitative data

14

Page 15: Learning Across Borders; A Monitoring, Evaluation, and Learning Framework for The MasterCard Foundation Scholars Program.

Challenges, Solutions, and Lessons Learned

15

Page 16: Learning Across Borders; A Monitoring, Evaluation, and Learning Framework for The MasterCard Foundation Scholars Program.

16

Challenges, Solutions, and Lessons Learned: Designing a Unified MEL Framework

Challenge Develop a single unified MEL framework to evaluate The MasterCard

Foundation Scholars Program

Solutions Develop the MEL framework from the ground up through a thorough

understanding of partner projects and what they have in common Use a flexible approach that allows for different methods

for each partner: Partner-specific comparison group designs allow for a rigorous evaluation while using different methods to draw the comparison group

Random assignment Regression discontinuity Matched comparison group (via propensity score methods)

Page 17: Learning Across Borders; A Monitoring, Evaluation, and Learning Framework for The MasterCard Foundation Scholars Program.

Challenges, Solutions, and Lessons Learned: Measurement

Challenge Locating applicants based on application data may be challenging Several key outcomes are difficult to measure Key outcomes, including educational attainment, career and service

outcomes, giving back and going back, occur several years in the future

Solutions To locate hard-to-find applicants, use their own and their schools’

contact information Draw survey items from existing instruments and pilot our

own instruments Establish a plan for tracking The MasterCard Foundation Scholars over

time that is appropriate to the context

17

Page 18: Learning Across Borders; A Monitoring, Evaluation, and Learning Framework for The MasterCard Foundation Scholars Program.

Challenges, Solutions, and Lessons Learned: Measurement—Baseline Data Collection in Uganda

Sample: 823 applicants spread around Uganda

Applicants were located by calling the phone number provided in their application

This is the first of three rounds of data collection planned

18

92%

8%

Locating applicants

Located

Not located

82%

18%

“Give-back” attitudes:Helping people in need is

something one should do…

for anyone

for friends and family

2% 15%

83%

Contact informationNo phone number1 phone number2 phone numbers

Page 19: Learning Across Borders; A Monitoring, Evaluation, and Learning Framework for The MasterCard Foundation Scholars Program.

Challenge Incorporate formative evaluation components that will generate

relevant feedback for partners throughout the evaluation Early feedback can then inform mid-course corrections as needed

Solutions Engage with partners while developing MEL activities Collect quantitative and qualitative data at regular intervals Provide partners with feedback from all data collection efforts

19

Challenges, Solutions, and Lessons Learned: Blending Formative, Summative Evaluation Approaches

Page 20: Learning Across Borders; A Monitoring, Evaluation, and Learning Framework for The MasterCard Foundation Scholars Program.

Phone interviews with 18 graduating tertiary Scholars Opportunity to gain early information on Scholar experiences This early outcome information will support continuous quality

improvement

Scholars offer suggestions for Program improvement Increase flexibility in resource allocation—for example, could leftover

resources for textbooks be used for transportation? Coordinate Scholar attendance at academic conferences

to facilitate networking Provide orientation to the Program in one intensive session early on,

rather than spread over several sessions

20

Challenges, Solutions, and Lessons Learned: Regular, Relevant Feedback to Partners

Page 21: Learning Across Borders; A Monitoring, Evaluation, and Learning Framework for The MasterCard Foundation Scholars Program.

Challenges, Solutions, and Lessons Learned: Regular, Relevant Feedback to Partners (continued)

Scholars also say what is going well The Program gives Scholars “the flexibility, time

and resources to actually do the work you set out to do”

“[I’m] trying my best to really use all the knowledge they’ve given to me and trying to implement it in myself first and then try to introduce it to society”

The Masters in Development program is a “dream program” that fits well with career goal of reducing poverty in her home country

21

Page 22: Learning Across Borders; A Monitoring, Evaluation, and Learning Framework for The MasterCard Foundation Scholars Program.

Conclusions

We have designed MEL activities that respond to the complex nature of The MasterCard Foundation Scholars Program with an approach that is

• Flexible, yet rigorous

• Capable of measuring key outcomes now and in the future

• Formative and summative in nature

• Collaborative and inclusive, responding to needs across the partnership

22

Page 23: Learning Across Borders; A Monitoring, Evaluation, and Learning Framework for The MasterCard Foundation Scholars Program.

Contacts

Barry Burciul [email protected] http://mastercardfdnscholars.org/

Matt Sloan [email protected] http://www.mathematica-mpr.com/CIPRE/Projects/

MasterCard_Scholars_program.asp

23