Outcomes Working Group Webinar 3: Indicators to measure outcomes Presenters: Bobbi Gray, Research...
-
Upload
darcy-chase -
Category
Documents
-
view
218 -
download
0
description
Transcript of Outcomes Working Group Webinar 3: Indicators to measure outcomes Presenters: Bobbi Gray, Research...
Outcomes Working GroupWebinar 3: Indicators to measure outcomes
Presenters: Bobbi Gray, Research and Evaluation Specialist, Freedom from Hunger
Anne Hastings, Executive Director,Microfinance CEO Working Group
Working Group Facilitator: Frances Sinha, Director EDA Rural Systems (India) and board member of SPTF
4 February 2015
Agenda • Introduction (5mins)
• Bobbi (FFH): criteria to develop indicators related to health outcomes; application by 4 MFIs, lessons (15-20mins)
• Anne (MCWG): developing a system for standardized measurement, review of outcomes studies (69), selecting themes and indicators for microfinance (15-20mins)
DISCUSSION
Recap
• Previous webinar : Theory of Change framework: ▫ Defining short-term/intermediary and long term changes▫ Assumptions – links from different services/uses to changes ▫ Attrition factor - not all participants will stay the course
• Next logical step to identify relevant practical indicators to measure changes - and the steps that lead to changes:• Criteria for identifying practical, relevant indicators• Selection relevant to microfinance • What is the experience? • Can we build consensus on a ‘standard menu’ ?
Indicators The basics• Specific, relevant - define objectives• Practical to measure - credible• Unambiguous - clear• Can compare, benchmark • Limited number – necessarily selectiveTrade-offs in e.g. what is practical to measure
(household income or expenditure) and/or unambiguous (women’s role in decision making)
Health Outcome Performance Indicators Project
6
Theories of Change: Improved Health
Access to and use of financial services: loans;
insurance; savings;
payments; health loans and savings
Increased income
Consumption smoothing
Coping with shocks
Seek prompt medical
treatmentSeek
preventive health care Access to and
use of health services: education,
provision of curative and
preventive health services
Improved health knowledge and seeking prompt medical treatment
and preventive health care
Integrated health and financial
services: direct provision and
linkages between sectors
Cross-sectoral efficiency gains in
provision of financial and
health services to poor populations
Seek prompt medical
treatmentSeek
preventive health care
Improved health
outcomes
7
Choosing Health IndicatorsCriteria
Feasibility Usability Usability/ Reliability
Likelihood of inclusion
Measurable by an FSP
Can be reported in client survey
Can change in short-term
Addresses relevant measures for FSPs
Cannot rely on specific interventions to change outcomes
Be applicable for both genders
Can be benchmarked to other data (MDGs, regional data, etc.)
Reliability
PPI/PAT Yes Yes Maybe Yes Yes Yes Yes Yes High
Food security index
Yes Yes Yes Yes Yes Yes Yes Maybe High
Use of preventive health services
Yes Yes Yes Maybe Maybe Yes Maybe Maybe High/ Moderate
Access to safe drinking water (MDG 7)
Yes Yes Maybe Yes Maybe Yes Yes Maybe High/ Moderate
8
Current Pilot PartnersFinancial
Service ProviderCountry No of Clients
being served by FSP
No. of clients participating in
health indicators
surveyADRA Peru 17,039 95
CARD Philippines 1,828,052 472
ESAF India 322,590 1,000
Equitas India 1,344,361 250
9
Survey Adaptations Poverty measurement – Use of country-specific PPIs Food security and nutrition –added a focus on food items in
India to reflect stronger focus on nutrition Preventive health care – focuses on institutional births in India;
annual exams and Pap tests in Peru; various annual exams in the Philippines and use of health insurance (PhilHealth)
Curative Health Care: Questions same in all three countries – forgoing medical treatment and purchase of medicines due to cost
Water and Sanitation– Focuses on defecating in the open and treating water to make it safer to drink in India; Open defecation, water sources, and water treatment in Peru; Water sources and treatment of water in the Philippines
Attitudes: Only measured in Peru and Philippines, accesses levels of confidence related to ability to cover future medical costs and seek adequate medical care
10
Standardization of indicators may be difficult. Proceed with caution in the interpretation of results. Baseline values will be important to establish. Baseline values with high levels of performance may not be
useful to track. The value of statistical analysis – ex. correlations between
indicators of interest with poverty – can help refine our “theory of change” as well as determine which indicators may be the most useful to help us understand changes in client outcomes.
Who to track and for how long? This is a very important question to answer as it influences which indicators will be the most useful.
This process requires patience.
Lessons Learned
Collaborating on Outcomes
AccionBRAC & BRAC InternationalCARE Access AfricaFINCA InternationalFreedom from Hunger
Grameen FoundationOpportunity InternationalPro MujerVisionFund InternationalWomen’s World Banking
The Goal of Our Work on OutcomesTo develop, test and disseminate a common core measurement and monitoring system designed to:
Be cost-effectively embedded within an MFI’s operations Use industry-standard indicators and metrics (PPI) Provide affiliated MFIs with actionable client outcome data
that can be tracked over time Allow MFIs the flexibility to measure those changes in
client outcomes they wish to affect using the same indicators for the same outcomes
Enhance the Working Group’s ability to demonstrate and improve the benefits of Responsible Microfinance for the clients being served
Contribute to the sector’s understanding of client outcomes
The Proposed Method1. Select the indicators to test 2. Target countries and the MFIs3. Pilot the indicators
Social performance (SP) leads will work with their partners to collect, analyze and report on the data
SP leads and MFI partners would convene to discuss experiences, identify lessons learned and develop recommendations for future use
SP leads will develop and publish a series of briefs by outcome area with theory of change and recom-mended indicators
Package and disseminate the final indicators along with supporting documentation
Lessons from the Working Group’s Review of Outcomes•Phase 1: Catalogued 69 different
outcomes research from all 8 members and their affiliates▫Wide variation in product studied,
methodology, indicators, quality of research, etc.
▫Difficult to draw definitive conclusions when comparing outcomes across programmes
Different methods
Baseline
Midline
Endline
Other data
FGD
Bank records
KII
Lit Review
Control group
Qual. survey
Quant. survey
0 5 10 15 20 25 30 35 40 45 5024
425
1151616
2730
1544
RESEARCH COM-PONENTS
Different themes
Child growthSocial capital
NutritionShocks / coping mechs
Personal view of ec'c standingFarming (business)
Food securitySavings
EmpowermentPoverty status
Poverty outreachFin literacy
Experience with MF programsBusiness (non-farming)
HealthWell-being
IncomeEducation
Assets
0 5 10 15 20 25 30 353
1010
1410
1525
2317
144
716
2930
1128
316
INDICATORS
A Closer Look at Education Studies1. There was a big range of indicators
being used across and within the networks.
2. There was little consistency across institutions.
3. Most studies only looked at one dimension of education.
Phase 2: Identifying Potential Indicators for 7 Types of Client Outcomes1. Food security2. Coping strategies/shocks3. Economic poverty (income or financial
status)4. Health5. Assets, housing and business6. Social capital and empowerment7. Child and youth education
Phase 2: Summary of IndicatorsFood security
% of households that are food insecure (levels measured vary by tool)Coping strategies/shocks% of households that had to [make a specific adjustment or level of change] to cope with the effects of [household or community shock]Economic poverty (income or financial status)% of households living above/below a given poverty lineHealth% of households with [level of access] to health care [services/supplies]
Phase 2: Summary of IndicatorsAssets, housing and business% of households that were able to [purchase assets/improve their homes] in the past yearSocial capital and empowerment% of [women] expressing confidence in making decisions regarding [specific category]% of clients who perceive [specify benefit or value] in group participationChild and youth education% of children in the household who are regularly attending school% of households with increased ability to pay for school fees
Discussion
1. Any questions/clarifications? 2. What is your experience in indicator selection?3. Can we aim at standardisation - synchronisation?
4. Our working group: contributing to guidelines for selection (and a menu) of indicators to measure client outcomes?
Thank you•For follow up, please contact:
[email protected], [email protected]
•Please note: presentations and recordings from all Outcomes Working Group Meetings are being posted to the SPTF website, working groups page: http://sptf.info/sp-task-force/working-groups