Impact Measurement why what how 9-14-09 Atlanta. Today Imperatives Questions Why Now? Significant...
-
Upload
may-carpenter -
Category
Documents
-
view
219 -
download
0
Transcript of Impact Measurement why what how 9-14-09 Atlanta. Today Imperatives Questions Why Now? Significant...
Impact Measurementwhywhathow
9-14-09
Atlanta
Today ImperativesQuestionsWhy Now?Significant ChallengesBreakthroughs in the fieldCARE’s Long-Term StrategyThis YearHow do we meet the challenges?
Imperatives
…can only flow from a clearly-defined program approach that is predicated on a clear framing of the problem.
…systems must actually add value for the organization and produce reliable data that it can be utilized to inform strategic decisions.
Measurement …
Questions 1. If you do not measure our impact and demonstrate the
change, then does it matter?
2. Can we measure the most important changes or do we emphasis what we can measure?
3. How do you actually measure social impact and do those type of changes inherently defy measurement - an increase in someone’s level of opportunity or their dignity
4. Does spending on measurement and building measurement systems mean less time and resources for the actual work that we do?
Most Significant Challenges
What does ‘impact’ even mean? How do we even begin to define metrics that can cut across a plethora of disparate issues that CARE is addressing? Do we need counter-factual, experimental designs and benchmarks within the same issue areas to do so?
How do you measure outcomes rather than inputs? Especially when the intervention is removed from the ultimate desired change - i.e., girl’s education programs with the goal of increasing women’s empowerment.
Most Significant Challenges (2)
How do we measure impact for the wide variety of constituents that CARE is accountable to?
How do we build the capacity across the organization to make sure that the measurement is consistent through a well implemented system.
Breakthroughs in the Field
Shared Measurement PlatformsAllow organizations to choose from a set of measures
within their fields
Comparative Performance SystemsRequire all participants within a field to report on the same
measures
Adaptive Learning SystemsAllow people working on different aspects of a single
complex issue to engage in an ongoing, facilitated process
Why Now ?
The program approach across CARE
Critical to demonstrate intended benefits
for organizational performance management
Support fundraising
Provide evidence for advocacy for LIFT-UP
Proactively meets future compliance
Levels of Measurement The proposed system covers all levels of measurement and all stages of the program cycle:
Outputs
Effects
Impact
Activities Inputs
Levels of Measurement The proposed system covers all levels of measurement and all stages of the program cycle:
Outputs
Effects
Impact
Activities Inputs
Levels of Measurement
LEVEL 1
Input:• Resources needed to undertake a set of activities
Activity: • The actions that convert inputs into outputs
Output:• The changes that are a direct result of activities
Levels of MeasurementLEVEL 2 Effect: Changes in human behaviors and practices Improvement in services through increased competency (e.g. more effective extension systems) Changes in systems and policies (enabling environment)
LEVEL 3 Impact: Long-term sustainable change Contribution to the MDGs
Quality Assessment
Proposal Design
Proposal Analysis
Analysis of proposals across
CARE
Des
ign
Ubora Data Collection & Quality Assessment
Eva
luat
ion
Impl
emen
tatio
n
Results(Ongoing
Activities and Outputs)
Quality Assessment
Mid-implementation
Strategic Cross-Cutting AnalysisAnalysis Process
Results(Final Activities &
Outputs, plus Effects and Impacts)
Program Causal
AnalysisMore in-depth analysis of the
impact of specific
programs (e.g.
Signature Programs)Meta-Evaluation
Analysis of results across
CARE
Sectoral/Thematic ReviewsMore in-
depth, cross-cutting
analysis of the impact of
specific development areas (e.g.
water)
Strategic Impact I nquiry
More in-depth analysis of
CARE’s impact on specific
cross-cutting themes (e.g.
women’s empowerment,
governance)
Ubora Performance ReportMid-year and annual reports analyzing CARE’s performance to inform action for improvement
Reports
Women’s Empowerment ReportAnnual report showcasing CARE’s approaches, learning, stories
and impact, drawing on data and analysis relevant to the year’s theme
Quality AssessmentOverall quality
External Internal
Components
Component 1Program Data Collection SystemSectoral Portfolio Review
Component 2Annual Impact ReportStrategic Impact Inquiry
Component 3Meta-EvaluationProposal AnalysisProgram Causal Analysis
Components of IM System
Inputs Activities Outputs
Impact
Effects
Access AfricaHow are we measuring our impact
Increased general well being of participants
household
Increased in economic, social and political
empowerment participants
Monitoring & VSLA quality
Activity/Output Level R2. Retention rate R5. Annualized return on savings R8. Portfolio at riskR16. Cost per member assisted
Outcome Level% of VS&L members who have invested in productive assets
% women holding leadership position at community level Level of implementation of policies for women right on land and other productive assets
Impact Level
% population
below $2 per
day (MDG)
Impact Measurement System
Activity & Output level: MIS System
Excel based system to Manage Field Officer performance & VSLA Quality
Measures 4 categories of performance ratios (SEEP):
o Members Satisfactiono VSLA Financial Performanceo VSLA operating efficiencyo CARE operating efficiency
Quarterly data collection by projects in the field
Access Africa aggregates global data quarterly
Data enables AA to engage in Program Quality discussions with the COs
Common system used by OXFAM, CRS, PLAN, Aga Khan, FfH, etc.
Common indicators/tool across COs
Option of methodology: Client Survey: data collection on participants household Household Survey: data collection on community
members (participants and non-participants) RCT, with IPA in Uganda and Malawi Mixing quantitative and qualitative methods
Capacity building of COs – TDY, direct support from AA
Support COs in hiring relevant staff and/or consultants for M&E
What about Outcome and Impact?
Making the information Available
The DirectoryName
Program #GoalManager Contact InformationAssociated Project
Program Informatio
n
NamePN/FC #ManagerContact Information
NamePN/FC #ManagerContact Information
NamePN/FC #ManagerContact InformationName
PN/FC #ManagerContact Information
Project InformationSearchable information in both categories:
•Start Dates and End Dates of Projects and ProgramsImpact PopulationTarget GroupSignature Program componentMDG Impact Area program or project works inNumber of direct participants in the program Gender and AgeActivities and OutputsCARE partners Advocacy Goals, Partners and Targets of Advocacy Efforts
This year
Work with Country Offices to create and disseminate appropriate guidance on impact measurement
Work with Technical Units in Atlanta to help put impact measurement systems in place in select Learning Labs.
Focus on organizing consultations on and outlining a business process for impact measurement at CARE, complete with a resource development plan
Working with the Policy Advocacy Unit and Communications on CARE’s Annual Impact Report.
Finished
Finally