State of Sustainability Planning & Evaluation in Community Health
-
Upload
amber-lopez -
Category
Documents
-
view
24 -
download
1
description
Transcript of State of Sustainability Planning & Evaluation in Community Health
State of Sustainability Planning & Evaluation in Community Health
Sustainability Framework (SF) 101
April 21, 2011IntraHealth with the SHOUT Group
ACKNOWLEDGEMENTS
IntraHealthRenée Kotz, Sara Margolis-Pacqué, Frances Thornhill, Kris Horvath
ICF MacroJennifer Yourkavitch, Ilona Varallyay, Kiersten Johnson, Leo Ryan
SF 101 - Outline
• Part 1: Introduction--“It’s complicated”
• Part 2: Practical Recent M&E Experience
• Part 3: SF 101: – Planning: Small but Critical Steps– M&E: Do we count all trees to measure a forest?
• Conclusion
PART 1 - INTRODUCTIONIt’s not complicated; it’s complex
Current Context Exhibit A: Global Health Initiative
• “The challenge of the next decade and beyond is to take these impressive accomplishments to the next level by helping countries achieve long-term sustainability in their health services.”
• “Building on a long tradition of U.S. government global health leadership and the unprecedented level of commitment manifested in recent years, the Obama Administration’s Global Health Initiative has the opportunity to move global health to a new level of effectiveness, with a vision of long-term sustainability led by partner countries.”
GHI— [Implementation of the Global Health Initiative: Consultation Document]
Do we need a system perspective?E.g. Urban Health System, Bangladesh (Concern)
Health Platform (MESPCC)
NGO Health
Providers
ADB, DfID, USAID, etc.
Govt & Private Health Facilities
Ward Health Committee
Community organizations
Traditional health providersYouth volunteers
Teachers
Private pharmacists
Social & religious leaders
Commissioner
City Government
MOLGRD
Chairman
Health Department
Health Inspector (in absence of Med
Off.)
MOHFW District & Sub-District
MOHFW
Sustainability in HSS: a Complex Equation?
Illustration inspired by Samir Rihani: “Complex Systems: Theory and Development Practice. Understanding Non-Linear Realities.”
“Health -programme sustainability is the ultimate manifestation of a complex web of inter-relations between health concerns, stakeholders, resources, and actions analogous to an ecosystem.”
Gruen et al. The Lancet. 2008
Common Evaluation Questions about Sustainability
Questions I don’t understand:• Is it sustainable?
• Can you ensure it will be sustained?
• Who will take over after you leave?
Questions I understand: Will you leave partners with a better chance of
facing what will come after you leave? Are you at least improving expectations for
continued progress in the context? Are you thinking “within a system?” [a.k.a. is the
“Sustainability Scenario” coherent from an internal development logic?]
Do you have information about where your efforts are heading?
Are your partners mastering the processes which produce the results?
Do the M&E systems inform the local actors about their progress?
Evolving Understanding of Sustainability— A Suggestion:
• An (emerging?) property within a local system which allows interdependent actors to maintain and improve the health status of the system’s (vulnerable) population through negotiated and coordinated social interactions, allowing the expression of their respective and collective capabilities.
Implications of this definition1. Other definitions (i.e. financial viability, organizational
capacity, policy alignment), identify potential determinants of sustainability
2. Projects can contribute to the process, or hinder it, but ultimately neither control nor define it.
3. We can, however, measure progress on hypothesized determinants of sustainability during our projects
4. More than one configuration can support the same sustainable outcome
5. Allowing systems to find their own new equilibrium might be more important than enforcing the best approach
Understanding the Role of External Agents in Building Sustainability
In a Linear Process• Attribution:
Input->Process->Output->Outcome->Impact
• Direct Accountability– “ensure sustainability in X years”
• Fits Top-Down Planning– Caveat:“involve stakeholders”– pre-determined “models”
• “We come, we fix, we go. Hey! They ‘got it’!”
• Evidence? – Partial, sub-systems, very thin.
In a Complex Process
• Contribution:Ownership? Negotiation? Alignment?
• Shared Accountability– “advance capacity & processes”
• Role of “Agents”– Agents have strategies– “models” emerge / constructed
• “Donors and Implementers as Types of Agents.”
• Evidence?– Challenging ; Growing Interest;
More robust?
PART 2 – THREE RECENT EXAMPLES
1. Prospective (USAID/NEPAL)
Point in time evaluation of progress toward sustainability by supported districts
2. Prospective & Retrospective (USAID/CSHGP/Concern Bangladesh)
Planning for sustainabilityWhat has been sustained?How much has been sustained?
3. Modeling Investments (USAID/CSHGP/Save the Children Guinea)
What could we achieve if…?
The Sustainability Framework –Time and Perspective in Evaluation
Prospective: What are the long-term odds based on the results, processes, and systems being built?
Retrospective: Were gains maintained?
Did progress continue? How did system adapt?
Constraints & Black Swans
Implement HSS
Intervention
Intervention Ends
Stakeholders Adapt & Organize
Improvements continue and/or
Achievements are maintained
SF
Source: Taking the Long View: A Practical Guide to Sustainability Planning and Measurement in Community-Oriented Health Programming. 2008. http://www.mchipngo.net/controllers/link.cfc?method=tools_sustain. And: Black Swan, Grey Swan, Sustainability. Or, the difference between Planning and Predicting. Available at: http://cedarscenter.com/resdetail.cfm?resid=125
E.g. 1- Measuring sustainability as a programming tool for health sector investments – USAID Nepal
Using Composite / Dashboard Measures
Intermediate
Promising
Poor
Nascent
Strong
Compo 1 index
Compo 4 index
Compo 5 index
Compo 6 index2
3
Source: Measuring sustainability as a programming tool for health sector investments—report from a pilot sustainability assessment in five Nepalese health districts. Int J Health Plann Manage. 2009 Oct-Dec;24(4):326-50 http://www3.interscience.wiley.com/journal/118903030/abstract
Can districts sustain the health of their mothers & children?
Kanchanpur and Rasuwa Districts, Nepal - 2006
E.g. 2- Sustained Results 5 Years Post-Project. Urban Health / Concern Bangladesh: 1999-2004↨-2009
• From Urban Child Survival Project to Urban Health System Strengthening
– Defining a Sustainability Scenario consistent with national policy
– Defining ±consistent measures (incl. outcomes and repeated capacity assessments)
– Repeated assessments & collective learning
• 18 months post-project coaching (minimal)
→ 3-year and 5-year post-project sustainability assessments
Source: Sustainability of the Saidpur and Parbatipur Urban Health Model (Bangladesh) Five Years After the End of Concern’s Child Survival Project . Final Evaluation Report – January 10, 2010. USAID, Concern Worldwide. ICF Macro/. Available at: http://cedarscenter.com/resdetail.cfm?resid=106
CWI Bangladesh: 1999-2004↨-2009How Much Sustainability in Health Outcomes?
Project Ends
E.g.3- Does it matter? HSS at District Level: Sustainability Investments and Child Deaths Averted
Source: Pro-sustainability choices and child deaths averted: from project experience to investment strategy Health Policy and Planning 2010;1–12. ; doi: 10.1093/heapol/czq042
Save the Children US in Guinea
A Simplified Traditional Scenario for Investment
Total Investment:$7.5 millions
U5 Lives Saved:2,530 [1,569 – 6,167]
Investment Scheme for Maximizing Sustainability – Phase 1
Investment Scheme for Maximizing Sustainability – Phase 2
Investment Scheme for Maximizing Sustainability – Phase 3
Investment Scheme for Maximizing Sustainability – Phase 4
Investment Scheme for Maximizing Sustainability – Phase 5
Total Investment:$7.5 millions
U5 Lives Saved:8,485 [4,169 – 8,909]
Testable Hypothesis: three-time the impact for the same investment
PART 3SF 101 - From Planning to M&E
‘Sustainability Manual’ Tools
• Any-Stage Rapid Tool– Sustainability Planning Checklist
• Step 1– Local system mapping and stakeholder analysis– Visioning activity
• Step 2– From Visioning to Developing a “Sustainability
Scenario”
Stakeholders’ Analysis for Sustainability Planning
POWER /
INFLUENCE Low-----------------------------------------High
INFORM & ANSWER CONCERNS
Core of the SystemChange Agents
MINIMAL INVOLVEMENT
KEEP INFORMED & FIND OPPORTUNITIES FOR COOPERATION WITH LOW
TRANSACTION COST
INTERESTLow---------------------------------------------------------------------------------High
‘Sustainability Manual’ Tools for M&E
• Facilitating a detailed planning workshop with local stakeholders
• Measurement tools & Environmental Scan • Notes on basing an evaluation on the
Sustainability Framework
Local actors revise sustainability vision
and scenario
Local actors are supported in
developing a (draft) sustainability vision
and scenario
Project designs interventions to
strategically CONTRIBUTE in the
most effective manner possible to the achievement of
the sustainability scenario
Projects develops its M&E plan, in
order to be accountable for its
progress. Project also adds
element of information in its M&E plan, to help
the local actors measure progress
toward their sustainability
scenario.
Participatory evidence-based
M&E activities are implemented to
help answer: [1] is the project
effective in achieving its objectives?
[2] are there indications of
progress toward the sustainability scenario of local
actors?[3] what lessons
are being learned about the
sustainability scenario (still
valid?), the value of the project’s
contribution (still strategic); and the
role played by local actors (consistent
with their own sustainability
scenario?)
Corrective measures are taken by the project and
by local actors
First Steps in Sustainability Planning– Who Does What?
THE BEAUTY AND DANGERS OF M&E FRAMEWORKS
Component 1—Health Outcomes
Component 2—Services
Component 3—Regional/District MOH Capacity & Viability
Component 4—Local NGO Capacity & Viability
Component 5—Community Capacity
Component 6—“Enabling” Environment
Proportion of pregnant women in targeted households who slept under an ITN the previous night
Quality of existing services
DPHO capacity and viability
Capacity of NRD (local NGO)
Community Health Competency in Malaria
Environment Scan
Proportion of children under five in targeted households who slept under an ITN the previous night
Coverage of health services – facilities + community based
Viability of NRD (local NGO)
Proportion of targeted households with a pregnant woman or children under 5 with at least one ITN
Community-based treatment policy
Proportion of women in targeted households who received two or more doses of IPTp during their last pregnancy in the last two years
Proportion of women in targeted households that cite IPTp as a necessary treatment during pregnancy.
National ACT + IPT supply
Proportion of children under five years old in targeted households with fever in the last two weeks who received treatment with an antimalarial according to national policy within 24 hours of onset of fever
Coverage of community groups
Functionality / Capacity of community groups
Example / Community-Based Malaria Project
Results Framework
IR1 IR2 IR3 IR4
Key Elements for Sustainability in the Results Framework of the Project
Potentially Key Elements for Sustainability in the Larger Environment of the Project and National Stakeholders
Elements in the Results Framework Sustainability Framework
Component 1
Component 2
Component 3
Component 4
Component 1
Component 2
Component 3Component 4
Component 5
Baseline 18 months
Project Objective
Sustainability Scenario
Assessing Progress Toward Sustainability
Project
Accountability
Shared
Accountability
Results Framework Aligned to the SF (Taking the Long View. CARE Nepal)
Dimension 1
IR1: Improved MN outcomes esp amongst marginalized populations
IR2:Improved
maternal & neonatal services
Dimension 2
IR3:Improved
capacity of MOH and
other sectors for MN care
IR4:Improved viability of
maternal and neonatal services
Strategic Objective: Sustained and equitable improvements in maternal and newborn health in the district of Doti and Kailali
especially amongst marginalized populations
IR5:Improved
community commitment for maternal & neonatal
care
IR6:Improved
environment for maternal & neonatal
care
The SF “Dashboard”A Picture Is Worth 1,000 Words But Can Sometimes Tell a Lie
Conclusion• Some Practical Tool and Guidance; No Insurance Policy• Nothing (Really) New Under the Sun, Except Getting
Serious:– Implications for Health and Community Systems– Implications for Scale Up Strategies– Implications for Integration of Health with Sustainable
Development in view of Global Challenges– Implications for Evaluation +++
• Don’t Lose Sight of the Forest for the Trees• The [Sustainability] Questions of Donors are also the
Questions for Countries Themselves
THANK YOU.