Reflections on monitoring a large-scale civil society WASH initiative: Lessons for sector monitoring...
-
Upload
irc -
Category
Technology
-
view
141 -
download
1
Transcript of Reflections on monitoring a large-scale civil society WASH initiative: Lessons for sector monitoring...
Tim Brennan and John McKibbin
REFLECTIONS ON MONITORING A LARGE-SCALE CIVIL SOCIETY WASH INITIATIVE
A/Prof Juliet Willetts, Bruce Bailey, Dr Paul Crawford
THINKCHANGEDO
Lessons for sector monitoring and potential contributions from NGOs
IRC Symposium 2013 Addis Ababa
Griffin nrm
Key messages• Three lessons for sector monitoring from donor monitoring:1. Investing in a ‘learning focus’ as part of sector or donor
monitoring is critical
2. Tensions between specificity and flexibility in choosing indicators and frameworks must be addressed carefully
3. An actor-centred ‘theory of change’ provided a helpful, readily understandable structure
• A wide range of strategies are available to NGOs to support sector monitoring (but so far are little adopted)
A large-scale civil society WASH fund… $32.5 million 11 CSOs 21 countries
45 activities$200K- >$1.6m
782,000 people with toilets
564,000 people with water
Our role in the Fund… diverse activities of the monitoring review panel (MRP)…
• Engaged for life of the program
• Designed the performance framework and reporting requirements
• Conducted analysis and synthesis – learning papers
• Undertook monitoring of Fund activities
• Provided an M&E capacity building role
• Undertook two independent evaluations of the Fund
Lesson 1 from donor monitoring: Invest in a learning focus
• ‘Developmental evaluation’ approach – what did we do?:• Collected ‘real-time data’ to inform on-
going decision making and adaptations• Included descriptive free-form case
studies as well as indicators• Used evaluation questions (not just
indicators)• Facilitated, structured learning ‘events’• On-going one-on-one feedback to NGOs • Learning ‘products’ that synthesise
lessons across contexts
Developmental evaluation adapts evaluation (or monitoring) to “emergent and dynamic realities in complex environments” (Patton, 2011, p1).
Lesson 2 from donor monitoring: Balance specificity and flexibility
• Why is specificity needed in sector monitoring?
• Why is flexibility needed in sector monitoring?
• How can these needs be balanced?
7 mandatory indicators
19 ‘result areas’
Flexible guidance on
what to monitor or
report
We achieved balance by defining some indicators very tightly, and making space in other ways for more diverse, flexible inputs
Lesson 3 from donor monitoring: Actor-centred theory of change was helpful
• Monitoring is usually about measuring ‘change’
• ‘Theories of change’ simplify reality to make it easier to measure, communicate and understand how we think change happens
• We can then also test our assumptions about how change happens
DELIVERABLES:
E.g. training, mentoring, advocacy, technical advice, campaigns etc.
ENABLING OUTCOMES:
Outcome 4: improved WASH governance
Outcome 5:strengthened local capacity
Outcome 6: improved gender equality
Outcome 7: improved WASH evidence base
CORE OUTCOMES:
Outcome 1: increased access to sanitation
Outcome 2:increased access to water
Outcome 3:improved hygiene behaviour
CSO
Delivery Team
Institutional &
community partners
Poor communities
& households
Theory of change behind the performance arrangements
NGO roles to support sector monitoring: a ‘strategy map’Strategy Causal Persuasive Supportive
I-1 I-2 I-3Focused on a particular individual or group
E-1 E-2 E-3Focused on the enabling environ-ment
Strategy map was developed by IDRC as a part of ‘outcome mapping’ (Earl et al., 2002)
Strategy Causal Persuasive Supportive
I-1 I-2 I-3Focused on a particular individual or group
Direct role in monitoring own direct implementation activities.
Providing awareness raising, education or specific training to community members or other partners.
Providing frequent, sustained, on-going mentoring and support; or multipurpose capacity building; or developing support structures, committees & networks.
E-1 E-2 E-3Focused on the enabling environ-ment
Engaging in policy dialogue, directly causing changes in incentives, rules; playing an advocacy role.
Dissemination of information to broad audience; persuasive environment for a specific behaviour or attitude; conducting workshops/conf’s.
Building partnerships, providing collective support and promoting networking and coordination; also supporting higher levels of government or local research networks.
Examples of NGO roles and their benefit to sector monitoring
• Direct monitoring role (I-1 strategy map)
“one NGO in Bangladesh showed that government (Department of Public Health Engineering) data for sanitation coverage of 38-58% in three ‘upazilas’ in 2009 was actually at 5-28% when the NGO conducted their baseline study in 2010, with these large differences gaining attention from relevant government staff”
Examples of NGO roles and their benefit to sector monitoring• Systematic strategies to build local government expertise in monitoring (I-3)
“one NGO working in Indonesia and Bangladesh directly supported existing government monitoring systems for sanitation by working in a direct support role for local government”
Examples of NGO roles and their benefit to sector monitoring• Document and share own learning/innovations and promote their uptake(E-2)
“An NGO working in Bhutan achieved adoption of elements of their project monitoring framework which used a qualitative information scale for examining the hygienic status of toilets, and also includes a scale for examining handwashing with soap.
Their ideas were taken up and integrated into government monitoring systems, including Ministry of Health’s Annual Survey in February/March each year, and inclusion of indicators to sanitary access and hygienic toilet usage in the next Five Year Development Plan from mid-2013 as Key sector Result Areas” (Halcrow, pers comm. 2013)”
So what? What needs to change for NGOs to better support sector monitoring?
Government staff should:• Consider and communicate the range of roles which
NGOs might play in support of national initiatives
Program designers (both NGOs and donors) should:
• Make efforts to adopt strategies explored in this presentation, particularly considering strategies not yet in use
M&E specialists:• Should align monitoring and evaluation systems of
WASH projects to sector monitoring, instigate the sharing of information with government
Key messages• Three lessons for sector monitoring from donor monitoring:1. Investing in a ‘learning focus’ as part of sector or donor
monitoring is critical
2. Tensions between specificity and flexibility in choosing indicators and frameworks must be addressed carefully
3. An actor-centred ‘theory of change’ provided a helpful, readily understandable structure
• A wide range of strategies are available to NGOs to support sector monitoring (but so far are little adopted)