Monitoring-fit for purpose?
-
Upload
irc -
Category
Technology
-
view
373 -
download
0
description
Transcript of Monitoring-fit for purpose?
Monitoring – Fit for purpose?Rachel Norman, PhD Researcher &Dr Richard Franceys, Senior Lecturer
1
The Research Inquiry
2
WHAT IS CONSIDERED AN APPROPRIATE LEVEL OF M&E TO ENSURE AN ANSWER OF YES TO THE FOLLOWING QUESTIONS?
Is it there? Is it working continuously? Is it used by all? Is it having an impact?
Where does the responsibility lie in terms of defining what is appropriate?
Does an appropriate level of M&E currently exist in the sector?
Is the implementation of M&E ensuring improved service delivery?
WASH Sector Service Delivery
WHAT COST WHY HOW USE
Research Overview
3
Research Aim To understand better how much WASH sector stakeholders are investing in monitoring and evaluation and how the data generated is being used.
Working Hypothesis Monitoring and evaluation in the WASH sector in lower-
income countries is not value for money …… because it is not used.
Objectives 1. To identify the evolution of M&E. 2. To examine the conceptual framework of Cost of M&E. 3. To examine budgeted and absolute costs of M&E at
global, national and programme level. 4. To explore the underlying purpose and use of M&E. 5. To examine the conceptual framework of VfM and
identify whether there is a core set of indicators. 6. To understand the potential for harmonisation and
alignment with the SDGs and HR frameworks.
Case Study Strategy – Global / Kenya / Uganda Mixed method design whereby qualitative data is used to validate quantitative data collection, analysis and visualisation
Data collection, analysis and visualisation
• Approx. 90 stakeholders identified• Approx. 900 academic articles sourced and reviewed• Approx.1000 (Global, Kenya, Uganda), documents & data archives
sourced and analysed:
• Approx. 90 interviews carried out.
LITERATURE REVIEWS DOCUMENT & DATA ARCHIVE REVIEWS
INTERVIEWS SURVEYS
4
What are monitoring and evaluating?
Literature shows:• 1960s: Coverage data• 1980s: Minimum Evaluation Procedures• 2000s: Guidelines and Toolkits providing
hundreds of indicators to choose to monitor
5
What are monitoring and evaluating?
Country level data analysis shows:• Increasing number of indicators being reported over
time.
6
Where are the indicators coming from?
Kenya: National: NIMES & KJAS
Sector: SWaP, JAR, Regulator
7
Uganda:National: NIMES & JAF & Baraza
Sector: SWaP, JSR, JTR
Programmes and Projects: Organisations, institutions, agencies
National and International public, private and civil society organisations
Clustered data
• Approximately 50% of indicator entries relate to Service level data
• Coverage is still the most frequently reported indicator
• Functionality reporting is limited
8
Service(Consumer)
Service Provider (Regional & Local)
Sector (National)
•Is it there?•Is it working?•Is it being used by the right people?
•Is it having an impact?
•Regional & Local
•National
Why are we monitoring and evaluating?
KENYA• ~ 50% of data records
reported a purpose
UGANDA• < 50% of data records
reported a purpose
9
How is M&E carried out?
Theory• Academic research
• 644,000 articles of which 3% related to WASH
• 79 different tools methods and approaches of which 57% related to WASH such as:
• Randomised controlled trials
• Cost-benefit analysis• Benchmarking• Needs assessments
10
How is M&E carried out? (cont.)
Practically WASH M&E in pictures?
11
• Practical application• Driven by accountability
requirements• Lack of understanding of
the options
What is the cost of M&E?
• M&E…. ‘are expensive activities yet there has been little analysis of whether they provide value for money’ (Maddock, 1993)
• Information systems….. ‘because of the lack of published research in this area, no established methodology could readily be adapted to this study…’ (Rommelmann et al, 2003)
• Multiplicity of M&E systems: ‘they also have a cost in terms of the muddle that it creates…’ (Anonymous, 2011)
12
What is the current spending on M&E?
• Possible spending of sector budget on M&E - Kenya
• MWI Sector Actual / Budget figures are sourced from 2011 Environment Water & Irrigation Sector Report 2010
• Possible spending of sector budget on M&E - Uganda
• WSS sub-sector figures are sourced from MWE (2010) SPR and MWE (2011) SPR
13
Is information actually USED to improve service delivery?
Very limited reported/documented information on use:Used for internal and external accountabilityUsed for internal management decisions
Stakeholder interviews suggest use:• Through learning and knowledge/information sharing• Policy formulation and to inform new planning • Performance monitoring.
14
Analysis of stakeholder interviews on Use
Kenya Uganda
15
Discussion
• Who, What and Why?• Common goal and yet multiplicity of indicators• Internally or externally driven or supported?• The definition crisis causing duplication
• Who, How and Why?• Capacity and capability is an issue• Research and development is an opportunity• Roles and responsibilities are a requisite
16
Discussion (cont.)
• What Cost• Accountability and yet no accountability• Too much or too little – difficult to quantify?• Embedded or extracted?
• How Used• Limited resources – a repetitive explanation• Used but no audit trail• Lead time on use
17
Interim Conclusions
• What is considered an appropriate level of M&E?• Where does the responsibility lie in terms of
defining what is appropriate?• Does an appropriate level of M&E currently exist
in the WASH sector?• Is the implementation of M&E ensuring improved
service delivery?
18
Interim Conclusions (cont.)
19