UNDP M&E framework for adaptation, VRA and IAS C. O. Nyandiga and A.F Wittmann Presented at the UNDP...
-
Upload
trevor-taylor -
Category
Documents
-
view
222 -
download
0
Transcript of UNDP M&E framework for adaptation, VRA and IAS C. O. Nyandiga and A.F Wittmann Presented at the UNDP...
UNDP M&E framework for adaptation, VRA and IAS
C. O. Nyandiga and A.F Wittmann
Presented at the UNDP CBA- UNV Workshop, August 2-10, Dakar-Senegal
Monitoring = A continuous data collection and analysis process implemented to assess a project (or program or policy) and compare it with the expected performance
Evaluation = A systematic and objective measurement of the results achieved by a project, a program or a policy, in order to assess its relevance, its coherence, the efficiency of its implementation, its effectiveness and its impact, as well as its sustainability
(source : World Bank)
Monitoring and Evaluation = a systematic and on-going process, that is part of project inception, planning and implementation
M&E is necessary to ensure that the OBJECTIVES of the project are met
M&E Provides project management and stakeholders with on-going
indications of progress or lack of progress in achievements of objectives
Helps identifying problems and successes of projects, during implementation
Provides a basis for corrective actions Helps reinforcement of initial positive results Helps in determining if a project is still relevant
ADAPTATION : What do we want to measure ?
We want to make sure our objectives are achieved : improving the adaptive capacity / reducing the vulnerability of the communities and the ecosystems on which they rely.
What can we measure ?- Coverage : extent to which the project reaches vulnerable
stakeholders (individuals, households, businesses, government agencies, policymakers…)
- Impact : extent to which the project reduces the vulnerability, through policymaking, capacity building…)
- Sustainability : ability of stakeholders to continue the adaptation process beyond project lifetime
- Replicability : extent to which projects generate and disseminate results and lessons of value for replication
Coverage i. Number of households, businesses engaged in vulnerability reduction or
adaptive capacity development activities, as a proportion of households in the community or region targeted by the project.
ii. Number of policies introduced or adjusted to incorporate climate change risks.
iii. Number of investment decisions revised or made to incorporate climate change risks.
iv. Number of stakeholders (individuals, households, communities, etc.) served by new or expanded climate information management systems (e.g. early warning systems, forecasting,etc.).
Impact i. Percent change in stakeholders’ behaviours utilizing adjusted practices or
resources for managing climate change risks, assessed via QBS. ii. Percent improvement in stakeholders’ capacities to manage climate
change (e.g. communicate climate change risks, disseminate information, or make decisions based on high quality information), as relevant, assessed via QBS.
iii. Percent reduction in perceived vulnerability: a. Percent improvement in stakeholder perceptions of vulnerability to a
recurrence of primary climate change-related threat(s), assessed via QBS.
Sustainability i. Number of beneficiaries of project receiving training in
implementation of specific adaptation measures or decision-support tools.
ii. Local (or spatially appropriate) availability of skills and resources necessary to continue adaptation after conclusion of project, assessed via QBS.
iii. Support for project activities among participating communities as assessed by QBS.
iv. Number of outside programmes, policies or projects incorporating project results into their processes.
Replicability i. Number of “lessons learned” from the project
CBA falls within the UNDP Framework for M&E of Adaptation to Climate Change
The specificity of CBAdaptation = it is a challenge to monitor and evaluate it, because :
“Adaptation is not generally an outcome, but rather consists of a diverse suite of ongoing processes that enable the achievement of development objectives under climate change”
“Adaptation cuts across many development objectives”
The results of adaptation can be measured in the long-term, whereas projects have a short duration
It is difficult to decouple climate risks from other drivers (UNDP Framework for Monitoring and
Evaluation of ACC)
M&E aims to ensure and show that the CBA objectives are met :◦ Building adaptive capacity◦ Reducing community vulnerability◦ Increasing the resilience of global environmental
benefits to climate change (including variability) in the GEF focal areas
M&E applies to the 3 CBA outcomes◦ Outcome 1 – Local level (projects)◦ Outcome 2 – National level◦ Outcome 3 – Global level
4 sets of indicators are monitored◦ The UNDP Adaptation Indicators
Adaptive capacities Sustainable community management of natural
resources
◦ The SGP’s Impact Assessment System (IAS) GEB Livelihood / Empowerment
◦ The Vulnerability Reduction Assessment
◦ Volunteerism and Inclusive Participation indicator
4 sets of indicators are monitored◦At the project level◦Aggregated at the national level◦Then again aggregated at the global level
Adaptive capacity fostered among natural resource dependent communities through awareness building, enhanced access to climate change and scenario information, and improved access to alternative resources◦ 2.1 Number of stakeholders at community level (e.g. businesses, community
representatives, CBOs, NGOs) engaged by project and provided with training in climate change risk management and scenario planning.
◦ 2.2 Population covered by awareness building programmes to increase understanding of risks associated with climate change among general public and key stakeholder groups.
◦ 2.3 Increase in awareness of climate change related risks to natural resources (QBS)◦ 2.4 Percentage change in natural resource dependent population with access to
alternative or supplementary livelihood options (QBS)
Sustainable community management of natural resources in the face of climate change promoted ◦ 4.1 Percentage of population in relevant areas engaged in sustainable community
management activities◦ 4.2 Number of measures deployed as part of sustainable resource management
activities.◦ 4.3 Percentage of area of concern in which sustainable resource management activities
are implemented.◦ 4.4 Number of stakeholders (e.g. families/households) benefiting from sustainable
resource management activities (e.g. in terms of increased income or food security).◦ 4.5 Success of sustainable resource management interventions in securing livelihoods
and protecting resources (QBS).
Why ? Measure the Global Environmental Benefits and the Livelihood / Empowerment benefits generated by the project
What ?◦ GEB /
o Biodiversity-(# of species, innovations/new technologies, local/national policies)
o Land Degradation- (ha. of land restored, land sustainably managed, tons of soil erosion prevented, # of innovations/new technologies, local/national policies)
o Livelihood / Empowerment /o Poverty Reduction –(# of households or individuals
who benefited from the project, income generation achieved through the project…)
o Capacity Building –(# of NGOs, community groups whose capacities were increased, #of women participating, support ensured from local / governmental institutions…)
Who, How ? And When ? At project conception, chose one GEB indicator and one LIVELIHOOD / EMPOWERMENT indicatorDuring project development, measure the baseline values of the indicators and prepare your monitoring plan (in project proposal) – During project implementation, measure the evolutions of the indicators (project reports) – RESPONSIBILITY OF THE GRANTEEAt the end of the project, measure the final indicator (final participatory evaluation / final report) – RESPONSIBILITY OF THE GRANTEE
IAS Indicator
to be measured
How it will be measured
When it will be measured
Target value to be achieved by project end
Nb of ha of degraded land restored / combat land degradation
Addition of the surface of all the pilot sites that have been planted, and that have benefitted from resilient farming practices
Regular monitoring after completion of each activity + At the end of the project
4 ha of farmland + 10 ha of forest land
Nb of NGOs and Community Groups participating / capacity building
For all activities, a list of participants is made, desegregating the different community groups (farmers, NGO members, women, elders)
Regular monitoring after completion of each activity + At the end of the project
-One NGO-3 community groups (farmers, women, youth).
Monitoring Plan –
-What you will measure (indicator)-What is the target value of your indicator-How you will measure it-When you will measure it-Who will measure it
Don’t forget to report the indicators in the SGP
database
Arid & semi-Arid zone ecosystems Local population estimates of arid and semi-arid zone species Changes in extent of undegraded arid and semi-arid habitat.Coastal, Marine and Freshwaters: Local population estimates of target aquatic species Changes in extent of healthy coral reef Changes in extent and quality of mangroves (patch size, maximum tree
size).Forest: Local population estimates of forest species Changes in natural forest extent Number of concessionaries in the formulation of timber extractionMountain: Local population estimates of montane species Changes in extent of undegraded montane habitat Rate of deforestation on slopes >20% in project areasAgrobiodiversity: Changes in number of local land races & domestic livestock breeds. Changes in extent of agricultural area using low-input high diversity
production methods Incorporation of local land races and indigenous breeds in national
breeding programmes Certification standards for agricultural products
WHAT Systematically capture and document off-site,
indirect and longer-term impacts- “ex-post evaluation”. Generates GEB Measurements
HOW Using proxy and development indicators while
project is ongoing Using structured QBS during and after project
closure External independent consultantsWHEN Minimum 5 years after project completion-Done
by UNDP EO
What ?The objective of a CBA project is to REDUCE the vulnerability of the community. VRA is a tool that contributes to measure achievement of this objective.
VRA is a form of Participatory Impact Assessment, which focuses on the community’s own perception of its vulnerability and adaptation capacities.
Why ?◦ Measure the community’s perceptions of current and
future climate change risks and adaptive capacities (now and in the long-run)
◦ Measure the obstacles to adaptation, and the assets◦ Evaluate the sustainability of the project◦ Capture qualitative information / knowledge from the
community, essential for project development/management (to make sure the project responds to community priorities) and for knowledge-management
◦ Capture quantitative (thus comparable) information (=scores) that will allow to verify the achievement of CBA objectives (at project / national / global levels)
How ? When ?VRA is composed of 4 indicators, that are transformed into 4 questions (tailored to the local context).These questions form the core of a “VRA participatory workshop”, that is organized at least 3 times in the course of a project, in the community (at the beginning, at mid-course of project implementation, at the end).
UNDP Adaptation Policy Framework Step
VRA IndicatorVRA QuestionIn these examples, we consider the case of a community facing increasing drought risks
Assessing current vulnerability
1. Vulnerability of livelihood/welfare to existing climate change and/or climate variability.
Example: What happens when there is drought? How does this affect you and your community?
Assessing future climate risks
2. Vulnerability of livelihood/welfare to developing climate change risks.
Example: What would happen if drought was twice as frequent? How would this affect you and your community?
Formulating an adaptation strategy
3. Magnitude of barriers (institutional, policy, technological, financial, etc) barriers to adaptation.
Example: What stands in the way of adapting to increasing drought? What means do you or your community have to manage events occurring more frequently?
Continuing the adaptation process
4. Ability and willingness of the community to sustain the project intervention
Example: Rate your confidence that the (project activity) will continue after the project period.
The H-Form = the tool that is used for VRA facilitation and data collection
Where ?Choose an appropriate location in the village, according to what you already
know about the community.The location has to be accepted by / accessible to everyone
When ?Accommodate the community schedule, including women’s schedule
Who ?Facilitating team : 1 or 2 lead facilitators (with leverage and experience ; who know the local
language and culture)1 person to take notes on the H-form1 person to take extensive notes on the side1 person to take pictures and coordinate the logistics
MOBILIZE the grantee as well as community-members.BE GENDER-SENSITIVE
QUICK TIPS-Make sure all the members of the community are invited and informed about the meeting-Create a friendly atmosphere, where everyone is at ease to speak-Promote open discussion (the participants are the “teachers”, because we want to learn from them and capture their perceptions)-Use a language that is appropriate-Prepare your H-forms in advance (as well as all the material you might need)-Inform the authorities in advance
1-Introduction
Present the facilitating team
Explain the context of the workshop (CBA project) and the objectives of the workshop
Explain the “rules” : promote open participation (there is no wrong / right answer; everyone is allowed to speak…)
2-Launch a general discussion about the local environment and climate
Use simple questions and tackle climate aspects one by onewhat is the “usual” climate here ? when does it rain ? when is it hot ? For how long…Are there any storms, droughts ? When ? you can draft a season calendar for example
3-How does the community experience climate change?
Have you noticed any changes in the past years ? Has it rained more ? less ? When did it rain last year ? for how long ? Is it hotter than in the past ? is the dry season longer ? Are there more storms etc.
QUICK TIPS
-Adapt to relevant climatic matters
-Be very specific : referring to “climate change” is too vague in most contexts
-Ask simple questions one by one, so people have time to discuss them
4-What are the impacts of these changes on your livelihood = VRA QUESTION 1 / Fill out H-form
5-What will be the impacts if these changes (more storms / more droughts) double in the
future ? = VRA QUESTION 2 / Fill out H-form
The impacts are noted in 2 columns (positive impacts / negative impacts)After the discussion, assessment : how serious are these impacts, in the view of the communityWhat are the possible solutions to limit these impacts ?
6-What are the obstacles / assets for adaptation ? = VRA QUESTION 3 / Fill out H-form
What prevents you from implementing the solutions that you mention ?After the discussion, assessment : are the obstacles stronger than the assetsWhat are the possible solutions to overcome the obstacles and build on the assets ?
7-Do you think that with the project, you will be able to adapt further in the future ? = VRA QUESTION 4 / Fill out H-form
(Assess the sustainability of the project, and how people will contribute)After the discussion, assessment : are the participants optimistic about the outcomes of the projects ? are they willing to actively participate ?
8-Conclusion
Thank people for their participationPresent the next steps of the project
What are the results of a VRA session ◦ QUANTITATIVE DATA = SCORING
Generate a score for each question (noted on 5; 1 shows that the vulnerability is high / 5 shows that the vulnerability is low
Calculate the average from each questionMultiply the total average by 2 to obtain a score
noted on 10
◦ QUALITATIVE DATA = all the information captured during the session, and classified in the different parts of the H-form
Negative impacts Positive impacts Ideas, Solutions
How to use the results of a VRA session◦ The initial VRA results
Are used for project development an incorporated in the project proposal (provides data for baseline assessment / local knowledge / and ideas for project outcomes/outputs)
Provide the baseline vulnerability measurement, reference for project monitoring
◦ The 2nd and 3rd VRA results are to be compared with baseline, in order to monitor the project.
Men Women
Q1 1 1
Q2 1 1
Q3 1 1
Q4 2 2
Total 5 5
Average (noted /5) 1,25 1,25
Average (noted /10) 2,5 2,5
VRA SCORE (/10) 2,5
QUICK TIPS
Always make sure you thoroughly document the VRA sessions
Take detailed notes on your own H-form, Note your personal side-comments –how many people ? Where ? When ? Who facilitated ? What went right / wrong ? What can be improved ? …Take pictures
Don’t forget to report the indicators in the SGP database
Lessons learned from implementation VRA is a powerful way to capture a lot of information from the community It gives an opportunity for mobilization and discussion, and contributes to
fostering community ownership of the project Language and communication are critical Gender appropriateness is essential (in the choice of the location, scheduling,
language, facilitators) : in Morocco, for example, we do specific women’s workshops VRA is a flexible tool : its organization can be tailored to the specific context (for
ex. In Morocco, we don’t always use the big H-form, and we added some questions/discussions to prepare for the VRA questions) – PROVIDED ALL THE INFO IS COLLECTED FROM THE DISCUSSION
VRA can be incorporated in a larger workshop (information about CC, training, rural assessment…) so that the communities are prepared and so that the discussions are to the point.
The main challenge is the QUANTITATIVE DATA (for discussion)How can they be better understood (scale of value is not necessarily translated in numbers) ? How can they be better captured (voting/scoring might block participants; people don’t want to
stand out)? How can they be more reliable for monitoring (composition of the group changes from one VRA
to the next ; participants might over/underestimate the scores in order to ensure the funding… ?
There is a potential for improvement, in order to be able to prove the impacts of our projects.
BASELINE indicators (project proposal)- What are the mechanisms for volunteerism that
already exist in the community before the CBA project (for example, traditional mechanisms for mutual assistance, associations, etc.)?
- Number of volunteers in the community already engaged in climate change adaptation activities before the CBA project
- What are the opportunities or obstacles that could facilitate or impede people from engaging in voluntary activities?
The CBA project aims at reinforcing Volunteerism : these indicators should be monitored over the course of the project
Be sure to capture quantitative / qualitative data on community volunteers and participation during the project : in the activity reports / follow-up on the Volunteerism chart
Inclusive participation : monitor the vulnerable group’s participation (esp. women, youth, people with a handicap, elders, property-less farmers…). In the activity reports / follow-up on the Volunteerism chart
Project Conception Project development / Planning grant Project implementation
Project closing
SGP IAS Chose the relevant indicators (in concept note)
Collect baseline dataIncorporate data in project proposalReport data on SGP database
Evolution of the indicators in project reports (before each disbursement)
Final participatory project evaluation;Include final measurements in final report and K&M document
UNDP adaptation Chose the relevant indicators (in concept note)
Collect baseline dataIncorporate data in project proposal
Evolution of the indicators in project reports (before each disbursement)
Final participatory project evaluation;Include final measurements in final report and K&M document
VRA Prepare 4 tentative questions, based on the main climate risk for the community
Organize VRA session and collect quantitative and qualitative dataIncorporate data in project proposal
Organize mid-term VRA session and collect quantitative and qualitative data (include in Mid-Term report)
Organize final VRA session and collect quantitative and qualitative data;Include data in final report and K&M document
Volunteerism Collect baseline info for project note
Collect baseline dataIncorporate data in project proposal
Evolution of the indicators in project reports (before each disbursement)Participation of volunteers is highlighted in activity reports
Final participatory project evaluation;Include final measurements in final report and K&M document
APR / PIR : Annual project report / Project Implementation reportGlobal Portfolio
Write case studies / share with PMU and ALMFill out SGP databaseQuarterly portfolio report (incorporate all the added indicators / measurement from the projects)Yearly report
National Portfolio
Aggregate indicators
More and more, we are asked to prove our results.
M&E is ESSENTIAL to show that our projects achieve their objectives.
This is why we need to assess the challenges of our system, and try to improve it.
How do you implement our M&E ? How do you explain it to the proponents /
communities, and support them in implementing ?
What are the challenges you are facing ? What do you need to overcome these
challenges ?
Quantitative aspect of VRA ? Consolidate our 4 set of indicators into 1,
without losing any of the information ? Incorporate project outcome indicators ?