MoRES v 1

Formative evaluation of UNICEF’s Monitoring Results for Equity System (MoRES) UNICEF management response

Evaluation title: FORMATIVE EVALUATION OF UNICEF’S MONITORING RESULTS FOR EQUITY SYSTEM (MoRES) Year: 2014 Office and person in charge of management response: Programme Division, Ted Chaiban, Director Overall response to the evaluation: The formative evaluation of MoRES used an approach based on case studies and theory. The evaluation

affirms that the concepts underpinning MoRES present UNICEF with a number of opportunities for achieving better results for the most disadvantaged children. It also provides important insights into the conceptual, technical and operational issues that need to be addressed in order to strengthen the approach as it is being applied in wider and more varied contexts. In addition, the evaluation provides a helpful analysis of the overall governance of MoRES and management of institutional change, noting both positive aspects and shortcomings. This analysis will be helpful for guiding the further mainstreaming of MoRES and informing major corporate initiatives in the future. Overall, the findings and recommendations of the formative evaluation – together with knowledge and insights gained from ongoing field experiences – have been very valuable in informing adjustments and adaptations of MoRES application by partners at country level to ensure that the approach fulfils its primary purpose - to accelerate results for the most disadvantaged children.

Planned use of the evaluation: The formative evaluation is already informing the ongoing application of MoRES at the country level as well as

the overall MoRES mainstreaming strategy for the organization (including by identifying areas on which headquarters and regional offices need to focus). The evaluation has been widely disseminated and discussed internally at global and regional technical network meetings as well as with management. Furthermore, it has been publicized through the internal communication website, ICON. The findings and recommendations were also discussed at the Global Management Team (GMT) meeting in February 2015. The findings of the evaluation have been used to both (a) validate the work that continued during 2014, while the evaluation was taking place, to make sure the focus and priorities of the 2014 mainstreaming workplan were on track and to make any necessary adjustments; and (b) inform the next phase of MoRES: extending the application of the approach fully to all UNICEF country programmes of cooperation. This next phase involves adapting MoRES to various contexts and supporting Governments and partners to fine-tune existing national/subnational plans and programmes so that they become more focused on equity and are able to deliver even better results. The management response has been developed in consultation with regional and country offices, enabling a wider internalization of the findings across UNICEF. The insights and knowledge gained from this evaluation are valuable to the organization and contribute to the strategic efforts of UNICEF to learn from experience in order to achieve better results for children.

Recommendation 1. Maintain the focus on equity: UNICEF should emphasize that the primary purpose of MoRES is to support UNICEF’s refocus on equity through a central commitment to generating robust evidence. Overall, more will be achieved if MoRES is further mainstreamed as a

MoRES v 1

commitment to encouraging equity-focused decision-making based on robust evidence, rather than as an integrated UNICEF system. MoRES should be understood as not an ‘one size fits all’ system but as an approach that (a) begins with an intent to accelerate progress towards achieving equity for children, (b) recognizes local context – particularly the political and institutional landscape as its starting point, before (c) determining what MoRES as an evidence-based approach can add, and (d) establishing what the ideal partnership strategy is to achieve this.

Management Response (Agree, Partially Agree, Disagree): Partially Agree

UNICEF does not agree with the statement that the primary purpose of MoRES, in support of UNICEF’s refocus on equity, is to generate robust evidence.

As the recommendation further notes, a key objective of MoRES is to catalyse the use of this robust evidence to take management decisions and make

course corrections quickly to more rapidly improve programmes/policies in order to achieve better results for disadvantaged children – this use of data

in a more strategic and timely way is the primary purpose. It is agreed that the focus on equity should be maintained, that MoRES should be

communicated as an approach rather than as a UNICEF system, and that should be adaptable to context. These points have been clarified in recent

briefing notes and guidance, and disseminated to field offices.

Actions planned Responsible office/person

Expected completion date

Implementation stage: Not started

Underway

Completed

Cancelled

Actions taken Supporting documents

1. Updates to main briefing note on MoRES to emphasize key points in this recommendation, including adaptation to context, while preserving the core of the approach and working with a range of partners to apply the approach, building on existing systems and approaches in countries.

2. Technical guidance on MoRES to reflect key points in this recommendation.

MoRES Working Group / Programme Division (PD) MoRES Working Group / Programme Division

January 2015 August 2014

Completed Completed

The use of MoRES to improve equity-focused and evidence-based programming, as well as the need to adapt the approach to country contexts and to develop a clear partnership strategy at the outset, has been emphasized in all recent guidance notes and briefings. These points were clearly highlighted in the updated Briefing Note (January 2015), the widely disseminated compendium of 15

Accelerating Results for the Most Disadvantaged Children: Monitoring Results for Equity System (MoRES) Briefing Note, January 2015 The Determinant Analysis for Equity Programming, August 2014

MoRES v 1

3. Case studies documenting

the application of MoRES, highlight MoRES as an approach to support the equity refocus and to achieve results for the most disadvantaged children

4. Strategy for wider application of MoRES in 2015 in support of the equity strategy agreed and tracked at senior-management level of the organization.

MoRES Working Group Deputy Executive Director (Programmes)

Feb 2015 February 2015

Completed Underway

country case studies on MoRES, guidance on determinants analysis, document on links between existing technical guidance and MoRES, and the February 2015 Global Management Team session on MoRES. Strategy agreed at the GMT meeting and side meeting, 10-11 February 2015.

Pursuing Equity in Practice – A compendium of country case studies on the application of the Monitoring Results for Equity System (MoRES), February 2015 Key Initiatives and Tools that Support the Application of MoRES, May 2014 Key Messages and Decisions on MoRES Mainstreaming from the GMT meeting, and side meeting, 10-11 Feb 2015

Recommendation 2. Develop a mainstreaming strategy based on the enhanced conceptual clarity and findings produced by the formative evaluation report. The mainstreaming strategy should (a) recognize the importance of generating robust data-based evidence for equity gaps at national and decentralized levels, assessing the financial and capacity implications of more frequent monitoring; (b) communicate MoRES as an approach rather than a system; (c) include enhanced guidance and systems for knowledge management that support the application of the elements of MoRES; (d) devolve responsibility for the adaptation of the MoRES elements to fit the context and sector requirements to the regional and country levels; (e) introduce a resourcing and capacity plan for partners, country offices and regional offices; and (f) ensure that appropriate accountability mechanisms are in place at various levels within the UNICEF management structure [headquarters (HQ), regional offices (ROs) and country offices (COs)], with clearly defined roles and responsibilities.

Management Response: Partially Agree

MoRES v 1

While the overall recommendation on having a clear mainstreaming strategy is noted and agreed, we ‘partially agree’ with this recommendation because at the time of the evaluation, a mainstreaming strategy had already been developed for 2014 (which was not explicitly noted in the evaluation report) and a stocktaking exercise was conducted at the end of 2014. Some of the points noted in this recommendation were already included in the 2014 mainstreaming strategy even before the evaluation findings were released (e.g., communicating MoRES as an approach (point b), introducing a capacity-strengthening plan (part of point e), updating and developing guidance as needed, and enhancing systems for knowledge management (point c). However, for 2015-2016, UNICEF is updating and improving the existing mainstreaming strategy, taking into account the evaluation findings, with the actions outlined below having been identified and prioritized based on a consultative process across the organization. (It should be noted that the observation on having appropriate systems for knowledge management is a broader agenda for the organization – investments in platforms that would serve to improve overall knowledge management and exchange in UNICEF will also greatly improve knowledge management and exchange in applying MoRES.)

Actions planned Responsible office/person

Expected completion date

Implementation stage: Not started

Underway

Completed

Cancelled

Actions taken Supporting documents

1. Fully integrate the MoRES approach and its methodologies and tools into the Programme Policy and Procedure Manual (PPPM) and training materials. Through this, clearly explain link of MoRES with results-based management (including the Theory of Change) and the human rights-based approach to programming.

Field Results Group (FRG)with MoRES Working Group

September 2015

Underway MoRES has been incorporated into the PPPM to some extent, but needs to be fully integrated, including in the sections covering the situation analysis, programme design, and monitoring and evaluation.

UNICEF Programme Policy and Procedure Electronic Manual (UNICEF Intranet) Guidance on Conducting a Situation Analysis of Children’s and Women’s Rights: Taking a rights-based, equity-focused approach to Situation Analysis, March 2012

2. Integrate the MoRES framework into inSight/RAM and related guidance to ensure that results statements and

FRG/PD/Division of Data, Research and Policy (DRP)

December 2014

Completed

Procedure on Using the Results Assessment Module (RAM 2.0) of VISION/inSight

MoRES v 1

indicators are linked to the determinant framework and are tracked on a frequent basis as part of programme monitoring.

(FRG/PROCEDURE/2014/003), November 2014

3. Articulate the relationship between Harmonized Approach to Cash Transfers (HACT) processes and MoRES, particularly regarding field monitoring done through HACT and how this can be used to support frequent monitoring of bottlenecks, as stipulated in MoRES.

FRG with MoRES Working Group

May 2015 Underway Draft field monitoring tools, incorporating data collection on bottlenecks, are available and are being reviewed by a field reference group.

Field Monitoring Checklist, Draft, April 2015 Rationale for Field Monitoring Checklist (PowerPoint), April 2015

4. Update country plans for the application and documentation of MoRES experiences with identified support needs from regional offices and headquarters. Plans should be based on, or include, context analysis to determine how to adapt the approach, as well as an assessment of data availability, quality and critical gaps at country level and resource requirements.

Deputy Regional Directors with regional MoRES focal points and PD sector leads

May 2015 Underway

Regional offices have been coordinating this exercise, and several have systems to track progress on the application of MoRES in various countries. Some sectors (e.g., Education) and regions [e.g., the Middle East and North Africa (MENA), and West and Central Africa (WCAR)] have undertaken a learning / capacity needs assessment to better tailor support to

Key Messages and Decisions on MoRES Mainstreaming from the GMT meeting, and side meeting, 10-11 Feb 2015 MoRES Education Implementation and Needs Assessment Survey, February 2015 MoRES in WCARO: Update, February 2015

MoRES v 1

country offices for the application of MoRES.

MoRES in MENA – Taking stock and moving forward: Briefing for PD, January 2015 MoRES - Learning Needs Assessment Survey, MENA, 2014

5. Capacity development of

counterparts and partners

is clearly prioritized and

carried out as an integral

part of the country

application of MoRES in

various sectors.

Country offices supported by regional offices, and technical assistance from MoRES Working Group

Continuous Underway In countries where the application of MoRES has advanced, counterparts and partners have been oriented on the MoRES approach and have contributed to its adaptation based on context and requirements.

Pursuing Equity in Practice – A Compendium of Country Case Studies on the application of the Monitoring Results for Equity System (MoRES), February 2015

6. Update global

mainstreaming plan with

inputs from

country/regional levels

and headquarters

divisions, and institute a

process for tracking

progress; the global plan

will consolidate key

elements of regional-level

plans as well as plans at

global level to integrate

PD Director with MoRES Working Group

June 2015 Underway Global mainstreaming plan is being updated.

MoRES v 1

the MoRES approach into

sector-wide approaches

and tools.

7. Update quality assurance tools and mechanisms for country programme documents and sectoral programmes to include core elements of MoRES, and define roles/responsibilities of HQ, ROs and COs

MoRES Working Group (HQ and ROs) with FRG

June 2015 Not started

8. Develop and implement a knowledge exchange system, including relevant communities of practice that foster in-country and country-to-country learning on equity-focused programming among partners

DRP 31 October 2015

Underway Development of knowledge exchange platform has commenced. Public Financing for Children (PF4C) Community of Practice already created and in good use.

Concept Note, Knowledge Exchange Platform for Equity-Focused Programming

9. Work with key UN partners to integrate the MoRES approach into wider UN guidance on monitoring equity

PD/Public Partnerships Division (PPD)

Feb 2015

Completed

The draft inter-agency guide on frequent monitoring for equity has been prepared and is ready for field testing.

Guidance on UN Joint Approach to Frequent Monitoring for Equitable Results (Draft), January 2015

10. Field test and finalize joint guidance

DRP with support from PD/PPD

Dec 2015 Not started Field testing of the guidance is part of the United Nations Development Group (UNDG) workplan for 2015.

2015 UNDG Work Plan: Results and Key Deliverables (Draft)

MoRES v 1

Recommendation 3: Resolve the technical challenges associated with MoRES processes and tools. There are some elements to MoRES which add significant – even transformational – value when they fill a gap in existing systems, particularly bottleneck analysis. However, across all contexts, there remain technical challenges to resolve, which signal a need for stronger guidance and continuous review.

Management Response: (Agree, Partially Agree, Disagree) Agree If recommendation is rejected or partially accepted, report reasons:

Actions planned Responsible office/person

Expected completion date

Implementation stage: Not started

Underway

Completed

Cancelled

Actions taken Supporting documents

1. Sectors will review existing technical guidance and prepare any additional guidance required by the field, including addressing specific technical issues as needed. The updates to guidance will highlight opportunities for cross-sectoral work and engaging multiple sectors in applying MoRES to improve outcomes for children. Further clarification will also be provided on adaptation of MoRES in humanitarian contexts.

MoRES Working Group/ Programme Division

30 September 2015 (pieces that are deemed urgent and critical by the field can be completed earlier)

Underway Several Programme sections have updated 2015 plans for mainstreaming MoRES – these plans include production of any additional technical guidance that may be required (these are being consolidated and will inform the updated 2015-2016 global mainstreaming strategy).

Draft 2015 Programme Section Work Plans for MoRES Mainstreaming – selected sectors

2. Short Q & A addressing common issues from the field will be prepared and disseminated through the MoRES website and technical networks

MoRES Technical Team (HQ and ROs)

May 2015 Underway Q&A document is being drafted.

MoRES v 1

3. Strategic review of staffing profiles will be undertaken to identify critical human resource gaps and/or make adjustments to specific job descriptions to support application of MoRES at country level

Regional Offices, PD and DRP with the Division of Human Resources

August 2015 Not started

Recommendation 4: Develop a policy advocacy strategy at national level for stronger links (a feedback loop) between locally identified bottlenecks and access to the resources required to remove them. Sometimes the participation of the government in resolving bottlenecks is limited by capacity and resources. Furthermore, the complex/fragmented government planning and finance cycles in many countries and highly centralized budgets create further barriers, since resources are not easily or readily reallocated.

Management Response: (Agree, Partially Agree, Disagree): Agree MoRES can be applied and adapted in various decentralized settings. However, for the feedback loop to work effectively there needs to be a clear and strong link to decision-making processes and accountability frameworks at different levels. There are opportunities to strengthen the work on linking the identification of bottlenecks and monitoring data on bottlenecks to the variety of decision-making, costing, budgeting and financing systems in different countries. If recommendation is rejected or partially accepted, report reasons:

Actions planned Responsible office/person

Expected completion date

Implementation stage: Not started

Underway

Completed

Cancelled

Actions taken Supporting documents

1. Develop and roll out a simple advocacy planning tool and guidance for advocacy to policymakers. The tool will support advocacy on linking equity analysis, bottleneck analysis and bottleneck monitoring to budgets at different levels of governance in different settings (e.g., low-income

DRP and PD September 2015

Underway Advocacy tool is being drafted.

MoRES v 1

and middle-income countries).

2. PPPM will include a specific focus on stakeholder engagement and advocacy with governments to support use of data for decision-making

FRG and PD September 2015

Not started

3. Document adaptation of MoRES in various settings (related to level of decentralization, planning, budgeting and monitoring processes)

MoRES Working Group

February 2015

Completed The recent compendium of 15 country case studies on MoRES and the synthesis of findings also highlight how MoRES has been adapted to country contexts, including through taking into account different levels of decentralization.

Pursuing Equity in Practice – A compendium of country case studies on the application of the Monitoring Results for Equity System (MoRES), February 2015

4. As part of the Toolkit on MoRES, develop and roll out operational guidance on how to use data from MoRES to remove identified bottlenecks related to budgets, Public Financial Management and local governance practices.

PD (Social Inclusion and Policy Section)

October 2015

Underway Technical notes on relevant key topics are in draft form and will be repurposed to show clear links to MoRES.

Recommendation 5: Evaluate MORES as an approach which supports the development and implementation of the Strategic Plan and Country Programme, rather than conduct a stand-alone evaluation of MoRES. Focus on the results to be achieved for the most disadvantaged children and the accelerated reduction in equity gaps.

MoRES v 1

Management Response: (Agree, Partially Agree, Disagree) Agree If recommendation is rejected or partially accepted, report reasons:

Actions planned Responsible office/person

Expected completion date

Implementation stage: Not started

Underway

Completed

Cancelled

Actions taken Supporting documents

1. UNICEF will evaluate the application the MoRES approach within individual programmes, paying attention to the contribution of MoRES to achieving results for disadvantaged children and in reducing equity gaps. Guidance will be prepared on how to formulate Terms of Reference for evaluations intended to include assessment of the role and contribution of MoRES to programme results. Sample Terms of Reference and related guidance will be included in the PPPM and disseminated to staff and management.

FRG / Evaluation Office

June 2015 Not started

2. The 2016 midterm review of the Strategic Plan will review (among other issues) the application of

DRP September 2016

Not started

MoRES v 1

MoRES as an approach in the Plan’s roll-out.

Recommendation 6: UNICEF management should use the experience of MoRES roll-out (and evidence from other initiatives) to inform the management and implementation of future major organizational change initiatives within UNICEF. The experience of implementing MoRES within UNICEF has generated some very important lessons on how major change processes involving corporate priorities can be implemented successfully.

Management Response: (Agree, Partially Agree, Disagree) Agree If recommendation is rejected or partially accepted, report reasons:

Actions planned Responsible office/person

Expected completion date

Implementation stage: Not started

Underway

Completed

Cancelled

Actions taken Supporting documents

1. Share lessons learned from the roll-out of MoRES – particularly related to communication, field engagement, formalization of management structures, staff empowerment and generating short-term wins – with senior management for application to other major organizational change initiatives within UNICEF

Deputy Executive Director (Programmes)

Recommend- ation is continuous by its nature

Completed the sharing of lessons learned Underway - Application of lessons learned

Lessons from evaluation were discussed at the 2015 GMT meeting.

Key Messages and Decisions on MoRES Mainstreaming from the GMT meeting, and side meeting, 10-11 Feb 2015