Internal Oversight Service Evaluation Office (EVS)

IOS/EVS/PI/147

Original: English

Evaluation Strategy

2016 - 2018

Document 6.2

2

Introduction

1. At its 196th session in April 2015, the Executive Board endorsed the UNESCO Evaluation Policy (196 EX/24.INF). The policy establishes an ambitious framework for ensuring a stronger and more integrated evaluation system within UNESCO that provides credible evidence to support the Organization in achieving its mandate. The policy addresses the guiding principles for evaluation in UNESCO, key requirements for planning and resourcing evaluation, roles and responsibilities, quality assurance mechanisms, evaluation follow-up and evaluation use.

2. In order to facilitate the effective implementation of the evaluation policy, the following strategy has been developed. It is not an organization-wide strategy, rather it focuses on the strategic actions to be taken by the IOS Evaluation Office (EVS) in support of its expected results for the current four-year programme cycle. In combination with the evaluation policy, this strategy will help to position evaluation as a critical management tool for enhancing accountability, promoting organizational learning and knowledge management, and informing decision-making processes. The ultimate purpose of programme evaluation is to use the information to improve programs. As such, the strategy places great emphasis on actions geared towards enhancing evaluation quality and use.

3. The strategy defines two main outcome areas along with a number of targeted actions that will be implemented to achieve them. Each outcome contains a set of indicators, baselines and targets to ensure effective monitoring in the delivery of the strategy.

4. The paper is organized in the following manner. Section one is a situation analysis outlining the key challenges to the effective operation of the evaluation system. Section two describes the main result areas of the strategy. Section three discusses the strategic actions that will be implemented to achieve the results. The following section presents an analysis of the major risks that may emerge during implementation of the strategy, along with mitigating measures. Section five presents the resource requirements necessary to support the implementation of the strategy. Section six briefly discusses the arrangements for monitoring, reporting and reviewing the strategy. The final section contains the implementation plan with key milestones. The annexes include the provisional corporate evaluation work plan for 2016 and the monitoring framework.

Key principles of the Evaluation Strategy

5. The UNESCO Evaluation Policy already establishes a number of core guiding principles for the evaluation function the foremost of which are independence, credibility and utility. In the implementation of this evaluation strategy, EVS will continue to uphold these key principles during all phases of the evaluation process.

6. The evaluation strategy especially aims to improve utility, as evaluation must contribute to organizational learning and accountability for results. Recent discussions and decisions of the Executive Board illustrate a greater opportunity for evaluation to inform key decision-making processes, in particular via the Strategic Results Report. The principles of inclusiveness and participation will be addressed through greater efforts to ensure wider participation of all key stakeholders throughout the evaluation process, as well as beginning to address means for building national evaluation capacities.

Document 6.2

3

7. When possible and appropriate, UNESCO evaluations will seek to systematically address as strategic cross-cutting issues that are relevant in particular for attaining Agenda 2030: gender equality dimensions, youth, the role of culture in all aspects of UNESCO’s programming, human rights based approaches, the effectiveness of normative aspects of UNESCO programmes and Africa priority.

Overview of roles and responsibilities for evaluation

8. The Internal Oversight Service Evaluation Office (EVS) is the custodian of the evaluation function. As such EVS is responsible for establishing an effective corporate and decentralized1 evaluation system to inform evidenced-based policies and programmes and for greater accountability and organizational learning. As it concerns the decentralized evaluation system, the EVS will develop an evaluation focal point network, update guidance material, conduct regular training, and provide technical backstopping and advisory support as part of the quality assurance mechanism for decentralized evaluations.

9. As noted above, a network of evaluation focal points will be established. The focal points will be responsible for developing decentralized evaluation plans, consulting regularly with EVS on the finalization of the decentralized plan and providing advisory support to the respective UNESCO entities on the management of decentralized evaluations. When necessary, evaluation focal points will seek guidance and support from EVS in the context of carrying out their function.

10. Other UNESCO entities, namely staff members, are expected to participate as key stakeholders in the evaluation process. Their responsibilities include, inter alia, participation in evaluation reference groups, providing EVS and external evaluation teams with all necessary information to facilitate the implementation of evaluations, and to ensure the effective sharing and use of evaluation findings and lessons.

I – Situation analysis

11. This section sets the stage by briefly discussing the existing underlying challenges across the evaluation system in UNESCO. The analysis provides the basis for determining the planned outcomes of the strategy and the associated actions to be taken to achieve them.

Uneven recognition and demand for evaluation among Programme Sectors: Traditionally,

Education and Natural Sciences have been very good clients for evaluation, followed by Culture. Social and Human Sciences and Communication and Information have been more passive and less willing to both nominate and provide funding for evaluations.

Quality of decentralized evaluations: The ‘Diagnostic Study of Evaluations of UNESCO’s Extrabudgetary Activities’ (IOS/2013), and the emerging findings from the ongoing meta-evaluation / synthesis review of completed evaluations, both strongly indicate that many reports do not meet the expected standards on numerous quality criteria.

1 As contained in the UNESCO Evaluation Policy, recall that decentralized evaluations are ones managed by UNESCO entities with a programmatic function, typically one of the Programme sectors or field units, and conducted by evaluators who have not been involved in any manner in the subject under evaluation. IOS does not manage decentralized evaluations.

Document 6.2

4

Weak planning and budgeting of decentralized evaluations: There is currently no

systematic planning and budgeting of decentralized evaluations. As a result, budgets for evaluation are not routinely secured during the project design phase, nor do they meet the 3% guideline; Executive Offices are unaware of which extrabudgetary projects are to be evaluated during a given biennium whether at Headquarters or in the Field; and, there is no central repository where Programme Sectors and Field Offices share completed decentralized evaluations.

Funding for evaluations: while recognized that EVS will need to improve its monitoring

efforts, all signs indicate that the 3% target to be earmarked for the evaluation of extrabudgetary projects is far from being met. The programme sectors need to take greater ownership in meeting this funding requirement. The situation is less challenging for corporate evaluations (e.g. more than adequate funding of the education sector during the 2014-16 thanks to generous extra-budgetary contribution from Norway), the pattern across the five programme sectors is uneven and not all cross-cutting issues are funded in the 2016 corporate plan. Incomplete coverage of key programme priorities is a major risk in terms of ensuring sufficient evaluation evidence to feed into the Strategic Results Reporting (SRR).

Evaluation management capacities: the overall evaluation capacity outside of EVS is well

below the desirable level. Apart from a couple field offices with designated M & E personnel (Iraq and Brazil), capacities in the field and Headquarters are weak, if non-existent, and staff have not benefitted from any formal institutional evaluation training.

Outreach and communication of evaluation knowledge: far too often, evaluations are not

as effective as they might be due to the lack of an approach for communicating the findings and lessons learned. Effective approaches must be embedded in all evaluations in order to improve knowledge management and organizational learning.

National capacity development: as the General Assembly resolution calling for UN

agencies to support efforts in building national capacities for evaluation is rather recent (late 2014), how to address this aspect has not been fully considered or developed in current evaluation practices.

Gender equality dimensions: our ongoing monitoring and self-assessments in the context

of the UN-SWAP (System-Wide Action Plan on gender equality and the empowerment of women) indicate that more improvement is needed to effectively integrate these dimensions into evaluation. The Evaluation Office’s 2015 assessment was ‘approaching requirements’ and the target is to achieve ‘meeting expectations’’ in 2016.

Follow-up to and use of evaluation: the main challenge remains the need for a more

systematized approach to evaluation follow-up and implementation of recommendations whereby the programme sectors and services better understand the respective roles, the periodicity for submitting action plans and progress reports, and assuming greater

Document 6.2

5

ownership in using evaluation to drive programme improvement and organizational learning.

II – Key result areas of the evaluation strategy

12. The Evaluation Strategy is operationalized within the framework of UNESCO’s corporate and decentralized evaluation plans and UNESCO’s programme and budget (C/5). The table below presents the overall results framework for the evaluation strategy. The principal outcome of the evaluation strategy is to:

improve decision-making, enhance organizational learning, and to contribute to programme improvement and accountability from the use of

evaluation findings and recommendations.

13. This corresponds to the outcome contained in UNESCO’s Programme and Budget (37 C/5).

Table – Results framework

Results

Hierarchy

Description

Mai

n

Ou

tco

me

Improved decision-making, organizational learning, programme improvement and accountability from the use of evaluation findings and recommendations

K

ey r

esu

lt a

reas

More effective corporate system capable of generating useful, credible and strategically relevant evaluations

Improved communication, visibility and use of evaluation findings and lessons learned

Improved funding mechanism providing greater resources for evaluation

Enhanced capacity of staff to plan, manage & use evaluations

Improved quality of decentralized evaluations

Quality assurance mechanism established and operational

K

ey O

utp

uts

Set of harmonized procedures for managing corporate evaluations

Corporate evaluation plan implemented Statutory reports delivered (summaries

of evaluations and annual report) Annual meta-evaluation undertaken Synthetic reviews conducted

Evaluation focal point network established and functioning

Evaluation management training programme operational

Evaluation guidance material updated Updated roster of evaluators shared Annual meta-evaluation undertaken Synthetic reviews conducted

Document 6.2

6

Results

Hierarchy

Description

S

trat

egic

act

ion

s

Systematize the roles and responsibilities in evaluation processes

Broaden the financial resources in support of evaluation

Professionalization of EVS staff to strengthen internal capacities

Improving outreach and communication strategies to facilitate the sharing of evaluation knowledge and lessons

Institutionalize evaluation follow-up processes

Enhancing quality assurance mechanisms to improve report quality

Establish the evaluation focal point network

Develop evaluation management training programme

Upgrade and improve the use of evaluation guidance material

Maintain an effective quality assurance mechanism

Contribute to national capacity development efforts

Undertake periodic meta-evaluations and evaluation synthesis exercises 

Establish an evaluation budget marker or alternative within the SISTER tool

Support to the corporate

evaluation system

Support to the decentralized

evaluation system

14. The framework illustrates in a summarized manner how the EVS intends to achieve the main outcome by way of delivery on a number of key result areas and associated outputs. The framework consists of two major streams of work:

actions in support of the corporate evaluation system, and  actions in support of the decentralized evaluation system.

III – Strategic actions

Actions in support of the corporate evaluation system

15. EVS Systematize the roles and responsibilities in evaluation processes: The EVS is solely responsible for the management of all corporate evaluations. In order to strengthen the overall planning, implementation and follow-up to corporate evaluations, the role of the EVS and other UNESCO entities within key evaluation processes requires further clarification and harmonization. This is especially so considering that the decentralized system is simultaneously being introduced to the programme sectors and services. The various roles and responsibilities for each process will be clarified in a set of new and updated evaluation guidance material.

16. For example, in order to improve evaluation planning the EVS will institutionalize the practice of organizing annual in-house consultations with each programme sector and other relevant services. The participatory consultation process is intended to ensure, inter alia, that corporate evaluations respond to the strategic decision-making needs of the Organization, that there is adequate coverage of all programmes, that gender equality dimensions are integrated into evaluations and that necessary funds are being allocated during the planning phase for evaluation.

Document 6.2

7

17. Broaden the financial resources in support of evaluation: While the UNESCO Evaluation Policy already establishes a target of 3% of programme expenditure as the recommended level of investment in evaluation, this target is not currently being met. IOS has established a Special Account in support of evaluation, an account into which donor funds raised may be allocated. The EVS will reinforce its efforts to further broaden the resource base necessary to implement the evaluation policy, evaluation strategy and annual work plans by:

continuing to promote the Evaluation Policy. In doing so, the EVS will be advocating for greater transparency by insisting that the existing requirements on resourcing be met;

drafting an Administrative Circular outlining the requirements for funding evaluation per the existing policies;

determining the baseline (2015) for the current level of financial investment in evaluation and its quality; the ongoing meta-evaluation and synthesis review will provide useful data in this regard;

monitoring the situation regularly and report in its Annual Report to the Executive Board on the status and progress in achieving the 3% target;

developing a project proposal (concept note) for the Development of the UNESCO Evaluation Focal Point Network and actively undertake resource mobilization efforts;

examining the feasibility, in consultation with BFM, of establishing a budget marker to more effectively monitor the allocation and expenditure of evaluation-related funds.

18. Professionalization of EVS staff to strengthen internal capacities: A strong, professional corporate evaluation function is an indispensable foundation upon which UNESO can build the required evaluation capacities among its staff. The following capacity development and learning activities will be undertaken:

regular staff meetings, at least two per month, will be held as a means to exchange information about ongoing work, to share lessons learned, to discuss methodological guidance or related issues;

EVS staff will periodically train or brief other EVS staff on specific issues or methodological approaches; when possible, EVS staff will build the capacity of non-EVS UNESCO staff (Headquarters and Field) in evaluation theory, practices and methods (in particular within the context of the training programme to be developed)

staff will continue to participate in the various task forces of the United Nations Evaluation Group (UNEG);

staff will participate in meetings of evaluation societies and associations or the annual meeting of the Evaluation Practice Exchange (EPE) of UNEG, especially within the context of presenting discussion papers, moderating sessions, etc.;

staff will participate in relevant self-learning opportunities (e.g. digital learning courses such as Massive Open Online Courses), writing journal articles, publications, etc.;

internal mobility schemes will be explored by which EVS staff are assigned to programme sectors (HQ or Field) for a defined period of time. The purpose would be for the EVS staff to strengthen evaluation capacities within the respective sector / office, or for the EVS staff to enhance his / her skills and competencies within the sector joined; and

IOS as a whole will aim to play a greater leadership role within the UN evaluation and international development community e.g. taking the lead in the development of tools and methodologies related to areas such as education, cultural dimensions of evaluation and normative / standard setting dimensions.

Document 6.2

8

19. Improving outreach and communication strategies to facilitate the sharing of evaluation knowledge and lessons: Outreach / communication strategies will be identified at the planning stage of corporate evaluations to ensure that the knowledge generated by evaluation are shared with key target groups to facilitate evaluation use. The strategies will include:

introducing more participatory evaluation approaches holding end-of-evaluation workshops with stakeholders to discuss and disseminate

evaluation findings development of customized, user-friendly lessons learned papers for wider electronic

distribution greater use of the UNESCO web site and social media to disseminate evaluative

knowledge. when possible, the dissemination of evaluation findings will be organized to coincide with

specific decision-making moments such as meetings of the Governing Bodies and the various intergovernmental bodies

presenting relevant evaluation findings to the wider evaluation communities such as at meetings of national and international evaluation societies and the Annual General Meeting of the UN Evaluation Group.

20. Institutionalize evaluation follow-up processes: In order to strengthen ownership and accountability for evaluation follow-up, a number of procedures will be introduced:

during the development of Terms of Reference for a given evaluation, a responsible lead unit will be assigned the preparation of the management response.

In subsequent phases of the evaluation, namely the finalization phase of the report, this responsible lead unit will bear responsibility for liaising with other relevant UNESCO stakeholders to develop the management response.

Within one month of completing the evaluation, a time-bound action plan must be developed by the responsible lead unit indicating the actions to be taken in response to individual accepted recommendations.

The respective responsible managers will update action plans on an annual basis. Evaluation follow-up will be a topic of discussion during the annual consultation on

evaluation matters between the EVS and the respective programme sectors and services. In particular, the annual consultation will provide an opportunity for the programme sectors / services to describe how evaluations have informed policies, strategies and programmes.

to enhance both learning and accountability aspects of evaluation, the EVS will manage a central repository or data base of action plans.

21. Enhancing quality assurance mechanisms to improve report quality and to further evaluation use: Evaluation report quality is a key determinant to whether stakeholders eventually use the findings and lessons presented in the report and act upon its recommendations. For example, evaluation conclusions must present reasonable judgments based on findings and substantiated by evidence and the methods employed must be appropriate for the evaluation and to answer its questions. 

 

22. In an effort to improve quality, efforts will be made to maximize the use of outside evaluation expertise for the quality assurance of evaluation products, namely evaluation reports. In particular, United Nations evaluation peers, recognized evaluation experts in academia and in the wider evaluation community will be used, inter alia, to peer review evaluation Terms of

Document 6.2

9

Reference, methodologies and reports, to participate in evaluation management groups, and provide technical comments to proposed evaluation methodologies.  

 

23. The Strategic Results Report (SRR) is a report produced by the Secretariat (BSP in coordination with the programme sectors, field offices and IOS) in the third year of the programme cycle for the purpose of facilitating strategic decision-making and future planning of the Executive Board, in particular discussions on programme improvement, reorientation and termination. The SRR contains, inter alia, an analysis of the outcomes achieved by UNESCO programmes. In order to inform the SRR, EVS will undertake a comprehensive meta-synthesis of completed evaluations every year.

Actions in support of the decentralized evaluation system

24. Establish the evaluation focal point network: Not only is the rationale for strengthening the decentralized evaluation system clarified in the UNESCO Evaluation Policy, the Director-General’s reform of the field network had also included a proposal for developing a programme evaluation and knowledge management function in select offices–a proposal which was eventually suspended due to financial constraints.

25. EVS developed a project proposal (concept paper) which describes the overall approach for establishing the focal point network, as extrabudgetary funding will be necessary to implement the initiative. Once operational, EVS will coordinate the network which will serve as a platform capable of facilitating the exchange of information on the planning, implementation, reporting and follow-up to decentralized evaluations. The platform may include, inter alia, a support group modality on UNESCO’s Intranet (DUO) whereby EVS will be able to better serve the needs of the focal points and the wider UNESCO community on evaluation issues. In this way, the focal point network will contribute to several objectives including a more efficient use of resources for evaluation through improved planning and collaboration, higher quality decentralized evaluations, a more effective use of evaluations by UNESCO staff and stakeholders and improved opportunities to use the evaluative information coming from decentralized evaluation in synthetic reviews at the level of sectoral programmes.

26. Develop an evaluation management training programme: As part of developing the evaluation focal point network, EVS will also develop and roll-out a training programme to build evaluation capacities across UNESCO. The evaluation training programme will include direct training delivered by the EVS staff e.g. training will be incorporated as a routine exercise during all EVS field missions. Furthermore, the EVS will develop e-learning modules as part of an online delivery platform to ensure wider access and possibilities for continual upgrading of skills and competencies. The EVS will further develop and regularly update an internal roster of evaluation experts. The roster will be made available to the evaluation focal points who in turn will be expected to contribute to the list on the basis of decentralized evaluations rated as high quality.

27. Upgrade and improve the use of evaluation guidance material: In order to strengthen capacities of the proposed evaluation focal point network, the existing suite of EVS guidelines will need to be updated and new material developed tailored to the specificities of the new network.  The guidance material will include, inter alia, information on developing terms of reference, commissioning and managing decentralized evaluations and follow-up to evaluation

Document 6.2

10

report recommendations. The guidance material will be compiled in one reference document (a handbook) and serve as the main reference for decentralized evaluations.

28. Maintain an effective quality assurance mechanism: As part of its coordination role of the decentralized evaluation system, the EVS will provide technical backstopping and advisory support to the evaluation focal point network. Advice and support will come at key phases in the evaluation process – Terms of Reference development, selection of evaluation teams, evaluation design and inception phases, draft and final reports. As part of the quality assurance mechanism, the EVS will update and disseminate evaluation guidance material covering the preparation, design, conduct and follow-up to decentralized evaluations. The guidance material will also form part of the training programme and will be made available online. An evaluation consultant and peer reviewer roster will be established and made accessible to the evaluation focal point network as one means to improve evaluation quality.

29. The Secretariat (Bureau of Strategic Planning) intends to establish shortly a formal project review mechanism by which new project proposals, principally extrabudgetary projects, will be subject to a formal review. Once established, EVS will be able to use this mechanism as an additional means by which to ensure that evaluation requirements are adequately included in project proposals.

30. Contribute to national capacity development efforts: The UNESCO Evaluation Policy notes the significance and need for stronger evaluation systems across the United Nations system, in particular with regard to greater efforts aimed at national evaluation capacity development (as called for in the General Assembly resolution A/RES/69/237). The policy highlights the following important guiding principle: “Evaluations are designed and implemented in a manner such that they contribute to strengthening national capacities; evaluations are undertaken in partnership with national authorities and make use of local evaluation expertise, whenever possible”. While the EVS capacities are relatively modest with regard to how substantive of a role it can play in building national capacities, the following measures will be taken:

emphasis will be placed, where appropriate and as possible, on using local / national evaluation expertise in the conduct of both corporate and decentralized evaluations;

corporate evaluations will include tailored communication strategies to facilitate the sharing of evaluations findings and lessons with relevant national and local stakeholders;

EVS will participate, when possible, in international / regional fora such as meetings of evaluation societies as a means for sharing experiences and knowledge with wider audiences; and

EVS will participate in and contribute to United Nations Evaluation Group (UNEG) efforts in this regard.

31. Undertake regular meta-evaluations:  To  improve  the quality of evaluations,  EVS will  conduct 

regular meta‐evaluations. The main purpose of meta-evaluation exercises will be to determine the degree to which certain quality aspects of UNESCO’s evaluation reports and the underlying practices have changed over time. The meta-evaluations will be the primary tool by which EVSVS will monitor improvement in the quality of the decentralized evaluation system.

32. Improve evaluation budgeting and planning for decentralized evaluations: Efforts will continue to inform responsible staff of the need to secure an appropriate evaluation budget

Document 6.2

11

during the project design phase. These efforts will include the development of some form of budget maker, perhaps within SISTER, in order to ensure a more systematic budgeting for evaluations and subsequent monitoring and reporting on investments made in decentralized evaluations.  

IV – Risk Analysis

33. Implementation of the strategy is accompanied by various risks that may emerge during the time of execution. The table below indicates possible risks, risk levels and mitigating measures.

RISK ANALYSIS

Risk

Risk

levels

Mitigating measures

o Low/high external and/or unpredictable demand for evaluation from stakeholders

Medium Advocacy for increasing stakeholders’ use and support of evaluations

Proactive engagement with the Executive Board during the deliberations

Participation in key decision making events

o Low/high internal demand for evaluations Medium Annual bilateral consultations with programme sectors and central services

Participation of the Focal Point Network Periodic briefings to SMT

o Insufficient organisational leadership, ownership and support

High Raising awareness of the strategic utility of evaluations

SMT fostering of a corporate culture of accountability and learning that embeds evaluation into decision-making

Periodic briefings to SMT on implementation of recommendations

o Sub-optimal use of evaluation Low Management ensuring the systematic consideration of evaluation findings in new policies, strategies and programme design

Highly participatory approach in development of action plans

Ensuring proper and timely follow-up

o Inadequate human resources – skills and staff

Medium Providing guidance, training and technical advice

Enhance Focal Point Network Promote M&E project appointments for large

extra-budgetary projects

o Unpredictable and inadequate financial resources

High Corporate commitment to assigning 3% of RP activity budget for evaluations

Proactive fund-raising with donors Leveraging from joint evaluations

o Limited quality monitoring data Medium Corporate commitment to improving monitoring system

Leveraging from other data sources, especially in education area

Commissioning additional data collection

Document 6.2

12

V – Resources Requirements

34. An effective implementation of the evaluation strategy requires the allocation of adequate resources. In line with emerging best practice, the evaluation policy sets an overall target of 3% of programme expenditure (regular and extrabudgetary budget resources) as the recommended minimum level of investment in evaluation. To achieve this target the intention is to draw on the following resources:

Regular Programme (RP) budget allocations; Extrabudgetary (EXB) resource allocation to specific project(s); and Extrabudgetary support (Add EXB) from Member States and partners.2

35. The table below illustrates the actual investment in evaluation during the previous budget period versus three percent target. The level of investment will need to improve in the current period to ensure adequate resources for meeting evaluation requirements.

Table: Three Percent Target vs Actual Investment for Evaluation Resources (2014-2015)

Regular Programme Extrabudgetary Programme (Decentralized)

Target of 3%3 Actual Investment4 Target of 3%5 Actual Investment6

$ 5,000,000 $ 2,850,000 (1.7%) $ 8,500,000 $1,750,000 (0.6%)

 

36. The available resources for 2016 will allow completion of 80% of evaluations planned in the EVS corporate evaluation plan, while additional funding needs to be raised for:

evaluation of UNESCO action in freedom of expression (estimate USD 50,000); UNESCO’s field presence in Asia (estimate USD 90,000); and development of UNESCO decentralized evaluations focal point network (estimate USD

200,000 over two years).

37. With regard to funding for decentralized evaluation, greater efforts will be needed to improve evaluation budgeting during the planning phase and better tracking of actual investments in completed evaluations. A number of strategies have been discussed earlier to address these weaknesses i.e. budget marker, systematic planning of evaluations and the focal point network.

38. In order to raise the necessary funding for the evaluations mentioned above, IOS Evaluation Office will continue with its efforts to secure funding from field offices concerned (especially in regard to the review of UNESCO’s field presence in Asia), and in parallel it will continue with the targeted mobilisation strategy on a bilateral or multilateral bases among Member States and donors, including foundations, such as:

2 For example, since 2014, Evaluation Office has raised nearly USD 750,000 from Norwegian Government. 3 The figure represents 3% of the operational budget of the 37 C/5 approved addendum $507M expenditure

plan 4 Of which $2M are IOS staff costs 5 The figure represents 3% of the ‘estimated’ operational budget expenditure for the 37 C/5 biennium 6 Estimate for actual extrabudgetary investment is based on 50 total completed evaluations at average cost

of $35,000

Document 6.2

13

traditional donors, most notably Scandinavian countries; emerging donors, for example in case of Asia, China and Malaysia; and foundations, such as The Global Network on Defending and Promoting Free Expression

(IFEX).

VI – Arrangements for monitoring, reporting and reviewing the strategy

39. The EVS will develop an annual work plan at the beginning of each year indicating the evaluations that will be undertaken (the annex contains the 2016 work plan). As a rolling plan, it will be subject to periodic modifications as necessary.

40. In order to monitor the implementation of this strategy, a framework has been established containing a set of performance indicators with associated baselines and targets. The monitoring framework is presented in the annex. The EVS will monitor the implementation of the strategy and report on progress once a year in the context of its Annual Report in addition to producing periodic reports as requested e.g. for Information Meetings, to the SMT, OAC etc.

41. Similar to provisions established for the UNESCO Evaluation Policy, the EVS will conduct a self-assessment of the evaluation strategy followed by an external validation to coincide with the midpoint of UNESCO’s Medium-Term Strategy (2017). The findings and recommendations of the review will be used to make any necessary adjustments to the result areas, working strategies, performance indicators, etc.

VII – Implementation Plan

42. In line with a set of envisaged strategic actions and subject to mobilizing all required resources, the implementation plan is indicated below.

Strategic actions In support of the corporate evaluation system

2016 2017 2018

Systematize the roles and responsibilities in evaluation processes x

Broaden the financial resources in support of evaluation x* x* x*

Professionalization of EVS staff to strengthen internal capacities x x x

Improving outreach and communication strategies to facilitate the sharing of evaluation knowledge and lessons

x

Institutionalize evaluation follow-up processes x

Review / evaluation of the UNESCO evaluation function x

Strategic actions In support of the decentralized evaluation system

2016 2017 2018

Establish and maintain the evaluation focal point network x* x* x*

Develop evaluation management training programme x*

Upgrade and improve the use of evaluation guidance material x x

Maintain an effective quality assurance mechanism x x

Contribute to national capacity development efforts x* x*

Undertake periodic meta-evaluation and evaluation synthesis exercises annually x x x

Conduct synthetic review of all completed evaluations x 2019

* Pending additional funding

Document 6.2

14

Annex I. IOS Evaluation Office provisional work plan – 2016/17 The UNESCO Evaluation Policy established the parameters below for selecting and prioritizing evaluation topics. While IOS retained final decision-making authority, the process for selecting the topics for 2016 was highly participatory to ensure ownership in the findings and evaluation use.

Gap analysis conducted by IOS evaluation unit;  Annual consultations between IOS and UNESCO entities;  Explicit decisions of the Governing Bodies;  Specific donor requests; and  

Application of UNESCO guidelines for the evaluation of extrabudgetary projects.

Evaluation Universe

2016 2017

Programmes

 

Education: 

Education for All Global and Regional Coordination Mechanisms 

UNESCO’s standard‐setting work related to the Regional Higher Education Recognition Conventions 

ASP Network  

UNESCO’s role in education in emergencies and protracted crises 

Women’s and Girls’ Education   

Natural Sciences: 

Science, Technology and Innovation (STI) policy  

Social and Human Sciences: 

Networks of Mediterranean Youth (NetMed)  

Culture: 

UNESCO’s role in culture in emergencies and protracted crises 

 

Communication and Information: 

UNESCO’s action in support of Freedom of Expression 

 

Natural Sciences: 

International Basic Sciences Programme (IBSP) 

Review of UNESCO’s work on renewable energy 

Review of UNESCO’s work on engineering   

Culture: 

2001 Convention on the Protection of the Underwater Cultural Heritage 

Evaluation of capacity building mechanisms and instruments (including museums and Conventions) 

 

Communication and Information: 

Memory of the World Programme 

Decentralized Bodies UNESCO field presence Asia  UNESCO field presence Latin America 

Quality assurance and support to decentralized evaluation system

 

Advisory support to decentralized evaluations 

Synthetic review and meta‐evaluation of completed evaluations 

 

Advisory support to decentralized evaluations 

Statutory reports

IOS Annual Report and periodic report on evaluations completed 

IOS Annual Report and periodic report on evaluations completed  

For shaded items, funding has not been secured.

Document 6.2

15

Annex II : Monitoring Framework

Results

Hi erarchy

Description

Performance Indicators

Targets

O

vera

ll O

utc

om

e

Improved decision-making, organizational learning, programme improvement and accountability from the use of evaluation findings and recommendations

K

ey O

utc

om

es

More effective corporate system capable of generating useful, credible and strategically relevant evaluations

Improved funding mechanism providing greater resources for corporate evaluations

Enhanced use of corporate evaluation findings and recommendations

Enhanced visibility of corporate evaluations

Enhanced capacity of staff to plan, manage and use decentralized evaluations

Improved quality of decentralized evaluations

Enhanced use of decentralized evaluation findings and recommendations

PI 1: % of RP Activity budget for evaluations PI 2: % of EXB projects budget for evaluations PI 3: $ raised from donors PI 4 : number of page visits and report downloads

of IOS/EVS web site PI 5: number of citations/references to corporate

evaluation reports in purposive representative sample of UNESCO policy and project documents

PI 6: number of citations/references to decentralized evaluation reports in purposive representative sample of UNESCO policy and project documents

PI 7: % of decentralized evaluation reports complying with minimum quality requirements as defined by meta-evaluation

PI 8: % of corporate evaluations with majority of recommendations (partially) accepted by management

PI 9: % of ((partially) accepted) recommendations of corporate evaluations resulting in verifiable and meaningful policy response

T 1: 3% T 2: 3% T 3: 200,000 USD per year T 4: baseline (subsequent target

based on baseline) T 5: baseline (subsequent target

based on baseline) T 6: baseline (subsequent target

based on baseline) T 7: 80 % T 8: 75 % T 9: 100 %

Document 6.2

16

Results

Hi erarchy

Description

K

ey O

utp

uts

Harmonized procedures for

managing and using corporate evaluations in place

Corporate evaluation plan funded and implemented

Statutory reports delivered (summaries of evaluations and annual report)

Communication strategy for corporate evaluation developed and implemented

Funding mechanisms for corporate evaluations improved

Quality control of corporate evaluations improved

Evaluation focal point network established

Evaluation management training programme operational

Staff rotation within evaluation focal point network operational

Improved communication on planning, design and reporting of decentralized evaluations within evaluation focal point network

Quality assurance mechanism established and operational

Evaluation guidance material updated

Updated roster of evaluators shared Periodic meta-evaluation undertaken Periodic synthetic review conducted

PI 1: IOS/EVS website is accessible and up to date PI 2: Evaluation-related items in administrative

manual up to date PI 3: % of evaluations of corporate evaluation plan

funded and implemented PI 4: % of corporate evaluations with management

response and action plan PI 5: annual meetings with Sectors, central services

and selected FOs on evaluation planning and use PI 6: % of evaluations complying with revised

internal editorial and peer review procedures PI 7: % of evaluation focal points actively engaged

with duties PI 8: % of Sectors and Field Offices with adequate

reporting on decentralized evaluation to IOS/EVS PI 9: number of evaluation (focal point) staff

involved in temporary exchange between IOS/EVS and Sector/FO

PI 10: % of guidance documents on evaluation up to date

PI 11: Periodic meta-evaluation undertaken PI 12: Periodic synthetic review conducted

T 1: website IOS/EVS updated by end of 2016

T 2: administrative manual updated by end of 2016

T 3: 80 %

T 4: 100 % (or clear justification for deviation)

T 5: meetings with all relevant evaluation stakeholders held

T 6: 100 %

T 7: 100 %

T 8: 100 % T 9: 2 temporary staff exchanges

per biennium T 10: 100 % T 11: 1 per biennium T 12: 1 per quadrennium

S

trat

egic

act

ion

s

Systematize the roles and responsibilities in evaluation processes

Broaden the financial resources in support of evaluation

Professionalization of EVS staff to strengthen internal capacities

Improve outreach and communication strategies to facilitate the sharing of evaluation knowledge and lessons

Institutionalize evaluation follow-up processes

Enhance quality assurance mechanisms to improve report

Establish the evaluation focal point network

Develop evaluation management training programme

Upgrade and improve the use of evaluation guidance material

Develop an effective quality assurance mechanism

Undertake periodic meta-evaluations and evaluation synthesis exercises

Please consult section VII on Implementation Plan

Support to the corporate

evaluation system

Support to the decentralized

evaluation system

Document 6.2