WORKSHOP ON THE DEVELOPMENT OF EQUITY FOCUSED AND GENDER … · 2016-09-15 · Incorporate outcome...
Transcript of WORKSHOP ON THE DEVELOPMENT OF EQUITY FOCUSED AND GENDER … · 2016-09-15 · Incorporate outcome...
WORKSHOP ON THE DEVELOPMENT OF EQUITY
FOCUSED AND GENDER RESPONSIVE EVALUATION GUIDELINES FOR CAMBODIA
22-26 AUGUST 2016 KATERINA STOLYARENKO
ANGKOR CENTURY HOTEL, SIEM REAP, CAMBODIA INDEPENDENT CONSULTANT
DAY 122 AUGUST 2016
SESSION 3: REAL-TIME
EQUITY-FOCUSED AND
GENDER RESPOSNIVE
EVALUATIONS
4
REAL-WORLD CHALLENGES TO EFGR EVALUATION
Unavailability of needed secondary data
Scarce local evaluation resources
Limited budgets for evaluations
Institutional and political constraints
Lack of an evaluation culture
Many evaluations are designed by and for external funding
agencies and seldom reflect local and national stakeholder
priorities
REAL-WORLD CHALLENGES IN INTEGRATING GENDER EQUALITY
IN EVALUATION
Conceptual difficulties
Applying the concept of “participation” to evaluation
Methodological difficulties
Applying the concept of “empowerment” to evaluation
Framework difficulties
1. Causal framework
2. Contribution framework
3. Gender analysis framework
8
ADDRESSING BUDGET CONSTRAINTS
A. Clarifying information needs
B. Simplifying the evaluation design
C. Look for reliable secondary data
D. Review sample size
E. Reducing costs of data collection and analysis
9
REDUCING COSTS OF DATA COLLECTION AND
ANALYSIS
Use self-administered questionnaires
Reduce length and complexity of survey instrument
Use direct observation
Obtain estimates from focus groups and community forums
Key informants
Participatory assessment methods
Multi-methods and triangulation
10
RATIONALIZE DATA NEEDS
Identify information needs
Simplify evaluation design (but be prepared to
compensate for ‘missing pieces’)
Review all data collection instruments and cut out any
questions not directly related to the objectives of the
evaluation.
11
LOOK FOR RELIABLE SECONDARY SOURCES
Planning studies, project administrative records, government ministries, other NGOs, universities / research institutes, mass media.
Assess the relevance and reliability of sources for the evaluation with respect to:
o Coverage of the target population
o Time period
o Relevance of the information collected
o Reliability and completeness of the data
o Potential biases
12
SOME WAYS TO SAVE TIME AND MONEY
Depending upon the purpose and level of rigor required, some of the
options might include:
Reducing the number of units studied (communities, families,
schools)
Reducing the number of case studies or the duration and complexity
of the cases
Reducing the duration or frequency of observations
13
ADDRESSING TIME CONSTRAINTS
Reduce time pressures on external consultants
Commission preparatory studies
Video conferences
Hire more consultants/researchers
Incorporate outcome indicators in project monitoring systems and documents
Technology for data inputting/coding
HOW TO OVERCOME CHALLENGES IN INTEGREGATING
GENDER EQUALITY IN EVALUATION?
Strengthen gender sensitive
indicators
Planning evaluations in programme
cycle
Methodological improvement
Evaluation capacity
development
WORKSHOP ON THE DEVELOPMENT OF EQUITY FOCUSED AND GENDER RESPONSIVE EVALUATION GUIDELINES FOR CAMBODIA
22-26 AUGUST 2016 KATERINA STOLYARENKO
ANGKOR CENTURY HOTEL, SIEM REAP, CAMBODIA INDEPENDENT CONSULTANT
DAY 223 AUGUST 2016
SESSION 4: COUNTRY LED M&E SYSTEM
KEY DEFINITIONS
NATIONAL EVALUATION POLICIES NATIONAL EVALUATION SYSTEMS
formal policy defining the purpose, responsibilities, functions and organization of the
public-sector evaluation function
a system put in place to implement NEP
Complement each other
give the normative framework mechanism that operationalize the principles dictated in the NEPs
Sources: Bamberger, et al (2014); Roseinstein (2015)
KEY DEFINITIONS
Formalized NEP - there is an official document, or decree mandating or requiring the use of evaluation
Not-formalized NEP – evaluation is conducted routinely, but there is no written policy, or there is no written policy and there is no evaluation
Well established NEP - functioning at a high level, the system is in place and is operational or evaluation practice is well established, evaluations are conducted and used and an evaluation culture exists
Evolving NEP – is or was in place and revisions are being made
Developing NEP– is being developed through advocacy among evaluators, VOPEs, stakeholders who are pro-evaluation, key players in the government international organizations
Sources: Bamberger, et al (2014); Roseinstein (2015)
THREE PATHWAYS FOR EVOLUTION
OF NATIONAL EVALUATION
POLICIES AND SYSTEMS
FACTORS CONTRIBUTING TO THE DEVELOPMENT OF NEPS
Taking control of the evaluation process
Evaluation capacity development
Integrating evaluation into the budget and financial planning systems
Standardized evaluation systems
Strengthening cross-sectoral planning and integration
Strengthening public accountability
Improving the performance of the public sector
Strengthening the results framework
Decentralization and devolution of power to local communities
HOW NEPS CAN IMPROVE GOVERNMENT PERFORMANCE?
The policy cycle: linking M&E with policy, planning, budgeting and management
FOUR ESSENTIAL BUILDING BLOCKS FOR AN EFFECTIVE NATIONAL M&E SYSTEM
NATIONAL INTEGRATED M&E FRAMEWORK
KEY COMPONENTS OF NEPS INFRASTRUCTURE
ROLES AND RESPONSIBILITIES OF NATIONALSTAKEHOLDERS IN NEPS
ROLES AND RESPONSIBILITIES OF NATIONALSTAKEHOLDERS IN NEPS
ROLE OF DONORS IN ESTABLISHING OF NEPS
CHALLENGES IN BUILDING NEPS–LESSONS FROM INTERNATIONAL EXPERIENCE
M&E drivers
A serious effort to build a national M&E system starts with a substantive government demand for M&E information. It requires political will and sustained commitment. Central leadership and a plan are very important.
M&E information is not an end in itself. A reasonable demand for evaluation is fundamental
Incentives for using M&E information
A strong civil society that demands that public policies and programmes be designed and implemented based on evidence.
The capacity to use M&E information relies on incentives that encourage managers to demand such information and actually use it as part of their normal operations. This could take the form of sanctions for not complying or rewards for meeting requirements.
Internal infrastructure on its own is insufficient to sustain an M&E system. A number of formal requirements associated with its use will oblige managers and senior officials to invest in M&E development.
CHALLENGES IN BUILDING NEPS– LESSONS FROM INTERNATIONAL EXPERIENCE
Stewardship close to the centre
It helps to have the institutional lead of the M&E system close to central government where it can not only lead in the design and roll out of the M&E system across government, but also serve as a M&E ‘champion’ and facilitator.
Since countries’ M&E systems are generally developed incrementally, it is important to monitor progress and make regular adjustments. This ‘oversight’ role can also be part of the central agency’s mandate.
Oversight by the national audit office could be a way of giving broad and public exposure to how well the M&E system is being implemented and to any adjustments that might be needed.
Incentives for using M&E information
There are too few skilled M&E specialists in most national M&E systems. A long-term training strategy needs to include a development component to build the necessary skill sets.
Other (non-technical, but important) audiences for training and M&E orientation include senior officials (who provide the leadership and political support needed to finance and sustain the M&E system); the users of M&E information (budget analysts, poverty analysts, programme managers, etc.); civil society and the private sector.
FORMULATION OF A GENDER-RESPONSIVE NEPS
WORKSHOP ON THE DEVELOPMENT OF EQUITY FOCUSED AND GENDER RESPONSIVE EVALUATION GUIDELINES FOR CAMBODIA
22-26 AUGUST 2016 KATERINA STOLYARENKO
ANGKOR CENTURY HOTEL, SIEM REAP, CAMBODIA INDEPENDENT CONSULTANT
DAY 223 AUGUST 2016
SUSTAINABLE DEVELOPMENT GOALS (SDGS)
SUSTAINABLE DEVELOPMENT GOALS (SDGS)
GENDER RELEVANT SDGS12 out of 17 Goals covering gender equality
Millennium Development Goals (MDGs) (2000-2015)
Goal 3: Promote gender equality and empower women
Target 3.A: Eliminate gender disparity in primary and secondary education, preferably by 2005, and in all levels of education no later than 2015
Goal 5: Improve maternal health
Target 5.A: Reduce by three quarters, between 1990 and 2015, the maternal mortality ratio
Target 5.B: Achieve, by 2015, universal access to reproductive health
Sustainable Development Goals (2016-2030)
Goal 5 “Achieve gender equality and empower all women and girls”
9 targets, 14 indicators addressing multiple concerns (none covered in the MDG framework)
Around 34% (58 out of 169) of Targets explicitly or implicitly address GE & EWG
Strong emphasis on disaggregation, including by sex and other relevant characteristics to capture intersecting inequalities (TG 17.18; para 74.g)
FROM MDGS TO SDGS
Marco Segone, UN Women - https://www.youtube.com/watch?v=2n0dgb26i-0
SDG INDICATORS
230 SDG indicators*
32% gender relevant indicators **
21% indicators explicitly to be
disaggregated by sex
* 7 duplicates and 2 triplicates; ** 1 indicator is repeated 3 times under different goals
EVALUATING-SDGS-EQUITY-FOCUSED-AND-GENDER-RESPONSIVE-LENS
1. 2030 Agenda/SDGs
2. EVALAGENDA 2020
3. UN RESOLUTION ON EVALUATION
4. BANGKOK DECLARATION
EVALUATING-SDGS-EQUITY-FOCUSED-AND-GENDER-RESPONSIVE-LENS
EvalGender+
EvalYouth
EvalSDGs
EvalIndigenous
Global Parliamentarian Forum for Development Evaluation
NO ONE LEFT BEHIND
2030 AGENDA/SDGS
EVALAGENDA 2020
The enabling environment for evaluation
Institutional capacities
Individual capacities for evaluation
Inter-linkages among these first three dimensions
CAMBODIA AND SDGS
CMDGs has finished at the end of 2015
CMDGs Assessment
CMDG1: Poverty Met on main indicators
CMDG2: Basic Education Most likely to be met on main indicators
CMDG3: Gender Met on main indicators
CMDG4: Child Health Met
CMDG5: Maternal Health Met
CMDG6: Communicable Diseases Met
CMDG7: Environment Met on main indicators
CMDG8: Partnership Met
CMDG9: Demining Some way to go
INTERNATIONAL EXPERIENCE: NATIONAL EVALUATION POLICIES
STATUS OF NATIONAL EVALUATION POLICIES WORLDWIDE
NEP OR EVALUATION PRACTICE (N=61)
GENDER EQUALITY AND NEPS
GENDER-RESPONSIVE NATIONAL EVALUATION POLICIES AND SYSTEMS
Country where the NEPS integrates gender equality thanks to the existence of the national gender policy
Countries where the NEPS integrates gender equality due to the existence of a national development strategy that integrates gender equality
Countries where the NEPSs do not integrate gender equality, but gender-responsive evaluations are implemented
1
2
3
National gender policy or action plan Reference to gender in the NEPS
Benin Not found No
Bhutan Not found No
Chile National Action Plan on Women, Peace and Security No
Colombia Not found No, but certain categories of “gender” or “wo-
men’s equality” projects are required to include
sex-disaggregated M&E indicators
Costa Rica Not found No
Ethiopia National Action Plan on Gender and Equality Yes
India Not found No
Kenya Plan of Action to Implement the National Policy on
Gender and Development
No
Kyrgyz Republic Not found No
Malaysia Not found No
Morocco Government Plan for Gender Equality 2012-2016 No
Nepal Not found Yes
South Africa Women’s Empowerment and Gender Equality Bill No
Sri Lanka Not found No
Uganda Uganda Gender Policy 2007 No, the Gender Policy includes an M&E strategy
National gender policy and reference to gender in the NEPSs
NEP CASE-STUDIES
South Asia Regional Consultation on National Evaluation Policies 40
Switzerland Colombia Uganda South Africa Canada Malaysia
Population (m) 8.01 47.7 35.6 50.7 34.35 29.76
Area (‘000km2) 41 1141 241 1220 9984 329
GNI per capita ($) 80,970 7,020 510 6,960 50,970 9,820
Life expectancy
(female)
85 78 55 58 83 77
Form of state federalparliamentary
republic
unitary presidentialconstitutional
republic
presidential system, multiparty democratic republic
constitutionalparliamentary republic
federal/constitutional monarchy,
parliamentary system
federal constitutional monarchy
Administrative
divisions
26 cantons 32 department, 1
capital district
111 districts, 1 capital city 9 provinces 10 provinces, 3
territories
13 states, 3 federal
territories
National Evaluation
Association
SEVAL (1996)
GEN (2006)
SIPSE UEA (2001) SAMEA (2005) CES (1981) MES (1995) (initiated),
registered 1999
Legal & Policy
Framework
constitution constitution constitution constitution administrative
policies
administrative directives
Federal Constitution
of the Swiss
Confederation of 18
April 1999
Parliament Act of 13
December 2002
Constitution (1991)
Resolution nr 63 (1994)
Ley organica 152 de
1994
Conpes 3294 de 1994
Ley 1152 de 2007
The Constitution of the Republic of Uganda
(1995)
The Local Governments Act (1997)
The Uganda Bureau of Statistics Act (1998)
The Budget Act (2001)
The National Planning Authority Act (2002)
The Public Finance and Accountability Act
(2003)
The National Audit Act (2003)
The Public Procurement and Disposal of
Public Assets Act (2003)
The Local Governments (Financial and
Accounting) Regulations (2007)
The Education Act (2008)
The NGO Policy (2008)
Constitution (sections 195,
92 and 133) of 1996
Public Service Act of 1999
Public Finance
Management Act of 2003
(sections 27 and 45)
Municipal Finance
Management Act of 2003
Government–wide
Monitoring and Evaluation
System of 2005
Federal Accountability
Act
(2006)
Policy on Evaluation
(2009)
National Action Council
Directive No1 dd. 15
October 2004
Federal Government
Circular No3 “Guideline on
Program Development
Evaluation” dd. 24 August
2005
10th Malaysian Plan 2011-
2015
Annual Budget Circular
South Asia Regional Consultation on National Evaluation Policies 41
Country Year Reform Purpose
Canada 1977, revised 1991, 2001, 2009
New Policy on Evaluation Create a comprehensive and reliable base of evaluation evidence for
use in supporting policy and programme improvement, expenditure
management, cabinet decision-making and public reporting. Requires
100% evaluation coverage every 5 years of all programs with Direct
Programme Spending.
Switzerland 1984 Research Program “The
Effectiveness of Public
Measures”
Strengthen the results orientation in the federal administration, promote
transparency and serve public accountability
Colombia 1994, revised 2005
Medium-Term Expenditure
Framework
Improve budget planning
Uganda 2004, revised 2013
National Policy on Public
Sector Monitoring and
Evaluation
Improve government performance in utilization of foreign aid and
demonstrating good governance and accountability to its citizens and
development partners
Malaysia 2005, revised 2013
Integrated Results Based
Management System
Outcome evaluations of government-funded projects and programmes
South Africa 2011 Government-wide M&E
improvement
National Evaluation Policy Framework, management performance
assessment tool, frontline service delivery monitoring, outcomes
monitoring
NEP’s Legislation
South Asia Regional Consultation on National Evaluation Policies 42
Switzerland Colombia Uganda South Africa Canada Malaysia
Long-term national
plan
Sustainable
development strategy
2015
National
Development
Plan 2010-2014
Vision 2030 2030 National Plan Government
Management
Agenda, Results for
Canadians
Vision 2020
National
ResponsibilityPCA DNP OPM DPME in Presidency CEE ICU
Main Types of
Evaluations
Policy, summative 5 types Policy,
Impact
6 types 5 types Formative
Outcome
M&E Tools SEVAL Evaluation standards SISMEGSISDEVAL
Output budgeting
tool Baraza
Outcomes based approachPlanning process
Monitoring the management performance
of departmentsMonitoring front line service
deliveryNational evaluation system
Monitoring of localgovernment
MRRSMAF
Strategic ReviewsWhole-of-
Government Framework
Government Spending and Performance
Overview
MyResults
Funding for
evaluations
Earmarked funds for
evaluation
External funding
plays an important
role in evaluation
funding,
but the level of
funding from the
national
government is
increasing
External
funding plays
an important
role in
evaluation
funding
Earmarked funds for
evaluation
Earmarked funds for
evaluation
Earmarked funds for
evaluation
South Asia Regional Consultation on National Evaluation Policies 43
Switzerland Colombia Uganda South Africa Canada Malaysia
Evaluation Programs
Centre for Continuing Education, University of Bern
University of Applied Sciences and Arts, Northwestern Switzerland, College of
education, Institute of further education and consulting, in
cooperation with the University of Bern
N/A Uganda Christian University, Makerere
University, Uganda Management
Institute
CLEAR at the University of the Witwatersrand,
CREST at the University of Stellenbosch
Carleton University, IDEA International Institute,
École Nationaled’Administration Publique,
Queen’s University, Université Laval, Universitéde Montréal, University of
Ottawa, University of Toronto
N/A
Good practices
ARAMISThe ‘External Studies’
Database
Management control meetings with the President
Town Hall MeetingsEvaluation RadarColombia Lider
Community based monitoring systems
Evaluation coordination
The outcomes system
Frontline service delivery
assessments
Results-based Management and
Accountability Frameworks Management Resources
and Results StructureFederal Accountability ActExpenditure Management
SystemStrategic Expenditure
Reviews
ProLLOBBISE
IRBMS
Key Challenges
1. Insufficient level of interest in evaluation
2. Evaluation culture
1) Data quality control
2) Utilization of M&E
1. Incentives framework for encouraging M&E practice
2. Evaluation culture
1. Planning Process
2. National Evaluation
System3. Utilization of
M&E
1. Measuring Evaluation Use
2. Promotion Evaluation Use
1. Evaluation competence
2. Insufficient resources
3. Evaluation discipline in educational institutions
45
NEPS GOAL AND OBJECTIVES
Objective 1
annual formative evaluations by every Ministry and its Programs and Activities
Objective 2
internal evaluations of it Activities and Programs
Objective 3
evaluation of key programmes once in 5 years and usage for strategic plans and budgets/policy or program adjustments
Outcome evaluations of government-
funded projects & programs
Focus
Integrated results-based management system of the public sector
EVALUATION PATH
47
LEGAL AND POLICY FRAMEWORK
REGULATED BY ADMINISTRATIVE DIRECTIVES
Outcome monitoring
• National Action Council Directive No1 dd. 15 October 2004
Outcome evaluation
• Federal Government Circular No3 “Guideline on Program Development Evaluation” dd. 24 August 2005
Outcome based planning
• 10th Malaysian Plan 2011-2015
Funding for Evaluation
• Annual Budget Circulars
• Established in 2005, when the government directed all federal- and state-level ministries and agencies to undertake outcome evaluations of their projects and programmes
• In 2009, the government adopted an outcome-based approach for planning, resource allocation, monitoring and evaluation as part of the 10th Malaysia Plan 2011–2015
The system requires annual formative evaluations for every ministry and its programmes and activities.
The evaluations are incorporated into planning processes and budgetary processes.
The evaluations also serve as supportive information for any proposal for policy or programme adjustments through the Outcome-based Budgeting System.
The approach encourages ministries to conduct internalized self-evaluations.
49
INSTITUTIONAL FRAMEWORK
M&E Champion
Ministry of Finance
ICU under Prime Minister ’s Department
• Prime Minister
• program evaluations under the budget
• project evaluations under Malaysia Plan
Administrative bodies for Evaluation
1
2
54
INTEGRATED APPROACH TO RESULTS-BASED MANAGEMENT (IRBM)
HoVerPrinciple
55
INTEGRATED DEVELOPMENT PLANNINGIntegrated National Development Framework
57
OUTCOMES-BASED BUDGETING (OBB)
58
Use of Program Logic and Programme Theory in Planning, Budgeting & M&E
2010 Rasappan, Arunaselam & Winston, Jerome, CeDRE International
ProLL
Model
59
MIS- MYRESULTS
Public Sector Policy & Program Environment
Doing The Right Things
The Right Way
For The Right People
Financial Accountability
Management Accountability
Program Accountability
MBS Program Agreement:
Verify Client Needs & Objectives
Prioritze Evaluation Questions
Determine Evaluation Design
Information Needed for
Evaluation
Determine Research Design
Design Adjustment
Data Entry
• Stakeholder Analysis of Report
• Stakeholder Feedback
Stakeholder Utilization
Facilitate
Decision-making
Logical Linkages with Substantive Purpose of Program
Data Collection
Data Sources and
Instrumentation
Preliminary Planning &
Assessment Stage
Evaluation Design
Stage
Data Collection &
Analysis Stage
Reporting & Information
Utilization Stage
© 1999 Arunaselam Rasappan/Jerry Winston
Pilot Testing• Data Collection
• Data Tabulation & Analysis
• Data Interpretation
• Validity & Reliability
Testing
Resource Program
Efficiency Improvement Compliance
Determine Evaluation Questions
• Identify pre-determined eval ssues
• Analyse program agreement
• Conduct stakeholder inventory
• Analyse program logic
Findings & Conclusions
Data Validation
Evaluation Reporting
• Multiple reports
• Multiple audiences
• User-oriented reports
• Utilization-focused reports
Data Analysis
Evaluability Assessment
ProLL Evaluation Flow Chart
PROFESSIONAL CAPACITY FOR M&E
Promotion of the evaluation function in Malaysia was stipulated by very productive tri-patriate cooperation established between the public sector (Ministry of Finance), private sector (CeDREInternational) and civil society (Volunteer Organizations of Professional Evaluation –Malaysian Evaluation Society).
Public sector
• Formulates public policies, Design Development strategies, Initiates & promote Development, Implements intervention programs, Uses M&E Services in partnership,
• Manage public resources and Build M&E capacity in Partnership
Private sector
• Provide inputs for public policy design, Advises development/policy planners, Promotes and provides M&E services, Manages private resources, Enhances M&E capabilities via research and development
VOPEs (MES)
• Advocate M&E for development results, Advocates and promotes professionalism in
• M&E, Strengthen M&E community of practitioners, Reaches out to sensitize public/private sector on M&E, Develops M&E capacity, competency and capability, act as honest broker in public-private collaboration, Promotes stronger global network relationships for shared learning
SINERGIA (NATIONAL SYSTEM OF EVALUATION OF THE RESULTS OF PUBLICMANAGEMENT)
NEPS GOAL AND OBJECTIVES
GOAL - to support the government in staying on track and improving performance as a means to achieve its public policy objectives through provision of timely and quality performance information to relevant decision-makers.
OBJECTIVES:
(1) to improve the effectiveness and impact of government policies, programmes and public institutions
(2) to help to improve efficiency and transparency of the planning and allocation of public resources
(3) to stimulate transparency in public management, in particular stimulating oversight by citizens, i.e. social control.
Colombia does not have formalized NEP
LEGAL AND POLICY FRAMEWORK
Evaluation is mandated by the national Constitution in Colombia.
The government also introduced a series of laws, decrees, and regulations to bolster the M&E system
INSTITUTIONAL FRAMEWORK
The Presidency
National Planning
Department
Ministry of Finance
Inter-sector Evaluation Committee
ORIENTING SINERGIA TOWARDS THE SDGS (2014-2030)
Colombia played a pioneering role in the creation of the High Level Inter-Institutional Commission for the Sustainable Development Goals by Decree 280 in 2015.
The commission is a formal monitoring and decision-making body organized around the implementation of the SDGs through public policies, plans, actions and programmes
Task - to direct and coordinate the process of enlisting support for, appropriating and implementing the Agenda 2030 Objectives, in association with stakeholders including territorial entities, civil society, the private sector, the academic world and the international community.
ORIENTING SINERGIA TOWARDS THE SDGS (2014-2030)
• Pioneer country in having a SDG approach in its 2014-2018National Development Plan and that this constitutes a formalinstrument for outlining the Government’s objectives.
• The plan was drawn up on the basis of three pillars,articulated around five cross-disciplinary strategies and adeveloping strategy of green growth, with the aim ofattaining the national targets for 2018 and making progresstowards the global goals of Agenda 2030.
• 2014-2018 NDP for the first time explicitly acknowledgedterritorial differences and the need to develop them so as toreach a virtuous growth path through six regional chapters,each with its own indicators and targets.
• NDP was drawn up in a participative process includingregional authorities, representatives of civil society and theprivate sector. More than 7,000 people took part in 33regional meetings and 27 sectoral consultations
ORIENTING SINERGIA TOWARDS THE SDGS (2014-2030)
2014-2018 NDP has taken 92 of the 169 objectives in the SDGs – each with at least one indicator – and they are currently part of SINERGIA’s control panels and monitoring system.
Remaining SDG objectives: i) 17 had already been incorporated into Colombian law or into CONPES documents; ii) 24 others encountered difficulties with the country’s measuring or verification systems, and iii) another 34 were not exclusive to the country
SINERGIA’S STRATEGIC EVALUATIONS IN THE
SDGS FRAMEWORK
WORKSHOP ON THE DEVELOPMENT OF EQUITY FOCUSED AND GENDER RESPONSIVE EVALUATION GUIDELINES FOR CAMBODIA
22-26 AUGUST 2016 KATERINA STOLYARENKO
ANGKOR CENTURY HOTEL, SIEM REAP, CAMBODIA INDEPENDENT CONSULTANT
DAY 3-523-26 AUGUST 2016
TYPES OF EVALUATION
Project
Programme
Thematic
Strategy/Policy
Joint
Impact
Formative
Summative
Ex-post
• Internal
• Independent
• External
WHAT MAKES EFGRE DIFFERENT FROM TRADITIONAL EVALUATION?
Stage 1: Planning
Stage 2: Preparation
Stage 3: Conduction
Stage 4: Reporting
Stage 5: Use and
Follow Up
focus on engagement, not just reporting
focus on improvement of gender equality & gender equity in a country through usage of evaluation findings
focus on inclusion of stakeholders, duty-bearers and rights-holders in evaluation process
participatory and respectful of all stakeholders (rights holders and duty bearers)
ESSENTIAL ELEMENTS OF EFGRE
1. Assessment of gender equality issues
2. Ensuring evaluation processes equity-focused and gender-responsive
GENDER-RESPONSIVE EVALUATION DESIGN
1. Integration of human rights and gender equality principles in evaluation:
Inclusion (disadvantaged and advantaged groups, duty bears and right holders)
Participation (design, management & conduction of evaluation)
Fair Power Relations (balance power relations between or within advantaged and disadvantaged groups)
2. Focus on:
Utilization
Empowerment
Feminist
STAGE 1: PLANNING
Evaluation Planning
Deciding on Evaluation Types and Timing
Budgeting for Evaluations
EVALUATION PLANNING
1. Identify evaluations to be included in the plan
(define main criteria for selection evaluations)
2. Prepare the evaluation plan
(commission and managing/involved but not managing)
3. Approve the evaluation plan
4. Review and update the evaluation plan
5. Track and report on the evaluation plan implementation
STAGE 2: PREPARATION
Evaluability Assessment
Stakeholders Analysis & Engagement
Development of ToR
Selection of Service Provider
EVALUABILITY ASSESSMENT
PURPOSE
1. To assess the readiness for an evaluation
2. To analyse whether or not an evaluation is worthwhile
3. To identify the needed changes
4. To formulate recommendations
KEY PRINCIPLES
Formative
Learning
Engaging
Need to examine if human rights and gender equality are
integrated into an intervention
MAIN STEPS
Duration
Type (internal/external)
Methodology
Scope
STAKEHOLDERS ANALYSIS
IDENTIFYING STAKEHOLDERS
Who: Stakeholders, disaggregated as appropriate
What: Their role in the intervention
Why: Gains from involvement in the evaluation
Priority: Importance of involvement in the evaluation process
When: Stage of the evaluation to engage them
How: Ways and capacities in which stakeholders will participate
Key QuestionsKey Principles
• Inclusiveness• Participatory and
reflective• Respect• Transparency and
accountability
STAKEHOLDERS ENGAGEMENT
UNEG STANDARD 4.6 ‘STAKEHOLDER ENGAGEMENT AND REFERENCE GROUPS’
Reference Group
Learning Group
Steering Group
Advisory Group
UN WOMEN APPROACH
Management Group Reference Group
TERMS OF REFERENCE
OUTLINE OF THE TORI. Background (programme/project context)
II. Description of the programme/project
III. Purpose (and use of the evaluation)
IV. Objectives (evaluation criteria and key questions)
V. Scope of the evaluation
VI. Evaluation design (process and methods)
VII. Stakeholder participation
VIII. Time frame
IX. Expected deliverables (including standard table of contents for an evaluation report)
X. Management of evaluation
XI. Evaluation team composition, skills and experiences
XII. Ethical code of conduct
EVALUATION CRITERIA
Relevance
Effectiveness
Efficiency
Sustainability
Impact
Gender equality and human rights (e.g. equality and non-discrimination, participation, inclusion, empowerment, accountability and social transformation )
EVALUATION QUESTIONS
To what extent is the intervention relevant to the needs and priorities as defined by beneficiaries?
To what extent is the intervention aligned with relevant normative frameworks for gender equality and women’s empowerment?
To what extent were the expected outcomes achieved?
To what extent was gender equality and women’s empowerment advanced as a result of the intervention?
What were the unintended effects, if any, of the intervention?
To what extent was capacity developed in order to ensure sustainability of efforts and benefits?
How will the benefits of the intervention be secured for rights holders (i.e. what accountability and oversights systems were established)?
GENDER-RESPONSIVE EVALUATION METHODS
Gender analysis frameworks
Feminist theory and methodologies
Appropriateness and relevance to both women and men
Participatory
Disaggregated data by social criteria
Constraints and challenges of informants
Gender roles and power relations
Context and culturally sensitive
Mixed methods (quantitative and qualitative)
ETHICAL CONSIDERATION
• Respect for dignity and diversity
• Right to self-determination
• Fair representation
• Compliance with codes for vulnerable groups
• Redress
• Confidentiality
• Avoidance of harm
UNEG Standards
SELECTION OF SERVICE PROVIDER
1. Consider the expertise required for conduction of evaluation
2. Consult key stakeholders
3. Incorporate in the ToR requirements for the service providers
4. Undertake an open and competitive process
EVALUATION TEAM COMPOSITION, SKILLS/QUALIFICATION
Women and men
Local and/or international perspectives
Evaluation knowledge and experience (quantitative and qualitative methods)
Content/sectoral knowledge and experience
Commitment to human rights and gender equality, and knowledge and experience in evaluating human rights and gender equality interventions
Understanding and application of UN mandates on HR & GE
Experience in and knowledge of participatory approaches and methods
Research and relational skills, including cultural competence
Knowledge of regional/country/local context and language.
UNEG standards
QUALITY ASSURANCE OF TOR
Describe the procedure for undertaking the quality assurance
Make assessment using the checklist
STAGE 3: CONDUCTION
Inception Report
Data Collection
Analysis and Reporting
OUTLINE OF INCEPTION REPORT I. Introduction
• Background and context
• Purpose, objectives and scope of the evaluation
• Theory of change or programme theory
II. Methodology
• Evaluation criteria and elaboration of key questions
• Indicators for measuring results (should be based on programme indicators)
• Evaluation design (method of data collection and analysis)
• Sample and sampling design
• Limitations to the evaluation
III. Evaluation matrix
• Summarizes the key aspects of the evaluation exercise by specifying what will be evaluated and how
IV. Work plan
V. Responsibilities, logistics and support
VI. Annexes
• Documents reviewed
• Draft data collection instruments (questionnaires and interview guides, lists of evaluation team members and contact details).
• Terms of reference
• Evaluation management and reference group members
STAGE 4: REPORTING
Evaluation Report
Stakeholders Involvement in Reviewing of Evaluation Reports
Quality Assurance of Evaluation Reports
STAGE 5: USE AND FOLLOW UP
Using Evaluation Results
Management Response
Communicating Evaluation Results
Follow Up on Implementation of Evaluation Results
MANAGEMENT RESPONSE