Post on 24-Feb-2016
description
Good Practice Planning Tips for GRPP Decision-Makers…at time of EvaluationNovember 14, 2009Dale HillIndependent Evaluaton Group, World Bank
Key Messages► 1st evaluation likely to be easier & more successful if
“enabling conditions” are in place early • If not in place, situational analysis essential before planning• Can then build in ways to compensate & progress on
conditions► When assessing timing & readiness for evaluation, need
to plan well to assure independence & needed expertise • More important for GRPPs – both needed at all stages• For independence, Governing Body makes most key decisions
► Early planning also needs to consider audience, Stakeholder involvement and dissemination needs.
2
Special Features of GRPPs in Relation to Evaluation Planning► High expectations of evaluation – to guide
development aid allocation decisions ($b) and yield broad lessons
► Open-ended programs with multi-stakeholder governance; scope and design evolve over time
► Governance & management structure raise issues of organizational independence & adequate expertise
► Broad objectives & reach; often complex multi-level operations – global, regional, country, local
► Management faces challenges of navigating complex decision-making to respond to Governing Body
3
Revisiting Enabling Conditions
► “Intentionality” – Formal Commitment to M&E and to act on findings
► Roles for M&E Defined in Advance► Formal M&E Policy and/or Principles► Results Framework
•Agreed Objectives at Founding•Monitoring Framework/logical framework• Follow through – Design System +
Baseline Data► Examples Beyond GEF at end of slides for
takeaway4
Good Practice (GP): Enabling Conditions
► Early Provision for framework, baseline data collection and monitoring
► Expert Panel set up w/Evaluation role early; Roles Defined
► UNAIDS (several evaluations completed)
► Millennium Villages (early evaluation completed only)
► GFATM (1st program-wide evaluation completed)
► GAVI (1st evaluation completed; 2nd underway)
5
Conditions Set up for First Evaluation (Rare Cases)
1st Step: Situational Analysis► Because of high expectations, planning doubly
important ► Before or soon after call for evaluation, Mgmt.
should do situational analysis & place evaluation on GB agenda
Management should ask: ► What is driving demand for evaluation at this time?► How should we take account of or build on past
evaluations (if applicable)?► What is our readiness for an evaluation at this time?► If not ready, what options do we have?
6
Timing and Past Evaluations….High Expectations, & Multiple Stakeholders► What is Driving Timing? Who asked for evaluation?
• Single donor may want for accountability – w/deadlines set• But 1st best is evaluation commissioned by program – with
FULL Governing Body consensus on need & timing► How to take account of past evaluations?
• Take account of whether the past evaluations met all expectations – may want better quality, coverage this time
• Δ in global context may dictate different approach, coverage• Multiple single-donor evals of GRPPs common – avoid
duplication and consider program-level meta-evaluation
7
Readiness Assessment► Status of consensus on need, timing & approach
• Objectives of program and log frame up to date? • New constituencies to involve and include?
► Status/Access to funding• 1st best – program budget for evaluation–but if not pre-
planned –• Can get GB approval of special item – build in time for this• 2nd best – if donor(s) offer to fund, prevent undue influence
► Status of GB & staff expertise on evaluation? Access to support?
► Status of info: (Usual) Baseline/monitoring, program records• Sometimes overlooked – enough activities completed?
► Consider audiences, translation, dissemination needs
8
Consequences of Poor PlanningPoor Planning: w/no compensation for Constraints
Constraints► Not enough activities
completed► Inadequate budget for
evaluation, dissemination
► Monitoring system not in place
► Not taking into account previous evaluations
Consequences► Low yield; limited findings
► Poor quality; compromised credibility; less used, accessible
► Eval. Q’s less clear; new data collection $$; findings limited
► Costly to program; burden on informants; may confuse strategy
Setting Broad Purpose & Design
► Should flow from Situational Analysis• e.g. Accountability for donor; Assess Impact of
Δ’d Context► Ideally Approved by Governing Body► Usually varies by Program Stage or
Maturity
10
New, Young Fine-tune design; confirm results chain; identify main risks; display results to satisfy donors
Evolved, Steady-State
Fill Accountability needs; Δ governance, strategy as needed; broaden lessons& influence
Evolved, but Δ’d context
Assess implications of changed context for strategy, objectives, reach and coverage
Mature Assess likelihood of sustainability of outcomes;Consider devolution
Assigning Roles: Key Players in the Evaluation► Governing Body► Overseers of Evaluation
• Oversight Committee• Standing Technical Expert Panel – predefined roles or
not► Program manager and staff► Independent Evaluation Team (usually consultants)► Others: Keep in mind for consultation, dissemination
• Key partners not represented on GB (donors)• Other Key informants (e.g. past GB members…)• Implementing Agencies (need translation?)• Direct Beneficiaries (need different, briefer
product?)
11
Preferred Roles of GB & Management
12
GB/Oversight Committee ManagementApproves M&E policy Upstream: May draft M&E policyConsiders timing & purpose;Commissions evaluation, Approves roles
Calls for or receives call for evaluation Does situational analysis, proposes Δ’d timing, budget, approach & roles
Approves criteria & process for selection
Proposes criteria &process for team selection
Approves selection Manages selection process; seeks expert input
Oversees evaluation Briefs team, provides documentation, Provides day-to-day admin. support
Receives first draft; may share w/Mgmt
Receives first draft in parallel w/or after GB
Approves final draft Free to comment & correct factual errors
May draft separate GB response
May draft Mgmt response
Approves responsive action Elaborates responsive action
Roles of Others: Participation & Consultation Upstream & Downstream
► Principles/mandate in charter, M&E Policy or legal agreements?• If not, Mgmt should consult GB and determine expectations
& Consider:• Before (design), During (interviews/surveys), After
(review/inform findings)► If no Formal Stakeholder List or Map – Prepare, to
equip evaluators
13
1st best – DESIGN – Global Alliance for Vaccines and Immunization (GAVI). The program decided to hire consultants to conduct a pre-evaluation “scoping Study” to survey a wide range of stakeholders to submit evaluation questions relevant to them. A Steering Committee provided a proposed list of stakeholders to the consultants. The submitted evaluation questions were compiled, consolidated and then sent back to stakeholders for prioritization. The resulting list of priority questions was referenced in the subsequent evaluation TOR/RFP as forming the basis for the evaluation.
2nd best – DURING – Evaluators of the One World Trust worked with the program to develop a matrix with categories of stakeholders in order to define a suitable sampling frame for surveys for the evaluation.
Planning: Design, Budget, Contracting
...Expertise Needed on Evaluation
...Weighing all Options and Responding to GB imperatives► Design must respond to purpose & match complexity of program
plus budget—credibility essential ► If expertise available (Oversight committee, host agency) – can
have confidence in design, budget adequacy and feasibility of work program• TOR can be specific; team selection mainly on qualifications
► If not, several options:• 2-stage evaluation: “evaluability study” or experts draft TOR;• Leave details to evaluators & selection process
– RFP has proponents submit design/work program (sometimes budget)
– Inception report/early consultation on methodology desirable
14
Conclusions & Hopes for the Future
► Ideally, the good practices we highlight will lead to more frequent & earlier attention to the enabling conditions which set the stage for successful GRPP evaluations.
► Where that is not possible, we hope decision-makers will see the benefits of doing a situational analysis to inform early planning and ensure a realistic, consensus approach.
► We further hope that such early planning will increasingly:• Lead to assigned roles in evaluation which ensure organizational
independence & access to expertise at key stages; • Adequately address questions of participation and
dissemination;• And thus lead to higher quality evaluations & better
development.
15
Thank You