Slide 1

E-101 Thursday Section 1 T HURSDAY, 1 ST NOV., 2012 11 12.30 P. M.

Slide 2

Roadmap for today Housekeeping Where are we? What are the goals? Application Feedback sticky notes 2

Slide 3

Housekeeping 3 Paper II is coming up Paper III Dec. 20 th we will start looking at this more closely once Paper II is back

Slide 4

4

Slide 5

Where are we- Assessing the performance of a system Policy and program evaluation Curriculum, Standards & Assessment 5

Slide 6

M&E - 6 One of the most impt. aspects of this course - make sure you walk away with it Resources International Initiative for Impact Evaluation J-PAL UNESCO IIEP UNICEF Innocenti Research Center UNICEF Evaluation Database World Bank Evaluation Database Other classes Haiyan Huas M & E class Tom Kahne - Program Evaluation

Slide 7

Weiss Reading 7 What does she have to say? Summary? What are the tensions that might arise in the evaluation process?

Slide 8

Monitoring & Evaluation 8 There is a difference between M and E! Monitoring: The gathering of evidence to show what progress has been made in the implementation of programs. Focuses on inputs and outputs. Evaluation: Measuring changes in outcomes and evaluating the impact of specific interventions on those outcomes.

Slide 9

Monitoring & Evaluation 9

Slide 10

10

Slide 11

11

Slide 12

Reflections - 12 What are the tensions b/w an insider & outsider role as an evaluator? If time is money, how does one implement process? What are the ways in which impact might be considered? How might cultural differences impact these? Creating local capacity for communities to monitor their own programs & hold states accountable for the delivery of basic services? Dissemination to the beneficiaries?!

Slide 13

Implementation Issues? 13 Political Economy Policy Context Methods

Slide 14

Political Economy 14 What is the policy purpose? Test innovations & defend budget Allocate budget to certain programs Pressure to demonstrate aid effectiveness and scale up Answer electorate Reaching Targets

Slide 15

Policy Context 15 What questions need answering? Who are the stakeholders who want answers? Do they also have the same amount of clout? What policy questions need answers? How much of a change would determine success? What does the government really need, and what will it use?

Slide 16

Methods 16 Operationalizing the outcome Process/outcome indicators Channels of impact theory of change Underlying assumptions [log frame] Who has access to the program? Are they the same as the intended beneficiaries? Data collection time vs. cost Analysis Dissemination

Slide 17

Things to think about 17 What ? Type of information and data to be consolidated How? Procedures and approaches for data collections and analysis Why? Why are we doing this does it support the program/policy When? Freq. of data collection, reporting Who? Responsibilities and capacities of focal points and resource persons

Slide 18

Sector Analysis: What is it? Sector Review: summary of the state of the sector Sector Assessment: success of the sector in meetings certain goals Sector Analysis: (most comprehensive) includes review and assessment (above), and also describes sector needs, constraints, and opportunities for improvement. (Kremmerer, 1994)

Slide 19

Sector Analysis: What is it? Tries to answer questions relating to: External efficiency Internal efficiency Access & equity Administration & supervision Costs & financing (Pigozzi & Cieutat, 1998)

Slide 20

Sector Analysis: What is it? The contents of a sector analysis may include: 1. Synthesis 2. Economic & Financial Analysis 3. History, Structure & Management Capacity 4. Analysis by Sub-Sectors: 5. Special Studies 6. A list of background documents (Pigozzi & Cieutat,1988)

Slide 21

Sector Analysis: Steps I. Initiation II. Formation of the National Steering Committee III. Determination of the scope of the analysis IV. Formation of the subsector teams V. Formation of the technical analysis team VI. Instrumentation and collection of data VII. Preliminary analysis of data VIII. Discussion of preliminary findings IX. Collection of any additional data X. Formulation of draft recommendations XI. Discussion of draft recommendations XII. Revision and prioritization of recommendations. (Kremmerer, 1994)

Slide 22

Opposition sectorsSupport sectorsOpposition sectors External actors Sector positi on Anti-system Legal Opposition Ideological support Core support Ideological support Legal Opposition Anti- system The Government Social sectors Political parties Pressure groups Political Mapping: An Example

Slide 23

Slide 24

Stakeholder Map: An Example

Slide 25

Program Theory Program theoryrefers to the mechanisms that mediate between the delivery (and receipt) of the program and the emergence of the outcome of interest. The operative mechanism of change isnt the program activities per se but the response that the activities generate. - Carol Weiss, Evaluation, p. 57 You will be reading a section of this book for next week.

Slide 26

Activity: Program Theory For each program, there may be multiple pathways to the intended outcome. Source: Weiss, C. (1997). Theory-based evaluation: Past, present, and future. New Directions for Evaluation, (76), 41-55, p. 42.

Slide 27

From Program Theory to Implementation to Evaluation Logical Frameworks can help you to organize: Design Implementation Evaluation Developed by USAID in the 1960s Vary slightly, but always include: 4 x 4 Narrative Summary, Indicators, Means of Verification, and Assumptions

Slide 28

Log Frames 28 In terms of relevance, practice could be the development of common understandings gained via an examination of assumptions and creation of a common language for evaluation - Garaway p.p. 718 Log frame as tool to do examine Relevance: what, where, who Effectiveness: Process & implementation Efficiency: Making the best use of resources Impact: Support the program purpose

Slide 29

How do you develop a logic model? Linear process from beginning to end Move around (iterative) Backwards (identify results then determine the resources/inputs and activities that are required) Can be described as If ___________ then ____________. If we have X resources and do Y activities, then we will have Z Outputs, W Outcomes and T Impact.

Slide 30

Putting It All Together Impact Outcome Output Impact Outcome Output

Slide 31

How to annoy your family with what you learned at HGSE Your Planned Work Your Intended Results Resources/ Inputs ActivitiesOutputsOutcomes Impact Logic Model of a Family Trip Holiday flight schedules Family schedules Holiday weather Your exam/ paper due dates Create family schedule Get holiday flight info Get tickets Arrange ground transport Tickets for all family members Frequent flyer miles used Money saved Ground transport Family members enjoy vacation Continued good family relations

Slide 32

Slide 33

Logical Frameworks Project Structure Objectively Verifiable Indicators Means of Verification Important Assumptions Goal: Purpose: Outputs: Activities: (Inputs)

Slide 34

Logical Frameworks Project Structure Objectively Verifiable Indicators Means of Verification Important Assumptions Goal Purpose Outputs Activities (Inputs) IF AND THEN

Slide 35

Logical Frameworks Project Structure Objectively Verifiable Indicators Means of Verification Important Assumptions Goal: Purpose: To increase the number of primary school graduates. Outputs: Activities: (Inputs)

Slide 36

Logical Frameworks Project Structure Objectively Verifiable Indicators Means of Verification Important Assumptions Goal: Develop human capacity in XX country. Purpose: To increase the number of primary school graduates. Outputs: Activities: (Inputs)

Slide 37

Logical Frameworks Project Structure Objectively Verifiable Indicators Means of Verification Important Assumptions Goal: Develop human capacity in Somalia. Purpose: To increase the number of primary school graduates. Outputs: By abolishing school fees, expect to see more children enrolling in primary Activities: (Inputs)

Slide 38

Logical Frameworks Project Structure Objectively Verifiable Indicators Means of Verification Important Assumptions Goal: Develop human capacity in Somalia. Purpose: To increase the number of primary school graduates. Outputs: By abolishing school fees, expect to see more children enrolling in primary Activities: Abolishing school fees.

Slide 39

Logical Frameworks Project Structure Objectively Verifiable Indicators Means of Verification Important Assumptions Goal: Develop human capacity in Somalia. Purpose: To increase the number of primary school graduates. Outputs: By abolishing school fees, expect to see more children enrolling in primary Activities: Abolishing school fees. All primary schools no longer requires fees.

Slide 40

Logical Frameworks Project Structure Objectively Verifiable Indicators Means of Verification Important Assumptions Goal: Develop human capacity in Somalia. Purpose: To increase the number of primary school graduates. Outputs: By abolishing school fees, expect to see more children enrolling in primary Activities: Abolishing school fees. All primary schools no longer requires fees. Poll families to ask what fees exist.

Slide 41

Logical Frameworks Project Structure Objectively Verifiable Indicators Means of Verification Important Assumptions Goal: Develop human capacity in Somalia. Purpose: To increase the number of primary school graduates. Outputs: By abolishing school fees, expect to see more children enrolling in primary Activities: Abolishing school fees. All primary schools no longer requires fees. Poll families to ask what fees exist. Primary can be funded without fees.