1

May 15, 2023Brandeis University

Heller Graduate SchoolSustainable International Development (SID) Program

Monitoring and Evaluation (M & E)(HS 278f-1 and HS 278f-2, module I)

Spring semester, 2019-Contact information for instructor and grading Teaching Assistant (TA)

Instructor and TA Office ContactInstructor: Ricardo Godoy Heller 153 rgodoy@brandeis.eduTA: Mr. Wei Jia (PhD candidate, IBS) Heller 153 jackyjia@brandeis.edu

Day, time, room of course activities and office hours (by appointment) for TA & instructorActivity Day Time Room

Class:278f-1 Thursdays/a/ 9-11:50am (except 2/21) G2278f-2 Wednesday/a/ 9-11:50am (except 2/20) 163

Office hours:Godoy Wednesdays Noon-2pm

153Thursdays Noon-6pmMondays 10am-2pm

Jia Tuesday 4-5pm 153/a/ If Brandeis cancels classes because of bad weather, we will try to hold the class electronically during class time (9-11:50am), but without the weekly quiz.

Main and specific goals. The main goal of the course is to familiarize students with the principles and best-practices of M & E of development projects. The specific goals are to understand:

Principles of causal inference, Links between an intervention, path variables, and outcomes Methods to evaluate and design a M & E system Trade-offs of different approaches to M & E

Roadmap for the course. As shown in the figure below, M & E is an approach that links three building blocks of development: Results (outcome, output), intervention, and paths linking an intervention to results. The course follows this tripartite division: (a) how to measure and estimate changes in results from an intervention (evaluation), (b) how to measure changes in path variables from an intervention (monitoring), and (c) how to design, monitor, and evaluate an intervention that redresses the hurdles affecting the outcome before an intervention is put in place.

What the course does and does not stress. Because we only have seven weeks in the course, because M & E can be applied to many fields, and because one course cannot do it all, some choices had to be made on what to stress and what not to stress or leave out.

Stressed. The course stresses foundational technical topics in M & E that students are unlikely to encounter in the SID curriculum. The topics include:

Principles of causal inferences Methods of quantitative data collection for M & E Rigor in the design of a M & E system based on observational data (i.e., not based

on randomized-controlled trials (RCT), the subject of Advanced M & E [Spring 2019, module II])

Unstressed. Even though they are part of M & E, the following topics are not stressed or are left out, either because [a] they are skills students can learn outside of class or [b] they are covered in other courses:

Qualitative methods of data collection for M & E (particularly participatory M & E; see section 2.7 of IFAD manual in required readings for class #1)

Data analysis and randomized-controlled trials (discussed briefly in class #6 and fully in Advanced M & E in Spring 2019, module II)

Cost effectiveness and cost/benefit analysis (covered in various courses at Heller) Communicating the results of M & E analysis for impact Ethics of research with human subjects (partially covered in survey design)

The section titled "Optional readings" contains a list of unstressed topics.

Approach. Lectures to cover principles In-class exercises to apply principles Final group project and presentation

Structure of class time (Table 1). Starting with the second class, each class will unfold in the following way:

9:00-9:30. Close-book quiz 9:30-9:45. Explanation of intermediate reports due on the following week 9:45-10:45. Lecture 10:45-11:00. Break 11:00-11:50: Lecture

Required readings. All the required readings for each class are available in LATTE and indicated in Table 1. The main required readings are shown below; some of them are freely available on line (as of January 9, 2019) at the links shown below.

Result : Outcome/output

(Evaluation)Path

(Monitoring)

Intevention (Monitoring)

2

IFAD . Irene Guijt and Jim Woodhill. 2002. Managing for impact in rural development. A guide for project M & E. Rome: International Fund for Agricultural Development (IFAD).

GT . Glennerster, Rachel and Kudzai Takavarasha. 2013. Running randomized evaluations, a practical guide. New Jersey: Princeton University Press, pp. 1-65. A PDF copy is in LATTE. Not available on line.

WB-1 . Shahidur R. Khandker, Gayatri B. Koolwal, and Hussain A. Samad. Handbook on impact evaluation: Quantitative methods and practices. Washington, D.C. The World Bank. The book is also available at: http://documents.worldbank.org/curated/en/650951468335456749/pdf/520990PUB0EPI1101Official0Use0Only1.pdf

WB-2 . Marelize Görgens and Jody Zall Kusek. 2009. Making monitoring and evaluation systems work: A capacity development tool kit. Washington D.C.: The World Bank. The book is also available at: https://www.amazon.com/gp/product/0821381865/ref=pd_sim_14_5?ie=UTF8&pd_rd_i=0821381865&pd_rd_r=KVFF1CBM7H3EVVSW0ZFC&pd_rd_w=vajHA&pd_rd_wg=akXo5&psc=1&refRID=KVFF1CBM7H3EVVSW0ZFC

Class notes from 2018 without suggested solutions are in the course web site.

Optional readings. The textbooks and readings below are included because they cover real-world examples or topics that we do not stress in class, or that we leave out. All the readings are available on line (as of January 9, 2019).

[1] International Federation of Red Cross and Red Crescent Societies (IFRC). 2011. Project/program monitoring and evaluation (M & E) guide. Geneva, Switzerland: IFRC. Link: http://www.ifrc.org/Global/Publications/monitoring/IFRC-ME-Guide-8-2011.pdf

[2] UNDP. 2009. Handbook on planning, monitoring and evaluating for development results. New York: United Nations Development Program. Link:New: http://web.undp.org/evaluation/handbook/documents/english/pme-handbook.pdf [3] Paul J. Gertler, Sebastián Martínez, Patrick Premand, Laura B. Rawlings, and Christel M. J. Vermeersch. 2011. Impact evaluation in practice. Washington, D.C.: The World Bank. Link:https://siteresources.worldbank.org/EXTHDOFFICE/Resources/5485726-1295455628620/Impact_Evaluation_in_Practice.pdf

Grading. The final grade will be based on (a) the best four of five in-class quizzes and (b) a final report and oral presentation on the last day of class (3/6 or 3/7) in which one or more students designs a M & E system with observational data for a project of your choice.

(a) 5 in-class quizzes ( 70% ). There will be five in-class, closed-book quizzes at the start of classes 2-6. You cannot use any written material, telephone, text messaging, or any other physical or electronic aid during the quiz, and you cannot speak with anyone (besides the proctor). Quizzes from 2018 with and without suggested solutions have been posted in LATTE.

You will be graded relative to all students in your class; the ranking and scoring for each of the two classes will be independent. To score each quiz we will compute the score of each student and rank the scores. Each quiz in each of the two classes will be graded in the following way: students who score at the top 16% of the class will receive an A (there is no A+). The 16% next lowest scores will receive an A-, and so on. Students scoring in the bottom 16% of the class will receive a C+. The material covered in each quiz will be cumulative and will include all the lectures and readings up to the day of the quiz. For example, the second quiz on class #3 (January 30-31) will include the lectures, readings, and notes covered in classes 1-2 (January 16-17 and January 23-24), but will also include the readings from class #3. Since the quiz will take

3

place at the start of the class, the quiz on January 30-31 cannot cover the lecture notes of January 30-31 but will cover the readings for that class. You should expect questions and material from previous lectures and quizzes to re-appear in later quizzes.

When computing the final grade from all quizzes for the course in a class, we will drop the quiz with the lowest score and rank students based on the total score of the remaining four quizzes. For the computation of the final score, the score of each quiz will be normalized 0-1 so that the final scores could range from zero (always at the bottom) to 4 (always at the top); this procedure controls for the fact that some quizzes might have fewer or more questions and points than others. Do not worry if you miss a quiz because the quiz you miss will be the quiz we will drop when computing your final grade. There will be no make-up quizzes.

Procedures to request re-scoring of question(s) in a quiz. If, after you received your graded quiz, you feel that one or more questions have been incorrectly scored, you must do the following within one week of receiving the quiz:

[1] Provide a hard-copy written explanation of why you think the question(s) was scored incorrectly.[2] Staple the explanation to the quiz and give it to the TA so we review it.

We will get back to you promptly with the final decision.

(b) Final class project: Presentation and paper (30%) . 30% of your final grade will depend on an oral presentation and a final paper, both due on the last day of class (3/6-3/7). 30% represents a full letter grade. Thus, if your average from your four best quizzes puts you in, say, a B and you complete well the final project, your final grade for the course could rise to an A- (B→B+→A-). Since the highest grade possible in this course is an A, no student can score above an A even if the quality of the final paper suggests otherwise. Appendix A has a description of how the final paper will be graded. Depending on the class, the final paper and presentation are both due on March 6 or 7th. The group presentation will be done during class time, and the final paper should be submitted electronically through LATTE by 5pm, on the same day as the final presentation. See Appendix A for details about the final paper. Details on the oral presentation will be supplied a week before the last class. To help you move along in the final project, in Table 1 and in Appendix A I have identified tasks that will take you through each of the stages to complete the final project. Model student papers from 2018 have been placed in LATTE.

Policy about group composition. Because of the amount of work, it is probably best if students work in a group rather than alone, but you are free to work alone if you prefer. There is no limit to the minimum or maximum size of a group. Four is probably a reasonable number for a group size. If students decide to work in a group, they should select the people in the group during the first week of class. After the first week of class, let the TA know:

The composition of your group (only one person needs to inform the TA of the composition)

If you want to work in a group but have not found one. In this case, the TA will assign the student to one of the smaller groups.

If you prefer to work alone.Thus, by the start of the second class we will know if you are working alone, and, if you

are working in a group, we will know who else is working with you. Because free-riding is common in group work, each assignment turned in should spell out the contribution of each student.

Policy about class cancellations. Because the course takes place during the height of winter, it is not uncommon for Brandeis to cancel classes due to poor weather. If Brandeis cancels any of the classes 2-6, we will do the following:

4

[a] Not have a quiz that week. If this happens you will be graded on the best of the remaining quizzes. For example, if a class is cancelled because of a snow storm on a Wednesday but Brandeis resumes classes the next day, then, in fairness to students in both classes, we will not have a quiz for either of the two classes. This will happen if we do not have a class on Wednesday but have one on Thursday. Alternatively, if there is a class and a quiz on Wednesday, but classes are cancelled the next day, we will drop the Wednesday quiz for the class that took it. This might not be perceived as fair by the class that studied and took the quiz on Wednesday but is done so that the total number of quizzes on which you are evaluated is the same for each of the two classes.

[b] Hold the class through electronic media. Later we will send you details on how to log on to the electronic version of the class.

If the final class on 3/6 or 3/7 is cancelled (class #7), there will be no oral presentations, but you will still be expected to submit the final assignment via LATTE on the date and time indicated in Table 1.

Feedback to students. Each week during classes 2-6 students will receive the suggested solutions to the weekly quiz of that week. In this way, you will be able to gauge how well you are doing compared with your peers. I will also give feedback to groups submitting intermediate reports for the final class project.

The role of the grading Teaching Assistants. [a] To grade and promptly return the weekly quizzes[b] Post the suggested solutions to the weekly quizzes on LATTE [c] Manage the electronic submissions of intermediate or final reports in LATTEto monitor compliance with the deadlines

If you have questions about the readings and lectures, or if you want to discuss your intermediate or final group project see the instructor, not the TA.

Gender perspective. The skills can be used to analyze how development projects affect gender relations. In fact, as you can see in the model student papers, many of the M & E systems design for the course center on how to improve female status.

Students with disabilities. See me if you are a student with a documented disability and wish to request a reasonable accommodation for this class. Brandeis cannot provide reasonable accommodations retroactively.

Policy about academic honesty. Academic integrity is central to the mission of educational excellence at Brandeis University. Each student is expected to turn in work completed independently, except when assignments specifically authorize collaborative effort. It is not acceptable to use the words or ideas of another person without proper acknowledgement of that source. This means that you must use footnotes and quotation marks to indicate the source of any phrases, sentences, paragraphs or ideas found in published volumes, on the internet, or created by another student. Violations of University policy on academic integrity, described in Section 3 of Rights and Responsibilities, may result in failure in the course or on the assignment, and could end in suspension from the University. If you are in doubt about the instructions for any assignment in this course, you must ask for clarification.

5

6

Table 1: Summary of scheduleClass Date/

a/Quiz Topic for class Readings/c/ Required & optional

reports /b/

1 1/161/17

No INTRODUCTION: [1] Course boundaries, [2] why M & E, [3] attributes of good treatment, path, & outcome, [4] linking treatment, path, and outcome, [5] building blocks: treatment/control, before/after, results/path/intervention.

IFAD: Sec. 1-3 + Gugerty-Karlan

None

2 1/231/24

1 CAUSAL INFERENCE: [1] theory of change or results chain -- defining results and indicators, [2] heterogeneous treatment effects, [3] observational versus randomized- controlled M & E, [4] selection bias. Intermediate report #1 due electronically in LATTE by 9am

WB-1: Chapter 2 #1 Introduction[required]

3 1/301/31

2 MEASUREMENT-I: Measuring treatment, path, and outcome: what, when, how, and where

IFAD: Sec. 4-5 #2 Causal inference[optional]

4 2/62/7

3 MEASUREMENT-II: Measuring treatment, path, and outcome: what, when, how, and where

5 2/132/14

4 SAMPLING AND DATA MANAGEMENT: [1] team composition, [2] training, [3] sample size, [4] monitoring, and [5] data management and documentation

IFAD: Sec. 6-7 #3 Measurement I and II [optional]

2/20-2/21. No class6 2/27

2/285 RANDOMIZED-CONTROLLED TRIALS (RCT): Introduction to RCT and notes on

final group presentationGT: pp. 1-65 #4 Sampling and data

management [optional]

7 3/63/7

No Group presentations and final paper due. Final paper due electronically in LATTE by 5pm.

None Final report[required]

/a/ First date is for the Wednesday class and the second date if for the Thursday class./b/ All intermediate submissions are due by 9am in LATTE on the class day; e.g., intermediate report #2 is dues at 9am on 1/30 or 1/31. See Appendix A for details of intermediate and final report./c/ Readings for each class are in the folder in LATTE for the class.

7

Appendix A: Final class project

Motivation. Alone or in groups (ideally of four people), students will design a M & E system based on observational data for a project of your choice. By observational data I mean a project where the intervention has not been assigned at random; the development organization decided where to intervene and who to give the intervention, and people/households/communities decided whether to take part in the project and evaluation. Subject to budget constraints, you give the treatment to any entity that wants to participate. This contrasts with an RCT where the organization randomly selects entities (e.g., people, villages) to benefit from the treatment. Both RCT and observational evaluations rely on quantitative or qualitative data for their evaluation; the difference is that in an RCT the intervention is assigned to entities chosen at random by the researcher or organization. Your project should not be an RCT. The groups should be formed soon after the first class. If a student wants to work in a group but cannot find a group, they should contact the TA so that the TA can assign them to a group.

For simplicity and to avoid conflict of interest between implementer and evaluator, assume that your group has been hired to develop a system of M & E for a project that others have designed; you might have had an input into the design of the project, but your role is divorced from having to implement the project.

Tips for group work. Group work matters since it is part of professional development. Many SID students form groups that work well, but there are also perennial complaints about group work at SID. Except for the last item in the list below, here are some tips from Randy Pausch. 2008. The last lecture. New York: Hyperion, pp. 142-144:

Sit together in class. Some of the in-class problems of this course will require group work, so by sitting and working together on the in-class problems you will have more opportunities to interact with the people in your team.

Try for optimal meeting conditions (e.g., lunch) for the group project Let everyone talk and don’t finish another person’s sentence. Check egos at the door. Praise each other. Don’t free ride on the group.

Intermediate reports. To facilitate progress in the final project, I have spelled out intermediate weekly tasks below, which the group might want to complete. Except for intermediate report #1 (due electronically in LATTE by 9am of 1/23-1/24 ), all other intermediate reports are optional but are designed to ensure you work on your M & E system gradually throughout the module and that you receive feedback if you want it. Intermediate reports are due by 9am on the day of the class (see footnotes to Table 1).

Reducing confusion on intermediate reports. To reduce confusion on the intermediate reports, I have done two things: (a) for each intermediate report I have written a section on topics that might be confusing and (b) after each weekly quiz I will devote 15 minutes to explain the intermediate report due next week. In this way students will have a clear idea of what to do each week as they work toward the final assignment.

Format of intermediate reports and final report. Submit intermediate reports in Word (not PDF), font 11, 1 doubled-spaced page maximum for each intermediate report). Any submission needs to have a cover page with the following included

[a] Name of project[b] Name of students in the group listed in alphabetical order by surname

[c] Next to each name indicate in 1-2 lines the chores the student performed for the report you are submitting. This last requirement is placed to redress free-riding and the grievances caused by free-riding. If everyone contributed equally to all tasks, just say so (e.g., “All group members contributed to all the tasks in this submission”, but if contributions differed, then clarify in the cover page (e.g., “Smith worked on ** and ^^^; Jones worked on xxx and yyy”).

In addition, if you decide to submit an intermediate report, you should paste all earlier intermediate reports with my comments so that I can see how [1] the most recent report builds on what you have done and [2] you have addressed my comments on earlier reports. Do not submit an intermediate report unless you have addressed my comments/questions. This does not mean you have to agree with my comments, but you need to tell me how you addressed my concerns so I don’t waste time repeating what I said before. Here is an example:

Concern #1: “It is unreasonable to propose taking three baseline measures because it will be too costly”. Response: We feel that three measures are reasonable because an NGO has agreed to do three surveys at no cost to us; also, we feel there will be too much measurement error and insufficient variance if we rely on only one measure.

I wrote the part in red and you write the part in blue, but you let me know exactly what I said and exactly how you responded.

Potentially confusing: Mechanically how do we deal with previous comments in subsequent submissions? The simplest way might be to use track changes in Word to indicate how you deal with my comments or use the “Comments” box in word.

Content of intermediate reports. The first intermediate report is designed to get you to think broadly about an intervention and the theory of change linking results (outcome, outputs, total impact) with the intervention and paths. The first intermediate report is the only intermediate report that is required. The other reports are there to add specificity to the different parts of your M & E system and are optional.

Feedback. I will send comment to groups that decide to submit intermediate report. Instructor’s office hours are ideally suited to discuss intermediate reports. All groups will receive feedback on their entire project once they submit the final assignment on the last day of class.

Intermediate and final report. The final report should be a compilation of the four edited/improved intermediate reports. The final report should contain no more than 4 pages (excluding the cover page). The final report should address the following five topics (1 double-spaced page for the first three topics, with tables, graphs, and appendixes excluded from the page limit):

8

[1] Introduction (Due 1/23-1/24): Explain the motivation for the project and the theory of change linking results (outcome[s], outputs, total impact) and intervention[s]. Topics covered should include the following: The motivation for the intervention, the problem address by the intervention, the importance of the problem, the intervention, the theory of change (logic) linking the intervention with the results (outcome, output, total impact), and the M & E strategy (1-page limit). Recall that this intermediate report is the only intermediate report that is required. This report is due electronically in LATTE by 9am of 1/23-1/24. An example follows:

Motivation: Our NGO wants to redress high (50%) primary-school drop-out rates in southern rural Malawi by comparing the standard approach (motivational workshops for parents) against a new approach where parents/children get paid $100/year if a child attends school at least 80% of school days/year.

Problem addressed by intervention (results [outcome, output], intervention, and paths): Conditional cash payments (CCT) to a child or to their parents if the child attends school (intervention) directly tries to overcome two hurdles for primary-school drop-out: school expenditures and opportunity cost of children (paths). Enrollment from payments will enhance the value parents attach to school. Thus:

Results:Outcome: Child school enrollment is expected to riseOutput: Increased value placed by parents on child schooling

Paths: School expenditures and opportunity cost of children.Importance of problem: Primary-school completion correlates with better adult income and health.Theory of change: Children drop out because parents do not have money to pay for school

supplies and because children perform valuable chores on the farm and household, so parents are reluctant to let their children go to school. Payments will ease these two hurdles and will be associated with higher expenditures in school supplies and a higher probability of hiring replacement workers for children who go to school. Furthermore, children who attend school will speak about their school experience with parents and such conversations will raise the value parents attach to schooling their children.

M & E strategy: The observational evaluation will last 2 years and will be done in a total of 20 primary schools (grades 1-5), all of which will receive motivational workshops, but 10 of which will, in addition, receive payments at the end of the school year if a child attends school for at least 80% of school days. We will measure # of school days missed (outcome) and quarterly expenditures in school supplies and hired workers by the households (paths) and estimate changes in school attendance before and after the intervention.

Potentially confusing : [a] Does the problem have to be real? Yes. Do some quick background research on the topic of interest and make sure it has obvious practical importance. [b] How specific do we need to be? Very. Aim for specificity in: geographical location (e.g., southern rural Malawi), demographic group (children in grades 1-5), entities (schools and children), and results (school attendance, parental valuation of school). [c] Does there need to be evidence linking the path variables with the results? No; the path variables or theory of change should be logical. E.g., you hypothesize that children drop out of primary school because households don’t have money for school supplies and children need to work in the house/farm (the two path variables). Therefore, your intervention directly tries to overcome these two hurdles through a cash infusion to the child/household. Seems logical, though of course this might not be the case when you carry out the intervention. [d] Do we have to use a randomized controlled trial (RCT)? No; in fact, the evaluation should rely on observational data since such evaluations are more common. Evaluations using RCT will be discussed in class #6 and in Advanced M & E. In this example, your M & E organization decides what schools to include in the evaluation.[e] You are not responsible for the costs of the project – your role centers on evaluating and monitoring the project. The costs of the project are shouldered by other institutions.

9

[2] Causal inference (Due 1/30-1/31). The topics you need to address for this step include [i] how you plan to deal with omitted-variable biases (OVB), particularly selection bias, [ii] what type of differential effects do you expect in the outcome in relation to sub-groups of the sample considered for the evaluation, and [iii] implications of heterogeneous treatment effects on sample size (1-page). An example follows:

Biases in sample. Organizational. Because our NGO tries to limit costs we tend to work in schools near

public roads. To overcome this bias, for this evaluation our NGO will purposefully select half the sample of villages that are no more than 10km away from the nearest paved road.

Self-selection of participants. We expect that better off-households or households where parents have more schooling and fluency in the national language will be more likely to want to participate in the study. Therefore, we will try to recruit parents who (i) do not send their children to school, (ii) do not speak the national language and are unschooled, and (iii) are at the bottom 20% of the village income distribution.

Differential effects. Besides the parental attributes just noted (e.g., lack of fluency in national language, low income), we expect payments to have larger positive impacts on older boys than on younger brothers or than any sister because in the study area there is a pro-male bias in parental school investments.

Implications of differential effects for sample size. Because we expect the impact of the intervention to differ by: (a) the age/sex of children, (b) socioeconomic status and education of parents, and (c) road proximity, we will make sure that our sample is equally divided between schools that lie near and far from roads, and that our sample contains an equal number of girls and boys, younger and older siblings, and households in the top and bottom of the village income distribution.

Potentially confusing:[a] Do we need to do formal power analysis to determine the sample size? No; you just need to convey that how you deal with the theory of change or differential treatment effects – the paths – has implications for how you deal with the composition of the sample for the evaluation. [b] What is the difference between path variables or theory of change and heterogeneous treatment effects? The two concepts overlap, but one has to do with mechanisms and the other with (sub) groups, but in either case, you would expect result to differ in relation to path variables or groups. Example: lack of school supplies and opportunity cost of children are immediate causes for primary school drop-out. Thus, infusing money will likely have more of an impact on households that lack money or that rely on children for household chores. These are mechanisms (paths) linking the intervention to the outcome. However, given this general theory of change, you also expect there to be further interaction effects with sub-groups in your sample; thus, payments might increase the enrollment of all children, but have a larger effect on the enrollment of girls than boys, or on younger siblings more than on older siblings. These interaction effects are of secondary importance -- but important nonetheless -- to the theory of change.

10

11

[3] Measurement I and II (Due 2/13-2/14). Measuring the treatment, paths, and outcome(s): what, when, how, and who (or where; entity). The simplest and clearest way to accomplish this intermediate report would be to set up a matrix in which you briefly provide details in the cells. You can use an appendix to amplify the points in the cells if you think the vignette in the cell is unclear. The parts in red are required; the parts in blue are optional (1-page limit, excluding appendix). An example follows using only one level of results (outcome) and not output. Rows 2-4 refer to row #1.

Four Keys

Level of result: Outcomes Paths

TreatmentPrivate (own child) Social (externalities; others) #1 #2

Intended Unintended Intended Unintended1.What School

attendance↑GPA ↑ Theft↓ Jealousy↑ School

expenditures↑Hired

laborer↑$ disbursed to households to increase incentives for child school attendance

2.When End of school year

End of school year

End of year

Quarterly Quarterly Quarterly Payment made at the end of the school year

3. How School records on attendance

School records on grades

Survey Focus groups Survey Survey On site verification or survey of parents/children if they received $

4. Who Child Child Neighbors Siblings who did not receive payment

Parent Parent Child or parent receiving transfer

Potentially confusing: [a] For outcomes, do we need to have intended and unintended private and social consequences? No; the only outcomes you need are the intended private consequences of the treatment. Include blue columns if you think they are real and obvious, otherwise ignore. [b] How many paths do we need? In the case study used as an example, I have identified 2 paths; hence two paths go into the matrix. The # of paths you include should flow from your theory of change. Limit the # of paths to 2-3 of the main mechanisms. [c] Do I need to have one method for collecting all data? No; in the example above, you will rely on school records for attendance and GPA, on surveys of parents to measure paths, and on focus groups of siblings who did not receive payments to measure negative externalities. [d] What does entity mean? Entity refers to the unit or level at which you assign the treatment or at which you will measure the path or outcome. In the example above, the child or parent receives the transfer, so they are the entities to measure the treatment, but we only measure the outcome (school attendance) among children grades 1-5 so children are the relevant entity to measure the private, intended outcomes. [e] How do we see the impact of the project on outputs rather than outcomes? Suppose that one output of the project was greater parent appreciation for the value of school. Then you might have a matrix, as follow:

Four keys Level of results: Output Path Treatment1.What Parental valuation of school for child Child speaks at home about experiences in school $ disbursed2.When End of school year Quarterly3.How Survey where each parent asked separately4.Entity Mother and father

12

[4] Sampling and data management (Due 2/27-2/28). (a) Team composition: how many supervisors and surveyors will you need and what will

be their responsibilities. (In real life you would write formal terms of reference for them; you could do this and include the information in an appendix).

(b) Training of surveyors: who will train surveyors, how will you know when they are competent to be released to the field on their own, and how will you end the contract of an under-performing surveyor? This section should also address how you will audit data collection for quality.

(c) Sample size: Include the sample size for different entities (e.g., villages, households, people) and then provide estimates of expected attrition and additions and how the sample size might have to be adjusted by losses in the sample. Explain sample size in relation to expected heterogeneous treatment effects and plans to deal with attrition and additions

(c) Monitoring: How will you monitor the integrity of the intervention and paths? E.g., how do you ensure that the designated person is attending the workshop?

(d) Data management and documentation; Explain who errors will be detected and corrected after the survey and how will the documentation be made available to the public. (1-page). Again, a matrix will help. An example of a matrix is included in the next page.Potentially confusing:[a] Should we include a Gannt chart as part of implementation? Yes; a Gantt chart clarifies how the proposed M & E work will unfold over time. We will not cover the making of a Gantt chart in class, but two challenges to keep in mind include: (a) the time units serving as columns (avoid too fine-grained sub-divisions unless necessary) and (b) how to organize the rows by meaningful categories. The row headings should link clearly with the topics you covered, so in this example, things like: Treatment, paths, outcome would be natural titles for rows. Perhaps the three could fall under the macro category of “The project”. Then you could have different rows for other macro-categories like administration, e.g., ‘personnel, data management’, etc. The Gantt chart should reinforce and clarify the table below.[b] How do we estimate attrition rates? If you search in Web of Science under “sample attrition AND region” you will likely come up with studies that have estimates of sample attrition in longitudinal studies in your region. If there isn’t anything for your region, then try searching using bigger categories (e.g., "sample attrition AND Africa" instead of “sample attrition AND South Africa”).[c] Should we include time and activities for write-up and dissemination? No. In practice, you would need to do this, but in this course, we will ignore these activities since it would take us too far afield, important though these topics may be. Also, recall that these topics are ones you are likely to master on your own, outside of the course.[d] The table that follows is an example; do not copy and paste it as part of your report!Focus on the topics of the table (e.g., Team composition) and then decide on sub-categories.

13

Item Approach Person responsible Comment or explanationTeam composition

Composition 2 teams, 3 surveyors.teamSupervision 1 field manager

TrainingTraining 3 weeks before start of M & E Field & data

managersThis will be done in the first 3 weeks of the project. The M & E will be headed by one field manager. Surveyors will be given 4 surveys to do and manager will do inter-coder testReliability Kaplan inter-coder test

Errors Random sample of surveys & surveyors

Random field visits by manager to ensure surveyors are following data collection instructions. One-on-one discussion with surveyors who are committing errors. Supervisor will determine if labor contract with surveyor needs to be ended based on quality of surveys; decisions based on case-by-caseSample size

Gross sample 20 schools*5grade/school=100 classes receive treatment~40 children/class* 100 classes =4,000 children~total # households with school-aged children=1500

Sample size of children in school and households come from xxxx (reference provided by students). In this row make it clear how many # entities receive the treatment and for how many entities you measure outcomes E.g., payments made in 20 schools, 5 grades/school so total of 100 classes (another 100 classes in 20 different schools used as controls). A total of 4000 children (1500 households) will receive payment, and another 4000 children (1500 households) will serve as control.

Expected attrition -20% children by end of 2-year evaluation

Based on literature review (reference provided by students)

Deal with attrition From neighbors find out why subjects left; quarterly surveys; track down attritersExpected addition +5% of baseline sample Code separately; exclude child if they can't complete 80% of school days (e.g., join school mid-

way through school)Deal with additions Include if they arrive <FebruaryMonitoring

Path 1 (expenditure)

Quarterly household surveys Survey team under supervision of field and data managersPath 2 (hires)

Intervention (payment)

Surveyors present at moment of transfer

Survey team Coordinate with teachers to review record of children who receive payments & assign surveyors to be present at some of the schools when payments are made

Data management and documentationData entry Done in iPad Individual surveyors Data entered directly into pre-programmed iPad as M & E unfolds

Data cleaning Done at end of each year Computer analyst After y1, 2-day seminar with surveyors to review coding problems. Manager will write codes to label, recode, and define variable needing greater clarification

Re-coding The 4 activities will be done during the 3 months following the end of data collection. Data dictionary will be in English and the local language and will be prepared by the manager and will be put in the project web site

Merging/appending At end of projectDocumentation

14

[5] Final report. The final report should join the 4 intermediate reports, organized in the order presented above. In addition, the four pages in Word should contain one cover page and, if you use references, a list of the references used at the end of the report. The list of references can be included as a self-contained separate page and is not part of the maximum number of pages allowed. Details of the cover page and citations follow:

Cover pageThe cover page should have the following items; sections in brown are clarifications or are given as examples and should be replaced with information from your group:[i] Title of project: [Be specific about place, cohort, and any other detail. The more details in the title the better][ii] Authors: [List the students in alphabetical order by surname, so for example:Mary BennettJohn SmithJosé UrquiolaRebecca Zhang][iii] Allocation of tasks:[For each student, indicate the specifics of what the students did for the entire project, soBennett: Took the lead and wrote intermediate report #3 Smith: Took the lead and wrote intermediate report #2Urquiola: Helped collect background information for intermediate reports 3-4Zhang: Edited all reports and took the lead on intermediate report #1.][iv] Work disclosure statement. The students listed below certify that they have read the final document before submission for content, spelling, grammar, formatting issues (e.g., font size, page length numbering), and agree that the submission conforms to the requirements set out in the syllabus. The cover page you should include the above statement, followed by the signature of each students: Mary Bennett: ________________ John Smith: __________________ José Urquiola: ________________ Rebecca Zhang: _______________ [v] By not signing below we agree that this paper can be used in other years as a model student paper:[As long as one student signs in section [v] I will assume that we can show the paper to class taking the course in other years][vi] 3 shortcomings of your M & E system. In lieu of an abstract, include a 250-word summary of the three main shortcoming of your M & E.

References cited [include as a separate page; no page limit][a] Follow APA style and only include citations used in the report and tablesPotentially confusing:[i] How do I cite sources on the web? Follow APA style and include web address and date when the reference was downloaded. [ii] Avoid orphans and unreferenced notes: Go through the text and make sure that every reference in the text and tables is documented in the “References cited” section. Avoid references in the section "reference cited" that do not appear in the text.

Additional materialAdditional tables, graphs, and material can be included under appendixes and mentioned in the text (i.e., no orphan appendix). Additional material is optional and should not be used unless necessary. Potentially confusing:

[a] How do I cross link additional material with the text? Avoid orphan appendixes or tables. Be specific, so "In Appendix 3, Table A, we can see that .....", or "For further details, see Appendix 2, Table C". Do not make the reader work hard at linking the text with the appendixes.

15

Summary of final assignment & grading criteria for final assignment

Intermediate reports

Final assignment

Page length of submission (1 doubled-spaced Word page, font 11, except where noted). Excludes supplementary material

Due date

1 (required) Not applicable

1 page (due electronically in LATTE by 9am 1/23-1/24

2 (optional) 1 page 1/30-1/31

3 (optional) 1 page 2/13-2/14

4 (optional) 1 page; smaller font size for table if needed 2/27-2/28

Final report(required)

Compilation of 4 updated/edited intermediate reports + [a] 1 cover page + [b] references cited (no page limit) +

supplementary material (optional). Due electronically in LATTE by 5pm

3/6-3/7-

Grading criteriaRecall that full submission of the final report can raise your final quiz-based grade by one full-letter grade. We use different criteria to assess if your grade will change

To earn a half-letter grade, your group must do all the following* Submit the first intermediate report on time (see table above for exact timing), following the guidelines indicated,* Give an oral presentation of the report (weather permitting), and* Present a final report without any technical blemishes, such as

* Grammatical mistakes or typos* Failure to follow page length limits and format indicated above* Late submission

To earn an additional half-letter grade, your group must:* Address all the substantive concerns for each of the intermediate tasks

16