Annex 2 Sample Terms of Reference - World...
Transcript of Annex 2 Sample Terms of Reference - World...
Annex 2
Sample Terms of Reference
Example I: The Uganda Nutrition and EarlyChildhood Development Project
Terms of Reference for Consulting Firm to Assist inthe Project Evaluation
I. Background
The Government of Uganda has applied for a credit from theInternational Development Association toward the cost of a Nutritionand Early Childhood Project. The project focuses on improving the qual-ity of life of children under six years of age and building the capacity offamilies and communities to care for children. Specifically, the project willaim at achieving early child development through improving the nutri-tion, health, psychosocial, and cognitive status of children under six yearsof age in Uganda.
II. Rationale for Investing in Early ChildhoodDevelopment
Investing in early childhood development (ECD) has tangible benefits notonly for the children and parents but also for entire communities and thecountry. Rapid physical growth and mental development occur duringinfancy and early childhood; at two years of age, a child’s brain is nearlyfully grown. Cognitive abilities are also developed to a large extent byfour years of age. Adequate physical and mental growth and developmentduring early childhood enhance school readiness, improve school reten-tion, and contribute to human capital dependency. Children from disad-vantaged backgrounds can particularly benefit from early child care, thusbridging the gaps and inequalities associated with poverty.
Good health and nutrition are crucial, as is mental stimulation, if thechild is to develop secure conceptual structures in later life. The synergybetween nutrition, health, and mental stimulation is so crucial that tangi-ble positive effects on child growth and development can only beachieved through an integrated approach.
169
02-Evaluating Impacts 5/23/00 5:42 AM Page 169
III. Project Objectives and Strategies
The development objective of the project is to improve growth anddevelopment of children under six years of age in terms of nutrition,health, psychosocial, and cognitive aspects. The achievement of theseobjectives at the end of the five-year implementation period will be mea-sured by the following markers: (a) reduced prevalence of underweightpreschool children by one-third of the 1995 levels in the project districts;(b) reduced prevalence of stunting on entry into primary schools by one-fourth of the 1995 levels in the project districts, (c) improved children’spsychosocial and cognitive development, (d) reduced repetition anddropout rates at the lower primary school level, and (e) development ofentrepreneurship skills and economic empowerment of mothers andcaregivers.
The project supports the Ugandan National Program of Action forChildren and the Poverty Eradication Action Plan. The project particu-larly enhances school readiness of young children and thus contributestoward reaching the goal of universal primary education. The main pro-ject strategy is to enhance the capacity of families and communities totake better care of preschool-age children (zero to six years) throughenhancing knowledge on child growth and development, parenting,nutrition and health care, and income-generating activities for women.
IV. Project Approach
The project is a process-driven, locally prioritized program rather than ablueprint package. Inputs are to be phased into communities as a resultof a participatory planning process to ensure ownership and sustainabil-ity. The program will involve collaboration between government andnongovernment entities, including local and international nongovern-mental organizations (NGOs), and communities. As a multisectoral pro-gram involving health, nutrition, early childhood education, child care,savings and income generation, the approach will involve linking variousgovernment departments and nongovernment entities to provide a com-prehensive service directed toward the development of children. The pro-ject will support a range of options—a program menu—relating to theneeds of preschool children and their families.
V. Project Components
Project Component 1—Integrated Community Child CareInterventions. This component supports the government’s goals (a) toimprove parental awareness on major aspects of child care, growth, and
170 EVALUATING THE IMPACT OF DEVELOPMENT PROJECTS ON POVERTY
02-Evaluating Impacts 5/23/00 5:42 AM Page 170
development through parental education, child growth monitoring andpromotion, training, and sensitization; and (b) to empower communitiesto support child development programs through capacity building,through skills for income generation, and through support grants. Theobjective is to reduce malnutrition (low weight for age) of children by athird at the end of the five-year period in the project districts andincrease readiness of children for primary schooling and thereby con-tribute to the drive for universal primary education. The governmentplan is to eventually cover all districts; however, interventions in thisphase will be implemented in 25 districts chosen by the governmentbased on the level of malnutrition, infant mortality, and rate of primaryschool enrollment. The project includes the following interrelated inter-ventions:
(a) Parental Education. This subcomponent will increase parents’ andcaregivers’ understanding of major aspects of child care, growth, anddevelopment, including child nutrition, health, and cognitive and psy-chosocial development. A range of related competencies will be strength-ened in parents. Building parental skills and knowledge will in turnimprove the health, psychosocial development, and well-being of chil-dren and, ultimately, their receptiveness to education at the primarylevel. The program will mobilize groups of mothers (and parents) at thecommunity level, supported by project materials in local languages, tech-nical supervision, and communications. Simplified learning materials foradults with low literacy have been tested successfully in Uganda.Emphasis will be on the enhancement of child care practices that promoteproper growth and development of children, including childhood nutri-tion and health (exclusive breastfeeding and appropriate weaning prac-tices—particularly the period of introduction of weaning foods, as well asthe type of foods given, and food preparation, child growth promotion,and deworming), psychosocial development, cognitive stimulation andsocial support, and hygiene and improved home health practices.
The above interventions will be strengthened and supported by anoutreach activity (children’s day) organized at the parish level to enablecommunities to access a number of child-related services by means ofone-stop shopping. A study of the impact of providing the anathelminthalbendazole to young children in selected parishes will also be conduct-ed in the course of parish-based child days and will measure the effect ofevery-six-months treatments on weight gain.
(b) Community Capacity Building and Empowerment for Child Care.This subcomponent comprises two interrelated activities: (a) communitycapacity building conducted through community planning and sensitiza-tion workshops, and (b) training in entrepreneurship to increase incomesof mothers and caregivers.
171SAMPLE TERMS OF REFERENCE
02-Evaluating Impacts 5/23/00 5:42 AM Page 171
Project Component 2—Community Support Grants for ChildDevelopment. Two types of grants would be available to communities:
(a) Community Support Grants—grants to communities offered on thebasis of matching contributions from communities. These grants and con-tributions from communities will cover activities designed to supportinterventions for child development that fall within the guidelines andmenu contained in the project implementation manual. To qualify for thisgrant, communities will provide counterpart contributions, which maybe in the form of goods, works, or services. Examples of the uses of suchgrants are construction and operation of community child care centers,home-based child care centers, or the production and marketing of wean-ing foods. The support grants component will be implemented in thesame 25 districts included in component 1.
(b) Innovation Grants—grants made available to communities toaddress child-related problems. The innovation grant will aid in imple-menting interventions outside the menu of interventions described bythe community support grants (a) above. As the term implies, the “inno-vation” fund will be used to support communities at different levels inimplementing “innovative ideas” on improving the lives of childrenwithin their communities. The innovation grants will be accessed bycommunities in the same manner as the community support grants: thatis, proposals will be prepared by communities following a participatoryplanning exercise, will then be screened by a subcounty committee, andforwarded for funding by the project.
Project Component 3—National Support Program for ChildDevelopment. This component consists of central program activities andpolicy initiatives designed to support the district-level programs in com-ponents 1 and 2 and provide quality assurance for the front-line projectactivities at the community level. This component includes (a) programmonitoring and evaluation, (b) support for prevention of micronutrientdeficiencies, (c) ECD curriculum development, (d) training of trainers forECD, and (e) information, education, and communications.
VI. Implementation Arrangements
The implementation of the project is the responsibility of the govern-ment of Uganda assisted by nongovernmental organizations within thedecentralization framework and devolution of powers to lower levels asstipulated in national policies. The community (LC-1) is the unit of oper-ation for service delivery, although the coordination structure will alsoinvolve the parish (LC-2), the subcounty (LC-3), and the district (LC-5)levels.
172 EVALUATING THE IMPACT OF DEVELOPMENT PROJECTS ON POVERTY
02-Evaluating Impacts 5/23/00 5:42 AM Page 172
In addition, the project hopes to use stakeholder sensitization and con-sultations, community mobilization, participatory community planning,capacity building for sustainability at all levels, together with strength-ening of complementarity with existing national programs and struc-tures. Existing political and institutional structures should be made use ofin a multisectoral manner. Transparency and accountability should alsobe ensured at all levels.
VII. Project Coordination
National. A project steering committee composed of line ministries,donors, and NGO representatives will be responsible for overall guid-ance of project implementation.
The committee, to be headed by the permanent secretary in theMinistry of Finance, Planning, and Economic Development will provideguidance to the project on policy issues and review and approve the qual-ity and efficiency of implementation. The project steering committee willalso make suggestions to improve the district annual budget and workplans for the project.
A small project coordinating office (PCO), composed of a coordina-tor, a deputy coordinator, a qualified accountant, and a small supportstaff, will be based in the social services sector of the Ministry ofFinance, Planning, and Economic Development and will take responsi-bility for the day-to-day coordination of project activities at the nation-al level.
District. An existing multisectoral committee—the district coordinatingcommittee (DCC)—will be identified by the Chief Administrative Officer(CAO) to take on the responsibility of coordinating the project at the dis-trict level. The CAO will identify a focal person from among the govern-ment officers who will coordinate NGO-related and other activities in theproject. The lead NGO will be included as a member of the DCC. In dis-tricts where no NGO with adequate or appropriate capacity and skillbase can be identified or strengthened to take over as the lead NGO, theimplementation will be through the district administration.
Subcounty. An existing sectoral committee similar to the one at the dis-trict level will be responsible for coordinating the project activities at thesubcounty level. This subcountry coordination committee will also facil-itate linkages between existing structures and those of the project, andalong with the lead NGO for the district, approve the subcountyNGO/CBOs’ (Community-Based Organizations) annual work plans andfunding requirements for the project in the subcounty.
173SAMPLE TERMS OF REFERENCE
02-Evaluating Impacts 5/23/00 5:42 AM Page 173
VIII. Project Impact Evaluation
The government of Uganda is interested in assessing the impact of vari-ous aspects of the project in order to ascertain its effectiveness and toguide the design of further NECD projects. Moreover, as the World Bankconsiders this project to potentially inform other countries regardingNECD services, it has included the project in a three-country evaluationof ECD programs and will provide technical assistance on a grant basisto the PCO to assist specific research activities. In particular, two studiesto evaluate the impact of specific project interventions will be undertak-en as part of the overall project:
• Parish Child Health Day Study for assessing the coverage ofanthelmintic treatments given at parish-level child health days andtheir impact on the weight gain of children under age six by using arandomized experimental design.
• Survey research using baseline and resurvey methodology for assess-ing (a) the impact of anthelmintic treatments and of overall projectactivities on the cognitive development, health, and nutrition of chil-dren under six years of age; (b) the impact of the caregiver educationcomponent and mass media communication campaign in the knowl-edge, attitude, and child-rearing practices of the principal caregivers;and (c) the impact of grass-roots management training, income-gener-ating activities and credit savings group formation, and provision ofcommunity grants in household and community welfare.
The selected firm will provide technical and logistical support for theabove studies and will be invited to participate as local research imple-menters in the design, data collection, and analysis necessary to completethe two studies of impact assessment. This firm will be the primary coun-terpart of the PCO, local researchers, and the researchers from the WorldBank and the University of Oxford who will be undertaking the impactassessment.
IX. Overview of Studies
Study One: Impact of Deworming at Parish Child Days. There havebeen a number of studies indicating the impact of treating school-agedchildren with anthelmintic medicine. However, there is only one large-scale, randomized trial that shows a large effect on weight gain for pre-school-aged children. This has raised the question of whether such aneffect could be achieved in African children. Thus, the NECD project willinclude a randomized study of the impact of providing the deworming
174 EVALUATING THE IMPACT OF DEVELOPMENT PROJECTS ON POVERTY
02-Evaluating Impacts 5/23/00 5:42 AM Page 174
agent, anthelminth albendazole, to young children in 25 selected parish-es in the course of parish-based child days and to measure the effect ofevery-six-months treatments on weight gain. Data will be collected fromthese parishes as well as 25 control groups that will also organize childhealth days but will not administer albendazole on a routine basis. If theanthelmintic treatments are delivered successfully and are shown to havebeneficial effects on Ugandan children, then the program of anthelmintictreatment may be recommended for all districts.
Because this is a scientific controlled trial, the selection of parishes thatwill be asked to administer albendazole will be undertaken by the PCOfrom a list of parishes where child days will be organized (this list will beprovided by the NGOs working in the districts). The PCO will also selectparishes that will serve as the control group. This experimental design iskey to a successful evaluation.
The firm will ensure that the local NGOs responsible for the organiza-tion of the child health days in the parishes are aware of the rationale forthe experimental design and that they comply with the strategy. Eachchild aged 12 months or older and under 6 who attends the fair in the 25designated parishes will be given a single 400-milligram tablet of chew-able, proprietary albendazole. The albendazole will be administeredevery six months; in the event that the NGOs choose to organize childdays on a more frequent basis, the anthelmintic will still be administeredon a six-month schedule and not more often.
Children in parishes where albendazole is administered as well as chil-dren in the 25 designated control parishes will be weighed at each childday, and their weights will be recorded both on their own health card andon the community register. Children who are too small to stand on thescale unaided will be weighed in their mother’s arms after the scale hasbeen set to zero with the mother standing alone on the scale. Theseweights will be recorded to the nearest tenth (0.1) of a kilogram. The dataon the community registers are the responsibility of the local NGOs,although the firm will work with the NGOs to ensure that the data col-lection system is compatible with the full range of objectives of the study.
The firm consultant will transcribe these weights on a pro forma to bedesigned in collaboration with technical advisors from the World Bankand the University of Oxford. This data transcription will be undertakenevery six months after the child day in the project area. In addition to thechild’s ID (a unique combination of the parish ID, the village ID, and theindividual ID recorded on both the child’s own card and the communityregister), the data on the pro forma will include the child’s gender; thedate of birth of the child taken from the child’s health card or, if that is notavailable, the age of the child taken from the parish register; the date ofthe child fair at which the weights were recorded; and whether or not the
175SAMPLE TERMS OF REFERENCE
02-Evaluating Impacts 5/23/00 5:42 AM Page 175
child took a dose of albendazole. These data will be entered in a comput-erized record in Kampala. The individual ID will provide the basis formerging the data from different periods, and thus the ID must be record-ed each time the data are transcribed and must remain constant for achild over the entire project.
The local circumstances and conditions at each child day that maydeter mothers from attending will also be recorded. These include dataon the state of the harvest and the weather conditions, both of which maydeter mothers from attending. Any special methods and opportunitiesused to advertise each child day will be recorded because different formsof advertising may affect attendance. The record should also include anestimate of the number of children who visited the child day from otherparishes and who did not have ID numbers obtained from the organizersof the child day.
The experiment will last two years. Thus, the firm consultant willrecord the data five times for each parish. That is, the firm consultant willcollect the data at the beginning of the project and at 6, 12, 18, and 24months after project initiation.
A complete copy of the data will be sent to the PCO every six months.These copies of the data will be considered the deliverable services of thefirst study of the project. Preliminary analysis will be undertaken at theUniversity of Oxford on a semiannual basis. However, the firm is request-ed to nominate a representative who will participate in the main analysisto be performed at the end of two years. This representative will be pro-vided travel and living expenses to work on the analysis at Oxford. Thefunds for this travel are budgeted in a separate line item and thereforeneed not be included in the contract covered by the request for proposals.
Study Two: Overall Impact of NECD Interventions. Household surveysand community surveys will collect baseline and follow-up informationneeded to evaluate the impact of the various project activities. The sur-veys will have several modules, which will measure:
• Cognitive development and growth of children under six years of ageresulting from anthelmintic treatments and of overall project activi-ties—Study Two will assess longitudinal growth and psychosocial andcognitive development outcomes in a cohort of children in communi-ties participating in the project (with and without anthelmintic treat-ment) compared with a cohort of children in nonparticipating com-munities. Both cohorts will be followed for two or more years. Thestudy will therefore complement the study of deworming at the parishlevel by allowing a greater understanding of the decision to take chil-dren to child days and to measure whether, over time, participation
176 EVALUATING THE IMPACT OF DEVELOPMENT PROJECTS ON POVERTY
02-Evaluating Impacts 5/23/00 5:42 AM Page 176
leads to an increase in measures of cognitive development. Moreover,by including communities that do not receive any ECD services, thestudy will assess whether the package of services leads to improve-ments in nutritional status and cognitive development.
• Changes in knowledge, attitude, and child-rearing practices of thecaregivers resulting from project parental education and the massmedia campaign.
• Improvement of the health and nutrition of children under six years ofage resulting from growth-monitoring activities, preventive healthand nutrition education, anthelmintic treatments, and overall projectactivities.
• Household welfare resulting from community grants, grass-rootsmanagement training, income-generating activities, and credit savingsgroup formation.
• Community characteristics and changes resulting from the projectinterventions (or otherwise) that could have an impact on child well-being during the duration of the project.
Sample Selection. The basis for this study will be a baseline survey col-lected at the time services are first delivered to the communities and a follow-up survey collected from the same households two years after theinitial survey. One-third of the sample will be drawn from the same 25parishes in the treatment (anthelmintic) group and another third from thecontrol groups studied in Study One. In addition, one-third of the samplewill come from villages in 25 parishes in the same districts as the treat-ment groups but that are not expected to receive services from the NECDproject. Thirty households will be selected from each parish. This implies750 households per strata (2,250 total) in the initial survey. Given expect-ed sample attrition, 5–10 percent fewer households are expected in theresurvey.
To collect the sample in the treatment and control parishes, all house-holds in each parish (there are approximately 700 households in a parishon average) will be listed, possibly by a resident of the community. Thislist will contain the name of the household head, an indication of the loca-tion of the household, and the number of children under age six in thehousehold. This list will serve two purposes. First, a sample of 30 house-holds containing at least one child under the age of six per parish will beselected by a random draw. Second, the total number of children undersix will serve as an estimate of the potential coverage of children in childdays and thus assist in determining the rate of attendance.
Because the NECD project will have less contact with the communitiesthat have no current NECD activity, the selection of households thatreceive no ECD service should use cluster sampling to reduce the costs of
177SAMPLE TERMS OF REFERENCE
02-Evaluating Impacts 5/23/00 5:42 AM Page 177
sample listing. In particular, one subcounty that is not in the projectshould be selected for every subcounty that is in the treatment group,preferably one that is adjacent to it. All parishes in these subcountiesshould be listed and a random draw of 25 parishes from the total list willbe selected. Two villages from each parish selected will then be chosen,again using a list of all the villages in the parish. This step reduces thenumber of villages where a census will need to be conducted. The cen-sus—similar to the one used in the treatment and control parishes—willform the list of households used to draw the sample of 30 households perparish. This will be the third stratum of the survey.
The initial baseline survey should be undertaken in mid-1999. Thistiming is based on the need to know the subcounties and parishes inwhich the NGOs will be operating in order to employ the suggested sam-ple design. This timing is also based on the assumption that the selectionand training of lead NGOs will not be completed until late 1998.
The development and pretesting of the questionnaire, however, shouldbe undertaken much earlier than this (early 1999) in order to be ready toimplement the survey as soon as the NGOs have identified the parishes inwhich they will be working. As the baseline needs to be fielded before thefirst deworming, the ideal time for the baseline survey is concurrent withthe initial community organization that will lead to a child day. Because thesample of 30 families in each parish is small relative to the total population,it is unlikely that the survey data collection will disrupt other activities oroverburden the communities. The data collection in the control groups(those with NGO activity but no deworming and those with neither)should be simultaneous with the data collection in the treatment group.
Survey Instruments. The basic questionnaires to be used for the surveyproject are household questionnaires (which gather data at the level ofthe household and individuals) and community questionnaires.
X. Household Survey
Household data will be collected by using a precoded schedule. This willbe drafted on the model of the Living Standards Surveys used in morethan 30 countries. A first draft will be provided by researchers from theWorld Bank. However, the instrument will be both abridged to accom-modate the particular needs of the project and adapted to local conditionsby using focus groups and a pretest procedure undertaken by the firm.The household questionnaire will contain modules to collect data on:
1. Sociodemographic characteristics: A roster of individuals residing inthe household in the past 12 months, their age and gender, as well astheir schooling and type of employment (if any). The coding format
178 EVALUATING THE IMPACT OF DEVELOPMENT PROJECTS ON POVERTY
02-Evaluating Impacts 5/23/00 5:42 AM Page 178
will indicate the parents of all children, if present—if not present,whether the parents are still living. A detailed list of assets will be col-lected to serve as an indicator of socioeconomic status.
2. Knowledge, attitude, and practices: The questionnaire will also collectinformation on the knowledge, attitudes, and child-rearing practicesof the principal caregivers.
3. Anthropometric data: Weights will be recorded to the nearest tenth(0.1) of a kilogram for all children under the age of six by using digitalscales that are to be provided. In addition, heights will be collected forall children between the ages of two and six. The pretest will be usedto determine whether it is feasible to collect the weights of the moth-ers of these children (if living in the households) as well.
4. Cognitive assessment: The firm will work with other local and interna-tional research consultants to the PCO to integrate tests of child cogni-tive development into the overall field data collection. In the baselinesurvey an internationally recognized test of cognitive development willbe administered to children aged 4.0–5.99 years. This test will also beadministered to the same age group in the second round of the survey,allowing a comparison of cohorts. In addition, a subset of children aged6–7.99 years at the time of the second round will be administered thistest. (Annex table 2.1 summarizes this strategy.)
In addition, knowledge assessments based on specific content fromthe program and a dynamic assessment may be developed for the sec-ond round of the survey. The inclusion of these measures will be eval-uated during the course of the project. Finally, a school performancemeasure will be developed for assessing knowledge acquired in thefirst year of school and administered to a subset of older children inthe resurvey. Existing tests might be adapted.
5. Child health: Morbidity data (including number and kind of symp-toms, levels of severity, length in time), patterns of access to and uti-lization of health services, sanitation, and so forth.
6. Household economy: The best approach to collecting this informationwill be extensively explored in the pretest phase and assessed jointlywith advisors from the PCO prior to finalizing the questionnaire. Thevariables may include food expenditures; agropastoral activities; con-sumption of home production; nonfood expenditures; housing charac-teristics; inventory of durable goods; employment; economic activities;income; land; crops and animals; income from project activities; house-hold enterprises; asset ownership; credit and savings information onamount of money and goods lent and borrowed, if money and goodshave been borrowed in the past 12 months; savings and net debt theday of the interview; information on loans, including the schedule,reason for borrowing, and number of loans from the same source; and
179SAMPLE TERMS OF REFERENCE
02-Evaluating Impacts 5/23/00 5:42 AM Page 179
location of savings, if any, including bank, housing savings bank, ruralsavings bank, and so forth. This information will be part of the base-line and final surveys only.
XI. Community Survey
Community questionnaires will be used to gather information on localconditions that are common to all households in the area. The bestapproach to collecting this information will be extensively explored in thepretest phase and assessed jointly with advisors from the PCO prior tofinalizing the questionnaire. The variables may include:
1. Demographic information: number of households, total population,population under six, ethnic groups, and religions;
2. Economic information, including principal economic activities andpatterns of migration for jobs;
3. Infrastructure: access to roads, electricity, pipe water, market, bank,and public transport. Condition of local infrastructure, such as roads,sources of fuel and water, availability of electricity, and means of com-munication;
4. Local agricultural conditions and practices: type of crops grown in thecommunity, how often and when they are planted and harvested, howthe harvest is generally sold, and qualitative data on rainfall, climateconditions, and seasonality;
5. Education: number and types of preschools, formal and informal ECDarrangements, distance to schools, number of classes, enrollment rates(gross and by gender), attendance, grade progression, health andnutrition services provided at school (for example, school health pro-grams, school lunch);
6. Health: type of health facility and distance and travel time to the near-est of each of several types of health facilities (hospital, pharmacy,health post, and so forth). Distance and travel time to the nearest ofeach of several types of health workers (doctor, nurse, pharmacist,midwife, community health worker, and so forth); and
7. Other: number and type of active local NGOs/CBOs, other child relat-ed projects or interventions (for example, government vaccinationcampaigns), and other community development projects.
Suggested Survey Staff. The survey staff should be constituted as fol-lows:
• Core survey staff: composed of the survey manager, the field manag-er, the data manager, and the data entry staff who will be responsible
180 EVALUATING THE IMPACT OF DEVELOPMENT PROJECTS ON POVERTY
02-Evaluating Impacts 5/23/00 5:42 AM Page 180
for overall field supervision, coordination, and monitoring of data col-lection and data entry and data management activities.
• Field survey staff: the field operations will be conducted by teamscomposed of a supervisor, two (or three) interviewers responsible forthe main questionnaire and the anthropometric measurements, and adriver. A similar number of specialists who will participate in admin-istering tests of cognitive development to the children will be selectedand trained in collaboration with local and international experts.
• Coordinator for the randomized trial: the coordinator will assist in thedevelopment of the data collection instruments, training of localNGOs responsible for the organization of the child days in the parish-es on the experimental design, data collection, and data transcription.He or she will oversee data entry and management of the study dataset and will participate in the main analysis to be performed at the endof the study.
Organization of Fieldwork. The firm will participate in the drafting ofthe field instruments prior to the pretesting of the survey and will haveprimary responsibility for the pretest. After the pretest the questionnairewill be redesigned (in partnership with researchers from the World Bank)and then translated into local languages.
The firm will work with other local and international research consul-tants selected by the PCO to integrate tests of child cognitive develop-ment into the overall field data collection. The local ECD researcher,assisted by international consultants, will select and adapt the principalcognitive test to be used and will train the testers.
The following organization of fieldwork is suggested. This is based oninternational experience and designed to ensure quality control. Somevariation of this approach might be agreed upon in consultation withresearchers from the World Bank based on the experience of the firm andother advisors to the PCO and information gained during the pretest.
The fieldwork will be organized into small teams consisting of a super-visor, two (or three) interviewers responsible for the main questionnaireand the anthropometric measurements, and a similar number of special-ists in administering tests of cognitive development to the children. Thesestaff will be trained in Kampala by the local ECD researcher in coordina-tion with international advisors on psychological testing. The trainingwill include a discussion of the research objectives, a review of each stepof the interview, practice training in the office, a dry run in the field, anda recap of experience after this dry run.
Once teams are trained they should be retained for the entire round ofthe survey, if possible. However, because a few staff may prove to beunsuitable during the fieldwork, it is advisable to train a few extra staff. It
181SAMPLE TERMS OF REFERENCE
02-Evaluating Impacts 5/23/00 5:42 AM Page 181
is not advisable to hire staff to work for a few days only in one parish andthen new staff in the next parish because this results in inexperienced staff.All staff should receive new training at the beginning of the resurvey.
During the administration of the cognitive test, children should, to thedegree possible, be alone with the interviewer. In no case should anotherperson (adult or child) respond to the questions asked of the child.However, during the resurvey the test for the subset of eight-year-oldsmay be administered in a group format if it proves convenient.
The supervisor will be responsible for ensuring that the interviewersundertake the survey in the households chosen for the sample withoutsubstitution and that all children in the appropriate age groups areadministered the tests of cognitive development. In addition, the super-visor will review each questionnaire after completion (prior to the teammoving to a new parish) to ensure that there are no gaps in the question-naire and to see that seemingly inconsistent information is verified.
The firm will enter all survey data as soon after collection as possible.Copies of the household- and child-specific data and rating scales alongwith the documentation necessary to access the data will be provided tothe PCO in a computerized format at the end of the baseline survey. Theoriginal questionnaires should be retained by the firm because the orig-inal data will generally need to be accessed during the course of theanalysis.
The child-level data must contain accurate identification codes thatcan be matched with the household survey codes. Although the uniqueindividual and household codes provided to the PCO need not containthe names of the households or their exact location, this informationmust be stored by the firm in a manner that makes it possible to revisitthe household at a later date. Because one step in the analysis will linkindividuals in the resurvey with their test results from the baseline, allindividual and household codes must be held constant over the threesurveys.
XII. Specific Tasks for Survey Specialists
The firm will participate in the following activities in collaboration withthe PCO, local researchers, and researchers from the World Bank and theUniversity of Oxford and implementing NGOs:
• Revision of work programs.• Development and adaptation of the data collection instruments and
support documentation, including listing materials, questionnaires,coding guides, interviewer and supervisor manuals, manual of opera-tions, data entry manual, and field procedures.
182 EVALUATING THE IMPACT OF DEVELOPMENT PROJECTS ON POVERTY
02-Evaluating Impacts 5/23/00 5:42 AM Page 182
• Revision of various drafts of documents, layout, translation, back-translation, and field testing. Provide hard copies and electronic ver-sions of all documentation to PCO. Forward questionnaires to the WorldBank researchers for their review and revision prior to the pilot test.
• Dwelling listing and cartographic updating. The responsibilities forlisting of households and dwellings in each selected parish includeobtaining base maps, preparing listing materials, contacting local offi-cials to inform them about the listing operation, identifying bound-aries, drawing maps, listing households in a systematic manner,obtaining preliminary information on households, including name ofthe household head, an indication of the location of the household,and the number of children under age six in the household; docu-menting procedures at the time of the sample design, at the end of thefieldwork, and at the completion of the data file.
• Preparation of sampling framework (with sampling specialist), train-ing of staff to implement the designed sample, supervision of theimplementation stage to ensure the quality of the sample selected, andprovision of a detailed report outlining all the steps involved in thedesign and implementation of the sample.
• In consultancy with the World Bank, participation in determining anappropriate strategy for identifying comparison groups (that is, non-project parishes).
• Selection and training of field workers. This activity consists of all thework necessary to develop training materials and manuals for all per-sons involved in fieldwork. Training will be required for interviewers,supervisors of interviewers, supervisors of teams, data entry person-nel, and anthropometric personnel.
• Field operation, including logistical arrangements for data collectionand obtaining household and individual consent; keeping a studyhousehold register.
• Production of progress reports: The firm will prepare fieldworkprogress reports (at six-month intervals) copied to the PCO and theWorld Bank. The firm should also prepare a basic description of thesurvey. This should include the survey content, the sample plan and itsimplementation, and the fieldwork techniques used. A full question-naire and basic documentation should be included as appendixes.
• Development of a data entry program using software that can checkfor ranges and consistency of the data and generate reports indicatingmissing data, data outside of the accepted ranges, and inconsistentanswers.
• Data cleaning, data entry, database management, and tabulationplans, including development of data entry program, data entry man-ual, data entry operator training, data quality checks, and guidelines
183SAMPLE TERMS OF REFERENCE
02-Evaluating Impacts 5/23/00 5:42 AM Page 183
for using the data. Also, coding open-ended questions, verification ofthe data, checking anthropometric data against standard referencetables.
• Enforcing data use policy agreement: The firm and researchersinvolved in the process of data collection and analysis will sign a mem-orandum of understanding with the PCO that will explicitly state thepolicy regarding issues such as access to data, intended users, proce-dures for obtaining copies of the data sets and documentation, andpublication and authorship rules.
• Conducting data analysis: The firm will conduct exploratory dataanalyses (for example, frequencies, percentage tabulations, and cross-tabulations) of key survey variables and their correlates. The firm willconduct modern statistical modeling of impacts after rounds 2 and 3 todetermine overall progress in social indicators (for example, nutrition,health, incomes, and community development) and the factors thataccount for the changes or lack of changes.
• Producing analyses reports: The firm will report on the findings afterrounds 2 and 3 of the surveys based on the analyses of the social indi-cators and the covariates. The firm will coordinate with the PCO andthe World Bank on the Parish Child Health Day Study and on the col-lection of impact on cognitive development but will not be responsiblefor the final reports on the result of these studies.
Specific tasks for the community survey include the following:
• Work with advisers from the PCO in the development of the commu-nity questionnaire and extensively explore in the pretest phase the bestapproach to collecting this information;
• Work closely with the implementing agencies (lead and local NGOs) inthe collection of the community data;
• Contact local officials and community leaders to explain the projectimpact evaluation approach and obtain communal consent for surveyresearch and the child health day study;
• Interview key informers and obtain maps, lists, and other communityrecords;
• Obtain lists of health and education facilities (pre- and primaryschools), including geographic location, catchment area, and type ofestablishment (for example, private or public);
• Obtain community demographic information, including number ofhouseholds and population by gender and age; and
• Obtain other data required in the community questionnaires.
Specific tasks for the child day study include the following:
184 EVALUATING THE IMPACT OF DEVELOPMENT PROJECTS ON POVERTY
02-Evaluating Impacts 5/23/00 5:42 AM Page 184
• Participation in the development of the study protocol,• Development of data collection instruments,• Training of local NGOs responsible for the organization of the child
days in the parishes for the experimental design,• Supervision of data collected during child day,• Data transcription,• Data entry and management, and• Participation in the main analysis to be performed at the end of the
study.
Annex Table 2.1 Proposed Sample Sizes for Impact Evaluationof Nutrition and Early Child Development Project, Uganda
Category Deworming No deworming No deworming and parent and parent and no parent education education education
No. of parishes 25 25 25 TotalTime Baseline Second Baseline Second Baseline Second
rounda round roundNo. of households 750 700 750 700 750 700 2250No. of children weighted at child daysb 5000 5000 5000 5000 20000No. of children aged 0–5.99 with anthropometry in the home (mean two per family)c 1500 1395 1500 1395 1500 1395 11580No. of children given cognitive tests: test all children aged 4.0–5.99 in house-holds 500d 465e 500 465 500 465 2895No. of children aged 6.0–7.99 given cognitive test and anthro-pometry subsetf — subset — subset subset
185SAMPLE TERMS OF REFERENCE
(Table continues on the following page.)
02-Evaluating Impacts 5/23/00 5:42 AM Page 185
School enrollment rates 25 25 25 25 25 25
commu- commu- commu- commu- commu- commu-nities nities nities nities nities nities
a. Assuming a small loss to attrition of 8 percent in two years.b. Assuming that about 200 children will attend each child day.c. Two children per family are assumed, but families will be recruited if they
have any children under six. Family refers here to a pair that consists of a mother(or substitute) and child.
d. This is a maximum; the actual number can vary according to the number offour- to five-year-old children encountered.
e. Assuming the same loss of 8 percent over two years; only children whoseparents were interviewed will be tested.
f. Number will be a subset of the children in the age range whose parents wereinterviewed. They will be linked with the earlier score. Even though the numberof children tested increases in the second round, the time for the interviews maydecrease because much information will not need to be assessed again. It is alsopossible that the size of this group will be reduced.
Source: World Bank Project Document.
Validity Study. In addition to the above, one small longitudinal studywill be added to examine the predictive validity of the preschool measurefor school performance at the end of the first year of school. In the base-line survey, two children per community aged 6.0 to 6.9 (not yet inschool) will be tested, for N = 150. These children will be located at theposttest and given a school performance test two years later, at ages 8.0to 8.99.
Task Schedule. The tentative timetable for the task schedule is as follows:
Month 1. Begin the process of constructing indicators of cognitivedevelopment in conjunction with international consultant and in accordwith terms of reference. This process may take up to six months.
Month 2. Initial pretest and revision of the questionnaire.Month 5. Begin listing of households for sample selection. This step is
dependent on the selection of the lead and local NGOs. It cannot be doneuntil the PCO and NGOs choose the parishes where child days will beorganized and then select the sites for the initial deworming program. At
186 EVALUATING THE IMPACT OF DEVELOPMENT PROJECTS ON POVERTY
Annex Table 2.1 (continued)
Category Deworming No deworming No deworming and parent and parent and no parent education education education
02-Evaluating Impacts 5/23/00 5:42 AM Page 186
the same time the questionnaire should be translated and field testedagain.
Month 7. Begin collection of data at child fairs for the dewormingstudy. Data will be collected at these fairs at six-month intervals. Asabove, the timing of this step is dependent on the selection of the lead andlocal NGOs.
Month 8. Training of field staff for household survey and initiation ofsurvey. The survey should take approximately three to four monthsdepending on the number of teams employed. Data entry should be con-current with data collection.
Month 14. Initial analysis of baseline data. This will be an ongoingprocess.
Month 20. Staff from the firm visit the University of Oxford to partici-pate in analysis of initial data.
Months 20–36. Collection of data for round 2 for deworming study.Midterm and final household surveys will be conducted two and four
years after baseline.
Support to Firm
No specific support will be given to the firm to carry out assignments.Firms are advised to include all requirements for effective carrying out ofthe assignment in their proposals.
187SAMPLE TERMS OF REFERENCE
02-Evaluating Impacts 5/23/00 5:42 AM Page 187
Example II: Rural Roads Impact Evaluation:Vietnam 1997 Baseline*
Terms of Reference: Baseline Survey for Rural RoadsImpact Study
I. Background
The study aims to assess the impact on living standards of the WorldBank–financed Vietnam rural transport project that were implemented in15 poor provinces over three to five years starting in 1997. The study’soverall focus will be on how the determinants of living standards arechanging over time in communes that have road project interventionscompared with ones that do not. This requires the collection of preprojectbaseline data for both project (“treatment”) areas and nontreatment con-trol areas and a number of further data collection rounds of postinter-vention data at two-year intervals. A detailed commune level databasewill be created in part by drawing on annually collected records at thecommune level. The latter will be augmented by the collection of retro-spective commune-level data and the collection of various other key sup-plementary data. A short district-level survey will help put the commune-level data in context. Finally, 10 to 15 households will be ran-domly sampled from commune-level household lists and a short house-hold questionnaire will be administered. The study will be conducted in6 provinces out of the 15 that will benefit from the project. The 6provinces will be representative of the 6 geographical regions of Vietnam.A random sample of about 200 or so project and nonproject communeswill be drawn. Six teams will be set up to simultaneously survey eachprovince. The survey should begin in April and finish about August. Datashould be available about October or November.
II. Survey Design
Sampling. Sampling will be done in three levels:
1. Provinces: The 15 project provinces are located in Vietnam’s six geo-graphical regions. Criteria for selection of survey provinces will be the
188 EVALUATING THE IMPACT OF DEVELOPMENT PROJECTS ON POVERTY
* These terms of reference were prepared by Dominique van de Walle.
02-Evaluating Impacts 5/23/00 5:42 AM Page 188
following: (a) one province will be selected in each geographical region;and (b) when there is more than one possible project province in eachregion, a random selection will be made.
2. Communes: The aim is to survey 200 or more communes, which arerandomly selected. About half or less (not more) should be communeswith road link projects, the rest controls. A list will be drawn of nonpro-ject communes in the six provinces (or alternatively one list for eachprovince) and a random sample will be drawn. Similarly, a list will bedrawn of all communes benefiting from road projects in the sixprovinces (or by province). This may be more than one commune perroad link; all will be included in the sampling frame. From these a ran-dom sample will also be drawn. The sample will not necessarily includeboth communes linked by a road project. If access to certain sampledcommunes is impossible, it will be replaced with another commune inthe district that is similar.
3. Households: In each sampled commune, a household survey will beadministered to 15 households. These (plus perhaps a few replacementhouseholds) will be randomly selected from the commune householdlists. After selection, the commune authorities will be asked about wherethe households fall in the very poor, poor, average, not poor, and richclassifications.
III. Survey Process
Six survey experts will be hired to conduct the surveys in the sixprovinces. After their training and the field testing of the questionnaire,they will begin surveying simultaneously in each province. In the dis-tricts, surveyors will need at least one local staff from the district projectmanagement unit to help with contacting local authorities and in somecases to help find suitable guides and interpreters in minority areas.Survey assistants or assistance from the provincial project managementunits will be hired as required.
Each surveyor will collect data from 35 communes on average, the dis-tricts they belong to, and 15 or so households per commune. Three to fourdays will be needed for each commune. The time spent in the field willbe about 100 to 140 days (four to five months). The total time will be sixmonths.
During the survey period, the supervisor will conduct field visits to allsix provinces to supervise data collection and ensure high quality.
The collected data will be cleaned and entered by using a data entryprogram.
Annex table 2.II.1 gives an estimated timetable for the study.
189SAMPLE TERMS OF REFERENCE
02-Evaluating Impacts 5/23/00 5:42 AM Page 189
Annex Table 2.II.1 Timetable for Impact Evaluation Study,Vietnam
Jan. Feb. Mar. April May June July Aug. Sept.
Design question-naires **** ****
Field test survey ****
Revise question-naires ****
Adapt data entry program, translate and print question-naires ****
Hire and train surveyors ****
Survey in field **** **** **** **** ****
Check data **** ****Perform data
entry *****
IV. Other
Equipment. The equipment purchased under the project will belong tothe project as long as the study continues (through future rounds) butwhen not in use by the team it will be housed in the Project ManagementUnit for the team’s use.
Budget Disbursements. The budget for the study (excluding payments tothe main investigator, who will receive monthly installments) will be dis-bursed in three installments. The first, upon signing of the contract, will con-sist of 20 percent of total funds. The second installment, consisting of 50 per-cent of the total budget, will be disbursed once the commune, household,and district questionnaires have been finalized and approved by the WorldBank task manager. This is expected to be sometime in late March. The thirdand final installment will be disbursed in late July or halfway through datacollection. Estimated budget details are shown in annex table 2.II.2.
190 EVALUATING THE IMPACT OF DEVELOPMENT PROJECTS ON POVERTY
02-Evaluating Impacts 5/23/00 5:42 AM Page 190
Annex Table 2.II.2 Estimated Study Budget
No. Time Amount (US$) Total (US$)
1. Main investigator 1 9 months 1,000 9,0002. Survey experts 6 6 months 400 14,4003. Travel allowance
for six surveyors, six local guides and interpreters 12 125 days 8 12,000
4. Car and other transport for six survey teams 6 125 days 40 30,000Car rental for main investigator 1 30 days 50 1,500
5. Air tickets Hanoi-Ho Chi Minh-Hanoi 6For surveyors (south provinces) 3 persons 200 1,200For main investigator 3 trips
6. Training of surveyors 12 1,338Payment 1 week 50Travel to field 3 days/ 3 cars 50Allowance 3 days 8
7. Field test of questionnaire (South and North communes) 1 2 weeks 2,000
8. Data cleaning and entry 2 2 months 200 800
9. Survey materials 2,00010. Communications
(fax, phone, email, Xerox) 2,00011. Equipment 5,000
Computer (PMU18) 1 1,700Printer (PMU18) 1 1,000Fax machine (study team) 1 500Laptop computer (study team) 1 1,800
191SAMPLE TERMS OF REFERENCE
(Table continues on the following page.)
02-Evaluating Impacts 5/23/00 5:42 AM Page 191
Annex Table 2.II.2 (continued)
No. Time Amount (US$) Total (US$)
12. Translation (questionnaire, manuals, documentation) 200 pages 8/page 1,600
13. Printing, Xeroxing 80014. Contingencies 1,362Total 85,000
Terms of Reference: Survey Supervisor or Main Investigator
I. Job Description
The in-country survey supervisor or main investigator will be responsiblefor the study’s baseline survey work within Vietnam. Responsibilitiesinclude determining availability of information at the commune level; help-ing to revise and finalize the district-, commune-, and household-level ques-tionnaires; field testing the questionnaire; incorporating revisions to thequestionnaire; arranging for the questionnaire to be translated; hiring andtraining the assistants; planning the field work logistics; preparing surveyimplementation and questionnaire documentation; supervising surveyimplementation and ensuring quality control; and supervising the projectdatabase and arranging for data cleaning and entry. The person will also actas the liaison with the Ministry of Transport’s Project Management UnitPMU18, the World Bank resident mission, the Canadian InternationalDevelopment Agency representative in Hanoi, and the project task manag-er at the World Bank in Washington. The person will report directly to thetask manager. The person will start as soon as January 1997; the contract canbe processed for a period of nine months at a rate of $1,000 per month.
II. Specific Tasks.
Specific tasks include the following:
1. Assuming responsibility for hiring, drafting detailed terms of refer-ence, training and supervising six main assistants who will work with
192 EVALUATING THE IMPACT OF DEVELOPMENT PROJECTS ON POVERTY
02-Evaluating Impacts 5/23/00 5:42 AM Page 192
local assistants (possibly from the local transport office) in the fieldand will be responsible for the collection of the district-, commune-,and household-level data;
2. Explorating of data availability at the commune level and workingclosely with the World Bank task manager to design the final versionsof the questionnaires;
3. Carrying out a field test of the questionnaires in both South and Northrural communes; reporting back on potential problems and necessaryrevisions; revising the questionnaires when needed;
4. Arranging for the questionnaires to be translated, printed, andXeroxed (the final versions of the questionnaires will be available inboth English and Vietnamese);
5. Choosing the six provinces to be included in the survey so that there is oneprovince to represent each geographical region—when there is more thanone such province, the sampled province is chosen randomly; drawing upa random sample of around 200 rural communes in the six provinces,including about half with projects and the rest without projects;
6. Planning all fieldwork logistics, including arranging for transport, drivers, travel allowances, the schedule of commune surveys, and alerting commune administrations of team arrivals and purpose;
7. Participating in survey implementation, alternating between theteams in a supervisory role; ensuring quality control; identifyingproblems affecting survey implementation, checking quality andcompleteness of data collected, suggesting ways to solve problems,and implementing them following consultation with the studyleader;
8. Ensuring that future survey rounds can replicate the baseline survey,which requires (a) preparing detailed documentation of all surveyimplementation design and logistics (how the sampling of provinces,communes, and household was done; how the survey teams weretrained and organized; how fieldwork was organized; what proce-dure was followed when a sampled site was not accessible or a sam-pled household not found; problems, issues raised, and solutionsfound); and (b) preparing a detailed manual on definitions of terms(for example, unemployment, income, primary occupation, child oradult, distance), units, currency amounts, codes used in the question-naires; how the questionnaires are to be administered and to whom;how prices were collected, and so forth; the former should ensure thatfuture rounds of the survey can reproduce the baseline’s organizationand logistical details, and the latter should be used in training of sur-veyors and for their work, as well as to aid future users of the data(there will be both English and Vietnamese versions);
9. Procuring the necessary equipment as itemized in the study budget;
193SAMPLE TERMS OF REFERENCE
02-Evaluating Impacts 5/23/00 5:42 AM Page 193
10. Establishing good relations and ensuring close cooperation withPMU18. Keeping them abreast of the study and monitoring projectdevelopments; supervising the setting up of a database of project-specific data (the World Bank task manager will identify the data to beincluded);
11. Arranging and supervising data cleaning and entry by using the pro-vided data entry program; and
12. Acting as liaison and communicating often with the task manager.
194 EVALUATING THE IMPACT OF DEVELOPMENT PROJECTS ON POVERTY
02-Evaluating Impacts 5/23/00 5:42 AM Page 194
Annex 3
A Sample Budget from an ImpactEvaluation of a School Feeding
Program
Phase I: July 1999–December 2000a
School Feeding Research Proposal—Baseline and Cross-SectionalEvaluation(July 1999–December 2000)Draft Budget—7/14/1999—US$
Staff weeks/Activity Source of funds/CostsFY2000 FY2001 BB RPO Other Total
World Bank staffEconomist 4 2 17,640Evaluation specialist 5 3 23,520Nutrition specialist 5 3 23,520Peer reviewer 0.2 0.2 1,948Peer reviewer 0.2 0.2 1,948
68,577FES staffStudy coordinator 4 4 12,000
12,000International
consultantsSituational
assessment (incl. travel) 7,000
Cognitive test development (incl. travel ) 6,000
Sampling specialist 2,000Cost-effectiveness
study 25,00040,000
Regional consulting firmb
Design, sampling, administration 42,000
195
(Budget continues on the following page.)
02-Evaluating Impacts 5/23/00 5:42 AM Page 195
Fieldwork 25,000Data orocessing 3,500Analysis 30,000
100,500Travel to countryTrips 4 2 12,000
12,000ContingenciesCommunication 1,000Software 2,000Translation 2,000
5,000TOTALS 68,577 150,500 19,000 238,077
Total Requested from RAD: $150,500Total Requested from Bank budget: $68,577Total Provided by outside sources: $19,000
a. Budget estimates for phase II of the evaluation are not included in this proposal.b. A breakdown of these costs is provided on the next page.
196 EVALUATING THE IMPACT OF DEVELOPMENT PROJECTS ON POVERTY
Staff weeks/Activity Source of funds/CostsFY2000 FY2001 BB RPO Other Total
02-Evaluating Impacts 5/23/00 5:42 AM Page 196
Estimated Budget—Local Data Collection and Analysis for Phase ISchool Feeding Impact Evaluation Costs in US$
# People # Staff weeks Weekly rate Total
ProfessionalsDirector 1 12 2,000 24,000Education specialist 1 8 1,500 12,000Nutrition specialist 1 8 1,500 12,000Statistician/sampling 1 12 750 9,000Fieldwork manager 1 8 750 6,000Programmer 1 10 300 3,000Data processing supervisor 1 8 300 2,400Assistant – surveys 1 10 100 1,000Assistant – anthropometrics 1 10 100 1,000Assistant – cognitive tests 1 10 100 1,000Data quality control 1 8 100 800Subtotal—Professional staff 72,200Fieldwork staff Supervisor 4 6 200 4,800Cognitive tester 4 6 120 2,880Anthropometrist 4 6 120 2,880Interviewer 4 6 120 2,880Driver 4 5 100 2,000Fieldwork equipment People/units Cost per week or unitVehicles (4 vehicles for
5 weeks) 4 5 350 7,000Gasoline (4 vehicles for
5 weeks) 4 5 80 1,600Scales; rulers (5 sets) 5 20 100Cognitive test equipment
(for 4 testers) 4 20 80Survey equipment
(for 4 interviewers) 4 20 80Subtotal – Fieldwork 24,300Data orocessing PeopleData coding 3 7 75 1,575Data entry 4 7 75 2,100Subtotal—data processing 3,675
Total 100,175
197A SAMPLE BUDGET FROM AN IMPACT EVALUATION
02-Evaluating Impacts 5/23/00 5:42 AM Page 197
Annex 4
Impact Indicators—Evaluation ofBolivia Social Investment Fund
Developed November 1997
I. Formal Education—Schools Type “A” and “B” (multigrade and regular)
1. Final Impact IndicatorsAchievement in Mathematics and Language testsa
Repetition rateDropout rateEnrollmentInstruction levelDemand for education (percent of students rejected from
school)a
2. Intermediate Impact IndicatorsRegularity in student attendanceRegularity in teacher attendanceStudents’ time allocation/hours spent studyingClassroom teaching methoda
Turnover in teaching staffa
3. Intervention IndicatorsInfrastructure
Ratio students/classroomNumber of classrooms in “good shape”Number of missing classroomsAvailability of multifunctional areaAvailability of basic services
– Electricity– Source of main water supply– Type of sanitation service; condition of the sanita-
tion serviceFurniture
Ratio students/deskRatio teacher’s tables/classroomRatio teacher’s chairs/classroom
198
02-Evaluating Impacts 5/23/00 5:42 AM Page 198
Ratio “adequate blackboards”/classroomRatio shelves/classroomsTexts and didactic materialRatio texts/studentQuality of mathematics, language, social studies, and
natural sciences textsAvailability of teachers’ textsAvailability and condition of maps and chartsDidactic games by school cycle (prebasic, basic, and
intermediate)Availability of an abacus
Education Reform Indicatorsb
4. Factors Affecting Outcomes Not Linked to the SIF Project (Exogenous)NutritionAvailability of school breakfast programCost of the schoolTeachers’ characteristicsEducational backgroundYears of serviceTraining receivedMethods applied in teaching (in a period of classes)Training received, by topic and courseStudent evaluation practices (frequency of homework and its
correction)Evaluation of the teachers by the studentsRationale for dropping outStudents rejected by the schoolDistance between the house and the schoolRatio students/teacher
5. Identification IndicatorsWhether school was prioritized by the Education ReformProgrammed cost by project component Actual expenditures by project component
II. Health
1. Final Impact Indicatorsc
Infant mortality rateChildhood mortality rate
199IMPACT INDICATORS—EVALUATION OF BOLIVIA SOCIAL INVESTMENT FUND
02-Evaluating Impacts 5/23/00 5:42 AM Page 199
Rates of incidence and prevalence of main diseases Prevalence of malnutrition (general, slight, moderate,
and severe)
2. Intermediate Impact IndicatorsUse of government health centersPrevalence of tetanus vaccination
Place where vaccine was receivedPrevalence of prenatal control
Number of prenatal controlsQuality of control
Prevalence of births attended in health centersQuality of attention
Prevalence of home births attended by medical personnelHeight at birthWeight at birthAnthropometric assessments
Place where assessment is heldAge when first assessment is made
Incidence of disease and prevalence of immunization by number of doses received
PolioDiptheria-tetanus-pertussis (DPT)MeaslesTuberculosis vaccine (TB)
Knowledge of places to go for immunizationIncidence and treatment for coughingIncidence and treatment for diarrheaPrevalence of the knowledge and use of oral rehydration
packetsClinics’ knowledge of prevalence of pregnancyAttendance of high-risk pregnanciesPrevalence of good hygiene habits and use of waterDuration of lactation
3. Intervention Indicatorsd
Quality of infrastructure by type of health centerAvailability of basic services in the health center (drink-
ing water, sewage system, and electricity)Adequacy of infrastructure based on established norms
by type of health centerAdequacy of equipment based on established norms by
type of health center
200 EVALUATING THE IMPACT OF DEVELOPMENT PROJECTS ON POVERTY
02-Evaluating Impacts 5/23/00 5:42 AM Page 200
Number of beds in the health centerAvailability of essential medicines by type of health centerAvailability of essential medical instruments by type of
health centerAvailability of essential furniture by type of health center
4. Factors Affecting Outcomes Not Linked to the SIF Project (Exogenous)Characteristics of the household
Quality of householdType of household
Basic Services in the householdElectricitySource of waterType of sanitary service
Accessibility to basic servicesDistance between the household and the closest
health centerDistance between the sanitary service and the
source of waterDistance between the household and the main
source of waterHours of availability of water per daySufficiency of amount of water per dayAvailability of water throughout the yearCost of consultation in the health centerHousehold head’s perception of the quality of:
“Service” in the health center attended by the household
“Infrastructure” of the health center attended by the household
“Availability of medicines” in the health center attended by the household
Household expensesPersonal characteristics of the members of the household
AgeLanguageEducation levelOccupation
Geographic characteristicsHealth districtHealth areaHealth sector
201IMPACT INDICATORS—EVALUATION OF BOLIVIA SOCIAL INVESTMENT FUND
02-Evaluating Impacts 5/23/00 5:42 AM Page 201
ProvinceLocality
Human resources in the health center (doctors, odontol-ogist, nutritionists, nurses, nurses’ assistants, techni-cians, administrative staff)
Population under the influence area of the health center by age groups
Cost of consultation in the health centerHealth interventions not financed by the SIF
5. Identification IndicatorsProgrammed cost by project component Actual expenditures by project component
III. Water
1. Final Impact Indicatorse
Infant mortality rateChildhood mortality rateRates of incidence and prevalence of diarrhea in house-
holdsPrevalence of malnutrition (general, slight, moderate,
and severe)
2. Intermediate Impact IndicatorsIncidence and treatment for diarrhea in health centersPrevalence of use and knowledge of use of oral rehydra-
tion packetsPrevalence of good hygiene habits and use of water
3. Intervention Indicators (of Input)Prevalence of training in health topicsAccessibility to basic services
Main source of waterExistence of and type of sanitary serviceDistance between the sanitary service and the
source of waterDistance between the household and the main
source of waterHours of availability of water per daySufficiency of amount of water per dayAvailability of water throughout the year
Quantity of water consumed by the householda
202 EVALUATING THE IMPACT OF DEVELOPMENT PROJECTS ON POVERTY
02-Evaluating Impacts 5/23/00 5:42 AM Page 202
Quality of watera
4. Factors Affecting Outcomes Not Linked to the SIF Project (Exogenous)Use of government (MSSP) health centersSize at birthWeight at birthDuration of lactationCharacteristics of the household
Quality of householdType of household
Accessibility to basic servicesDistance between the household and the closest
health centerCost of consultation in the health centerHousehold’s expenses
Personal characteristics of the household’s membersAgeLanguageEducation levelOccupation
5. Identification IndicatorsProgrammed cost by project component Actual expenditures by project component
203IMPACT INDICATORS—EVALUATION OF BOLIVIA SOCIAL INVESTMENT FUND
a. Not considered in baseline.b. To be developed in coordination with Education Reform staff; will be consid-ered exongenous to the intervention unless the SIF Education Reform interven-tions are considered jointly. c. General mortality rate, birthrate, global fertility rate, adult mortality, and lifeexpectancy at birth deleted.d. Training in health topics deleted.e. General mortality rate, birthrate, global fertility rate, adult mortality (male andfemale), life expectancy at birth, prevalence of acute respiratory infections, andtreatment of coughing deleted.
02-Evaluating Impacts 5/23/00 5:42 AM Page 203
Ann
ex 5
Tem
plat
e of
Log
Fra
me
for
Pro
ject
Des
ign
Sum
mar
y fo
rth
e P
roje
ct C
ompl
etio
n D
ocum
ent
or P
roje
ct A
ppra
isal
Doc
umen
t
Hie
rarc
hy o
f Obj
ecti
ves
Key
Per
form
ance
Ind
icat
ors
Mon
itor
ing
and
Eva
luat
ion
Cri
tica
l Ass
umpt
ions
204
Sec
tor-
Rel
ated
Cou
ntr
yA
ssis
tan
ce S
trat
egy
Goa
l:Pr
ovid
e a
one-
sent
ence
stat
emen
t of
the
long
-ter
mst
rate
gic
goal
(as
ref
lect
edin
the
CA
S) to
whi
ch th
epr
ojec
t is
des
igne
d to
con
-tr
ibut
e. T
he s
tate
men
tsh
ould
des
crib
e su
bsta
n-ti
ve d
evel
opm
ent c
hang
ein
the
sect
or(s
) of
inte
rest
.
Sec
tor
Ind
icat
ors:
1. I
ndic
ator
s ac
com
pany
ing
the
sect
or-r
elat
ed C
AS
goal
invo
lve
mea
sure
men
ts th
atar
e no
t gen
eral
ly f
und
ed b
yth
e pr
ojec
t but
that
may
be
fund
ed b
y th
e B
ank
as p
art
of o
ther
wor
k.2.
Nor
mal
ly th
e bo
rrow
erw
ould
mon
itor
thes
e in
dic
a-to
rs a
s pa
rt o
f go
od-p
ract
ice
sect
oral
man
agem
ent.
Sec
tor
and
Cou
ntr
y R
epor
ts:
1. T
his
colu
mn
iden
tifi
esw
here
the
info
rmat
ion
for
veri
fyin
g ea
ch in
dic
ator
will
be f
ound
, and
the
proc
ess
invo
lved
.2.
Ind
icat
ors
acco
mpa
nyin
gth
e se
ctor
-rel
ated
CA
S go
alar
e ge
nera
lly m
onit
ored
and
eval
uate
d v
ia v
ario
us s
ecto
ror
cou
ntry
rep
orts
gen
erat
edou
tsid
e th
e pr
ojec
t.
(Fro
m G
oal
to B
ank
Mis
sion
)•
Ass
umin
g th
at th
e se
ctor
-re
late
d C
AS
goal
(st
ated
inth
e fa
r le
ft b
ox)
is a
chie
ved
in th
e lo
ng te
rm, l
ist a
nyad
dit
iona
l ass
umpt
ions
need
ed to
link
this
goa
l to
the
Ban
k’s
mis
sion
(th
at is
,po
vert
y al
levi
atio
n).
• T
hese
ass
umpt
ions
oft
enin
volv
e co
ndit
ions
, act
ions
,or
res
pons
es o
utsi
de
of th
e
02-Evaluating Impacts 5/23/00 5:42 AM Page 204
205
Pro
ject
Dev
elop
men
tO
bje
ctiv
e:1.
Pro
vid
e a
one-
sent
ence
stat
emen
t of
the
beha
vior
alch
ange
exp
ecte
d f
rom
the
targ
et b
enef
icia
ry g
roup
or
inst
itut
ion(
s) b
y th
e en
d o
fpr
ojec
t im
plem
enta
tion
. A
chie
vem
ent o
f th
e ob
jec-
tive
ser
ves
as a
sim
ple
test
of d
eman
d f
or p
roje
ct o
ut-
puts
. The
obj
ecti
ve s
houl
dex
pres
s a
sing
le d
evel
op-
men
t pur
pose
that
is r
ealis
-ti
c, s
peci
fic,
mea
sura
ble,
and
dem
and
-dri
ven.
For
agu
ide
to s
etti
ng th
e pr
ojec
tob
ject
ive,
see
“D
o’s
and
Don
’ts f
or S
etti
ng a
Pro
ject
Dev
elop
men
t Obj
ecti
ve”
(cal
l x37
065
or e
-mai
lM
&E
Hel
p@w
orld
bank
.org
for
a co
py).
Ou
tcom
e / I
mp
act
Ind
icat
ors:
1. O
utco
me
ind
icat
ors
rela
teto
the
resu
lts
to b
e ac
hiev
edby
the
end
of
proj
ect i
mpl
e-m
enta
tion
, whi
le im
pact
may
not
be
fully
ach
ieve
dun
til f
ive
or m
ore
year
s af
ter
proj
ect i
mpl
emen
tati
on h
asbe
en c
ompl
eted
.2.
Ind
icat
ors
at th
e ou
tcom
e(P
DO
-lev
el)
are
not a
rest
atem
ent o
f th
ose
at th
eou
tput
leve
l.3.
Col
lect
ion
of d
ata
for
mea
sure
men
t of
thes
e in
di-
cato
rs is
gen
eral
ly f
und
edby
the
proj
ect.
Pro
ject
Rep
orts
:1.
Thi
s co
lum
n id
enti
fies
whe
re th
e in
form
atio
n fo
rve
rify
ing
each
ind
icat
or w
illbe
fou
nd, a
nd th
e pr
oces
sin
volv
ed.
2. I
ndic
ator
s ac
com
pany
ing
the
proj
ect d
evel
opm
ent
obje
ctiv
e ar
e ge
nera
lly m
oni-
tore
d a
nd e
valu
ated
via
var
i-ou
s pr
ojec
t rep
orts
, sup
ervi
-si
on m
issi
on r
epor
ts, a
ndev
alua
tion
(m
idte
rm a
ndfi
nal)
rep
orts
.3.
Whe
n d
ata
colle
ctio
n is
requ
ired
, spe
cifi
c m
enti
onsh
ould
be
mad
e of
met
hod
san
d r
espo
nsib
iliti
es, w
hich
may
incl
ude
inqu
irie
s fr
ombe
nefi
ciar
ies.
proj
ect a
nd o
utsi
de
of th
ese
ctor
.
(Fro
m P
roje
ct D
evel
opm
ent
Ob
ject
ive
to S
ecto
r-R
elat
edC
AS
Goa
l)•
Ass
umin
g th
at th
e pr
ojec
td
evel
opm
ent o
bjec
tive
isac
hiev
ed, l
ist a
ny a
dd
itio
nal
assu
mpt
ions
nee
ded
to ju
sti-
fy th
e pr
ojec
t’s c
ontr
ibut
ion
to th
e st
ated
goa
l.•
The
se a
ssum
ptio
ns r
efer
toth
e co
ntri
buti
on(s
) of
ad
di-
tion
al p
roje
cts,
ad
dit
iona
lin
puts
, or
add
itio
nal
resp
onse
s fr
om b
enef
icia
rygr
oups
and
inst
itut
ions
that
are
crit
ical
to th
e ac
hiev
e-m
ent o
f th
e st
ated
goa
l.
(Ann
ex 5
con
tinu
es o
n th
e fo
llow
ing
page
.)
02-Evaluating Impacts 5/23/00 5:42 AM Page 205
206
(Fro
m O
utp
uts
to
Pro
ject
Dev
elop
men
t O
bje
ctiv
e)•
Ass
umin
g th
at th
e ou
tput
slis
ted
in th
e fa
r le
ft b
ox a
reac
hiev
ed b
y th
e en
d o
f th
epr
ojec
t, lis
t any
ad
dit
iona
las
sum
ptio
ns n
eed
ed to
achi
eve
the
proj
ect o
bjec
tive
.•
The
se a
ssum
ptio
ns m
ayen
caps
ulat
e co
ndit
ions
, pol
i-cy
cha
nges
, or
expe
cted
beha
vior
s of
ben
efic
iary
grou
ps o
r in
stit
utio
ns th
atar
e ne
cess
ary
for
proj
ect s
uc-
cess
.•
The
se a
ssum
ptio
ns a
recr
itic
al to
the
achi
evem
ent o
fth
e st
ated
pro
ject
obj
ecti
vebu
t are
out
sid
e th
e d
irec
tco
ntro
l of
the
proj
ect.
Pro
ject
Rep
orts
:1.
Out
put i
ndic
ator
s ar
e ge
n-er
ally
mon
itor
ed a
nd e
valu
at-
ed v
ia v
ario
us p
roje
ct r
epor
ts,
supe
rvis
ion
mis
sion
rep
orts
,an
d e
valu
atio
n (m
idte
rm a
ndfi
nal)
rep
orts
.2.
Sou
rces
of
dat
a fo
r m
oni-
tori
ng a
nd e
valu
atin
g th
ese
ind
icat
ors
typi
cally
incl
ude
adm
inis
trat
ive
and
man
age-
men
t rec
ord
-kee
ping
sys
tem
san
d s
umm
ary
repo
rts
gene
r-at
ed b
y th
e pr
ojec
t.
Ou
tpu
t In
dic
ator
s:1.
Out
put i
ndic
ator
s ha
vequ
anti
ty, q
ualit
y, a
nd ti
me
attr
ibut
es. I
f ti
me
is n
ot s
tat-
ed, t
he e
nd o
f pr
ojec
t is
assu
med
.2.
Out
put i
ndic
ator
s ge
nera
l-ly
incl
ude
mea
sure
s of
cos
t-ef
fici
ency
.3.
Col
lect
ion
of d
ata
for
mea
-su
rem
ent o
f ou
tput
ind
ica-
tors
is f
und
ed b
y th
e pr
ojec
t.4.
For
com
plex
pro
ject
s, a
sepa
rate
tabl
e (p
erha
ps a
nad
den
dum
to a
nnex
1)
may
be u
sed
to p
rovi
de
a m
ore
det
aile
d li
stin
g of
out
put
ind
icat
ors.
5. I
t is
bett
er to
hav
e on
ly a
few
mea
ning
ful a
nd e
asily
mea
sure
d o
utpu
t ind
icat
ors
than
an
abun
dan
ce o
f in
di-
cato
rs f
or w
hich
dat
a co
llec-
tion
is p
robl
emat
ic.
Ou
tpu
t fr
om e
ach
Com
pon
ent:
1. S
tate
her
e (i
n th
e pa
stte
nse)
the
valu
e ad
ded
by
the
com
plet
ion
of e
ach
com
-po
nent
.2.
Aco
rrec
t sta
tem
ent o
fou
tput
val
ue a
dd
ed w
ill b
eea
sy to
mea
sure
(as
ref
lect
-ed
in th
e in
dic
ator
s to
the
righ
t).
3. F
or s
impl
icit
y an
d c
lari
tyof
the
logi
c, th
ere
shou
ld b
eon
e ou
tput
sta
tem
ent f
orea
ch c
orre
spon
din
g pr
ojec
tco
mpo
nent
.4.
Eac
h ou
tput
sho
uld
cor
re-
spon
d in
num
ber
to it
sre
spec
tive
com
pone
nt.
5. T
he p
roje
ct te
am is
gen
er-
ally
res
pons
ible
for
ens
urin
gth
e d
eliv
ery
of th
e ou
tput
sas
par
t of
good
pro
ject
des
ign
and
goo
d im
plem
en-
tati
on, p
lann
ing,
and
del
iv-
ery.
An
nex
5 (c
onti
nued
)
02-Evaluating Impacts 5/23/00 5:42 AM Page 206
207
Pro
ject
Com
pon
ents
/S
ub
com
-pon
ents
:1.
Aco
mpo
nent
is a
clu
ster
of s
ubco
mpo
nent
s or
act
ivi-
ties
that
are
des
igne
d to
pro
-d
uce
a si
ngle
pro
ject
out
put.
2. L
ist e
ach
proj
ect c
ompo
-ne
nt a
s a
mai
n he
adin
g, f
ol-
low
ed b
y th
e m
ajor
sub
com
-po
nent
s, if
any
, tha
t are
fund
ed a
s a
part
of
it.
6. T
he o
utpu
t ind
icat
ors
are
agre
ed o
n w
ith
the
borr
ower
dur
ing
PCD
sta
ge (
as to
the
avai
labi
lity
of d
ata
and
eas
eof
col
lect
ion)
, and
a b
asel
ine
is o
btai
ned
pri
or to
app
rais
al.
Pro
ject
In
pu
ts (
bu
dge
t fo
rea
ch c
omp
onen
t):
1. L
ist c
ompo
nent
inpu
ts in
term
s of
the
tota
l cos
t of
each
com
pone
nt in
clud
ing
cont
in-
genc
ies
(for
exa
mpl
e,U
S$__
_).
2. F
or la
rge
or c
ompl
ex p
ro-
ject
s, th
e co
sts
for
subc
ompo
-ne
nts
may
als
o be
sho
wn
(ind
ente
d, t
o se
para
te th
emfr
om th
e co
mpo
nent
cos
ts).
Pro
ject
Rep
orts
:1.
Inp
uts
are
gene
rally
mon
i-to
red
via
pro
gres
s re
port
san
d d
isbu
rsem
ent r
epor
ts(b
oth
quar
terl
y).
2. I
nput
s ar
e ge
nera
lly e
valu
-at
ed v
ia s
uper
visi
on m
issi
onre
port
s (s
emia
nnua
l) a
ndau
dit
rep
orts
(an
nual
).
(Fro
m P
roje
ct C
omp
onen
tsto
Pro
ject
Ou
tpu
ts)
• A
ssum
ing
that
the
com
po-
nent
s an
d a
ctiv
itie
s lis
ted
inth
e fa
r le
ft b
ox a
re im
ple-
men
ted
suc
cess
fully
, lis
t any
add
itio
nal a
ssum
ptio
nsne
eded
to a
chie
ve th
e st
ated
outp
uts.
• T
hese
ass
umpt
ions
are
cond
itio
ns o
utsi
de
the
dir
ect
cont
rol o
f th
e pr
ojec
t and
are
requ
ired
if th
e st
ated
pro
ject
outp
uts
are
to b
e ac
hiev
ed.
• T
he p
roje
ct it
self
sho
uld
not b
e sp
end
ing
mon
ey to
achi
eve
any
of th
ese
cond
i-ti
ons
(sin
ce s
uch
assu
mp-
tion
s ar
e in
clud
ed in
the
com
pone
nts
them
selv
es).
Sour
ce:O
pera
tion
al C
ore
Serv
ices
Dep
artm
ent,
Wor
ld B
ank.
For
com
plet
ed e
xam
ples
of
this
ann
ex, v
isit
the
M&
E H
elp
Des
k on
the
Ban
k’s
inte
rnal
Web
at h
ttp:
//
Lnt
s012
/he
lpd
esk.
nsf.
02-Evaluating Impacts 5/23/00 5:42 AM Page 207
Ann
ex 6
Mat
rix
of A
naly
sis
Nic
arag
ua
Em
erge
ncy
Soc
ial
Inve
stm
ent
Fun
d I
mp
act
Eva
luat
ion
— 1
998
A. P
over
ty T
arge
tin
gIs
sues
Gen
eral
Ind
icat
ors
Met
hodo
logi
esC
omm
ents
Sour
ce o
f dat
a
208
Pove
rty
leve
ls o
f SF
com
mun
itie
s/d
istr
icts
•P
erce
nt o
f ho
use
-ho
lds
in c
omm
uni
tyor
dis
tric
t be
low
pov
erty
line
or
con-
sum
pti
on le
vels
of
extr
eme
poo
r
•M
ean
cons
umpt
ion
leve
l in
soci
al f
und
part
icip
ant c
omm
uni-
ties
ver
sus
cons
ump-
tion
leve
l in
coun
try
•Po
vert
y m
ap in
dex
(as
used
by
SF)
Req
uire
s ho
useh
old
inco
me
and
con
sum
ptio
nsu
rvey
and
iden
tifi
cati
onof
SF
acti
viti
es b
y co
m-
mun
ity
and
dis
tric
t
Req
uire
s ho
useh
old
inco
me
and
con
sum
ptio
nsu
rvey
and
iden
tifi
cati
onof
SF
acti
viti
es b
y co
m-
mun
ity
and
dis
tric
t
Map
s us
ually
use
pro
xym
easu
res,
like
a c
ompo
s-
To c
ompa
re a
cros
s co
un-
trie
s, n
eed
sim
ilar
def
in-
itio
ns o
f po
vert
y lin
es
Dis
adva
ntag
es a
re th
at it
arbi
trar
ily c
hoos
es in
di-
Ove
rsam
plin
g na
tion
alho
useh
old
sur
vey
(LSM
S) in
SF
com
mun
i-ti
es—
only
for
ed
uca-
tion
, hea
lth,
wat
er, a
ndsa
nita
tion
pro
ject
s
Ove
rsam
plin
g na
tion
alho
useh
old
sur
vey
(LSM
S) in
SF
com
mun
i-ti
es—
only
for
ed
uca-
tion
, hea
lth,
wat
er, a
ndsa
nita
tion
pro
ject
s
SF u
ses
pove
rty
map
base
d o
n L
SMS9
3 d
ata
02-Evaluating Impacts 5/23/00 5:42 AM Page 208
209
Pove
rty
leve
ls o
f SF
bene
fici
arie
s (h
ouse
-ho
ld le
vel)
•Pe
rcen
t of
bene
fici
a-ri
es b
elow
pov
erty
line
or in
ext
rem
epo
vert
y
•M
ean
cons
umpt
ion
leve
l of
bene
fici
ary
hous
ehol
ds
vers
usna
tion
al a
vera
ge f
orsi
mila
r ho
useh
old
spe
r pr
ojec
t typ
e (f
orex
ampl
e, w
ith
chil-
dre
n in
pri
mar
ysc
hool
, wit
h ac
cess
topi
ped
wat
er, w
ho u
sela
trin
es)
ite
pove
rty
ind
ex b
ased
on m
ix o
f va
riab
les
Inco
me
and
con
sum
p-
tion
su
rvey
tha
t p
icks
up
SF
bene
fici
arie
sei
ther
by
size
of
SF o
r by
over
sam
pli
ng in
SF
com
-m
uni
ties
Inco
me
and
con
sum
ptio
nsu
rvey
that
pic
ks u
p SF
bene
fici
arie
s ei
ther
by
virt
ue o
f si
ze o
f SF
or
byov
ersa
mpl
ing
in S
F co
m-
mun
itie
s; c
an a
lso
over
-sa
mpl
e in
“m
atch
” co
m-
mun
itie
s w
itho
ut S
Fin
terv
enti
ons
cato
rs a
nd w
eigh
ts, a
ndea
ch c
ount
ry h
as d
iffer
-en
t ind
ex. A
dva
ntag
e is
that
it o
ften
pro
vid
esm
ore
geog
raph
ical
dis
-ag
greg
atio
n th
an in
com
ean
d c
onsu
mpt
ion
sur-
veys
—th
e tw
o ca
n be
linke
d to
der
ive
pred
ict-
ed c
onsu
mpt
ion
leve
ls a
td
issa
greg
gate
d le
vels
May
var
y w
idel
y by
SF
proj
ect t
ype
usin
g co
mpo
site
pov
er-
ty in
dex
; will
upd
ate
usin
g L
SMS9
8 an
dC
ensu
s d
ata
to p
red
ict
cons
umpt
ion
at s
ubna
-ti
onal
leve
ls
Ove
rsam
plin
g na
tion
alho
useh
old
sur
vey
(LSM
S) in
SF
com
mun
i-ti
es
Ove
rsam
plin
g na
tion
alho
useh
old
sur
vey
(LSM
S) in
SF
com
mun
i-ti
es
(Mat
rix
cont
inue
s on
the
follo
win
g pa
ge.)
02-Evaluating Impacts 5/23/00 5:42 AM Page 209
210
Dis
trib
utio
n of
SF
reso
urce
s
Inst
itut
iona
l des
ign
feat
ures
that
aff
ect
SF ta
rget
ing
perf
or-
man
ce
Oth
er f
acto
rs a
ffec
t-in
g ta
rget
ing
perf
or-
man
ce
•Pe
rcen
t of
SF p
roje
cts
and
res
ourc
es in
bot
-to
m q
uint
ile o
f d
is-
tric
ts
•U
se o
f po
vert
y m
ap•
Prom
otio
nal e
ffor
ts•
Dir
ect a
cces
s by
ben
e-fi
ciar
y gr
oups
•Sh
are
of p
roje
cts
byre
ques
ting
age
ncy
•D
ecen
tral
ized
off
ices
•Ta
rget
res
ourc
e al
loca
-ti
ons
•Su
bpro
ject
men
u
•A
ge o
f SF
•“S
ocia
l cap
ital
” of
com
mun
ity
•D
ista
nce
to S
F he
ad-
quar
ters
•H
ighe
st e
duc
atio
nle
vel o
f be
nefi
ciar
ies
•Pr
esen
ce o
f go
vern
-m
ent o
r N
GO
inte
r-ve
ntio
ns
Nee
d c
onsi
sten
t ran
king
met
hod
olog
y ac
ross
coun
trie
s
Dev
elop
sta
ndar
d in
stit
u-ti
onal
var
iabl
es th
at c
anbe
use
d to
exp
lain
targ
et-
ing
outc
omes
—va
riab
les
easi
ly o
btai
ned
fro
m S
Fs
Als
o ne
ed s
tand
ard
def
i-ni
tion
s fo
r va
riab
les—
vari
able
s ob
tain
ed f
rom
SFs,
hou
seho
ld s
urve
ys(w
ith
iden
tifi
cati
on o
f SF
bene
fici
arie
s), a
nd n
atio
n-al
sur
veys
Nee
d to
rev
iew
ran
king
syst
em a
nd r
ecal
ibra
te
Info
rmat
ion
avai
labl
efr
om S
F
Onl
y in
dic
ator
in d
oubt
is th
e “s
ocia
l cap
ital
of
com
mun
ity”
Mat
rix
(con
tinu
ed)
Issu
esG
ener
al I
ndic
ator
sM
etho
dolo
gies
Com
men
tsSo
urce
of d
ata
02-Evaluating Impacts 5/23/00 5:42 AM Page 210
211
Com
pari
son
of a
lter
-na
tive
s
Phys
ical
cap
ital
•D
egre
e of
cou
ntry
inco
me
ineq
ualit
y
•Pe
rcen
t of
SF p
roje
cts
and
res
ourc
es in
bot
-to
m q
uint
ile o
f d
is-
tric
ts v
ersu
s ot
her
com
para
ble
prog
ram
san
d d
eliv
ery
mec
ha-
nism
s
•E
xten
t to
whi
ch s
ub-
proj
ects
res
pond
toco
mm
unit
y pr
iori
ties
•B
enef
icia
ry p
erce
ptio
nof
ben
efit
leve
l and
impr
ovem
ents
to w
el-
fare
•Im
prov
emen
t in
acce
ssto
soc
ial a
nd e
cono
mic
Com
pare
targ
etin
g pe
r-fo
rman
ce b
ased
on
geo-
grap
hica
l loc
atio
n or
pove
rty
leve
ls o
f be
nefi
-ci
arie
s, d
epen
din
g on
surv
ey d
esig
n, s
cale
of
SFan
d o
ther
pro
gram
s
Com
mun
ity-
leve
l sur
vey,
bene
fici
ary
asse
ssm
ent,
or h
ouse
hold
sur
vey
wit
hov
ersa
mpl
es in
SF
area
s
Hou
seho
ld s
urve
y or
bene
fici
ary
asse
ssm
ent i
nSF
com
mun
itie
s
Hou
seho
ld s
urve
y of
SF
bene
fici
arie
s
Diff
icul
t to
find
via
ble
com
para
tors
; nee
d s
epa-
rate
info
rmat
ion
gath
er-
ing
from
com
para
tor
prog
ram
s
Nee
d to
hav
e ei
ther
base
line
or r
ecal
l que
s-
Plan
ned
for
cos
t-ef
fici
ency
ana
lysi
s
Cov
ered
in I
DB
-fi
nanc
ed B
enef
icia
ryA
sses
smen
t and
Faci
litie
s Su
rvey
Cov
ered
in h
ouse
hold
surv
ey a
nd I
DB
-fi
nanc
ed B
enef
icia
ryA
sses
smen
t
Som
e re
call
ques
tion
sin
hou
seho
ld s
urve
y,
B. B
enef
its
Dat
a so
urce
s/C
ase
Stud
y:Is
sues
Gen
eral
Ind
icat
ors
met
hodo
logi
esC
omm
ents
Nic
arag
ua
(Mat
rix
cont
inue
s on
the
follo
win
g pa
ge.)
02-Evaluating Impacts 5/23/00 5:42 AM Page 211
212
Mat
rix
(con
tinu
ed)
Dat
a so
urce
s/C
ase
Stud
y:Is
sues
Gen
eral
Ind
icat
ors
met
hodo
logi
esC
omm
ents
Nic
arag
ua
infr
astr
uctu
re (
befo
rean
d a
fter
)
•Im
prov
emen
t in
acce
ssto
soc
ial a
nd e
cono
mic
infr
astr
uctu
re v
ersu
sco
mpa
rato
r pr
ojec
ts
•Im
prov
emen
t in
qual
i-ty
of
infr
astr
uctu
re a
ndse
rvic
es (
befo
re a
ndaf
ter)
•Im
prov
emen
t in
qual
i-ty
of
infr
astr
uctu
re a
ndse
rvic
es v
ersu
s co
m-
para
tor
proj
ects
Hou
seho
ld s
urve
y of
SF
and
com
para
tor
proj
ect
bene
fici
arie
s
Faci
litie
s su
rvey
and
hous
ehol
d s
urve
y, s
ome
cove
rage
fro
m b
enef
icia
-ry
ass
essm
ent
Faci
litie
s su
rvey
and
hous
ehol
d s
urve
y, s
ome
cove
rage
fro
m b
enef
icia
-ry
ass
essm
ent (
in S
F an
dco
mpa
rato
rs)
tion
s; n
eed
to d
evel
opse
para
te in
dic
ator
s pe
rty
pe o
f SF
pro
ject
Nee
d to
hav
e ei
ther
base
line
or r
ecal
l que
s-ti
ons;
nee
d to
dev
elop
sepa
rate
ind
icat
ors
per
type
of
SF p
roje
ct
Nee
d to
hav
e ei
ther
base
line
or r
ecal
l que
s-ti
ons;
nee
d to
dev
elop
sepa
rate
ind
icat
ors
per
type
of
SF p
roje
ct
Nee
d to
hav
e ei
ther
base
line
or r
ecal
l que
s-ti
ons;
nee
d to
dev
elop
sepa
rate
ind
icat
ors
per
type
of
SF p
roje
ct
also
pos
sibl
e ex
ant
efr
om p
revi
ous
LSM
S;ca
n co
mpa
re S
F be
nefi
-ci
ary
wit
h ch
arac
teri
s-ti
cs o
f na
tion
al p
opul
a-ti
on a
nd w
ith
mat
chco
mm
unit
ies
Can
com
pare
SF
bene
fi-
ciar
y w
ith
gene
ral c
har-
acte
rist
ics
of n
atio
nal
popu
lati
on a
s w
ell a
sm
atch
com
mun
itie
s
For
educ
atio
n, h
ealt
h,w
ater
and
san
itat
ion,
reca
ll pl
us h
isto
rica
lin
form
atio
n fr
om f
acili
-ti
es s
urve
y an
d S
F ex
ante
app
rais
al
For
educ
atio
n, h
ealt
h,w
ater
and
san
itat
ion,
SF a
nd n
on-S
F fa
cilit
ies
thro
ugh
hous
ehol
d a
ndfa
cilit
ies
surv
eys
02-Evaluating Impacts 5/23/00 5:43 AM Page 212
213
Hum
an c
apit
al•
Impr
oved
ed
ucat
iona
lst
atus
: sch
ool a
tten
-d
ance
, yea
rs c
ompl
et-
ed, d
ropo
ut a
nd r
eten
-ti
on r
ates
(be
fore
and
afte
r an
d v
ersu
s co
m-
para
tors
)
•Im
prov
ed h
ealt
h st
a-tu
s: f
or e
xam
ple,
inci
-d
ence
of
dis
ease
, inf
ant
mor
talit
y, m
alnu
trit
ion,
incr
ease
d b
reas
tfee
din
g(b
efor
e an
d a
fter
and
vers
us c
ompa
rato
rs)
•Im
prov
ed e
cono
mic
stat
us: i
ncre
ased
inco
me,
red
uced
tim
esp
ent f
etch
ing
wat
er,
low
er c
ost o
f se
rvic
es,
incr
ease
d e
mpl
oym
ent
(bef
ore
and
aft
er a
ndve
rsus
com
para
tors
)
Hou
seho
ld s
urve
y an
din
form
atio
n fr
om s
choo
l
Hou
seho
ld s
urve
y w
ith
heal
th m
odul
e.A
nthr
opom
etri
c m
ea-
sure
s if
mal
nutr
itio
nin
clud
ed
Hou
seho
ld s
urve
y
Hou
seho
ld s
urve
y an
din
form
atio
n fr
omsc
hool
for
SF
and
non
-SF
sch
ools
and
hou
se-
hold
s
Hou
seho
ld s
urve
y an
din
form
atio
n fr
om h
ealt
hce
nter
for
SF
and
non
-SF
cen
ters
and
hou
se-
hold
s
Hou
seho
ld s
urve
y fo
rSF
-ben
efic
iary
and
non
-be
nefi
ciar
y m
atch
com
-m
unit
ies
(Mat
rix
cont
inue
s on
the
follo
win
g pa
ge.)
02-Evaluating Impacts 5/23/00 5:43 AM Page 213
214
Soci
al c
apit
al
Sust
aina
bilit
y of
oper
atio
ns
•In
crea
sed
com
mun
ity
capa
city
to a
dd
ress
prob
lem
s (v
ersu
s co
m-
para
tors
)
•In
crea
sed
par
tici
pati
onra
tes
in c
omm
unit
y-in
itia
ted
cha
nges
(ve
r-su
s co
mpa
rato
rs)
•C
ond
itio
ns u
nder
whi
ch S
F pr
ojec
ts a
reop
erat
ing
afte
r SF
inte
r-ve
ntio
n (a
bsol
ute
sus-
tain
abili
ty)
•C
ond
itio
ns u
nder
whi
ch S
F pr
ojec
ts a
reop
erat
ing
afte
r SF
inte
r-ve
ntio
n ve
rsus
com
-pa
rato
r pr
ojec
ts (
rela
-ti
ve s
usta
inab
ility
)
Hou
seho
ld s
urve
y, c
om-
mun
ity
surv
ey, a
nd/
orbe
nefi
ciar
y as
sess
men
t
Hou
seho
ld s
urve
y, c
om-
mun
ity
surv
ey, o
r be
nefi
-ci
ary
asse
ssm
ent
Faci
litie
s su
rvey
Faci
litie
s su
rvey
Nee
d to
dev
elop
ind
ica-
tors
Can
get
som
e ad
dit
iona
lin
form
atio
n fr
om b
enef
i-ci
ary
asse
ssm
ent
Can
get
som
e ad
dit
iona
lin
form
atio
n fr
om b
enef
i-ci
ary
asse
ssm
ent
Not
ad
dre
ssed
Info
rmat
ion
in h
ouse
-ho
ld s
urve
y on
par
tici
-pa
tion
For
educ
atio
n an
dhe
alth
pro
ject
sur
veys
,ha
ve b
oth
SF a
nd n
on-
SF For
educ
atio
n an
dhe
alth
pro
ject
sur
veys
,ha
ve b
oth
SF a
nd n
on-
SF
Mat
rix
(con
tinu
ed)
Dat
a so
urce
s/C
ase
Stud
y:Is
sues
Gen
eral
Ind
icat
ors
met
hodo
logi
esC
omm
ents
Nic
arag
ua
C. S
ust
ain
abil
ity
of B
enef
its
Dat
a so
urce
s/C
ase
Stud
y:Is
sues
Gen
eral
Ind
icat
ors
met
hodo
logi
esC
omm
ents
Nic
arag
ua
02-Evaluating Impacts 5/23/00 5:43 AM Page 214
215
Can
get
som
e ad
dit
iona
lin
form
atio
n fr
om b
enef
i-ci
ary
asse
ssm
ent
Can
get
som
e ad
dit
iona
lin
form
atio
n fr
om b
enef
i-ci
ary
asse
ssm
ent
Can
get
som
e ad
dit
iona
lin
form
atio
n fr
om b
enef
i-ci
ary
asse
ssm
ent
For
educ
atio
n an
dhe
alth
pro
ject
sur
veys
,ha
ve b
oth
SF a
nd n
on-
SF For
educ
atio
n an
dhe
alth
pro
ject
sur
veys
,ha
ve b
oth
SF a
nd n
on-
SF For
educ
atio
n an
dhe
alth
pro
ject
sur
veys
,ha
ve b
oth
SF a
nd n
on-
SF Wou
ld n
eed
ad
dit
iona
lw
ork
Incl
ude
in n
ext
bene
fi-
ciar
y as
sess
men
t;im
pact
eva
luat
ion
Faci
litie
s su
rvey
Faci
litie
s su
rvey
Faci
litie
s su
rvey
and
hous
ehol
d s
urve
y
SF d
atab
ase,
com
mun
ity
surv
ey, o
r be
nefi
ciar
yas
sess
men
t
Com
mun
ity
surv
ey,
hous
ehol
d s
urve
y, o
rbe
nefi
ciar
y as
sess
men
t
•M
aint
enan
ce o
f in
fra-
stru
ctur
e an
d s
ervi
ces
over
tim
e (a
bsol
ute)
•M
aint
enan
ce o
f in
fra-
stru
ctur
e an
d s
ervi
ces
over
tim
e ve
rsus
com
-pa
rato
r pr
ojec
ts (
rela
-ti
ve)
•Q
ualit
y an
d q
uant
ity
ofin
fras
truc
ture
and
ser
-vi
ces
over
tim
e
•Te
nden
cy o
f SF
com
mu-
niti
es to
sub
mit
oth
erpr
opos
als
(to
SF a
ndot
hers
) ov
er ti
me
•C
omm
unit
y pa
rtic
ipa-
tion
in s
ocia
l and
eco
-no
mic
infr
astr
uctu
rene
eds
over
tim
e
Sust
aina
bilit
y of
mai
nten
ance
Sust
aina
bilit
y of
impa
ct
Sust
aina
bilit
y of
com
mun
ity
effe
cts
(Mat
rix
cont
inue
s on
the
follo
win
g pa
ge.)
02-Evaluating Impacts 5/23/00 5:43 AM Page 215
216
Mat
rix
(con
tinu
ed)
D. C
ost-
Eff
icie
ncy
Iss
ues
Dat
a so
urce
s/C
ase
Stud
y:Is
sues
Gen
eral
Ind
icat
ors
met
hodo
logi
esC
omm
ents
Nic
arag
ua
Cos
t-ef
fici
ency
of
subp
roje
cts
•A
vera
ge c
ost p
er n
ewsc
hool
, hea
lth
post
,w
ater
sys
tem
ver
sus
alte
rnat
ive
appr
oach
esve
rsus
com
para
tor
pro-
ject
s
•U
nit c
osts
: cos
t per
squa
re m
eter
of
con-
stru
ctio
n, p
er k
ilom
eter
of r
oad
, and
so
on, v
er-
sus
com
para
tor
proj
ects
•A
vera
ge c
ost p
er b
enef
i-ci
ary
per
SF p
roje
ct ty
peve
rsus
com
para
tors
•A
vera
ge c
ost o
f em
ploy
-m
ent g
ener
ated
ver
sus
com
para
tors
SF d
atab
ase
and
info
r-m
atio
n fr
om g
over
nmen
tm
inis
trie
s an
d m
unic
ipal
gove
rnm
ents
SF d
atab
ase
and
info
r-m
atio
n fr
om g
over
nmen
tm
inis
trie
s an
d m
unic
ipal
gove
rnm
ents
SF d
atab
ase
and
info
r-m
atio
n fr
om g
over
nmen
tm
inis
trie
s an
d m
unic
ipal
gove
rnm
ents
SF d
atab
ase
and
info
r-m
atio
n fr
om g
over
nmen
tm
inis
trie
s an
d m
unic
ipal
gove
rnm
ents
Cos
ts c
hang
e ov
er ti
me
and
com
para
tor
proj
ects
mus
t be
iden
tica
l
SF a
nd n
on-S
F d
ata
from
fac
iliti
es s
urve
y.N
on-S
F co
st e
stim
ates
may
not
be
relia
ble
Can
cal
cula
te S
F av
er-
ages
. Will
incl
ude
inco
st-e
ffic
ienc
y an
alys
is
Can
cal
cula
te S
F av
er-
ages
. Will
incl
ude
inco
st-e
ffic
ienc
y an
alys
is
Can
cal
cula
te S
F av
er-
ages
. Will
incl
ude
inco
st-e
ffic
ienc
y an
alys
is
02-Evaluating Impacts 5/23/00 5:43 AM Page 216
217
Can
cal
cula
te S
F av
er-
ages
. Will
incl
ude
inco
st-e
ffic
ienc
y an
alys
is
Can
cal
cula
te S
F av
er-
ages
. Will
incl
ude
inco
st-e
ffic
ienc
y an
alys
is
Nee
d to
dev
elop
sta
n-d
ard
def
init
ions
of
inst
i-tu
tion
al c
osts
; spe
cify
tim
e pe
riod
SF d
atab
ase
and
info
r-m
atio
n fr
om g
over
n-m
ent m
inis
trie
s an
dm
unic
ipal
gov
ernm
ents
SF d
atab
ase
and
info
r-m
atio
n fr
om g
over
n-m
ent m
inis
trie
s an
dm
unic
ipal
gov
ernm
ents
•SF
inst
itut
iona
l cos
ts(i
nves
tmen
t and
ope
rat-
ing)
as
shar
e of
SF
pro-
ject
s ve
rsus
com
para
tor
proj
ects
•A
vera
ge c
ompl
etio
nti
me
vers
us c
ompa
rato
rpr
ojec
ts
Cos
t eff
icie
ncy
ofd
eliv
ery
mec
hani
sm
02-Evaluating Impacts 5/23/00 5:43 AM Page 217