Sustained Use Zim BSF Report

8/3/2019 Sustained Use Zim BSF Report

1/37

Evaluation of Oxfam Biosand Filter

Cholera Emergency Response Program

Mudzi, Zimbabwe

April 2011

Daniele Lantagne and Fungai Makoni


2/37

Page 2 of 38

Executive SummaryThe Zimbabwean cholera outbreak is an ongoing cholera epidemic that began in August 2008. As of January 10,2010 there had been 98,741 reported cases and 4,293 deaths, making it the deadliest African cholera outbreak inthe last 15 years. Oxfam/America (OA) and Oxfam/Great Britian (OGB) jointly responded to the epidemic alongwith the local partner organization, Single Parents and Widow(ers) Support Network (SPWSNet). Biosand Filters,slow-sand filters adapted for use in the home, were distributed to almost 1,000 households to address medium tolong-term water needs of the households. To use the Biosand Filter, users simply pour water into the filter andcollect finished water out of the outlet pipe into a bucket. Although Biosand Filters have been successfullyimplemented in the development context, there is little evidence of program success in the humanitarian context.Thus, although the program was established in response to a cholera outbreak, it was also viewed as a pilotprogram to test the introduction and use of the Biosand Filter technology in a humanitarian context. As such, the

program was evaluated at numerous stages throughout the program and upon program completion.

The objectives of this sustained use report, which was conducted four months after the program closed, were toprovide an independent assessment on: 1) the efficacy and effectiveness of the Biosand Filters for the choleraresponse; and, 2) a process evaluation (for learning purposes). A mixed-methodology investigation was completedto evaluate the program, including: 1) household surveys to understand Biosand Filter knowledge, use, andsustained use; 2) water quality testing to document Biosand Filter effectiveness; and, 3) key informant interviewswith program staff to characterize the response.

A total of 61 beneficiary households were interviewed, with complete surveys and water quality samples

collected in 29 families receiving filters with plastic housing and 29 families receiving filters with concretehousing. Overall, 56% of surveyed beneficiaries reported having Biosand Filter treated water in the household atthe time of the unannounced survey visit.

Results from multiple evaluations of this program (including this one) confirm the extensive worldwide data setthat the use of Biosand Filters improves the microbiological quality of stored household drinking water. Thepercentage of the population that had drinking water contaminated drinking water before treatment with theBiosand Filter and uncontaminated after treatment was 63% in the M&E Coordinator evaluation, 43% in thefollow-up evaluation, and 56% in this sustained use evaluation. It can be assumed the program reduces diarrhealdisease incidence as research studies have shown that diarrheal disease incidence decreases with (evenincomplete) microbiological reduction through biosand filtration.

A percentage of the surveyed population that is effectively using the intervention to treat contaminated water tomicrobiologically safe levels can be calculated by: multiplying the percentage of the surveyed population thatreported using the filter by the percentage of the reported users whose household stored drinking water wasimproved from contaminated to uncontaminated by using the filter. In this sustained use evaluation, 56% of surveyed households reported using the filter, and 56% of reported users had household stored drinking water thatwas improved from contaminated to uncontaminated by use of the filter. Thus, 31% (56% multiplied by 56%) of surveyed households were using the biosand filters effectively to reduce stored household water contamination atthe time of the unannounced survey visit. This is the highest effective use percentage the consultant has seen forthe use of biosand filters in the emergency context in her evaluations with UNICEF and Oxfam to date.


3/37

Page 3 of 38

The importance of adequate training on BSF use was consistently identified in this evaluation. First, training wasassociated with increased use of the filter, as respondents who reported receiving more than one training of anytype were more likely to report treated water at the time of the unannounced survey visit than those who reportedreceiving one training or less. However, there were significant gaps identified in households applying theirtraining to correct installation and maintenance of their Biosand Filter, as:

The majority of beneficiaries (68%) did not have the appropriate standing water depth (4-6 cm) tocorrectly maintain the biolayer (schmutzdecke) in their filter. Of particular concern is the high percentage(14% each) of beneficiaries at the extreme ends with either 0 cm or ! 10 cm. Having the correctstanding water depth was not correlated with any training or assistance from volunteer or SPWSNetindicating that training was not sufficient to ensure adequate installation.

The majority of respondents reported cleaning the filter in some way (84%). However, the cleaningstrategies were inconsistent, with a variety of inappropriate methods (from removing all the sand tocleaning only the sand at the top to cleaning the outside only to reinstalling the whole filter) employedirregularly by households.

The vast majority of beneficiaries use the water for only drinking and cooking (82%). Given the fast flowrate in the Biosand Filters, it is recommended to encourage households to use the water for all purposes.

These user training operations and maintenance difficulties, which may: 1) lead to the misperception inhouseholds that they are consuming safe water when in fact they are not; and, 2) inhibit the long-term sustaineduse of the filter.

The data suggest that there are differences in the effectiveness and efficiency of the plastic and concrete BiosandFilters. It is not clear at this point how much the differences are due to innate differences between the filters stylesand how much is due to the fact concrete filters were distributed later than plastic ones, with improved education

and training. However, it does appear that concrete filters were less likely to crack.

Based on the small number of families confirmed to be reached with a Biosand Filter (only 897), and the timeperiod after the cholera in which they were reached, it can not be stated that the Biosand Filters were anappropriate intervention for the program goal of responding to the cholera. A total of 69% of surveyedrespondents who received Biosand Filters also received Aquatabs as a cholera response household watertreatment option from (an)other organization(s). Aquatabs are less expensive and easier to distribute, simpler touse, and more effective than Biosand Filters at reducing the cholera bacteria from household stored drinkingwater. However, Aquatabs are a consumable product, not a durable, and thus they only provide protection whilethe household uses the products until they run out.

The key evaluation question for Biosand Filters is no longer on microbiological effectiveness, but instead onappropriate implementation strategies that result in users effectively using the filters to improve the quality of theirstored household water in a cost-effective manner. Should OA wish to continue with Biosand Filterimplementations in emergencies, it is recommended that OA:

Coordinate future programs with the Government of Zimbabwe. The focus of future programs should notbe on confirming microbiological effectiveness or health impact (which have been well-verified) butinstead on establishing cost-effective distribution strategies that reach a large population with the trainingnecessary to encourage consistent and correct use.

Understand the potentially limited role of Biosand Filters in the acute emergency situation. Biosand Filterstake time to distribute, train on, and for the schmutzdecke to develop. A chlorine-based product should


4/37

Page 4 of 38

be distributed in the emergency response time until such time as there is capacity to complete thistraining and development.

Complete significant training with beneficiaries , that teaches how to: a) adequately install the filter,including establishing the correct standing water layer; b) maintain the filter, including where to obtainreplacement parts; c) how to clean the filter appropriately and on what schedule; and, d) highlights thatfiltered water can be used for more than just drinking and cooking.

Keep program design and scope realistic and considered . In the future, research plans should be morefully integrated into implementation.


5/37

Page 5 of 38

Table of Contents

Table of Contents !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"

1 Introduction !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!#

$ Previous research !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! %

2.1 OA program !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!%

2.2 Government of Zimbabwe draft summary report !!!!!!!!!!!!!!!!!!!&$

$!' Sustained use report !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!&$

' Methodology !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!&(

3.1 Household surveys !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! &(

3.2 Water quality testing !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! &(

3.3 Key informant interviews !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! &(

( Survey Results !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! &"

" Process Evaluation Results !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! $)

* Discussion and Recommendations !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!$'

6.1 Independent assessment on efficacy and effectiveness of theBiosand Filters for the cholera response !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! $'

6.2 Process evaluation !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!$(

6.3 Recommendations !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!$*

Annex A: Terms of Reference !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!$#

Annex B: Itinerary and survey respondents !!!!!!!!!!!!!!!!!!!!!!!!!!!!'$

Annex C: Household questionnaire !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ''

Annex D: Water quality indicators !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! '#


6/37

Page 6 of 38

List of Figures and Tables

Figure 1: OA program timeline of evaluations ................................................................................................ 8Figure 2: M&E Coordinator paired treated/untreated water E. coli samples .................................................. 10Figure 3: Project-end evaluation paired treated/untreated water E. coli samples.......................................... 11Figure 4: Standing water level in filters (percent of samples in ranges, by centimeter)................................. 17Figure 5: Sustained use evaluation paired treated/untreated water E. coli samples ...................................... 18Figure 6: An unused filter with iron staining, a filter used for storing tomatoes, a correctly installed and

used filter................................................................................................................................................ 19

Table 1: Household characteristic results ...................................................................................................... 16Table 2: Training received by respondents for plastic and concrete containers ........................................... 17Table 3: Cleaning strategies for plastic and concrete filters .......................................................................... 18Table 4: Water quality data ........................................................................................................................... 37

Abbreviations

BSF Biosand FilterCFU Colony Forming UnitFR Female RespondentM&E Monitoring & EvaluationNFI Non-food ItemsNGO Non-governmental OrganizationOA Oxfam/AmericaOGB Oxfam/Great BritainSPWSNet Single Parents and Widow(ers) Support Network

AcknowledgementsThe authors would like to thank all those involved in the planning and execution of this study, particularly theOA/Boston, OA/Zimbabwe, SPWSNet staff, and enumerators. Without their support this report would not have

been possible.


7/37

Page 7 of 38

1 Introduction

The Zimbabwean cholera outbreak is an ongoing cholera epidemic that began in August 2008. As of January 10,2010 there had been 98,741 reported cases and 4,293 deaths, making it the deadliest African cholera outbreak inthe last 15 years. Mudzi District, in north-eastern Zimbabwe on the Mozambican border, was one of the mostaffected districts. Although cholera is a yearly occurrence in Zimbabwe, the outbreak beginning in 2008 wasworse because it started before the rainy season, affected a large number of people, had a mortality rate, and alsooccurred in urban environments previously spared from yearly outbreaks.

Oxfam/America (OA) and Oxfam/Great Britian (OGB) jointly responded to the epidemic along with the localpartner organization, Single Parents and Widow(ers) Support Network (SPWSNet). The initial response came inthe form of borehole repair, distribution of non-food items, health and hygiene education, coordination, andsupply of oral rehydration salts. A preliminary assessment was also undertaken on the overall situation and thegroundwork was laid for the development of a diarrhea early warning system and the introduction of the BiosandFilters to address the medium to long-term needs of the households.

The Biosand Filter is a slow-sand filter adapted for use in the home. The version of the Biosand Filter most widelyimplemented consists of layers of sand and gravel in a concrete or plastic or concrete container approximately 0.9meters tall, and 0.3 meters square. The water level is maintained to 4-6 cm above the sand layer by setting theheight of the outlet pipe. This shallow water layer allows a bioactive layer (schmutzdecke) to grow on top of thesand. Diarrheal disease causing organisms are removed through mechanical trapping, adsorption, and predation

by schmutzdecke organisms. A diffuser plate with holes in it is placed on the top of the sand layer to preventdisruption of the biolayer when water is added to the system. To use the Biosand Filter, users simply pour waterinto the filter, and collect finished water out of the outlet pipe into a bucket. Although Biosand Filters are widelyand successfully implemented in the development context, there is little evidence of program success in thehumanitarian context.

OA implemented a pilot program to distribute Biosand Filters to 950 families in Mudzi District in April 2009. OApurchased 450 pre-fabricated plastic filter housings from Hydraid in the United States and commissioned themanufacture of 500 locally-made concrete filter housings in Zimbabwe. Working with SPWSNet, the filters weredistributed to families in stages, accompanied by filter use and maintenance training coupled with public health

and hygiene education.

Although the program was established in response to a cholera outbreak in north-eastern Zimbabwe, it was alsoviewed as a pilot program to test the introduction and use of the Biosand Filter technology in a humanitariancontext. As such, the program was evaluated at numerous stages throughout the program and upon programcompletion. The main purpose of this post-program sustained use evaluation is to draw lessons from thisexperience on the viability of the technology, effective management arrangements, and minimum conditionsnecessary for sustainable use of the filters by communities. The Terms of Reference for this sustained useevaluation are appended in Annex A. In this report, previous research on Biosand Filters in Zimbabwe is firstsummarized, then results from the sustained use evaluation are presented, and lastly conclusions andrecommendations are presented.


8/37

Page 8 of 38

$ Previous research

Numerous organizations, including OA, have been interested in implementing and researching Biosand Filterprograms in the development and emergency situation in Zimbabwe. In this section the research completed todate on Biosand Filters in Zimbabwe is briefly summarized, including the OA program and a summary report of results from NGO programs using the Biosand Filter in Zimbabwe from the Government of Zimbabwe. The Termsof Reference for this sustained use evaluation are summarized at the end of this section and fully appended inAnnex A.

2.1 OA program

The OA Biosand Filter cholera response and research program began in April 2009 and concluded in September2010 (Figure 1). Monitoring and evaluation occurred throughout this program, as: 1) the Biosand Filter technologywas new for OA, SPWSNet, and the communities and adjustments and improvements were regularly neededduring program implementation; and, 2) part of the intention of the program was to learn lessons for potentialexpansion.

This report documents the fourth major evaluation of this program (Figure 1). The first evaluation, the initial distribution evaluation , was completed by this consultant in July 2009. The second evaluation, the ongoing evaluation , was performed by an OA/Zimbabwe Monitoring and Evaluation (M&E) Coordinator in December. Thethird evaluation, a program-end evaluation , was conducted by OA/Zimbabwe in September 2010. The evaluation

described in this report, the sustained use evaluation , was completed in January 2011. The results of theseevaluations are summarized in the following sections.

Figure 1: OA program timeline of evaluations

$!&!& Initial distribution evaluations

At this initial evaluation, 531 filters had been distributed in 117 communities across six wards of Mudzi. Thisaveraged to only 4.5 filters distributed per community. Although a large number of filters had been distributed tofamilies in a short amount of time, only 3 (17%) of 17 randomly selected households had a correctly assembled

filter they were using at the time of the unannounced visit. Problems were noted with maintaining the correctwater level of 5 cm above the sand to ensure schmutzdecke development and with storage of filtered water in

201120102009

4 5 6 7 8 9 1 0 1 1 1 2 1 2 3 4 5 6 7 8 9 1 0 1 1 1 2 1

P r o j e c t s t a r t

V i s i t b y

O x f a m / A m e r i c a

F i r s t d i s t r i b u t i o

n s

I n i t i a l D i s t r i b u t i o n E v a l u a t i o n

V i s i t b y


E d u c a t i o

n a n d c o v e r e d b u c k e t s

p r o v i d e d

M o n i t o

r i n g a n d E v a l u a t i o

n p e r s o n

o n - b o a r d

T w o v i s i t s b y


F i n a l E v a l u a t i o n a n d

P r o j e c t C l o s e

S u s t a i n e d U s e E v a l u a t i o

n


9/37

Page 9 of 38

open containers where recontamination is possible. Based on these results it was recommended to: 1) delaydistribution of remaining filters until expansion and retraining of the Ward Volunteer network occurred; and, 2)visit each existing filter household and ensure the filter is correctly assembled, provide an informational poster,answer user questions, ensure a covered container is being used for water storage, and collect a GPS point.

Additionally, it was noted that the program was attempting to be many things at once a pilot program, aresearch program, an implementation program, a program targeting the most needy people, a program comparingtwo types of filters all in an aggressive time frame in an emergency context. The recommended next steps wereto: 1) determine how to ensure correct usage of the filters by providing appropriate training; and, 2) determinerealistic outcome measures that can be obtained from this program.

Based on the above evaluation, numerous changes were implemented, including: distribution was delayed; theWard Volunteer network was expanded and retrained; all households were revisited to ensure correct filterassembly; more time and energy was put into training; a dedicated covered storage container was provided;distributions were clustered within communities; and, informational print materials were developed. In addition,the initial desired outcome measure of diarrheal disease reduction was dropped from the evaluation, as there wasneither statistical power nor controls sufficient to evaluate diarrheal disease reduction in this program.

$!&!$ Ongoing evaluation

After the program restarted distribution, the OA/Zimbabwe Biosand Filter project M&E Coordinator conductedongoing evaluation. Three program reports (from December, January to February, and March to May)summarizing data collected by the M&E Coordinator were made available to the consultant and are summarizedin this section.

The M&E Consultant reported that communities believed that the Biosand Filter treated water was safe asevidenced by only 27% exposing treated water to post-filtration treatments such as Aquatabs or boiling. A total of 29 (5%) of the 571 filters were cracked. The M&E Coordinator reported E. coli results from 16 paired householdwater samples one sample collected from the household before treatment and one reported treated samplecollected from the household water storage container . These results were analyzed by the consultant, and arepresented in Figure 2. As can be seen, the use of the Biosand Filter improves the microbiological quality of storedhousehold water. More households had household stored water in the no risk category after treatment than hadwater in this no risk category before treatment. Overall, 10 of the 16 households (62.5%) had water that wascontaminated before treatment ( ! 1 CFU/100 mL of E. coli) and that was not contaminated (


10/37

Page 10 of 38

Figure 2: M&E Coordinator paired treated/untreated water E. coli samples

Due to Oxfam administrative issues, the results from 571 questionnaires administered by the M&E Coordinatorwere not available to the consultant at the time of writing this report.

$!&!' Project +end evaluation

OA in Zimbabwe conducted a program-end evaluation in September 2009. A total of 194 (23%) of the 850confirmed beneficiaries of Biosand Filters were surveyed. The households were not randomly selected. Overall,52% of households had access to improved sources of water, 80% covered their stored water container, and 93%reported still using Biosand Filter. Slightly over half had correct standing water levels to correctly maintain theschmutzdecke, with 29% having water levels too low (between 0 and 4 centimeters above the sand level), and29% having levels too high (above 6 centimeters). Almost every household (99%) reported receiving training onthe Biosand Filter, with 66% reporting follow-up household visits. In addition, 77% reported receiving education,including from SPWSNet (44.2%) and the Ministry of Health and Child Welfare (56%). The training from theMinistry was likely general cholera messages, and not specific Biosand Filter information. The program-endevaluation, like the ongoing evaluation, noted post-treatment after filtration with another household watertreatment option, as 7.4% of respondents reported post-treatment with Aquatabs, boiling, or solar disinfection.Program challenges noted included cracking in the container, incorrect standing water levels, long maturationtime for the schmutzdecke to develop, and slow filtration rates.

The percent of households reporting Biosand Filter treated water and the time of the unannounced survey visitwas not included in the program-end evaluation report. The water quality parameters of turbidity and E. coli weretested. Turbidity was generally low across all samples tested. Sixty triplicated household water samples weretested: 1) untreated water stored in the house; 2) water filtered through the Biosand Filter by the enumerator andcollected directly from the filter without touching a storage container; and, 3) reported Biosand Filter treated waterstored in the household storage container. These results were analyzed by the consultant, and are presented inFigure 3. It is clear, again, that the use of the Biosand Filter improves the microbiological quality of storedhousehold water. Overall, 38 of the 60 households (63%) had water that was contaminated before treatment ( ! 1CFU/100 mL of E. coli) and that was not contaminated (


11/37

Page 11 of 38

CFU/100 mL of E. coli) and that was not contaminated (


12/37

Page 12 of 38

2.2 Government of Zimbabwe draft summary report

Given the recent interest in Biosand Filters in Zimbabwe, the Government of Zimbabwe is evaluating whetherthey will approve Biosand Filters formally through the Standards Association of Zimbabwe. At this point in time,

the government has collated information for numerous NGO reports into one draft document. It is unclear at thispoint in time which Ministry will be responsible for leading the decision on Biosand Filters in Zimbabwe.

In its summary document, the government report notes that a number of NGOs in Zimbabwe have beenpromoting the One Way Ministries manufactured Biosand Filter as a household water treatment method indifferent rural areas of the country on a pilot basis. They noted 2,000 filters had been installed by the end of 2010 by International Medical Corps (Bindura, Shamva and Rushinga), Single Parents and Widow(ers) SupportNetwork (SPWSNet) (Mudzi), Institute of Water and Sanitation Development (Hatcliffe, Harare), and One wayMinistries (Epworth, Harare). The objective of the Government report was to assess the efficacy of the BSF inremoving pathogenic bacteria and turbidity ensuring drinking water is safe at point of consumption as presentedin the different reports. The assessment also aims at providing NAC with adequate information to make aninformed policy decision on the adoption, rejection or recommendations for macro field trial and furtherimprovement and development of the Biosand Filter as a household water treatment technology.

Based on the compilation of 160 water quality samples from the various implementing organizations, theGovernment concluded that technical efficacy of the Biosand Filter alone is not enough to ensure sustainabilityand for NAC to make a policy decision on the adoption or rejection of the technology and that further work isneeded to improve the results and the success of the program including achieving continued and active longterm adoption of the technology will require adequate data on the main interlinking elements of the program:

technical ( E. coli and turbidity removal rates), user perceptions, operations and maintenance, hygiene education,use of filtered water, durability, sustainability of the filters and monitoring and evaluation even beyond theprogram implementation period. To ensure these elements are adhered to there is need for clear Terms of Reference and support by NAC [the Government of Zimbabwe] in form of among others monitoring &evaluation.

$!' Sustained use report

The objectives of this sustained use report were to provide an independent assessment on: 1) the efficacy and

effectiveness of the Biosand Filters for the cholera response; and, 2) a process evaluation. The specific questionsfor the efficacy and effectiveness objectives included:

Was the introduction of Biosand Filters an appropriate intervention? Were the Biosand Filters effective in the cholera response? Has the quality of water improved with the use of filters for household consumption (cooking, drinking,

and sanitation)? Did communities receiving the filters change their behavior in preventing disease during the program?

What has been the impact on prevalence of water-borne diarrheal diseases in these communities? Is there any difference in effectiveness or efficiency between the plastic Biosand Filters compared to

concrete ones? How sustainable is the use of Biosand Filters likely to be?


13/37

Page 13 of 38

The process evaluation included reviewing question of coverage; cost-effectiveness; efficiency and coordinationmechanisms; communications; information management and accountability; communication and reporting; anddesign of program.

The methodologies to complete these objectives are presented in the next section, followed by survey and processresults, and conclusions and recommendations in subsequent sections. The full Terms of Reference for thissustained use evaluation are appended in Annex A.


14/37

Page 14 of 38

' Methodology

A mixed-methodology investigation was completed to fully evaluate the program, including: 1) household surveysto understand Biosand Filter knowledge, use, and sustained use; 2) water quality testing to document theeffectiveness of the Biosand Filters; and, 3) key informant interviews with program staff to characterize theresponse. Focus group discussions were not completed due to the distance between filter beneficiaries.

3.1 Household surveys

The consultant developed a survey consisting of 44 questions on: 1) respondent and household characteristics,assets, and HWTS knowledge and products received; 2) installation, training on, use, and maintenance of the

household Biosand Filter; and, 3) current treated and untreated stored household drinking water, including samplecollection for later analysis. Each survey took 15-20 minutes to administer per household. Survey training and pre-testing occurred on January 18, 2011, with four enumerators. A finalized survey was developed and printed inMudzi. The survey was written in English and administered in English or Shona, depending on the family. From aline list of the 897 Biosand Filter beneficiaries, 100 households were randomly selected using the Microsoft Excel(Redmond, WA, USA) random function. All survey data was entered into Microsoft Excel and analyzed using Stata10.1 (College Station, TX, USA). The evaluation itinerary and survey tool used in the evaluation are appended inAnnex B and Annex C, respectively.

3.2 Water quality testing

Enumerators were trained by the consultant to collect a treated water sample (if available) and an untreated watersample aseptically from each surveyed household. Samples were collected in sterile WhirlPak ! bags with sodiumthiosulfate to inactivate any chlorine residual present, and stored in a cooler on ice for analysis. Each eveningafter surveys were completed, the consultant completed the microbiological testing using Millipore (Billerica, MA,USA) portable filtration stand laboratory equipment. Samples were diluted appropriately with sterile bufferedwater, filtered aseptically through a 45-micron filter, placed in a plastic petri-dish with a mColiBlue24 mediasoaked pad, and incubated in a portable incubator for 24 hours at the appropriate temperature. Red colonies were

counted as total coliform, and blue colonies as E. coli. One deviation from Standard Methods is holding timebefore the sample was fully filtered was extended from 8 to 12 hours due to travel logistics (APHA/AWWA/WEF,1998). Negative controls were included each day and 10% of samples were duplicated for quality control.

3.3 Key informant interviews

Key informant interviews with program staff were conducted to characterize both the programmatic response andevaluation, and the management strategies necessary to complete a program leading to sustained use. Keyinformant interviews were conducted formally and informally with program staff from OA/Zimbabwe, SPWSNet,

and OA during the field visit and after the consultants return.


15/37

Page 15 of 38

( Survey Results

In four days of work, four enumerators located 61 of the 100 families randomly selected for survey. This isequivalent to a survey rate of 3.8 surveys per enumerator per day. The reasons 19 of the 39 families were notsurveyed were: the household was not home because they were in the fields (8/19, 42%); the household waslocated further than a one hour walk from the road (4/19, 21%); because the car was stuck in the mud (2/19,11%) or unable to cross the river; and, because the household moved (2/19, 11%). The remaining 20 familieswere unable to be located.

Of the 61 families location by the enumerators, full surveys and water quality samples collected were completedin 29 families receiving plastic filters and 29 families receiving concrete filters. An additional survey (no samplecollections as the surveys were completed outside the owners home) were completed with families receiving

plastic filters and two samples only (no surveys as the water was collected from the beneficiaries home with aneighbor present) were completed on concrete filters. Thus, a total of 60 surveys and 61 sample collections werecompleted.

Household characteristic results are presented in Table 1. A statistically significant difference between plastic andconcrete filters is noted by a p-value of less than 0.05.

Of note in the survey results are: Plastic filter beneficiaries were more likely to have received their filter in 2009, as opposed to concrete filter

beneficiaries, who received their filters in 2010. The majority of survey respondents were female, and the majority attended school. A statistically significantly smaller number of plastic filter beneficiaries reported having a family member with

cholera (14%) as compared to concrete filter beneficiaries (57%), indicating targeting improved over time. The majority of filter beneficiaries (69%) reported also receiving Aquatabs for cholera response. The majority of respondents reported receiving enough education for installation (95%) and maintenance

(95%), had help assembling the filter (79%), and knew someone else with a Biosand Filter (93%). The majority of respondents reported ever using the filter (98%), currently using the filter (85%), and planning

to keep using the filter in the future (100%). Of the nine people who report currently not using the filter thereasons why are: broken (3), and one each for it does not fit in house, it has problem, using rainwater instead,not using, have no water, and gave it away. There was a statistically significant difference in reported currentuse between plastic and concrete filters, with a higher percentage of concrete filter beneficiaries reportingcurrent use. This could be because these beneficiaries received filters more recently.

The majority of filters were wet (91%) on observation, which is a non-biased indicator of use. A minority of users reported sharing the filtered water (33%). The majority of respondents reported cleaning the filter in some way (84%). The majority of respondents reported storing treated water in covered containers (74%). Overall, 58% of respondents reported treated water at the time of the unannounced survey visit, with 97% of

those treating (34 respondents) using the Biosand Filter. No user reported post-treatment of filtered water withanother option, which differs from previous evaluations.

The vast majority of beneficiaries use the water for only drinking and cooking (82%). Concrete- filterbeneficiaries were more likely to use water for more than only drinking than plastic filter beneficiaries.


16/37

Page 16 of 38

Table 1: Household characteristic results

Plastic

filters

Concrete

filters

Total p-value

Number of households surveyed and or sampled 30 (49%) 31 (51%) 61

Number (%) received Biosand Filter in 2009 (n=58) 23 (85%) 13 (42%) 36 (62%)


17/37

Page 17 of 38

Survey respondents reported receiving a variety of trainings (Table 2). Training was associated with increased useof the filter, as respondents who reported receiving more than one training of any type were more likely to reporttreated water at the time of the unannounced survey visit than those who reported receiving one training or less(p=0.05).The only statistically significant difference in training received between plastic and concrete filters wasthat beneficiaries of concrete filters received more pamphlets, which is attributed to the concrete filters beingdistributed later on in the program after written materials were developed. Twenty (57%) of the 35 respondentswho reported receiving pamphlet information could produce it.

Table 2: Training received by respondents for plastic and concrete containers

Training received Plastic

filters

Concrete

filters

Total p-value

Poster/pamphlet (n=61) 12 (40%) 24 (77%) 36 (59%)


18/37

Page 18 of 38

The average number of times users cleaned their filter was 1.2 times per week (range 0-7). For plastic filters thisaverage was 1.8 times per week (range 0-7) and for concrete this average was 0.7 times per week (average 0.02-3.5). The cleaning strategies were inconsistent (Table 3), with 16% of respondents overall reporting nevercleaning, and a variety of cleaning strategies (from removing all the sand to cleaning only the sand at the top tocleaning the outside only to reinstalling the whole filter) were employed. A particularly worrisome cleaningstrategy was to wash slime out, or remove the schmutzdecke, which actually decreases the microbiologicalremoval efficacy of the filter. Concrete filter beneficiaries were statistically significantly more like to clean justthe sand at the top, which is the appropriate cleaning strategies for the Biosand Filter. This could be due tosomething innate in the concrete filter, or the fact that these beneficiaries received filters later, with bettereducational materials. In future programs, beneficiaries need sufficient training on a consistent, appropriatecleaning strategy.

Table 3: Cleaning strategies for plastic and concrete filters

Cleaning Strategy Plastic

filters

Concrete

filters

Total p-value

Remove all the sand, and clean (n=61) 6 (20%) 6 (19%) 12 (20%) 0.95

Clean just the sand at top (n=61) 7 (23%) 18 (58%) 25 (41%)


19/37

Page 19 of 38

Figure 6: An unused filter with iron staining, a filter used for storing tomatoes, a correctly installed and used filter


20/37

Page 20 of 38

" Process Evaluation Results

OA/Boston requested the consultant to write a narrative describing her experience with the process portions of theprogram, beginning with her initial interactions with the program through to this evaluation.

The consultant was first made aware of the program in early 2009, while she was working with a UNICEF/Oxfamcontract at the London School of Hygiene and Tropical Medicine. The Terms of Reference for that work were toevaluate the effectiveness and acceptability of HWTS options in acute emergency situations. The protocol wasopen-ended, and the consultant was to travel to four emergencies that occurred during the time of the contract toevaluate HWTS distributions within eight weeks of emergency onset. Initially, the consultant was contacted byOA to provide input into the Zimbabwe Biosand Filter program and to assess whether this program might beappropriate for inclusion into the larger UNICEF/Oxfam program.

At the time of the initial distribution evaluation by the consultant, it was clear that the process by which theprogram was being implemented was non-ideal. Individuals within OA/Boston had different ideas about theprogram goals, with some wanting an emergency program, some a health impact study, some a pilot program.OA/Zimbabwe staff received conflicting messages about the program from OA/Boston staff and were frustrated byhow the program was managed. OA/Boston staff had little interaction with the implementing partner, SPWSNet,or the Government of Zimbabwe, leaving those interactions at the discretion of OA/Zimbabwe staff. Overall, thislack of communication between OA/Boston and the on-the-ground implementing team led to the program beingimplemented in ways that surprised OA/Boston. For example, Biosand Filters were distributed by SPWSNet tohouseholds selected by local chiefs, rather than households affected by cholera. In addition, filters weredistributed in a wide geographical area to obtain the most political capital from chiefs, which is not conducive toevaluation or a research study. Filters were distributed to families with minimal group training, and families wereexpected to install and maintain their own filters without support. Laslty, SPWSNet did not have access to trainingmaterials and resources for correct installation of filters.

In the initial distribution evaluation, recommendations were made to improve distribution and training. Inaddition, the consultant linked OA and SPWSNet to the Centre for Affordable Water and Sanitation (CAWST), anorganization that provides technical assistance on Biosand Filters. It was mutually decided at the end of the initialdistribution evaluation by the consultant and OA that this program was not appropriate for inclusion within thelarger UNICEF/Oxfam program due to the lack of a program established in the acute emergency context.

From July 2009 through to December 2010, the consultant was involved with the program at a minor level,sometimes answering emails or a technical question. The consultant had no substantive knowledge of how theprogram was progressing. In late September 2010, the consultant was contacted to conduct a sustained useevaluation of the program. Arrangements were made and the evaluation was planned for the end of January 2011.

OA/Boston consistently emphasized to the consultant the importance of this evaluation being independent andunbiased. The consultant and OA/Boston worked with OA/Zimbabwe to arrange for an independent localconsultant to assist with evaluation logistics in Zimbabwe. The name of the local consultant was provided to the

consultant and OA/Boston the week before departure. Upon emailing the local consultant, the consultant was toldby OA/Zimbabwe staff that logistics were managed and would be discussed upon arrival. Upon arrival to


21/37

Page 21 of 38

Zimbabwe, the consultant met with the local consultant, who was unfamiliar with the evaluation and surprisedthe government had not been contacted to approve it. Unbeknownst to OA/Zimbabwe staff, the local consultantarranged and executed meetings with Ministry of Health and Water officials. Significant concern about approvingthe evaluation was expressed, and noted local Mudzi government staff approval would be needed. Afterreconvening with OA/Zimbabwe staff, it was discovered that OA/Zimbabwe staff had, unbeknownst to OA/Bostonor the consultant, arranged for SPWSNet staff, volunteers, and OA/Zimbabwe staff to travel to Mudzi and assistwith the survey. No arrangements for government approval had been made. On Monday afternoon (see timelinein Annex B), OA/Zimbabwe staff, SPWSNet staff, the local consultant, and the consultant departed for Mudzi.

On Tuesday, SPWSNet staff and OA/Zimbabwe staff finalized the Biosand Filter beneficiary list, and theconsultant trained the enumerators and randomized the beneficiary list to select households for survey. SPWSNetand the enumerators suggested two non-standard survey methods to the consultant. The consultant firmlyexpressed to SPWSNet that selected households should not be called to pre-arrange a household visit and firmlyexpressed to the enumerators (some of whom had previously evaluated this program) that replacements of

households not at home with households at home that they could locate when surveying was not to becompleted. Both of these methods can bias usage results, as households will treat water if they know a visit willoccur. On Wednesday, after gaining approval from the local government, surveys began in the early afternoon.

In the survey results, we found that the vast majority of beneficiaries reported training, however, the quality of thetraining did not lead to beneficiaries having correct knowledge and ability on how to install, maintain, and cleantheir filter. Clearly, there was information loss in providing the correct training to the beneficiaries, but we werenot able to distinguish whether that information loss occurred between OA and SPWSNet, SPWSNet and theVolunteers, or between the Volunteers and the beneficiaries, because respondents were not able to distinguishbetween SPWSNet staff and Volunteers trained by SPWSNet in survey responses. In retrospect, given the poor

results around training knowledge, it would have been a good idea to conduct knowledge assessments of Volunteers. However, we were unaware that this would be needed during the planning of the evaluation.Pamphlets were distributed to beneficiaries who received filters later in the study, indicating the programresponded to recommendations being made in the initial distribution evaluation. A pamphlet was not available toaccess pamphlet quality.

After survey completion on Saturday, the consultant departed Zimbabwe on Sunday. As the survey data had notbeen entered or analyzed by the consultant at this time, the consultant did not provide a debriefing meeting withOGB or the Government of Zimbabwe. It was planned that the local consultant (who was not able to be presentfor Thursday surveys and for part of Friday due to a prior commitment) would present the results of this survey tothe Government of Zimbabwe and interested parties. OGB did not indicate interest in disseminating theinformation, as they stated they were not planning to continue their Biosand Filter program. We did not providefeedback to the beneficiaries as the physical distance to each beneficiary was prohibitive.

Overall, it appears there was poor communication between OA/Boston, OA/Zimbabwe, SPWSNet, theVolunteers, and the beneficiaries. Significant information seems to have been lost in this chain, and beneficiarydid not have the knowledge to install, maintain, and clean their filters to achieve sustained, effective use. In thefuture it is recommended that all parties involved in the program align on program goals and develop trainingmaterials and training classes before filter distribution.


22/37

Page 22 of 38

The consultant feels that the monitoring and evaluation of this program was quite good with adequate metricscollected and significant data obtained. The communication difficulties between the various partners lessened theutility of the extensive monitoring and evaluation conducted, as the program did not reach its potential. However,despite these difficulties, the consultants feels that: 1) there was high sustained use of the Biosand Filters by thebeneficiaries (56% of household reporting using the filter, and 56% of reporting users having improvedmicrobiological quality); and, 2) despite the challenges with the survey, the consultant does feel that the surveywas representative, as: 1) the survey was randomly implemented; 2) replacements on non-selected householdswere not left to the discretion of the enumerators; and, 3) households were not called the day before the survey topre-arrange a visit. A main limitation of this evaluation, however, is the relatively lower response rate to thesurvey (61 or 100 targeted households located in four days of surveying) due to the dispersed nature of BiosandFilter distribution and difficulty in accessing households.


23/37

Page 23 of 38

* Discussion and Recommendations

The purpose of this evaluation was two-fold: 1) to conduct an independent assessment on the efficacy andeffectiveness of the Biosand Filters for the cholera response; and, 2) to conduct a process evaluation. Based on thedata collected, the specific questions raised in the Terms of Reference are answered below. Recommendations forfuture programs are included at the end of this section.

6.1 Independent assessment on efficacy and effectiveness of the Biosand Filtersfor the cholera response

Was the introduction of Biosand Filters an appropriate intervention?

Based on the small number of families confirmed to be reached with a Biosand Filter (only 897), and the timeperiod after the cholera in which they were reached, it can not be stated that the Biosand Filters were anappropriate intervention for the program goal of responding to the cholera. However, the filters were wellliked and used by the beneficiaries, with 56% of survey respondents having treated water on the day of theunannounced survey visit, which indicates that they are an appropriate intervention for Zimbabwe.

Were the Biosand Filters effective in the cholera response?

The data suggest that no, the Biosand Filters were not effective in the cholera response. They were delivered

with too little training too late to too few people to significantly reduce the risk of cholera. Additionally, 69%of surveyed respondents who received Biosand Filters also received Aquatabs as a cholera responsehousehold water treatment option from other organization(s). Aquatabs are less expensive, easier to distribute,simpler to use, and more effective at reducing the cholera bacteria from household stored drinking water thanBiosand Filters. However, Aquatabs are a consumable, not a durable, product and thus they only provideprotection until they run out. The evidence also suggests that the Biosand Filter program, unlike the Aquatabsdistributions, had sustained uptake.

Has the quality of water improved with the use of filters for household consumption (cooking, drinking, and sanitation)?

The evidence consistently shows that the use of Biosand Filters improves the microbiological quality of household stored water. The percentage of the population that had contaminated water before treatment anduncontaminated after treatment ranged from 63% in the M&E Coordinator evaluation, 43% in the follow-upevaluation, and 56% in the sustained use evaluation. An important note in the survey results is the lowpercentage of the population that used the filtered water for anything other than cooking and drinking. Giventhe high flow rates possible with Biosand Filters, it is recommended to encourage users to use the water foradditional purposes.


24/37

Page 24 of 38

Did communities receiving the filters change their behavior in preventing disease during the program? What hasbeen the impact on prevalence of water-borne diarrheal diseases in these communities?

The 897 confirmed filters distributed were delivered to a minimum of 121 communities (about 7.4 filters percommunity). Given the low percentage coverage per community, it is not anticipated that the filters changedcommunity behavior or significantly reduced the prevalence of water-borne diarrheal disease at thecommunity level.

Is there any difference in effectiveness or efficiency between the plastic Biosand Filters compared to concrete ones?

The data suggest that yes, there are differences in the effectiveness and efficiency of the plastic and concreteBiosand Filters. It is not clear at this point how much the differences are due to innate differences between thefilters styles and how much is due to the fact concrete filters were distributed later than plastic ones, withimproved education and training. However, it does appear that concrete filters were less likely to crack.

How sustainable is the use of Biosand Filters likely to be?

A total of 33 of 59 (56%) households surveyed reported having Biosand Filtered water at the time of theunannounced survey visit conducted in January 2011. This data indicates a willingness of the population tocontinue using their filter. Sustainability in inhibited by lack of training on operations and maintenance.

6.2 Process evaluation

Coverage: Review the adequacy of the program coverage. What was the quality of beneficiary selection and their

participation in the decision making during the program?

The goal of the program as reported to the consultant was to reach families affected by cholera in MudziDistrict. The population of Mudzi was 130,514 in 2002, or 26,103 families. This program reached 3.4% of the households in Mudzi District. Although the selection criteria for filter distribution was supposed to befamilies affected by cholera, only 36% of survey respondents reported being cholera-affected. A higherpercent of concrete filter beneficiaries, who received their filter later in the program, reported being cholera-affected, indicating that selection criteria may have improved after initial evaluations.

Cost effectiveness: Based on available evidence is it possible to draw any qualitative conclusions as to how cost-effective Oxfams intervention has been?

The sub-contracted amount from OA to SPWSNet for the education component of the program was 40,000USD. As 897 families were reached, it was thus 45 USD to reach each family with the education to completethe program. On the day of the unannounced survey visit, 56% of respondents had current Biosand Filteredwater in the household. In addition, 56% of those with treated water had contaminated water beforetreatment and uncontaminated after. Thus, about one-third of beneficiaries (31.4% - 56% multiplied by 56%)were using the filter on the day of the unannounced visit to treat their water to WHO drinking waterstandards. This is equivalent to 281 households being reached effectively with the intervention (897 totalreached multiplied by 31.4%), and a cost of 142 USD per household reached with effective treatment.


25/37

Page 25 of 38

It is difficult to assess the full cost-effectiveness of this program in full due to the extensive evaluations thatoccurred and impacted program success substantially, as well as equipment and supplies that were providedby OA to SPWSNet.

Efficiency and coordination mechanisms: Review internal and external coordination. Internal coordinationconcerns the coordination between Oxfam America and the implementing partner as well as coordinationbetween the HQ based specialists and country based program staff. External coordination concerns thecoordination with local authorities, community volunteers.

The coordination chain between the partners was as follows: 1) OA in Boston designed the program,including a research component that included a health impact study; 2) OA in Zimbabwe coordinated thework with SPWSNet to implement the program; and, 3) SPWSNet implemented the program and interactedwith local authorities and community volunteers. There was no direct communication between Boston andSPWSNet. All of the communication chains were effective, except there was a missing link between Boston

and the local NGO to ensure that the program was implemented in such a way to meet some of the researchgoals of the program.

Communications, information management and accountability What communication methods and measureswere used and were they effective and useful? What were the results?

The use of volunteers to communicate with the beneficiaries was an effective mechanism to extend the reachof the NGO, and volunteer trainings should be prioritized in future programs. Increased training wasassociated with reported use of the filter, indicating that trainings were effective at encouraging sustained use

of the filter.

Communication to the affected people and their perceptions.

The communication to the affected people (the beneficiary population) occurred not by OA, but by SPWSNet.There is a good relationship between the local NGO SPWSNet and OA in Zimbabwe, and the perception of the population for this program was positive.

Communication and reporting: ongoing monitoring, communication methods between community members and the volunteers, between partners and the community members including documenting, reporting feedback from

community members.

The communication chains established for this program were effective, except for what is noted in other sections.

Design of program: What conclusions or recommendations can be drawn about the overall design of the program? The analysis should include reflection on the planning, implementation, documentation and monitoring of activities, including partner capacity and Oxfam support.

Overall, the consultant stands by the comment made in July 2011 in her first report: this program is

attempting to be many things at once a pilot program, a research program, an implementation program, aprogram targeting the most needy people, a program comparing two types of filters all in an aggressive time


26/37

Page 26 of 38

frame. There was not the internal communication or operational capacity available within OA in Boston andZimbabwe with SPWSNet to complete such a complicated program that would meet all of these goals. That isnot to say that the program was a failure, in fact far from that. It is simply to say that some of the programgoals fell by the wayside due to the complicated nature of the program. In the future, it is recommended thatprogram design be more considered, and communication between program designers and local NGOs beimproved.

6.3 Recommendations

Results from multiple evaluations in Zimbabwe confirm the extensive worldwide data set that the use of BiosandFilters improves the microbiological quality of stored household drinking water. It can be inferred that users of Biosand Filters thus have less diarrheal disease. The question is no longer on microbiological effectiveness of thefilters, but on appropriate implementation strategies that result in users effectively using the filters to improve the

quality of their stored household water in a cost-effective manner, and whether that is appropriate and cost-effectivecompared to other interventions such as Aquatabs in the emergency context. The Government of Zimbabwe hasnoted this, and is actively seeking data on how to encourage effective, sustained use of Biosand Filters in Zimbabwe.Should OA wish to continue with Biosand Filter implementations, it is recommended that OA:

1. Coordinate future programs with the Government of Zimbabwe. The focus of future programs should notbe on confirming microbiological effectiveness or health impact (which have been well-verified) butinstead on establishing cost-effective distribution strategies that reach a large population with the trainingnecessary to encourage consistent and correct use.

2. Understand the potentially limited role of Biosand Filters in the acute emergency situation. Biosand Filterstake time to distribute, train on, and for the schmutzdecke to develop. A chlorine-based product shouldbe distributed in the emergency response time until such time as there is capacity to complete thistraining and development. This result is consistent with other Biosand Filter evaluations that theconsultant has conducted with NGOs distributing Biosand Filters in acute emergency situations, forexample after the Haiti earthquake (for UNICEF and Oxfam/UK) and during cholera in the DemocraticRepublic of Congo (for Oxfam/DRC).

3. Complete significant training with beneficiaries , that teaches how to: a) adequately install the filter,including establishing the correct standing water layer; b) maintain the filter, including where to obtain

replacement parts; c) how to clean the filter appropriately and on what schedule; and, d) highlights thatfiltered water can be used for more than just drinking and cooking.

4. Keep program design and scope realistic and considered . In the future, research plans should be morefully integrated into implementation.


27/37

Page 27 of 38

Annex A: Terms of Reference

EVALUATION OF THE OXFAM AMERICA BIOSAND FILTER

IMPLEMENTATION PROJECT, ZIMBABWE

1- Introduction:

This document sets out the Terms of Reference for the final evaluation of the Biosand Filter Implementation program in Zimbabwe. Project implementation spanned the period April 2009 September 2010 with final reports, analysis and other documentation expected to be completed byDecember 2010.

The program was set up in response to a cholera outbreak in north-eastern Zimbabwe but was alsoviewed as a pilot program to test the introduction and use of the Biosand filter technology in ahumanitarian context. The main purpose of this evaluation, as further elaborated below, is to drawlessons from the experience on the viability of the technology, effective management arrangements andminimum conditions necessary for sustainable use of the filters by communities.

2- Background:

As part of its strategy for water delivery in emergencies, Oxfam America introduced point of useBiosand Filters to 900 households in Mudzi District in Zimbabwe. This location was selected becauseOxfam America was already responding to the Cholera epidemic in this part of the country. Cholera isa yearly occurrence in Zimbabwe but the 2008/09 outbreak was worse because it started before therainy season, affected a large number of people with a high mortality rate, and also occurred in urbanenvironments previously spared from yearly outbreaks. The reason for the high incidence of thedisease in Mudzi was largely due to the poor quality of household water in the rural areas due tocontaminated sources and the use of unsafe surface water sources.

Oxfam America and Oxfam Great Britain jointly responded to the epidemic along with the local partner organization, Single Parents and Widowers Support Network (SPWSN). The initial responsecame in the form of bore hole repair, distribution of NFIs, health and hygiene education, coordination,and supply of oral rehydration salts. A preliminary assessment was also undertaken on the overallsituation and the groundwork was laid for the development of a diarrhea early warning system and theintroduction of the Biosand filters.

As Oxfam moved forward with the programming to address the acute needs caused by the outbreak, itconsidered ways of halting the annual reoccurrence of cholera in Zimbabwe. The HarvardHumanitarian Initiative and Oxfam worked with SPWSN to put in place a regionally run system thatwould help with early detection and response. The system was adapted by SPWSN and other localstakeholders to better suit community needs, resulting in Village Cholera Committees. To address themedium to long-term needs, Oxfam America implemented a pilot program to utilize householdBiosand Filters to 950 families in Mudzi District. Oxfam America purchased 450 pre-fabricated plasticfilters from Hydraid and commissioned the manufacture of 500 locally made concrete filters inZimbabwe. Working with SPWSN, the filters were distributed to families in stages, accompanied byfilter use and maintenance training coupled with public health and hygiene education.

Given that the technology was new for Oxfam, SPWSN as well as for the communities, adjustmentsand improvements were regularly needed during program implementation. Monitoring of the program


28/37

Page 28 of 38

was initially focused on filter distribution activities and logistics. Training of households in the use of filters and monitoring of water quality or hygiene practices were also only improved during the courseof the program. To compensate for the lack of consistent monitoring data, the program undertook anextensive results monitoring exercise in September 2010. This exercise has made a closer examinationof the proper knowledge/utilization of Biosand Filters, hygiene practices and household water quality.Data from the results monitoring exercise is still being analyzed at the time of writing this TOR.

In addition, the program is planning to prepare a How-To toolkit that will document the experienceof the Biosand Filter program. The Toolkit will serve the purpose of a technical, user and monitoringmanual, list useful training and IEC materials prepared by this program, make recommendations onchoice of monitoring indicators, present monitoring templates or formats to be used for various aspectsof the program, and include guidance (based on the pilot program experience) on the minimumfrequency and type of data collection needed to manage such a program. The toolkit will, in turn,

benefit from the program evaluation report and will be finalized upon completion of all other programrequirements.

3- Evaluation of the use of Biosand Filters

3.1. Rationale, Purpose and Scope of the Evaluation

Rationale:

Oxfam is committed to the highest standards of learning and accountability. Accountability is our obligation to demonstrate, in a transparent way, that our actions, decisions and their results direct or indirect are in compliance with agreed norms, protocols, values and standards with respect to prudentand honest management of resources, knowledge and relationships (with people, partners, allies,donors, etc.), and allow/enable an independent assessment of our performance and results. We areaccountable both internally to ourselves to draw out lessons for future improvement, and externally toour stakeholders and especially to the people we serve. According to OAs Humanitarian Monitoring,

Evaluation and Learning (MEL) policy regarding programs implemented by partners, OA will evaluateselect programs for accountability and learning purposes.

Oxfam America is building its capacity in the water and sanitation sector in alignment with OxfamInternationals strategy under the Single Management Structure. WASH interventions in humanitarian

programs are relatively new within OA. The use of Biosand Filters for provision of safe drinking water to cholera-affected communities was piloted in Zimbabwe with the intention of learning from theexperience for possible replication in other contexts. This evaluation will allow Oxfam to capturelessons and learn from its new experience as well as to demonstrate the efficacy of the Biosand Filter use during emergencies.

3.2. Purpose:

The purpose of this evaluation is two-fold:1) to provide an independent assessment on:Efficacy and effectiveness of the Biosand Filters for the

cholera response: Was the introduction of Biosand Filters an appropriate intervention? Were the Biosand Filters effective in the cholera response? Has the quality of water improved with the use of filters for household consumption

(cooking, drinking, and sanitation)? Did communities receiving the filters change their behavior in preventing disease during

the program? What has been the impact on prevalence of water-borne diarrheal diseases inthese communities?

Is there any difference in effectiveness or efficiency between the plastic Biosand Filters

compared to concrete ones? How sustainable is the use of Biosand Filters likely to be?


29/37

Page 29 of 38

2) Process evaluation (for learning purposes) Coverage: Review the adequacy of the program coverage. What was the quality of

beneficiary selection and their participation in the decision making during the program? Cost effectiveness: Based on available evidence is it possible to draw any qualitative

conclusions as to how cost-effective Oxfams intervention has been? Efficiency and coordination mechanisms: Review internal and external coordination.

Internal coordination concerns the coordination between Oxfam America and theimplementing partner as well as coordination between the HQ based specialists andcountry based program staff. External coordination concerns the coordination with localauthorities, community volunteers.

Communications, information management and accountability What communicationmethods and measures were used and were they effective and useful? What were theresults?

o Communication to the affected people and their perceptionso Communication and reporting: ongoing monitoring, communication methods between

community members and the volunteers, between partners and the community

members including documenting, reporting feedback from community members Design of program: What conclusions or recommendations can be drawn about the overalldesign of the program? The analysis should include reflection on the planning,implementation, documentation and monitoring of activities, including partner capacityand Oxfam support.

4- Intended use and evaluation audience

This evaluation is expected to serve an important learning function for the design and management of similar programs in other contexts. It is also expected to give Oxfam partner, SPWSN, valuablefeedback on their role, management, staffing as well as M&E functions. This will serve as beneficialinput for improving their organizational capacity to undertake similar or different humanitarian

response programs in the future.

External audience:

In addition to sharing evaluation results with OAs implementing partner, findings will also becirculated amongst OGB and other NGOs, local authorities and with communities using the filters. Amodified version of the evaluation report may be shared with donors as relevant. Oxfam America willdevelop a separate advocacy or lessons sharing plan with relevant stakeholders.

Internal audience:

The evaluation results will be shared across OA along with the how-to tool kit being developed by

the program where key lessons learned and best practices will be compiled to complement toolsalready developed from other literature and experiences with the use of Biosand Filters.

5- Evaluation Principles and Methods

The evaluation will be guided by the following principles: Participation: Evaluation must incorporate the views and perceptions of the affected people and

program participants, duly contextualized in the proper perspective. Participants need to beallowed to shape the very design/intent of the evaluation.

Rigorous: Evaluation must be methodologically rigorous and stand the scrutiny of evaluators and practitioners detailing the evidence base, and wherever possible, triangulating the information.

Credible: the evaluation must be conducted impartially Ethical: Must respect the dignity and privacy of the people and informants. It must also adhere to

internationally recognized standards and norms for humanitarian response evaluation.


30/37

Page 30 of 38

6- Methodology: Documentation review. Observation of key events and processes, Photographs and maps. Random water quality testing. Interviews with stakeholders including staff. Staff includes all relevant OA staff based in Boston

and in Zimbabwe. Non-staff stakeholders that will be important to interview include localauthorities, community leaders, leaders and members of womens groups or users of BiosandFilters.

Focus group discussions separately with women, children, men, etc. Gathering/documenting people perceptions using key informants, personal open-ended

interviews and discussion.

7- Management and Supervision of the evaluation

The evaluation will take place in January 2011. The process will be facilitated in Zimbabwe by the OAHumanitarian Program Coordinator with oversight from the Deputy Director of HumanitarianPrograms and the Humanitarian MEL Manager in HRD, Boston.

8- Time Frame and budget

The evaluation mission will be conducted in January 2011 and will involve:- documentation review 1 day- interviews with Oxfam staff/management 1 day- field investigation and interviews 5 days- debriefing Oxfam/partner staff in Harare & Boston 1 day- analysis and report writing 2 days- travel 3 days

The first draft of the report should be submitted to Oxfam by February 5 2011, whereas the final reportis expected to be completed within 5 days of receipt of any consolidated comments from Oxfam.

Budget:

Details related to the consultant fees and expenses will be stipulated in consultant contracts.

9- Deliverables

Evaluation Report

The final report should be no more than 25 pages, clearly written in English, using font size of 11 points, single spaced. An executive summary of no more than three pages should summarize the major insights, conclusions, and recommendations of the study. As noted before, since the evaluation willserve a capacity building or learning purpose, the report should draw on lessons from the program and

be forward looking in its recommendations.

The report should clearly identify the purpose of the evaluation, what was evaluated, how theevaluation was conducted, the data considered, the conclusions drawn and recommendations made andlessons identified. The report should explain how each conclusion derives from the findings, and whattheir limitations are. Recommendations should be linked to conclusions.


31/37

Page 31 of 38

10- Evaluation Team

The evaluation team will be composed of an international consultant (Team Leader) and one nationalconsultant. The Team Leader will have extensive expertise in the evaluation of humanitarian programsespecially in emergency WASH assistance. The national consultant will also have an evaluation andWASH background and will be responsible for contextualizing the evaluation methodology andapproach to local social and cultural practices. The national consultant will play a central role inorganizing and conducting individual and group discussions with official and communityrepresentatives.

The Team Leader will have final responsibility for the design of the evaluation methodology and reportwriting.


32/37

Page 32 of 38

Annex B: Itinerary and survey respondents

The itinerary of the evaluation was as follows:

Monday, January 17 th: Arrive into HarareMeet with Ransam Mariga, OA/ZimbabweMeet with Fungai Makoni, Local ConsultantMeet with Fungai Makoni, MOW Staff, MOH Staff Meet with Oxfam/Great Britian/ZimbabweDepart for Mudzi

Tuesday, January 18 th: Train enumerators on survey

Wednesday, January 19 th: Meet with government officials to obtain permissionFirst day of survey samplingWater Quality testing

Thursday, January 20 th: Second day of survey samplingWater Quality testing

Friday, January 21 st: Third day of survey sampling

Water Quality testing

Saturday, January 22 nd: Fourth day of survey samplingWater Quality testingReturn to Harare

Sunday, January 23 rd: Depart Zimbabwe

This investigation was not formally reviewed by an Internal Review Board for adherence to ethical standards, as:1) this was not required by Oxfam; and, 2) this type of investigation is generally considered exempt from reviewbecause it is program evaluation and not research. However, the author conducts all of her investigations inaccordance with generally accepted ethical research standards and Internal Review Board procedures. As such,the informed consent paragraph read to each potential survey participant before the survey began (see Annex C),reads: No one except the researcher will know that it was you who provided these answers. In addition, theresearcher, in accordance with standard practice, stripped all identifiers from household survey data on dataentry, and deleted the key linking household survey number to household name after analysis was complete.Thus, the names of individual survey respondents cannot be provided to OA. However, the line list of all filterbeneficiaries, collated by OA/Zimbabwe and SPWSNet staff, is provided with this report.


33/37

Page 33 of 38

Annex C: Household questionnaire


34/37

Page 34 of 38


35/37

Page 35 of 38


36/37

Page 36 of 38


37/37

Annex D: Water quality indicators

The water quality indicators tested in this evaluation included the bacteria E. coli (by the consultant, presented inFigure 5) and the indicator organism Total Coliform (by the consultant). For completeness, E. coli and TotalColiform results are presented in tabular form in this annex.

Table 4: Water quality data

Untreated Stored Household WaterTotal ColiformCFU/100 mL

Untreated Stored Household WaterE. coli

CFU/100 mL

Treated Stored HouseholdWater Total Coliform

CFU/100 mL

Teated Stored Household WaterE. coli

CFU/100 mL

>400 192 0 0

20 0 47 44

96 16 20 0

>400 40 14 0>400 130 30 0

256 16 60 0

>400 18 0 0

8 2 0 0

>400 84 >400 10

>400 0 0 0

88 8 >400 0

>400 8 74 0

149 2 >400 >400

>400 0 >400 78

136 6 >400 4>400 282 30 0

>400 200 >400 350

>400 100 >400 48

150 10 >400 0

>400 0

74 64

116 16

>400 >400

244 108

>400 180

>400 2>400 2

334 70

>400 42

220 74

>400 4

0 0

>400 16

>400 >400

>400 170

220 38

154 4

Sustained Use Zim BSF Report

Documents

Transcript of Sustained Use Zim BSF Report