may 9, 2008 1

Implementation and use of operations research

Dr. Lesly Michaud

World Vision, Haiti

may 9, 2008 2

Presentation Outline

• Description of implementation

• Objectives & methods of operations research (OR)

• Selected OR results

• Conclusions and implications of OR for overall evaluation

may 9, 2008 3

Timeline

2002 2003 2004 2005Qualitative/ formative research

Development of BCC and 2 program models

Implementation of full program models

Operations Evaluation (Round 1)

Consultation with WV; program improvements

Operations Evaluation (Round 2)

BaselineSurvey

Final Survey

Development Implementation

Impact evaluation

Program implementation

-Service delivery points

-Staffing

may 9, 2008 5

Program service delivery points

Rally PostIdentification of beneficiaries

Mandatory attendance at

- Mothers’ Clubs*(mother of beneficiary child)- Home visits (severely malnourished

- Prenatal/ postnatal clinics- Mothers’ Clubs(pregnant/lactating women)- Home visits (post-delivery)

Eligible to receive food atFood distribution points

*Organized differently in preventive and recuperative

PREG/LACTATING WOMENCHILD BENEFICIARIES

may 9, 2008 6

Organizational structure  

Headquarters (Port-au-Prince)Director of MCHN

Director of Commodities

Regional Office (Central Plateau)Regional MCH Coordinator; Assistants to MCH

Coordinator

MCH Supervisors(Nurses)

Health Promoters

Assistant Health Promoters

Food Monitors

Regional Office (Central Plateau)

Regional Commodities Officer

may 9, 2008 7

Services at Rally Posts (RP)

Identical services in both program models

Services include:• Growth monitoring and promotion• Determination of eligibility for food assistance*• Vaccination• Distribution of vitamin A capsules, ORS, deworming

medications• Brief group education sessions

*Based on program model: age-based eligibility in preventive and nutritional status-based eligibility in recuperative

may 9, 2008 8

Services at Mothers’ Clubs (MC)

• Preventive BCC- Pregnancy: 6 sessions- Lactation: 6 sessions- When child is 6-23 mo:

18 sessions, of which 7 focus on complementary feeding and 11 on other topics; some reinforcement

• Recuperative BCC- Pregnancy: 6 sessions- Lactation: 6 sessions- Malnourished child is 6-59

mo: 9 sessions of which 7 sessions focus on recuperation, nutrition; 2 on other topics

Small group education sessions, using interactive methods, meeting monthly and located close to homes of beneficiary mothers

may 9, 2008 9

Services at pre- and post-natal consultations

• Identical in both program models

• Services include– Physical examination– Iron-folate supplements– Tetanus toxoid immunization– Post-partum vitamin A

may 9, 2008 10

Services at Home Visits

Identical in both program models

• Post-delivery

• Newborn visit

• Follow-up of severely malnourished children

may 9, 2008 11

Food Distribution Points

• Monthly ration is the same in both groups (except for longer duration for children in preventive program)

• Ration for children is different from ration for pregnant/lactating women.

• Indirect household ration is included, but only one indirect ration even if multiple beneficiaries in a single household

may 9, 2008 12

Commodity Children 6-24 mo or malnourished <5 years

Pregnant/lactating women

Direct ration (kg)

Indirect ration (kg)

Direct ration (kg)

Indirect ration (kg)

WSB 8      

SFB   10 5 5

Lentils   2.5 2 2

Oil 2   1.5 1.5

WSB=Wheat-soy blend; SFB=Soy-fortified bulgur; oil is fortified with vitamins A & E

Rations distributed at FDPs

Operations Research

may 9, 2008 14

Main objectives

OR Round 1

• Evaluate program implementation relative to plan

• Identify and prioritize solutions to problems

OR Round 2

• Evaluate implementation of solutions

• Look for differences in implementation and/or program use between preventive and recuperative models

– To rule out implementation differences as a reason for differences in impact at the final survey

may 9, 2008 15

Methods

• Structured observations Rally Posts, Mothers’ Clubs, Food Distribution Points

• Exit interviews with beneficiaries and staff

• Group interviews with beneficiaries & staff

may 9, 2008 16

Examples of issues covered

• Rally Posts (RP) – OR-1 & OR-2Staffing and participationOrganization of activities, bottlenecksAvailability of suppliesLarge group education: staffing, qualityGrowth monitoring: accuracy, quality, counseling

• Mothers’ Clubs (MC) – OR-1 & OR-2Organization (age groupings) relative to planStaff facilitation and teaching skills

• Food Distribution Points (OR Round 1 only)OrganizationQuality of foodAmounts distributed (compliance with intended amount)

may 9, 2008 17

Other issues covered

• Staff-level factorsMotivation, job satisfaction

Perceptions of supervision/management

Technical knowledge

Time allocation/workload

• ParticipantsKnowledge of BCC messages

Trial and adoption of key practices

Factors influencing trial and adoption (constraints, facilitating factors)

may 9, 2008 18

Key results from Round 1 Rally Posts

• High participant/staff ratios, crowded

• Sequence of activities often did not follow plan

• Problems with availability of supplies Vaccines, ORS, vitamin A, deworming

• Group education Only one session, crowded, poor ambience, little use of visual materials, did not follow monthly plan for topics

• Growth monitoring Some errors in weighing and plotting, communication with mothers somewhat limited and vague

may 9, 2008 19

Key results from Round 1Mothers’ Clubs

• Overall quality very goodFacilitation, teaching, technical content

• In the preventive model, difficulties to organize MCs by age groups

• Quality not consistent across all staff/program areas

may 9, 2008 20

Key results from Round 1 Food Distribution Points

• Minor inconsistency in measurement of commodities but beneficiaries largely got intended amount

• Staff and food transport problems and a very high beneficiary/staff ratio lead to high time burden for beneficiaries and staff

• Other issues– Food handling/hygiene concerns– Communication issues (health & commodities)

may 9, 2008 21

Consultative workshop with staff Port-au-Prince February 2004

• Presentation of OR 1 results, dialogue, use of decision tool to:• Prioritize problems• Identify feasible solutions• Make an action plan

• Program management played an active role in process

• Field staff were also involved

may 9, 2008 22

OR 2 Rally Post ResultsOrganization & supplies 2003 vs 2004

• At 10 RPs observed both years:

Slight decrease in average participant/staff ratio; range decreased substantially

2003: 21 (7-59) 2004: 17 (6-33)

• Sequence of activities followed plan designed following workshop

• Numbering system used to ensure first-come first-served

• Availability of supplies still a problem: some better, some worse than 2003

may 9, 2008 23

OR 2 Rally Post ResultsGroup education 2003 vs 2004

• Many positive changes

• No differences between program areas

# of sessions observed

2003n = 24

2004n = 62

Mean number of sessions (range)

1.3

(1 - 3)

3.1

(2 – 6)

Mean number of participants (range)

30

(4 - 76)

15

(5 – 30)

Mean duration of session (range)

10

(1 – 23)

22

(6 – 41)

may 9, 2008 24

OR 2 Mothers’ Clubs Results

• Large improvement in organizing mothers into groups by child age in preventive MCs

• Session topics appropriate to groups

• Quality of facilitation and teaching remains very high (e.g. technical content 84% correct; same as OR1)

• No differences between program areas

may 9, 2008 25

Summary of OR 2 results Staff interviews

• Staff-level factors

Motivation, job satisfaction – very positive

Perceptions of supervision – positive, but some variability

Technical knowledge – generally very good, some gaps

• No differences between program areas

may 9, 2008 26

Conclusions 1:Implementation of solutions

• Achieved success in implementing majority of solutions identified at workshop

• Demonstrated commitment to program improvement

• Operations research helped build staff interest and commitment in improving program

• Areas for continuing improvement were identified

may 9, 2008 27

Conclusions 2: Comparing preventive and recuperative program areas

• No major differences in quality of implementation

• Some suggestion that communication during growth monitoring is better in preventive areas

• No major differences in staff-level factors or in participant trial/adoption of practices

may 9, 2008 28

Conclusions 3: Implications for evaluation of effectiveness

and cost-effectiveness

• Any differences in effectiveness can reasonably be attributed to differences in program models, but NOT program implementation