Malawi Primary Health Care (PHC) Project Evaluation · The PHC was a timely and effective medicine...

Malawi Primary Health Care (PHC) Project Evaluation

Clinton Health Access Initiative (CHAI) - Malawi July 2014

An operational and health impact evaluation of the multi-partner

implemented emergency essential medicines

parallel supply chain in Malawi

Government of Malawi

Lilongwe

2

Acknowledgements

The CHAI evaluation team would like to thank the support of the Malawi MOH,

UNICEF, CHAM, and the other organizations and partners who provided feedback on

this report. The authors of this report would like to specifically acknowledge the

support and time of the staff of the 59 health facilities interviewed for this report.

3

Table of Contents Acronyms ................................................................................................................................................ 4

Executive Summary ................................................................................................................................. 5

Purpose, Objectives and Scope ............................................................................................................... 9

Evaluation Methodology ....................................................................................................................... 10

Evaluation Findings ............................................................................................................................... 13

A. Relevance .............................................................................................................................. 13

B. Effectiveness .......................................................................................................................... 16

C. Efficiency ............................................................................................................................... 29

D. Impact ................................................................................................................................... 32

E. Sustainability ......................................................................................................................... 36

Lessons Learned .................................................................................................................................... 40

Conclusion and Recommendations ...................................................................................................... 42

Annexes ................................................................................................................................................. 43

Annex 1: Evaluation Terms of Reference .......................................................................................... 43

Annex 2: Evaluation Team ................................................................................................................ 49

Annex 3: Evaluation Questions ......................................................................................................... 50

Annex 4: Phase I Kit Contents ........................................................................................................... 52

Annex 5: Phase II Kit Contents .......................................................................................................... 53

Annex 6: Comparison of Phase I and Phase II Items ......................................................................... 56

Annex 7: List of Interviewed Partners and Sites Visited ................................................................... 58

Annex 8: Reported List of “Most” and “Least” Important PHC kit, Over and Undersupplied Items by

Facility Level ...................................................................................................................................... 62

Annex 9: Analysis of Most Important, Least Important, Stocked Out, Under and Oversupplied Kit

Item Lists ........................................................................................................................................... 66

Annex 10: Tracer Item Selection Matrix ........................................................................................... 68

Annex 11: Under and Overstocked Items and Expiries..................................................................... 69

Annex 12: Evaluation Surveys ........................................................................................................... 71

Annex 13: Bibliography ..................................................................................................................... 85

4

Acronyms

ARIs Acute Respiratory Infection

CCTV Closed Circuit Televison

CHAI Clinton Health Access Initiative

CHAM Christian Health Association of Malawi

CMST Central Medical Stores Trust

DAC Development Assistance Committee

DFID Department for International Development (UK)

DHO District Health Office (or Officer)

FEFO First Expiry First Out

HMIS Health Management Information System

HSA Health Service Assistant

HTSS Health Technical Support Services - MOH

JSI John Snow Inc

KfW Kreditanstalt für Wiederaufbau (Reconstruction Credit Institute)

LMIS Logistics Management Information System

MDG Millennium Development Goals

MMR Maternal Mortality Rate

MOH Ministry of Health

O&M IT/S O&M Information Technology Services

OECD Organization for Economic Cooperation and Development

OPD Out Patient Department

PHC Primary Health Care

PMPB Pharmacy Medicines and Poisons Board

PODs Proof of Delivery

RTT RTT Group (Now Imperial Health Sciences)

UNICEF United Nations Children’s Fund

WHO World Health Organization

Executive Summary

PHC Project Overview

The Primary Health Care (PHC) Essential Medicines project was launched in January 2012 to address

the essential medicine and health commodity stock out and under supply emergency in Malawi. A

project of national reach, the project supplied standardized kits of essential medicines and supplies

on a monthly basis to over 620 Ministry of Health (MOH) and Christian Health Association of Malawi

(CHAM) facilities in all of Malawi’s 28 districts through a parallel supply chain. The project received a

total of USD39M1 in funding from the governments of the United Kingdom, through DFID, Norway,

through the Royal Norwegian Embassy in Malawi, and Germany, through KfW.

The project was coordinated by UNICEF under a multi-partner project approach. PHC Project

partners included the MOH, the Pharmacy, Medicines and Poisons Board (PMPB), the Central

Medical Stores Trust (CMST), international donors, and John Snow Inc. (JSI), who oversaw the

warehousing and distribution planning for the kits. The project was designed to be implemented

from January 2012 to June 2013. It was then extended to the fourth quarter of 2013 in order to

distribute the remaining kits and supplies.2 The last full distribution during this phase of the project

took place in September 2013, with partial deliveries continuing into October 2013.

PHC Project Goals and Objectives

The overall goal of the PHC project was to contribute towards keeping mortality and morbidity

among children under 5 and pregnant women on a constant declining rate in Malawi. It was

intended to meet the 2015 Millennium Development Goals (MDG) 4 &53 by averting drug shortages

over the implementation of the project. Operational objectives included: the implementation of an

efficient and accountable supply chain system; national level project oversight; a functioning

independent monitoring mechanism; health workers sensitized and knowledgeable about the kit

items; and an effective communication campaign that sensitized communities about the kit project.

In addition, in order to address longer term national supply system concerns, the Central Medical

Stores Trust (CMST) was to receive 55% of the kit costs from recipient MOH and CHAM facilities to

support their recapitalization initiative.

1

Using end of project September 2013 USD / GBP, NOR and EURO conversion rates 2

During the extension period ( June to October 2013) the warehousing and distribution of the kits was transitioned to UNICEF 3

MDG4: Reduce child mortality by two-thirds, between 1990 and 2015, the under-five mortality rate; MDG5: Reduce by three quarters, between 1990 and 2015, the maternal mortality ratio

6

Project Accomplishments (as of September 2013)

15,844 kits in total were distributed to over 620 MOH and CHAM facilities in 28 districts

The total number of medicines and supplies distributed by the PHC project potentially

served up to 9,487,697 patients.

The kits program reduced product stock outs from 31% to as low as 7%, with 81% of

facilities reporting that the project reduced the incidence and duration of product stock

outs.

An estimated MKW 4,202,126,365 (US$8,670,690) in total went to the recapitalization of

the CMST from the PHC project.

Under its multi-partner model, the project was rapidly set up and exhibited a high degree of

supply chain visibility and controls and all PHC kit contents passed PMPB quality testing.

Project Implementation Challenges

Communication and coordination challenges were experienced with CHAM facilities as many

facilities believed that the kits were free.

As a result of the standardization of the kit contents and their quantities, over and under

stocks were experienced with 75% of recipient facilities reporting some level of product

redistribution to mitigate expiries. However, there were relatively few expiries experienced

during its implementation.

Two major in-country distribution delays were experienced. The first in April 2012, due to

vendor delays, and the second in June 2012, due to the delays in the Beira corridor and the

relocation of the UNICEF warehouse in Copenhagen.

Overall Project Findings

The PHC project was a timely and relevant project and while it did not eliminate essential medicine

and supply stock outs in Malawi, it had a significant impact in reducing their occurrence and

duration. Eighty-one percent (81%) of facilities interviewed agreed or strongly agreed that the PHC

project prevented stock outs at their facility. The kits program reduced product stock outs nationally

from 31% in 2011 to as low as 7% in 2013 for selected items.4 In addition, a total of 89% facilities

either strongly agreed or agreed that the kits increased patient access to essential medicines. In total

15,844 kits were distributed with no quality issues regarding their contents reported. These needed

medicines and supplies potentially served up to 9,487,697 patients.

The overarching objective of the project was to contribute towards keeping mortality and morbidity

among children under 5 years and pregnant women on a constant declining rate to meet the 2015

MDG 4 & 5. While there is no way to conclusively link the contribution of the kits program to keeping

mortality and morbidity among children under 5 and pregnant women on a constant declining rate,

it is clear that the PHC’s improvement in the availability of specific commodities contributed to this

overall decreasing trend.

Overall, the PHC project was also found to be effectively and efficiently implemented. The rapid

planning and launch of the project was particularly notable, given the national scope and complexity

4 See Page 10 for selected tracer items

7

of the project. The project demonstrated a high degree of project visibility as demonstrated by

routine coordination meetings, regular PMPB product quality reporting, as well as the continuous

independent monitoring of the project’s implementation. The project achieved a high degree of

inventory and supply chain visibility in particular, as demonstrated by monthly distribution

summaries, the complete accountability of PODs, as well as the immediate identification of an

incidence of theft in 2012. A total of 88% of the health facilities said that the PHC kits met their

overall delivery expectations. In general about 69% of the sites said the deliveries occurred within

one week of their expectations.

The PHC project contributed an estimated MKW 4,202,126,365 (US$8,670,690) to the

recapitalization of the CMST. While this is a significant contribution, as a result of invoicing and

payment challenges, this potentially only represents roughly 65% of the revenue that CMST should

have received from the kits program. CHAM facilities, in particular, were noted for their delayed or

lack of payment for the PHC kits, mostly due to budget constraints. In addition there were

communication and project coordination issues with CHAM sites, who were initially under the

impression that the PHC kits were free.

Despite the prioritization of communication outreach and mass media sensitization efforts, 48% of

facility staff felt that the mass communication campaign did not reach patients and communities,

with 28% of clinical staff reporting that they had first learned about the kit program when the kits

arrived at their facility. As result of the standardized nature of the kits, 75% of facilities reported

redistributing oversupplied or unneeded PHC items to other facilities or to their district hospitals. In

addition, despite the overall improvement of LMIS rates in Malawi, the recording of PHC items on

stock cards was inconsistent among facilities, with about 24% of monthly stock entries

undocumented or incorrect, potentially introducing those products to a risk of leakage at the health

facility level.

Conclusion

The PHC was a timely and effective medicine and health commodity supply delivery intervention. It

reduced the occurrence and duration of health commodities stock outs, particularly at the rural

health facility levels, and its contribution to primary health care in Malawi is a considerable

achievement.

The PHC project’s contribution to the recapitalization of CMST was meant to address concerns about

the sustainability of the project. However, as evidenced by the eventual revival of the kits project in

2014, the recapitalization of CMST did not yield capacity and performance outputs by the end of the

PHC project as intended. While the PHC project was largely a project success, the MOH, CMST and

other partners, ultimately need to address the broader supply system issues that prompted the need

for the PHC project intervention in the first place. By implementing the 2012 Joint Strategy for

Supply Chain, and strengthening the public supply system, Malawi’s public health system will be able

to better plan against and respond to emergency stock out situations and begin the process of

building a more sustainable national supply delivery system.

8

Evaluation Recommendations

Design kits specific to health centers with maternity wards, facilities with surgical wards, and

central hospitals for future kit project implementation.

Revive and implement the 2012 Joint Strategy on Supply Chain Integration Plan.

Strengthen CMST’s invoicing and account reconciliation capacity.

Standardize and streamline product redistribution and relocation policies and procedures.

Implement more targeted supply management improvements at the DHO level.

Strengthen the inventory management capacity and performance of CHAM facilities.

9

Purpose, Objectives and Scope

Evaluation Purpose

The purpose of the evaluation is to conduct a comprehensive and systematic evaluation and review

of the impact of the PHC Essential Medicines project during its implementation. As Malawi’s public

health system maintains multiple, parallel supply and distribution systems, which pose longer-term

coordination, financing and sustainability challenges, this evaluation is intended to provide more

evidence on the effectiveness, implementation, and health impact of such a parallel system.

In 2012, the MOH, CMST and partners developed and agreed to the implementation of the

Integrated Supply Chain Strategy. As such, this evaluation also aims to inform the implementation of

that strategy and to draw operational lessons from the successes and challenges of the PHC project.

Evaluation Objectives

The objective of this evaluation is to assess the implementation of the PHC project and its health

impact in Malawi, particularly its impact ensuring patient access to Primary Health Care products.

The PHC project was assessed on its relevance, effectiveness, efficiency, impact, sustainability, and

equity.

Scope of Evaluation

This evaluation assesses the planning, establishment, implementation, and to a lesser extent, post-

project planning phases. It was guided by evaluation questions based on the original UNICEF RFP

(see Annex 1). Among the areas investigated in this evaluation include project coordination, supply

chain efficiency and effectiveness, the communication components of the project, as well as health

system capacity building. The evaluation had a special focus on understanding how patients

benefitted from the PHC project, especially the project’s contribution to reducing the burden of

maternal and under-five mortality rates in Malawi.

10

Evaluation Methodology

Evaluation Framework

Based on the original submitted RFP required evaluation questions, the CHAI evaluation team

evaluated the following project areas (Figure 1):

Figure 1 Evaluation Framework

Evaluation Methods

Data Sources

The evaluation was implemented from February to May 2014. The evaluation included surveys that

were implemented at 59 health facilities. In addition, partner and donor key informant interviews

were conducted, along with a comprehensive review of PHC project documentation. Subject-

relevant research on available data was also conducted. Reviewed documented sources included:

• Health center and hospital inventory management stock cards • UNICEF project management documentation • Logistics Management Information System (LMIS) • Health Management Information System (HMIS) • O&M IT/S and Management Services PHC Reports • Various relevant kit system program evaluations and literature on essential medicine

distribution programs The evaluation conformed to the OECD Quality Standards for Development Evaluation and

implemented a rigorous internal review process that included feedback from MOH and development

partner organizations on the draft report.

PHC Evaluation Guiding Questions

Based on the evaluation framework, the team developed a series of high-level evaluation questions,

which guided the development of the research agenda for the evaluation, partner interview tools, as

well as health facility surveys [See annex for the evaluation questions as well as the surveys].

11

Site Selection and Informational Interviews

The evaluation site selection included all MOH

central hospitals, 10 district hospitals, and

randomly selected health centers from the

northern (~20%), central (~30%), and southern

regions (~50%). A total of 59 sites were visited (44

MOH sites and 15 CHAM sites). Changes were

made to the initial list of randomly selected health

centers due to the identified overrepresentation

of sites in one district, or one particular CHAM

denomination. Health facility interviews were conducted both in Chichewa and English depending on

the comfort level of the interviewed staff. During the visits, interviews were also conducted with

village chiefs, head men and members of the community drug committees when possible. For a

complete list of health facilities visited and partners interviewed, see Annex 7.

Tracer Item Selection

In order to understand recordkeeping practices for kit items and assess potential leakage levels, 10

kit items were selected as tracer items. They were designated to be reflective of priority primary

health disease burdens as well as representative of each kit box (1-7, and briefly 1-8), as well as the

loose items that arrived under a different schedule than the regular boxes. Additional criteria were

also selected to better understand the feasibility of data collection [See Annex 10].

Amoxycillin susp bot

Cotrimoxazole, 480mg tabs

Erythromycin tabs 100s

Glucose 5% 500ml iv 20s

Oxytocin Inj

Paracetamol, 500mg tabs

Quinine inj 10s

Sodium chloride 500ml iv 20s

Sulphadoxine/Pyrimeth 1000s

Syringe+needle, 2ml unit

Evaluation Limitations

Sample Size: A total of 59 health facilities were visited during this evaluation due to time and team capacity

constraints. Due to the absence of staff the day site visit occurred, the clinical survey for Nthisi District

Hospital and the inventory manage survey Zomba could not be completed. As a result, the findings and

recommendations for both the clinical and inventory management areas were each based on data from 58

facilities.

Stock Card Data Quality: In about 31% of the cases, inventory management stock cards were incomplete,

not able to be located, or otherwise inaccessible. This evaluation team has attempted to mitigate this

concern by quantifying these gaps with the goal of identifying the levels of confidence in the data collected.

Health Center Institutional Memory: The length of time staff has worked at a health facility will likely impact

their ability to recall PHC project implementation details. Informants who have been at their facility for a

Total Sites Visited by Facility Type

Facility Type MOH CHAM Grand Total

CENTRAL HOSPITAL 4 - 4

DISTRICT HOSPITAL 10 - 10

HEALTH CENTRE 30 10 40

MISSION HOSPITAL - 4 4

RURAL HOSPITAL - 1 1

Grand Total 44 15 59

12

shorter period might be less knowledgeable about the implementation of the project at their facility, which

could have also introduced interviewee recall bias.

The following table reflects the amount of time interviewed staff have worked at their respective facilities:

Staff posted at health facility5

Clinical Management

Survey (n = 58)

Inventory Management

Survey (n = 58)

A. % of Staff who have been at their facility since 2010

41% 38%

B. Average # months worked 58 months (4.8 Years)

52 months (4.3 Years)

C. Median # months worked 36 month (3 years)

42 months (3.5 years)

D. % of staff started before the project [ Before January 2012]

60% 72%

E. % of staff started within the implementation of the PHC project [Between January 2012 – June 2013]

28% 24%

F. % of staff started during the PHC project extension period

[Between July - Sept 2013] 2% 2%

G. % of staff started in 2013 [After October 2013]

10% 2%

5 Benchmarked against the staff who participated in the survey site visits in March 2014

13

Evaluation Findings

The findings of this evaluation have been organized into six categories in compliance with the Development

Assistance Committee (DAC) Quality Standards for Development Evaluation. A qualitative ratings system has

been established using the following categories: Poor, Adequate, Good and Excellent.

The following table summarizes the broad findings in each category.

Evaluation Category Evaluation Score

Relevance Excellent

Effectiveness Good / Excellent

Efficiency Good / Excellent

Impact Good / Excellent

Sustainability Poor

Equity Excellent

In the following subsections, each of these evaluation categories is examined in further detail. First, the

framing questions of the evaluation category are presented, then the findings are presented, with the key

points bolded at the beginning of the section.

A. Relevance

Evaluation Category: Relevance Evaluation Score: Excellent

Category Framing Questions

To what extent are the objectives of the program still valid?

Are the activities and outputs of the program consistent with the overall goal and the attainment of

its objectives?

Are the activities and outputs of the program consistent with the intended impacts and effects?

Evaluation Findings Summary

By meeting the critical supply gaps in the Malawi public health system that occurred in 2011, the PHC

Project was a timely and relevant project. The project’s utilization of a kit system, along with its partner

structure and project oversight mechanisms, was consistent with its objectives of rapidly establish a supply

chain that achieved timely and routine medicine and health supply deliveries of a national scope.

Detailed Findings

Project context

The Primary Health Care (PHC) Essential Medicines Project was launched in January 2012 in response to the

persistent medicine and health commodity under stocks and stock outs experienced by Malawi in 2011. It

14

was estimated that as much as 60%6 of all health centers and hospitals in Malawi experienced chronic stock

outs or understocks in 2011.

Figure 2 Stock outs or under supply by % in 2011 (by quarter)

These health commodity supply challenges occurred in a context of the national financial and political

challenges that occurred in 2011. These challenges included civil unrest, chronic fuel shortages, and high

inflation. In addition, the suspension of donor general and sector budget support, as well as reduced tobacco

exports in 2011 resulted in a national FOREX crisis.7 This lack of FOREX, cuts to government budgets, as well

as the inflation-induced reduction of purchasing power presented severe national health commodity

procurement challenges for public and private organizations alike. In 2011, the Central Medical Store was

reestablished and registered as the Central Medical Store Trust (CMST). In 2011, it had no CEO or governing

board and had only recently began to establish its revolving capital fund. As a result, it did not have the

inventory and operational capacity to address the health commodity emergency that emerged in 2011.

The MOH executed an emergency 6 month procurement, but also made an appeal to development partners

to support a broader emergency response to these stock outs. Discussions began in July 2011 to design a

comprehensive essential medicines procurement, storage and distribution project. This project, approved in

November 2011, received a total of USD37 million8 in funding from the governments of the United Kingdom,

through DFID, Norway, through the Royal Norwegian Embassy in Malawi, and Germany, through KfW. This

funding was intended to fund the project for 18 months.

The overall goal of the PHC project was to contribute towards keeping mortality and morbidity among

children under 5 and pregnant women on a constant declining rate in Malawi to meet the 2015 Millennium

Development Goals (MDG) 4 &5, by averting drug shortages over the implementation of the project. The

actual attainment of the MDG goals can be assessed in second half of 2014, when the results of the Malawi

End-line MDG survey become available.

6 This is based on an analysis of 10 tracer items that were a part of the PHC project evaluation and is a rough proxy for

national rates. The actual national stock out rate at that time may be higher or lower. 7 http://www.imf.org/external/pubs/ft/survey/so/2012/car080912a.htm

8 The exchange rate at the time of project approval in November 2011

15

Project Objectives

The operational objectives of the project were the:

1. Timely delivery of the pre-packed medicine and health commodity kits to the PHC facilities through a

functional, efficient and accountable supply chain system

2. Functioning project coordination mechanism established which will provide project management and

oversight at the national level

3. Functional Independent monitoring mechanism of essential commodities at facility level in place

4. Health facility workers sensitized and knowledgeable on appropriate application of the kit system

and rational use of medicines

5. Effective Communication Campaign that informs communities of availability of drugs at PHC sites

A sixth objective was not documented, but understood by the partners, was to leverage the PHC kits project

to support to the CMST recapitalization initiative and strengthen the CMST’s procurement, warehousing and

service delivery capacity.

Alignment to objectives

The project design was well aligned to the project objectives. The overall project was carefully designed to

achieve Objective 1. A prepackaged “kit” procurement and distribution model was selected by the partners

to expedite the launch of the project and as a way to contend with national distribution capacity limitations.

The project was implemented using a “Phase I” and a “Phase II” kit. The Phase I kits were distributed from

January to March/April 2012 and were based off a Zimbabwe essential medicines kit design. These kits were

made up of 37 items and its use of preexisting supplies in the UNICEF pipeline enabled the rapid start-up of

the project. The Phase II kits were designed by representatives from UNICEF, CMST, HTSS, the World Health

Organization (WHO), and to a limited extent, also DHOs. Their design, assembly and procurement occurred

in parallel to the distribution of the Phase I kits. The Phase II kits were made up of 61 items and mostly

shipped by sea. The PHC Project also consisted of loose items bulk-purchased by UNICEF. These items were

not in the kits because they were either too bulky (like the umbilical clamps) or were controlled items (such

as diazepam injectable or phenobarbitone) and so required special importation waivers and so were on a

separate distribution schedule from the rest of the kits.

The selection of partners and assignment of activities was also well-aligned to the context. As a result of its

national presence, experience with kit production, and current supplier contracts, UNICEF was selected as

the procurement partner to oversee the implementation of the entire project. JSI, due to its logistics

expertise, and existing subcontract arrangements with Cargo Management Logistics (CML) and RTT, was

selected to manage the planning, warehousing and distribution for these kits. In order to address the

substantial warehousing capacity required to store and distribute the kits, RTT (now Imperial Health

Sciences) was also subcontracted by JSI to provide expanded warehousing services for the kits. Investments

were made at the RTT warehouse in less than 3 months in late 2011. These included floor reinforcement, the

installation of racking and CCTV to upgrade their facilities to a pharmaceutical grade warehouse in

preparation for the arrival of the first consignment in January 2012.

16

The project was also able to develop an approach relevant to the goal of ensuring adequate oversight, as

described in Objectives 2 and 3. Under its partnership model, MOH, donor and partner coordination was a

central component of the PHC project. Partner meetings were first implemented monthly and then

bimonthly. Independent monitoring of the project was implemented by O&M Information

Technology/Systems and consisted of eight rounds of health facility spot checks designed to continuously

identify urgent project issues during its implementation.9 For additional information please refer to “Project

Coordination in the Effectiveness section.

In addition, an important element of this project was its public awareness campaign, tailored to achieve

Objectives 4 and 5. The campaign consisted of a mass media campaign, which included radio spots,

newspapers adverts, and even a television documentary. The goal of this mass media outreach was to

sensitize the public on the utilization of the drugs and product availability. In addition, there was also a

health facility outreach component to the project. The outreach component targeted all 28 districts, and

focused on DHOs, District Information Officers, and health facility drugstore staff, who were in turn

responsible for disseminating information to the health center level.

As an emergency response, the project was intended to relieve pressure off CMST while it established its

revolving capital fund and strengthened its operational capacity, towards achievement of the sixth objective.

The Central Medical Stores Trust (CMST) was to receive 55% of the kit costs from recipient MOH and CHAM

facilities in support of their recapitalization initiative.

B. Effectiveness

Evaluation Category: Effectiveness Evaluation Score: Good /Excellent

To what extent were the objectives achieved / are likely to be achieved?

What were the major factors influencing the achievement or non-achievement of the objectives?


The PHC project established and implemented an effective supply chain for essential medicines. In its

national distribution of a total of 15,844 kits, with only two major monthly delays, the project

demonstrated its supply chain distribution effectiveness. Routine partner meetings, independent

monitoring and a high degree of supply chain accountability, demonstrated the project’s effective project

coordination and controls.

The communication aspect of the project was found to be the least effective in terms of meeting its goals.

In interviews, 48% facility staff felt that the mass communication campaign had been ineffective and did

not reach patients and communities. In addition, project coordination with CHAM facilities was ineffective,

as evidenced by the perception that the kits were free, and the high percentage of the CHAM facilities

eventually opting out of the program. The amendment of kit deliveries was also a noted challenge. CMST

received an estimated US$8,670,690 from the PHC kit project, however, this represents only 65% of what

they should have received, mostly due to non-payment of invoicing issues.

9 June, August, and October 2012; January, April, July, and October 2013; and January 2014

17

The project achieved 5 of its 7 stated health service outputs targets, and by improving product availability,

it contributed to overall disease management capacity, particularly at the health center level.

Detailed Findings

Effectiveness was assessed in terms of supply chain performance, coordination, communication, and health

service outputs.

Supply Chain Performance

The first and primary objective of the PHC kits program was “Timely delivery of the pre-packed medicine kits

to the PHC facilities through a functional, efficient and accountable supply chain system.” The evaluation

examined the effectiveness of the PHC parallel supply chain across various aspects.

Stock Out Analysis

Eighty-one percent (81%) of facilities

interviewed agreed or strongly

agreed that the PHC project

prevented stock outs at their facility.

This parallels their view on increased

patient access to medicines. Many

facilities highlighted the dependability

of the kits supply relative to the more

inconsistent CMST supply at that

time. The 2012 and 2013 LMIS data

for the selected 10 kit tracer

items for this evaluation (See

Annex 10 for the complete list of items) shows that the consistent delivery of the kits drove down the rate of

facility stock outs from 25% to 7% during the implementation period of the PHC project (See Figure 3

above).10 There is an observable increase in stock outs in Q4 2013 after the distribution of the remaining kits

and kit items ended in September/October 2013. The MOH attributed the increase of stock outs in Q4 2013

due to the end of the kits program.11

However, the kits did not completely eliminate stock outs, and occasionally items within the kits themselves

stocked out. Thirty-three percent (33%) of the facilities said they frequently experienced stock outs even

with the kit program, with 55% saying they only sometimes experienced stock outs, and only 12% of

facilities saying they never experienced commodity stock outs. Seventy-five percent (75%) of the facilities

attributed stock outs to the short supply of health commodities, both in the kits and from CMST system.

Even with the kits, Malawi’s general environment of undersupply continued to be a challenge.

10

The 3% increase in Q3 2012 is largely attributed to increased facility stock out of Sodium chloride 500ml iv 20s and Oxytocin. 11

Monthly Inventory & Reporting Performance, HTSS, MOH, December 2013

Figure 3 PHC Tracer Item Facility Stock Out Rate: 2012 - 2014

18

Across the health facility service levels, the following items were reported as the most frequently stocked

out items from the PHC kit (alphabetically):

Amoxycillin tabs 1000

Bandage gauze 8x4m

Benzylpenicillin 5miu 50s

Calamine lot 500ml

Cotton wool 500g

Gentamycin inj 50s

Gloves exam latex M 100s

Ibuprofen 200mg tabs 100s

Paracetamol syrup bot

Paracetamol tabs, 500mg 1000s

Quinine 300mg tabs 100

Quinine inj 10s

Order Responsiveness

Kit contents, by definition, are standardized and cannot be amended. The total number of kits, however,

should have been tailored to need. However, 69% facilities strongly disagreed or disagreed that it was easy

to make changes to the number if kits. Only 14% of facilities interviewed strongly agreed or agreed that it

was easy to amend the number of kits received at their facility.

Commodity Management Capacity

The PHC project was not specifically designed to develop

in-country capacity for inventory control, quantification,

ordering and distribution planning. While the routine

and dependable supply of medicines and supplies can

indirectly reinforce stock keeping practices and reporting

practices, the results appeared mixed for the PHC

project. Fifty-nine (59%) of the sites interviewed agreed

or strongly agreed that the kits had improved the

inventory management skills of their facility. Eight-six

percent (86%) of those interviewed said they received no

specific training related to the kits, and so the skills-

building was most often understood in terms of the day-

to-day fact that commodities were available in their

stock rooms to manage. As an example, one MOH facility

mentioned that the start of the kits program was

timely as it helped their recently arrived HSA to

apply their MOH training at their facility.

Twenty-two (22%) disagreed and 2% strongly

disagreed that the kits had strengthened their

inventory management capacity. A large

number of interviewed health facility staff

reported that they had already received

drugstore and inventory reporting training from

the MOH, and since the kits used the same stock

cards, there was nothing particular to the PHC

Figure 4 Interviewee Response to inventory management skills building

Figure 5 LMIS Reporting Rates, 2012 - 2014

19

project that improved their skills.

While LMIS reporting rates did improve during the implementation of PHC project (see Figure 5), more

notable increases occurred during the end of the project, with the kit only contributing to the larger MOH

efforts to improve LMIS reporting. This includes training, mentoring, and supportive supervision efforts that

were rolled out by the MOH in late 2012 and throughout 2013. In the course of two years, national

reporting rates more than doubled from 39% to 86%, which is a significant achievement and should be

recognized. While this reporting improvement is indeed notable, as noted earlier in this evaluation,

inventory data quality remains an issue (see sections below).

Over and Understocked Items

While the benefits of using a kit distribution system include simplified procurement, storage, distribution,

and well as a reduced risk of leakage, the quantity of the items in these kits are not based on health facility

consumption data. As a result there is an inherent risk of over and understocks in this model. Annex 11

includes a reported list of the most over and understocked commodities.

Facility level stock management

The cycle check reports highlight poor recordkeeping at CHAM facilities in particular during the

implementation of the kits project. As an example, the August 2012 report from Independent Monitors

indicated that 13% of the CHAM facilities visited had no functioning stock card system.

Items from the PHC kits were recorded on the same stock card as MOH CMST items and commodities from

other supply systems. However, initially there appeared to be some divergent guidance of this practice, with

34% of CHAM facilities being advised to create separate stock cards for kit items. Only 10% of those facilities

actually ended up implementing such a parallel system.12 This rationale for these separate systems was to

better segregate free MOH supplies from the fee-based CHAM procured stock. The integration of free and

non-free supplies was a challenge CHAM experienced during the PHC project.

During the site visits for this evaluation inventory data was collected on 10 tracer items. Data was collected

from drugstore stock cards for a total of 12 months for each product (May 2012 to April 2013) and compared

with delivered quantities. Across all the products about 31% of monthly stock entries were excluded from

this analysis due the inability to locate or access facility stock cards. The lack of stock cards varied from

product to product. In general, it was observed that slower moving items, such as sodium chloride, glucose

5%, and erythromycin, tended to have the least complete information. While the stock entry for faster

moving items, such as paracetomol and cortimoxazole, tended to be more complete. This is presumably due

to the fact that they were more frequently prescribed and so their stock cards required more frequent

updating and were better maintained.

12

Conducting Spot Checks in Health Facilities – First Cycle. O&M IT/S and Management Services Ltd, p. 7 & 24

20

The tracer item unrecorded percentages are as follows:

“Unrecorded” in the table above is defined as products that were delivered and not recorded on drugstore

stock cards. Unrecorded product is at greater risk of leakage, as well as expiry, since it not documented in

facility records. Amoxicillin was eventually excluded from this analysis due to the inconsistency of the unit of

measure recorded, indicating that at many facilities amoxicillin in both syrup and tablet form were

frequently entered on the same stock card. It is important to recognize the limitations of the above

estimates due to missing or inaccessible health facility stock cards.

Health Facility Storage Practices

Fifty-three percent (53%) of the sites visited indicated that they had inadequate storage space when

managing the kit products. Some facilities indicated that due to the lack of space or shelving they kept kit

items in the boxes they were delivered in. This sometimes resulted in the stacking of boxes on top of each

other, which made the implementation of FEFO inventory management and the identification of expired or

expiring products difficult.

Product Expiry

As a result of the long expiry dates of the kit contents and the fast moving nature of most of the kit items,

product expiries were found to be a rare occasion during the implemented evaluation site visits. In the

monitoring cycle reports, almost zero wastage was reported until the end of 2013.

Expiries were also largely mitigated through the relocation of slow moving or near expiring items to higher

volume facilities, such as a district or central hospitals for use or further redistribution. The eighth (and final)

cycle report issued in January 2014 identified a list of items that had expired in 2013 and were expected to

expire in the first half of 2014. They included:

Products Expired in 2013

• Albdendazole

• Benzyl Benzoate lotion

• Cortimoxazole

• Doxycycline

• Erythromycin

• Metronidazole

• Promethazine

Tracer Item

Units

Delivered

Units

Recorded on

Stock Card

Total

Unrecorded

Units % Unrecorded

Amoxycillin susp bot 141,840 Undetermined Undetermined Undetermined

Sodium chloride 500ml iv 20s 14,100 8,864 5,236 37%

Erythromycin tabs 100s 7,800,000 5,017,096 2,782,904 36%

Glucose 5% 500ml iv 20s 15,320 10,340 4,980 33%

Sulphadoxine/Pyrimeth 1000s 2,684,000 1,834,018 849,982 32%

Cotrimoxazole 480mg tabs 500s 23,166,000 17,659,733 5,506,267 24%

Paracetamol tabs, 500mg 1000s 20,096,000 16,075,192 4,020,808 20%

Oxytocin Inj 32,426 25,980 6,446 20%

Quinine inj 10s 102,480 95,426 7,054 7%

Syringe 2ml w needle 100s 105,400 102,200 3,200 3%

21

Products to expire by mid-2014

• 50% Dextrose

• Albendazole

• Benzyl Benzoate lotion

• Cotrimoxazole

• Doxycycline

• Hydrochlorhiazide

See Annex 11 for additional details on expired items.

Product Quality

In general, nearly all facilities spoke highly of the quality of the items contained in the kits and voiced no

complaints or concerns. The PMPB conducted random sampling throughout the implementation of the PHC

project and all products passed post-marking testing. Only one site visit mentioned a quality issue when it

was reported that IV fluid arrived with some stains on it, they were reported to the PMPB and then

subsequently returned. No additional complaints were submitted to the PMPB.

As it was an emergency situation at the time, UNICEF received a blanket waiver from the PMPB to import the

kits into Malawi. As controlled substances, phenobarbital tabs and injection, as well as diazepam, required

more specialized approvals and so were among the items distributed on separate schedule as loose items.13

This blanket waiver prompted media reporting alleging the kits project hurt local suppliers. In response, two

press releases, “Donors, Government have not ignored local pharmaceutical suppliers” (22 February 2012)

and “Drug certification and quality control in Malawi” (15 March 2012), were released by the government

and the PMPB. The latter was to address reports that the medicines being imported by UNICEF were not

certified by the PMPB.

Project Coordination

PHC Project Structure

The PHC project was implemented under an effective partnership model. As a comprehensive supply

delivery project of national scope, various health sector organizations and partners contributed to the

execution of the PHC project under UNICEF project coordination. The clarity of partner roles (see below) has

been highlighted among the strengths of the project.

Partner Project Role & Responsibilities

UNICEF

Overall Project Coordination and Oversight

Procurement and international freight

Project Communications

Independent project monitoring through O&M

IT/S

DFID, Germany (KfW),

and Royal Norwegian Embassy

Project financing

Project monitoring

MOH – HTSS Project monitoring

National inventory monitoring

13

Primary Health Care Essential Medicines Support for Malawi – Consolidated May Report, UNICEF Malawi, June 2013. P.24

22

MOH - District Health Office District level inventory monitoring and reporting

Commodity redistribution

JSI [RTT & CML]

Distribution planning and oversight

Warehousing and Distribution

Technical advisor

CMST Kit recipient invoicing

PMPB

Product waivers

Certfication of RTT warehouse

Post-marketing surveillance

Health Centers, Community Hospitals,

District Hospitals and Central Hospitals

(MOH & CHAM)

Kit receipt and storage

Patient treatment

Health facility inventory record-keeping and

LMIS reporting

Project Start-Up

The ability of the project to take advantage of existing contractual agreements enabled the rapid roll-out of

the PHC project and was a key strength of the PHC project. As the procurement partner, UNICEF was able to

supply from existing supplies from the UNICEF pipeline. While DFID, under a “conditional gift” mechanism,

was able to transfer pooled donor funding for the UK, Norway and Germany, to USAID and plug into an

existing contract with JSI, and in turn, their subcontractors, CML and RTT.

Coordination Meetings

Routine partner project meetings organized by UNICEF were a critical element of the management of the

project during the start-up and planning phase of the project and subsequently the monitoring of the

project. Initially these meetings were monthly and then they were bimonthly, with one or two delays in the

latter due to scheduling issues. The agenda of these meetings included updates on kit procurement and

distribution, presentations on the findings by the independent monitors, updates from the UNICEF

communication team, as well as reports on post-marketing surveillance by the PMPB. As the recapitalization

of CMST was also one of the objectives of the PHC project, it was also a regular agenda item. In fact, the

initial coordination meetings were held at CMST offices. In general, these coordination meetings were

mostly attended by central level partners, with only periodic attendance by CHAM or MOH DHOs.

Independent Project Monitoring

O&M IT/S and Management Services acted as an independent monitor of the PHC project. It implemented 8

quarterly health facility “spot-checks.” The first report was submitted in June 2012 and the last was

completed in January 2014. In each cycle they visited between 60 - 70 facilities and developed findings and

recommendations. They assessed various aspects of the PHC project, including kit delivery totals, record

keeping quality, product under and over supply situations, storage quality, as well as progress on kit

invoicing by CMST and payments made by districts, central hospitals and CHAM facilities.

All cycle reports included system improvement recommendations and provided rich information on the

performance of the kit program during its implementation. As noted above, findings and recommendations

23

were shared during the project coordination meetings. In one instance, over the course of the first two

cycle visits, particularly poor record keeping and drugstore infrastructure was identified at a site in Thyolo

at a site in Mchinji.14 These sites were discussed at a coordination meeting and it was decided that kits

would no longer be sent to these sites until improved record keeping at those facilities was observed.15

These spot-checks provided the project coordination team a high degree of visibility into the project and

the information to quickly response to identified issues as they arose.

Partner Coordination Challenges

Project coordination with CHAM appeared to cause the most challenges at the beginning of the project. This

is evidenced by the initial misperception by CHAM facilities that the kits were free during the initial months

of the project. The fact that 55% of the cost of the kits were intended to contribute to the recapitalization of

CMST appeared to be poorly communicated. It was only when CHAM facilities received invoices from CMST

did this become clear to many CHAM facilities.16 This misperception resulted in some CHAM sites returning

kits, stock-piling kits for their eventual return, or ultimately opting out of the project entirely. While 172

CHAM facilities were a part of the PHC project at the planning stage, only 58 CHAM facilities maintained

their participation in the project by mid-2012.

Inconsistent CHAM representation at the bimonthly coordination meetings is one likely factor for this poor

coordination. In addition, as an umbrella organization for multiple faith-based health service delivery

organizations, internal communication has been noted as being an organizational challenge. In addition,

CHAM underwent restructuring from 2010 to 2012 which resulted in the turnover of organizational

leadership, and may have limited their ability to consistently engage on PHC project-related issues.

Project Visibility and Security Responsiveness

Another key strength of the PHC project coordination was performance and implementation visibility. In

addition to the bimonthly coordination meetings and the O&M cycle reports, the coordinating partners

actively collected data against performance targets and UNICEF conducted 2 internal project progress

reports. Visibility into the distribution process, in particular, was an operational strength. Partners and

donors received distribution updates almost on a daily basis. At the end of each distribution, a summary

would be published in the local papers to inform the public that supplies were being delivered to health

centers. In addition, all proofs of deliveries (PODs) were provided to all PHC partners, including the

programs.

Product security was also a part of this project visibility. The theft of some kits was identified at the RTT

warehouse via CCTV in May 2012. Donors noted that UNICEF and JSI were quick to respond and report out

on the issue. A total of $46,000 was recouped in insurance money which was put back into operational

expenses. Another allegation of theft and resale of kits were raised at one point, was quickly investigated

but then proved to be unsubstantiated. The response to these incidences reflect the project visibility the

PHC project was able to achieve.

14

These facilities were reintroduced to the kit distribution system after introducing the use of stock cards, training staff in drug store management, repairing a drugstore wall, as well as installing a burglar bar in late 2012. 15

PHC Essential Medicines Kits Project Coordination Meeting Notes, 21 August 2012 16

Delayed invoicing by CMST may also have exacerbated this misperception

24

Health Service Outputs

Target achievement

At the start of the project, the PHC kits project management team identified 7 indicators related to

appropriate treatment of common health conditions among children. Data was collected from the FY

2009/2010 HMIS report to provide baseline rates and targets were set to maintain these baseline levels over

time. Setting these targets as a flatline indicator was deemed appropriate as the kits program intended to

respond to the national health commodity crisis and ensure continued operation of the health system at the

same level. The following table summarizes the 7 indicators, the baseline levels, and the final levels

reported in the HMIS 2012/13 report. It also lists if the targets set at the outset of the project were achieved:

Outcome Outcome Indicators 2009/2010

baseline (HMIS report)

2013 Target as stated in

logframe

2012/2013 result

(HMIS report)

Target achieved?

Availability of basic sets of essential medicines in assisted government and CHAM primary health care facilities in all 28 districts. This will enable appropriate treatment of common health conditions especially women and children to pre-crisis levels.

1.1) Proportion of children under-five seen at facility level for non-bloody diarrhea (per 1,000 children)

135

135 or above

155 Yes

1.2) In patient diarrhea death rate among under five (per 1000 cases)

3.0 3.0 or below 1.1 Yes

1.3) Acute Respiratory Infection (ARI) inpatient death rate among under five (per 1000 cases)

1.5 1.5 or below

0.9 Yes

1.4) Out Patient Department (OPD) visits (per 1,000 population)

1,316 1,400 or above

1,108 No

1.5) Under Five ARI new cases in under five population (per 1000 children)

500

500 or better 212 Yes17

1.6) Number of children under-five new cases seen at health facility level for ARI (per year)

1,100,100 1,100,100 or above

564,600 No

1.7) Number of children under-five seen at health facility level for non-bloody diarrhoea (per year)

324,000 324,000 or above

413,000 Yes

Overall, five out of seven of the outcome indicators were achieved. Over the course of the project, the

proportion (1.1) and number (1.7) of children seen at the facility level for non-bloody diarrhea increased.

This indicates increased care seeking by caregivers and increased provision of services for non-bloody

diarrhea, which can be lethal if not managed correctly with ORS, Zinc, and antibiotics. The rate of new cases

of ARI in the under-5 population (1.5) declined and in-patient death rates for diarrhea (1.2) and for ARI (1.3)

both had noticeable declines in the period under review. Because these indicators reflect in-patient

outcomes, the changes in this indicator can be reasonably presumed to be a factor of improved services

17

Note that the target for indicator 1.5 was unclear as worded: “Targets to be the same as or better than baseline”. In

this case, we are assuming that “better” means a lower number of new ARI cases in the under five population

25

within the health facility, as well as improved care seeking behaviors such that caregivers bring children for

care earlier in their illness.

Outpatient department visits (1.4) declined

in the period under review, which was

contrary to the target of maintaining levels

of patient visits. This decline reflects a year-

on-year decline dating to the 2009/2010

fiscal year. Declining OPD utilization appears

to be driven in large part by the dramatic

reduction in visits related to malaria (See

Figure 6). Between 2009/2010 and

2012/2013, the share of all OPD cases

attributable to malaria dropped from 48% of

all cases to 21%. This reflects a marked

decline in malaria cases in the under-5

population, which has only recently been reflected in the overall population. In the same time period, the

death rates from malaria have also declined (data not shown) indicating that the malaria program has

achieved successes in both preventing malaria cases and effectively treating them. Because the malaria

burden remains so high, and because malaria-related visits contribute so heavily to OPD utilization, it is

reasonable to link the decline in utilization to the successes of malaria control.

The number of ARI cases in children under-5 seen at the health facility each year (1.6) also declined instead

of going up as indicated in the target. The number of cases seen at the facility declined by nearly half

between 2009/2010 and 2012/2013. This decline in visits should be driven primarily by the declining case

rate in the population, which declined by 58%. This decline may be due in part to other public health

successes beyond the kits program. The pneumococcal vaccine was introduced nationally for all children

under the age of 1 in November 2011. Uptake was immediately high in the target age group, and at least

some of the decline may be attributable to reduced incidence of pneumococcal illnesses. Another possible

explanation may have to do with improved community case management of ARI, where children treated in

their communities may not have their data captured in the HMIS, however it is notable that no similar

decline was seen in the diarrhea cases, which may also be addressed through community case management.

In the case of both indicators that did not achieve the targets set out at the beginning (1.4 & 1.6), the flaw

may be with the indicators and non-achievement may actually reflect public health successes.

Health Service Utilization

In order to complement health service utilization data obtained from the HMIS, health workers at the

sampled facilities were asked for their perceptions of the impact of the kits program on health service

utilization. Clinicians (n=58) were asked whether the kits program encouraged more patients to attend the

facility. Overall, 86% of respondents responded yes, with an additional 3% stating that this was somewhat

true. The results were similar for the subset of clinicians who had been at the facility since January 2010 or

earlier (data not shown). Seventy-five (75%) of clinicians who had been at their present facility since January

2010 or earlier (n=24) felt that there were more patients attending the facility than before the kits program,

0

500

1000

1500

Axi

s Ti

tle

Year of HMIS report

Declining malaria cases in Malawi

Malaria cases<5per 1000 <5population

Malaria casesper 1000population

Figure 6 Malawi Malaria Cases

26

and 78% of those who thought there were more

patients attending (n=18) felt that this was

attributable to the kits program.

This is an interesting finding to note in light of the

overall decline in OPD utilization registered in the

HMIS and it may reflect a perception of general

community openness to service utilization, rather

than an actual increase in numbers of patient visits.

In fact, by using HMIS data to examine the OPD

utilization just in the facilities sampled for the

assessment (excluding 6 facilities with incomplete

data), there is also a marked decrease in OPD

utilization after 2010, with a significant drop from

2010 to 2011 and then relatively steady utilization from 2011 - 2013.

Qualitative responses volunteered by health

workers at 3 facilities shed further light on the

matter. These clinicians felt that the number of

onward referrals from the facilities had reduced

as a result of the kits program, because when

patients found the medicines they needed,

there was less need to refer them to a higher

level. It could also be hypothesized that with

good commodity availability, patients may be

less likely to make repeat visits to the health

facility for the same condition. Reducing

unnecessary referrals and unnecessary repeat

patient visits would be a positive outcome of

the kits program, although it would lead to a

reduction in OPD utilization in the national data. While the available data is not strong enough to

conclusively determine the contribution of the kits program to reduced OPD utilization, there is further

indication that the decline is actually an indication of health system success rather than failure in the context

of the kits program.

Even more importantly than patient visits was the impact that the program had on patient access to needed

medicines. Overall, 90% of respondents agreed or strongly agreed that the kits program increased patient

access to needed essential medicines. A similar question was asked to clinicians who had been at their

current facilities since at least January 2010, asking them to compare access to essential medicines before

the kits program in 2010 and in 2013, and 75% of these respondents (n=24) observed increased access with

most of the remaining observing no change.

Figure 7 Patient Facility Attendance Responses

Figure 8 Annual OPD visits

27

Disease management

In order to understand how the kits program contributed to the ability of health workers to manage

diseases, health workers were asked about their ability to manage individual target diseases and conditions

with the resources available to them. Overall health workers indicated a high degree of comfort with disease

management, particularly for commonly managed conditions including diarrhea, ARI, and fever, with 84% -

88% able to manage all or most cases. Comfort levels were lower with management of syphilis, with only

76% of health workers able to manage all or most cases, though this may reflect the syndromic management

approach where syphilis is not treated as a disease entity on its own. Syphilis management may also have

been complicated by the unavailability of diagnostic tests, which were not included in the kits. Comfort

levels were least with severe or complicated malaria and postpartum care with 71% and 66% respectively.

Further analysis shows that the majority of those facilities where the health worker was not able to manage

the majority of cases in these last two instances were low level health centers where referrals for severe

malaria or postpartum complications would be appropriate.

Encouragingly, the numbers of health workers who reported not being able to manage any cases were

negligible for all conditions except for post-partum care. Again, those not able to provide post-partum care

were located in lower level facilities with no delivery wards. The most common cause of not being able to

manage cases was the need for referral to a higher level (60%), which is appropriate in many cases. The

second most common cause for not being able to manage cases was lack of medicines (14%). Although this

is a minority of facilities, this is concerning and reflects continued issues with stock outs of medicines

required to manage common health conditions.

Communication

The communication component of the PHC project was a key element of its design. The three objectives of

the communication initiative were to A) ensure the utilization of the medicines B) sensitize communities on

their availability and C) sensitize health facilities, DHOs and the DHMT on recapitalization. This initiative

included a mass media component (radio, newspaper and television documentaries) as well as management

outreach meetings. This evaluation only focused on objectives A & B.

At the health facility level, twenty-eight (28%) clinical staff first learned about the kit program when the

kits first arrived at their facility. Ten percent (10%) learned about the kits from the radio. The rest (48%)

first learned about the PHC project through other various management channels. This includes directly

from the DHO (7%), a CHAM management memo (7%), and MOH Meeting (5%). Three percent (3%) heard

about the kits directly from a visiting UNICEF representative. Much of the sensitization and management

communication efforts were focused on the district level and it appears that this messaging did not always

trickle down to the health center level given the varied responses as to how facility staff first learned about

the kits.

Diarrhea ARI Syphilis Deworming Fever

Severe/

complicated

malaria

Post-partum

care

Able to manage all or most cases 84% 86% 76% 86% 88% 71% 66%

Able to manage some cases 14% 12% 19% 12% 10% 26% 24%

Not able to manage any cases 0% 0% 3% 0% 0% 2% 9%

28

In facility interviews 48% facility staff felt that the mass communication campaign was ineffective and did

not reach patients and communities. Many clinicians said that their communities did not receive

newspapers and that they also did not recall community members mentioning the PHC radio and newspaper

advertisements and ultimately doubted that they were reached by them. Facilities often mentioned that,

like with some facility staff, community members often first find out about the kits when they saw the

delivery trucks arriving in their community.

Strengthening CMST

The recapitalization of CMST was among the main objectives of the PHC project. CMST was to receive 55% of

the kit revenues from both MOH and CHAM facilities. As far as the distribution of complete kits, a total of

3,703 Phase I and 12,141 Phase II kits were distributed under the PHC kits program. As indicated in the table

below, the amount that CMST is expected to be capitalized is roughly USD$13,229,360.

The invoicing from CMST and payment by the MOH and CHAM as closely monitored during the partner

coordination meetings as well as documented in the O&M reports. In general, it was observed by the

independent monitors that the invoicing from CMST was regular and timely, with some challenges related to

delayed payments from the Central Hospitals (at least initially) and more notably from CHAM facilities.

The table below reflects the total financial amount invoiced and received by CMST as of April 2014:

As reflected in the above table, overall, 76% of invoiced totals have been paid to CMST. Eighty-three (83%)

of the amount invoiced to MOH was paid, while only 24% of the amount invoiced to CHAM hospitals having

been paid. CMST was not able to provide information related to the amount invoiced to CHAM community

health and CHAM community hospitals. According to the cycle reports, it was unclear if all the CHAM health

facilities received invoices from their umbrella organizations. In terms of their lack of payment, many

facilities cited budgeted constraints and other financial commitments.18 Payments from CHAM to CMST may

18

6th

and 8th

cycle reports

Total Kits

Distributed

Cost / Kit

(USD) Total Expensed

55% of Total

Expensed

Total Phase I 3,703 $594 $2,199,582 $1,209,770

Total Phase II 12,141 $1,800 $21,853,800 $12,019,590

Grand Total $13,229,360

Health FacilityTotal Kits

Received (#)

Total Amount

Due for Kits

Received (MKW)

Total Amount

Invoiced

(MKW)

Total Amount

Paid (MKW)

Total

Outstanding

Balance

(MKW)

% of Revenue

Received

1. MOH - Health Centres 10,744 2,712,554,307 2,712,554,307 2,352,943,987 359,610,320 87%

2. MOH - District Hospitals 2,200 718,480,827 718,480,827 590,541,642 127,939,185 82%

3. MOH - Central Hospitals 792 245,326,853 245,326,853 125,016,244 120,310,610 51%

4. CHAM - Health Centres Data Unavailable Data Unavailable Data Unavailable Data Unavailable Data Unavailable Data Unavailable

5. CHAM - Community Hospitals Data Unavailable Data Unavailable Data Unavailable Data Unavailable Data Unavailable Data Unavailable

6. CHAM - Hospitals 2,201 525,764,377 525,764,377 126,474,044 399,290,333 24%

Grand Total 15,937 4,202,126,365 4,202,126,365 3,194,975,917 1,007,150,447 76%

29

have also been complicated by some CHAM facilities requesting quarterly distributions, the return of unused

kits,19 and the fact that many CHAM sites eventually opted out of the project. While 172 CHAM facilities may

have been a part of the project initially, as of mid-2012, kits were only being delivered to 58 CHAM sites. In

addition, the inclusion of the loose items, and challenges related to fluctuating Malawian kwacha, may have

also further complicated the tracking of invoices and payments related to the PHC project.

While CMST may have not yet received all the revenue that was owed to them yet, it was clear that revenue

from the kits was significant. The MKW 3,194,975,917 in revenue CMST received from the kits converts to

US$8,670,690 in September 2013 dollars.20 However, this also represents a deficit of US$4,558,670 in

expected revenue, with CMST only receiving roughly 65% of what they should have from the kits program21.

C. Efficiency

Evaluation Category: Efficiency Evaluation Score: Good / Excellent


Were objectives achieved on time?

Were activities cost-efficient?

Was the program or project implemented in the most efficient way compared to alternatives?


The project was implemented on time and largely met its targets for supply chain efficiency. The cost per

kit was reasonable and the majority of funds were expensed on direct commodity-related costs. However,

due to the emergency nature of the project and the limited number of logistics providers of sufficient

capacity, opportunities to explore additional cost efficiencies were limited. Due to the accepted risk of over

and under stocks integral to kit systems, 75% of PHC kit recipient facilities reported redistributing some kit

items to some extent.

Detailed Findings

Timeliness

Project efficiency was demonstrated by the rapid 5 month planning and start-up of the project as well as the

project’s performance in distributing the kits.

The PHC project was designed from July to October 2011, with the first kit order submitted in November and

delivered to Malawi in January 2012. This is a notable achievement as in addition to the actual design of the

project, project start-up work included: donor funding discussions; JSI, RTT and CML contract coordination;

the refurbishment of the RTT warehouse (which increased volume by roughly 50%);22 the development of a

19

35 kits were returned by CHAM from August to October 2012, 4th

Cycle Report, January 2103, p.12 20

1USD = MKW368, 30 September 2013 (End of kits program) rate, http://www.oanda.com/currency/converter/ 21

As CMST uses Malawian Kwacha, due to fluctuating exchange rates, this should be considered a rough estimate. 22

USAID | DELIVER PROJECT Performance Evaluation Malawi 2007 – 2013, December, 2013. p. vii

http://www.oanda.com/currency/converter/

30

performance management framework; as well as the execution of an international procurement and kit

packing operations.

The distribution of the PHC kits was also very efficient as only two gaps in its monthly distribution were

experienced during the entire implementation of the project. These delays were mainly due to upstream

procurement and unexpected port issues, rather than the distribution providers. The PHC distribution

system regarded as an efficient and reliable supply source for essential medicines and health supplies by the

receiving facilities. Eighty-eight (88%) of health facilities that received the PHC kits said that their delivery

expectations were met.

In the transition to the full, Malawi-specific kit design under Phase

II supplier delays were encountered. For a short period certain

items were sourced from an approved supplier in Kenya, and due

to issues with the Beira port in Mozambique, the kits were also

temporarily run through South Africa. Due to these operational

adjustments, variations in the Phase II kit configurations were

introduced with the box configuration going from 7 boxes to 8,

and then eventually back to 7 again.

The fuel shortages in 2012 were also a major challenge for the

national kit distribution. The MOH established mechanisms with

the Malawi Energy Regulatory Authority (MERA) to ensure access

to fuel for deliveries through both government and parallel supply

systems.23

The original end date of the project was June 2013 and was

extended to September / October 2013 in order to distribute the

excess stock that emerged from the distribution gaps in April and

June 2012, as well as other remaining stock that had accumulated

in the country. This extension period also included the distribution

of the product already in the UNICEF procurement pipeline.

Initially, an option was proposed during the project coordination meetings in early 2013 that CMST would

distribute the kits post-June 2013, however, it subsequently determined that further operational reforms

and improvements still needed to occur at CMST.

Cost Efficiency

In support of the PHC project, donors provided a total of USD39,230,794.24 Eighty percent (80%) went to

procurement and related management costs, and 20% went to warehousing, distribution and related

management costs. Of the 80% in UNICEF procurement and management costs, 95% of that amount went

to the procurement of drugs and only 5% went to direct and indirect project implementation costs.25 This

indicates that in-country project implementation funding was used efficiently.

23

Primary Health Care Essential Medicines Support for Malawi – Consolidated May Report, UNICEF Malawi, June 2013. 24

In September 2013 USD 25

UNICEF Funds Utilization Report, Primary Health Care Essential Medicines Support for Malawi – Consolidated May Report, June 2013

Figure 9 Facility Perceptions on PHC Kit Delivery

31

Even though they were not free, even with the reduced cost, both DHOs and CHAM facilities reported that

the kit project was able to reduce pressure on their budgets and which made them go further. One CHAM

facility mentioned that even with the 45% discount, many of the kit items were much less expensive than

the same products sold by private suppliers for the PHC project. This savings enabled this facility to procure

more medicines and then pass along the savings to their patients.

JSI distributed the kits from January 2012 to June 2013 and from July to September/October 2013 they were

distributed by UNICEF. On average, it cost $506 to distribute a kit under the JSI-managed distribution

compared with $354 to manage under the UNICEF-managed system. This higher cost is perhaps most likely

explained by the added costs of the sub-contracting of warehousing and distribution services to RTT and

CML. Due to the emergency nature of the project, the PHC project tagged onto an existing warehousing and

distribution arrangement under USAID. If contract negotiations for warehousing and distribution services

had been more dedicated and the project planning horizon was longer, more opportunities for identifying

cost saving in this area might have been further explored.

Efficiency consideration of the kits approach

While kitting costs add additional costs to the procurement process, in the terms of performance, they also

introduce efficiency gains, demonstrated by the rapid start-up of the project, as well as a more streamlined

distribution planning and implementation process.

However, by their nature, kit programs are also limited in the efficiencies they can achieve at the facility

level due to the standardization of their contents and their quantities. As a result of the risk of product over

and under supply, redistribution is an inherent part of a kit system and was a regular occurrence under the

PHC project. Forty-four (75%) of the 58 facilities visited reported the relocation of PHC items to manage

oversupplied stocks. Most of the relocation was from Health Centers to District Hospitals, and from low to

high patient volume sites. These were mostly slow moving items or items that were not appropriate for

health centers, particularly if they lacked a maternity ward. It was noted by a CHAM representative, that

since the kit contents were standardized, it was also difficult to relocate items because after a certain point,

Donor

Procurement,

Shipping & Mgmt -

UNICEF

Logistics,

Distribution & Mgmt -

JSI /RTT

Logistics, Distribution

& Mgmt - UNICEF

(Extension)

Total ContributionUSD

Exchange Rate

30 September 2013* Total USD

DFID - GBP 9,132,541 4,492,262 30,950 £13,655,753 1.614 $22,034,377

Norway - NOK 59,600,000 - 2,950,000 kr 62,550,000 0.167 $10,436,468

Germany (KfW) - EUR 5,000,000 - - € 5,000,000 1.352 $6,759,950

Total (USD) $31,440,113 $7,248,534 $523,158 $39,230,794

* End of Project Extension Date, Exchange rates from Oanda.com

Logistics,

Distribution & Mgmt

Total Kits

Distributed

Donor

Contribution

(GBP)

Donor

Contribution

(NOK)

Donor

Contribution

(USD)

Average

Distribution

Cost / Kit

(USD)

JSI, RTT & CML 14,313 4,492,262 0 7,248,534.27$ 506.43$

UNICEF 1,531 30,950 2,950,000 542,147.18$ 354.11$

GBP 1.614

NOK 0.167

USD

Exchange Rate

30 September 2013

32

facilities were all oversupplied with the same items. Albendazole, Amoxycillian, Erythromycin and Benzyl

Benzoate were specifically highlighted as having this problem. As the kits were not distributed for free, the

lack of shared financial management systems between the MOH and CHAM (as well between individual

CHAM sites themselves), further added a layer of complexity to redistribution of oversupplied items.

D. Impact

Evaluation Category: Impact Evaluation Score: Good / Excellent


What has happened as a result of the program or project?

What real difference has the activity made to the beneficiaries?

How many people have been affected?


The PHC demonstrated project impact by increasing product availability at health facilities, and in turn,

contributed to the decreasing mortality and morbidly trends among pregnant women and children under

5. The overarching objective of the project was to contribute towards keeping mortality and morbidity

among children under 5 years and pregnant women on a constant declining rate to meet the 2015 MDG 4.

While there is no way to conclusively link the contribution of the kits program to keeping mortality and

morbidity among children under 5 on a constant declining rate, it is clear that the PHC’s improvement in

the availability of specific commodities contributed to this overall decreasing trend. Maternal mortality

has also declined, though at less marked rates. Specific kit contents relevant to the key direct causes of

maternal mortality may have contributed to its decline during the implementation of the project.

Detailed Findings

Under-5 child mortality

General Trends in Under-5 child mortality

Malawi has made steady progress in reducing Under-5 mortality over the last several decades. The

Demographic and Health surveys have recorded a steady and sustained decline.

In 2014, in the report titled Committing to

Child Survival: A Promise Renewed: Progress

Report 2013, UNICEF declared that Malawi,

along with 6 other countries, had achieved

Millennium Development Goal 4, related to

child survival in 2014, a year early. The

country had gone from 244 under-5 deaths for

1,000 live births in 1991 down to 71, a 71%

33

reduction, with an annual decline in that period of 5.6%.

The exact causes of these overall declines are not entirely clear. It is likewise not possible to estimate the

exact contribution made by the kits program to the continuous decline in child mortality in Malawi.

Population-level mortality assessments

require vast amounts of rigorously collected

data, which is beyond the scope of this assessment. Additionally, due to the wide reach of the kits program,

there is no effective “control” population that could provide a comparator to assess how the program

contributed. However, by assessing the overall contributors to child mortality in Malawi, and determining

how the kit contents addressed these, we can determine if the kits were able to contribute to the sustained

declines.

First, we should assess whether the overall decline in child mortality is primarily caused be environmental

factors beyond the health system that could underlie improved survival. One common underlying factor

that leads to overall reduced mortality is improving levels of nutrition. Poor nutrition has been shown to

underlie as many as 50% of all under-5 deaths in countries like Malawi. While Malawi has registered some

gains in declining rates of severe malnutrition in children under 5, from 6% prevalence of wasting (low

weight for height) in the 2004 DHS to 4% prevalence in 2010. Chronic malnutrition in the form of stunting

(low height for age) in children under 5 also declined from 53% in 2004 to 47% in 2010. While these declines

are welcome, these malnutrition levels remain extremely high, and continue to contribute to continued

mortality of children under five.

A second potentially underlying cause of

improving health might be improving access to

water and sanitation facilities. This data is

collected in both the 2008 national census and the

2010 DHS. In the DHS, overall 79.3% of the

population obtains drinking water from an

improved source, most commonly a tube well or

borehole. However only 32.6% of the population

reports treating water before drinking, increasing

the likelihood those disease-causing contaminants

may be in the water at the time of drinking. Only

8.8% of all households use an improved toilet

facility, with the majority (75.8%) of people using a

pit latrine without a slab or an open pit. This data

indicates that water and sanitation continue to

pose a health challenge to children under 5.

In light of the continuing environmental and nutritional challenges that children under-5 face, the major

gains in mortality may be presumed to be related in large part to factors within the health system.

Causes of under-5 deaths in Malawi

Neonatal deaths are the most common cause of under-5 deaths. Between 1990 and 2012, neonatal deaths

rose from 21% of all under-5 deaths to 31%. Because neonatal deaths have declined at a slower rate than

other categories of under-5 deaths, the major improvements in mortality have come in the other age

Figure 11 Source: World Health Organization. Malawi: Health Profile. 2010.

Figure 10 Trends in Childhood Mortality, UNICEF 2013

34

categories. Infectious diseases continue to be the primary cause of deaths after the neonatal period. Both

neonatal deaths and deaths from infectious diseases are primarily addressed within the health system.

One part of the reduction in mortality should be attributable to prevention of diseases through vaccines and

other interventions. Malawi has maintained reasonably high coverage of vaccines through the Expanded

Programme on Immunisation (EPI), and has also been an early adopter of new vaccines including

pentavalent vaccine in 2002, pneumococcal vaccine in 2011, and rotavirus vaccine in 2012.26 The overall

decline in malaria cases registered in the HMIS system (as noted earlier in this evaluation) also indicates

improving malaria primary prevention. But there remains an important component of appropriate treatment

of illnesses, especially acute respiratory infection (ARI), fever, and diarrhea.

However, a strong case can be made that effective case management of common conditions contributing to

under-5 mortality has also assisted to reduce mortality. The following table summarizes the major causes of

under-5 mortality in Malawi, and the kit contents that addressed them. Some of the kit contents were

geared towards prevention of common conditions, some directly treated the disease causes, and some were

used to manage potential complications arising from the disease, for example, morbidity and mortality

associated with high fever caused by malaria.

Contribution of the kits program to declining under-5 child mortality

The contents of the kits were well-targeted towards treating the major causes of under-5 deaths in

Malawi. The most common category of deaths in children under-5 is neonatal deaths. Eighty-nine (89%) of

neonatal deaths in Malawi are caused by 3 contributing conditions: prematurity, infections, and birth

asphyxia, all three of which are addressed in some part by the kit contents. So even though the kit contents

were not targeted towards the immediate lifesaving of premature infants, the selected contents could

contribute to reduction in neonatal deaths.

Infants born prematurely are more than 10 times more likely to die in the perinatal or neonatal period than

infants who reach full term, and prematurity is responsible for 37% of all neonatal deaths. The kits included

essential commodities to ensure delivery of quality antenatal care services to mothers, thereby preventing

prematurity and neonatal deaths. Sulphadoxine/Pyrimethine prevents malaria in pregnancy and iron folate

prevents anemia in pregnancy; both malaria and anemia are known contributors to premature birth.

Infections, and particularly ascending genital tract infections, are also important contributors to prematurity

in Malawi, and antibiotics provided in the kits could be used to treat STIs identified during antenatal care.

Under-5 mortality cause

Relevant kit contents

Neonatal

Antenatal care and prevention of prematurity

SP - Prevention of malaria in pregnant women

Iron folate – Prevention of anemia in pregnant women

Benzyl Penicillin – Treatment of syphilis and sexually transmitted infections Treatment for neonates

Phenobarbitone, Diazepam – Treatment and prevention of seizures

Epinephrine – Cardiac resuscitation

Ampicillin, Cotrimoxazole, Metronidazole, Chloramphenicol – Treatment of

26

Pentavalent vaccine protects against 5 infectious agents: diphtheria, pertussis, tetanus, Hepatitis B, and Hib. Pneumococcal vaccine protects against severe pneumonia and meningitis. Rotavirus vaccine protects against severe diarrhea caused by rotavirus.

35

bacterial infections

Tetracycline eye ointment – eye infection and general hygiene

Glucose 5% - Hypoglycemia

Acute Respiratory Infection/Pneumonia

Amoxicillin, Erythromycin—Treatment of ARI

Fever/Malaria

Treatment of malaria

Quinine – Treatment of severe malaria and malaria not responding to first line Treatment of malaria complications and fever

Paracetemol syrup – Treatment of fever

Iron Folate – Treatment of anemia caused by malaria

HIV/AIDS Cotrimoxazole – Prevention and treatment of opportunistic infections.27

Diarrhea

ORS – Prevention and treatment of dehydration

Zinc – Reducing duration of diarrheal episode, prevent dehydration

Doxycycline, Metronidazole—Treatment of bacterial diarrhea not responding to other treatments

Albendazole – Prevention/treatment of diarrhea caused by parasites

The kits also included a suite of medications that can be used for treatment of neonatal infections, another

primary cause of neonatal deaths. The third primary cause of neonatal death, birth asphyxia, requires skilled

practitioners and is less responsive to specific commodities. Nonetheless, the medications provided assisted

to prevent neonatal deaths.

The drugs also provided the backbone for the

country’s successful program for the Integrated

Management of Childhood Illnesses (IMCI). This

approach allows lower cadre health workers to

use clinical algorithms based on symptoms to

presumptively diagnose and treat several of the

major and commonly recurring illnesses for

children including ARI, fever, and diarrhea.

Conclusive demonstrations of correlation

between the kits program and declines in child

mortality are beyond the scope of this evaluation.

However, by providing essential commodities, the

kits program allowed for effective case

management of common conditions that too often become deadly in the Malawian context. Therefore it is

reasonable to conclude that the kits program was able to contribute to the sustained declines in under-5

child mortality in Malawi.

Maternal mortality

General trends in maternal mortality

27

While cotrimoxazole for prevention of opportunistic infections is provided through the HIV supply chain, the cotrimoxazole contained in the PHC kits could be used to treat opportunistic infections in children who had not yet been diagnosed with HIV.

0

200

400

600

800

1000

1200

19

90

19

92

19

94

19

96

19

98

20

00

20

02

20

04

20

06

20

08

20

10

20

12

MM

R

Maternal Mortality Ratio in Malawi (# of deaths per 100,000 births)

Figure 12 Maternal Mortality Estimation Interagency Working Group, 2014

36

Although it is unlikely that Malawi will reach its MDG target for maternal mortality, there is evidence that

the rates of maternal deaths have declined over time. Much of this may be attributable to an increasing

percentage of facility births, as well as improving maternal health practices by health workers. Skilled

attendance at birth rose from 51.8% in 1990 to 65.8% in 2013. In 2010, the MDHS indicates that 73% of all

women delivered in a health facility. While the overall trend in maternal mortality is positive, the most

recent estimates indicate that progress has slowed somewhat since 2005.

Causes of maternal mortality in Malawi

The exact causes of maternal mortality in Malawi are

not well understood. However, the recent report of

Confidential Inquiry into Maternal Death (MoH 2013)

showed that 56.2% of maternal deaths are due to direct causes like bleeding, eclampsia, sepsis,

complications of abortions, obstructed labour and ruptured uterus. These findings are consistent with the

causes of maternal mortality in the region.

Contribution of the kits program to declining maternal mortality

As in the case of child mortality, the kits program provided essential products for management of some of

these direct causes of maternal mortality. Key among these is oxytocin, which is used for labor induction

and to control hemorrhage. Chlorhexidine is also an essential commodity to prevent infection in the birth

canal, and magnesium sulfate to manage pre-eclampsia. In addition, there are several commodities that

address other possible factors in maternal deaths including HIV, malaria and anemia (See table below).

Maternal mortality cause Relevant kit contents Hemorrhage Oxytocin – increase uterine contractions and control hemorrhage

Sepsis / Infections Chlorhexidine – Cleansing of birth canal

Benzyl Penicillin, Amoxicillin, Metronidazole, genatamycin—treatment of Sepsis and infections

Hypertensive disorders

Magnesium sulfate – management of pre-eclampsia

Oxytocin – labor induction to prevent obstructed labor

Other possible factors in maternal deaths

Maternal mortality cause Relevant kit contents HIV/AIDS Co-trimoxazole—management of opportunistic infections

Malaria Doxycycline – Malaria prophylaxis

Sulfadoxine Pyrimethamine - Malaria prophylaxis

Quinine – Treatment of severe malaria/ second line

Anemia Fefol – Prevention of anemia

Others Paracetamol, Cannula, Sterile gloves

Promethazine – prevent/control nausea/vomiting during and after surgery/delivery

E. Sustainability

Evaluation Category: Sustainability Evaluation Score: Poor


Figure 13 : MOH 2013. Confidential Inquiry into causes of maternal deaths, Malawi.

37

To what extent did the benefits of a program or project continue after donor funding ceased?

What were the major factors which influenced the achievement or non-achievement of sustainability

of the program or project?


While the PHC project contributed $8.6M to the recapitalization of CMST, this support did not translate

into the expected operational and financial capacity at CMST by the end of the project as originally

intended. This was due to factors external to the implementation of the PHC project, including progress on

CMST internal reforms and operational improvements.

Detailed Findings

Benefit continuation after project end

The PHC project was designed as an emergency measure to meet the national supply gap in essential

medicines that Malawi struggled with for the last several years, which reached a critical point in 2011. During

the implementation of the PHC project, essential medicine stock outs at health facilities were reduced to as

low as 7%. When the project ended in September/October 2013, there was a clear increase in product stock

outs to 12% in Q4 2013. As a result, this evaluation concludes that the PHC project did not achieve its system

sustainability goals by the end of its implementation. This conclusion is further supported by the stock outs

that occurred immediately after the end of the PHC project and the eventual revival of the kits project again

in 2014.

Factors contributing to non-achievement

A rationale for the PHC project was to create a “breathing space” for CMST to institute procurement and

internal operational reform, funded from a revolving capital fund. However, despite the PHC’s contribution

to the CMST’s recapitalization, it did not translate into expected operational capacity on the part of CMST by

the time the project ended. While CMST only received roughly 65% of what they should have from the kits

program,28 as noted earlier in this report, the US$8.6M recouped by CMST is still significant amount of

funding by any standard. Expectations on the pace of reform and capacity building at CMST between the

MOH, partners and donors were clearly not aligned. The Joint Strategy for Supply Chain Integration Strategy,

was in part, developed to add structure to this reform, however, it remains unclear to what extent its

milestones have been achieved by MoH. Additional developments, like the delayed submission of DFID

“Component II” essential medicines order in 2013, due to CMST-DFID outsourcing negotiations, added an

additional layer of complexity to CMST reform planning. In light of the significant contribution of the PHC

project to CMST recapitalization, barriers to operational reform at this point, are perhaps less a case of

financial constraints, but more a case of internal operational reform planning and its execution.

Equity

Category: Equity Evaluation Score: Excellent


28

As CMST uses Malawian Kwacha, due to fluctuating exchange rates, this should be considered a rough estimate.

38

How broadly were the benefits of the project shared among various populations and regions?

Where the benefits of the PHC project equally shared among the different levels of the health

system?

Evaluation Finding Summary

The PHC project achieved equity in its benefits due to its national reach, inclusion of all service levels in the

Malawi health system, as well as in its selection of kit items that targeted a broad spectrum of diseases

and conditions. By increasing product availability at public facilities in all levels of the health system, the

project increased equitable access to health care in Malawi.

Detailed Findings

Benefits across populations and regions

The PHC project provided essential medicines and supplies to over 620 health facilities in 28 districts. A total

of 15,844 kits were distributed during the implementation of the PHC project, which potentially reached an

estimated 9,487,697 patients. As a result of its national scope, this evaluation concludes that the project

achieved a high level of equity in terms the beneficiaries of the PHC project.

A total of 89% facilities either strongly agreed or

agreed that the kits increased patient access to

essential medicines. One facility said, "People were

dying here and were saved because of those drugs.”

Another clinician reported overhearing patients

talking about improved availability of drugs at the

health center due to the kits program. The only

facilities that disagreed with this increase in

availability were CHAM sites, which were able to

procure directly from private suppliers and so less

reliant on the kits program. One central hospital felt

that while the kits mitigated product stock outs, the

quantities were inadequate for their routine patient

load and that the kit items did not dramatically

change overall patient access to medicines.

Fifty-nine percent (59%) of facilities said that regardless of quantity, they felt that the items selected met

patient needs, with 34% saying the items somewhat met their needs, and 7% saying the items selected did

not meet the needs of their health facility. In terms of quantity, 60% of facilities said that the quantities in

the kits were met patient needs, while 40% said they did not. Isolating for just the health centers that were

interviewed, 65% agreed that the quantities meet the needs of their facilities, while 35% said they did not.

While the overwhelming facilities believed the PHC increased patient access to medicines, health facility

observations on the standardization of both the content of the kits and their quantities were not entirely

consistent – even among the health centers for which they were designed. Fifty-nine percent (59%) of

facilities said that regardless of quantity, they felt that the items selected met patient needs. While 60% of

Figure 14 Patient Access Survey Question Responses

39

the facilities felt that the quantities in the kit were adequate for patient needs. [See Analysis of Most

Important, Least Important, Stocked Out, Under and Oversupplied Kit Items in the Annex section]

As a part of this evaluation, health facilities were asked which of the PHC items they thought were the most

and the least “important.” Importance was defined in terms of both having the broadest use and its

criticality in terms of meeting the needs of that facility. Items listed as the “least” important should be

understood as items perceived as having least relevance for their patient needs.

Across all health service levels, inclusive of MOH and CHAM sites, the following items were considered the

most “important” items in the PHC kits (alphabetized):




Cotrimoxazole 480mg tabs 500s

Doxycycline tabs 1000

Gentamycin inj 50s


Metronidazole tabs 1000s



Quinine inj 10

Across all health service levels, inclusive of MOH and CHAM sites, the following items were considered the

least “important” items in the PHC kits for their respective facility (alphabetized):

Albendazole 400mg tabs 100s

Benzoic/salicylic acid ont 40g

Benzyl Benzoate lotion 1000ml

Epinephrine (or Adrenaline) inj 10s

Promethazine tabs 100s


Suture non abs DEC3 36s

Suture abs DEC4 36s

Water for Inj 50s

Benefits across service delivery levels

The PHC project distributed medicines and supplies to all service levels in the MOH service delivery system;

this includes health centers, district hospitals and central hospitals. While most CHAM facilities eventually

opted-out of the PHC project, mostly due to financial reasons, their health centers and community and

mission hospitals also benefited from the PHC project through reported cost savings. As such, this evaluation

finds that equity across health service delivery levels and systems was also achieved.

The items in the kits included products that address various conditions and diseases. They included broad-

spectrum use antibiotics, such as cotrimoxazole and amoxycillian. Maternal and neonatal products, such as

oxytocin and and 2ml syringes, were also included. Antimalarial products such as, Sulphadoxine/Pyrimeth

and injectable quinine were also included in the kits. This evaluation concludes that equity in the

implementation of the PHC project as also achieved due to the selection of PHC kit contents.

40

Lessons Learned

The higher level lessons learned from this evaluation are:

Standardized kit distributions are effective in emergency situations and in environments characterized by

chronic undersupply and limited inventory management capacity. In the Malawi context, they are most

appropriate for rural, public health facilities.

By simplifying procurement, warehousing, and distribution planning, the utilization of the kit system under

the PHC project enabled it to quickly establish a medicine and health commodity distribution system of

national reach. As expected, the use of a kit system produced inventory over and under stocks - however it

did not result in a substantial incidence of product expiry in 2012 and 2013. This is attributed to the

appropriate selection of kit contents and their quantities, the active relocation of products as well as the

long shelf life of the PHC commodities.

MOH health centers were clearly the facilities that most benefited from the PHC project. While the hospitals

found the kits very useful, kit products were often quickly consumed in a few weeks, or often in a few days,

and so had less of an impact on their high patient volumes. Similarly, as evidenced by the number of CHAM

facilities that opted-out of the project, the PHC project was also less of an effective solution for CHAM

facilities.29 As a private organization, with facility based procurement budgets, they are able to directly

procure from private suppliers and so less dependent on the government supply in times of crisis.

Unless kits of multiple variations are available for the various health facility level and their needs, in order to

maximize the benefits of a kit distribution model in Malawi, any future kit distribution project should focus

on only supplying MOH rural health centers.

Increased product availability at the PHC level reduces patient burdens at higher levels in the health

system and leads to overall improved health service delivery.

The routine and dependable monthly delivery was cited the most often as the main strength of the PHC

project. Facilities often expressed that due to the routine supply made available by the kits, patients had

more confidence in their health center. Eighty-six (86%) of health facilities reported that they observed an

increase of patients at their respective facilities due to the PHC project. In the Malawi environment, patient

demand appears to rises to meet the available medicine and health commodity supply.

Several health centers observed that the PHC project was able to reduce the number of patient referrals to

district and central hospitals. While this evaluation was not able to confirm this observation from other data

sources, it concludes that investment in PHC services (supported by a routine and reliable supply system)

likely also has broader health system benefits in Malawi.

29

CHAM facilities were not recipients of the streamlined kits that were distributed in 2014

41

Due to the over and under supplies that result from the implementation of a kit distribution model,

product redistribution is a critical mechanism for averting product stock outs, overstocks and expiry.

Product redistribution is a frequently used mechanism by health facilities in Malawi to manage oversupplied,

expiring or unneeded commodities. Seventy-five percent (75%) of health facilities interviewed for this

evaluation reported relocating products from the PHC kits to other health facilities, usually their district or

referral hospital. Health centers without maternal or surgical wards, for example, would often relocate

either slow moving or unused products to facilities that did have these wards.

The redistribution of health center oversupplies is managed at the DHO level and as a process, is not

standardized, effectively planned or documented. It is also often limited by transportation capacity or the

proximity between overstocked and understocked facilities. This lack of standardization limits opportunities

to more optimally address undersupply and stock out issues and also introduces a risk of product leakage.

Under the PHC project, CHAM experienced redistribution challenges due to difficulties related to transferring

product ownership and costs. While due to the standardization of the kit items, both MOH and CHAM also

found it a challenge to redistribute certain products to other facilities as they were also overstocked with the

same items.

Ad-hoc product relocations add a hidden operational, financial and potentially also a health impact cost

when patients do not receive the medicines they need. Since the centralization of drug budgets occurred in

mid-2013, districts have less capacity to procure from private suppliers in the event of a stock out, which has

further increased the dependence on redistribution as a supply strategy. Until Malawi is able to implement

an effective and fully functioning health commodity order-based (“pull”) system at the facility level, kit based

or not, effective product redistribution will need to be a part of the national logistics system in Malawi.

The DHOs play a critical role in the effective planning, oversight and management of medicine and health

commodity stocks at the health center level.

The DHOs play a key role in the performance outcomes of health centers, in particular, effective inventory

and supply management. They aggregate and report on stock levels, submit resupply orders to CMST, and as

noted above, also manage product redistribution. Similarly, they also need to be able to effectively voice and

communicate the needs of their health centers to decision-makers at the central level, and so must also

serve as advocates for their district health facilities. In addition, DHOs are the mechanism in which

information is shared and improvement initiatives are rolled out to lower level facilities. The fact that

roughly a third of health facilities first heard about the kit project when the kits arrived is indicative of the

inconsistent effectiveness of communications from the DHO level to their respective health facilities.

DHOs are an overburdened and critical pivot point in the management of the Malawi health system and are

on the forefront of its clinical and operational performance and success. Due to their significant

responsibilities, they have to be experts in many technical and functional areas. They need to be effective

managers, planners, problem-solvers, and in general, also competent and effective leaders. Under Malawi’s

decentralized system DHOs have direct oversight and leadership over the health facilities in their district, as

a result, supply chain management trouble shooting and improvement initiatives should be more effectively

owned and led at this level in the health system.

42

Conclusion and Recommendations

Conclusion

The PHC was a timely and effective medicine and health commodity supply delivery intervention. It reduced

the occurrence and duration of health commodity stockouts, particularly at the rural health facilities levels,

and this is a notable achievement. However, the PHC project was intended to be an intervention of a limited

duration.

The PHC project’s contribution to the recapitalization of CMST was meant to address concerns about the

sustainability of the project. However, as evidenced by the increased stock outs that occurred immediately

after the end of the kits project, as well as the revival of the kits project in 2014, the recapitalization of CMST

did not yield capacity and performance outputs by the end of the PHC project as intended. While the PHC

project was a largely successful project, and its support to the provision of primary health care in Malawi

considerable, the MOH, CMST and other partners, need to return to the implementation of the targets and

milestones of the 2012 Joint Strategy for Supply Chain Integration in order ensure the longer-term effective

operations of CMST and decrease the kind of stock out situations that put Malawi’s health system at risk and

require interventions like the PHC project in the first place.

Evaluation Recommendations

If standardized kit distributions are planned in the future, multiple kits configurations should be

developed for health centers with maternity wards, facilities with surgical wards, and central

hospitals to better mitigate the risk of medicine and supply over and understocks.

The MOH, CMST and partners should revive and implement the priorities set out in the 2012 Joint

Strategy on Supply Chain Integration in order to strengthen the capacity of CMST to better mitigate

emergency stock out scenarios in the future. As a part of this, there should be knowledge and skills

transfer between the parallel supply chains and CMST.

The invoicing and account reconciliation functions of CMST needs to strengthened in order for

ensure its capacity to capture revenue and in turn sustainably fund its operations.

Under Malawi’s “push” system, a significant amount of product redistribution occurs which

introduces a hidden cost and burden to the national supply chain. In its current implementation, it’s

a fairly ad-hoc and undocumented process, and so needs to be better standardized and streamlined.

As the District Health Office (DHO) plays a critical role in monitoring inventories and the

management oversight of the health center level, more targeted supply management improvements

and capacity building need to be targeted at, and led from, the DHO level.

As an large private sector health service provider, the inventory management capacity and

performance of CHAM facilities needs to also be strengthened and better integrated into the MOH

system.

43

Annexes

Annex 1: Evaluation Terms of Reference

49

Annex 2: Evaluation Team

The CHAI evaluation team was made up of a cross-disciplinary professionals from medical,

pharmacy, supply chain, health research and management consulting backgrounds. The team is

made up of CHAI Malawi staff who have strong Malawi health context experience in health systems

strengthening, health research and supply chain management, as well as exceptional relationships

with local Ministry of Health and implementing partner organizations.

The following team structure was implemented:

Andrew MvulaProject Coordinator

Dr. Bridget MaleweziMedical Advisor

Emily Churchman Kobayashi

MNCH Advisor

Sean Donato Project Lead

Newton ChagomaEvaluation Analyst

Serena ParcellEvaluation Analyst

Chris ConnellyEvaluation Analyst

Diallo AfadhaliPharmacist Advisor

Maeve MagnerCHAI Global Supply

Chain Advisor

Survey and Data Collection Team Evaluation Technical Review Team

Clement BandaM&E Advisor

Godwin NyirendaEvaluation Analyst

Experience and Qualifications

Sean D

onato

Res

earc

h T

eam

Lea

der

Dr.

Brid

get M

ale

wezi

Me

dic

al A

dvis

or

Em

ily C

hurc

hm

an K

obayashi

MN

CH

Advis

or

Adnre

w M

vula

Stu

dy C

oord

inato

Chris C

onnolly

E

valu

atio

n A

naly

st

New

ton C

hagom

a

Evalu

atio

n A

naly

st

Godiw

n N

yirenda

Evalu

atio

n A

b

Dia

llo A

fadhall

Pharm

acis

t A

dvis

or

Sere

na P

arc

ell

Evalu

atio

n A

naly

st

Cle

me

nt B

anda

M&

E O

ffic

er

Ma

eve M

agner

Supply

Chain

Advis

or

Health Evaluation and Research

Medicine

Clinical Pharmacy

Public Health

Supply Chain and Logistics

Supply Chain Impact Evaluation

Procurement

Health Systems Strengthening

Maternal and Child Health

Mixed Method Research

Statistical Analysis

Financial Analysis

Process Improvement

Malawi Health Context

50

Annex 3: Evaluation Questions

I. Operational Evaluation Questions

1. Project Management

A. Project Coordination and Planning

How did the partners work together to implement the project and meet stated PHC goals?

How effective were the PHC coordination meetings?

B. Supply Chain Efficiency and Effectiveness

How did you store the kits and what were the storage conditions?

How were the kit contents managed with the other stock? Did it change your inventory practices or recordkeeping?

Number of kits procured and delivered to each of the Center?

What was the cost of the kit medicines, relative to the total kit costs, vs the total cost of the program?

What was the performance against agreed-upon delivery schedules?

What was the product leakage rate?

What were the over and understock totals and rates?[Risk of loss due to oversupply]

What were the product delivery lead-times?

C. Product Selection

Did the contents meet disease priorities? Were there any disease priorities which were not covered by the kit program?[High level]

What was the suitability of the drugs in the kits? [Kit content appropriateness]

What were the contents of the kits for Phase I vs. Phase II?

D. Project Communication

Was the PHC Essential Medicines Project Communication strategy effective?

2. Product Availability

Availability of Quality Medicines and Health Supplies

Did any of the kit contents arrived expired or expire during its storage?

Did the kits arrive late?

Did the PMPB list of PSM items overlap with the kit items? Did any of the products show up on the PMPB report? Were there any product recalls?

To what extent were kit contents available to patients?

Was the product availability adequate for the patient population? [Quantity question]

Was the product of a good quality when delivered?

What did you do when a kit product stocked out?

What was the amount delivered?

Where there stock outs of essential medicines in your facility?

II. Health Impact Evaluation Questions

3. Health Service Outputs

A. Disease Management

How many women have benefited from prenatal iron supplementation that reduces the incidence of iron deficiency anemia?

To what extent has the Project eased the management of Diarrhea, ARI, Postpartum care, syphilis, de-worming, fever, severe and complicated malaria and reduced malaria prevalence in pregnant women?

B. Health Service Utilization

What impact did the kits have on patient visits to Center?

51

What impact did the PHC program have on health service utilization?

C. Commodity Management Capacity

The recapitalization of CMST was a stated objective of the PHC project, to what extent was this achieved?

What was the impact of the PHC program on health facility worker inventory management skills?

What was the impact of the PHC program on LMIS reporting at Center?

What was the impact of the PHC project on the CMST reform process?

4. Public Health Impact

Impact on mortality and morbidity among children under 5 and pregnant women

Did the products provided reinforce or help to maintain current declining disease trends?

How has the Project contributed towards keeping mortality and morbidity among children under 5 years and pregnant women on a constant declining rate to meet the 2015 MDG 4 (Under Five Mortality rate of 78 / 1000 LB)?

Is there a correlation between product availability and the uptake of services for pregnant women?

Is there a correlation between product availability and the uptake of under-5 health services?

52

Annex 4: Phase I Kit Contents

Phase I Kit: Distributed January to April/May 2012 (37 Items in Total)

# Item description Qty per kit # Item description Qty per kit

1 Amoxicillin 250mg tabs/PAC-1000 3 21 Water for inj 10ml amp/BOX-50 1

2 Amoxici.pdr/oral sus 125mg/5ml/BOT-100ml 15 22 Zinc 20mg tablets/PAC-100 2

3 Benzylpenicillin pdr/inj 3g vial/BOX-50 1 23 Albendazole 400mg chewable tabs/PAC-100 2

4 Ciprofloxacin 500mg tabs/PAC-10 15 24 Fe(as fum.)+folic 60+0.4mg tab/PAC-1000 2

5 Diazepam inj 5mg/ml 2ml amp/BOX-10 pt 1 25 Miconazole nitrate cream 2%/TBE-30g 10

6 Doxycycline 100mg tabs/PAC-1000 3 26 Bandage,gauze,8cmx4m,roll 10

7 Epinephrine inj 1mg/ml 1ml amp/BOX-10 1 27 Cotton wool,500g,roll,non-ster 5

8 Erythromycin 250mg tabs/PAC-100 14 28 Compress,gauze,10x10cm,n/ster/PAC-100 3

9 Glucose hyperton.inj 50% 50mL vl/BOX-20 1 29 Tape,adhesive,Z.O.,2.5cmx5m 5

10 Hydrochlorothiazide 25mg tabs/PAC-100 10 30 Syringe,disp,2ml,ster/BOX-100 1

11 Lidocain inj 2% 50ml vial/BOX-5 2 31 Syringe,disp,5ml,ster/BOX-100 1

12 Metronidazole 250mg tabs/PAC-1000 2 32 Needle,disp,19G,ster/BOX-100 2

13 New formula Oral reh.salts,1L s/BOX-100 1 33 Needle,disp,23G,ster/BOX-100 1

14 Paracetamol 500mg tabs/PAC-100 40 34 Sut,abs,DEC2,need 3/8 26mm,tri/BOX-36 1

15 Paracetamol 120mg/5ml or.sol/BOT-60ml 25 35 Sut,nonabs,DEC3,need 3/8 30mm,tri/BOX-36 1

16 Povidone iodine sol 10%/BOT-500ml 4 36 Needle,disp,21G,ster/BOX-100 1

17 Salbutamol 4mg tabs/PAC-1000 1 37 Gloves,exam,latex,medium,disp/BOX-100 5

18 Sulf.400mg+Trimet.80mg tabs/PAC-500 20

19 Sulf.100mg+Trimet.20mg disp.tab/PAC-100 50

20 Tetracycline eye ointment 1%/TBE-5g 10

53

Annex 5: Phase II Kit Contents

Phase II Kit: Distributed January to June 2012 – September /October 2013 (61 Items in Total)

S0305020 1 Apron,protection,plastic,disp/PAC-100 2 BOX

S0512111 2 Bandage,gauze,8cmx4m,roll 10 EA

S1559005 3 Benz.benzylpeni.pdr/inj 1.44g vl/BOX-50 1 BOX

S1520011 4 Benzylpenicillin pdr/inj 3g vial/BOX-50 1 BOX

S0709210 5 Cannula,IV short,18G,ster,disp 25 EA



S0323302 8 Catheter,Foley,CH14,ster,disp 2 EA

S0323303 9 Catheter,Foley,CH18,ster,disp 2 EA

S0523050 10 Compress,paraffin,10x10cm,ster/BOX-10 4 EA

S1501000 11 Epinephrine inj 1mg/ml 1ml amp/BOX-10 1 BOX

S1550025 12 Fe(as fum.)+folic 60+0.4mg tab/PAC-1000 14 PAC

S1551960 13 Gentamicin inj 40mg/ml 2ml amp/BOX-50 1 BOX

S0330030 14 Gloves,exam,latex,pwdfree,large/BOX-100 5 EA

S0330025 15 Gloves,exam,latex,pwdfree,medium/BOX-100 10 EA

S1555205 16 Lidocaine inj 1% 50ml vial/BOX-5 1 BOX

S0305030 17 Mask,surgical,disp/BOX-100 2 BOX

S1560600 18 Quinine inj 300mg/ml 2ml inject/BOX-10 8 BOX

S1560610 19 Quinine sulfate 300mg tabs/PAC-100 1 PAC

S0782300 20 Syringe,RUP,5ml,w/fixed ndl/BOX-100 2 BOX

S0504000 21 Tape,adhesive,Z.O,perforated,10cmx5m 3 EA

S1543805 22 Water for inj 10ml amp/BOX-50 4 BOX

S1580020 23 Zinc 20mg tablets/PAC-100 10 PAC

S1555370 24 Albendazole 400mg chewable tabs/PAC-100 10 PAC

S1515020 25 Benz.acid 6%+salic.acid 3% oint./TBE-40g 13 TBE

S8914100 26 Carton f/dividers 249 x 140 x 163 mm 1 Ea

S1555945 27 Clotrimazole 500mg vag.tabs/applicator 20 EA

S0523005 28 Compress,gauze,10x10cm,n/ster/PAC-100 10 EA

S1552300 29 Hydrochlorothiazide 25mg tabs/PAC-100 20 PAC

S1553000 30 Ibuprofen 200mg tabs/PAC-100 10 PAC

S1504200 31 Magn.trisilicate compound tabs/PAC-1000 6 PAC

S1555930 32 Nystatin oral sus 100,000IU/ml/BOT-30ml 25 BOT

S1555990 33 Paracetamol 125mg/5ml or.sol/BOT-60ml 100 BOT

S1555965 34 Paracetamol 500mg tabs/PAC-1000 16 PAC

S1559200 35 Promethazine 25mg tabs/PAC-100 18 PAC

S1562015 36 Salbutamol 4mg tabs/PAC-1000 4 PAC

# Material Desription Qty Unit

Box: 2/7 (13)UNICEF

Material

code

UNICEF

Material

code

Material Desription Qty Unit#

Box: 1/7 (23)

54

S1519500 37 Benzyl benzoate 25% lotion /BOT-1000ml 1 BOT

S0519600 38 Cotton wool,500g,roll,non-ster 10 EA

S1544400 39 Doxycycline 100mg tabs/PAC-1000 7 PAC

S0327520 40 Gloves,surg,pwdfree,7.5,ster,s.u.,pair 100 EA

S1555650 41 Metronidazole 250mg tabs/PAC-1000 7 PAC

S1537100 42 Sulf.400mg+Trimet.80mg tabs/PAC-500 36 PAC

S0782303 43 Syringe,RUP,2ml,w/fixed ndl/BOX-100 1 BOX

S1505060 44 Amoxicillin 250mg tabs/PAC-1000 10 PAC

S1523000 45 Calamine lotion/BOT-500ml 2 BOT

S1546315 46 Erythromycin 250mg tabs/PAC-100 60 PAC

S1552115 47 Glucose hyperton.inj 50% 50mL vl/BOX-20 1 BOX

S1561130 48 Oral reh.salts,orange Flavour1Ls/CAR-100 3 CAR

S1568045 49 Sulfadox+Pyrimeth 500+25mg tabs/PAC-1000 2 PAC

S0564050 50 Sut,abs,DEC4,need 1/2 30mm,round/BOX-36 1 BOX

S0565011 51 Sut,nonabs,DEC3,need 3/8 30mm,tri/BOX-36 1 BOX

S0782302 52 Syringe,RUPw/SIP,10ml,w/fixed nd/BOX-100 1 BOX

S1510000 53 Tetracycline eye ointment 1%/TBE-5g 150 TBE

S1505046 54 Amoxici.pdr/oral sus 125mg/5ml/BOT-100ml 120 BOT

S1564326 55 Sodium chl.inj 0.9% 500ml w/giv.s/BOX-20 1 BOX

S1552106 56 Glucose inj 5% 500ml w/giv.set/BOX-20 1 BOX

Box: 4/7 (10)UNICEF

Material

code

# Material Desription Qty Unit

Material Desription Qty Unit

#

Box: 3/7 (7)

Box: 7/7 (1)

UNICEF

Material

codeMaterial Desription Qty Unit

UNICEF

Material

Qty Unit

Box: 6/7 (1)UNICEF

Material

code

Material Desription Qty Unit#

UNICEF

Material

codeMaterial Desription#

#

Box: 5/7 (1)

55

Material number # Material Description

S1545300 57 Oxytocin inj. 10 IU, 1ml amp/BOX-10

S0566010 58 Clamp,umbilical cord,ster,s.u.

S1543625 59 Diazepam inj 5mg/ml 2ml amp/BOX-10 pt

S6780202 60 Phenobarb.200mg/ml inj 1ml amp/BOX-10 pt

S1559312 61 Phenobarbital 30mg tabs/PAC-100 pt

Loose Items - Outside Kits (5)

56

Annex 6: Comparison of Phase I and Phase II Items

Comparison Question Answer

What was the difference between contents of phase I and phase II kits?

Phase II Kits are richer in items and tackle a broader range of diseases

What was the added advantage of the contents of kit II?

It addresses maternal, child health, surgery, severe malaria, mental health and includes injectable antibiotics and more ways of administration

What were the disadvantages of the items in the phase II kits?

No anti-diabetic, no anti-ulcer, no antidote, no anti-emetic, only one type of anesthetic which is again local

Did the kit contents include items appropriate for Primary Health Care?

Yes, but we still need to assess at what level (from field information).

Kit I Kit II

1 type of non-steroidal anti-inflammatory drug,

1 type of anesthetic which is again only local

No mental health, at the exception of Diazepam which can have multiple uses,

No anti-diabetic,

No anti-ulcer,

No anti-emetic drugs,

No anti-allergic,

No antacid,

No antidote,

No transfusion kit,

No normal saline

No anti-diabetic,

Only one type of anesthetic which is again local,

No anti-ulcer,

No antidote,

No anti-emetic.

Phase I Items Not Included in Phase II Kits

Ciprofloxacin 500mg tabs/PAC-10

Miconazole nitrate cream 2%/TBE-30g

Needle,disp,19G,ster/BOX-100



Povidone iodine sol 10%/BOT-500ml

Sulf.100mg+Trimet.20mg disp.tab/PAC-100

Sut,abs,DEC2,need 3/8 26mm,tri/BOX-36

Tape,adhesive,Z.O.,2.5cmx5m

57

Phase II Kit Items Not Included in Phase I Kits

Apron plastic 100s

Benzathine Penicillin 1.44g 50s

Benzoic/salicylic acid oint 40g

Benzyl Benzoate lotion 1000ml

Calamine lot 500ml

Cannula iv 18g

Cannula iv 22g

Cannula iv 24g

Catheter foley CH14

Catheter foley CH18

Clotrimazole vag pess box

Compress paraffin 10x10cm

Gentamycin inj 50s

Gloves exam latex L 100s

Gloves surgical 7.5 pair

Glucose 5% 500ml iv 20s


Magnesium Sulphate inj

Magnesium trisilicate tabs 1000s

Mask surgical 100s

Nystatin oral susp

Oxytocin Inj

Phenobarbitone inj

Phenobarbitone tabs



Quinine inj 10s

Sodium chloride 500ml iv 20s


Suture abs DEC4 36s

Syringe 10ml w needle 100s

Umbilical cord clamps

Zinc Oxide tape 10x5m

58

Annex 7: List of Interviewed Partners and Sites Visited

MOH, Donors and Implementing Partners (15)

Organization Interviewee Title Interview Date

CHAM Lumbani Makwakwa Pharmacy Manager 28-Feb-14

CMST Saidi Sangala Financial Accountant 7-Apr-14

DFID Susan Clapham Health and HIV Adviser 26-Mar-14

JSI Phillip Kamutenga Country Director 20-Feb

KfW Lamulo Nsanja Country Economist 4-Apr-14

MOH-HTSS Brian Chimwenje Senior Pharmacist 14-Apr-14

MOH-HTSS David Bagonza Supply Chain Advisor 14-Apr-14

Norwegian Embassy Hildegunn Tobiassen First Secretary 6-Mar-14

PMPB Godfrey Kadewele Chief Executive Officer 4-Apr-14

UNICEF Dennis Chimenya Communication for Development Officer 27-Mar-14

UNICEF John Blasco Supply and Logistics Specialist 27-Mar-14

UNICEF Nyson Chizani Monitoring and Evaluation Specialist 3-Mar-14

UNICEF Sam Chriwa PHC Specialist 3-Mar-14

USAID Ugbede-Ojo Abu Supply Chain Advisor 26-Mar-14

MOH Central Hospitals (4)

Region Facility Name District

Interview Date Interviewee Title

Central Kamuzu Lilongwe 4/8/2014 Dr. Munthali Medical Doctor

4/7/2014 Tadala Hamisi Pharmacist in-Charge

North

Mzuzu Mzimba North 3/14/2014 Brave Nyirenda Radiology Technologist/Drug committee member

Cosum Moyo Pharmacy Technician

South Queens Blantyre 3/21/2014

Dr. Dube Paedtrician

Sylvester Gausi Pharmacy Technician

Zomba Zomba 3/27/2014 Mr. Chaheka Principle Clinician

District Hospitals (10)

Region Facility Name District Interview Date Interviewee Title

South Chikwawa Chikwawa 17-Mar-14 Charles Kwenje Clinical Officer

Labson Tembo Pharmacy Technician

North Chitipa Chitipa 11-Mar-14 Given Chilimiro Clinical Officer

59

Jeremiah Chiumia Pharmacy Technician

Central Dowa Dowa 21-Mar-14 George Mhasa Clinical Technician

Mercy Matapila Pharmacy Technician

North Karonga Karonga 12-Mar-14 Alberto Balasiya Clinical Officer

Voster Msutu Pharmacy Technician

Central Kasungu Kasungu 20-Mar-14 Sosten Lankhulani District Medical Officer

Kondwani Kautsa Pharmacy Technician

South Machinga Machinga 27-Mar-14 Charles Mtibo District Medical Officer

Kingston Kapida Pharmacy Technician

South Mangochi Mangochi 26-Mar-14 Dr. william Peno District Medical Officer

Andy Kishyombe Pharmacy Technician

South Mwanza Mwanza 18-Mar-14 Dr. Godwin Ulaya District Medical Officer

Mcjones Nyirongo Pharmacy Technician

Central Ntchisi Ntchisi 20-Mar-14 Robert Kombe Pharmacy Technician

South Thyolo Thyolo 24-Mar-14 Botha Matron

Nelson Nachinga Pharmacy Technician

MOH Health Centres (30)



Central Area 18 Lilongwe 31-Mar-14 Grace Chikhwaya Nurse/Midwife(Drugstore Clerk)

South Beleu Chikwawa 17-Mar-14 Hannah Ng'oma Nurse/Midwife

Edward Awali Drugstore clerk

Central Bwanje Ntcheu 12-Mar-14 Stephen Mpinganjira Medical Assistant

Dennis Chikakuda Drugstore clerk

Central Chadza Lilongwe 13-Mar-14 Chifundo Nkhoma Medical Assistant

Charles Chikawale Drugstore clerk

North Chikwina Nkhata-Bay

17-Mar-14 Aidan Chinsula Senior Medical Assistant

17-Mar-14 Aidana Chinselu Senior Medical Assistant

South Chimembe Blantyre 19-Mar-14 Paul Salijeni Nurse /Midwife Technician/ Drugstore Clerk

South Chiumbangame

Mangochi 24-Mar-14 Hastings Chunga Medical Assistant

South Chonde Mulanje 26-Mar-14 Mr. Munthali Medical Assistant

Juma Mizati Pharmacy Technician

Central Diamphwe Lilongwe 14-Mar-14 Clemecia Nyoni Nurse/Midwife

Laston Owen Drugstore clerk

South Domasi Zomba 28-Mar-14 Leonard Magombo Medical Assistant

North Kachere Nkhata-Bay

18-Mar-14

Roosevelt Njerenje/Janet Nkusang'ombe

Clinical Officer/Nurse

Idah Manda Drugstore clerk

South Kakoma Chikwawa 17-Mar-14 Williams Duba Nurse/Midwife

60

Central Kawale Lilongwe 3-Apr-14 Paliwin Malikebu

Senior Medical Assistant

4-Apr-14 Tamanda Kazaku nurse/Midwife

Central Lobi Dedza 11-Mar-14 Justin Mwalabu Medical Assistant

Ella Nambote Drugstore clerk

South Lundu Blantyre 19-Mar-14 Micheal Matewere Medical Assistant

South Makhanga Nsanje 20-Mar-14 Lucius Gama Medical Assistant

North Malidade Mzimba North

13-Mar-14

MacFarland Phiri/Ken Chirwa

Nurse/Clerk

Ken Chirwa Drugstore clerk

Central Maluwa Lilongwe 10-Mar-14 Bartholomeu Andrea Medical Assistant

Doreen Mphika Drugstore clerk

North Mkoma Mzimba South

19-Mar-14 Precious Chibweya Senior HAS

Micheal Chirwa Medical Assistant

North Mpamba Nkhata-Bay

17-Mar-14 Chawanangwa Luwe Medical Assistant

Central Nalunga Lilongwe 1-Apr-14

Joseph Dickson Nthanda

Medical Assistant

Zuze Ndalamira Drugstore clerk

South Namwera Mangochi 24-Mar-14

Coby Maunda Medical Assistant

Steria Shauli Drugstore clerk

South Nankumba Mangochi 25-Mar-14 Clement Masoambeta Medical Assistant

Ingris Maunde Drugstore clerk

South Naphimba Mulanje 26-Mar-14 Francis Sibande Medical Assistant

South Ndamera Nsanje 28-Mar-14 Macford Chikopa Medical Assistant

Asautsaine Nkhumani Drugstore clerk

Central Ngoni Lilongwe 31-Mar-14 D.R.C Gondwe Medical Assistant

Dyson Njale Drugstore clerk

South Ngwelelo Zomba 27-Mar-14 Golden kachimanga Medical Assistant

South Nkalo Chiradzulu 24-Mar-14 Mr. Khulumura Medical Assitant

James Kaphwiti Drustore Clerk

South Ntaja Machinga 26-Mar-14 Akuzike Nkali Medical Assistant

Aaron Mchere Drugstore clerk

Central Tembwe Mchinji 2-Apr-14 Odala Mbewe Medical Assistant

Chrispin Kachipande Drugstore clerk

CHAM Mission / Rural Hospitals (6)



North Ekwendeni Mzimba North 13-Mar-14 Albert Mirongo Principle Clinician

Agness Hara Nurse

South Malamulo Thyolo 25-Mar-14 Dr. Shank Medical Doctor

Harry Muyaya Stores Supervisor

South Mlambe Blantyre 20-Mar-14 Mary Chikapa Matron

John Sailes Drugstore Assistant

South Mposa Machinga 26-Mar-14 Precious Chambakata Drugstore clerk

South Mulanje Mulanje 26-Mar-14 Dickson Chisale Pharmacy Technician

Dr. Shakespeare Medical Supplies

61

Central Ntonda [Rural Hospital]

Ntcheu 13-Mar-14

Stanley Namajagali & Elizabeth Gondwe

Medical assistant & Administrator

Elizabeth Bulaimu Pharmacy Attendant

CHAM Health Centres (10)

Region Facility Name District Interview

Date Interviewee Title

North Chambo Chitipa 11-Mar-14 Siesta Ngwira Clinical Officer

Central Gowa Ntcheu 13-Mar-14

Marlyne Ziwoya

Nurse/Midwife

Grace Zinkanda

Drugstore clerk

South H Parker Zomba 27-Mar-14 Stanley Mweiwa

Medical Assistant/ Drug store Clerk

Central Kasina Dedza 11-Mar-14

Sister Victoria Ogu

Nurse in-charge

Gyliselia Hola Pharmacy Attendant

North Lusubilo Rehabilitation Center

Karonga 12-Mar-14 Christina Kasilika

Nurse

Central M'bwatalika Lilongwe 2-Apr-14 Chimwemwe Ndoliro

Nurse/Midwife In-charge/Drug store Manager

South Mposa Machinga 26-Mar-14 Frank Nyozani Nurse

South Nsambe Neno 18-Mar-14

Mr. Nkhoma Clinical Officer

Stanley Nkhoma

Pharmacy Clerk

South PIM Chiradzulu 24-Mar-14 Damson Erick Nurse/Midwife

Chance Uladi Nurse/Midwife

Central St Mary's Rehabilitation

Dowa 1-Apr-14 Sister Jolly

Nurse/Midwife

In-charge/Drug store Manager

62

Annex 8: Reported List of “Most” and “Least” Important PHC kit, Over

and Undersupplied Items by Facility Level

Below is a list of PHC kit items that were considered by facilities as the “most” important for their facility,

based on service level:

Health Center

1) Paracetamol tabs, 500mg 1000s 2) Amoxycillin tabs 1000 3) Cotrimoxazole 480mg tabs 500s 4) Metronidazole tabs 1000s 5) Quinine inj 10s 6) Quinine 300mg tabs 100 7) Gloves exam latex M 100s

8) Benzylpenicillin 5miu 50s 9) Doxycycline tabs 1000 10) Gentamycin inj 50s 11) Amoxycillin susp bot 12) Ibuprofen 200mg tabs 100s 13) Erythromycin tabs 100s

DHO / District Hospital

1. Quinine inj 10s 2. Benzylpenicillin 5miu 50s 3. Gentamycin inj 50s 4. Gloves surgical 7.5 pair

5. Cotrimoxazole 480mg tabs 500s 6. Oxytocin Inj 7. Gloves exam latex M 100s 8. Paracetamol tabs, 500mg 1000s

Central Hospitals

1) Gloves exam latex M 100s 2) Benzathine Penicillin 1.44g 50s 3) Cannula iv 18g 4) Gloves exam latex L 100s 5) Mask surgical 100s

6) Metronidazole tabs 1000s 7) Suture abs DEC4 36s 8) Syringe 10ml w needle 100s 9) Quinine inj 10s

Mission Hospital

1) Paracetamol tabs, 500mg 1000s 2) Gentamycin inj 50s 3) Quinine 300mg tabs 100 4) Amoxycillin tabs 1000

5) Benzathine Penicillin 1.44g 50s 6) Benzylpenicillin 5miu 50s 7) Gloves exam latex M 100s 8) Gloves surgical 7.5 pair

Rural / Community Hospitals

1) Benzylpenicillin 5miu 50s 2) Doxycycline tabs 1000 3) Gentamycin inj 50s

4) Amoxycillin susp bot 5) Ibuprofen 200mg tabs 100s 6) Erythromycin tabs 100s

Below is a list of PHC kit items that were considered by facilities as the “least” important for their facility,

based on service level:

Health Center

1) Benzyl Benzoate lotion 1000ml 2) Epinephrine (or Adrenaline) inj 10s 3) Magnesium Sulphate inj 4) Mask surgical 100s 5) Catheter foley CH18 6) Glucose 50% 20s 7) Promethazine tabs 100s

8) Benzoic/salicylic acid ont 40g 9) Catheter foley CH14 10) Zinc tabs 100s 11) Doxycycline tabs 1000 12) Apron plastic 100s 13) Clotrimazole vag pess box 14) Sulphadoxine/Pyrimeth 1000s 15) Water for Inj 50s

63

DHO / District Hospital

1) Benzyl Benzoate lotion 1000ml 2) Promethazine tabs 100s 3) Sulphadoxine/Pyrimeth 1000s 4) Albendazole 400mg tabs 100s 5) Benzoic/salicylic acid ont 40g

6) Epinephrine (or Adrenaline) inj 10s 7) Sut non abs DEC3 36s 8) Suture abs DEC4 36s 9) Water for Inj 50s

Central Hospitals

1) Albendazole 400mg tabs 100s 2) Benzyl Benzoate lotion 1000ml 3) Doxycycline tabs 1000 4) ORS 100s

5) Sulphadoxine/Pyrimeth 1000s 6) Zinc Oxide tape 10x5m 7) Zinc tabs 100s

Mission Hospital

1) Promethazine tabs 100s 2) Benzoic/salicylic acid ont 40g 3) Catheter foley CH14

4) Magnesium trisilicate tabs 1000s 5) Sut non abs DEC3 36s 6) Tetracycline eye oint 5g

Rural / Community Hospitals

1) Compress paraffin 10x10cm 2) Epinephrine (or Adrenaline) inj 10s

3) Gentamycin inj 50s 4) Magnesium Sulphate inj

By health facility level, the following items were reported by the sites as the most frequently stocked out

items provided in the PHC kits:

Health Centers – Most Frequently Stocked Out

1) Ibuprofen 200mg tabs 100s 2) Paracetamol tabs, 500mg 1000s 3) Amoxycillin tabs 1000 4) Quinine 300mg tabs 100 5) Calamine lot 500ml 6) Bandage gauze 8x4m

7) Cotton wool 500g 8) Paracetamol syrup bot 9) Gentamycin inj 50s 10) Metronidazole tabs 1000s 11) Erythromycin tabs 100s

DHO / District Hospital – Most Frequently Stocked Out

1) Bandage gauze 8x4m 2) Benzylpenicillin 5miu 50s 3) Ibuprofen 200mg tabs 100s

4) Gentamycin inj 50s 5) Zinc Oxide tape 10x5m

Central Hospitals30 – Most Frequently Stocked Out


Cannula iv 24g

Cotton wool 500g

Epinephrine (or Adrenaline) inj 10s

Gentamycin inj 50s


Mask surgical 100s

Oxytocin Inj

Mission Hospitals – Most Frequently Stocked Out

1) Paracetamol tabs, 500mg 1000s 2) Cannula iv 24g 3) Gloves exam latex M 100s

4) Glucose 50% 20s 5) Quinine inj 10s

Rural / Community Hospitals – Most Frequently Stocked Out


30

These items are not ranked by frequency of stock outs as these items received the same scoring by facilities interviewed

64

By health facility level, the following items were reported by the sites as the most frequently undersupplied


Health Center – Most Undersupplied

1) Ibuprofen 200mg tabs 100s 2) Quinine 300mg tabs 100 3) Paracetamol tabs, 500mg 1000s 4) Amoxycillin tabs 1000 5) Bandage gauze 8x4m 6) Calamine lot 500ml

7) Syringe 2ml w needle 100s 8) Benzylpenicillin 5miu 50s 9) Fe+Fol tabs 1000s 10) Metronidazole tabs 1000s 11) Nystatin oral susp 12) Amoxycillin susp bot

DHO / District Hospital – Most Undersupplied

1) Benzylpenicillin 5miu 50s 2) Gentamycin inj 50s 3) Amoxycillin tabs 1000 4) Cannula iv 24g 5) Gloves surgical 7.5 pair 6) Metronidazole tabs 1000s 7) Benzathine Penicillin 1.44g 50s

8) Compress paraffin 10x10cm 9) Cotton wool 500g 10) Quinine 300mg tabs 100 11) Syringe 10ml w needle 100s 12) Paracetamol tabs, 500mg 1000s 13) Sodium chloride 500ml iv 20s 14) Zinc Oxide tape 10x5m

Central Hospitals – Most Undersupplied

1) Gloves exam latex M 100s 2) Cannula iv 24g 3) Calamine lot 500ml

4) Epinephrine (or Adrenaline) inj 10s 5) Gentamycin inj 50s 6) Gloves exam latex L 100s

Mission Hospital – Most Undersupplied

1) Benzathine Penicillin 1.44g 50s 2) Benzylpenicillin 5miu 50s 3) Cannula iv 24g 4) Gentamycin inj 50s

5) Gloves exam latex M 100s 6) Ibuprofen 200mg tabs 100s 7) Paracetamol syrup bot 8) Paracetamol tabs, 500mg 1000s

Rural / Community Hospitals – Most Undersupplied

1) Ibuprofen 200mg tabs 100s 2) Quinine inj 10s

By health facility level, the following items were reported by the sites as the most frequently oversupplied


Health Center – Most Oversupplied

1) Albendazole 400mg tabs 100s 2) Doxycycline tabs 1000 3) Promethazine tabs 100s 4) Glucose 50% 20s 5) Mask surgical 100s 6) Sulphadoxine/Pyrimeth 1000s 7) Water for Inj 50s 8) Fe+Fol tabs 1000s

9) Hydrochlorothiazide 25mg tabs 100s 10) Erythromycin tabs 100s 11) Apron plastic 100s 12) Epinephrine (or Adrenaline) inj 10s 13) Magnesium trisilicate tabs 1000s 14) Tetracycline eye oint 5g 15) Cotrimoxazole 480mg tabs 500s

DHO / District Hospital – Most Oversupplied

1) Promethazine tabs 100s 2) Hydrochlorothiazide 25mg tabs 100s

7) Clotrimazole vag pess box

8) Epinephrine (or Adrenaline) inj 10s

65

3) Cotrimoxazole 480mg tabs 500s 4) Doxycycline tabs 1000 5) Sulphadoxine/Pyrimeth 1000s 6) Albendazole 400mg tabs 100s

9) Glucose 50% 20s 10) Magnesium trisilicate tabs 1000s 11) Sut non abs DEC3 36s 12) Suture abs DEC4 36s

Central Hospitals – Most Oversupplied

1) Albendazole 400mg tabs 100s 2) Sulphadoxine/Pyrimeth 1000s 3) Benzathine Penicillin 1.44g 50s

4) Fe+Fol tabs 1000s 5) Hydrochlorothiazide 25mg tabs 100s 6) Zinc tabs 100s

Mission Hospital – Most Oversupplied

1) Cotrimoxazole 480mg tabs 500s 2) Promethazine tabs 100s

3) Tetracycline eye oint 5g

Rural / Community Hospitals – Most Oversupplied

1) Albendazole 400mg tabs 100s 2) Cotton wool 500g 3) Cotrimoxazole 480mg tabs 500s

66

Annex 9: Analysis of Most Important, Least Important, Stocked Out,

Under and Oversupplied Kit Item Lists

The table organizes the various item lists reported by the facilities. The table below includes the most

referenced items during the survey interviews. The comments column attempts to provide and explanation

for the respective categorizations of the products discussed.

# Item M

ost

Im

po

rtan

t

Un

der

sup

pli

ed

Lea

st im

po

rtan

t

Ov

ersu

pp

lied

Fre

qu

entl

y S

tock

ed

Ou

t

Comments

1 Albendazole 400mg tabs 100s

Multiple deworming interventions likely resulted in an overstock of supplies

2 Amoxycillin tabs 1000

As this is a multiple use item, it’s a fast moving and widely used across all health service levels

3 Bandage gauze 8x4m The was most likely provided in low quantities

4 Benzoic/salicylic acid not 40g

This item is used for uncommon skill conditions (rashes,etc.). The expertise to treat the skin conditions requiring this product is often minimal

5 Benzyl Benzoate lotion 1000ml

This item is used mostly to treat scabies in Malawi, which is a rare condition. The expertise to treat the skin conditions requiring this product is often also minimal.

6 Benzylpenicillin 5miu 50s

As this is a multiple use item, it’s a fast moving and widely used across all health service levels

7 Calamine lot 500ml

This is an antihistamine and broadly used for various skin conditions and is a fast-moving item.

8 Cannula iv 24g

This item is used for collecting blood samples and IV transfusions and is a multiple use item. This was likely supplied in low quantities.

9 Catheter foley CH14

This item is used mostly for complicated procedures (urine retention) which the admission of patients and mostly used at a secondary or tertiary level.

10 Catheter foley CH18

This item is used mostly for complicated procedures (urine retention) which the admission of patients and mostly used at a secondary or tertiary level. This is just a higher caliber that CH14.

11 Cotrimoxazole 480mg tabs 500s

It is the drug used for treatment of most opportunistic infections in HIV and TB, and is supplied by multiple programs.

12 Cotton wool 500g

Multipurpose (cleaning wounds, maternity during delivery, etc.). Only 10 rolls provided in the kits so at likely risk of frequent stock outs.

13 Doxycycline tabs 1000

Routine use for STIs and urinary tract infections is broadly used. As with cotrimoxazole, multiple programs supply this product.

14 Epinephrine (or Adrenaline) inj 10s

Used for resuscitation and used mostly at the secondary and tertiary level. Its administration requires advanced training only found at those levels.

67

15 Fe+Fol tabs 1000s Used for the treatment of anemia caused by multiple conditions. As with cotrimoxazole, and doxycycline, this product is often supplied by multiple programs.

16 Gentamycin inj 50s As this used for treating severe bacterial infections, it’s fast moving and widely used across all health service levels

17 Gloves exam latex M

100s

Used for infction prevention, this is a multiple use item, it’s a fast moving and widely used across all health service levels

18 Glucose 50% 20s Routine IV fluid, mostly used at a secondary and tertiary level

19 Ibuprofen 200mg tabs

100s

Broad pain-reduction use and often overprescribed at all levels

20 Magnesium Sulphate inj Used for the treatment of severe high blood pressure in pregnant women with complications most often utilized at the secondary and tertiary levels

21 Mask surgical 100s The items are for surgical/theatre use which is not available at the primary level

22 Metronidazole tabs

1000s

Commonly used for treatment of multiple conditions (diarrhea, etc.) at a primary level and so is a fast-moving item.

23 Paracetamol syrup bot Used in the reduction of pain and fever caused by multiple conditions in children. This is also a fast moving item.

24 Paracetamol tabs, 500mg

1000s

Used in the reduction of pain and fever caused by multiple conditions. This is also a fast moving item.

25 Promethazine tabs 100s Used for the treatment of vomiting in adults and not widely used at the health center level.

26 Quinine 300mg tabs 100

The drug is for treatment of Malaria which is endemic in Malawi. It is meant for treatment of severe malaria, but some facilities also use it for simple malaria which is out of the treatment guidelines. The low quantity might have been put deliberately to prevent over prescription which can lead to drug resistance.

27 Quinine inj 10s

The drug is for treatment of Malaria which is endemic in Malawi. It is meant for treatment of severe malaria, but some facilities also use it for simple malaria which is out of the treatment guidelines. The is likely undersupplied because severe malaria is very common.

28 Sulphadoxine/Pyrimeth

1000s

The drug is no longer used at a first line treatment of Malaria. It is only used as a prophylaxis for Malaria in pregnancy and is provided by multiple programs like cotrimoxazole.

29 Tetracycline eye oint 5g Routine eye antibiotic, no longer commonly used.

30 Water for Inj 50s

It is used for dilution of intramuscular or intravenous drugs (injections) which is not common at a primary level. Some injectables come already packed with water. It's possible that the amount of water provided exceed the injectibles they were meant to compliment.

31 Zinc tabs 100s

While reported as "least" important, but it is a life-saving drug for children with diarrhea. Many interventions use this product, but clinicians and health workers may need more training and education on this item as it appears underutilized.

Annex 10: Tracer Item Selection Matrix

# PHC Project Item Product Description Tracked on Stock Card

Primary Care Use

Maternal and Neonatal

Commodity

Antimalarial Commodity

Multipurpose Commodities

O&M Report Tracer Item

Additional Rationale for Inclusion

1 Sulphadoxine/ Pyrimeth 1000s

Preventive treatment for malaria in pregnancy and for complicated malaria.

It represents women. Represents Box 1

2 Paracetamol, 500mg tabs

Anti-pain medication

Widely used commodity;

3 Erythromycin tabs 100s

broad-spectrum antibiotic

Represents Box 7

4 Cotrimoxazole, 480mg tabs

bacteriostatic antibiotic

480mg: Children represented

5 Syringe+needle, 2ml unit

injecting drug, for low vein calibre patients

Neonatal commodity

6 Quinine inj 10s injectable antimalarial

Only treatment for severe and complicated malaria

7 Oxytocin Inj increase uterine contraception during labor

Maternal and neonatal commodity; should be available at facilities with maternity wing. From the list of loose items

8 Amoxycillin susp bot multipurpose antibiotic, against body infection

Represents Box 4

9 Sodium chloride 500ml iv 20s

I.V infusion, increase body concentration in salt and rehydration

Represents Box 5

10 Glucose 5% 500ml iv 20s

I.V infusion, increase body concentration in sugar

Represents Box 6

Annex 11: Under and Overstocked Items and Expiries Across all health service levels, inclusive of MOH and CHAM sites, the following items were considered the

most undersupplied item in the PHC kits (alphabetized):



Bandage gauze 8x4m


Calamine lot 500ml

Fe+Fol tabs 1000s


Metronidazole tabs 1000s

Nystatin oral susp



Syringe 2ml w needle 100s

Across all health service levels, inclusive of both MOH and CHAM sites, the following items were considered

the most oversupplied item in the PHC kits (alphabetized):

Albendazole 400mg tabs 100s

Cotrimoxazole 480mg tabs 500s

Doxycycline tabs 1000

Erythromycin tabs 100s

Fe+Fol tabs 1000s

Glucose 50% 20s

Hydrochlorothiazide 25mg tabs 100s

Mask surgical 100s



Tetracycline eye oint 5g

Water for Inj 50

During the evaluation site visits, when asked which kit items expired the most at their facilities the following

products, organized by service level, were mentioned:

Health Center

• Albendazole 400mg tabs 100s

• Amoxycillin tabs 1000

• Benzylpenicillin 5miu 50s

• Epinephrine (or Adrenaline) inj 10s

DHO/District Hospital

• Benzyl Benzoate lotion 1000ml

• Epinephrine (or Adrenaline) inj 10s

• Phenobarbitone inj

• Phenobarbitone tab

Mission Hospital

• Benzathine Penicillin 1.44g 50s

• Diazepam Inj

• Promethazine tabs 100s

Central Hospital

• Zinc tabs 100

When reviewing the above lists, it’s important to consider that most of the items came up only once among

the 58 facilities as having expired. Epinephrine was the product most mentioned as having expired.

While relocation was a critical mechanism of mitigating product expiry, (as noted earlier in this report) due to

the standardization of the kits the facilities) would often have an oversupply of the same item making

relocation a challenge. Amoxycillin, Benzyl Benzoate, Glucose 5% and Erythromycin were mentioned as the

products that expired because other facilities were also oversupplied with them and so they could not be

relocated.

71

Annex 12: Evaluation Surveys

Health Impact Evaluation - Health Center Survey Clinician Survey

Facility Name: Region ☐ North ☐Central

☐ South

Sector

☐MOH ☐CHAM District:

Interviewee: Date:

Title: Data Collector:

Has the clinician worked at this facility since

January 2010? ☐ A. Yes ☐ B. No

# of years the clinician worked at this facility

Evaluation Category: 1D. Project Communication

Evaluation Question: 15. Was the PHC Essential Medicines Project Communication strategy effective?

FIRST WE ARE GOING TO SPEAK BREIFLY ABOUT THE COMMUNICATION STRATEGY FROM UNICEF. UNICEF TRIED TO REACH FACILITIES AND PATIENTS TO INFORM THEM OF THE KITS PROGRAM, SO WE’D LIKE TO KNOW IF AND HOW THAT COMMUNICATION REACHED YOU AND YOUR FACILITY 1) How did you learn about the kit program? 2) What communications did your facility receive regarding the PHC kit project? 3) In general, did you feel that PHC communications reached patients and communities?

A. ☐Yes B. ☐No C. ☐Don't Know

If so, by which method? How? 4) If so, what do you think the impact this communication had was on patient behavior?

Response / Interviewer Notes - NOTES SHOULD ALIGN WITH THE QUESTION NUMBERS LISTED ABOVE – DO NOT LEAVE THIS SPACE BLANK:

Evaluation Category: 2A. Availability of Quality Medicines and Health Supplies

Evaluation Question: 16. To what extent were kit contents available to patients?

1) In general, did the PHC kit program increase patient access to needed essential medicines?

A. ☐Strongly Agree B. ☐Agree C. ☐Disagree

72

D. ☐Strongly Disagree E. ☐Don't Know

2) Why?

QUESTION BELOW ONLY FOR HEALTH WORKERS WHO HAVE BEEN AT THE FACILITY SINCE 2010 OR EARLIER:

3) When you compare 2010 (before the kits program) to 2013 (with the kits program), do you see a change in the level of access patients had to essential medicines?

A. ☐Yes - More Access / B. ☐Yes - Less Access / C. ☐No Change /

D. ☐NA / E.☐ Don't Know

4) If you observed a change, what do you think are the causes of this difference? 5) Since the end of the PHC Kit project what changes to patient access have you observed?


Evaluation Category: 1C. Product Selection

Evaluation Question: 12. What was the suitability of the drugs in the kits? [Kit content appropriateness]

FOR THESE QUESTIONS

1) In general, regardless of quantity supplied, where the contents of kits appropriate for the needs of your facility? If everything came in the right quantities, was the content of the kits appropriate for your facility?

A. ☐Yes B. ☐No C. ☐Somewhat D. ☐Don't Know

2) Regardless of quantity supplied, which five kit items were the most important for assisting patients? 3) Regardless of quantity supplied, which five kit items were the least important for assisting patients? 4) Which products, which were not included, did you think should have been included in the kits?


73

Evaluation Category: 2A. Availability of Quality Medicines and Health Supplies

Evaluation Question: 23. Was the product of a good quality when delivered?

1) Was the product of a good quality?


2) If not, what were specific issues that occurred?

3) Have you reported any quality concerns to the DHO or PMPB? 4) Did the patients provide any feedback on the quality of the products? 5) Were patients overall happy with the quality of the products? 6) If patients had negative feedback, what were specific issues that occurred?


Evaluation Category: 3A. Disease Management

Evaluation Question: 1. To what extent has the Project eased the management of Diarrhea, ARI, Postpartum care, syphilis, de-worming, fever, severe and complicated malaria and reduced malaria prevalence in pregnant women?

Now I am going to ask you some questions about management of common conditions at your facility. For each condition I'm interested in knowing how much you are able to manage the cases, and the primary reason you may not be able to manage some or all of the cases; 1. Diarrhea

A. ☐Able to manage all cases B. ☐Able to manage most cases

C. ☐Able to manage some cases D. ☐Not able to manage any cases

2. ARI



3. Postpartum care



74

4. Syphilis



5. De-worming



6. Fever



7. Severe and complicated malaria



8. For cases that you are not able to manage (listed above) , what is the primary reason:

A. ☐Lack of medicines B. ☐Lack of diagnostics

C. ☐Lack of training D. ☐Cases requiring management at a higher level (referral)

E. ☐Other, specify:________________

9. Reduced malaria prevalence in pregnant women 1) What percentage of pregnant women attending this facility receive malaria prophylaxis (estimate is fine)?

A. ☐100% B. ☐80-99% C. ☐50-79% D. ☐20-49% E. ☐1-20% F.☐ 0%

QUESTION BELOW ONLY FOR HEALTH WORKERS WHO HAVE BEEN AT THE FACILITY SINCE 2010 OR EARLIER:

3) Have you observed any reduction in prevalence of malaria among pregnant women when you

compare before the kits and after the kits?


75


Evaluation Category: 3B. Health Service Utilization

Evaluation Question: 3. What impact did the kits have on patient visits to Center?

1) Do you feel that the kits program encouraged more patients to attend this facility?


2) Why or why not?

Only for health workers who have been at the facility since 2010 or earlier: 3) When you compare 2010 (before the kits program) to 2013 (with the kits program), do you see a

change in the number of patients attending your facility?

A. ☐Yes -Increase B. ☐Yes-Decrease C. ☐No Change D. ☐NA E. ☐Don't Know

4) If you observed a change, what do you think are the causes of this difference? Do you think this change was related to the kits?



Overall High Level Site Findings & Additional Interviewer Notes – DO NOT LEAVE THIS SPACE BLANK

Additional Questions

a) What was the best thing (strength) about the PHC Project? B) What was the worst

(weakness)?

76

Operations Evaluation - Health Center Survey

Inventory Management Survey

Evaluation Category: Availability of Quality Medicines and Health Supplies

Evaluation Question: 17. What was the amount delivered?

1) How many kits did you receive per month?

2) Did this amount change at any point? Why?

A. ☐Yes / ☐ B. No / ☐C. Don't Know

3) Was it easy to request more/less kits?

A. ☐Strongly Agree / ☐ B. Agree / ☐C. Disagree / ☐D. Strongly Disagree /

☐E. Neither Agree nor Disagree / F. ☐Don't Know

4) Why?


Facility Name: Region ☐ North ☐ Central

☐South

Sector: ☐MOH ☐CHAM District:

Interviewee: Date:

Title: Data Collector:

Has the interviewee worked at this facility since January 2010?

☐ A. Yes ☐ B. No

# of years/months the interviewee worked at this

facility

77

Evaluation Category: Supply Chain Efficiency and Effectiveness

Evaluation Question: 6. What was the performance against agreed-upon delivery schedules?

1) When did you expect the kits to arrive?

2) Do you agree with this statement?: “The kits arrived according to my delivery expectations”



3) Do you agree with this statement?: “ I was informed of expected delivery dates in advance.”



4) Do you agree with this statement?: “ I received the expected quantities [of kits] with each delivery.”



5) If they were late, in general, by how many days/weeks?

A. ☐1-2 days/ B. ☐3-4 days / C. ☐5-6 days / D. ☐7-8 days /

E. ☐More than a week / F. ☐ 10 days / G. ☐Over 2 weeks /

H. ☐Over a month / ☐I. N/A



Evaluation Question: 10. How did you store the kits and what were the storage conditions?

1) Did you have any storage challenges?


2) If so, what were they?


78


Evaluation Question: 11. How were the kit contents managed with the other stock? Did it change your inventory practices or recordkeeping?

1) How were the kit contents managed with the other stock? 2) What integration issues emerged as a result of these two essential medicines delivery systems?


Evaluation Category: Commodity Management Capacity

Evaluation Question: 5. What was the impact of the PHC program on health facility worker inventory management or recordkeeping skills?

1) Were there any training or mentorship provided specifically related to the inventory management of the kits?


2) Did the kits program impact your inventory management system?


If so, how?

3) Do you agree with this statement?: “The kit program improved the inventory management skills of health facility staff.” [Record keeping, organized stock rooms, LMIS reporting, etc.]





Evaluation Question: 21. Was the product availability adequate for the patient population? [Quantity question]

1) Did the quantities in the kit, in general, meet patient needs?


2) Which 3 items in the kit were undersupplied in terms of quantity? 3) Which 3 items in the kit were oversupplied in terms of quantity?

Response / Interviewer Notes - NOTES SHOULD ALIGN WITH THE QUESTION NUMBERS LISTED ABOVE – DO NOT LEAVE THIS SPACE BLANK: 3 Undersupplied:

79

3 Oversupplied:


Evaluation Question: 19. Where there stockouts of essential medicines in your facility?

1) In general, did stockouts of essential medicines and supplies occur?

A. ☐Frequently / B. ☐Sometimes / C. ☐Never / D. ☐Don't Know

2) 2) Which five commodities were the most often stocked out? [List below] 3) 3) Of those five, how often did they stockout?

3A) Item 1: _________________________________________


3A) Item 2: _________________________________________


3A) Item 3: _________________________________________


3A) Item 4: _________________________________________


3A) Item 5: _________________________________________


4) Why do you think these stockouts occurred? 5) Do you agree with this statement?: The kits program prevented product stockouts at your facility?



6) What impact did the kits program have on stockouts? [Definition of stockout: When a product is not available in facility inventory, regardless of duration of absence]



Evaluation Question: 20. What did you do when a kit product stocked out?

80

1) What did you do when a kit product stocked out?

Response / Interviewer Notes - NOTES SHOULD ALIGN WITH THE QUESTION NUMBERS LISTED ABOVE – DO NOT LEAVE THIS SPACE BLANK


Evaluation Question: 22. Did any of the kit contents arrived expired or expire during its storage?

1) Did any of the kit contents arrive expired?


2) If so, what did you do with them? 3) What 3 kit items expired [during storage at the facility] the most? Why? What did you do with them? What did they do with stock that was nearing expiry?


Overall High Level Site Findings & Additional Interviewer Notes – DO NOT LEAVE THIS SPACE BLANK

Additional Questions

Aside from UNICEF and CMST, did this facility receive or procure essential medicines and

supplies from other sources and supplies?

What was the process for transferring stock that was near expiry?

Stock Card Data Collection Section

Box 1 - Quinine inj 10s

M

ay

201

2

Ju

n

201

2

Ju

l

201

2

Au

g

20

12

Sep

2012

Oct

2012

No

v

2012

Dec

2012

Jan

2013

Feb

2013

Mar

20

13

Ap

r

201

3

Total Amount Received

Notes

Box 2 - Sulphadoxine/Pyrimeth 1000s

May

201

2

Ju

n

201

2

Ju

l

201

2

Au

g

20

12

Sep

2012

Oct

2012

No

v

2012

Dec

2012

Jan

2013

Feb

2013

Mar

20

13

Ap

r

201

3


Notes

Box 2 - Paracetamol, 500mg tabs

M

ay

201

2

Ju

n

201

2

Ju

l

201

2

Au

g

20

12

Sep

2012

Oct

2012

No

v

2012

Dec

2012

Jan

2013

Feb

2013

Mar

2013

Ap

r

201

3


Notes

82

Box 2 - Syringe+needle, 2ml unit

May

201

2

Ju

n

201

2

Ju

l

201

2

Au

g

2012

Sep

2012

Oct

2012

No

v

2012

Dec

2012

Jan

2013

Feb

2013

Mar

2013

Ap

r

201

3


Notes

Box 3 - Cotrimoxazole, 480mg tabs

M

ay

201

2

Ju

n

201

2

Ju

l

201

2

Au

g

20

12

Sep

2012

Oct

2012

No

v

2012

Dec

2012

Jan

2013

Feb

2013

Mar

20

13

Ap

r

201

3


Notes

Box 4 - Amoxycillin susp bot

M

ay

201

2

Ju

n

201

2

Ju

l

201

2

Au

g

20

12

Sep

2012

Oct

2012

No

v

2012

Dec

2012

Jan

2013

Feb

2013

Mar

20

13

Ap

r

201

3


Notes

83

Box 5 - Sodium chloride 500ml iv 20s bottles

May

201

2

Ju

n

201

2

Ju

l

201

2

Au

g

20

12

Sep

2012

Oct

2012

No

v

2012

Dec

2012

Jan

2013

Feb

2013

Mar

20

13

Ap

r

201

3


Notes

Box 6 - Glucose 5% 500ml iv 20s bottles

M

ay

201

2

Ju

n

201

2

Ju

l

201

2

Au

g

20

12

Sep

2012

Oct

2012

No

v

2012

Dec

2012

Jan

2013

Feb

2013

Mar

20

13

Ap

r

201

3


Notes

Box 7 - Erythromycin tabs 100s

M

ay

201

2

Ju

n

201

2

Ju

l

201

2

Au

g

20

12

Sep

2012

Oct

2012

No

v

2012

Dec

2012

Jan

2013

Feb

2013

Mar

20

13

Ap

r

201

3


Notes

84

Box 8 / Loose - Oxytocin Inj

M

ay

201

2

Ju

n

201

2

Ju

l

201

2

Au

g

20

12

Sep

2012

Oct

2012

No

v

2012

Dec

2012

Jan

2013

Feb

2013

Mar

20

13

Ap

r

201

3


Notes

Additional Notes

Annex 13: Bibliography

Bornbusch, Alan and James Bates. “Multiplicity in Public Health Supply Systems: A Learning

Agenda.” Global Health: Science and Practice 2013, Volume 1, Number 2

Kassebaum et al. “Global, regional, and national levels and causes of maternal mortality during 1990–2013: a systematic analysis for the Global Burden of Disease Study 2013” Lancet. Published Online May 2, 2014. Management Sciences for Health. “Managing Access to Medicines and Health Technologies.”

Chapter 26: Kit System Management, 2012.

Maternal Mortality Estimation Inter-Agency Group, 2014. Maternal mortality in 1990-2013,

Malawi. Downloaded on May 13, 2014.

Ministry of Health – Government of Malawi, December 2013. Health Management Information

Bulletin: Annual Report July 2012 – June 2013.

Ministry of Health – Government of Uganda, 2011. Assessment of the medicines kit-based supply

system in Uganda.

“Conducting Spot Checks in Health Facilities.” First, Second, Third, Fourth, Fifth, Sixth, Seventh &

Eight Cycle Reports. Prepared by O&M IT/S and Management Services Limited. June 2012 – January

2014

ORC Macro. Malawi Demographic and Health Survey. 2010.

UNICEF, September 2013. Committing to Child Survival: A Promise Renewed: Progress Report 2013

UNICEF, November 2012. Primary Health Care Essential Medicines Support for Malawi –

Consolidated Progress Report.

UNICEF, June 2013. Primary Health Care Essential Medicines Support for Malawi – Consolidated

Progress Report.

Malawi Primary Health Care (PHC) Project Evaluation · The PHC was a timely and effective medicine...

Documents

Transcript of Malawi Primary Health Care (PHC) Project Evaluation · The PHC was a timely and effective medicine...