AdvancingtheScienceofDelivery:InspectionsRegimesinHealthCareandtheirImpactonPatientSafetyStandardsandQualityofCareinKenya

P149879

ConceptNote/ResearchProtocolUPDATE1

NOVEMBER2015

Abstract

Thisstudyevaluatestheimpactofaccountabilitymechanisms–throughdifferentmodelsofhealthinspections–onqualityofcareandpatientsafety,quantity,andpricesofhealthservicesinKenya.Usinghealthmarketsastheunitofintervention(clustersofhealthfacilitieswherenofacilityismorethan4KMfromthecenterofthemarket),wewillexperimentallyallocateallprivateandpublichealthfacilitiesinthreeKenyancounties–Kakamega,KilifiandMeru–intooneofthreearms:(1)high-intensityinspectionswithenforcementofwarningsandsanctionsfornon-compliantfacilities;(2)high-intensityinspectionswithenforcementofwarningsandsanctionsfornon-compliantfacilitiesandpublicdisclosureofinspectionresults,and(3)“business-as-usual”low-probabilityinspections(thecontrolgroup).Theresultsfromthisstudywillcontributetounderstandingtheextenttowhichgovernanceandaccountabilitymechanismscanhelpimproveservicedeliveryinlow-incomecountries,particularlyinthiscaserelatedtoimprovingpatientsafetyandqualityofcareinthepublicandprivatesectors,andifso,howthesesystemsmayoperatewhentheyareimplemented“atscale.”Anotherimportantcontributionofthisprojectisthedevelopmentofasetoftoolsandinstrumentsthatcanbebroadlydeployedbothinsurveysofqualityandinthedesignofinspectionsystemsacrossdiverselow-incomesettings.

1The first versionof this ConceptNote/ResearchProtocolwas reviewedaccording toWBprocedures in July 2014. Previousversionswerepreparedandusedforfundingapplicationsduring2013.Thisversionreflectsthemostrecentadjustmentsbasedon project’s advances on regulatory reform, discussion with counterparts, intervention assessments, and pilots of datacollectioninstruments.

2

TableofContentsIEPROFILEINDICATORS........................................................................................................................................3

1. EXECUTIVESUMMARY.................................................................................................................................4

2. BACKGROUNDANDKEYINSTITUTIONALFEATURES.....................................................................................4

3. LITERATUREREVIEWANDKNOWLEDGEGAPS..............................................................................................5

4. POLICYRELEVANCE......................................................................................................................................7

5. THEORYOFCHANGE....................................................................................................................................7

6. HYPOTHESES/EVALUATIONQUESTIONS.......................................................................................................9

7. MAINOUTCOMESOFINTEREST..................................................................................................................10

8. EVALUATIONDESIGNANDSAMPLINGSTRATEGY.......................................................................................128.1 TREATMENTANDCONTROLGROUPS..................................................................................................................138.2 SAMPLESIZECALCULATIONS.............................................................................................................................15

9. DATACOLLECTION.....................................................................................................................................179.1 QUANTITATIVEINSTRUMENTS...........................................................................................................................179.2 MANAGEMENTOFDATAQUALITY.....................................................................................................................189.3 ETHICALISSUES..............................................................................................................................................199.4 QUALITATIVEINSTRUMENTS..............................................................................................................................209.5 IEIMPLEMENTATIONMONITORINGSYSTEM........................................................................................................20

10. DATAPROCESSINGANDANALYSIS.........................................................................................................2010.1 DATACODING,ENTRY,ANDEDITING..................................................................................................................2010.2 MODELSPECIFICATIONFORQUANTITATIVEDATAANALYSIS....................................................................................21

11. STUDYLIMITATIONSANDRISKS.............................................................................................................21

12. IEMANAGEMENT...................................................................................................................................2212.1 EVALUATIONTEAMANDMAINCOUNTERPARTS....................................................................................................2212.2 WORKPLANANDDELIVERABLES........................................................................................................................2412.3 BUDGET........................................................................................................................................................25

13. PLANFORUSINGDATAANDEVIDENCEFROMTHESTUDY.....................................................................25

14. REFERENCES...........................................................................................................................................27

3

IEPROFILEINDICATORSNo. Indicator Description1 IEcode P1498792 IETitle KenyaPatientSafetyImpactEvaluation3 IETTL GuadalupeBedoyaArguelles4 IEContactPerson GuadalupeBedoyaArguelles(DECIE)5 Region AFR6 SectorBoard/GlobalPractice GOV7 WBGPID(ifIEisevaluatingaWBGoperation) PID586467(IFC)

8 WBGProjectName(ifIEisevaluatingaWBGoperation) KenyaHealthinAfrica

9 ProjectTTL(ifIEisevaluatingaWBGoperation) NjeriMwaura

10 Intervention Accountabilitymodelsincludinghigh-intensiveinspectionsandpublicdisclosureofhealthfacilitygrading,

11 MainOutcomes Qualityofcareandpatientsafety,pricesanddemandofhealthservices12 IEUnitofIntervention/Randomization Marketsofhealthfacilities13 NumberofIEUnitsofIntervention 26014 IEUnitofAnalysis Healthfacility15 NumberofIEUnitsofAnalysis 1,050healthfacilities16 NumberofTreatmentArms 2

17 IEQuestion1(TreatmentArm1) Whatistheimpactofhigh-intensiveinspectionsonqualityofcareandpatientsafety,pricesanddemandofhealthservices?

18 MethodIEQuestion1 Randomassignmentatthehealthmarketlevel19 MechanismtestedinIEQuestion1 Package

20 IEQuestion2(TreatmentArm2)Whatistheimpactofhigh-intensiveinspectionsandpublicdisclosureofhealthfacilitygradingonqualityofcareandpatientsafety,pricesanddemandofhealthservices?

21 MethodIEQuestion2 Randomassignmentatthehealthmarketlevel22 MechanismtestedinIEQuestion2 Consumer-basedaccountability23 Gender-specifictreatment(Yes,No) No

24 Genderanalysis(Yes,No) No

25 IETeam&Affiliations

JishnuDas(DECHD,LeadResearcher),GuadalupeBedoya(DECIE),JorgeCoarasa(GTCDR),AmyDolinger(GTCDR),AnaGoicoechea(GTCDR)andNjeriMwaura(GTCDR).TheteamisadvisedbyKhamaRogo(GTCDR)andsupportedbyFrankWafula(GTCDR),PamelaNjuguna(GTCDR),BenjaminDaniels(DECHD),TatianaZarate(DECIE)andGarimaSharma(DECIE).

26 EstimatedBudget(includingresearchtime) USD2.6million27 CNReviewDate July2013(2014registeredintheWBsystem)28 EstimatedTimeframeforIE July2013–December2018

29 MainLocalCounterpartInstitution(s) MinistryofHealth(MOH),RegulatoryBoardsandCouncilsoftheMOH,CountyHealthTeams

4

1. EXECUTIVESUMMARYOur study lies at the intersection of innovations in delivery and reformsof themechanisms that govern healthsystems.BuildingonthreeyearsofworkwithmedicalboardsandtheMinistryofHealthinKenya,weproposetoevaluatedifferentmodelsofhealthinspectionstoimprovepatientsafetyandthequalityofcareinKenyanprivateandpublichealthfacilities.Therearethreecomponentstotheevaluation.Thefirstisthetreatment.Facilitiesinthestudywillbeinspectedusinganewinspectionframework,includinganimprovedJoinHealthInspectionChecklist(JHIC)developedaspartthisprojectwiththeeightmedicalboardsandcouncilsinKenya,theMinistryofHealthandtheprivatesector.2Inthefirsttreatmentarm(henceforth,Arm1)inspectionswillbecarriedoutbythegovernmentinallfacilitiesinthe(experimentallyallocated)healthmarkets,andtheywillreceivewarningsandsanctionsandbere-visitedaccordingtothenewregulatoryframework.Inthesecondtreatmentarm(Arm2)facilitieswillreceivethesametreatmentand in Arm 1 but following the inspection, they will be assigned a prominently displayed patient safety gradeaccordingtotheircompliancecategory.TheimpactofintenseinspectionsinArm1andadditionalpublicdisclosureof their scoring in Arm 2 will be compared with a control group. Of note is that the control group will be a“businessasusual”groug,butwith lowprobabilityof inspectionsandwarningsandsanctions.This isconsistentwiththelegalrequirementthattheboardsandcouncilsmustinspectfacilitiesagainstwhichseriousallegationsofmalpracticeorpatientsafetyviolationshavebeenlevied.Thesecondisthesampleofhealthfacilities. Inthestudyareas,all facilitieswillbegroupedinhealth“markets”,definedasgeographicalclusterswherefacilitiesareincloseproximitytoeachother.3Thesehealthmarketsrangeinsizeinourstudyareasfromsingleclinicstomarketswith67facilities.Theunitoftreatmentisalwaysthehealthmarketandthe impactofcompetitioncanbestudiedbyexaminingdifferential impactsof the treatmentacrossmarketsofdifferentsizes.The thirdcomponent is themeasurementandevaluationofoutcomes.During2014,wedeveloped,pilotedandvalidated a series of new measurement tools for patient safety and quality of care applicable to the Kenyancontext.Thus,ouroutcomemeasureswillincludepatientsafetyasmeasuredthroughtheinspectionChecklist,butwillalsoincludethedirectobservationofpatientsafety(didthedoctorwashhishandsbeforetreatingthepatient)and the treatment of standardized patients (when a surveyor trained to act like a patient presents before thedoctor,whatistheaccuracyofthediagnosisandtreatmentthathe/shereceives).Thiswillallowustolookattheimpactofthetreatmentonthemeasuresthatareincentivizedorsupported(theChecklistmeasures)aswellastheimpactonabroadersetofoutcomesthatmaybesubstitutesorcomplementswiththechecklistmeasure.

2. BACKGROUNDANDKEYINSTITUTIONALFEATURESThesubjectofanalysisofthisstudyistheregimeforinspectionsforhealthfacilitiesinKenya,aswellasvariationsproposed to the current regime. This sectiondescribeshow the inspections systemworks rightnow, aswell ascriticalinstitutionalfeaturesrelatedtoitsimplementation.Therearethreemaintypesofinspectionsthatareconductedinthecurrentregime:(1)inspectionsofnewfacilitiesbefore issuing licenses; (2) inspections of operating facilities due to complaints received, for instance, due tomalpracticecases (i.e., reactive inspections),and(3) inspectionstoverify thequalityofhealthproviders.Duetocapacityconstraintsmentionedpreviously, inspectionsin(3)whichareclosesttothesubjectofthisstudy,occur

2The checklistwas validatedaspartofKePSIE’spilotsduring2014and thenew regulatory frameworkwas launchedby thegovernmentin2015.3Weuseddataonthegeographical locationsofallregisteredfacilities inKenyatoconstructhealthmarketsusingaK-meansclusteringalgorithm.Heuristically,thealgorithmiterativelyplaceshealthfacilitiesintouniquemarketssuchthateachmarketisdefinedbyadistance—nofacilityinamarketcanbemorethanXKilometersfromthecenterofthehealthmarket,definedasthe point equidistant fromall facilities in the existingmarket. In all our computations,we use 4 Kilometers as the distancemetric. Inourpilotandbaselinepatientexit surveys,weverified that75percentofpatientsarrivingata facilitycome fromwithinthatdistance.

5

withlowprobabilityinsmallgeographicregions(inspectorsgototheseregionsandinspectallhealthfacilitiesinthesurroundingareasbasedonthequotafortheinspectionperiod).TheboardsandcouncilsandtheMinistryofHealth that are responsible for the inspections,must supervise them from their headquarters in Nairobi, sincethere isnoregionalnetworkof inspectorsestablished.Currently, thereareeight regulatoryboardsandcouncils(MedicalPractitionersandDentistsBoard,ClinicalOfficersCouncil,NursingCouncilofKenya,RadiationProtectionBoard, Pharmacy and Poisons Board, Council of the Institute of Nutritionists and Dieticians, Kenya MedicalLaboratoryTechniciansandTechnologistsBoard,andPublicHealthOfficersandTechniciansCouncil).On Julyof2012, theMedicalPractitionersandDentistsBoard issueda JointHealth InspectionsChecklist,whichwas developed in cooperation between different stakeholders from the private and public sectors and withsupport from the IFC’s KenyaHealth in Africa project. This Checklist contains a set ofminimum standardswithquality of care and patient safety and aims to create a more transparent inspection system, and enhancecommunication and accountability between health inspectors and health facilities in the private sector. It alsointendstodecreasetheburdenfromtheprevioussystemwhereeachoneoftheboardsandcouncilsconductedindividualinspections.InOctober2013,aspartofthisproject,allregulatoryboards,theMinistryofHealth,andtheprivatesectorsignedtheWindsorDeclaration,acommitmenttoworkonimprovinginspections,developingarisk-basedscoringsystemand consumer signalingmechanisms aswell as evaluating rigorously different inspections regimesdeveloped inthis proposal. A technical group with representation from all stakeholders has been meeting regularly andadvancingonallthesefronts.ThistechnicalgroupbuiltononeyearoffieldexperienceimplementingthegazettedJointHealth InspectionsChecklistandmade further improvements toan itemizedchecklist,aconsistent scoringsystem,andanimplementationmanualthatreducesdiscretion.Therewasalsoagreementthataclearsetofrisk-based warnings and sanctions is needed. In 2015, the government launched the new regulatory frameworkincluding the improved Joint Health Inspection Checklist now to be applied both in private and public healthfacilities,andascoringandawarningsandsanctionssystemsthatdefinesthenewregulatoryframeworkthattheimpactevaluationwilluseforitsimplementation.Kenyastandsoutintheregionandamongotherlow-incomecountriesforatleastthreereasonsthatmakeitanideal testcaseforthisstudy: (i)Kenya isoneoftheonlycountriesthathasstandardizedandnational figurestodiagnose the magnitude and characteristics of the problem; (ii) although not well known, levels of medicalknowledge among health care providers in Kenya are high, relative not only to other countries in Sub-SaharanAfrica,butalsocomparedtocountrieslikeIndia:diagnosticaccuracyofhealthprovidersisabout72percentacross5differentillnesses,comparedto57percentinTanzania,and34percentinSenegal(WorldBankSDI,2010-2012),and (iii) the main stakeholders are committed to the process, as is shown in the Windsor Declaration, thecontinuousworkfromthetechnicalgroupontheregulatoryreform,andthesupportfromandparticipationoftheCountyrepresentativesandtheirhealthteams.

3. LITERATUREREVIEWANDKNOWLEDGEGAPSAgrowingbodyofexperimentalevidencepoints to theeffectivenessofauditsandmonitoring (accompaniedbyrewardsorsanctions)inimprovingtheperformanceofserviceproviders(Olken2007;Duflo,Hanna,andRyan2012and;ZamboniandLitschig2013). Investing inbettergovernanceandmonitoringof serviceprovidersmaybeanimportantcomponentofimprovingstatecapacityforservicedeliveryinlow-incomecountries(BesleyandPersson2009; Muralidharan, Niehaus, and Sukhtankar 2014). Further, where the audit and monitoring functions aredirectly linked toconsumerchoice,additional incentivesthroughconsumerdemandhavebeenfurthershowntoimproveoutcomes.LeslieandJin(2009)andAndrabi,DasandKhwaja(2013)bothshowthatpublicdisclosureofinformation(withoutfurtherintervention)improvesmarketoutcomesbyincreasingqualityanddecreasingprice.Finally,thepublicdisclosureofinformationisanimportantadministrativefunctionthatcanplayaroleinqualityimprovements in both the public and the private sector. This is a particularly attractive proposition since it

6

providesclearpolicyguidanceonpotentialleverstoimproveprivatesectorperformance—akeychallengethatisincreasinglyunderfocusinKenyaandotherlow-incomecountries.4Theimpactevaluationseekstofillseveralknowledgegaps:

1. Thefirstknowledgegapthatthisstudywillfillistheroleofinspectionsasaviablepolicyleverthataffectsservice provision in the public and private sector. Frequent calls for more inspections and greaterregulationarenotbackedbysystematicdataontheoverall impactofsuchmeasures,particularlyintheprivate sector.5In Kenya one concern is that inspections may not work because they become amechanismforextractingrentsratherthanimprovingquality–thisimposesacosttothefacilityonlyiftherent-extraction is correlated with the quality deficit, but whether this is indeed the case is unclear.Compounding theproblem is that theprecisedesign for inspections to improvepatient safety and thequalityofcare isunclear.For instance,should inspectionsbepurelypunitive,or isamixofpunishmentandrewardbetter?Shouldmentoringbeapartoftheinspectionprocess?

2. Howtoimprovepatientsafetyandqualityofcareinresourcepoorsettingsisthesecondknowledgegapthat this study addresses. Patient safety and quality of care are integral to the policy debate in low-incomecountriesbutthelackofeffectivesystemstoregulate,measure,andimprovethequalityofcareplaces severe limits on evidence-based decisionmaking. This is particularly worrisome because recentdata suggest very low levels of patient safety andpoor quality of care. In Kenya (oneof the only low-income countries with a national survey on patient safety), only 2% of health facilities in 2012 werecompliantwithminimumstandardsonpatient safety indicators related toprotocols andpatient safetysystems.6InIndia,astandardizedpatientsurveyshowshighratesofunder-provisionandover-treatmentinboththepublicandprivatesector(Dasetal.,2012).7Perhapsbecauseoftheseintrinsicuncertainties,thecapacitytoconductinspectionsonaregularbasisislow–inKenya10percentofallhealthfacilitiesarescheduledtobe inspectedinagivenyear(fromover10,000healthfacilitiesinthecountry),andintheAfricaregion,outof45countrieswithdejureinspectionregimes,only5countriesreportactuallycarryingoutinspections(WorldBank,2011).

3. Thethirdknowledgegapthatthisstudyfills is the impactof inspectionsandmonitoringwhentheyaredesignedandconductedastheywouldbe inafullyscale-upgovernment-ledprogram.Conductingsuchan evaluation in a “stand-alone” manner would arguably yield greater scientific validity, but lowerpotentialforscale-up.Incontrast,theinspectionsystemthatwillbeevaluatedandtheevaluationdesignitselfhasbeenbuiltthroughconsensuswithmultiplemedicalboardsandtheMinistryofHealthinKenyaovera3-year longprocess. ItthereforeaddressesspecificconcernsthathavebeenraisedinKenya–andother low-income countries–on the benefits and forms of different inspection regimes, and introducesbasic incentives for quality improvement for health care providers and stakeholders. This country-ledapproachevaluatestheprocess,asitwouldworkwhenfullyscaledupandthereforehasstrongpredictivepower.

4In Kenya, close to half of all registered facilities are private. Of these, one-third are Faith Based Organizations or Non-Governmental Organizations and the remaining are secular private facilities. Among the poorest quintile in Kenya, 47% ofhouseholdsseekcareataprivatefacilitywhenachildissick(Barnesetal.,2010).5Infact,wecannotfindasinglesystematicevaluationoftheimpactsofregulationandgreaterinspectionsintheliteratureonqualityofcareinlow-incomecountries.6IFC-WHOKenyaMinimumPatientSafetyStandardsComplianceSurvey(2012)7AreplicationoftheIndianstudyinChinashowedcomparablelevelsof(poor)quality(Sylviaetal.,2014).

7

4. POLICYRELEVANCEThepolicychallenge that the impactevaluationresearchseeks toaddress liesat the intersectionofgovernanceandhealth.Asmentionedpreviously, globally and inKenya there is littleevidenceon (a)whetherandhow theadministrative function of regulation—in this case through inspections and monitoring coupled with publicdisclosures–affects consumer and producer behavior in healthmarkets in low-income countries; (b) how thesesystemsoperatewhentheyare“atscale”;and(c)whetherthesesystemscanbeusedtoimprovequalityofcareinpublicaswellasinprivateproviders.Addressingtheseissuesattheglobalandcountryleveliscritical.Aswewillshowbelow,qualityofcareandpatientsafety isasignificantconcernthataffectsthehealthofpopulations inmostresource-poorsettingsandalthoughinspectionsandmonitoringarefrequentlyproposedpolicyactions,thereislittleevidencetosupportthesepolicyrecommendations.IntheUnitedStates,thekeymilestoneonpatientsafetywastheInstituteofMedicinereport(Kohn,CorriganandDonaldson,1999)andthisevaluationhasthepotentialtobothhighlighttheissuesofpatientsafetyandqualityofcareinresourcepoorsettingsandsuggestpotentialpolicyleversthatcanimprovethesekeyprocessoutcomesinthehealthcaresector.8Appropriatepolicytowardstheprivatesector isan increasingconcernbutthe lackofsystematicevidenceagainhamperspolicyaction.Theresultsfromthisstudywillthusprovideguidanceonkeyissuespertainingtothedesignof health system governance and administration and the role of producer and consumer incentives in theimprovementofquality.Acrucialpolicy-relevantimplicationofthisstudyisthepotentialtoaffecthealthoutcomesinKenya,whichmaybereplicatedacross theAfrica region ingeneralusing similarmechanismsofaction.Morebroadly, for low-incomecountriestheresultsfromthisimpactevaluationwillprovide:

A. Theviabilityandimpactsofpolicyactionslinkedtoinspectionsandmonitoringforqualityimprovementinpublicandprivateclinics;

B. Thedifferentialimpactsofsuchactionsonpatientchoice,qualityandpricesinpublicandprivatefacilitiesC. The relative impacts of inspections only versus inspections plus public disclosure of information for

improvingquality;D. Asetoftoolsandinstrumentsthatcanbebroadlydeployedbothinsurveysofqualityandinthedesignof

inspectionsystemsacrossdiverselow-incomesettings.

5. THEORYOFCHANGETheevaluationtargetsallhealthfacilitiesinthreecountiesofKenya(around1,100or10%ofthetotalnumberofhealth facilities in the country). Itwill have three arms, eachofwhich is expected to affect quality of care andpatientsafetyforallthepopulationintheircatchmentareas.ThedetailsonthetreatmentarmsarediscussedinSection 8. The control group will be the “Business-as-usual” low-intensity health inspections regime, the firsttreatmentarmwillbeaHigh-intensitypublichealthInspectionsregimeandthesecondarmwillbeHigh-intensitypublichealthinspectionsandpublicdisclosureofthehealthfacility’sscore.Thereare threecomponentsacross thesearms: (1)a regulatory frameworkaccompaniedbyclearguidelinesontheminimumpatientsafetystandardsthatfacilitiesareexpectedtocomplywith;(2)amonitoringsystemtotrackcompliancewithminimumqualityofcareandpatientsafetystandardsandenforcewarningsandsanctionsovertime;and(3)ascoringandinformationreportcardsystemtopublicizehealthfacilities’compliancewithminimumpatientsafetystandards.Eachtreatmentarmwillinclude2or3ofthesecomponents.Interventionactivitiesaredefinedbythesethreecomponentsasfollows:

8 A memorable comparison is the comparison of patient to airline safety, whereby iatrogenic injuries from patienthospitalizationswasthe“equivalentofthreejumbo-jetcrasheseverytwodays”(Leape,1994).

8

(1) Aregulatoryframeworkaccompaniedbyclearguidelinesontheminimumqualityofcareandpatientsafetystandards:ActivitiesinthiscomponentincludethestreamliningoftheChecklist,thedevelopmentofadetailedimplementationmanual,ascoresystemtogradehealthfacilitiesandasystemofwarningsand sanctions for non-compliant HFs. All these activities took place during 2014, from a process thatstartedyearsbefore.Thenewregulatoryframeworkwaslaunchedin2015tobeappliedtoallpublicandprivatehealthfacilities.

(2) Amonitoring systemto track compliancewithminimumqualityof careandpatient safety standardsandenforcewarningsandsanctions:Thiscomponent includesactivitiesrelatedtothedefinitionoftheparametersfortheinspections,includingwhotheinspectorswillbe,howoftentheinspectionswilltakeplace, and the system toenforce thewarnings and sanctions (1). Several of theseelementshavebeendefined, and a planning meeting with the participant counties and the MOH is scheduled forAugust/September2014.

(3) Ascoringand informationreportcardsystemtopublicizehealth facilities’compliancewithminimumquality of care and patient safety standards: The scoring system will be based on the regulatoryframeworkandcompliancecategoriesfrom(1),andthereportcardsystemwillbedevelopedconsistentwiththatscoringinawaythateffectivelycommunicatestopatientsthestatusofthehealthfacilities.

Figure1presentsasimplifiedtheoryofchangebehindthebroadinterventiontobeevaluated,includingitsmaincomponents/inputs,activities,outputs,andthehypothesizedcausalchaintoselectoutcomesofinterest.Themainassumptionbehindthistheoryofchangeisthatthecombinationoftwoormoreofthesecomponentsleadstoinspectionsystemsthatprovideincentivesforhealthfacilitiestocomplywith/increasequalityofcareandpatientsafetystandards(i.e.,clearrulesofthegame,goodinformationsystems,aworkingsystemofwarningsandsanctions, as well as a consumer liability system through the scorecard grading that create costs/benefits thatincentivizehealthfacilitiestoimprovequalityofcareandpatientsafety).For instance, activities and outputs from component (1) are expected to affect the knowledge of the healthfacilities in termsofqualityandpatient safety,which isanecessary (althoughnot sufficient) condition toaffectcompliancewiththestandards.Activitiesandoutputsincomponent(2)areexpectedtodirectlyaffectcompliancebycreatingincentives(andcosts)fornoncompliance.Finally,thethirdcomponentisexpectedtoaffectconsumerdemand,which in turnmay leadtochanges inproviderbehavioranda reallocationofdemandto facilitieswithhigher patient safety scores In the long-term, better quality of care contributes to improvements in the healthoutcomesofthepopulationservedbyhealthfacilities.Weexaminetheimpactoftheinterventionsonatriadofmeasuresthattakesintoaccountoursettingwithpublicandprivateproviders:QualityandPatientSafetyinthemarket,QuantityinthemarketandPricesinthemarket.Thefirstsetofoutcomesareintermediateoutcomesinthecontinuumofpatientsafetyandqualityofhealthcareservicesmeasuredthrough(A)compliancewithaqualityofcareandpatientsafetychecklist;(B)compliancewithkeyinfectioncontrolmeasuresindoctor-patientinteractions;(C)accuracyofdiagnosisandtreatmentasevaluatedthroughtheuseofstandardizedpatients;and(D)prevalenceofunnecessaryorharmfulmedication.Consumerandprovider behavior are further measured through (A) patient loads in different health facilities and (B) priceschargedtopatientsforconsultationsandlaboratoryprocedures.

9

Figure1.TheoryofChangeofaHigh-stakesandHighConsumerLiabilityHealthInspectionRegime

6. HYPOTHESES/EVALUATIONQUESTIONSTheoverallresearchquestionwilladdresshowhealthmarketsreacttointensiveinspectionsandpublicdisclosureofinformation.Thus,theevaluationwillprovidebothchangesinmeanhealthcaredeliveryoutcomesandproviderandconsumerbehavior,aswellaschangesinthegradientbetweenpriceandqualityinthesemarkets.ThefollowinghypothesesderivedfromthelogicmodelexplainedindetailinSection5,willbetested:

• HYPOTHESIS1:High-intensityhealthfacilitiesinspectionswithenforcementofwarningsandsanctionsincreasecompliancewiththequalityandsafetystandardsthataretested(seeoutcomesofinterest).Theextentofimprovementdifferswhencomparedtoahigh-intensitypublic-ledinspectionregimewithpublicdisclosureofhealthfacilitygrading.

• HYPOTHESIS2:Inspectionmodelstestedinlocalmarketschangequalitymetricsthatarenotsubjecttoformalinspection.Forinstance,caremeasuredthroughStandardizedPatients,whowillassessthequalityofcareprovidedalongthreedimensions–treatment,diagnosisandadherencetothechecklist–alongwiththepricechargedattheendoftheinteraction,aswellashealthcareworkerpracticeofpatientsafetyintheinfectionpreventionandcontroldomain,measuredthroughdirectobservationofpatient-healthcare-workerinteractions.

• HYPOTHESIS3:Thereisanimpactofthehigh-intensityhealthfacilitiesinspectionsandtheadditional

publicdisclosureofhealthfacilitygradingsystemsonmarket-leveloutcomes(forinstance,ontheaveragepriceinamarketwithinspections)aswellasthedifferentialimpactonclinicswithhigh/lowinitialsafetyscores.Previousliteratureonsuchreportcardsinsettingsoutsidehealthcare(JinandLeslie,2003;

Activities

Develop standards and guidelines on PS/QC

Improved Quality of Care Delivered and PS practices

Short- and intermediate-term outcomes

Improved knowledge of HFs about PS/QC

standards

Improved Health Indicators

Inputs

Regulatory framework and rules on standards of PS/

QC

1

System to check for compliance/progress with PS/QC standards

2

Scorecard grading and information system for HF

3

Improved knowledge and

attitudes (choices) of

patients regarding HFs

and their PS/QC

Develop a scoring system for HFs, and warnings and

sanctions for different levels of non-compliance with

standards

Develop dissemination plan on standards for HFs

Develop inspections parameters (e.g., inspectors, tool for inspectors, training, frequency of inspections)

Develop enforcement plan for warnings and sanctions

Develop supportive supervision parameters and

plan–if any (e.g., supervisors, frequency)

Develop dissemination plan about inspections/supportive supervision systems for HFs

Develop signaling strategy and information system for

HF (e.g., scorecards)

Develop dissemination plan of signaling for patients and

HFs

Improved Compliance with PS/QC standards

Long-term outcomes

HF: Health Facility PS: Patient Safety QC: Quality of Care

Outputs

Standards and guidelines on PS/QC available to HFs

Scoring and warnings and sanctions system for different levels of non-compliance with

PS/QC available to HFs

Number of presentations and information products on

framework delivered to HFs

Inspections conducted according to defined

parameters (record of inspections, frequency,

duration)

Warnings and sanctions enforced (number of

warnings and sanctions followed up/enforced)

Supportive supervision sessions conducted according

to the defined parameters

Presentations/ dissemination activities conducted

Signaling system developed

Scorecards/signals posted after inspection/supportive

supervision

Dissemination activities for patients and HFs; patients’ awareness/recognition of

signaling

10

Andrabi,DasandKhwaja,2013)showsthatprices,qualityandquantityadjustwhensuchinformationisprovided.

Together, the tests to thesehypotheseswill allow the study to answer key questions regardinghealth facilitiesinspections and how they should be carried out in this context. The following are the evaluation questions,resultingfromthehypotheses,whicharealignedwiththetreatmentarms:

• Whatistheimpactofgovernment-ledhigh-intensityinspectionsandenforcementofwarningsandsanctionsoncomplianceofhealthfacilitieswithminimumqualityandpatientsafetystandards,qualityindicatorsofhealthcaredeliverythatwerenotinspected(includingqualityindicatorsmeasuredthroughstandardizedpatientsanddirectobservationofselectpatientsafetypractices),pricesanddemand?

• Whatistheimpactofgovernment-ledhigh-intensityinspectionsandpublicdisclosureofhealthfacilitygradingandenforcementofwarningsandsanctionsoncomplianceofhealthfacilitieswithminimumqualityandpatientsafetystandards,qualityindicatorsofhealthcaredeliverythatwerenotinspected(includingqualityindicatorsmeasuredthroughstandardizedpatientsanddirectobservationofselectpatientsafetypractices),pricesanddemand?

7. MAINOUTCOMESOFINTERESTWeexaminetheimpactoftheinterventiononatriadofoutcomegroupsthatareimportantinsettingswithpublicandprivateproviders:QualityandPatientSafetyinthemarket,QuantityinthemarketandPricesinthemarket.Quality and Patient Safety will be measured through three main indicators that gauge diverse dimensions ofquality and patient safety at different points in the continuum of the health care chain: (1) Adherence to aChecklist developed by the regulators and that includes indicators of quality and patient safety related toprotocols, infrastructure, and equipment at the HF level; (2) Adherence to patient safety practices related toinfectionpreventionandcontrol;(3)Adherencetocase-specificchecklistsofessentialandrecommendedcareforfourmedicalcases.Quantity(demand)ofcareserviceswillbemeasuredthroughpatientflowrecordsfromhealthfacilitiesandadministrativedata,andpriceswillbemeasuredthroughpatientexitsurveys.Below is a listof themainoutcomesof interest for this analysis, aswell as the sourceand themethodofdatacollectionusedineachcase.

• Adherence to a Checklist of quality of care and patient safety standards. This is a Checklist forSingular/Joint Inspections for Public and PrivateMedical Institutions (henceforth Checklist), which hasbeendevelopedbyatechnicalworkinggroup(TWG)includingtheregulatorybodiesundertheMinistryofHealth, the private sector, and other stakeholders. This Checklist includes indicators of infrastructure,equipment,staffcharacteristics,andprotocolstomeasurequalityandpatientsafetyacrossallunitsofaHF,andhenceitallowsmeasuringhowHFsstandwithrespecttothestandardssetbytheregulation.TheresultsaremappedintoascoreandthemeasureofadherencewewilluseisthescoreasapercentageofthemaximumscorepossibleforeachHF.WedevelopedanelectronicversionofthisChecklist.DatawillbecollectedthroughsurveystotheHFin-chargeandstaff,aswellasthroughobservationandverificationduringthesurveysfollowingcloselytheprotocolsthatinspectorsfollowwhentheyconductinspections.

• Adherencetopatientsafetypracticesrelatedtoinfectionprevention,andcontrol(IPC)inprimarycare,9whicharemeasuredbydirectobservationofselectpracticesintheconsultationroomandthelaboratory.Theseoutcomeswillmeasureadherencetopatientsafetypracticesforfivegroups:(i)handhygiene,(ii)injection and blood draw practices, (iii) use of gloves, (iv) disinfection of reusable equipment, and (v)

9We use the term primary care to distinguish inpatient and outpatient care. A better term would be a “patient seekingambulatorycare.”

11

wastesegregation.Thisselectionwasbasedontheirhigh-value(i.e.stronglinktohealth-careinfections),highfrequencyofopportunities(i.e.,pointsintimeduringthecarechainwhenapatientsafetypracticeshould be performed to avoid likely adverse events), and feasibility to develop valid, reliable, andgeneralizableindicatorsintheplannedtimeandconditions.

• Adherence to case-specific checklists of essential and recommended care for 4 medical cases. ThisdimensionofqualityofcareandpatientsafetyismeasuredthroughStandardizedPatients,whichallowsus to gauge the extent of correct treatment (both under and over-treatment) in these facilities. ThemethodofStandardizedPatients(SPs)wasusedinthefirstlarge-scalepopulationstudyinIndiain2008-2009(See,e.g.,Dasetal.,2012).SPsarepeoplefromthelocalcommunitywhoareextensivelytrainedtopresent the same case tomultiple providers. To assess thequality of care, SPs are trained to recall allhistory questions, examinations, and diagnoses. They are debriefed with a structured questionnairewithin one hour of the interaction. The quality of medical advice is assessed by the time spent withpatients, by providers’ adherence to case-specific checklists of essential and recommended care, thelikelihood of correct diagnosis, and the appropriateness of treatment. SPs will undergo an exit surveywherethedataontheirinteractionwiththedoctorisrecorded.For the study, we developed and piloted the clinical cases for Kenya. The scripts and SP and surveyinstruments for thosecases,alongwithadetailed instructionmanual forSPsand fieldenumerators forfuturescaled-updatacollectionwerevalidatedin2014.Forthiscomponent,weworkedwithanadvisoryboard constituted by theMinistry of Health to tailor tracer conditions to the Kenyan setting and arecurrentlypilotingtheapproachinNairobi.Thefourtracerconditionssatisfythecriteriaofrelevance(highincidence in low-income settings), low risk to SPs (for instance, from invasive examinations), andestablishedmedicalprotocolswithclear triage,managementandtreatmentchecklists.For threeof thefourcases (asthma,diarrheaandTB) theMinistryofHealthhasdevelopedanddisseminateddiagnosticandtreatmentguidelines.Forthefourth(unstableangina)theseguidelineshavenotyetbeendevelopedordisseminatedsothatthecardiologydepartmentatNairobiHospitalconsultedEuropeanandAmericandiagnosticand treatmentguidelines.Thesecaseshadpreviouslybeendevelopedandvalidated in Indiaand later in China. Therefore, using the same tracer conditions allowed for speedier development,allowingtheteamtofocusoncontext-specificityandfacilitatedcomparisonswithothercountries.

• Prevalenceofunnecessaryorharmfulmedications,whichismeasuredthroughtheanalysisofthedrugprescriptions collected by the SPs described previously. The prescriptions will be compared to theguidelinesdevelopedforthefourSPcasesandtheinstancesofunnecessaryorharmfulmedicationswillberecordedbasedonmedicaljudgment.

• Price of services at health facilities. The data on prices will come from a short survey that will be

collected frompatients fromvariousdemographic characteristicsexiting thehealth facilities, aswell asfromSPs.Inlargehealthfacilities,samplingofpatientswillbedonerandomlyusingaskiproutine.

• Quantity(demand)ofhealthcareservices:Thedataondemandcorrespondstothenumberofpatientsinthe lastcompletemonthat thetimeof thesurvey,collectedat the facility level throughaHFsurveyfromtheformsreportedtotheMOH,orfromtheHFsalternativerecordsystem.Wewereencouragedtofindduringthepilotthatcollectingthisdata(aswellasprices)wasnotaprobleminthefield.Thesedatawillbealso retrieved fromadministrative records fromtheMOHthatcollectsselectmonthly indicatorsfor themajorityofHFs tocrosscheck.According to theMOH, thecompliancewith thebasicaggregatefiguresisaround90%.

OtherIntermediateLevelIndicatorsThe individual outcomes used to build the previous key outcome indicators constitute a set of intermediateindicators. For instance, the Checklist in the first bullet contains disaggregated indicators on infrastructure,

12

protocols,equipmentandstaff,thecase-specificchecklistscontainseveralmeasuresofqualityandpatientsafety,and the infection, prevention and control measures include diverse measures of hand hygiene, safe injectionspracticesanddisinfectionandwastesegregation.Additional informationwillalsobecollected includingothercharacteristicsofhealthfacilities,staffandpatients.Among the HF indicators are services availability, accreditation and other supervision services the HFsmay bereceiving.Additional staff characteristics includeeducation level, school,andsalary,while thepatient indicatorsinclude socioeconomicmeasures, health care alternatives, time andmeans to access the health facility, detailsabouttheirinteractionwiththehealthproviders,andtheirlevelofsatisfactionwiththeservicesprovided.Otherintermediateleveloutcomeswillincludeknowledgeteststopatientsaboutthescoreofthehealthfacilitiesandwhattheymean,aswellastoinhabitantsofthemarket-areaswheretheinterventionistakingplacetoassesshowknowledgeisbeingaffected.Amonitoringsystemtocheckforadherenceprotocolsandimplementationwillbeinplace.Theinspectionswillbeconductedelectronically. Furthermore,asonepartof theproject,wehave regularly visitedhealth facilitiesandcompiledfieldnotesfromunstructuredinterviews.Wewillcontinuetodosoandwill incorporatediscussionsonthebarrierstoimprovementandactionsthatthehealthfacilitiesaretakingtoimprovequality.the132

8. EVALUATIONDESIGNANDSAMPLINGSTRATEGYThisevaluationusesaclusterrandomizedexperimentaldesigntoestimatethecausalimpactoftheinterventions,where the clusters aremarkets of health facilities, and the cluster size is themarket size, or number of healthfacilitiespercluster.All localmarketsofhealthfacilitiesinthethreecounties–Kakamega,Kilifi,andMeru–willberandomized into treatment and control groups, taking into consideration the size of the market. These threecountieswerechosenafteranextensiveconsultationwithrepresentativesoftheKenyancountyhealthexecutives,basedonprior analysis of healthmarkets in all Kenyan counties. Specifically,we constructedhealthmarkets ineachcountyandhighlightedthecountiesthatcontainedmarketswithdiversesizes.InsomecountieslikeTurkana,therewerefewmarketswithmorethanonehealthfacility(typicallyinthearidNortherncounties)andevaluationsfromthesecountieswouldnotbepredictiveofperformanceindenserhealthmarkets.Inothercounties,suchasMombasa,mosthealthfacilitiesareinmarketswithalargenumberofotherhealthfacilities.Insuchcounties,thepredictive power of the evaluation for smaller markets is poor. Ten counties satisfied our technical criteria ofhealthmarketswith diverse sizes. The analysis and countieswere discussedwith the county health executives,whodecidedtheultimatestudyareas.The choice of a cluster-randomized experimental design stems from the nature of the intervention and therelevanceofcapturingtheeffectoftheinterventionatthemarketlevel.Particularly,thetreatmentappliedtoonehealth facility is thought to have potential spillovers on geographically proximate health facilities throughconsumerdemand.Therefore,thecluster-randomizeddesignwasselectedbecauseofthepotentialspilloveratthemarket-level, and the policy-relevance of these level impacts in terms of quality of care and patient safety,quantityandprices.Themarketsaredefinedat this stagebygeographicdelimitations.Basedonpreliminarydata fromthebaselinefromaround8,000patients surveyedwe find thataround75%of the respondents live4kmor closer from thehealthfacilitytheyvisited.Usingthisdistance,weapplyaclusteringalgorithmcalledtheK-meansalgorithmthatidentifies markets of health facilities by the nearest center, conditional on the K-distance to the center. Ouralgorithm stops creating new clusters when all facilities are within the predefined distance (4 km) from theirrespectivemarket center. Based on this exercise, we estimate there are about 260 suchmarkets in the threecounties selected, with diversemarket size ranging from singletons tomarkets with 2, 3, 4, 5 or more healthfacilities.

13

Anappropriaterandomizeddesignwithbalancebetweentreatmentandcontrolgroupsprovidestheindependentassumptionbetweentreatmentstatusandtheoutcomesofinterestrequiredtoidentifythecausalimpactoftheinterventions. A potential risk of a cluster-randomized design in terms of bias of the estimates stems fromunbalances due to important differences across the clusters that may be correlated to the impact of theintervention(e.g.inourcasethefactthatthemarketsareofdifferentsizesmayhavesignificantlydifferentqualityortheimpactonqualitybasedonthesizeofthemarket).Weaddressthisriskthroughappropriatestratification,whichwillalsoimprovetheprecisionoftheestimates.Thevariablesforthestratificationwillbeselectedafterthebaselineiscollected,andbeforetherandomizationtakesplace.A potential spillover effect in controlmarkets is a possibility.Wewill have to dealwith this as the experimentcontinues.Wethinktheconcernisparticularlysalientinareaswheremarketsarehardtodefine—manypatients“bypass”facilitiestovisitthosethatarefurther,especiallywhenusingpublictransportsuchasMatatusonmarketdays. Inthiscase,wewillhavetothinkofa“control”marketasonewherethesignalonquality ismissing,andthereforeconsumershaveto inferthequality fromotherobservablecharacteristics.Wecantryandelicitdirectinformation on this (for instance, in asking about perceived quality for controlmarkets in exit surveys that arealreadyplanned)or formallymodel inference inmissingdatathroughmultiple imputationmethods. It ishardatthispointtosaywhatisgoingtohappen;wethinkthebestwaytodealwiththisimportantissueistoputinplacedata collectionmechanisms that allow us to check how demand is responding on a timely basis. Thus,we areworkingwith the counties to see if their data on usage (which comesmonthly) reflect actual usage, and if so,whetherthesecanbeusedtomonitorsuchspillovers.Ifso,wecananswerquestionslike“Doesdemandincreaseinmarkets that are graded relative to the control groups” (subject to theusual identification concerns in thesespilloverspecifications).Itispossiblethatinspiteoftherandomizeddesignandstratification,differencesbetweentreatmentandcontrolclustersremainduetochanceordifferenttake-uporresponseratesofthehealthfacilities.Thesedifferences inobserved (or other unobserved) characteristics represent a problem if they are correlated with the treatmentstatus and the outcomes of interest. Special care is being taken in the design of the questionnaires to collectrelevant information that canbeused tomakeappropriate corrections (e.g. for instance,modeling selectionorparticipation)ifatallneededtoobtainunbiasedestimators.Finally,exante,wearenotpoweredtodetectsmallerchangesbysubgroups,sincethesamplesizeisdeterminedby the census of facilities in the 3 countieswherewe areworking. Ex post,wewill look at sub-group analysis,especiallybyownershipand“level”.

8.1 TREATMENTANDCONTROLGROUPSThe IEwill have a control group and two treatment arms: (a) the control groupwill be the “Business-as-usual”HealthInspectionsRegime;(b)thefirstarmwillbetheHigh-intensityPublicHealthInspectionsRegime;and(c)thesecond arm will be a the High-intensity Public Health Inspections Regime with Public Disclosure of FacilitiesGrading. All groups will share a well-developed regulatory framework and standardized guidelines to conductinspections and assessments, and will be to the extent possible revenue-neutral (that is, the same amount ofmoneywillbespentoneach)butwilldifferinthefollowingimportantfeatures:

• ControlGroup(Counterfactual):"Business-as-usual"HealthInspectionsRegimeHealth facilities in this group will operate under the current regime for health inspections, whereinspectionsare low-probabilityevents.Underthisregime,theprobabilityofbeinginspectedduringtheyear isaround10%andtheinspectionsarecarriedoutbyboardmembersandstafffromtheMOHanddifferentregulatorybodieswhoaremobilizedfromNairobitothespecifiedlocation.Thisarmwillactasacounterfactual,measuringwhatwouldhappenwithcompliancewiththequalityofcareandpatientsafetystandardsandtheotheroutcomesofinterest,ifthecurrentregimeoflow-probabilityinspections,andanimproved regulatory framework are in place. We note that a “pure” control group cannot beimplementedinthissetting,sincepatientcomplaintsandmalpracticehavetobeinvestigatedasamatter

14

ofduecourse.Additionally,aspartof thisproject, theregulatory framework isbeing improvedwithanenhancedChecklistandimplementationmanual,ascoringsystemtogradehealthfacilitiesandsystemofwarnings and sanctions for noncompliant health facilities. This regulatory framework will apply to theentirecountrysothecounterfactualwillalsohaveanimprovedregulatoryregime.

• Arm1:High-intensityofPublicHealthInspectionsRegimeInthisregimeinspectionswillbeproactiveandeveryfacilityinthisarmwillbeinspectedwithcertainty.Regional teams of full-time inspectors will be deployed to inspect all health facilities in the marketsassignedtothispilot.TheMOHandregulatorybodieswillberesponsiblefortrainingandsupervisingtheregionalteamsandthefull-timeinspectorswillbeaccountabletothem.Inpractice,thismeansthatthefacilities in themarkets assigned to this armwill be inspectedwith a probability of 100% by teams ofdedicated inspectors that report to the boards and councils and the boards and councilswill use theirlegal authority to enforce any sanctions as a consequence of the inspection. The frequency of theinspections will be once a year with appropriate follow-ups (due to warnings and sanctions) andresponses fromhealth facilities for re-inspection if theywanta change in their risk categorization.Thisintervention arm will be operating for a year. There will be a dissemination campaign for patients tointroduce them to the warnings and sanctions. 10 Appropriate governance mechanisms will beincorporated intothecontractstoensurethispilotmimicstotheextentpossiblewhatwouldhappenifthe regulatory bodies had the resources to create a cadre of dedicated inspectors anddeploy them inregionalteamstocoveroneormorecounties.

• Arm2:High-intensityofPublicHealthInspectionsRegimewithPublicDisclosureofQualityScoreThisarmincludesinterventioninArm1plusthepublicdisclosureofthequalityscoreateachfacility.Attheendoftheinspection,eachfacilitywillbeassignedaletter-grade(provisionallyAthroughD)thatwillbe prominently posted on the health facility door. Fraud control mechanisms will be put in place toensurethatclinicsdonotaltertheinformationontheclinicdoor.Thescoringsystemhasbeendevelopedmappingthecompliantcategoriesasdefinedbytheregulation.Therefore,thehigh-intensityfeature,andthepublicdisclosureoftheHFqualitystatuscontain incentivesand informationthatbyand largewerenotavailableforthemarketsbefore.We are fully aware and cognizant of theworry that this prominent displaymay be corrupted and arecurrentlyworkingwiththeMinistryofHealthandthechosenstudyregionstostudyvariousoptions.Asapolicystudy,oneofourobjectivesistoensurethattheultimatemodelisscalableandwerecognizethatthismayleadtoa(technically)sub-optimalchoicewithsomecorruption.Wewillstudythecorruptionofthe grade display as one outcome of the study.We have agreedwith theMinistry of Health and theCountyHealthExecutivesthattheworkwillbecompleted in3countiesandhavespokentothecountyexecutives on this concern. The final system is still being worked out and two options have beenproposed.First,thepostedgradewillincludeanumberthatpatientscansendanSMSto,andfromwhichtheywillreceiveanimmediateSMSconfirmingtheGradethatthefacilityreceived.Weareinvestigatingtheoptionthat theycan,at thatpoint, reportback thegradethat theysawwithheavypenalties if thegradehadbeenaltered. Inaddition, ineachof thesecounties (surprisingly) thecounty teamsare fairlyactiveinvisitinghealthfacilities—somethingweverifiedbyvisitingmultiplehealthfacilitiesandcheckingvisitor’s log books which showed visits by the county health team on a regular basis (even in remotelocations).Finally,wewillbuildinrandomchecksinmarketsverifyinggrades.

Afewancillaryissuesthat“might”affecttheinterpretationofourresultsfollow:

10Attheendofeachinspection,theHFreceivesacopyoftheChecklistfilledinbytheinspectorssotheywillknowindetailwhytheygotthegradetheydid.Thisalsoactsasafeedbacksystem,andprovidesapotentialimprovementplansincethewaytheChecklistisitemizedwillallowfacilitiestoparseoutthespecificareasforimprovement.Additionally,thewaythewarningsaredesignednowimpliesthattheywillalsohaveadeadlineandaplanofwhatneedstobeimprovedintheshortterm(beforethenextwarningfollow-up,ortogetoutofthesanction).

15

• A valid concern is the extent to which a one-year intervention length of exposure is long enough toobserve impacts. A longer intervention and follow-up timeframe would be ideal. We are planning onraisingadditionalfundingsourcestofollow-upifweseeduringourmonitoringthatthisisturningouttobeaseriousissue.Theissueofhealthfacilitiesnotbeingabletobuyequipmentintheshorttermmaybeparticularly salient, since if the intervention is “off” the budget cycle for public facility purchaserequisitions, there isa risk that the improvementhappensonly in thenextbudget cycle.However,ourpreliminarysimulationssuggestthatfacilitiescanimprovethescoresconsiderablythroughinexpensiveorlow-costchangesthatrequiregreatereffortbutnotnecessarilygreaterresources(forinstance,standardoperatingprocedures,documentation,colorcodingwaste-binsorcalibrating labequipment).Thesewillshow up as improvements in the checklist, although it is an open question whether this shows up inchangesinactualpractice—somethingwewillbeabletoassessduringthefollow-up.

• Ifthereisnochangeindemand,itcouldbebecause(A)peopledidnotunderstandtheinformationor(B)

demand is not responsive to patient safety.Wewill not be able to say, but we canmitigate this risksomewhatbyconductinganumberof focusgroupsdiscussionspriortodecidingonthepreciseformofthe report cards. Campaigns to publicly disseminated facility regulations at the health facility andclient/customerlevelarecomponentsthatwillrequirecarefulconsideration(includingthematerials,thecommunicationstrategytotargetthedifferentaudiences,andthecommunicationstrategytodeliverthemessages). For instance, theHF-level campaigns are likely to be delivered (among other) in amonthlymeetingthatcountieshavewithallHFs.

8.2 SAMPLESIZECALCULATIONSAs mentioned previously, a key feature of this study is that all the health facilities in the three participatingcounties–Kakamega, Kilifi, and Meru–will be part of the analysis. Hence, we do not have a sample of healthfacilitiesbuttheentirepopulationofhealthfacilitiesinthesecounties:around260markets(clusters),andaround1100health facilities,whichnaturally implies that the resultswillbe representativeofandvalid for these threecounties.Themarketswillberandomizedequallyintothethreearms:eacharmwillconsistofaround87marketsand371healthfacilitiesasthetablebelowshows.

Table1.RandomizationofMarkets

T1Treatment1

(87markets,371healthfacilities)

T2Treatment2

(87markets,371healthfacilities)C

Controlgroup(87markets,371healthfacilities)

MinimumDetectableEffectNOTE: We present again the estimates conducted with the pilot results as we are in the process of finalizingbaseline,afterwhichwewillhavemorecomprehensivedatatoupdatethesefigures.Overall,wefoundmorehealthfacilities during the baseline, and had a high consent rate (94%), which is very favorablewith respect to theseestimates.ThetablebelowpresentspreliminaryestimatesofMDEforthe6mainoutcomesofinterest.SincethenumberofclustersandHFsisfixed,asitcomprisesallthepossibleinthethreecounties,theMDEbecomesdependentontheotherdeterminingvariables.Assumingpowerof80%andconfidenceof95%, theMDEvariesbetween0.19and0.27standarddeviationforallmainoutcomes.Thisrepresentschangesofbetween20%and27%withrespecttothemeanreference.Toourknowledge,thereisnodatatocompareimpactsofsimilarinterventions.However,arandom small-scale control trial in Uganda involving a community-basedmonitoring and accountability systemwith disclosure of public primary health facilities scores led to an increase in the summary HF score of 0.56

16

standard deviation compared to the control facilities, and select utilization and health outcomes improvedbetween33%and46%comparedtothecontrolgroup(BjorkmanandSvenson,2009).

Table2.MinimumDetectableEffect(80%power,5%significancelevel)

Adherence to HF Checklist

with QC and PS Standards

(1)

Adherence to Checklist of

essential recommended

care (2)

Adherence to IPC

practices (3)

Prevalence of unnecessary or harmful

medications (4)

Price per Visit (KES) (5)

Quantity (Monthly patient flow) (6)

Unit Level Health Facility Health Provider Health Provider Patient Health

Facility Health Facility

Variable Mean 0.53 0.40 0.40 0.70 436 1128 Variable SD 0.50 0.49 0.49 0.46 326 1095 MDE (in SD) 0.27 0.22 0.22 0.19 0.27 0.27 MDE (as % of reference mean) 25% 27% 27% 12% 20% 26%

Total number of clusters 250 250 250 250 250 250

Total number of clusters per arm 83 83 83 83 83 83

Average number of facilities per cluster

3 3 3 3 3 3

Total number of HFs 750 750 750 750 750 750

Total number of HFs required for 80% take-up

938 938 938 938 938 938

Number of Patients per HF

- - - 4 - -

Average number of HP per HF

- 2 2 - - -

Notes:Estimatesassume intra-clustercorrelationof0.1;variationexplainedatthemarket levelof0.1;variationexplainedatthe HF/HP level of 0.1; variation explained by a covariate at the market level of 0.1 (e.g., market size). Power of 0.8 andconfidenceof95%areassumedinallcases.(1)KePSIEDataPilot (2014). Score/maximumscore foranequalweighting scoring system from thepilotof theChecklist forSingular/JointInspectionsforPublicandPrivateMedicalInstitutions.(2)BasedonresultsforIndiafromDasetal(2012),adjustedforKenyabasedinon-goingpilot.(3) Based on statistics of Infection, Prevention and Control (IPC) Adherence fromWHO reviews (2008, 2012), including handhygiene,safeinjectionandsterilizationpractices.Sourcesareofdifferentscope.(4)BasedonresultsforIndiafromDasetal(2012)(5)and(6)KePSIEDataPilot(2014)The estimates in the table below are based on conservative assumptions. At least two important additionaladjustmentswillimprovetheprecisionoftheestimatesanddecreasetheMDE:

• Stratificationof theclustersbasedon information fromthebaseline,before the randomization takesplace. This stratification will be based on a few variables that capture part of the between-clustervariationintheoutcomesofinterestandwillhelpwiththeprecisionoftheestimates.

• Likelyhigher take-uprates.WeassumebelowaHFtake-uprateof80%.However,thetreatmentarmsareregulatorytreatmentswhereHFscannotrefuse.HFscanrefusetobeinterviewedsowestillbudgetfornon-responsebutfromthelisting,mostHFswillconsentedtoparticipate(94%).Further,theHFshavean incentive tonot fallbehind,especially considering thatallHFs ineachmarketwillbe subject to thesameregime.

17

9. DATACOLLECTIONWewilluseacombinationofmethodsand instrumentstocollect thedata, includingsurveystohealth facilities,patients and health workers, covert observation of health workers provision of care (through StandardizedPatients),overtobservationofpatient-healthworkerinteractions,andadministrativedata.Fortheproposedquantitativedatacollection,threemainactivitiesareplanned:(i)apilottodevelopandtestallsurveyinstruments(January-June2014);(ii)abaselinedatacollection(January-November2015);and(iii)afollow-up data collection. The follow-up data collection has been proposed to come12months after the interventionstarts.

9.1 QUANTITATIVEINSTRUMENTSFourtypesofquantitativedatacollectionwillbeincludedasfollowing:

1. SurveyDataonQualityofCarePatientSafetyStandards(Checklistandotherindicators):Anassessmentofcompliancewiththeminimumqualityofcareandpatientsafetystandardswillbecollectedthroughsurveystoallhealthfacilitiesinthetreatmentandcomparisongroupsatbaselineandthenatafollow-upaftertheinterventiontakesplace.

2. StandardizedPatientstoMeasureQualityofHealthCareDelivery:Asmentionedpreviously,thisdatacollectionmethodconsistsoftheuseofunannouncedStandardizedPatientswhoarerecruited,andextensivelytrainedtopresentaconsistentcaseofillnesstodiversehealthcareproviders.Thisisconsidereda“goldenstandard,”amongotherreasons,becauseitreducescommonbiasesfromself-reportedmeasuresofqualityofhealthcareprovisionwhenrealpatientsaresurveyed,andalsobecauseitallowsforvalidcomparisonsacrossdifferentfacilities.Unlikeprescriptionauditsandvignettes,whichmeasureclinicalknowledgeandcompetence,theuseofSPsisaninnovativeapproachthathasbeenshowntoaccuratelyassessproviderpractice.Thisisimportant,sinceclinicalpracticecandiffersubstantiallyfromcompetence.ThismethodwasusedinIndiatoexaminequalityofcareforangina,asthmaanddysentery(Dasetal.,2012).Findingsshowlowlevelsofmedicaltraining,frequentincorrectdiagnosesandprescribedtreatments.Resultsfromthisstudyarecontributingtoestablishbenchmarksfortheoverallqualityofcareforspecificcasesandalsoforamorefocusedpolicyapproach.ForStandardizedPatients(SPs),healthcareworkers(HCWs)arerandomlychosen(i.e.,theSPsgotoanyHCWthatisassignedtothem).

3. DirectObservationofPatientSafetyPractices:Thisdatacollectionwilltakeplacethroughdirectobservationoftheinteractionbetweenhealthcareworkers(HCWs)andpatientswhileinconsultationorinthelaboratory.InmostHFs,thereareusuallyoneortwoHCWsinservice.Duringthebaseline,wesplitthetimeofthesurveyors(3to5hoursintotal)betweentheHCWsinservice.ForbigHFs,werandomlychoseupto4HCWstobeobservedduringthe3to5-hourperiod.Directobservationiscurrentlyconsideredthegoldstandardtomeasurecompliancewithinfection,preventionandcontrolstandardssuchashandhygieneandsafetyinjectionpracticesbecausethisistheonlymethodavailablethatallows,forinstance,registeringopportunitiesandactionstoassessthenumberoftimesandappropriatetimingofactionsinthesequenceofcare(TheJointCommission,2013).Thisthenallowsthebuildingofindicatorsthatcanbereplicatedandcomparedacrossdiversetypesofhealthproviders.Thedisadvantageofthismethodispotentialobservationbias,whichresultsfromhealthproviderschangingtheirbehaviorbecausetheyareawarethattheyarebeingobserved(the“HawthorneEffect”).Forthetypeofpracticesselectedforthismodule,theprioraboutthelowlevelofcompliancewiththesepractices,andthetypeofresearchquestionstheseindicatorsintendtoanswer,thereisstrongevidencethattheadvantagesofovertobservationexceedthedisadvantages.Thefinalanalysiswillincludeadiscussionofthepotentialimplicationofobservationbiasinthiscase.

18

4. SurveyDataonPricesandQuantities:Dataonpricesofselecthealthserviceswillcomefromtwo

differentsources.First,fromthecasesofstandardizedpatients.Second,fromarandomsurveyonselectservices,collectedfrompatientsexitingpublicandprivatehealthfacilities.Dataonquantitieswillbecollectedatthehealthfacilitylevel.

9.2 MANAGEMENTOFDATAQUALITYGiventhecomplexityoftheexercise,wewilltakeexceptionalstepstoensurethequalityofthedataandcompleteconfidentialityoftheworktoprotectresearchsubjects.Inadditiontoincludingafull-timefieldcoordinator,anddevelopingamonitoringprotocolwithmultiplechecksduringdatacollection,thequalityofthedatawillbefurtherenhancedthroughthefollowingfiveactivities:

1. Extensivepilotsofdatacollectioninstruments:AlldatacollectioninstrumentswerepilotedbyasurveyfirmandtheWBGteam,including:allstagesofquestionnairedevelopment,logisticplanning,electronicdataentrysystems,andimplementationinthefield.Forinstance,thepilotforthestandardizedpatientinstrumenttoolplace,andthreemonthsofWBGstaffsupportwasallocatedjustforthisexercise.Aspartoftheexercise,theWBGteamparticipatedinthedevelopmentofthecases,thetrainingofstandardizedpatients,andtheimplementationinthefield.An“adverseeventsroster”wasmaintainedtorecordanyadverseeventwiththestandardizedpatientsandnoeventwasreported.

2. AdvisoryGroup:AnadvisorygroupcomposedofKenyanmedicalprofessionalswascontractedtoprovidekeyadviceforthefulltermofthepilot.TheyhelpeddeveloptheStandardizedPatientcases,andalsotoservedasatechnicaladvisorygrouptofinalizetheclinicalcasesappropriatefortheKenyancontext,andtheircorrespondingscripts.

3. TrainingofFieldCoordinator,ResearchAssistants,andInterviewers:IntensivetrainingwillbeconductedbytheWBGteam,andpersonnelofthedatacollectionfirmthroughworkshopsandfieldvisits.Thetrainingwillincludethefollowingcomponents:(i)theoretical,includingthecontentofquestionnairesandclinicalcases;(ii)technical,includingtheuseoftabletsforcompletingelectronicquestionnaires;(iii)classroompractice;and(iv)fieldpractice.Followingthetraining,SPs,interviewers,supervisors,dataeditorsanddatamanagerswillbeevaluatedbasedontheirunderstandingofthequestionnairesandtheirabilitytocorrectlyrecorddatausingthesametestscenariosasusedintheclassroompractice.Thetrainingperiodwillconcludeonlywhenthefieldteamshavedemonstratedmasteryofthedesignatedtasks,anddecisionsastowhichfieldstaffwilltakepartinthedatacollectionwillbemadeonthebasisofthistrainingandevaluation.AlltrainingactivitieswerepilotedbetweenJanuaryandJune2014.

4. Electronicdatacollection:Thiswillallowcheckinginrealtimeandthereforeimmediatecorrectionswithsurveyors.Thedatawillbesavedinasecuredserverwithrestrictedaccess.Onlyauthorizeduserswillbeabletoretrievereports.Protocolswillbeinplaceforaudiorecordingofinterviewsandforusingtheaudiointerviewsforseconddataentrytocrosscheckagainstthefirstdataentryduringelectronicdatacollection.ThesecondentrywillbeconductedonasubsetoftheinterviewsasdeterminedbytheIFC/WBteam.ProcedureswillalsobespecifiedforidentifyingandreconcilingmismatchesbetweenfirstandsecondentryandforidentifyingandfixingerrorsinsurveyIDs.Theprotocolwillalsoincludeastrategyforensuringpromptfeedbacktofieldteamsonanyerrorsencounteredandfororganizingandloggingquestionnairesonceentered.

5. ExternalQualityDataCollectionAuditing:Allinstrumentsofdatacollectionwillundergoexternalauditingtoverifyquality.

We believe that the combination of considerable advance planning, full team participation in the pilots anddevelopmentof themodules andawell-functioningelectronicdataentry systemwill guaranteeextremelyhigh

19

qualitydata.Thesearestrategiesthatwehavefollowedbefore,andthedatafromourpreviousworkisuniversallyregardedasextremelyhighquality.

9.3 ETHICALISSUESExtensive care will be taken to ensure that this study accounts for the rights and welfare of all participantsinvolved. The research team already has obtained Human Subjects Certification through the U.S. NationalInstitutes of Health. Enumerators will be chosen from a pool of experienced surveyors and receive intensivetrainingoninformedconsentandconfidentialityprocedures.The research team, ledby theLeadResearcher JishnuDas, submitted the initial researchprotocol for reviewtoAMREFESRConJanuary9,2014.FollowingcommentsfromtheESRCtotheinitialapplication,theresearchteamresponded to thecommentson January13,2014and this revisedprotocolwasapprovedby theCommitteeonJanuary21,2014foraperiodof1year.ArenewalapplicationwassubmittedafterthisperiodandapprovalwasgrantedbytheCommitteeonJanuary27,2015foranadditionalperiodof1year.StandardizedPatientsBynature,theuseofStandardizedPatientsrequirescarefulethicalconsideration.Wehaveextensiveexperienceinforeseeing,managing and limiting any potential risks to both participants andmedical providers. The first stepbeginsinrealizingthelimitationsofthetypesofcasesthatcanbeusedinagivencontext.Bynecessity,allcasesinvolvingchildhoodillnessesandinvasiveexamsareexcludedas,insomecontexts,eventheuseofathermometercancauseahealthrisktotheSP.AllfinalSPcasesforthisstudywillbedevelopedinclosecollaborationwiththetechnical advisory group,which includes a panel ofmedical professionals, to develop only those cases that aremostrelevantandsafeinKenya.Further,thisstudybuildsonsimilarstudiesusingSPsinmiddle-andlow-incomecountriesandthelessonslearnedfromthosestudies.AkeyconcernwithusingSPsisthepotentialthattheymaybedetectedoverthecourseofthestudy.InIndia,detectionrateswerelessthanonepercentduringthepilotphase,andnocasesweredetectedinthemainsurvey.InourpilotinKenya,nocasesweredetected.BothinKenyaandIndia,duringapproximately150hoursoftraining,SPswerecoachedbyaprofessionalstandardizedpatienttrainer,doctors,andananthropologistto consistently portray the emotional, physical, and psychosocial aspects of the case and to accurately recallinteractionswithproviders.AllSPswerealsothoroughlytrainedtomakeplausibleexcusestoavoidthermometers,needles,andpelvicexamsandtohidemedicinesthatdoctorsrequestedthemtoingestintheclinic.ThematerialsfromtheIndiastudywereusedinsubstantialtechnicaltransferfortheKenyastudy,includingsimilarstrategiestoidentifyandmitigatepotentialharm.These activities were part of the pilot to develop clinical cases appropriate for the Kenyan context. The pilotstartedwiththeSPcasesalreadydevelopedfor India(andtested inChinabasedonthe Indian instruments), forangina,asthmaanddysentery(Dasetal.,2012).Allthestepsdescribedpreviouslybelongedtotheidentificationstage, with emphasis on documenting and reporting what worked and what did not work. These insightsconstitutedthebasisofthefine-tuningofthefinalinstrumentstocollectdataforthestudy.ConsentAllsubjectswillbeinformedofthestudydesignandconsentwillbeobtained.Healthfacilitiesareinformeduptoonemonth inadvanceof surveyactivitiesandsignedconsent isobtained forparticipation in the study. For thesurveys conducted, study participants will either be health care providers, health facility staff, SPs, or patientsuponexiting a facility. Participantswill be informed that thepurposeof the study is to provide informationonways to improvepatient safetyandqualityof care inKenya, andwill begivena copyof study informationandconsentforms,aswellascontactinformationforanyquestionsorconcernsregardingtheirparticipationandtheresultsofthestudy.ForSPs,providerswillbeinformedofthestudydesignandgivenatimelineduringwhichtheycanexpect tobevisitedbyoneormoreSPs,e.g.4 to6weeks.For thepatientsafetysurveys,enumeratorswillhaveamedicaland/ornursingbackgroundandbeattentiveandresponsivetotheinterviewwiththepersonnelof

20

healthfacilities.ConfidentialityThe informationcollectedwill remainsecureandconfidential.Participantswillbeassignedunique identificationnumbers,andsurveyformswillbeimmediatelyencryptedwhencompletedbyenumeratorsonelectronicdevices.All data will be transported as encrypted files and stored on a private server with restricted access. Anyinformationrelatedtothestudywillbestoredinsecurelocations,andonlytheresearchteamanddatastafffromthe survey firm will have access to the study data. Identifying details collected as a part of the study will beremoved during data cleaning, with only core research team having secure code for accessing personalinformation,anddatawillbeanalyzedanonymouslytoensurethatnoparticipantcanbepersonallyidentifiedinanypubliclyreleaseddata.WB/IFC staff will also perform visits at random to sampled locations to ensure that questionnaires areadministered properly: informed consent has been taken; respondent is given the option of not responding tocertainquestions;andenumeratorsareaskingquestionsinanappropriatewaythatbothensurestheaccuracyofdataandpreservesthedignityofrespondents.

9.4 QUALITATIVEINSTRUMENTSQualitative data collection will include: (i) activities with an anthropologist, who will work closely with theevaluationteamtoprovideinsightontheprocessesandproductstobeputinplaceasapartofthebroaderstudy;(ii)focusgroupdiscussionscarriedoutwithhealthfacilitiespersonneltocollectinformationonchallengesduringthe implementation of the treatment arms, and (iii) focus group discussionswith doctors and SPs through thedevelopmentandfine-tuningoftheSPsmodule.

9.5 IEIMPLEMENTATIONMONITORINGSYSTEMAmonitoringsystemwillbeinplacetotracktheimplementationoftheintervention.Thiswillbefacilitatedbytheuse of electronic devices to conduct the inspections in the two treatment arms. All other activitieswill also betracked to assess the adherence to theplan andprotocols, and investigate the characteristics of the treatmentdelivered.Dataonthecostoftheinterventionofeachtreatmentarmwillbemonitoredclosely.Aplanofalloftheresources required and the cost of the intervention will be part of a planning exercise to be held inSeptember/October2014withtheparticipantcountiesandtheMOH.Followingthatexercise,theplanningfortheprocurementoftheservicesrequiredforthearmswillbedevelopedandallthecostswillberecordedaspartoftheimplementationmonitoringsystem.

10. DATAPROCESSINGANDANALYSIS

10.1 DATACODING,ENTRY,ANDEDITINGHealthfacilityandhealthcareworkerleveldatawillbecollectedthroughinterviewsandobservation;standardizedpatients and patients’ data will be collected through observation and face-to-face interviews using electronicquestionnaires.Thisallowsdataentry tobedoneautomatically in the field, and information tobeavailable foranalysis immediatelyafter surveying. This implies thatmostdata codingand labellingwill takeplaceduring thedesignofthequestionnaires.ForthePracticeofPatientSafetythatwillbepaper-basedusingdirectobservation,protocolsfordata-entryhavebeendeveloped.Data entry protocols developed aim to generate variables that correspond clearly and logically to the questionlabels used in the questionnaires alongside a coding strategy that maintains consistent, unique identifiers formatchingdata.Therewillalsobeproceduresinplacetobackupandcompiledataregularly(bythedatamanager)andconductspot-check,cleaningandarchivingofdata.Thisincludesidentifyingredundantobservations,ensuringall componentsarecorrectly linkedanddatasets canbemergedcleanly, completing inventoryofdatabase,andcompletingelectronicarchivingofquestionnaires.

21

Final databasesof all completeddatawill be compiledwith the appropriate unique identifiers (health facilities,health care workers, SPs, patients), clearly labeled variables and response options, and be supported bycorrespondingdictionaries.Thiswillalso include final responseratesandreplacementofhealth facilities,SPsordoctors from the original sample. The control checks and protocols developed for all coding, data entry andediting,willbereportedinatechnicalannex.Data quality controls will be in place at two levels: internally and independently. The survey firmwill conductinternal quality checks, including validation of error rates as all data is processed, back-checking 10% of theinterviewers’workbysupervisors,andaddressingdata inconsistencies. Independently,alldatawillbehostedbythe research team to ensure independent data-entry checks atmultiple levels. The research teamwill provideoversight and closelymonitorquality control at each stageof the researchprocessby reviewingandapprovingdeveloped study materials and tools, accompanying and witnessing field interviews, as well as reviewing andapprovingadataanalysisplan.

10.2 MODELSPECIFICATIONFORQUANTITATIVEDATAANALYSISEquation(1)showsthegeneralformulatoestimatetheimpactsoftheprogram.Asimplemeandifferenceoftheoutcomesofinterestbetweenthetreatmentandthecomparisongroups(orbetweenbothtreatmentgroups)willsufficetoestimatetheimpactoftheprogramunderastandardrandomization.However,sincetherandomizationisconductedatthelocalmarketlevel,equation(1)alsoincludessomeadditionalcontrols, incasethetreatmentandcontrolgroupsarenotbalancedatthehealthfacilitylevel:

(1)

whereYitindicatesanyoftheoutcomesofinterest(e.g.,compliancewiththeminimumqualityofcareandpatientsafety standards) for facility i at time t (t=0,1) with t=0marking thebaseline and t=1 the follow-up; Ti is thetreatment indicator thatequalsone for facilitiesbelonging to the treatmentgroupsandzero for facilities in thecontrolgroup;Xitareasetoftime-variablecharacteristicstocontrolfor,particularly,thosethatmaydifferentiatetreatmentandcontrolgroupsatbaseline,andcouldbecorrelatedtoreceivingtreatmentandtotheoutcomesofinterest;δ is theparameterof interestand represents the impactestimate;andeit representsa setof randomtime-varyingunobservedcharacteristics.Thisequationwillbeestimatedthroughregressionanalysiswithstandarderrorsclusteredatthelocalmarketlevel.The treatment estimatorswill correspond to thepopulation treatment effects for the three counties aswewillhave thepopulationandnon-response isexpected tobe low. In thecase therearedropoutswewillbeable toestimateintent-to-treatestimators.ToaddressmultiplehypothesistestingwewillusetheBonferronicorrection.Variables at themarket andhealth facility levelwill be collected toaddresspotential selectiondue toattrition,non-response,take-uprateissuesthatariseforcorrelationbetweenthetreatmentindicatorandtheoutcomesofinterest.

11. STUDYLIMITATIONSANDRISKSThemajorchallengeofthisstudystemsfromsignificantcoordinationrequiredtodefineandimplementthediverseelementsofthetreatmentarms,whichareassumedtoimpactpatientsafetyandqualityofcare.Asmentionedpreviously,thedesignofthesearmshasbeenandwillcontinuetobeajoineffortinvolvingallthestakeholders(theMOHandtheregulatoryboardsandcouncils,representativesoftheprivatehealthcaresector,andtheIFC/WBimpactevaluationteam).Currently,therearediversegroupsworkingonelementsthatwillconstituteinputsforthisstudy,someofwhichareatthesametimealignedwiththeMOHpolicygoalsandstrategy.Therefore,awellplanningsystem,andalong-termdialogueandcommitmentofallkeystakeholdersrepresentthecornerstonesofthestrategywehaveinplacetoaddressthisissue.

Yit = a +dTi + bXit +e it

22

12. IEMANAGEMENT

12.1 EVALUATIONTEAMANDMAINCOUNTERPARTSThe IEteammembersareGuadalupeBedoya(DECIE) JorgeCoarasa(GTCDR), JishnuDas (DECHD),AmyDolinger(GTCDR),AnaGoicoechea(GTCDR)andNjeriMwaura(GTCDR).TheteamisadvisedbyKhamaRogo(GTCDR)andsupportedbyFrankWafula(GTCDR),PamelaNjuguna(GTCDR),BenjaminDaniels(DECHD),TatianaZarate(DECIE),andGarimaSharma(DECIE).Below is the list of Government and Private sector Counterpart Representatives, and other local collaborators,followedbythebiosoftheIEteam.Thesegroupsconstitutethecoreparticipantsandcollaboratorscontributingtothedevelopmentofalltheinputsandmaterialfortheimplementationandanalysisofthestudy.

Table3.LocalCollaboratorsRole Name Organization

LocalCounterpartsGovernmentandPrivatesectorCounterpartRepresentatives

Dr.PacificaOnyancha QualityAssuranceandRegulations

Dr.CharlesKandie DivisionofHealthStandardsandQualityAssurance,MinistryofHealth

Dr.MaryWangai DivisionofHealthStandardsandQualityAssurance,MinistryofHealth

Mr.JohnKabanya DivisionofHealthStandardsandQualityAssurance,MinistryofHealth

Dr.DavidOmondiOkeyo NutritionCouncilDr.KipkerichKoskei PharmacyandPoisonsBoardMs.MaryNyamboki NursingCouncilofKenyaMr.PeterMwangiKagiri RadiationProtectionBoardDr.AmitThakker KenyaHealthcareFederation(KHF)Mr.MicahKisoo ClinicalOfficersCouncil

Mr.DanielYumbya Medical Practitioners and DentistsBoard

Mr.PatrickKisabei Kenya Medical Laboratory TechniciansandTechnologistsBoard

Dr.KephaOmbacho PublicHealthTechnologistsCouncilStandardizedPatients

AdvisoryCommittee Dr.CharlesKandie MOHJohnKabanya MOHDr.FatmahAbdallah MOHSarahChuchu MOHRahabMaina MOHDr.IzaqOdong MOHDr.RachelNyamai MOHDr.NjeriMwaura IFCProf.RuthNduati UniversityofNairobi

DrugTestingProtocolandPilot Dr.BernardOlayoMD,MPH IFCChecklistandRegulatoryFramework

TechnicalWorkingGroup Dr.LoiseMutai MPDBMrs.MaryNyamboki NCKMr.PeterSewe COCMr.PeterKagiri RPB

23

Role Name OrganizationDr.AmitThakker KHFMrs.BettySamburu NCMr.JohnKabanya MOH/DHSQRMrs.MildredWasike PublicHealthMrs.IdahOmbura KMLTTB(TBC)Dr.JamesOwuor PPB(TBC)

KePSIETeamBios

Jishnu Das (Lead Researcher) is a Lead Economist in the Development Economics Research Group (HumanDevelopment and Public Services Team) at The World Bank and a Visiting Scholar at The Center for PolicyResearch,NewDelhi.HereceivedhisPh.D.fromHarvardUniversityinEconomicsin2001.SincejoiningtheWorldBank,Jishnuhasworkedonissuesrelatedtothedeliveryofbasicservices,particularlyhealthandeducation.Hiswork draws upon data collected in Zambia (education), India (health and education), Pakistan (education) andParaguay (health). His recent research focuses onthe quality of health care(Journal of Economic Perspectives,Health Affairs, andthe Journal of Development Economics); correlates of mental health (World Bank EconomicReview, forthcoming,andSocialScienceandMedicine); the linkbetweenteacherabsenteeismandstudenttest-scores (Journal of Human Resources); and the structure of educational provision in Pakistan (ComparativeEducationReview).In2006,hisworkonreligiouseducationinPakistanreceivedtheGeorgeBeredayAwardfromthe Comparative and International Education Society. He also works on natural disasters in the context of anearthquakethathitNorthernIndiaandPakistaninOctober2005.Heco-foundedthewebsitewww.risepak.comtohelpcoordinate relief in theaftermathof thequake.Thewebsitewasawarded theStockholmChallengeAward(2006)forthebestICTprojectinthepublicadministrationcategory.Guadalupe Bedoya (Co-Principal Investigator) is an Economist in the Development Economics Research Group(DevelopmentImpactEvaluationUnit,DIME)attheWorldBank.ShereceivedaMaster’sdegreeinEconomicsandPublicPolicyfromPrincetonUniversityin2007,andaPh.D.inPublicPolicyfromtheUniversityofChicagoin2011.Guadalupehasover8 yearsofworkexperience ineconomicanalysis,policydesign, and impactevaluation.Hercurrentworkincludesdesigning,analyzing,andsupervisingdiverseimpactevaluationsacrossmultiplesectorsandcountries, aiming to understandwhat works, and especiallywhy interventionsmay ormay notwork, with theoverarchinggoalofbuildingknowledgetoimprovepolicyeffectiveness.BeforejoiningDIME,GuadalupeworkedattheMinistryofFinanceandtheDepartmentofEconomicPlanninginColombia;theInter-AmericanDevelopmentBankintheU.S.,andalsoconductedfieldworkindiversemicrofinanceorganizationsinIndia.Jorge Coarasa (Health in Africa Expert andManagement Support) is a Senior Economistwith theWorld BankGroup Investment Climate Department where he focuses on private participation in health and education. HestartedwiththeWBGattheIFCofficeinNairobiwherehemanagedtheHealthinAfricaInitiativeAdvisoryServicesprojectsinKenya,NigeriaandTanzania.BeforejoiningIFChewasaDeputyDirectorGeneralwiththeMinistryofSocialDevelopmentoftheMexicanGovernment.Heoversawthestrategicplanningof18nationalsocialprogramsaimedat increasingaccess tohealth,educationandsocialprotection for thepoor.Hehasbeenaconsultantonhuman development issues with the World Bank and the UN, a teacher at a rural teacher training college inMozambiqueandamanagementconsultantwithDeloitteMr.CoarasaHoldsgraduatedegreesinEconomicsfromITAMinMexicoCity,InternationalRelationsfromUniversidadComplutensedeMadridandPublicAdministrationfromHarvardUniversity.BenjaminDaniels (ResearchAssistant)worksat theWorldBank’sDevelopmentEconomicsResearchGroup.Hejoined the World Bank after completing a degree in economics at Pomona College in 2011, investigatinginequalities in children's outcomes after the 2005 Pakistan earthquake. He received his master's degree inEconomicsandPhilosophyfromtheLondonSchoolofEconomicsin2013,andworkedasresearchassistantattheGranthamInstitutethere,studyingthesocioeconomicpatternsofdomesticsolarsubsidies.HehassincereturnedtotheWorldBankandplanstoapplytoPh.D.programsin2014.

24

AmyDolinger (FieldCoordinator)works in the IFC’s InvestmentClimategroup.Shegraduated fromthePaulH.NitzeSchoolofAdvancedInternationalStudiesatJohnsHopkinsUniversitywithaMaster’sdegreeininternationaldevelopmentandeconomics.ShehasalsoworkedasaResearchandPolicySummerAssociatewithInnovationsforPovertyAction(IPA).Ana Goicoechea (Impact Evaluation ProjectManager) is a Senior Private Sector Development Specialist in theWorldBankGroupInvestmentClimateDepartment.ShecurrentlyleadstheInvestmentClimateImpactProgram,managingaclusterofabout10impactevaluationsthataimatassessingtheimpactsofvariousregulatoryreformsthatarerelevantforprivatesectorgrowth.Shehasover10yearsofexperienceintheWorldBankGroupwithafocuson impact evaluation, results-basedmanagement, andpolicy analysis on infrastructure andprivate sectordevelopment. She has an educational background in economics and international development and holds aMastersinPublicPolicyfromtheUniversityofMaryland.RachealNjeriMwaura(KenyaHealth inAfricaProjectTeamLeader) recentlyjoinedtheWorldBankGroupandmanages the Health in Africa Initiative Advisory services project in Kenya. Prior to joining IFC, she was aprogrammemanageratAfricanMedicalResearchFoundation(AMREF)–whereshemanagedtwolargenationalprogramsfocusingonstrengtheningthenationalHIV&AIDSM&Eframeworkandhealthcareprovidercapacityandstrengtheningorganizationalandtechnicalcapacityofcommunitybasedorganizationsthroughagrantmakingandcapacity building programme. She was also the gender focal point at AMREF Kenya. She also worked at theMinistryofhealth inKenya, inthesectormonitoringandplanningdepartmentatnational levelandasaDistrictMedical Officer of Health in charge of all health service delivery and provision where working jointly withstakeholders she significantly improved the health outcome indicators of the district. She is also the nationalassistantsecretaryoftheKenyaNationalMedicalWomen’sAssociation.DrMwauraholdsamedicaldegreefromMakerereUniversity,KampalaUgandaandapostgraduatediplomaandmastersofScience inPublichealthfromLondonSchoolofHygieneandTropicalMedicine–UniversityofLondon.KhamaRogo(KenyaHealthinAfrica)isLeadHealthSectorSpecialistwiththeWorldBankandHeadoftheWorldBank Group's Health in Africa Initiative. Prior to his WB career, he taught Obstetrics and Gynecology at theUniversityofNairobi,beforebecomingtheVicePresidentofMedicalAffairsAfricaforIpas.AnativeofKenya,ProfRogoreceivedhisMDandM.MEDfromtheUniversityofNairobiandearnedaFellowshipandPhDinGynecologicOncology in Sweden. A prominent advocate and global authority on reproductive health issues, he is a visitingprofessoratseveraluniversitiesandauthorofover100papersandbookchapters. HehasbeenaconsultanttoWHO, UNFPA, UNICEF, USAID, and DFID, and advisor to many other bilateral and unilateral internationalorganizations. Prof. Rogo is past president of the Kenya Medical Association and Kenya Obstetrical andGynecologicalSociety.HewasalsothechairmanofKenya'sNationalCouncilforPopulationandDevelopment,andserved on the Gender Advisory Panel of WHO, the Advisory Committee of the David and Lucile PackardFoundation,andtheboardoftheCenterforAfricanFamilyStudies.HeiscurrentlyontheboardofINTRAHEALTH,amongotherresponsibilities.

12.2 WORKPLANANDDELIVERABLES

Table4.Milestones,Deliverables,andEstimatedTimelineMilestones Deliverables CompletionDate

Methodologynote Note November15,2012PeerreviewedConceptNote ConceptNote July2013

Datacollectionplanandpilot TORsQuestionnaires June31,2014

PilotDataAnalysisandAdjustedInstruments

PilotReportsandAdjustedInstruments August30,2014

Datacollection(Baseline) CleaneddataDictionaries December30,2015

25

Firstdataanalysis

PresentationDatafileDofilesBaselinereport

January31,2016

ImplementationPlanforintervention Rolloutplan January31,2016

Implementationofinterventionalignedtoevaluation

Monitoringreportsverifyingtreatmentandcontrolstatus MonthlyfromMarch2016

Follow-updatacollectionplan TORsQuestionnaire February28,2017

Datacollection(Follow-up) CleaneddataDictionaries February28,2018

Finalreportandpolicynotes

TechnicalnotePolicynoteDatafileDofiles

April30,2018

Disseminationoffindings Presentations May-December,2018

12.3 BUDGETThetotalestimatedbudgetfortheIEisUSD$2.6millionincludingallthestafftimeoftheIEteam.Sofarwehavesecuredabout75percentof thisbudget including resources fromtheproject,partof the IE teamtimeby theirunits,andotherexternalfunding.Belowisthebudgetbycategory.

Table5.TotalEstimatedBudgetperCategoryCategory USD(000) %

DataCollection 1,640 63%Staff 600 23%Travel 200 8%Workshops 25 1%Non-datacollectionSTC/STT/ETC 139 5%Total 2,604 100%

13. PLANFORUSINGDATAANDEVIDENCEFROMTHESTUDYTherearefoursetsofaudienceswhowouldbenefit fromthisstudy.Theprimaryaudience includestherelevantgovernment agencies in Kenyawhowoulduse the evidence generated from the study to directly informpolicydecisions regarding systems that improvepatient safetyandqualityof care. Inaddition, theconsumer signalingmechanism thatwouldbedevelopedaspartof this studywouldalso inform thegeneralKenyanpublic as theychoose a health facility. Besides the primary clients, the study results would also inform policy makers andregulatory authorities in other countries who are also looking to understand the best way to improve patientsafety and quality of care in their respective countries. And finally, the study will also inform the largerdevelopment community supporting programs to improve health systems around the world, as well as thoselookingatgovernancerelatedissues,particularlycenteredoninspectionprocesses.Asmentionedpreviously,thisstudyhasbenefittedimmenselyfromthelong-termrelationshipthathasbeenbuiltwith the Kenyan government and the private healthcare sector in the country as part of the Health in Africa

26

Initiative(HIA),afive-yearprogramestablishedbytheWorldBankGroupin2009.TheHIAKenyaproject,andnowthe impactevaluation teamhavebeenproviding technical support to thegovernment. In the study, stepshavebeen taken to ensure that the policymakers, stakeholders and researchers are engaged throughout the studyperiod,sothatthestudyremainsrelevantfromapolicyperspective.Aspartofthestudy,severalworkinggroupswith representation from all stakeholders have been formed and workshops with participation from allstakeholdersconvened.Theworkshopsandworkinggroupsallowforconsensusbuildingandsettingupaconstantfeedbackmechanism,asbysharingprojectoutcomesandfine-tuningthestudybasedonfeedback,theteamwillensurethattheimpactevaluationisdirectlylinkedtopolicyoutcomes.Giventhecomplexityofthestudy,carehasbeentakentoensurethattheintermediateandfinaloutcomesfromthe study are shared with the relevant audiences in the best possible way. The dissemination strategy thusdesignediscenteredonthreepillars:

1. Preparation of Reports/Papers and Briefs: TheWorld BankGroup teamwill prepare technical reportswithdataanalysisaswellaspolicybriefsfornon-technicalaudiencesafterthecompletionofbaselineandfollow-up data collection. These reports and briefswill be published in both, electronic and hard copyform.

2. Workshops:11The team will organize national workshops following the completion of major reports(baselineandend-line)withtheparticipationoftheMOH,theregulatoryboards,theprivatesectorandotherstakeholderstosharetheresults.Inadditiontoprovidingamechanismfordisseminatingfindings,theseworkshopswillserveasaplatformforconsensusbuilding.Besidesthis,theresultswillbepresentedat international forums (seminars, conferences, WB workshops), and internal global practices. Thedissemination strategy will use the local network and the international network that starts with theHealthinAfricaprojectthatisongoingin8countriesinAfrica.

3. Media dissemination: Results stories as well as information campaigns to inform consumers in

participatingcountiesaboutpatientsafetywillbedistributedthroughthemedia.

11Tworelevantworkshops recently tookplace. InNovember2012,client representatives, researchers,andWBGoperationalstaffgatheredforaweekinParistobetterunderstandimpactevaluationconceptsandstartdiscussionsaroundKePSIEdesigninaworkshoporganizedbytheInvestmentClimateImpactProgramincollaborationwithDIME.InOctober2013,alargergroupofstakeholders, includingtheeightregulatoryboardsandtheMOH,gathered inNairobiandagreedonpursuingfurthertheimpactevaluationtomakepolicydecisionsoninspectionreformsinordertomaximizepatientsafetyandqualityofcare.

27

14. REFERENCESAndrabi,Tahir,Das, JishnuandKhwaja,AsimIjaz (2013).ReportCards:The ImpactofProvidingSchoolandChild

Test Scores on EducationalMarkets. Submitted. Electronic document, retrieved on July 28, 2014 from:http://www.povertyactionlab.org/publication/report-cards-impact-providing-school-and-child-test-scores-educational-markets

Barnes, Jeff, O’Hanlon, B., Feeley III, F., McKeon, K., Gitonga, N., Decker, C. (2010). Private Health SectorAssessment in Kenya.World Bank,Working Paper No. 193. Electronic document, retrieved on July 28,2014from:http://pdf.usaid.gov/pdf_docs/PNADS739.pdf.

Besley,TimothyandPersson,Torsten(2007).TheOriginsofStateCapacity:PropertyRights,Taxation,andPolitics.TheNationalBureauofEconomicResearch,WorkingPaperNo.13028.Electronicdocument,retrievedonJuly28,2014from:http://www.nber.org/papers/w13028.pdf.

Bjorkman,MartinaandSvenson,Jakob(2009).PowertothePeople:EvidencefromaRandomziedFieldExperimenton Community-Based Monitoring in Uganda. The Quarterly Journal of Economics, 124(2): 735-69.Electronic document, retrieved on July 28, 2014 from:http://qje.oxfordjournals.org/content/124/2/735.full.pdf+html.

Das,Jishnu,Holla,Alaka,DasVeena,Mohanan,Manoj,Tabak,Diana,andChan,Brian.(2012)“InUrbanandruralIndia, A Standardized Patient Study Showed Low Levels of Provider Training and Huge Quality Gaps.”Health Affairs. December. Electronic document, retrieved on July 28, 2014 from:http://content.healthaffairs.org/content/31/12/2774.full.pdf+html.

Duflo, Esther, Hanna, Rema and Ryan, Stephen P (2012). IncentivesWork:Getting Teachers to Come to School.American Economic Review 102(4): 1241-78. Electronic document, retrieved on July 28, 2014 from:http://pubs.aeaweb.org/doi/pdfplus/10.1257/aer.102.4.1241.

WorldHealthOrganization(2008)“RevisedInjectionSafetyAssessmentTool.ToolC–Revised.”Electronicdocument,retrievedonFebruary14,2014from:http://www.who.int/injection_safety/Injection_safety_final-web.pdf?ua=1.

WorldHealthOrganization(2008)“HandHygieneinOutpatientandLong-termCareFacilities.”Electronic

document,retrievedonFebruary18,2014from:http://apps.who.int/iris/bitstream/10665/78060/1/9789241503372_eng.pdf?ua=1.WorldHealthOrganization(2010)“BestPracticesforInjectionsandRelatedProceduresToolkit.”Electronic

document,retrievedonMarch6,2014from:http://whqlibdoc.who.int/publications/2010/9789241599252_eng.pdf

WorldHealthOrganization(2012)“SaferPrimaryCare:AGlobalChallenge.”Electronicdocument,retrievedon

March6,2014from:http://www.who.int/patientsafety/summary_report_of_primary_care_consultation.pdf

Jin,GingerZheandPhillip Leslie (2003).TheEffectof InformationonProductQuality:Evidence fromRestaurant

HygieneGradeCards.QuarterlyJournalofEconomics118(2):409-451.Electronicdocument,retrievedonJuly28,2014from:http://qje.oxfordjournals.org/content/118/2/409.full.pdf+html.

KohnLT,CorriganJM,DonaldsonMS,editors(1999).Toerrishuman:buildingasaferhealthsystem.Washington,

DC:NationalAcademyPress,InstituteofMedicine.Electronicdocument,retrievedonJuly28,2014from:http://www.fluentmedical.com/docs/To_Err_Is_Human.pdf.

28

Leape, Lucian L (1994). Error inMedicine. The Journal of the AmericanMedical Association, 272(23): 1851-57.Electronic document, retrieved on July 28, 2014 from:http://hospitalmedicine.ucsf.edu/improve/literature/error_in_medicine_leape_jama.pdf.

Leslie,PhillipandJin,GingerZhe(2009).ReputationIncentivesforRestaurantHygiene.AmericanEconomicJournal

(Microeconomics) 1(1): 237-67. Electronic document, retrieved on July 28, 2014 from:http://pubs.aeaweb.org/doi/pdfplus/10.1257/mic.1.1.237.

Muralidharan,Karthik,Niehaus,PaulandSukhtankar,Sandip(2014).PaymentsInfrastructureandthePerformanceof Public Programs: Evidence from Biometric Smartcards in India. The National Bureau of EconomicResearch, Working Paper No. 19999. Electronic document, retrieved on July 28, 2014 from:http://www.nber.org/papers/w19999.pdf.

Sylvia,Sean,Shi,Yaojiang,Xue,Hao,Tian,Xin,Wang,Huan,Liu,Qingmei,Medina,AlexisandRozelle,Scott(2014).Survey Using Incognito Standardized Patients Shows Poor Quality Care in China’s Rural Clinics. HealthPolicy and Planning 29(4): 1-12. Electronic document, retrieved on July 28, 2014 from:http://www.ncbi.nlm.nih.gov/pubmed/24653216.

The JointCommission (2013). “AccreditationStandards.”Electronicdocument, retrievedon July28,2014 from:www.jointcommission.org/standards.

TheWorldBank.(2011).HealthyPartnerships:HowGovernmentsCanEngagethePrivateSectortoImproveHealthin Africa. Electronic document, retrieved on July 28, 2014 from:https://www.wbginvestmentclimate.org/advisory-services/health/upload/Healthy-Partnerships_Full-Rpt-bkmarks-2.pdf.

Olken, Benjamin A (2007).Monitoring Corruption: Evidence from a Field Experiment in Indonesia. Journal ofPolitical Economy 115(2): 200-249. Electronic document, retrieved on July 28, 2014 from:http://unpan1.un.org/intradoc/groups/public/documents/un-dpadm/unpan041963.pdf.

Zamboni, Yves and Litschig, Stephan (2013). Audit Risk and Rent Extraction: Evidence from a RandomizedEvaluation in Brazil. Barcelona Graduate School of Economics, Working Paper No. 554. Electronicdocument,retrievedonJuly28,2014from:http://www.econ.upf.edu/docs/papers/downloads/1270.pdf.

Datasets

IFC-WHOKenyaMinimumPatientSafetyStandardsComplianceSurvey(2012)

NationalQualityControlLaboratories,2005