Meeting Report

Perspectives on Performance-Based Incentives to Improve Quality of Maternal Newborn Health Care in Low Resource Settings: Launch of a technical working group

January 16-17, 2014

2

Table of Contents: Meeting Report

Technical Working Group Launch January 16-17, 2014

World Bank Offices – Washington, DC Acronyms ...................................................................................................................................................... 3

I. Perspectives on Performance-Based Incentives to Improve Quality of Maternal Newborn Health

Care: Day One ............................................................................................................................................... 4

Introduction and Overview of Session ...................................................................................................... 4

Perspectives on PBI and QoC - Opportunities and Challenges ................................................................ 4

Issues, Approaches, and Current Thinking on the Development of PBI/QoC Indicators .......................... 7

Operationalizing Measurement for PBI/QoC: Successes and Challenges................................................. 9

Managing Change Processes to Facilitate Supportive Contexts for PBI/QoC ......................................... 10

Closing ..................................................................................................................................................... 11

II. Perspectives on Performance-Based Incentives to Improve Quality of Maternal Newborn Health

Care: Day Two ............................................................................................................................................. 13

Key Lessons from Day One ...................................................................................................................... 13

Plenary Discussion on Group Work......................................................................................................... 13

Closing ..................................................................................................................................................... 15

III. Annexes ........................................................................................................................................... 16

1. Agenda ................................................................................................................................................ 16

2. Participant List .................................................................................................................................... 18

3. Next steps identified out of the workshop ......................................................................................... 19

3

ACRONYMS

ANC Antenatal Care

BCC Behavior Change Communication

HMIS Health Management Information System

HRITF Health Results Innovation Trust Fund

HSS Health Systems Strengthening

MCH Maternal Child Health

MNCH Maternal Newborn and Child Health

PBF Performance Based Financing

PBI Performance Based Incentives

PEPFAR President’s Emergency Fund for AIDS Relief

PNC Post-Natal Care

QoC Quality of Care

QI Quality Improvement

RBF Results Based Financing

SNL Saving Newborn Lives

SSA Sub-Saharan Africa

TRAction Translating Research into Action project

USAID US Agency for International Development

WHO World Health Organization

4

I. PERSPECTIVES ON PERFORMANCE-BASED INCENTIVES TO IMPROVE QUALITY OF MATERNAL NEWBORN HEALTH CARE: DAY ONE

Introduction and Overview of Session Key Presenters: Monique Vledder and Neal Brandes

Notes:

Implementers of Performance-Based Finance (PBF) programs are often criticized for not

collaborating more effectively

This meeting is an opportunity to offer collaborative solutions and next steps

While PBF programs often focus on structural changes, different communities have an

opportunity to change the design to improve clinical quality

This meeting brings together two communities: Quality of Care (QoC) and PBF to

discuss:

o Differences in perspective;

o Difference in language/terminology; and

o Opportunities for collaboration

Sara Riese initiated introductions of the meeting’s participants and gave an overview of

the day’s agenda

Question to consider throughout the day: what are some of the approaches in current

thinking in indicators in QoC?

Perspectives on PBI and QoC - Opportunities and Challenges Facilitator: Sebastian Bauhoff

Presenter: Kathleen Hill - Performance-Based Incentives & Quality of Maternal Newborn Care

in Low Resource Settings

There is a need to find common language in approaches and among stakeholders for

the different but intersecting worlds of PBF, MNCH and QoC.

Traditionally Performance-Based Incentives (PBI) incentivized “units of care delivery” but

the sector is in transition to incentivizing QoC

The MNCH and QoC communities are also in conversations about “best” measures of

MNCH QoC

June 2012 meeting summary in Geneva with World Health Organization (WHO):

o The group of experts explored quality measures

o Review of the evidence showed that it was inconclusive whether there was an

effect for financial incentives in outcomes and processes

o PBI was found to be most effective in interventions under provider control

o Indicators were weak and difficult to understand/measure

o Highlighted challenges preventing smooth collaboration were:

Quality measures were/are variable

There is a lack of consensus in developing measures/indicators of quality

5

There are many implementers doing PBI with experience on measuring

QoC but not often the reverse

o Design considerations:

Align program priorities with national priorities;

Programs must be sustainable; and

Minimize the administration burden

The community needs to examine the intersection between MNCH, PBI, and Quality

Plenary Q&A:

o How are patient satisfaction and preventive care and the related concept of score

cards and client-centered care reflected in the discussions on quality measures?

o These discussions are currently focused on low-resource settings; evidence on

PBF in high-resource settings is mixed especially in US.

o Non-financial incentives are also utilized within PBI programs – what are the

trade-offs between financial and non-financial incentives?

Most of the dialogue revolved around financial-incentives; moving into

non-financial is a much broader discussion

o The MCH community has a focus on system readiness; how much should PBF

focus on systems v. QoC? What is the variability of systems delivery across the

three spheres (PBI, MNCH, QoC)

Presenter: Huihui Wang – National RBF Program and Service Quality of Health Centers in

Burundi

National PBF (in Burundi)

o Started in 2006 with pilot program

o Summary bullets:

Data from 2010 to 2012 illustrated improvement in quality especially in

qualitative indicators;

Data was more mixed when looking at technical indicators facility by

facility

Figure 8: showed significant improvement in quality of pre-natal services

at health center level

Need to consider influence of cash v. in-kind incentives

The program needs to plan for how to manage plateau

Q&A: need to consider where the data has been “gamed” within the context of the

“testing of testers” project in Zambia

o Each measurement has a value and an error measurement (worst may be better

than they are and best maybe worse than they are)

o Best facility measurements: (figure 10) illustrates that measurement may contain

error

o Need to be able to measure “gamed” data and measurement error

In this discussion, gamed data meant that physicians were consciously

making a choice on where to focus quality of care based on the incentives

package and what they would receive.

o How do we know that measurement is correct?

6

The data was doublechecked by the health workers/data collectors

o Also give consideration that during the course of the project, there was better

data monitoring which could be a reason for the scores showing improvements

(i.e. the facility was always good but the data was initially poor and improved

over time)

o “Theory of change:” providers need to understand what they’re being incentivized

towards:

How do we break-up these indicator lists to make them more

understandable around what measures are being prioritized?

Presenter: Gyorgy Fritsche – Performance Based Financing in Nigeria

PBF in Nigeria: is fee for service PBF? Includes a ‘quality bonus’ (how is quality

derived?)

Summary Discussion

o It is important to keep in mind that this meeting is the intersection of PBI and

QoC and what can PBI do to improve QoC.

o Programs have to balance quantity measurement and quality (e.g. instituting the

quality check list)

o Interesting points to note:

What does it say about the measurement when the largest escalation is in

the beginning?

It may be good to experiment on how PBI drives quality and access;

Researchers need to look at the impact on health (as a product of quality

and quantity)

o The systems between Nigeria and Burundi are quite different illustrating

importance of context

o The Nigeria experience also illustrates the problem in common metrics

Can all of the metrics be found in a common place?

7

Issues, Approaches, and Current Thinking on the Development of PBI/QoC

Indicators Facilitator: Son Nam Nguyen

Presenters: Paulin Basinga – Issues, approaches, and current thinking on the development of

PBI/QoC Indicators

The study showed that there was a gradual increase in average facility quality scores

over time (2006 – 2010)

o There was an a particular uptick in quality of prenatal care

Lessons learned:

o Provider-controlled activities were more responsive to PBF

o Low QoC: maybe combine PBF with health providers training.

Note: Supervision only is not enough to increase provider knowledge

Summary Discussion

o Rwanda study was the only one that met the standards of the Cochrane Review

on PBI and QoC

o Distinguished because of the use of “vignettes” (aka case-study) to measure

competence

Have not continued to use vignettes and need a tool for assessing

supervisors

o Health economists and clinicians have conflict in use of terms (i.e. vignettes v.

case study)

o Vignettes don’t measure competence, only knowledge

Presenter: Manuela de Allegri - TRAction Malawi experience developing Quality of Care

indicators

Multi-method approach (qualitative and quantitative)

o Mixed methods approach looks at infrastructural indicators and interviews with

staff as inputs to measure QoC; some data will be collected through household

surveys as well as through exit interviews

o Study will measure the early identification of emergencies, prevention of

emergencies, and preparedness in emergencies

o Key component is understanding providers’ reasons for not performing selected

activities

Plenary:

o Absenteeism was self-reported

Perhaps it would be possible to blanket country with absenteeism survey

using small amount of resources

o Did not conduct post-natal component, since there is not a specific day/visit for

PNC. PNC interventions conducted whenever women returns to facility,

therefore hard to set up observations

Possible for the study to conduct exit interviews at the household level

which would mean missing observation for the 6 week visit

8

o Question becomes how will we capture the neo-natal mortality?

Study is not capturing this component, there isn’t enough capacity in

Malawi

Presenter: Kathleen Hill - Report from the Geneva meeting discussion on QoC indicators

The meeting brought together Quality Community and MNCH community

WHO is trying to expand conversation from coverage to intersection between coverage

and quality to include structural pieces

o Discussed which tools to use in capturing information/knowledge

o Vignettes cannot tell you about performance

Competence was also a hotly debated topic – different from knowledge (even

competence does not translated into quality)

Malawi found that some of the indicators don’t even make sense for measuring MCH

o To reiterate that context matters

Which indicators will work for our intended purpose: it is important to look across sectors

at indicators that may be successfully translated into the MCH context (e.g. TB);

Tensions: WHO wanted some highlevel measures that included quality that they could

use for advocacy

o But different indicators are needed for different contexts and stakeholders which

was a problem left unsolved

SNL Action Plan – the timing for the release is meant to coincide with a new Lancet

supplement (2014 as the year of the Newborn);

WHO surgical checklist is an example of a functioning tool/checklist is successful in

decreasing know-do gap; as a quality measure it may be good to dig-in to the systems

and tools that we have that are functioning.

Need to measure quality; it is not just about incentives it’s about Behavior Change which

requires constant measurement to gauge success.

Scientific advisory board at PEPFAR is discussing quality and the importance of the telephone

number of the person that comes to the ANC clinic as a key piece of information for tracking and

follow-up using mHealth (SMS or phone calls)

Plenary discussion:

Important to consider where a country is at baseline with an independent, rigorous

evaluation and identify gaps, and then QoC areas which can be improved upon using

PBI programs

It’s also important to jointly measure the intervention as well as the quality. The routine

measurements included in PBI interventions are like an organized Hawthorne effect.

The intervention itself needs to be documented.

MNCH technical content doesn’t lend itself to simple indicators which measure the right

thing in the right way. Need to be sure to consider perverse incentives.

Consider:

9

o Does this indicator make sense in the country context?

o Is this indicator something that will be increased by PBF?

Operationalizing Measurement for PBI/QoC: Successes and Challenges Facilitator: Steve Hodgins

Presenter: Petronella Vergeer - Operationalizing measurement for PBI/QoC: successes and

challenges

Quality integration into PBF is a popular topic, everyone wants answers now, we need to

be sure to manage our expectations and take care in the development of these

indicators

There are different methods to measure/verify quality

o These methods depend on the context, the intervention, and availability of data

o Recommendations:

Develop better HMIS systems

Improve both the availability and the use of data

Increase dialogue among and between stakeholders

Verification and counter-verification of quality can show high discrepancies.

The conversation should not just be about measurement, but about who is measuring

what, and the separation of measurement and verification functions

Presenter: Ronald Mutasa – RBF in Zimbabwe: Design, evidence, and early lessons on pay-

for-quality

Source documents at provider-level matter: types of registers that are maintained by

providers on day-by-day basis

Not all of the tools that we used are appropriate for frequent application

But data is important: maternal mortality in Zimbabwe has been increasing since 1994

but understanding why is a challenge

o RBF used a broad approach: Fee-for-services, management strengthening and

training, and monitoring for improved documentation

PBF program has been adapted over time (in implementation since 2011)

The process evaluation is using a mixed methods approach

Next steps are to explore opportunities for collaboration and to introduce QoC indicators

in HMIS

In order to facilitate PBF for quality, it’s important to support countries to develop and

adapt a national level quality of care framework with measures. Without this, QoC

measures are rarely integrated with any consistency

Lesson learned: importance of technical capacity and knowledge hand-in-hand with an

understanding of the local/national political economy and strong ownership by local

stakeholders

Presenter: Rianna Mohammed-Roberts – Monitoring & Measuring Quality at the Hospital level:

Liberia Health System Strengthening Project

10

Focus on hospital level

Sample size is an issue: may be a good idea to randomize community sample

o Possibly randomize by providers: what is the sample size to develop randomized

control trials

o Ministry did not allow for randomizing of financial incentives;

Need to identify strategies and research designs which will provide useable data, since

this cannot be designed as an impact evaluation

o The semi-experimental model can be used

Incentives around c-section: idea is to incentivize for complicated pregnancy and birth

o May be good to look beyond PBI; the fee is the same regardless of the procedure

So then what are you incentivizing? Need to incentivize a package of

services

Plenary Discussion

Need to look at incentives and high-level quality; we think of this in the context of HSS

In trying to systematize quality measurement probably need to look at incentives for

BCC

It’s important to use mixed methods

Managing Change Processes to Facilitate Supportive Contexts for PBI/QoC Facilitator: Jim Heiby – Managing change processes to facilitate supportive contexts for

PBI/QoC

Integrated PBI with QoC Initiatives

o There is a need to address the processes in healthcare and this is where quality

improvement focuses

“Trying harder” is not a strategy for quality improvement

o Incentivizing supervisors to push their staff to work harder results in short term

increases in quality/quantity of care – but these are short term fixes

Focusing on human resources using PBF to incentivize positive behavior change is a

more long-term, potentially sustainable way to improve QoC

Presenter: Marty Makinen – Proposed TRAction Senegal Work

Managing change for PBI

o Proposed Senegal work – assess feasibility of quality components in RBF

initiative at the Primary Care level

The ‘check list’ used in the Senegal context is a way of decreasing the number of

indicators and providing clear road map for both RBF incentives and improving QoC

o Key question: what is the appropriate list of indicators? And how does the

checklist influence or change behavior?

Conducting feasibility research that shows checklists should be focused on processes

and results

11

Presenter: Rashad Massoud – Lessons from National Level QI Procedures

Lessons in quality improvement

4 main factors to Improving Quality: Environment, Patient/Community, “System”, and

Oraganization

o Important to remember that QoC is about the ‘content of the care’ and the

‘process of receiving/providing care’

To improve quality, there must be both a change in the system and a change in

behavior

Presenter: Steve Hodgins - Gaps/areas in need of additional focus

Incentivizing behaviors – according to household surveys, there was a gap between

what was being reported and what is happening

o This is key when looking at data: how much can data be trusted?

Incentives are nice little options that politicians and implementers can implement with

relatively little resources where across the board salary increases may not be feasible.

o But we must keep in mind that programs also need to be replicable, scalable,

and sustainable

Plenary

The science of human behavior and the intersection with behavioral economics is

starting to coordinate and inject itself into PBI

o How do we get complex services to adhere to concrete results: need to create

some sort of continuous process to instigate BCC in a sustainable fashion within

existing bureaucracies

o Social norms play a huge role in changing behavior. So bringing in the crowd to

incentivize behavior change is one approach to incentivize change

One gap may be in what we know but not the “how” we did something to get to the

results, especially at this meeting where we are trying to merge 3 areas with 3 different

vocabularies (multi-dimensional approaches)

Need to think of PBF as a tool within the larger Health Systems context

Closing Facilitator: Jim Sherry

Culture is important: we’ve pulled together a number of different groups that take this

differently

Bridge for tomorrow discussions

o PBF is here to stay – it’s attractive, policy-makers have bought in. We need to

shift metrics to optimize its use.

o Language matters-we need to come up with a way to understand what the other

communities are saying

12

o Many PBF programs make assumptions about health systems that may need to

be revisited

o There are perspective limitations: we don’t live in single-payer, single-provider, or

centrally planned environments.

o Ethos Matters, Density of health workers, technical capacity matters. Publication

issues/data oversupply: gap between researcher and implementer.

o Non-financial incentives need to be considered alongside the financial. How do

we leverage limited resources?

o Need to balance complexity and significance. Moving from complex to simple

isn’t a dumbing down, but makes interventions more applicable

o Scaling-up from pilots. What are the strengths/limitations of these programs that

we have learned from the PBF pilots that need to be considered before

automatically scaling-up?

o Synergies etc. in these issues are intense whether in HR or civil society nothing

is acting alone

o A broader framework needs to exist for other people to understand and see

where what we’re talking about relates to others

o What is our learning process?

13

II. PERSPECTIVES ON PERFORMANCE-BASED INCENTIVES TO IMPROVE QUALITY OF MATERNAL NEWBORN HEALTH CARE: DAY TWO

Key Lessons from Day One Key Presenters: Sara Riese: overview of the previous days presentations

Key themes:

o PBI and quality of MNCH in low resource meetings

Stakeholders need to engage in order to shape the success of future

interventions

The interventions are complex while the number Subject-Matter-Experts

is limited

o It may be best to focus on key areas asking:

Who are we missing in this dialogue of change and common vocabulary?

What sources/resources should be leveraged for success

Group Work

o Three groups, each focused on a key area that has emerged from the

conversation: indicators, the “black box”, and navigating change

o Each group will consider: short-term work agenda, what are key human

resources, is there scope for building a community of practice?

o G1: Focus on key questions on indicators

What would a core set of non-negotiable indicators look like

How do we raise the bar on indicators

What is the scope for validating them?

Could better tools for measurement be arrived at?

o G2: What are the gaps in our knowledge and areas for secondary analysis?

Black box: how do we peg improvements

How can impact evaluations and learning from implementation serve our

agenda best

o G3: How do we navigate the changes in complex institutional context

PBI cross cuts a number of health systems areas – what are the areas to

prioritize?

How can the complex implementation process receive technical and

implementation support

How do you disseminate and create broader understanding

Plenary Discussion on Group Work o Building community of practices, scope, critical individuals

o Group 1: in QoC there are some things that are not measurable but there are

quality components that can be tracked outside of PBI and then PBI may be used

to fill gaps;

need to identify area of overlap: establish criteria in QoC v. PBI to find

overlapping indicators;

14

depending on health systems there may be different indicators so the

indicators need to be flexible for the context and resource availability;

PBI should be thought of at different levels (facility to health office); i

Indicators will need to be updated but the community must be cognizant

that providers should not be focused to narrowly on their incentives for

quality;

The tools could still be careful but need to be coordinated; may be good

to capture how other people have developed indicators (top-down v.

bottom-up);

1. HOW DO WE DO THAT: usefulness of global indicators v.

process so maybe develop guidelines for process of developing

indicators v. supplying list;

Need to capture challenges and bring qualitative researchers together

with quantitative researchers to define the common vocabulary and the

indicators

Make sure we don’t lose steam (but who should be involved in the

conversation?)

o Group 2: Looking into the “black box”. Areas for secondary analysis – where are

the hidden treasures of data;

Not seeing in the literature the analysis of process but this could be just a

delay in the literature published;

Each community has its own community of practice so perhaps we should

use existing resources;

PBI and QoC are already overlapping so may be just need to drive that

collaboration forward in a stronger fashion;

Use of secondary data – maybe it won’t respond to process and we don’t

know what existing data is out there usually publicly available data is

delayed 2-3 years;

Example of Argentina illustrating need to research the HOW of

interventions-illustrates the importance of qualitative researchers;

Mapping both existing data or existing Communities of Practice; can this

group work on process evaluation;

o Group 3: Change management and cross-cutting themes: need to distinguish

between PBI and QoC and does PBI need to be linked to quality;

It is an important part but it does not need to be explicitly linked such that

the implication is that the incentives do not decrease quality;

Recognize positive externalities and highlight them so as not to lose sight;

The Cape Town Health Systems Global meetings is a great opportunity to

gather because it brings together people in different ‘tribes’ and is a good

place to disseminate findings and tools;

Need to identify local champions;

Everything needs to be based on data and we need to think about cross-

sectoral capacity building (but how do we make it sustainable?)

15

PBI should be available to implementers so that people understand the

guiding principles of PBI; quality should not be compromised.

Opportunities for future discussion

o World Bank-Argentina meeting: this will be an event where all RBF-country

programs will come together (16 will come and share lessons learned)

complemented by more nascent country programs/implementers;

This will be a workshop, not a meeting

Expecting 200+ partners/participants, this will feed into an annual report

on the RBF/Health website;

There will be a side event at Cape Town on RBF

o Summer 2014 TRAction consultative meeting on Incentives for Quality

o Cash on Delivery at Center for Global Development: the relationship between

recipient and donor could be designed such that financing be distributed based

on meeting indicators,

PBF is focused on facility not on indicators; but maybe at higher-levels

such indicators like maternal mortality would be applicable

Perhaps we should use this indicator at facility level to reimburse at the

facility level for capturing the mom’s phone number;

Closing

Robert Clay

o There is complementarity between the work of the World Bank and the work of

USAID

The importance of impact evaluations to help to demonstrate results and

increase funding for Global Health. A focus on Implementation Science

gives us the opportunity to learn from interventions which helps both in

design and for improvement. We can take a learning approach vs a

blueprint approach.

This is data for decision-making and a great collaborative moment. Too

much research is not focused on the right areas to make an impact on

interventions

Tim Evans (remote)

o We are at an interesting place with RBF at the World Bank with 3 priorities:

1. Financing systems in high-quality way

2. Collaborating with Bank’s other sectors

3. That the populations that need services come up to scale.

o Need to explore the black box of service delivery through joint learning

processes. Move away from retrospective evaluation design, towards

prospective, which would include both process and summative evaluation.

o Propose that we look at the ~24 focus countries which are overlapping between

WB work and USAID work for collaboration and use the capacity and skills found

in each institution in order to do this work, as well as build communities of

practice in these important areas.

16

III. ANNEXES

1. Agenda

Perspectives on Performance-Based Incentives to Improve Quality of Maternal Newborn Health Care in Low Resource Settings: Launch of a technical working group

Location: World Bank I building 2nd floor Room 210

In 2012, TRAction held a consultative meeting that explored opportunities and challenges for performance measurement and research in the area of performance-based incentives and quality of maternal-newborn care. As PBI programs are increasingly including incentives for quality of care measures, many questions are emerging around the effect of financial incentives on quality of care and PBI program implementation best practices for incentivizing quality. TRAction, along with other stakeholders working in this area, will convene a technical working group to examine emerging PBI quality of care implementation and research experiences and determine priority areas for collaboration. In addition the group will develop the agenda for a more in-depth consultative meeting anticipated in the summer of 2014. Objectives for this launch of the technical working group include:

To review country experiences and reflect on a PBI/QoC framework and components (indicators,

measurement, supportive context)

To foster dialogue on measuring and improving quality within PBI programs

To discuss future opportunities and processes for the working group

Agenda

Thursday, Jan 16th (9am-5pm)

Time Presentation Presenter/Facilitator

8:30am Registration

9:00am Introduction Monique Vledder, World Bank (WB) Neal Brandes, USAID

9:30am Overview: Perspectives on PBI and QoC - Opportunities and challenges (30m)

Burundi (15m)

Nigeria (15m)

Plenary discussion on essential components of PBI/QoC framework

Kathleen Hill, URC Huihui Wang, WB Benjamin Loevinsohn, WB Facilitator: Sebastian Bauhoff, RAND

11:00am Coffee Break

11:30am Overview: Issues, approaches, and current thinking on the development of PBI/QoC Indicators (15m)

TRAction Malawi experience (15m)

Report from the Geneva meeting discussion on QoC indicators (15m)

Plenary Discussions

Paulin Basinga, Gates Foundation Manuela de Allegri, University of Heidelberg Kathleen Hill, URC Facilitator: Son Nam Nguyen, WB

1:00pm Lunch

17

2:00pm Overview: Operationalizing measurement for PBI/QoC: successes and challenges (15m)

Zimbabwe (15m)

Liberia (15m)

Plenary Discussions

Petra Vergeer, WB Ronald Mutasa, WB Rianna Mohammed, WB/Ken Leonard, University of Maryland Facilitator: Steve Hodgins, Save the Children (StC)

3:30pm Overview: Managing change processes to facilitate supportive contexts for PBI/QoC (15m)

TRAction Senegal experience (15m)

Lessons from National Level QI processes (15m)

Gaps/areas in need of additional focus in current programming (15m)

Plenary Discussions

Jim Heiby, USAID Marty Makinen, R4D Rashad Massoud, URC Steve Hodgins, StC Facilitator : Jim Heiby, USAID

5:00pm Closing Jim Sherry, George Washington University (GWU)/TRAction

Friday, Jan 17th (9am -1pm)

Time Presentation Presenter/Facilitator

9:00am Key lessons from previous day Sara Riese, TRAction

9:30am Review of PBI/QoC framework Dinesh Nair, WB

9:45am Group work on one of the 3 components: Indicators, Measurement, and Managing Change Each group will develop a ToR for their group, with key objectives, scope, and key resource persons

10:45am Coffee Break

11:15am Plenary discussion of each group’s work and joint development of milestones, processes for continued working and resources

12:00pm Opportunities for future interaction

TRAction Summer 2014 Consultative Meeting on PBI and Quality of Maternal Care

World Bank Argentina Meeting

Co-facilitation: Jim Sherry, GWU/TRAction and Dinesh Nair, WB

12:45pm Closing Tim Evans, Director, Health Nutrition & Population, Human Development Network, WB Robert Clay, Deputy Assistant Administrator, USAID

1:00pm Lunch

18

2. Participant List

Name Organization

Alison Chatfield Maternal Health Task Force Caroline Ly USAID Charlotte Warren Population Council Deb Armbruster USAID Dinesh Nair World Bank Gyorgy Fritsche World Bank

Ha This Nguyen World Bank Huihui Wang World Bank Jake Robyn World Bank Jim Heiby USAID Jim Sherry URC-TRAction Project Joe Naimoli USAID Kathleen Hill URC-TRAction and ASSIST Projects M. Rashad Massoud URC-ASSIST Project

Manuela de Allegri University of Heidelberg Marie Donaldson URC-TRAction Project Marty Makinen Results for Development Mead Over Center for Global Development

Monique Vledder World Bank Neal Brandes USAID

Paulin Basinga Gates Foundation Petra Vergeer World Bank Prea Gulati TRAction Rianna Mohammad World Bank Ronald Mutasa World Bank Sara Riese URC-TRAction Project Sebastian Bauhoff Rand Corporation

Son Nam Nguyen World Bank Stephan Brenner University of Heidelberg Steve Hodgins Save the Children-Saving Newborn Lives Project

Supriya Madhaven USAID

19

3. Next steps identified out of the workshop This annex outlines specific products that emerged from the workshop. Participants, as well as other

PBF/QoC/MNCH stakeholders, will be engaged on next steps to define and develop these products, as

appropriate.

Develop guidelines / toolkit for selecting indicators

The discussions at the recent workshop suggest a demand by policy-makers for guidance on selecting QoC indicators for use in PBI programs. The idea is provide a practical step-by-step approach for policy makers. The goal is not to be prescriptive but to describe the thought process and trade-offs inherent in indicator selection. The workshop identified the following steps to develop such guidance.

1. Conceptual overview and criteria selection

Review conceptual frameworks and objectives of PBI and QoC to clarify potential and limitations. Identify area of overlap and constraints.

Review criteria for indicators used in PBI and QoC, and identify overlapping criteria.

Use expert consultation to decide on combined PBI-QoC criteria.

2. Indicator selection

Review empirical evidence on important priorities in the country/region.

Obtain WHO indicators for these priority areas.

Use the combined PBI-QoC criteria to assess which of these WHO indicators could be effective.

Use expert consultation to decide on final set of indicators

Could compare empirical approach to indicators selection with expert-driven approach.

3. Dissemination

The guidance document and indicators (with scoring) could be made public. A potential model is the National Quality Forum which lists, defines and tracks indicators. See http://www.qualityforum.org/Qps/QpsTool.aspx

The website could also record actual use of indicators by programs/countries and provide for regular updates.

The site could be set up through TRAction’s knowledge management group. A draft document could be developed for discussions at the TRAction summer 2014 workshop. This draft could use a clinical area (MCH) and region (SSA) as exemplary case study.

20

Develop process evaluation tool kit

The recent workshop also highlighted the need for further process evaluation documentation of PBI programs, particularly those which are working to integrate quality measures. Development of a process evaluation tool kit would allow for collection of comparable data across programs, learning broad as well as specific lessons on the “black box” which can then be applied to other programs where appropriate.

Survey of information needs and current approaches to selecting QoC indicators for PBI

It may be useful to better understand what guidance the policy community requires on using PBI for improving QoC. This could be used to clarify the global research agenda on implementation science and focus implementation research efforts where the value-added is highest. Topics could include:

Objectives of the specific PBI-QoC project

Approach used by current projects to select their indicators. Thinking behind the final selection.

What information or guidance would have been useful to policy-makers in this process.

Where implementers turned for guidance. This survey could be sent to projects of the World Bank HRITF, who are at different stages of implementation and hence provide a spectrum of challenges and needs. In addition it may be informative to include Rwanda (as early implementer), and to briefly sketch the approaches that high-income settings such as the UK and the US have taken to indicator selection. The latter examples may help to characterize needs at different stages of QoC improvement, e.g., as countries move from structural indicators to process and outcome indicators.