Improving measurement through Operations

Research

Peter Winchpwinch@jhsph.edu

Department of International Health, Johns Hopkins Bloomberg School of Public Health

andTechnical Advisor on Operations Research, MCHIP Project

Brief history of measurement in NGO child survival projects

(1) Early projects

– Wide variation in quality of evaluation

– Lack of standardized outcome measures, sampling techniques and questionnaires

– Hard to compare results between baseline and final, or between projects

Brief history of measurement in NGO child survival projects

(2) The response: Focus on indicators

– Definition of standard indicators, measurement tools and sampling techniques (30-cluster sampling, LQAS)

– Training/capacity building on evaluation methodology

– NGOs have applied this expertise in measurement of indicators as they have become engaged in projects funded by: Global Fund, PEPFAR, PMI etc.

Measurement in NGOOperations Research projects

Measurement itself has not been the major focus of innovation in NGO operations research projects

The measurement tradition has been “indicatorcentric”:– Main form of measurement is cross-sectional

household surveys– Main purpose of surveys is to measure

indicators– Indicators are pre-defined, and calculated in

standard ways Not much experimentation with measurement,

beyond measurement of standard indicators

Strengths of focus on indicators

Results of projects easily communicated to various audiences, including Congress

Can summarize results across projects Can standardize:

– Definition of indicators– Training and procedures for data

collection– Training and procedures for data

analysis

Weaknesses of focus on indicators

Less effort invested in developing innovative approaches to measurement

Many fundamental questions have not been addressed:– Why do some interventions work well in

one site, but not in another?– How can we select the most appropriate

intervention for a certain type of community?

– Through what mechanism do community-based interventions achieve their impact?

Example: Odek et al. 2009

Odek et al. Effects of Micro-Enterprise Services on HIV Risk Behaviour Among Female Sex Workers in Kenya’s Urban Slums

Promotion of micro-enterprise as alternative to commercial sex work

Figure 2 from Odek et al. 2009

We invest great effort in measurement of indicators on the far right of this model

We invest relatively little effort in measurement of the variables in this box

NGO scorecard on measurement

Doing better Practices in the home Coverage, access to

services, use of services

Quality of care

Room for improvement

Characteristics of communities

Steps on the way to improved practices, coverage and quality

Steps on the way to improved practices, coverage and quality

Community mobilization Empowerment Social capital Strengthened household to hospital

continuum of care

Usual measurement approach to these steps

Assume intervention area has it, and comparison area doesn’t have it– Intervention area is mobilized– Comparison area is not mobilized

This prevents us from answering important questions:– Why do some communities respond better

than others to interventions?– What interventions are most appropriate

for a community with characteristics X and Y?

Community mobilization, empowerment and social

capital are constructs

What is a construct?

Also called “latent variable” Abstract theoretical variable invented

to represent a phenomenon of interest to scientists

Not directly measurable

What is a construct?

Measurement of constructs in psychology

Most developed in psychology Psychology as a science depends on

valid and reliable measurement of constructs

Some psychologists spend their entire careers working on measurement of one construct

Example: Depression

How do you measure a construct?

Different from an indicator– Generally have one single question– Think carefully about the

numerator and the denominator Usually put together a set of

questions (items) related to the construct

Ways to measure constructs

Single item on a questionnaire– Frequently misleading e.g. “what is

your monthly salary?” to measure wealth

Additive index: One point for each question where respondent answers YES

Complex scales

Example of additive indices:Measuring community

mobilization

Community Action Cycle in community mobilization

Measuring community mobilization and the action

cycle1. Exposure to the process, engagement in the Community Action Cycle

2. Changes in individual awareness of problems, self-efficacy, commitment to change

3. Actions taken after being mobilized

1. Exposure to the process, engagement in the Community

Action Cycle1. Have you attended a meeting with your

neighbors organized by X project? Y/N2. Did you discuss/list the different health

problems here in this village? Y/N3. Did you discuss which problems are most

important? Y/N4. Did you discuss the causes of these

problems? Y/N5. Did you discuss what can be done about

these problems? Y/N

2. Changes in individual awareness of problems, self-efficacy, commitment to

change

1. “I feel confident that I could provide feedback to the committee on the quality of services in the clinic”– Strongly disagree, disagree, agree,

strongly agree2. “I intend to work with the committee to

improve the quality of services in the clinic”– Strongly disagree, disagree, agree,

strongly agree

3. Actions taken after being mobilized

1. I have provided feedback to the committee on the quality of services in the clinic Y/N

2. I have attended committee meetings to work on improving health services here Y/N

3. I am working with the committee to improve the quality of services in the clinic Y/N

What I am recommending

Keep measuring indicators, but don’t just measure indicators

Branch out into measurement of constructs

Be aware that time and effort is needed to develop good measures of constructs

Steps in developing a scale to measure a construct

Qualitative research to understand what is happening

Draft set of questions to measure the construct

Administer questions to a small sample Statistical analysis to select the questions

(items) that belong in the scale Revision of the questions to measure the

construct Include questions in questionnaire

administered to larger sample

Systems

“Systems” is a core theme throughout many maternal and child health projects

Manifests itself in different ways:– District health systems– Household-to-hospital continuum of

care– Referral networks– Supervision systems

Continuum of care for health services (Kerber et al. Lancet 2007)

Sociometric questionnaire

Questionnaire to elicit the structure of a social network

The questions are generally about relations with other people, rather than questions about individual knowledge, behavior etc.

Valuable tool for examining systems

Sociometric questionnaire applied to complete network:

Steps Define the population, and its boundaries List all actors in the population/network

– All health workers– All CHWs and TBAs

Select one or more questions to be asked regarding each of the other actors in the network

Create sociometric questionnaire with list of actors and questions– Often in table format

Example of sociometric questionnaire

ID Name How many times a week to you talk to

____?

How many times per month do you seek advice from

____?

001 Person #1

002 Person #2

003 Person #3

004 Person #4

Questions for a sociometric questionnaire

You can’t have very many! Examples:

– Whom do you go to for advice?– Who provides you with support for

X?– Who have you talked to about how

to treat sick children?– Who has referred a patient to you?– Who have you talked to on the

mobile phone?

Sociometric questionnaire:Full matrix

P1 P2 P3 P4

P1 5 4 10

P2 5 0 1

P3 2 0 2

P4 10 1 3

Sociometric questionnaire:Symmetrized lower-half matrix

P1 P2 P3 P4

P1

P2 5

P3 3 0

P4 10 1 2.5

Analysis of the symmetrized lower-half matrix

Calculation of measures for actors in network: Degree centrality, betweenness, closeness centrality, etc.

Calculation of measures of network as a whole: Density, cohesion etc.

Multi-dimensional scaling Sociogram

Examples of what could be done in maternal and child

health projects

1. Referrals between facilities2. Communication between health

workers

Referrals between facilities

Take existing data, then enter into a lower-half matrix

From the lower-half matrix:– Manual analysis: Take map, draw lines

between facilities, width indicates number of completed referrals

– Multidimensional scaling on matrix– Calculate network density before and

after the intervention

Health facility referral matrix

Facility 1

Facility 2

Facility 3

Facility 4

Facility 1

Facility 2

20

Facility 3

5 3

Facility 4

0 0 5

Density: Method 1

“Linkage density”

Number of facility pairs with any linksTotal number of facility pairs

=4/6

Density: Method 2

“Referral density”:Total number of referrals

Total number of facility pairs

=33/6

Might also calculate referrals in relation to target population, e.g. maternal referrals per women of reproductive age

Comparisons of density

Comparisons can be made for trends in same referral system over time

Comparisons with other systems likely won’t be meaningful, due to differences in:– Number of facilities– Total population size– Population per facility

Multidimensional scaling

1

3

2

4

20

53

5