Adaptive Management for Learning, Improvement & Evaluation of Health Programs

SoumyaAlva & Anne LaFondCenter for Health Information, Monitoring and Evaluation (CHIME)John Snow, Inc.September14, 2017

2

OUTLINE

• What is adaptive management?• Why use adaptive management?• “CHN on the Go” – an example of JSI’s use of

adaptive management

3

WHAT IS ADAPTIVE MANAGEMENT?

4

EVALUATION THROUGHOUT THE PROJECT CYCLE

In some cases:Limited to Mid term evaluation

5

ADAPTIVE MANAGEMENT

Evidence driven program implementation strategy

A big step beyond simple performance monitoring

Iterative approach to learning & evaluation while implementing

Interest in better implementation strategies for better outcomes

Increasingly being used in the field of public health

Data to drive adaptation

6

DEVELOPMENTAL EVALUATIONAn approach to evaluation grounded in systems thinking and that supports innovation by collecting and analyzing real time data in ways that lead to informed and ongoing decision-making as part of the design development and implementation process.

- Michael Quinn Patton

7

USAID LEARNING LABAdapting—or adaptive management—refers to intentionally and systematically using relevant knowledge to inform decision-making and ultimately take action. Within the development context, that action could be adjusting interventions or whole strategies, experimenting with new ways of working, scrapping programming that simply isn’t working, or scaling approaches that have demonstrated value.

- Monalisa Salib, in USAID Learning Labhttps://usaidlearninglab.org/lab-notes/adaptive-management-if-not-now-when

8

PARTICIPATORY ACTION RESEARCH IN HEALTH

Participatory action research cycle. Oxford University Press, Australia, from F. Baum, 2016, The New Public Health 4th ed. Oxford University Press. www.oup.com.au

9

THE ADAPTIVE MANAGEMENT CYCLE

10

WHY USE ADAPTIVE MANAGEMENT?

11

KEY ADVANTAGES

RELEVANT AND TIMELY DATA ARE NEEDED TO MAKE PROGRAM

IMPLEMENTATION EVIDENCE-BASED

Moving away from traditional M&E

Incorporate learning into implementation (testing implementation strategies)

Ability to develop a better program

Better tailoring of interventions to context & target populations

No need to wait for endline results, potential for early stakeholder buy-in

12

HOW DOES THIS TRANSLATE TO THE REAL WORLD?

Quality ImprovementPDSA CyclesSTUDY

PLAN

DO

ACT

Organizational ManagementLearning Organization

Program Learning/ Routine Reflection

Iterative & Adaptive Program Strategy Development (e.g. developmental evaluation)

13

THREE MANAGEMENT STRATEGIES

Traditional

Testing

Adaptive

• PMP• Baseline/Midline/Endline• Implementation Research

• PMP• Baseline/Midline/Endline• Implementation Research• Process Documentation &

Developmental Evaluation

• PMP• Baseline/Midline/Endline

• Management • Reporting

• Management • Reporting• Learning

• Management • Reporting• Learning• Adaptation

1

2

3

Purpose

Purpose

Purpose

14

WHAT DOES IT TAKE TO MAKE THIS WORK?

Supportive funder & implementing partner

For funding and enabling team to iterate

Mixed methods –use of multiple data types

Researchers & evaluators who have a good understanding of program implementation

Stakeholders willing to engage

15

JSI’S EXPERIENCEWITH ADAPTIVE MANAGEMENT:THE EXAMPLE OF “CHN ON THE GO”

16

CCH & THE “CHN ON THE GO” APP

ICT via a smartphone application (CHN on the Go)

An innovative solution to address barriers in job satisfaction & motivation among Community Health Nurses (CHNs)

Designed using an HCD approach to identify needs of CHNs

Improve connectivity & career development among CHNs Increase their linkage & interactions with a professional network & supervisors

Provide clinical refreshers

Innovations for MNCH Initiative: Funded by the BMGFConcern Worldwide : Implementing Partner 2009-2016 (2 Phases)JSI: Global Research Partner with in-country research partners (Sept 2012-Dec 2016)

17

PROGRAM IMPLEMENTATION• Implemented by Grameen

Foundation in Ghana

• 5 Districts: Ningo Prampram, Ada East and West (Greater Accra), and South Tongu, South Dayi (Volta)

• Approximately 50 health centers and associated CHPS

• More than 300 CHNs/supervisors

• Project timeline:• Design process: Oct ‘13-March ‘14• V1 July 2014, V5 May 2016

GHANA

VOLTA

Togo

Burkina Faso

Gulf of Guinea

Côte d’Ivoire

GREATER ACCRA

18

OBJECTIVES

Effect on greater health worker motivation/satisfaction?

Clinicalrefresher

Workplanning

Emotionalsupport

Career development

Connectivity

Professionalnetwork

19

Baseline and endline levels of CHN motivation & job satisfaction? Differences by district, facility type, and CHN age?

Association of CHN on the Go with greater CHN motivation/ satisfaction?

Project implemented as planned? Barriers and supportive factors affecting implementation? Expected results as foreseen in the pathways in the TOC?

1

2

3

RESEARCH QUESTIONS

20

JSI’S ROLE AS GLOBAL RESEARCH PARTNER

Develop Theory of Change

Rigorous evaluation approach

Mixed method approach

Real time data for adaptive management during the life of

the project

Process documentation

Monitoring data

Review meetings with implementers and

stakeholders

RME activities to understand impact of pilot

21

Learning Center

Health knowledge

Point of Care Health knowledge,

relationshipsPlanner

Targets, Work scheduling

Achievement Center Targets,

relationships

Staying Well Keeping well, relationships

AN ADAPTATION OF THE THEORY OF CHANGE

Driver (Direct outcomes (Modules & Intermediate outcomes) Direct outcomes ) Driver

Change Pathways

WHATSAPPCommunication &

Connectedness

Supervisor AppMonitoring & Supervision

FINAL OUTCOMES: Better CHN performanceBetter quality of careImproved maternal & child health

ASSUMPTION: Health workers feel unmotivated and frustrated in their jobs

SYSTEM LEVEL CHANGES: Supervision, education/learning

22

COMPONENTS OF ADAPTIVE MANAGEMENT

Refine Program Strategy using HCD

Process documentation, periodic reflection and review with program team

Endline Data Collection

Baseline Data Collection

Process documentation, periodic reflection and review with program team

Human centered design (HCD)

Process documentation, periodic reflection and review with program team

23

RME TimelineProject Start

May – June 2014

Project EndMay 2016

BaselineMay – Aug 2014

PD Round 1May – Aug 2014

BaselineMay – Aug 2014

Feedback SessionApril 2015

Magpi e-surveyApril – May 2015

Pop-up QuestionsJune 2015

PD Round 2July – August 2015

Magpi e-surveyDec 2015 – Jan 2016

EndlineApril 2016

Pop-up QuestionsMay 2016

Other data collection activities: • Monthly field interviews• Monthly usage monitoring dashboard

Are pathways in

the TOC being followed?

Is implementation progress in relation to plans?

Are results on app use as expected according to the CCH TOC?

Reasons why different modules were used by CHNs?

Successes & challenges faced?

24

OUR APPROACH

Designed & planned when project started

Worked closely with local research partners

Worked closely with implementing program staff (Concern/ Grameen)

Aware of all aspects of program implementation

Led research but in collaboration with Concern/Grameen

Review meetings on a regular basis

Information sharing with stakeholders

Analyzed PD and monitoring data in combination with endline findings

Create engaging visualizations

25

• More probing into usage of the app and specific modules by CHNs

• Helped develop newer iterations of the app

• Greater focus on technical issues faced

• Greater focus on supervision

• Easy to understand visualization

• Built stakeholder interest

HOW DID ADAPTIVE MANAGEMENT HELP?

26

BENEFITS• Increased qualitative or mixed data

availability – deeper than just simple monitoring

• Ability to review program strategy

• Near real time picture of implementation

• Sharing findings with stakeholders on an ongoing basis

• Increase in appreciation of the value of iteration among program managers

• Greater interest in methodology over time

CHALLENGES• Uptake and appreciation of

iterative measurement and routine reflection among program managers

• Time

• Ability to use data

• Program theory understanding

• Tension between evaluation standards and iterative and adaptive program design

• Complexity of the theory of change/change pathway

• Time and resource investment

+

++

+

+

+

-

-

-

-

27

SUMMARY OF FINDINGS

• As a job aide for CHNs (work planning, knowledge, communication)

• As a job aide for supervisors (communication, monitoring & supervision)

• Helps CHNs

• Gain confidence

• Gain respect & recognition from colleagues, supervisors & clients

• Provides emotional support for some

App addresses various needs of CHNs & supervisors

Sustainability• CHNs want to continue using the app - both current and other

districts

• GHS support to scale up, using CHNs own phones, especially in Volta

28

RESOURCES• Better Evaluation http://www.betterevaluation.org/en/taxonomy/term/1429

• USAID Learning Lab, Collaborating, Learning and Adapting (CLA). https://usaidlearninglab.org/faq/collaborating-learning-and-adapting-cla

• USAID’s US Global Development Lab (Monitoring, Evaluation, Research and Learning Innovations - MERLIN). https://www.usaid.gov/GlobalDevLab/about/monitoring-evaluation-research-and-learning-innovations-program.

• Doing Development Differently. http://doingdevelopmentdifferently.com/

29

QUESTIONS FOR DISCUSSION

30

QUESTIONS• What are some of the decisions that need to be made at different stages of

project implementation? What questions need to be answered? And for what purpose?

• 6 months after implementation of a 2-3 year project, mid-term, 6 months before end of project

• What types of data are useful at each stage to answer questions about project fidelity, implementation, strategy relevance and project performance?

• How do you make better decisions with the data available?

• What are some of the ways you can use data for decisionmaking?

• When and how do you engage stakeholders?

• How do you enable them to use data at all stages?