Embedding Evaluation into mHealth Activities: Opportunities and Challenges

Kelly Ladin L’Engle, PhD, MPHBehavioral and Social SciencesFHI 360

mHealth Working Group Meeting: Innovative Monitoring and Evaluation of mHealth, October 20, 2011

We need more evaluation!

• While mHealth has great potential, current research…does not provide much evidence ror actual and wide-scale health impacts, nor answers to critical research questions…. (Mechael et al., 2010)

mHealth M&E Opportunities

• Real-time data, and lots of it• Ability to automate systems• Increased and additional access to program users

Two examples:1. m4RH (Mobile for Reproductive Health): opt-in, menu-based, pingpong, SMS system providing information about 9 family

planning methods and clinic locations in Kenya and Tanzania

1. ECINFO: Randomized controlled trial testing timed delivery of family planning role model stories to emergency contraception users, compared to m4RH access, and control group in Ghana

Keyword Content# of queries

Kenya Tanzania

m4RH Main Menu 6787 4333

00, 01, 02, 03, 99 Secondary Menus 7995 3135

11 Implants 1520 685

21 IUDs 1319 650

31 Permanent Methods 1084 620

41 Injectables 1467 671

51 OCPs 1492 598

61 Emergency Contraception 1408 1020

71 Condoms 2199 748

81 Natural Methods (incl. LAM in Kenya) 3170 1362

91 Clinic Locations 1281 813

Total Hits 29,722 14,635

Example: Monitoring User Data from m4RH

Example: Automated Systems for ECINFO

Administer Informed Consent

and Baseline Interview

Electronic Enrollment and Randomization

Intervention A (Role Model Stories)

Intervention B (m4RH) Control Group

Follow-up Text Data Collection

Follow-up In-person Interview

Pharmacy Recruitment of Eligible EC Users

Example: Increased Access to m4RH Users

Purpose Text

Introduction Thanks for using m4RH! Please help us with 4 questions and stand a chance to win FREE air time. M4RH is free, voluntary, and confidential…You can use m4RH even if you don’t answer these questions. You will receive the questions shortly...

Gender Please tell us your gender. Reply F if you are female, M if you are male, and send to 4128.

Age Which is true for you? If you are 19 or younger reply A; If 20-29 reply B; If 30-39 reply C; If 40 or older reply D.

Promotion Point

How did you learn about m4RH? From a clinic, reply R; partner, friend or family member- S; community health or peer educator- T; community event- U; radio- V.

Associated KAB Change

How has m4RH changed your use of family planning? Reply CHANGE then your change, like ‘CHANGE got an IUD’.

Reminder Hello from m4RH, this is a friendly reminder to please help by answering the 4 questions sent to you earlier this week. You still stand a chance to win.

Open-ended Texts

m4RH•Started using condom with my partner

•Protected me from early pregnancy

•Got to know clinics in my region hence access of services easy and from a reliable source

•Got IUD that has helped me manage my only two kids with my little monthly salary from casual jobs

•Use 3-month injections enjoy sex like never before

•Got implant for wife

Example: Additional Access to m4RH Users

1. Continuous electronic data capture and monitoring2. Fielding of text questions to m4RH users to assess age,

gender, where they learned about m4RH3. Potential for pre-post data collection to assess change in use

of family planning methods4. Telephone interviews to obtain feedback on system,

evidence for clinic referrals and FP uptake, and m4RH “intervention map”

5. Clinical-level data collection to assess if clients visit clinic as a result of m4RH

6. Future: Conduct rigorous experiment or quasi-experiment to assess m4RH impact

M&E: New Methodological Challenges

• Analysis issues• Response rates• Limits on SMS data collection:

– Question format and order– Data validity and reliability– Amount of data that can be collected

• Participant recruitment and study consent• Etc.

CONTACT

10

Visit our m4RH website:www.fhi.org/m4RH

Kelly L’Engleklengle@fhi360.org

919-544-7040