Embedding Evaluation into mHealth Activities: Opportunities and Challenges
Kelly Ladin L’Engle, PhD, MPHBehavioral and Social SciencesFHI 360
mHealth Working Group Meeting: Innovative Monitoring and Evaluation of mHealth, October 20, 2011
We need more evaluation!
• While mHealth has great potential, current research…does not provide much evidence ror actual and wide-scale health impacts, nor answers to critical research questions…. (Mechael et al., 2010)
mHealth M&E Opportunities
• Real-time data, and lots of it• Ability to automate systems• Increased and additional access to program users
Two examples:1. m4RH (Mobile for Reproductive Health): opt-in, menu-based, pingpong, SMS system providing information about 9 family
planning methods and clinic locations in Kenya and Tanzania
1. ECINFO: Randomized controlled trial testing timed delivery of family planning role model stories to emergency contraception users, compared to m4RH access, and control group in Ghana
Keyword Content# of queries
Kenya Tanzania
m4RH Main Menu 6787 4333
00, 01, 02, 03, 99 Secondary Menus 7995 3135
11 Implants 1520 685
21 IUDs 1319 650
31 Permanent Methods 1084 620
41 Injectables 1467 671
51 OCPs 1492 598
61 Emergency Contraception 1408 1020
71 Condoms 2199 748
81 Natural Methods (incl. LAM in Kenya) 3170 1362
91 Clinic Locations 1281 813
Total Hits 29,722 14,635
Example: Monitoring User Data from m4RH
Example: Automated Systems for ECINFO
Administer Informed Consent
and Baseline Interview
Electronic Enrollment and Randomization
Intervention A (Role Model Stories)
Intervention B (m4RH) Control Group
Follow-up Text Data Collection
Follow-up In-person Interview
Pharmacy Recruitment of Eligible EC Users
Example: Increased Access to m4RH Users
Purpose Text
Introduction Thanks for using m4RH! Please help us with 4 questions and stand a chance to win FREE air time. M4RH is free, voluntary, and confidential…You can use m4RH even if you don’t answer these questions. You will receive the questions shortly...
Gender Please tell us your gender. Reply F if you are female, M if you are male, and send to 4128.
Age Which is true for you? If you are 19 or younger reply A; If 20-29 reply B; If 30-39 reply C; If 40 or older reply D.
Promotion Point
How did you learn about m4RH? From a clinic, reply R; partner, friend or family member- S; community health or peer educator- T; community event- U; radio- V.
Associated KAB Change
How has m4RH changed your use of family planning? Reply CHANGE then your change, like ‘CHANGE got an IUD’.
Reminder Hello from m4RH, this is a friendly reminder to please help by answering the 4 questions sent to you earlier this week. You still stand a chance to win.
Open-ended Texts
m4RH•Started using condom with my partner
•Protected me from early pregnancy
•Got to know clinics in my region hence access of services easy and from a reliable source
•Got IUD that has helped me manage my only two kids with my little monthly salary from casual jobs
•Use 3-month injections enjoy sex like never before
•Got implant for wife
Example: Additional Access to m4RH Users
1. Continuous electronic data capture and monitoring2. Fielding of text questions to m4RH users to assess age,
gender, where they learned about m4RH3. Potential for pre-post data collection to assess change in use
of family planning methods4. Telephone interviews to obtain feedback on system,
evidence for clinic referrals and FP uptake, and m4RH “intervention map”
5. Clinical-level data collection to assess if clients visit clinic as a result of m4RH
6. Future: Conduct rigorous experiment or quasi-experiment to assess m4RH impact
M&E: New Methodological Challenges
• Analysis issues• Response rates• Limits on SMS data collection:
– Question format and order– Data validity and reliability– Amount of data that can be collected
• Participant recruitment and study consent• Etc.
Top Related