WH2014 Workshop: mHealth Evidence

WLSACONVERGENCE SUMMIT

MHEALTH EVIDENCE: EVALUATING MOBILE AND WIRELESS HEALTH

AUDIE ATIENZA, NATIONAL CANCER INSTITUTE, NIH

BETHANY RAIFF , ROWAN UNIVERSITYINBAL NAHUM-SHANI , U. MICHIGANDAVID MOHR, NORTHWESTERN UNIVERSITY

Title of your module

Your name, title and affiliation

http://obssr.od.nih.gov

mHealth Evidence WorkshopWireless Health 2014October 29, 2014NIH Campus, Bethesda, MD

3

In this training, you will work in groups to explore different methods for evaluating mobile and wireless technologies

There are participants from across academia, industry, and government participating

The objective is to have a better understanding of when to use different study designs and how the choice effects what information you get from your research

Today is divided into three modules:

– Single case study design

– SMART and Factorial Designs

– Design for Evolving Behavioral Intervention Technologies

Each module is scheduled for 1 hour and will include:

– A briefing to provide a common background and understanding for discussion

– Discussion among the participants

In each module discussion, participants will discuss these questions:

– How does what was just presented effect your project?

– What questions can you answer using the method?

4

Agenda 9:00-9:15 Introduction to the workshop

» Audie Atienza, PhDNational Cancer Institute, NIH

9:15-9:45 Single case study design

» Bethany Raiff , PhDRowan University

9:45-10:15 Small Group breakouts

10:15-10:30 Report back

10:30-11:00 SMART and Factorial Designs

» Inbal Nahum-Shani , PhDUniversity of Michigan

11:00-11:30 Small Group breakouts

11:30-12:00 Design for Evolving Behavioral Intervention Technologies

» David Mohr, PhDNorthwestern University

12:00-12:15 Closing and questions

5

Today’s taskYour goal for this look at different study designs to test the intervention in the following scenario:– Your hospital has identified smoking cessation as a critical issue for its

quality improvement program. They would like to include an intervention to increase smoking cessation in hospitalized patients. Their current psychoeducational program and smoking cessation medications, the average rate for smoking cessation at two months is 15%. The hospital has asked you to help them create an intervention that can provide sustained support for their affected adult patient population (ages 40-60, approximately 1000/year). Based on recent research, your team decides that the components of successful cessation programs include: education on smoking cessation, follow up with patients after they leave the hospital, electronic information and follow-up, stress reduction tools and pharmacy refills for smoking cessation medications delivered through a wireless system that is geared to release the meds to match people’s high craving times.

6

Questions for Breakouts

As you apply the method to your problem, answer the following questions:–How does what was just presented effect

your project?–What questions can you answer using the

method?

Single case experimental designs

Bethany Raiff, PhD, BCBA-D

Rowan University

Outline

Introduce single-case designs (SCDs) to evaluate behavioral data

Introduce each SCD and discuss advantages and disadvantagesProvide examples

Analysis of SCDs

Single Case Design (SCD)

Focus on one individual at a time aka n-of-1 designs (even though n in a study is > 1) Not the same as a “Case study”

Common characteristics: Repeated assessment Replication of effects Each case serves as his/her/its own control

When to use SCDs?

Able to repeatedly assess behavior/symptoms over time

Desire to change behavior/symptoms (not merely assess) and establish preliminary efficacy

When it is unethical to withhold treatment from some participants.

Can be applied across a range of funding levels

SCDs

Types of designs Reversal-replication (e.g., ABA,

ABAB) Alternating treatments Multiple-baseline Changing criterion Combination

Talia

Day

0

2

4

6

8Baseline

Intervention

Baseline

Edward

0 2 4 6 8 10 12 14 16

Da

ily fre

qu

en

cy o

f b

loo

d g

luco

se te

stin

g

0

2

4

6

8

Bonita

Andrea

Day

0 2 4 6 8 10 12 14 16

Baseline

Intervention

Baseline

Ran out of strips

Reversal Design: Increasing blood glucose testing with Internet-based incentives (Raiff & Dallery, 2010)

$35

$45

$48

Reversal Design: Increasing peer interactions (Allen et al, 1964)

Alternating treatments design (Kratchowill et al., 2012)

Alternating Treatments: Promoting moderate-to-vigorous activity (Larson et al., 2014)

Reversal and Alternating Treatments Designs

Advantages Within-subject replication Clear demonstration of the effect of the independent variable Flexible (ABA, ABAB, ABACACB, BAB)

Disadvantages Must remove a treatment to demonstrate experimental control

(not always ethical) Risk of carryover and order effects Will not work with irreversible treatments

Multiple-Baseline: Adherence to weight management with personal electronic device (Cushing et al., 2010)

Multiple Baseline: Different communities (Hawkins et al, 2007)

Changing Criterion Design: Decreasing daily smoking (Hartmann & Hall, 1976)

Changing Criterion: Increasing number of steps taken per day in sedentary adults (Kurti & Dallery, 2013)

Multiple Baseline & Changing Criterion Designs

Advantages: Do not need to remove an effective

treatment Flexible (behavior, settings, participants)

Disadvantages: Requires quantitative criteria that can be

targeted in step-wise fashion (CCD) May need more subjects to convincingly

show experimental control

Data analysis

Is there a reliable effect of the intervention?

What is the magnitude of the effect?

Are the results clinically meaningful and socially valid?

Emphasis on individual subject effects.

Statistical analyses can be used along with visual analysis.

Variability, overlapping data, trends

Mean Shift

Level Shift

Emphasis on individual subjects (Dallery et al., 2013)

Emphasis on individual subjects(Dallery et al., 2013)

Standards for evaluating SCDs The following guidelines have been

proposed: Systematically manipulate an independent

variable Collect inter-observer agreement Phase must include at least THREE data

points (preferably FIVE or more) At least THREE attempts to demonstrate an

intervention effect at different time points (e.g., participants, conditions, criteria, etc)

Kratchowill et al. (2012); Logan et al. (2008)

General Conclusion about SCDs

Rigorously and efficiently establish feasibility and preliminary efficacy

Overcome ethical barriers to withholding or discontinuing effective treatments

Obviate logistical issues (e.g., access to a large number of participants)

For more information about SCDs

Books Barlow & Nock (2008). Single-case experimental designs: strategies for

studying behavior change. Pearson Publishers Kazdin (2010). Single-case research designs: methods for clinical and

applied settings. Oxford PublishersJournal Articles

Dallery, J. & Raiff, B.R. (2014). Optimizing behavioral health interventions with single-case designs: from development to dissemination. Translational Behavioral Medicine: Practice, Policy and Research (online first).

Dallery, J., Cassidy, R., Raiff, B.R. (2013). Single-case experimental designs to evaluate technology-based health interventions. Journal of Medical Internet Research, 15, online. doi: 10.2196/jmir.2227

Thank you!

Questions?

NIH mHealth Institute Event

Inbal Nahum-Shani

Experimental Designs for Optimizing Interventions

Key DefinitionKey Definition

Multi-Component Interventions• Component:

► The content of the intervention (e.g., topics in prevention program)► The intervention modality (e.g., phone calls/emails) ► Features to promote compliance or adherence (e.g., reminder emails)

Example: • Optimizing a technology supported lifestyle intervention for weight loss:

Bonnie Spring, PI. DK097364 ► Telephone Caching ► Report to Primary Care Provider ► Text Messages ► Meal Replacements► Buddy Training

Thursday, April 13, 2023 © Inbal Nahum-Shani 2014 33

How do We Typically Develop How do We Typically Develop Interventions?Interventions?

1. Theoretical Model


2. Intervention Components


4. Confirm Effectiveness


3. Intervention Package



How do We Typically Develop How do We Typically Develop Interventions?Interventions?










Open Questions Open Questions Efficacy of Individual components► Which components are effective?► Which level is more appropriate?► Which components work well together?

Sequencing of components► Which component to offer first?► Which to offer subsequently?► How should I tailor components over time?

Briefly describe the overarching guiding principles for your work

► Sub-bullets of explanation List and define Key Terms necessary to work with experts in your field

Thursday, April 13, 2023© Inbal Nahum-Shani 2014 36

Open Questions Open Questions Efficacy of Individual components► Which components are effective?► Which level is more appropriate?► Which components work well together?

Sequencing of components► Which component to offer first?► Which to offer subsequently?► How should I tailor components over time?

Briefly describe the overarching guiding principles for your work

► Sub-bullets of explanation List and define Key Terms necessary to work with experts in your field

Factorial DesignsFactorial Designs

SMARTSMART



Factorials: More than 1 factor; levels of each factor crossed with levels of other factors.

► Should I include Text Messages?• Factor 1: Text (On/Off)

► Should I include Meal Replacement?• Factor 2: Meal (On/Off)






Experimental conditions 2X2 factorial N=400

Experiment Condition

Factor

Text Meal

1 (N=100) On On

2 (N=100) On Off

3 (N=100) Off On

4 (N=100) Off Off




► Should I include Text Messages?• Factor 1: Text (On/Off)• Main effect: On (N=200) vs. Off (N=200)




Factor

Text Meal

1 (N=100) On On

2 (N=100) On Off

3 (N=100) Off On

4 (N=100) Off Off





► Should I include Meal Replacement?• Factor 2: Meal (On/Off)• Main effect: On (N=200) vs. Off (N=200)



Factor

Text Meal

1 (N=100) On On

2 (N=100) On Off

3 (N=100) Off On

4 (N=100) Off Off






► Should I include Buddy Training?• Factor 3: Buddy (On/Off)




Experimental conditions 2X2X2 factorial N=400

Condition Factor

Text Meal Buddy

1 (N=50) On On On

2 (N=50) On On Off

3 (N=50) On Off On

4 (N=50) On Off Off

5 (N=50) Off On On

6 (N=50) Off On Off

7 (N=50) Off Off On

8 (N=50) Off Off Off








Condition Factor

Text Meal Buddy

1 (N=50) On On On

2 (N=50) On On Off

3 (N=50) On Off On

4 (N=50) On Off Off

5 (N=50) Off On On

6 (N=50) Off On Off

7 (N=50) Off Off On









Condition Factor

Text Meal Buddy

1 (N=50) On On On

2 (N=50) On On Off

3 (N=50) On Off On

4 (N=50) On Off Off

5 (N=50) Off On On

6 (N=50) Off On Off

7 (N=50) Off Off On



► Should I include Meal Replacement?• Factor 2: Meal (On/Off)• Main effect: On (N=200) vs. Off (N=200)






Condition Factor

Text Meal Buddy

1 (N=50) On On On

2 (N=50) On On Off

3 (N=50) On Off On

4 (N=50) On Off Off

5 (N=50) Off On On

6 (N=50) Off On Off

7 (N=50) Off Off On




► Should I include Buddy Training?• Factor 3: Buddy (On/Off)• Main effect: On (N=200) vs. Off (N=200)


SMART Designs SMART Designs

Adaptive Intervention:

► Intervention in which components are sequenced and adapted over time in a way that addresses the specific and changing needs of participants

Motivation in the context of technology-supported interventions:

► Cost: Try less expensive components first and more expensive components later - for those who really need them

► Boredom: Introduce new components over time to increase interest ► Cognitive overload: Offering all components simultaneously -

difficulty to allocate attention

SMART:

► Randomized Trials ► Multiple stages of randomization► Each stage corresponds to a critical question concerning the sequencing and

adaptation of intervention options over time


SMART DesignsSMART Designs

Hypothetical Example► Aim: Develop an adaptive technology-supported weight loss intervention

• I consider 3 components: Phone, Text and Buddy • Phone is most expensive; Text least expensive

► What is the best way to sequence and adapt?• Which component to offer first: Phone or Text • Which component to add for non-responders: Phone or Buddy


Questions We Can Address with Questions We Can Address with SMARTSMART First-stage intervention component:

► Is it better to start with Phone Coaching or Text Messages?► (SG1+SG2+SG3) vs. (SG4+SG5+SG6) ► Phone Coaching vs. Text Messages

• Controlling for subsequent intervention component

R

Phone

Text

Response

Non-Response RBuddy (SG3)

Phone (SG2)

Response


Phone (SG5)

Step-Down (SG1)

Step-down (SG4)


Questions We Can Address with Questions We Can Address with SMARTSMART Second-stage intervention component:

► Is it better to add Phone Coaching or Buddy Training?► (SG2+SG5) vs. (SG3+SG6) ► Phone Coaching vs. Buddy Training

R

Phone

Text

Response


Phone (SG2)

Response


Phone (SG5)

Step-Down (SG1)

Step-Down (SG4)


Questions We Can Address with Questions We Can Address with SMART SMART Embedded adaptive interventions

R

Phone

Text

Response


Phone (SG2)

Response


Phone (SG5)

Step-Down (SG1)

Step-Down (SG4)

Stage 1 = {Phone}, Then

IF response = {NO} THEN stage 2 = {Add Phone} ELSE IF response = {YES} THEN stage 2= {Step-Down}



R

Phone

Text

Response


Phone (SG2)

Response


Phone (SG5)

Step-Down (SG1)

Step-Down (SG4)

Stage 1 = {Phone}Then, IF response = {NO} THEN stage 2 = {Add Buddy}ELSE IF response = {YES} THEN stage 2= {Step-Down}



R

Phone

Text

Response


Phone (SG2)

Response


Phone (SG5)

Step-Down (SG1)

Step-Down (SG4)

Stage 1 = {Text}, ThenIF response = {NO} THEN stage 2 = {Add Buddy} ELSE IF response = {YES} THEN stage 2 = {Step-Down}



Stage 1 = {Text}, ThenIF response = {NO} THEN stage 2 = {Add Phone} ELSE IF response = {YES} THEN stage 2 = {Step-Down}

R

Phone

Text

Response


Phone (SG2)

Response


Phone (SG5)

Step-Down (SG1)

Step-Down (SG4)



Stage 1 = {Text}, ThenIF response = {NO} THEN stage 2 = {Add Buddy} ELSE IF response = {YES} THEN stage 2 = {Step-Down}

VS.VS.

Stage 1 = {Phone}, Then

IF response = {NO} THEN stage 2 = {Add Phone} ELSE IF response = {YES} THEN stage 2= {Step-Down}

R

Phone

Text

Response


Phone (SG2)

Response


Phone (SG5)

Step-Down (SG1)

Step-Down (SG4)


Summary Summary

Factorial Designs:

Efficacy of Individual components

► Which components are effective?► Which level is more appropriate?► Which components work well together?

SMART Designs:

Sequencing and adaptation of components

► Which component to offer first?► Which to offer subsequently?► How should I tailor components over time?


Evaluation of Behavioral

Intervention TechnologiesDavid C. Mohr, Ph.D.Ken Cheung, Ph.D.

Stephen M. Schueller, Ph.D.

Northwestern University&

Columbia University

WHAT’S A BIT?BITs are applications that use technologies such as mobile phones, computers, tablets, and sensors, to support behaviors that improve health, mental health, and wellness.

bit = behavioral intervention technology

COMPUTERS

SENSORS

MOBILE PHONES

TABLETS

RCTs IN BEHAVIORAL AND PSYCHOLOGICAL HEALTH

• RCT Methods • A fixed intervention (pharmacological

agent, device).

• Development of methods of evaluation for psychological interventions (Eysenck, 1952)• Intervention locked down, with

manualization, therapist training and supervision.

Timeline for Technology Development

2000 2012201020042002 2014

Android release

2006

End of Symbian

2016

Obtain Grant

Develop & Pilot

Recruit

Follow-up Eval

Data Analysis & Pub

1-2 yrs

1-2 yrs 1-2 yrs

1-3 yrs

1 yr

Academic Timeline

2008

Symbian release

Problems

• Mismatch between research and technology innovation timeline.

• BITs change in the real world• Patient expectations change• Patient Horizon

• Meds – 10s of millions over decades• Behavioral – 10s of millions over decades• BITs – potentially millions, over small

period of time

Should we be testing BITs in traditional RCTs ?

WHAT’S A PRINCIPLE?

Evaluate [Behavioral Strategy], delivered using [Detailed description of essential BIT functionality], to affect [Clinical outcome].

Principle = An underlying rule or model

BIT MODEL:Why, What, How,

WhenHow:

Behavioral Strategies

How:Behavioral Strategies

Why: Clinical Aims

Why: Clinical Aims

What: BIT Elements

What: BIT Elements

How:Characteristics

How:Characteristics

When: Workflow

When: Workflow

Behavioral

Instantiation

cbits.northwestern.edu

Mohr et al. J Med Internet Res 2014;16(6):e146


WhenHow:



Why: Clinical Aims

Why: Clinical Aims

What: BIT Elements

What: BIT Elements

How:Characteristics

How:Characteristics

When: Workflow

When: Workflow

Why:Clinical Aims

•Lose Weight• Increase Exercise• Decrease Calories

•Decrease Depression• Increase Positive

Activities• Decrease

Avoidance•Usage Aims

Why:Clinical Aims

•Lose Weight• Increase Exercise• Decrease Calories

•Decrease Depression• Increase Positive

Activities• Decrease

Avoidance•Usage Aims




WhenHow:



Why: Clinical Aims

Why: Clinical Aims

What: BIT Elements

What: BIT Elements

How:Characteristics

How:Characteristics

When: Workflow

When: Workflow


•Education•Goal Setting•Monitoring•Feedback•Motivation Enhancement


•Education•Goal Setting•Monitoring•Feedback•Motivation Enhancement




WhenHow:



Why: Clinical Aims

Why: Clinical Aims

What: BIT Elements

What: BIT Elements

How:Characteristics

How:Characteristics

When: Workflow

When: Workflow

What:Elements

•Information Delivery•Notifications•Logs•Passive Data Collection•Reports

• Visualizations

What:Elements

•Information Delivery•Notifications•Logs•Passive Data Collection•Reports

• Visualizations




WhenHow:



Why: Clinical Aims

Why: Clinical Aims

What: BIT Elements

What: BIT Elements

How:Characteristics

How:Characteristics

When: Workflow

When: Workflow

How:Characteristics

•Medium (Text, Audio, Video)•Complexity•Aesthetics•Tailored to user in other ways

How:Characteristics

•Medium (Text, Audio, Video)•Complexity•Aesthetics•Tailored to user in other ways




WhenHow:



Why: Clinical Aims

Why: Clinical Aims

What: BIT Elements

What: BIT Elements

How:Characteristics

How:Characteristics

When: Workflow

When: Workflow

When:Workflow

•User defined•Conditions

• Time-based rules• Task completion

rules• Event-based rules

•Tunneling•Frequency•Length of treatment

When:Workflow

•User defined•Conditions

• Time-based rules• Task completion

rules• Event-based rules

•Tunneling•Frequency•Length of treatment



PRINCIPLE STATEMENT



Why: Clinical Aims

Why: Clinical Aims

What: BIT Elements

What: BIT Elements

How:Characteristics

How:Characteristics

When: Workflow

When: Workflow


Evaluate [Behavioral Strategies], delivered using [Elements, Characteristics, Workflow], to affect [Clinical Aim] .

PRINCIPLE STATEMENT EXAMPLE



Why: Clinical Aims

Why: Clinical Aims

What: BIT Elements

What: BIT Elements

How:Characteristics

How:Characteristics

When: Workflow

When: Workflow


MyFitnessPal aims to support users in goal setting and self-monitoring, using logging features and feedback to support weight loss and physical activity. •Changes over the last years

• Aesthetics• Added of barcode scanning• Added social networking

TRIGGERS FOR CHANGES

• Use data – Expected vs. observed• User feedback – qualitative

information about the problem• Clinical outcomes

DECISION CRITERIA

Does the change interfere with a primary principle being tested?•Clinical Aim: Usually off limits.•Behavioral Strategies: Often these are clearly defined, and therefore would not be changeable. •Instantiation Components (Elements, Characteristics, Workflow): These are often the target of engineering studies, but less commonly the target of clinical research.

• But may affect behavioral strategies.



Why: Clinical

Aims

Why: Clinical

Aims

What: BIT

Elements

What: BIT

Elements

How:Characteristi

cs

How:Characteristi

cs

When: Workflow

When: Workflow

DECISION CRITERIA

If the introduction of the change is successful, would it create an alternative explanation for the success of the trial?

• Unintended consequences

DOCUMENTATION AND REPORTING

Revisions and updating. Clearly mention the date and/or version number of the application/intervention (and Highly Recommended comparator, if applicable) evaluated, or describe whether the intervention underwent major changes during the evaluation process, or whether the development and/or content was “frozen” during the trial.

EVALUATION OF CLINICAL OUTCOMES• Use standard evaluation methods

• Assuming changes improve primary targets and do not open alternative interpretations, this would provide a conservative estimate.

• May wish to examine if substantive changes are associated with improved use or outcomes.• Include a time varying ordinal component.

Decisions with Uncertainty

• Uncertainty about 2 or more solutions (optimization problem).• Adaptive Randomization using reinforcement

learning techniques (Q-learning).

Trial within a trial

PRINCIPLED TRIAL

• If viewed as an optimization problem (incorporate learning into trial), accuracy-centric framework and type I error rate not appropriate. • P-values closer to .50

may provide reasonable confidence for optimization but would not permit generalizability.

End of Tx

RR

ControlControl

BITBIT

Clinical AimsClinical Aims

Beh Strategies Beh Strategies

BIT ElementsBIT Elements

CharacteristicsCharacteristics

WorkflowWorkflow

CC

CC

CC

EE

CC

Cheung K, Duan N.. Am. J. Public Health. 2014;104:e23-e30.

SUMMARY• Principled Trials

• Define principles (Behavioral and technical)• Decision making methods• Documentation & Reporting• Evaluation

• Limitations• Does not validate a specific app• Risk of unintended consequences

• Advantages• Consistent with what BITs are• Learning during trial• Cost effective knowledge generation• Produces more generalizable data

WLSACONVERGENCE SUMMIT

www.wirelesshealth2014.org

WH2014 Workshop: mHealth Evidence

Healthcare

Transcript of WH2014 Workshop: mHealth Evidence