Substance Abuse, Multi-Stage Decisions, Generalization Error

How are they connected?!

S.A. Murphy

Univ. of Michigan

CMU, Nov., 2004

Outline

• Chronic Disorders

• Multi-Stage Decisions & Reinforcement Learning

• Statistical Challenges

• Generalization Error

• Discussion

Chronic Disorders

Why does the management of a chronic disorder such as drug dependence require multi-stage decisions?

•High variability across patients in response to any one treatment

•No Cure •Relapse is likely without either continuous or intermittent treatment for a large proportion of people.

•What works now may not work later•Exacerbations in disorder may occur if there are no alterations in treatment

Additional Issues in Managing the Chronic Disorder

•Treatment is often burdensome, especially over time

•Patient adherence is a critical issue

•Co-occurring problems are common

High-Level Questions

•What is the best sequencing of therapies?

•What is the best timings of alterations in therapies?

•What information do we use to make these decisions?

Multi-Stage Decisions

&

Reinforcement Learning

k Decisions on one individual

Observation made prior to jth decision point

Decision at jth decision point

Primary outcome Y is a specified summary of decisions and observations

Goal: Construct decision rules that input data at each decision point and output a recommended decision; these decision rules should lead to a maximal mean Y.

where each is a low dimensional summary of

for j=1,…., k

An example of a decision rule is:

alter treatment if

otherwise maintain on current treatment.

Unknown UnknownCauses Causes

Observed Therapy 1 Observed Therapy 2 ObservedVariables Outcomes Outcomes

Conceptual Formulation

Statistical Challenges

Statistical Challenges

•Construction of summaries useful for decision making•High dimensional noisy information•In many cases summary must be meaningful

•Construction of decision rules•Variety of data sources•In many cases decision rule must be meaningful

•Experimental designs•Evaluation versus construction/refinement of decision rules.

Experimental Design

•Adaptive Treatment Strategies are multi-component treatments

•Multiple decision points through time•Different kinds of decisions•Decision options for improving patients are often different from decision options for non-improving patients

•Services Research•Delivery Mechanisms•Training of Staff…….

Unknown UnknownCauses Causes

Observed Therapy 1 Observed Therapy 2 ObservedVariables Outcomes Outcomes

Conceptual Formulation

Goal: Provide experimental designs for developing or refining adaptive treatment strategies

•Advocating for a series of developmental, experimental trials

--- Collins, Murphy, Nair & Strecher (2004)

•Designs similar to full factorial designs--- Lavori & Dawson (2004), Lavori (2001)--- Murphy (2004)

•Need designs that are similar to balanced fractional factorials

Examples of sequential multiple assignment randomized trials:

•CATIE (2001) Treatment of Psychosis in Alzheimer’s Patients

•CATIE (2001) Treatment of Psychosis in Schizophrenia

•STAR*D (2003) Treatment of Depression

•Oslin (ongoing) Treatment of Alcoholism

Construction of decision rules

•Data: n (patient) trajectories of observations and decisions---from one patient we see:

•Decisions are randomized among feasible options

•Y is known summary of all decisions and observations

•Estimate decision rules so as to maximize mean of Y

Unknown UnknownCauses Causes

Observed Therapy 1 Observed Therapy 2 ObservedVariables Outcomes Outcomes

Data Structure (k=2)

Some Methods

• Q-Learning (Watkins, 1989) (one of many……many methods from reinforcement learning)

---regression

• A-Learning (Murphy, 2003; Robins, 2004; Blatt et al. 2004)

---regression on a mean zero space

• Weighting (Murphy, van der Laan & Robins, 2002)

---weighted mean

Generalization Error

Goal: Find the decision rules that are best (maximize mean Y) within a restricted class of decision rules:

e.g.,

One decision only!

Data:

is randomized with probability

The estimand is

or equivalently

A Direct Approach

But not feasible, particularly if more than one decision!!

Q-LearningApproximate

Minimize

Even if our sample is infinite,

is not necessarily close to

The difference is the generalization error.

The message:

In adding information we changed our goal. Our

estimator will not (even asymptotically) achieve

equivalently

• Can we add information without changing our goal?– Can we “guide” Q-learning, A-learning and

other methods closer to our goal?– Do we need a theory that compares biased

estimators?– Is there a way to use a sieve without changing

our goal?

Discussion

Open Problems

• How might we form summaries of high dimensional noisy data so as to make good decisions? (Prediction is not the goal)

• How might we use structural models to inform the design of randomized experiments?

• How should we design experiments if our goal is building or refining an adaptive treatment strategy?

Open Problems

• How might we use observational data to estimate good adaptive treatment strategies (e.g. decision rules)?

• How might we use data in which an adaptive treatment strategy was implemented to improve the decision rules?

• How might we “guide” Q-Learning or A-Learning so as to more closely achieve our goal?

• How might Bayesian methods be used here?

This seminar can be found at:http://www.stat.lsa.umich.edu/~samurphy/seminars/cmu1104.ppt

Further information on adaptive treatment strategies can be found at:http://neuromancer.eecs.umich.edu/dtr/

Q-Learning (k=2)

Data:

Q-Learning

Minimize

Minimize

Sequential Multiple Assignments

Initial Txt Intermediate Outcome Secondary Txt

Monitor +

Responder R counseling

Monitor

Med B

Med A

Nonresponder REM + Med B+ Psychosocial

R

Responder Monitor +

R counseling

Monitor

Med A + Psychosocial Med B

Nonresponder R

EM +Med B+Psychosocial