Measuring Consumer Perception of Care

Challenges & Opportunities

John Bartlett, M.D.,M.P.H.March 20-21 Meeting with California’s Division of Alcohol and Drug Programs

Goals for Presentation Review consumer perception of care as a

concept

Review success criteria for its measurement

Review the development & testing of CSAT’s Modular Survey

The Concept

Measuring consumer response =

core business function In healthcare tied to growth of consumerism

& CQI initiatives A NOMS domain Consumer perception of care ≠ satisfaction

Measuring Consumer Perception of Care

Approaches to measurement differ

depending on scope & purpose For purposes of comparability, improvement

over time, & benchmarking measures must be :

Meaningful Scientifically-sound Actionable

The Problems with Satisfaction

No evidence linking the measurement of

satisfaction to client outcomes Few satisfaction surveys scientifically

validated Data is not actionable (ceiling effect)

The Modular Survey

SAMHSA –supported initiative Conducted under the auspices of the Forum

on Performance Measurement &

the Washington Circle Conducted in 2 phases

Phase 1 in conjunction with mental health Rx Phase 2 substance abuse – specific

Design Requirements

Phase 1 focus on commonality,

not comprehensiveness Short Scientifically sound Actionable Use of existing, widely-used, non-proprietary

surveys Consensus-driven

“Field-level”CommonMeasures

Adult Common Measures

Child/AdolescentCommon Measures

AdultMental Health

Core Measures

Adult Substance

Abuse Core Measures

AdolSubstance

AbuseCore Measures

C/AdolMental HealthCore Measures

Common DesignTemplate

Modular Survey Flow of Common Questions for Individual Respondent

Approach to Phase 1

4 workgroups to develop consensus Selection of instruments Identification of concerns Identification of potential items Ranking of items Final item selection (modified Delphi) Pilot testing

Phase 1 Pilot Testing

Conducted during summer/fall 2004 Primary data collection in Cincinnati

United Way agencies (N = 1157) Secondary analysis using MHSIP data

(16 state & LA County data sets) Final N > 22,000 respondents Pool of items reduced from 28 to 11 All items common to both fields, both populations

Approach to Phase 2

Stand-alone SUD Rx initiative Under Washington Circle with Forum as

“subcontractor” New item development (no existing SUD survey) Content work group co-chaired by

Tom McLellan (TRI) & Doreen Cavanaugh (Georgetown)

Support from Forum Methods Work Group

& Ann Doucette (George Washington Public Provider & Consumer Advisory Groups

Phase 2 SUD Initiative

Closely coordinated with NOMS Identification of concerns

Relationship to treatment program Self awareness of problem/commitment to

change Perceived outcomes Social connectedness

Generation of items (35 in testing pool)

Phase 2 Pilot Testing OMB & IRB approval spring 2006 Conducted in 3 rounds

Round 1 – Adult & Adolescent (summer 2006) 14 programs, N = 1207

Round 2 – Adult & Adolescent (winter 2006 – 07) 6 programs, N = 585

Round 3 – Adolescent (spring-summer 2007) 8 programs, N = 268

Final adult N = 1549 (2 samples) Final adolescent N = 492 (1 sample) All demographic groups covered except Native American

Phase 2 Completion

Analysis & recommendations by Ann Doucette PH.D. Use of IRT For a copy of the technical report, e-mail

jbartlett@avisagroup.com Review by Forum Methods Work Group

(November 2007) Review & Approval by SUD Content Committee

(November 2007)

Final SUD Modular Survey

21 items (11 from Phase 1, 10 from Phase 2) Quality – 6 items Perceived Outcomes – 6 items Social Connectedness – 7 items Commitment to Change – 2 items

10 demographic & background items Spanish translation available

Modular Survey Flow of Common Questions for Individual RespondentFinal Version

Phase 1Common Items *

Phase 2MentalHealth Items

Phase 2SUD

Items

Common DesignTemplate

* All populations, all fields

Convergence with NOMS

In Spring 2006 NOMS Technical Consulting Group convened

Recommended 17 items from 8 different instruments

9 of the 17 from the Modular Survey 5 are in the Final Modular Survey

In summary……

Consumer perception of care key measurement domain

Its measurement must meet certain criteria in order to be worth the effort

The Modular Survey is the only current instrument measuring consumer perception of care that is:

SUD Rx – specific Product of both consensus and empirical

analysis Short and actionable