David Chambers and Enola ProctorJune 7, 2017

1

Strategies and Outcomes:Measurement in D & I Research

Strategies & measuresYour university published an article bragging about your selection to MT-DIRC training at Washington University. The next day, you received a phone call from a clinic director in a nearby city. She informs you that the city’s consortium of three Federally Qualified Health Centers wants to improve their implementation of colon cancer screening guidelines, and they believe that you can offer good advice. She asks you to come to a meeting with the agency directors next week to share your suggestions.

June 8, 2016 2

She tells you:

• Clinic A is already doing very well with the guideline implementation and their director has volunteered their agency as a model for the other two.

• Clinic B is receiving staff pushback because they feel overloaded with other programs and don’t know what approach could possibly help.

• Clinic C has a mixed track record with implementing breast cancer screening, and the director doubts that these kinds of guidelines are very effective in the first place.

3June 8, 2016

DiscussionGroup 1 Group 2 Group 3How might you learn about the strategies Clinic A has used thus far in implementing evidence-based programs and interventions in cancer prevention and control?

What questions would you ask Clinic A about their approach?

Which implementation outcomes might be important, and which would you want to explore for potential measurement?

Would you assess the same implementation outcomes in each clinic?

If not, which implementation outcomes would you assess for which clinic?

What additional information would you want to learn about to inform your discussion with Clinic C, and why?

4June 8, 2016

Implementation OutcomesFeasibility

FidelityPenetration

AcceptabilitySustainability

UptakeCosts

*IOM Standards of Care

Conceptual Model for Implementation Research

What?

QIsESTs

How?

ImplementationStrategies

Implementation Research Methods

ServiceOutcomes*Efficiency

SafetyEffectiveness

EquityPatient-

centerednessTimeliness

Patient Outcomes

Clinical/health status

SymptomsFunction

Satisfaction

Proctor et al 2009 Admin. & Pol. in Mental Health Services

CONTEXT

CONTEXT

CON

TEXT

CON

TEXT

The UsualThe Core of

Implementation Science

5June 8, 2016

IMPLEMENTATION STRATEGIES

Key constructs

6June 8, 2016

Implementation Strategies:Definition

Systematic intervention process to adopt and integrate evidence-based healthcare innovations into usual care *

Active ingredient in processes for moving EST’s and QI’s into usual care

*Powell, McMillen, Proctor et al., Medical Care Research and Review, 2012

7June 8, 2016

Implementation Strategies:Complexity*

Discrete• involve one process or action, such as “meetings,”

“reminders”Multifaceted**• uses two or more discrete strategies, such as “training +

technical assistance”Blended• several discrete strategies are interwoven & packaged as

protocolized or branded strategies, such as “ARC,” IHI Framework for Spread”

*Powell, McMillen, Proctor et al., 2012 ** Grimshaw et al., 2001, Grol & Grimshaw, 2003

8June 8, 2016

68 “discrete” strategies in 6 categories:• Plan• Educate• Finance• Restructure• Manage quality• Attend to policy context

Strategies Compilation

Train and educate

stakeholders

Change infrastructure

1

23

283442 49

66&709

10

111213

22

44

4

5

14

18

23

61

26

27

46

56

37

39

41

50

6269

6

7

1740

47

5264

48

5163

67

68

24

25

36 38

45

65

72

3557

8

33

53

54

58

21

30

32

59

15

1629

60

1920

3143

55

7173

Develop stakeholder interrelationships

Provide interactive assistance

Utilize financial strategies

Engage consumers

Use evaluative and iterative strategies

Adapt & tailor to context

Support clinicians

Identification of Implementation Strategies: ERIC Cluster Solution

Current NIH funded research:Evidence-based practices

vs. Implementation StrategiesInstituteMechanismPI

Evidence-basedintervention

ImplementationStrategy

NIMHR01Dr. Yuhua Bao

Collaborative Care Model in community health clinics

Value-Based Purchasing (apolicy-level strategy)

NIMHR01Dr. Martha Bruce

Depression treatment in home health agencies

Web-based training, supervision and consultation with audit and feedback

NIDAR01Dr. Todd David Molfenter

Use of buprenorphine in addiction treatment organizations in Ohio

Clinician training and payment policy changes

Current NIH funded research:Evidence-based practices vs.Implementation Strategies

InstituteMechanismPI

Evidence-basedPractices

ImplementationStrategy

NIAIDR01Dr. John Lucian Davis

Contact Investigation for TB detection in Uganda

Mobile-phone health text messages and incentives

NIDDKR01Dr. Chester Fox

Evidence-based care of chronic kidney disease in primary care

Computer Decision Support tool, facilitation including physician champions and audit and feedback

NHLBIR01Dr. James Herbert

Evidence-based diet and physical activity intervention in churches

CBPR with train-the-trainer

Specifying Implementation strategies

Conceptually defined consistent with literatureOperationally defined

13

Actor Who?

Action Does what?

Action target(s) At what part of the context? (intervention? Organization? Providers?)

Dose How much?

Temporality When? In what order or sequence?

Implementation Outcome addressed

To achieve what implementationoutcome?

Justification Theoretical or empirical rationale?June 8, 2016

Example:strategies & their specification

June 8, 2016 14

Implementation strategy:Provision of intensive implementation support

15

Strategy specificationActor Practice facilitators

Action Provide formal clinic-wide stafftraining on intervention components andunderlying evidence (often in conjunction with clinic champion)

Action target(s) Build knowledge and acceptance of theintervention among clinic staff

Dose 1-h meeting at each clinic

Temporality Annually at start of implementationyears 1 and 2

Implementation Outcome addressed

Improved staff trust, understanding,uptake of intervention

Justification Evidence strength and quality, relativeadvantage, staff engagement,knowledge and beliefs, self-efficacyJune 8, 2016

Measuring implementation strategies

• Exposure:– Observational – Controlled

• Data types and sources– Key informants

• Interviews, focus groups, checklists– Direct observation

• Participant observation– Archival

• Tx disseminator records• Implementing site records (meeting notes, memos, policies)

16June 8, 2016

IMPLEMENTATION OUTCOMES

Key constructs

17June 8, 2016

Implementation Outcomes:Measure distinct from clinical

outcomesCould have an effective intervention:Not adoptedPoor penetration in health systemImplemented but

with poor fidelityNot sustained

18June 8, 2016

Implementation Outcomes: Key Concepts

• Acceptability• Adoption• Appropriateness• Feasibility• Fidelity• Implementation cost• Penetration• Sustainability

19June 8, 2016

Dissemination and Implementation Research in Health

PAR # 16-236; 237, 238Primary purpose: identify, understand, & develop • Strategies

for the • Adoption, adaptation, integration, scale-up,&

sustainability----these are implementation outcomes

• Of EB interventions, tools, policies, & guidelines

From PAR 16-238

• Dissemination research outcomes:– spread and sustain knowledge– reach many different stakeholders

• Implementation research outcomes:– adopt and integrate evidence-based health

interventions into clinical and community settings– adoption, implementation and sustainability

21

Outcome Issues

• What are the right outcomes?• Which outcomes are relevant to particular settings,

stakeholders, situation? (context, coming up)• Realistic timing• How to intervene to achieve the outcome

– What are the barriers to achieving the outcome?– What conceptual models/ frameworks are applicable to

which outcomes?– What implementation strategies are effective in achieving

which outcome?• Validity and reliability

June 8, 2016 22

Evidence-based intervention/ program/guideline

Implementation outcomes

Service system outcomes

Clinical outcomes NIH study(PI, University)

Rapid ART initiation for pregnant mothers for PMTCT of HIV

Acceptability, Feasibility, Cost-effectiveness

Rate of retention in care through 6mo

Viral suppression at time of delivery; MTC transmission at 6mo post-partum;

R01 HD074558(Abrams, E.; Columbia University)

Mobile computer-based video intervention to increase HIV testing rates in ED

Acceptability, Feasibility

None specified

Post-intervention HIV testing rate (RCT; 4 groups)

R34 DA037129 (Aronson, I.; National Development and Research Institutes)

Tobacco use treatment guidelines

Fidelity, Cost, Organizational readiness to implement change, Provider adherence to guidelines

None specified

Smoking abstinence at 6mo

R01 CA175329 (Shelley, D.; NYU School of Medicine

Evidence-based intervention/ program/guideline

Implementation outcomes

Service system outcomes

Clinical outcomes NIH study(PI, University)

Cognitive Processing Therapy for PTSD

Fidelity, adaptation, penetration

Capacity to deliver CPT

PTSD symptom change

R01 MH106506 (Wiltsey Stirman, S.; Palo Alto Veterans Institute for Research

SafeCare w/Technological Enhancement (compared with SafeCare as usual)

Feasibility, Acceptability, Fidelity

Perceived job demands & resources

Parent and child behavior change, Client satisfaction

R21 MH098244 (Self-Brown, S.; Georgia State University)

FAST TB Transmission Control in Hospitals

Acceptability, Barriers to use, Cost, Cost-effectiveness, Reach, Adoption

None specified Reduction in hospital worker TB infections; Reduced time to treatment

R01 AI112748 (Nardell, E.; Brigham & Women’s Hospital)

June 8, 2016 25

Bryan Weiner (PI): Increasing Colorectal Cancer Screening Rates in Community Health Centers, R21 CA161657

June 8, 2016 26

Donna Shelley(PI): Implementing Tobacco Use Treatment Guidelines in Community Health Centers in Vietnam, R01 CA175329

June 8, 2016 27

Ross Brownson(PI): Disseminating Evidence-Based Interventions to Control Cancer, R01 CA 160327

June 8, 2016 28

Ross Brownson(PI): Disseminating Evidence-Based Interventions to Control Cancer, R01 CA 160327

Measurement: Toward Standardization & Harmonization

• Society for Implementation Research Collaboration Instrument Review Project– http://www.societyforimplementationresearchcoll

aboration.org/sirc-projects/sirc-instrument-project/

• Grid-Enabled Measures developed by the National Cancer Institute– http://cancercontrol.cancer.gov/brp/gem.html

29June 8, 2016

Three primary outcomes for this project series include:(1) a comprehensive library of D&I instruments measuring the implementation outcomes identified by Proctor and colleagues (2010) and organized by the Consolidated Framework for Implementation Research (CFIR; Damschroder et al., 2009) for SIRC members;(2) a rating system reflecting the degree of empirical validation of instruments, adapted from the evidence-based assessment (EBA) work of Hunsley and Mash (2008) and Terwee et al (2012);(3) a consensus battery of instruments decided upon by D&I expert task force members using the EBA criteria to guide future D&I research efforts.

An update on SIRC Instrument Review Project Progress (10/15/14)We have now identified nearly 450 instruments to be included in our repository. The instruments assessing Implementation Outcomes (Proctor et al., 2009) have all been twice rated using our evidence-based assessment rating criteria. Results are available throughthe IRP portion of the website. The repository is available to all SIRC memberswww.societyforimplementationresearchcollaboration.org

THE SIRC INSTRUMENT REVIEW PROJECT

http://www.societyforimplementationresearchcollaboration.org/sirc-projects/sirc-instrument-project/

June 8, 2016 32https://implementationscience.biomedcentral.com/articles/10.1186/s13012-016-0401-y

Context informs strategy selectionBarriers assessment

Flottorp, S.A (2013) A checklist for identifying determinants of practice: A systematic review and synthesis of frameworks and taxonomies of factors that prevent or enable improvements in healthcare professional practice. Implementation Science 8:35

Antecedents for the interventionEmmons, K. M., Weiner, B., Fernandez , M.E., & Tu, S.

(2012), Systems Antecedents for Dissemination and Implementation : A Review and Analysis of Measures, Health Educ Behav 39: 87

33June 8, 2016

Take home points

• Implementation involves distinct– Strategies– Outcomes

• Measurement is evolving• Harmonization and standardization enhances

quality and rigor for implementation science

34June 8, 2016

Support: National Institute of Mental Health

P30 MH068579R25 MH080916P30 DK092950U54 CA155496UL1 RR024992 (Clinical and Translational Science Award, CTSA)

Washington University Institute for Public HealthBrown School of Social Work

Conflicts: none35

How to reach us

Questions…….????????

Enola Proctorekp@wustl.eduDavid Chambers

dchamber@mail.nih.gov

36June 8, 2016