Measurement and analysis of provider behaviour change in knowledge translation research: Is there a best practice?

2nd National Knowledge Translation Conference in Rehabilitation, April 19, 2018

Nancy Salbach, PT, PhDCIHR New Investigator in Knowledge Translation

Associate Professor, Department of Physical Therapy, University of Toronto

Presentation Objectives:

1. To consider the advantages and limitations of methods for measuring provider behaviour change following a KT intervention

2. To understand which quantitative approaches to measuring provider behaviour change may lead to clustered data and how to evaluate clustering in data analysis

2

Outcomes of KT studies in RehabilitationSystematic review: Jones et al 2015

3

More than 75% of KT intervention studies* in rehab target professional or process outcomes

*provider targeted: PT (13 studies); OT (7 studies); SLP (3 studies); OT&PT (3 studies)

Systematic review: Jones et al 2015 (n=26 studies)

4

Why professional or process outcomes?• Ultimate aim = improved health of Canadians =

improved patient function in rehabilitation setting

• KT interventions targeting multidisciplinary team

• Patient function = primary outcome

• Provider practice = intermediate ‘process’ outcome

• KT interventions targeting uniprofessional practices

• Provider practice = primary outcome as patient function influenced by practices of “other” providers

5

Most professional-level outcomes measured by provider self-report

Systematic review: Jones et al 2015

6

Warning! Brain squeeze

What are the pros and cons of different methods for measuring behaviour change in rehabilitation providers following KT interventions?

7

https://imgflip.com/meme/58388563/

Thinking-hard

Provider Self-report

Pros

• Feasible

• Inexpensiveif online

• Data on large group

Cons

• Measurement at one point in time

• Attrition over time

• Social desirability bias

• Item reliability?

8

Patient Self-report

Pros

• Inexpensive if online

• Can capture practice over time

Cons

• Privacy - Difficulty obtaining patient emails

• Recall

• Expense - phone/mail survey

• Low response rate

• Is response linked to health?9

Interviews & Focus Groups

Pros

• Deep understanding of mechanisms of behaviour change

Cons

• Interviews=resource intensive

• Focus groups: scheduling and opportunity to speak

• Limited for large groups10

Observing Provider Practice

Pros

• Observer unbiased

• Range of patients

Cons

• Expensive, resource intensive

• Representativeness depends on duration of observation

• Observation may influence practice11

https://www.smartcherry.in/your-

observation-power-smart-cherry/

Health record review

Pros

• Retrospective: Less vulnerable to bias

• Range of patients

Cons

• Inconsistent documentation

• Expensive, resource intensive

• If prospective, providers can change documentation behaviour

12

http://www.bcm

j.org/mds-

be/merit-and-

agony-

retrospective-

chart-reviews-

medical-

student%E2%8

0%99s-

perspective

Now a word about clustering

• Clustering is when people or data in your study form natural groupings because the people or data within a group are more related to each other than they are to people or data outside the group

• Can you think of examples of clustering in rehabilitation?

13

Clustering of Providers within HospitalsEg. Guideline implementation study, examine provider use of recommendations in 3 rehab hospitals

14

Hospital 1 Hospital 2 Hospital 3

Clusters

Clustering of Patients within ProvidersEg. Guideline implementation study, examine provider use of recommendations with each patient

15

Provider 1 Provider 2

Patients Patients

Provider 3

Patients

Clusters

Clustering of Measurements within PatientsEg. Guideline implementation study, examine provider use of recommendations across sessions with same patients

16

Patient 1 Patient 2

Data points Data points

Patient 3

Data points

Clusters

Multi-level Clustering

Data

points

Hospital

Hospital 2 Hospital 3Cluster Level

Patients

Provider

What happens when you don’t account for clustering?

• It depends on whether clustering has an influence

• If clustering influences your study outcome, and you don’t account for clustering in the analysis, you will likely get the wrong answer

• Let’s take a look at which level of clustering was important in a guideline implementation trial

18

Stroke Canada Optimization of Rehabilitation through Evidence-Implementation Trial (SCORE-IT)

• National cluster randomized guideline implementation trial

• 20 stroke inpatient rehabilitation centres randomized to a multi-component facilitated or passive KT intervention for guideline implementation

• Primary outcome was patient function: Effect on walking capacity (6-minute walk test) but no effect on UE function (Box & Block test) observed (n=777)

• Examining implementation of recommendations was planned to help explain results related to patient outcomes

19

Hypothesis

• The rate of implementing 18 selected treatment recommendations would improve from pre- to post-intervention to a greater extent in the active, facilitated KT intervention group than in the passive KT intervention group

20

Measurement of Implementation of 18 SCORE Treatment Recommendations

• Self-reported implementation (yes/no) using checklists

• PTs, OTs, nurses asked to complete after every patient seen in a 2-week period, pre- & post-intervention

• Intervention period: 16 months

21

Warning – This is a test! In the SCORE-IT study…

Present Level of Clustering

Clusters of providers within hospitals?

Clusters of patients within providers?

Clusters of measurements within

patients? 22

Analysis plan for each of 18 outcomes• Logistic Regression with random effects modeling (can account

for clustering)

• Tested for clustering effect at site, provider and patient levels

• Estimated the rate of adherence to each treatment recommendation pre- and post-intervention

• Compared the change from pre- to post-intervention between groups using an interaction term: group x time

• Analysis conducted with and without adjustment for clustering and covariates (eg patient motor function)

23

Significant clustering effects at Provider (67%) and Patient (39%) levels only

Treatment Site Provider Patient

1. Sit-to-stand Yes Yes2. LE ROM and/or stretching Yes3. Use of LE external support Yes Yes4. Task-specific training (i.e., stairs) Yes5. Training for sitting balance Yes6. Training for standing balance Yes7. FES for the LE

8. Walking practice Yes Yes9. Treadmill walking practice

10. UE ROM and/or stretching Yes Yes11. Interventions to prevent shoulder pain Yes Yes12. Task-specific training (i.e., self-care tasks) Yes Yes13. Techniques to reduce hand edema Yes14. Ice/heat or soft tissue massage for shoulder

15. FES for wrist/ arm/shoulder

16. Educate patient or caregiver on how to handle

arm or shoulderYes

17. UE constraint-induced therapy

18. Visual imagery to enhance arm recovery Yes

24

Between-group analysis – UE treatments: No effect before/after adjustment for clusteringTreatment Effect (ChangeE-ChangeC)

% (95% CI)

Effect

Regression Estimate (p-value)

Unadjusted Adjusted for

Clustering

Adjusted for Clustering

and Covariates

UE ROM/Stretching 5.2 (-4.4, 14.7) 0.1 (0.876) 0 (0.968)

Interventions to prevent shoulder

pain

4.9 (-5.1, 14.9) 0.8 (0.343) 0.8 (0.269)

Task-specific UE training 6.0 (-5.4, 17.3) 0.5 (0.237) 0.5 (0.237)

Techniques to reduce hand

edema

0 (-9.6, 8.8) -0.1 (0.734) -0.1 (0.759)

Shoulder ice/heat/massage 4.8 (-2.3, 11.8) 0.8 (0.487) 0.9 (0.432)

FES for UE 0.5 (-4.1, 5.1) DNC 1.7 (0.382)

Educate patient/caregiver 3.4 (-3.9, 10.6) 0.5 (0.718) 0.2 (0.916)

Constraint-induced therapy 1.8 (-5.6, 9.2) -0.8 (0.596) -0.9 (0.606)

Visual imagery 3.5 (-3.7, 10.8) 0.4 (0.381) 0.5 (0.338)

25

Between-group analysis – LE-Posture-Mobility Treatments: 2 of 5 effects no longer significant after adjustment for clustering

TreatmentEffect (ChangeE-ChangeC)

% (95% CI)

Effect

Regression Estimate (p-value)

UnadjustedAdjusted for

Clustering

Adjusted for

Clustering and

Covariates

Sit to stand 21.4 (10.4, 32.4) 1.1 (0.035) 1.0 (0.028)

Walking practice 22.0 (11.2, 32.8) 1.0 (0.029) 1.0 (0.043)

LE ROM/Stretching -14.7 (-26.0, -3.5) -1.1 (0.093) -1.1 (0.106)

Training for sitting

balance -14.6 (-28.2, -1.0) -0.8 (0.125) -0.9 (0.123)

Training for

standing balance -22.8 (-39.4, -6.2) -1.0 (0.040) -1.1 (0.037)

26

Take home messages

• Self-report methods to evaluate provider behaviour are common in KT intervention studies

• Many methods to evaluate provider behaviour:• Consider whether method captures practice in a range patients

over time, vulnerability to bias, feasibility, and cost

• Understand how limitations influence interpretation of results

• Plan to adjust for important covariates (patient, provider, hospital) based on theoretical or empirical evidence

• Qualitative and quantitative research helps us understand what influences provider practice 27

Campbell et al 2012; Campbell et al 2007

Take home messages

• Identify levels of clustering in design phase; review reporting guidelines for cluster RCTs

• In context of stroke rehabilitation guideline adoption:• Effect of patients clustered within provider has strongest

impact on estimated use of best practices, followed by data clustered within patients

• Hospital site does not appear influential

• Consult with a statistician re: impact of clustering and covariate adjustment on sample size estimation; plan to evaluate clustering in analysis

28

SCORE-IT Research TeamMark Bayley, MD, Toronto Rehabilitation Institute-University Health Network

Sharon Wood-Dauphinee, PhD, McGill University

Susan Barreca, PT, MSc

Sandra Black, MD, Sunnybrook Health Sciences Centre

Lucie Brosseau, PhD, University of Ottawa

Johanne Desrosiers, PhD, Université de Sherbrooke

Janice Eng, PhD, University of British Columbia

Ian Graham, PhD, University of Ottawa

Jeremy Grimshaw, PhD, University of Ottawa

Margaret Harrison, PhD, Queen’s University

Jeffery Hoch, PhD, University of Toronto

Susan Jaglal, PhD, University of Toronto

Nicol Korner-Bitensky, PhD, McGill University

Marilyn MacKay-Lyons, PhD, Dalhousie University

Nancy Mayo, PhD, McGill University

Carol Richards, PhD, Laval University

Monica Taljaard, PhD, Ottawa Health HealthResearch Institute

Robert Teasell, MD, Western University

Merrick Zwarenstein, MBBCh, PhD, Western University

SCORE-IT Coordinators:

Amanda Hurdowar

Jennifer Mokry

Olga Yaroslavtseva

Marion Hau

29

SCORE-IT Acknowledgements

We would like to acknowledge the hospitals and thank the many healthcare professionals and people with stroke who participated

Funding Participants

Thank you!

Nancy Salbach, PT, PhDnancy.salbach@utoronto.ca

@nancysalbachKnowledge to Action Lab: http://www.physicaltherapy.utoronto.ca/research/ktalab/

32

CONSORT Diagram

33

Clustering EffectsTreatment Intracluster Correlation Coefficient

Site Provider Patient

1. Sit-to-stand 0.06 0.26* 0.10*

2. LE ROM and/or stretching 0.00 0.30* 0.04

3. Use of LE external support 0.11 0.10* 0.23*

4. Task-specific training (i.e., stairs) 0.00 0.35* 0.00

5. Training for sitting balance 0.01 0.16* 0.09

6. Training for standing balance 0.02 0.23* 0.08

7. FES for the LE 0.12 0.00 0.00

8. Walking practice 0.02 0.31* 0.11*

9. Treadmill walking practice 0.03 0.19 0.17

10. UE ROM and/or stretching 0.08 0.32* 0.06*

11. Interventions to prevent shoulder pain 0.08 0.16* 0.21*

12. Task-specific training (i.e., self-care tasks) 0.05 0.28* 0.06*

13. Techniques to reduce hand edema 0.06 0.06 0.22*

14. Ice/heat or soft tissue massage for

shoulder 0.14 0.16 0.20

15. FES for wrist/ arm/shoulder 0.24 0.16 0.04

16. Educate patient or caregiver on how to

handle arm or shoulder 0.05 0.25* 0.07

17. UE constraint-induced therapy 0.11 0.19 0.14

18. Visual imagery to enhance arm recovery 0.00 0.31* 0.04

*Significant clustering effect at Provider (67%) and Patient (39%) levels only.

Cluster sizes at the site, provider and patient level by study group and sampling time pointCluster Cluster Size Mean (SD)

Level 7 Treatments Implemented by RNs, OTs, and PTs

11 Treatments Implemented by OTs and PTs

Facilitated Passive Facilitated Passive

Pre Post Pre Post Pre Post Pre Post

Site:

Providers per site

15 (8) 13 (6) 10 (7) 13 (7) 6 (3)* 6 (3)* 5 (4)* 7 (4)*

Provider:

Patients per provider

3 (3) 2 (1) 2 (1) 2 (1) 2 (1) 2 (1) 2 (1) 2 (1)

Patient:

Forms per patient

13 (13) 7 (3) 6 (4) 6 (3) 4 (2) 4 (1) 4 (2) 3 (1)

*Smaller provider clusters per site for treatments that fewer health professional groups were asked

to implement.

35

Baseline Implementation of Recommended Treatments for LE/Mobility Function (unadjusted)

Treatment % of Times Implemented (95% CI) Level

n Facilitated KT n Passive KT

Sit to stand 647 20.4 (17.3, 23.5) 193 36.3 (29.5, 43.1) >15%

LE ROM/Stretching 151 15.9 (10.1, 21.7) 118 8.5 (3.4, 13.5) >15%

Use of external LE support 647 7.3 (5.3, 9.3) 193 15.0 (10.0, 20.1) <15%

Task-specific LE training 151 31.8 (24.3, 39.2) 118 26.3 (18.3, 34.2) >15%

Training for sitting balance 151 23.8 (17.0, 30.7) 118 17.0 (10.2, 23.7) >15%

Training for standing balance 151 51.7 (43.7, 59.6) 118 36.4 (27.7, 45.1) >15%

FES for LE 151 0.7 (-0.6, 2.0) 118 0 (0, 0) <15%

Walking practice 647 15.9 (13.1, 18.7) 193 31.6 (25.0, 38.2) >15%

Treadmill walking 151 2.7 (0.1, 5.2) 118 6.8 (2.2, 11.3) <15%36

Within-Group Unadjusted Change: Improved (green) and reduced (red) for LE/MobilityTreatment No.

forms

Pre to Post Change in % of Times Implemented (95% CI)

Facilitated KT Passive KT

Sit to stand 1381 18.7 (12.2, 25.3) -2.7 (-11.5, 6.2)

LE ROM/Stretching 547 -5.4 (-13.1, 2.3) 9.3 (1.1, 17.5)

Use of external LE support 1381 1.4 (-2.5, 5.3) 2.3 (-4.5, 9.1)

Task-specific LE training 547 6.7 (-4.3, 17.6) 11.5 (0.1, 22.9)

Training for sitting balance 547 -6.4 (-15.6, 2.9) 8.2 (-1.8, 18.2)

Training for standing balance 547 0.8 (-10.7, 12.2) 23.6 (11.6, 35.6)

FES for LE 547 0 (-1.9, 1.9) 0.7 (-0.7, 2.2)

Walking practice 1381 23.2 (16.8, 29.6) 1.2 (-7.4, 9.9)

Treadmill walking 547 -1.3 (-4.5, 2.0) -1.6 (-7.5, 4.3)

37

Baseline Implementation of Recommended Treatments for UE Function (unadjusted)

Treatment % of Times Implemented (95% CI) Level

n Facilitated KT n Passive KT

UE ROM/Stretching 647 12.7 (10.1, 15.2) 193 21.8 (15.9, 27.6) >15%

Interventions to prevent

shoulder pain647 25.0 (21.7, 28.4) 193 25.4 (19.2, 31.5) >15%

Task-specific UE training 647 28.9 (25.4, 32.4) 193 37.3 (30.5, 44.1) >15%

Techniques to reduce hand

edema151 7.3 (3.1, 11.4) 118 10.2 (4.7, 15.6) <15%

Ice/heat or soft tissue massage

for shoulder151 1.3 (-0.5, 3.2) 118 8.5 (3.4, 13.5) <15%

FES for UE 151 2.0 (-0.2, 4.2) 118 2.5 (-0.3, 5.4) <15%

Educate patient/caregiver how

to handle UE647 8.8 (6.6, 11.0) 193 13.0 (8.2, 17.7) <15%

Constraint-induced therapy 151 4.6 (1.3, 8.0) 118 10.2 (4.7, 15.6) <15%

Visual imagery 151 2.7 (0.1, 5.2) 118 5.1 (1.1, 9.1) <15%

38

Within-Group Unadjusted Change: Improved (green) and reduced (red) implementation for UETreatment No.

forms

Pre to Post Change in % of Times

Implemented

(95% CI)

Facilitated KT Passive KT

UE ROM/Stretching 1381 8.7 (3.2, 14.2) 3.5 (-4.3, 11.4)

Interventions to prevent shoulder

pain1381 0.7 (-5.5, 6.8) -4.3 (-12.1, 3.6)

Task-specific UE training 1381 12.0 (5.3, 18.8) 6.1 (-3.0, 15.2)

Techniques to reduce hand edema 547 -1.7 (-7.3, 3.9) -1.3 (-8.6, 6.0)

Ice/heat or soft tissue massage for

shoulder547 1.5 (-1.8, 4.7)

-3.3 (-9.6, 3.0)

FES for UE 547 -0.6 (-3.5, 2.4) -1.1 (-4.6, 2.4)

Educate patient/caregiver how to

handle UE1381 0.6 (-3.5, 4.7) -2.8 (-8.8, 3.2)

Constraint-induced therapy 547 -3.9 (-7.6, -0.3) -5.7 (-12.2, 0.8)

Visual imagery 547 3.6 (-1.1, 8.4) 0.1 (-5.4, 5.6)

39