IMPLEMENTATION QUALITY RESEARCH OF PREVENTION

PROGRAMS IN CROATIAMIRANDA NOVAK

University of Zagreb, Faculty of Education and Rehabilitation Sciences

Josipa Mihic, Clemens Hosman, Celene Domitrovich

Health Promotion Research Centre 18th Annual Summer ConferenceNational University of Ireland Galway

MOTIVATION FOR RESEARCHo Although the efficacy of various evidence-based interventions has

been established through carefully designed trials in control conditions, there is a lack of evidence for its utilization in natural community conditions (Kam, Greenberg and Weiss, 2003; Greenberg et al., 2005; Fixsen et al., 2005, Proctor and Rosen, 2008)

o A more systematic process is warranted to translate efficacy results into positive participants’ outcomes

SCIENCE TO PRACTICE CONTEXT FACTORS AND SUPPORT SYSTEM

CHARACTERISTICS OFPROGRAM

IMPLEMENTATION RESEARCH

DEFINITION OF IMPLEMENTATIONo implementation refers to what a program consist of when it

is delivered in a particular setting (Durlak and Dupre, 2008)

o characteristics of intervention itself and characteristics of the intervention support system

o it is important to differentiate between factors affecting implementation quality (Fixsen et al., 2005, 2009) and implementation aspects (Durlak and Dupre, 2008)

WHY WE MONITOR IMPLEMENTATION

To know what actually happened

To provide feedback for

quality

To strengthen the conclusions made about outcomes

To examine the change process

To understand the internal dynamics

of program

To advance knowledge for

replication

To strengthen the quality of

evaluation

IMPLEMENTATION FACTORS - CAPACITY FOR PROGRAM IMPLEMENTATION

PERCEIVEDPROGRAM

IMPACT

PERCEIVEDPROGRAM

IMPACT

INDICATORS OF IMPLEMENTATION QUALITY

Research project»Preffi – Quality assurance in the Region of Istria«

Department of Health and Social Care, Region of Istria and Faculty of Education and Rehabilitation Sciences, University of Zagreb

Incorporation of science-based principles in prevention practice in Istria through collaboration of scientists and

practitioners

PROGRAM

QUALI TY

MODEL OF TRAINING STUDY AND ITS EFFECTS

I MPLEMENTATI ON

QUALI TY

PROGRAM

EFFECTS

PROGRAM

DEVELOPMENT

Science Policy Resources Leadership

Target groups & context factors

Expertise

TRAINING PROGRAMPrinciples of science-based practice

Logic modelling Implementatio Evaluation AdvocacyInteractive group education and Individual consultation

ORGANIZATION MANAGERSPROGRAM AUTHORS

PROGRAM DELIVERERS

The general aim of this research was to study implementation processes and

their outcomes in prevention programs in Croatia.

1. To construct valid and reliable measures of implementation quality.2. To explore the level and variation of implementation quality3. To explore the differences in perception of implementation quality

between program managers, implementers and participants.

4. To explore the relationships of implementation factors and indicators of implementation quality.

5. To test the impact of the Training for Prevention on the level of implementation quality in experimental group

SAMPLE: 24 mental health promotion and prevention programs

MANAGERS IMPLEMENTERSIMPLEMENTATION FACTORS Standardization

Implementers’ skillsAttitudesTrainingSupportMonitoring

Standardization

AttitudesTrainingSupportMonitoring

IMPLEMENTERS PARTICIPANTSINDICATORS OF IMPLEMENTATION QUALITY

FidelityQuality of deliveryResponsivenessPerceived program impact

DosageQuality of deliveryResponsivenessPerceived program impact

ParentingPositive

development and SEL

Substance abuse

MEASURES

o Implementation Factors Questionnaire for Program Managers

o Implementation Factors Questionnaire for Program Implementers

o Indicators of Implementation Quality Questionnaire for Program Implementers

o Indicators of Implementation Quality Questionnaire for Program Participants

o The Preffi 2.0 instrument

June 2010 - December 2010 - January 2011 February 2011 - February 2011 - January 2011 January 2011 March 2011 December 2011

3 assessors independently

assess 24 written programs’ proposals

Division 24 programs :

experimental (N=12) and

control condition

(N=12)

EXPERIMENTAL

Training for Prevention

to experimental

condition

Mid-Assessment

implementationquality

Focus groups

Evaluate Initial set of

items

19 PROGRAM MANAGERS

50 PROGRAM DELIVERERS

434 PROGRAM PARTICIPANTS

CONSTRUCTION OF SCALES FOR IMPLEMENTATION QUALITY

ASSESSMENT

Implementation of 24 programs

Post-Assessment

Implementation quality

June 2010 - December 2010 - January 2011 February 2011 - February 2011 - January 2011 January 2011 March 2011 December 2011

3 assessors independently

assess 24 written programs’ proposals

Division 24 programs :

experimental (N=12) and

control condition

(N=12)Mid-

Assessment implementation

quality

Focus groups

Evaluate Initial set of

items

Implementation of 24 programs

Post-Assessment

Implementation quality

STUDY OF IMPLEMENTATION QUALITY

19 PROGRAM MANAGERS

50 PROGRAM DELIVERERS

434 PROGRAM PARTICIPANTS

EXPERIMENTAL

Training for Prevention

to experimental

condition

19 PROGRAM MANAGERS

50 PROGRAM DELIVERERS

434 PROGRAM PARTICIPANTS

June 2010 - December 2010 - January 2011 February 2011 - February 2011 - January 2011 January 2011 March 2011 December 2011

3 assessors independently

assess 24 written programs’ proposals

Division 24 programs :

experimental (N=12) and

control condition

(N=12)Mid-

Assessment implementation

quality

Focus groups

Evaluate Initial set of

items

Implementation of 24 programs

Post-Assessment

Implementation quality

STUDY ON THE TRAINING FOR PREVENTION IMPACT

22 PROGRAM MANAGERS

55 PROGRAM DELIVERERS

744 PROGRAM PARTICIPANTS

EXPERIMENTAL

Training for Prevention

to experimental

condition

RESULTS

STUDY OF IMPLEMENTATION QUALITY

Reports on implementation factors across 24 programs show that views of managers and programs implementers are rather similar

MAN: Lowest ratings for standardization, training and monitoringIMP: Lowest ratings for standardization and training

Indicators of implementation quality are really high – group results all over value of 3

Overall group results show that program implementers give similar although higher ratings than participants

STUDY OF THE TRAINING FOR PREVENTION IMPACT

Levels of implementation factors reported by managers and the effect of the Training for Prevention at post-interventionPOST INTERVENTION INTERVENTION EFFECT

CONT INT BETA(SE)

P EFFECT SIZE

Standardization M

SD

2,890,43

2,050,81

-0.84(0.28)

<.001** -1.35

Implementers’ skills M

SD

3,610,47

3,760,39

0.15(0.18)

0.42 0.35

AttitudesM

SD

2,970,37

2,740,46

-0.23(0.18)

0.21 -0.55

TrainingM

SD

2,950,72

2,390,50

-0.56(0.27)

.05* -0.92

SupportM

SD

3,470,63

3,250,40

-0.21(0.22)

.36 -0.42

Monitoring M

SD

3,140,70

2,350,29

-0.79(0.22)

.01** -1.59

N 11 11

Managers from experimental conditions give more critical answers about implementation factors than those from control = CHANGE OF CRITERIA

Levels of implementation factors reported by implementers and the effect of the Training for Prevention at post-intervention

POST INTERVENTION INTERVENTION EFFECTCONT INT BETA

(SE)P EFFECT SIZE

Standardization M

SD

3,100,54

2,850,59

-0.34(0.23)

0.15 -0.43

AttitudesM

SD

3,040,41

2,830,29

-0.13(0.13)

0.31 -0.59

TrainingM

SD

3,070,66

2,850,59

-0.22(0.17)

0.21 -0.36

SupportM

SD

3,440,47

3,490,69

0.07(0.25)

0.77 0.09

Monitoring M

SD

2,970,57

2,800,60

-0.38(0.25)

0.14 -0.29

N 33 22

Implementers from experimental conditions give more critical answers about implementation factors than those from control = CHANGE OF CRITERIA

Levels of implementer’s ratings on indicators of implementation quality and the effect of the Training for Prevention at post-intervention

POST INTERVENTION INTERVENTION EFFECT

CONT INT BETA(SE)

P EFFECT SIZE

Fidelity M

SD

3,290,37

3,220,36

-0.02(0.13)

0.85 -0.19

QualityM

SD

3,550,40

3,490,36

-0.03(0.14)

0.82 -0.16

ResponsivenessM

SD

3,360,41

3,490,36

0.12(0.15)

0.45 0.34

Perceived program impact M

SD

3,340,48

3,250,48

0.02(0.19)

0.93 -0.19

N 33 22

At post-test, program implementers from the experimental group did not report improved indicators of implementation quality in comparison with the control

group

Levels of participant’s ratings on indicators of implementation quality and the effect of the Training for Prevention at post-intervention

POST INTERVENTION INTERVENTION EFFECT

CONT INT BETA P EFFECT SIZE

DosageM

SD

92,3318,00

99,452,13

6.96(4.41)

.13 0.71

QualityM

SD

3,430,52

3,640,34

0.17(0.07)

.02* 0.49

ResponsivenessM

SD

3,090,67

3,500,41

0.26(0.12)

.04* 0.76

Perceived program impact

MSD

2,940,74

3,230,62

0.19(0.12)

.14 0.43

N 391 353

At post-test, participants from the experimental group report on higher levels of implementation quality, two of them being significant - quality of delivery and

responsiveness.

Moderator analyses have shown that Training for Prevention is more effective for SHORT PROGRAMS

Moderator analyses have shown that Training for Prevention is more effective for PROGRAMS WHERE MANAGER IS NOT ACTIVE

CRUCIAL CONDITIONS FOR SUCCESFUL IMPLEMENTATION

Investment in implementation factors

THANK YOU FOR YOUR ATTENTIONmnovak@erf.hr