Implementation and adaptation of evidence-based programs

Presenter: Sophie Aitken

This webinar is part of the CFCA information exchange webinar series: www.aifs.gov.au/cfca

Please note: The views expressed in this webinar are those of the presenter, and may not reflect those of the Australian Institute of Family Studies or the Australian Government

Expert Panel Project

Commissioned by Department of Social Services (DSS) 5 year project (2014-2019) Support Families and Children Activity service providers to

plan, implement and evaluate quality programs share the results with others

Aim: programs and services are built on the best available evidence of what works for children, young people and families.

Elements of Expert Panel project

Expert Panel Industry List Support for service providers (e.g.

Communities for Children)

Support for service providers

DSS funded projects that are administered by Expert Panel members (e.g. outcomes measures)

Service-funded projects that assist services to plan, implement and evaluate programs (via Industry List)

Face-to-face and helpdesk support, resources and information (e.g. webinars, evaluation resources)

Implementation & Adaptation of Evidence-

Based Programs

Sophie Aitken Centre for Family Research & Evaluation

23rd February, 2016

Centre for Family Research and Evaluation: CFRE

A partnership between –

drummond street services – not-for-profit family services agency +

&

Deakin University Psychology Department – academic, teaching and research institution

To promote the health and wellbeing of all Australian families by contributing to the evidence-base of family based interventions

To build sector capacity to strengthen evidence-based programs through expertise and collaboration

What we will cover in this Webinar

Selecting Evidence-based Programs

Measuring Program Fidelity

When to Consider Adapting EB Programs

Process for Adaptation

"Good and Bad" Adaptations

Building Evidence-based Components into your own Programs

Adapting for ATSI Communities

Differences that may warrant adaptation

Culturally and Linguistically Diverse Aboriginal and Torres Strait Islander Ages of Children targeted Rural and Remote communities Education Level Socio-economic Status Gender & Sexual Diversity

Community Needs & Service Gap Analysis

At-Risk Groups

Risk Factors

Ages of Children

Program Type

1. Implement an existing Evidence-Based program 2. Adapt an existing Evidence-Based program

3. Develop a program with Evidence-Based core components

• How well the program meets identified needs Needs

• Overall fit with current priorities, structures and support Fit

• For training, staffing, technology, data systems and administration Resource Availability

• Indicating the outcomes that might be expected if the program is implement well Evidence

• Expertise, exemplars available for observation, how well the program is operationalised

Readiness for Replication

• As intended and to sustain implementation over time Capacity to Implement

The Hexagon Tool – six factors to consider

The Hexagon Tool

Blasé, K., Kiser, L, & Van Dyke, M. (2013) The Hexagon Tool: Exploring Context. Chapel Hill, MC: National Implemention Research Network, FPG Child Development Institute, University of North Carolina at Chapel Hill.

AIFS Guidebook of Evidence-based Programs

CfC program objectives Create Strong Child-Friendly Communities Early Learning & Care Healthy Young Families School Transition & Engagement Supporting Families & Parents

Target groups Infants (0-2 years) Early Childhood (3-5 years) Middle childhood (6-12 years) Parents At-risk or vulnerable

Keywords

https://apps.aifs.gov.au/cfca/guidebook/search

Communities That Care®

A research-based process to enable communities to develop long-range plans for promoting healthy development of children and young people

Adapts to the needs of different and distinct communities

The process:

• Understanding local needs – Community profiling including youth surveys

• Identifying risk and protective factors across individual, family, peer, school and community domains

• Community Engagement – mobilise support of key leaders and build working partnerships between residents, organisations and agencies

• Develop Community Action plan to select and implement Evidence-Based programs and practices

• A Monitoring & Evaluation plan to assess effectiveness

Program Fidelity

Is the program being carried out as intended? We can only know this if we put in place measures to monitor that all of the core components of the EB program are being adhered to

this is known as fidelity measurement

Identify Core Components Before we can measure fidelity need to identify what are the core components that need to be adhered to: • Target group we are seeking to deliver to • Qualifications of the facilitators • Number of sessions to be delivered • Core program content to be covered • Key activities and teaching methods – modelling, role plays,

peer support, homework, group activities, videos, handouts • Participant engagement and responsiveness

How can we measure fidelity?

• Collect demographic data about clients • Record details of facilitators including qualifications and

whether they have completed relevant training • Collect attendance records for every session • Complete session records to track which topics were covered

and which activities/homework were delivered • Include trained facilitators to observe and provide feedback on

delivery methods • Record which participants completed homework tasks • Obtain client satisfaction surveys

1. Implement an existing Evidence-Based program

2. Adapt an existing Evidence-Based program

3. Develop a program with Evidence-Based core components

Common Risk Factors Common Protective Factors Unhealthy lifestyle Healthy lifestyle

Maltreatment, neglect & abuse Emotional competency skills

Family Conflict Family cohesion/harmony

Harsh parenting styles Parental involvement in children’s activities

Parental mental illness/substance abuse Supportive relationship with an adult

Poverty/economic insecurity Access to health and support services

Lack of warmth & affection Community & social connectedness

Social isolation Social skills

Homelessness Family stability and security

School failure/low commitment to school Opportunities for success at school

Bullying & victimisation Prosocial peer group Toumbourou, J. (2015) A review of therapeutic processes targeted in evidence-based parent and family interventions, Prevention Science Consulting Group, Unpublished paper.; Commonwealth of Australia (2000) National Mental Health Stratagy, Promotion, Prevention and Early Intervention for Mental Health A Monograph 2000

When to consider adapting…

Ineffective engagement

Unique risk or protective factors

Unique symptoms of a common disorder

Effectiveness not demonstrated

Adapted from Barrera, M. & Castro, F.G. (2006), A heuristic framework for the cultural adaption of interventions. Clinical Psychological Science Practice, Vol 13, pp.311-316.

Process for Program Adaptation

6. Evaluating Effectiveness

5. Pilot Testing the

Adapted Program

4. Deciding what

Changes to Make

3. Identifying Differences

between Target and

Original Populations

2. Choosing an Evidence

Based Program

1. Community Needs

Analysis

Adapted from Chen, E.K., Reid, M.C., Parker, S.J. & Pillemer, K. (2013) Tailoring Evidence-Based Interventions for New Populations: A Method for Program Adaptation Through Community Engagement, Evaluation Health Professional, Vol 36 (1), pp.73-92

Source of mismatch Original validation Groups

Current target group

Potential mismatch effect

Language

English Non-English speaking

Inability to understand program content

Ethnicity Anglo-saxon, non-minority

Minority group Conflicts in beliefs, values and or norms

Socioeconomic Status

Middle class Low SES Insufficient resources and different life experiences

Urban-rural Context Urban, inner city

Rural, remote Logistical and environmental barriers affecting participation in program activities

Risk factors: Number & Severity

Few and moderate in severity

Several and high in severity

Insufficient effect on multiple or most severe risk factors

Family stability Stable family systems

Unstable family systems

Limited compliance in program attendance and participation

What not to change: “Bad” Adaptations

Core content components: • knowledge, attitudes, values, and skills

Core pedagogical components:

• how it’s administered – methods, strategies, interactions

Core implementation components: • logistics, resources, staffing and ability to maintain fidelity

Resource Centre for Adolescent Pregnancy and Prevention (ReCAPP, 2013)

“Good” adaptations

Location • Participants feel comfortable and safe

Accessibility • Participants can easily get to the program

Staffing • Facilitators reflect the diversity of group

Language & Other resources • Group content is understood

Activities • Tailored to capacity and culture of group

Process for Program Adaptation

6. Evaluating Effectiveness

5. Pilot Testing the

Adapted Program

4. Deciding what

Changes to Make

3. Identifying Differences

between Target and

Original Populations

2. Choosing an Evidence

Based Program

1. Community Needs

Analysis

1. Implement an existing Evidence-Based program 2. Adapt an existing Evidence-Based program

3. Develop a program with Evidence-Based core components

Developing an Evidence-Informed Program

Theoretical or Research Background

Program Logic or Theory of Change

Evidence-Based Interventions or Activities

Program Manual & Documentation

Qualified Facilitators

Program Evaluation

Core Components of Effective Interventions Determine what type of program you want to deliver – e.g. parenting programs, childrens’ programs, whole family support, playgroups Review the literature to find out what is known about the core components of effective programs of this type CFCA website a good starting point https://aifs.gov.au/cfca/ or search for systematic reviews or meta-analyses Incorporate these components into your own programs

Core Components of Effective Parenting Programs

Teaching skills in: parent-child play; helping your child learn; limit setting; problem-solving; emotion regulation; communication

Understanding different stages in childrens’ development

Programs held in 2 hour sessions

Between 6 and 24 sessions

2 facilitators

Programs based on behavioural approaches

Include fathers and grandparents

Homework tasks to practice at home

Focus on strengths rather than deficits

Video and role play to demonstrate parenting strategies

Participant discussion and sharing of stories to enable peer support

Weekly phone calls to provide support and encouragement

Key Components of Effective Family Support 1. Dedicated Worker 2. Whole Family 3. Father-inclusive 4. Home-based 5. Includes practical support 6. Assertive engagement 7. Collaborative 8. Strengths-based 9. Goal-directed programs rather than those offering generic support 10. Programs that target a number of risk factors either simultaneously or

sequentially

Challenges of Implementing Evidence-based Programs for ATSI communities

Remoteness

History of Cultural Decimation and Stolen Generations

Children Raised by Community rather than Nuclear Family

Different Parenting Norms

Fluid Living Arrangements

Demands for Regular Attendance & Adherence to Timelines

Programs Not Evaluated with ATSI populations

Aboriginal-specific programs that have shown to be effective: • NSW Aboriginal Maternal and Infant Health Strategy (NSW Health, 2005)

Mainstream Programs that included Indigenous families in evaluation: • The Family Independence Program (Homel, R. et al, 2006) • The Preschool Intervention Program (Homel, R. et al, 2006)

Mainstream EB programs that have been partially successfully adapted for Indigenous families: • Triple P – Positive Parenting Program (Turner et al, 2007) • The Resourceful Adolescent Program (Rowling et al, 2002) • MindMatters (Hazell, 2005) • Exploring Together Preschool Program (Robinson et al, 2009) • Family and Schools Together Galiwin’ku program (Guenther, J. 2011)

Adaptations to Consider for ATSI Programs

• Community Consultation on program content and resources • Engage whole Community in activities • Employ local community leaders to help facilitate the program • Non-Indigenous facilitators trained in cultural competence • Location must be considered a ‘safe’ place and be easily accessible • Extra sessions to allow for sharing of personal stories and development of trust between participants

and facilitator • Culturally appropriate language, resources and activities • Adapt activities to reflect literacy competence of the group Mildon, R., & Polimeni, M. (2012). Parenting in the early years: effectiveness of parenting support programs for Indigenous families. Resource sheet no. 16. Produced for the Closing the Gap Clearinghouse. AIHW cat. no. IHW 77. Canberra

Guenther, J. & Boonstra, M. (2009) Adapting Evaluation Materials for Remote Indigenous Communities and Low-Literacy Participants, Paper Presented to APCCAN 2009 Symposium 08, Perth, 17 November, 2008.

Exploring Together for Tiwi Islanders

Exploring Together – Parenting Program for at-risk children Parent group – Behavioural parenting approach Childrens’ group – Psycho-social skills Parent-children group – strengthen parent-child relationship Robinson, G. & Tyler, B. (2003) Evaluation of an early intervention program on the Tiwi Islands,

Interim Report, School for Social Policy and Research, Charles Darwin University, Australia.

Exploring Together for Tiwi Islanders - Challenges

• Children had limited “feelings” vocabulary • Parents did not readily articulate feelings or observations about one particular

child • Behaviour management approach to parenting did not fit well with parenting

norms and thinking styles • Parent attendances were lower than optimal

Exploring Together for Tiwi Islanders Adaptations

Delivered in School Environment so familiar/safe place Gained support of Community leaders 4 of 6 program staff were Tiwi Islander Adapted childrens’ activities to reflect limited feeling vocabulary Adapted parenting management approach to a more family systems therapy approach Outcomes – despite challenges • Parent/child relationships enhanced • Children developed social-emotional skills • Success in engaging families • Improvements in childrens’ behavivour

Key Messages for Adapting EB programs

Adaptation is Important for Engagement and Effectiveness

Consult with Members of the Target Population

Maintain the Core components of the Program

Discuss any Changes with the Program Developer

Trial the Adapted Program with the Target Population

Build in Fidelity Measurement & Program Evaluation

Thank you for listening

Contact Information:

Centre for Family Research & Evaluation drummondsectorsupport@ds.org.au

Ph: (03) 9663 6733

Further information

Expert Panel project website https://aifs.gov.au/cfca/expert-panel-project

Email fac-expert-panel@aifs.gov.au

Join the Conversation

You can continue the conversation started here today and access a range of related resources on the CFCA

website:

www.aifs.gov.au/cfca/news-discussion