3rd ENSEC Conference29th June - 3rd July 2011University of Manchester, UK

Phillip SleeProfessor in Human Development Flinders UniversityAdelaide, South Australia www.caper.com.au

Director: Flinders Centre Promotion Well-Being: Violence Prevention (SWAPv)

“KidsMatter”- the outcomes of an evaluation of a SEC Initiative in Australian schools

Flinders University Evaluation Team :P. T. Slee, M.Lawson, A.Russell, H.Askell-Williams, K.Dix, L.Owens, G.Skryzpiec and B.Spears

…

Outline of Presentation• Mental Health

•‘KidsMatter Constructs’•KidsMatter Evaluation

•KidsMatter Evaluation Findings•”KidsMatter & Disability

•KidsMatter and Academic Outcomes’•The ‘Elephant in the Room’

•Translating Research to Policy & Practice•Summary & Discussion

KidsMatter Suite of Initiativeshttp://www.flinders.edu.au/ehl/educationalfutures/groups/kidsmatter/

kidsmatter_home.cfm

• KidsMatter –Primary• KidsMatter –Early childhood• Kidsmatter Quality Assurance• Kidsmatter- Disabilities

Schools as Sites for Interventions

• Attention now focussed on working through schools to improve the health & well-being of children

(Greenberg, et al,2003)

• school–community partnerships positively influence outcomes for students, eg showing increases in attendance rates, resilience, behaviour and attitude

(Weare & Nind, 2011)

• partnerships between school and community enable

students to achieve the best life outcomes, eg academic and non-academic eg SEL

(Slee et al, 2009).

KidsMatter conceptual framework

KidsMatter process assumptions:

protective factors/reduction in risk factors.

The factors reside within the school environment, the family context and the psychological world of the child, & include:

1. Positive school climate

2. Increases in students’ social-emotional competencies, eg sense of mastery and control

3. Increases staff knowledge and competence (e.g., in teaching SEL)

4. effective parenting

5. effective strategies for identifying and working with at risk students

Two dimensions of mental health

• Adaptive defences are essential to positive mental health

(Vaillant 2000)

• increasing international interest in the concept of positive mental health and its contribution to all aspects of human life

(Tennant et al, 2007)

• The World Health Organisation – Mental Health is not simply the absence of mental disorder or illness,

but also includes a positive state of mental well-being (WHO, 2004).

– Mental health can be conceptualised as a state of well-being in which the individual realizes his or her own abilities, can cope with the normal stresses of life, can work productively and fruitfully, and is able to make a contribution to his or her community (WHO 2001)

• Both dimensions of mental health were of interest for us in this evaluation

Representation of the two dimensions in our analytical models?

Representation 1The mental health outcome includes both strengths and difficulties as outcomes at the same level in the conceptual model

Representation 2The measure of the positive dimension (SEC) is seen as mediating the relationship between the intervention and the level of mental health problems

Schools Participants

Quantitative Measures in

all 100 schools

Qualitative Measures:

Visits to 10 Schools

Outcomes

Longitudinal data collected up to 4 times during 2007 and 2008 School

implementation of:

• KidsMatter specifically

• Mental health initiatives generally

Risk and protective factors:

• School mental health approach

• Child and family competencies

Change over time in:• Students (MH)• Parents• Teachers• Schools

School engagement & implementation of KM, Facilitators, Barriers

Background: gender, age, ATSI, ESL, Fee, ’at risk’

Student MH: Parent-rated SDQ & SCS

Student MH: Teacher-rated SDQ & SCS

Parent: Knowledge, Skill, Confidence, etc

Teacher: knowledge, confidence, SEL, etc

KM Executive Summary, ‘pill & dose’

Parent Focus Groups

Student Voice

Teacher Interviews

Leadership Interviews

KidsMatter Schools

100 schools across Australia

-

50 Round 1 started in 2007

50 Round 2 started in 2008

State Project Officers

Students

N=4980

Parents/Caregivers

Teachers

Principals & KM

Leaders

4/33

1

4

1

KidsMatter evaluation measures

School

School practices and environment

Family

Family circumstances &

relationshipsChild

Individual children’s

abilities and needs

Student Mental Health Outcomes SDQ (Goodman, 2005)

• Positive school community• School engagement• Parenting information• Parenting support• Early intervention

• Parental knowledge• Parental behaviour

• Relationship skills• Decision making and perspective taking• Self-awareness and self-management

• Emotional symptoms• Conduct problems• Hyperactivity• Peer problems

The Four KidsMatter Components

how welcomed members feel and their sense of belonging.

provision of social and emotional learning, professional development

parenting education and support.

supporting students who are experiencing, emotional - social - behaviour difficulties.

Background and Research

Design

N=4980 students

Effect Size

We used the part-correlation coefficient r for reporting all effect sizes. We use correlations of 0.10, 0.24, and 0.37 as indicative of small, medium and large effects, respectively (Hattie,2009)

The analysis of change undertaken in the evaluation used:

Latent class analysisStructural equation modellingMultilevel modellingThematic analysis of qualitative data

Impact of KidsMatter Overview

“[KidsMatter] has changed school culture, I think. It’s changed the way the school views mental health. It’s given a greater awareness, but it’s also changed the way, people relate to one another, particularly the students, and the way the classrooms operate”. (School Principal)

Impact of KidsMatter on Schools and Teachers • 22% increase in teachers who ‘strongly agreed’ that

schools were engaged with KidsMatter • 31% increase in schools using the 7-Step Implementation

process• Progress on the four Components:

– most progress on 2: Social and Emotional Learning for Students

– least progress on 4: Early Intervention for Students

Why was least progress made on Component 4?

On developing social and emotional competencies, teachers perceptions were that, as a result of KidsMatter:

• more knew how to help students

• more ‘strongly agreed’ that the staff acted to help students

• more ‘strongly agreed’ that their teaching programs helped students

• more ‘strongly agreed’ that they had good self-efficacy

for teaching SEL

Impact of KidsMatter on Schools and Teachers:

Teachers’ knowledge, competence and confidence

Impact of KidsMatter on Families

Parents perceptions were that, as a result of KidsMatter:

more became involved with the school

more had increased their capacity to help their children with social and emotional issues

more felt their school better catered for their child’s needs

Parenting knowledge and approach showed little change from the initial (self-reported) high levels

Impact of KidsMatter on Student Mental Health

KidsMatter achieved its major aims to improve student mental health and well-being and reduce mental health difficulties

Strengths go up Difficulties go down

Impact of KidsMatter on Student Mental Health

KidsMatter achieved a shift for a number of students within the ‘borderline’ and ‘abnormal’ ranges, into the ‘normal’ range

The numbers vary depending on how you assess mental health

“Mental Health is not simply the absence of mental disorder or illness, but also includes a positive state of mental well-being”

(World Health Organisation, 2004)

Borderline or Abnormal: At Time 1 By Time 4 Improvement

Based on SDQ Difficulties 24% 19% 5% or 1 in 20

Based on Mental Health Strengths & Difficulties 34% 24% 10% or 1 in 10

Methodological Notes and Limitations

Strengths of the evaluation design

• Multi-methods and tools• Multiple informants • Multiple time points• Multiple settings• Ecologically valid

(based in authentic settings)• Stratified school selection

process

The analysis of changeundertaken in this report uses • Latent class analysis• Structural equation modelling• Multilevel modelling• Thematic analysis

Limitations include• The Initiative is not, and was

not designed to be, a strict randomly controlled trial. Because the sample is not a true random sample, caution should be taken if generalising findings to other students and other primary schools in Australia.

Conclusions

KidsMatter had a positive impact on schools, teachers, parents, and children

• changed school culture• the ‘whole school’ approach enhanced academic and social

competencies• facilitated Social and emotional learning for students,

Parenting support & education, and Early intervention• strengthened protective factors within school, family & child• improved student mental health

“Look it really works. It can change school culture, which changes the way kids relate. It really

does. By having that focus and by really thinking about how kids relate to one another; how the staff relate to the children and teaching them a set of relationship skills to help them cope. You can really make a profound difference in your school and in those children’s lives” (School

Principal)

Conclusions

KM & Disability

• The parents and teachers of 555 South Australian primary school students participated in the KidsMatter Evaluation.

• Of this group, 494 South Australian students 61 students had an identified disability (11%) ,

Resultsco-morbidity is present and a student with a disability is more likely to also have mental health problems.

using the SDQ students without a disability had 1 in 8 chance of having mental health difficulties (i.e.being in the abnormal range),

students with one identified disability had a 1 in 3 chance

and students with multiple disabilities had a 1 in 2 chance

The findings indicate, on average, a practically significant (parents=large & Teachers = medium) improvement in student wellbeing ie increase on the ‘Mental Health Strengths’ scale.

and a practically significant (Parents=small effect size & Teachers =

medium effect size) decrease in mental health difficulties over time ie reduced SDQ (difficulties) scores

Results (Cont’)

What else can be done for students with disabilities?

Findings –SEC & Academic Performance

• This research now extended to investigate the impact of KidsMatter on academic performance, based on the National Assessment Program - Literacy and Numeracy 2008 data (NAPLAN).

• A two-level hierarchical linear model, using HLM-5

was employed to examine the relationships between school-level characteristics and student-level academic outcomes (Raudenbush &Willms, 1995).

Implementation quality of whole-school mental health promotion and students’ academic

performance• a child’s social and emotional wellbeing impacts on

their academic performance. (CASEL,2008)

• BUT there is growing concern about how schools evaluate the implementation of such programs, and of the program’s effectiveness if the quality of implementation is not maintained.

• few studies, internationally, report the relationship between quality of implementation of mental health promotion initiatives and student outcomes, eg academic performance

Implementation Index

• An Implementation Quality Index was developed (Greenberg, 2004; Domitrovich,2008; Askell-Williams, 2011).

• revealed that the extent to which children’s social-emotional competencies improved over time was dependent, on how well the school had implemented KidsMatter (small effect size=.22).

Findings (cont.)

• Schools that implemented KidsMatter well also had improved learning outcomes for students of up to six months more schooling, over and above any influence of socio-economic background (Dix et al, 2011).

• Over the 2 years of KM, 14% more teachers strongly agreed (scored 6 or 7) that KidsMatter had led to improvements in students’ schoolwork, equivalent to a medium (b = 0.30, p < .05) effect size.

The ‘Elephant’ in the Room!

Quality Assurance: “If we keep on doing what we have been doing, we are going to keep on getting what we have been getting”. (Wandersman et al,

2008)

• The initiative is now approved for broader dissemination to 1700 schools by 2014

• Upscaling of KM affects how KM is being disseminated post pilot

• All KM stakeholders have an interest in ensuring that KM continues to be implemented in the manner that optimises the mental health and wellbeing of students, as well as achieving positive outcomes for parents and teachers.

.

Quality Assurance

has been seen as one way of maintaining and assessing quality outcomes

“concerns remain about thetransferability of intervention models beyond well-controlled settings” (Humphrey et al,2010,p.215)

Defining Quality Assurance• The development of evaluation standards is

one part of a move toward ‘evidence-based’ practice.

• “the determination of standards, appropriate methods and quality requirements by an expert body, accompanied by a process of inspection or evaluation that examines the extent to which practice meets the standards”.

Murgatroyd and Morgan (1993, p.45)

PromotionReadiness

AdoptionSustainability

Monitoring

Incentives

Implementation

Quality Assurance

(Slee, Murray-Harvey, Dix & van Deur,2011)

How to translate our research to outside settings?

Promotion - How well information about the existence and value of the programme is promoted to new schools and the broader service community.

Readiness - This outlook refers to the extent to which the school recognises there is a problem to be addressed, is willing to address it, and has the capacity to do so.

Adoption –With the support of the staff, parents and carers, the principal commits to implementation on behalf of the school community. This stage may also be informed by local community agencies.

Implementation - The program must meet the local needs of the community in which the intervention is taking place and must be responsive to local conditions.

Sustainability - Whether the fidelity, dosage and quality are maintained over time

Monitoring - An accurate monitoring and feedback system. This aspect is central to quality assurance.

Incentive – infrastructure that provides incentives or recognition to schools and individuals within schools for achieving implementation milestones.

Slee, P.T., Murray-Harvey, R., Dix, K; van Deur, P (2011).

Researchers need to innovate & take risks in developing partnerships & exchanging knowledge to influence policy & practice

THE THREE CULTURES

• The culture of science - theory building -hypothesis testing- research

• the culture of policy- political - economic -

- social imperatives

• the culture of practice - clinical judgement

Tensions Amongst the Three Cultures

• “Science is focussed on what we know. Social policy and the delivery of health and human services are focussed on what we should do. Scientists are interested in questions. Scholars embrace complexity. Policy makers demand simplicity. Scientists demand we stop and reflect. Service providers are expected to act” (Schonkoff, 2000,p.183

Research to Policy & Practice

• “The vitality and sustainability of a society depend on the extent to which it equalizes opportunities early in life for all children to achieve their full potential and engage in responsible and productive citizenship”(Shonkoff & Bales,2011p.117).

Research to Policy & Practice (Cont’)

“Science has an important role to play on advising policy makers on crafting effective responses to social problems that affect the development of children” (Shonkoff & Bales, 2011,p.17)

“This is not an initiative for poor schools with disadvantaged families, it’s an initiative for all children in primary schools and all types of schools” (School Principal)

SUMMARYThe Flinders University’s Evaluation overall main recommendation was “that the broad

framework, processes and resources of KidsMatter be maintained as the basis for a

national rollout”Federal Minister for Mental Health & Aging

announced a rollout to 1700 schools with an additional $18.4m.

The Change Process & ENSEC

Gladwell (2002) identifies 3 principals for creating social change:

1. The law of the few

2. ‘stickiness’ –the memorable key elements of the message

3. The power of context

“Do not go where the path may lead, go instead where there is no path, & leave a trail”. Ralph W. Emerson

Thank You

The KidsMatter Initiative

• KidsMatter (KM) - Australian national primary school mental health promotion, prevention and early intervention initiative.

• KM - developed in collaboration with the Australian Government Department of Health and Ageing, beyondblue: the national depression initiative, the Australian Psychological Society, and Principals Australia

• KidsMatter uses a whole-school approach. It provides schools with # a framework, # an implementation process, and# key resources to develop and implement evidence-based mental health promotion, prevention & early intervention strategies.

Comparisons of effect sizesComparison of effect sizes: Round 2 schools

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7

calcium/bone mass

lead exposure/ID

aspirin/heart attack

mental health strengths (P)

teacher SEL actions (T)

General Engagement (T)

total SDQ (P)

Parenting support school (T)

KMI impact on child (P)

KMI parenting learning (P)

KMI implementation (T)

KMI PD (T)

KMI impact on child (T)

KMI engagement (T)

SEL (T)

KMI implementation (T)

Vari

ab

le

Effect size rEffect size: Part correlation r

‘social and emotional education across the world: celebrating diversity in theory, research and

practice’.