Prevention and Early Intervention Pilot program Client Outcomes Studies Australasian Evaluation...

Prevention and Early Intervention Pilot program Client Outcomes

Studies

Australasian Evaluation Society Lunchtime Seminar February 2007

Presented by Dr Terrence O’Brien

Evaluation

• Evaluation should not be seen as an end product or one off event.

• Evaluation is a participatory process.• Evaluation is an essential tool for continuous quality

improvement.

Client Outcomes Evaluation

• Application to human services is challenging• Changes in family’s behaviours due to interventions

needs to be measured, but not easy• The question is more about how rather than if it can be

done or not..

What is an Outcome evaluation?

• Outcome evaluations/studies look to the extent to which program participants experience benefits or changes in their behaviour and life circumstance and indications that these changes are related to the services received. (Mikka 2001)

Client Outcomes Evaluation• Previous difficulty in evaluating clients’ outcomes.• Outcomes in humans services hard to define and

measure.• Sensitivity to professional and client needs.• Human Services not conducive to rigorous

experimental approaches.• Research in this area is becoming more popular.• Growing trend of outcome measures being used to

monitor funded NGOs.

Outcomes Evaluation

Help to inform the choice and validity of intervention, ensuring best/evidence based practice principles are followed

• Allow the organisations to be outcome focussed.• Strengthen existing services. • Target effective services for expansion. • Identify staff and volunteer training needs. • Develop and justify budgets. • Prepare long-range plans.

• Focus board members' attention on program issues.

PEI Pilot Programs Client Outcomes Studies

• Putting Families First• Future Directions Initiatives – PEI 1,2,4, 5, 7 and 8• PEI and Indigenous Family Support Trials and Pilots• Anecdotal evidence of positive outcomes from trials

(one year)• Client Outcome Studies needed to be developed for

the pilot evaluation (3 years)

PEI Pilot Programs

• 25 Trial Programs

• 18 Pilot Programs

• 12 Communities and 6 Child Safety

Evaluation Processes• Demographic and Case Management and

• PEI Principles Survey

• Client Outcomes Studies

MISSION AUSTRALIA-PROJECT CIRCUIT BREAKEROUTCOMES EVALUATION PROJECT CONCEPT

When my service does (our services do) - intervention

Strengths based early intervention and prevention-

With children, parents, families who-

Consist of families with children between the ages of 10 and 16 years who live in the Northern Suburbs of the Department of Communities Brisbane City region.

When the context is-

Referrals of families with students in secondary school who are experiencing relationship difficulties and other issues such as truanting and anti-social/deviant behaviour.

What change is there in (what change am I wanting to see) –

Stability and quality of family relationships and communication

Project concept cont.

• And how does this impact on the child by improving their –• Level of positive engagement with education

In terms of: -

School attendance

Behaviour at school

OUTCOMES EVALUATION PROJECT DESIGN

Organisation: Mission Australia - Project Circuit Breaker

1. What are your Questions?When my service provides strength based, solution focused case management,is there improvement in the young person’s:• school attendance, • behaviour at school and • behaviour at home during and after the intervention.Are there improvements in the quality of the family relationships during and after the

interventions?What are the individual outcomes that can be attributed to each intervention strategy that is

implemented with a family? 2. How will you find out?MethodologyType: Descriptive (Survey Design)/pre and post Interventions and after 3 months.

4 case studies. (Using the NCFAS Child Behaviour Checklist and other questionnaires devised by Project Circuit Breaker staff)

Over what time: 3-6 month intervention periods over a total of 2 years

3. SampleType of sample: Purposive sample - Case Study sample – consenting familiesHow many families in the sample: 30 + Dependent on number of referrals receivedDescription of the sample: Families and young people referred from Secondary Schools

within Project Circuit Breaker’s catchment area. Families experiencing difficulties with young person’s behaviour. Young person experiencing difficulties with relationships parent/s and at school.

4. What will you do about:Confidentiality and privacy issues: Information about the purpose, duration and the role

that families play in the study are given to families prior to request of information. Families will not be identified; codes will be attributed to each member of family.

Informed consent: Forms to be signed Ethical issues: Using AES ethical principles5. Tools and TechniquesTypes: Standardised tests – Pact c, NICFAS Quality of Family interactions, Child

Behaviour Checklist for ages 6 –18 (home/school/self report) attendance records (3 months prior/3 months during intervention) and questionnaires devised by PCB staff, GO, and Parent/young person. Case Studies developed from case notes – semi structured interview at 3 months after intervention.

Key characteristics to measure: Changes in presenting issues, Family interactions, Changes in behaviour at school, School attendance.

When will you use them? : Pre and post intervention and 3 months after intervention

Organisation Outcomes

Sisters Inside - BOWS 1. Accommodation2. General well being 3. Child reunification

Centacare - RAIFSS 1. Engagement using culturally appropriate ways2. Parenting skill and knowledge3. Community strength

Durumbal – Eagle Hawk Family Support Service

1. Improvements in family resilience2. Self esteem3. Improved budgeting and household

management skills4. Increased community engagements5. Contact with support networks

Client Outcomes

Mission Australia - Project Circuit Breaker

1. School attendance 2. Behaviour at school 3. Behaviour at home4. Family Relationships 5. What are the individual outcomes that can be

attributed to each intervention strategy that is implemented with a family?

Lutheran Family Support Services - REFS

1. Young person’s behaviour2. Parenting skills3. Family relationships

Mununjali Jymbi Centre 1. Family and community resilience and well being

2. Sustainable self determining family and cultural support systems and structures

3. Reduce the levels of representation of Aboriginal people and Torres Strait Islander people in statutory systems and crisis services

Client Outcomes

Ozcare - GIFS 1. In the well being, school behaviour and attendance of children

2. Relationships and parenting skills of families3. Coping skills of children with a disability

Brisbane Youth Service – Parents Yes

1. Relationships 2. Parenting skill and knowledge3. Connections to support networks4. Participation in life

Red Cross 1. Children’s school attendance and behaviour 2. Educational achievement

Client Outcomes

Save the Children 1. Creative strategies of intervention2. How does this assist the family to become

involved in a therapeutic relationship with an FSW?

3. A child's Health, Emotional State, Connectedness and Intellectual state

The Spot – The Spot for Kids 1. Family relationships2. Contact with Spot and local community support

networks3. Child/children’s general behaviour at home,

school and at The Spot

Cairns Women’s Centre - OSP 1. Does the program help improve family’s social connectedness?

2. Is there more stability in the family’s housing situation?

3. Is there improved contact with community support networks?

Client Outcomes

Organisation Intervention Client Sample Outcome or QuestionsWill the organisations see

improvementsin

Data collection techniques

Data collection

time frame

Lutheran - REFS

Family therapy, in home support & specialised group work

Young persons aged 11- 16 and their families displaying difficult behaviours

Young persons behaviourParenting skillsFamily relationships

Surveys and questionnaires devised by REFS and REU

Pre, exit and 3 mths post


Strength Based intervention.

Young persons aged 11- 16 displaying and their families difficult behaviours

School attendance, Behaviour at school Behaviour at homeFamily Relationships What are the individual outcomes that can be attributed to each intervention strategy that is implemented with a family?

Pact c, NICFAS Quality of Family interactions, Child Behaviour Checklist for ages 6 –18 (home/school/self report) Attendance records Questionnaires

3 months prior/3 months during intervention and

Survey designOrganisation Intervention Client Sample Outcome or Questions

Will the organisations see improvements

in


Data collection

time frame

Lutheran Family Support - REFS

Family therapy, in home support & specialised group work

Young persons aged 11- 16 and their families displaying difficult behaviours

Young persons behaviourParenting skillsFamily relationships

Surveys and questionnaires devised by REFS and EU



Strength Based intervention.

Young persons aged 11- 16 displaying and their families difficult behaviours

School attendance, Behaviour at school Behaviour at homeFamily Relationships What are the individual outcomes that can be attributed to each intervention strategy that is implemented with a family?

Pact c, NCFAS Quality of Family interactions, Child Behaviour Checklist for ages 6 –18 (home/school/self report) Attendance records Questionnaires

3 months prior/3 months during intervention and

Red Cross In home and at school mentoring service.

Primary school aged children and their families where there area indicators of abuse or neglect or not coping with parenting responsibilities

Children’s school attendance and behaviour educational achievement

Surveys and questionnaires devised by Red Cross.



improvementsin


Data collection

time frame

Survey design


improvement in

Data collection

techniques

Data collection time frame

Save the Children

Practical assistance and counselling.

High support need families residing in caravan parks

Creative strategies of interventionHow does this assist the family to become involved in a therapeutic relationship with an FSW.A child’sHealth, Emotional State,Connectedness and Intellectual state

Session activity Data

Qualitative research

Action research minutes

6 mths of data

The Spot – The Spot for Kids

Family based training personal development program

Families with children up to the age of 12

Family relationshipsContact with Spot and local community support networksChild/children’s general behaviour at home, school and at the spot

Case notes, NCFAS, EU data

Pre and Post

Case Studies


improvement in

Data collection

techniques

Data collection

time frame

Sisters Inside - BOWS

Parenting support for reconnecting mothers and their children.

Mothers who are leaving prison

AccommodationGeneral well being

Child reunification

Forms with rating scales, questionnaires, case notes

Pre and Post

Ozcare - GIFS

Collaborat-ive Case work

Young children, families and children with a disability

In the well being, school behaviour and attendance of childrenRelationships and parenting skills of familiesand coping skills of children with a disability

Rating Scales, case studies, diaries and Observations

Pre and Post

Case Studies

Brisbane Youth Service – Parents Yes

Parenting support program

Parents under the age of 25

Relationships Parenting skill and knowledgeConnections to support networksparticipation in life

Goal attainment Scale, case notes, Eco Map, Family Strengths FDI 2 Database

3 mths of data


improvement in

Data collection

techniques


Case Studies

Organisation Service Client Sample

Outcome or QuestionsWill the organisations see

improvement in

Data collection

techniques


Centacare - RAIFSS

Fortnightly face to face integrated family support where

Children and young people who are at risk of statutory intervention

Engagement using culturally appropriate waysParenting skill and knowledgeCommunity strength

Narrative Inquiry

Case Notes

Observation

Inter-

vention period

Durumbal – Eagle Hawk Family Support Service

Youth connecting with community activities.

6 Aboriginal and Torres Strait Islander families with young people aged 12 – 19 years."

Improvements in family resilienceSelf esteemImproved budgeting and household management skillsIncreased community engagementsContact with support networks

Narrative Inquiry

Inter-

vention period

Narrative Inquiry

Mununjali Jymbi Centre

1.Family and Cultural Community Development Support3. Healing & Intervention Programs

Aboriginal and Torres strait Islander families.

Family and community resilience and well being Sustainable self determining family and cultural support systems and structures Reduce the levels of representation of Aboriginal people and Torres Strait Islander people in statutory systems and crisis services

Narrative Inquiry

Action Research

Inter-

vention period

Organisation Service Client Sample

Outcome or QuestionsWill the organisations see

improvement in

Data collection

techniques


Narrative Inquiry

Parents YES Outcomes Study

– relationships;

– parenting skill and knowledge;

– connections to support networks; and

– participation in life.

• Improvements in positive child/parent relationships and interactions

• Children are reaching their development milestones

• Parenting skills and knowledge has increased

• Connections to support networks are strong

• Changes have been seen in life participation

• An increase in stable, safe, secure and adequate housing

Project Circuit Breaker’s, Client Outcomes Study

• a young person’s attendance and academic performance at school;• general changes in the young person’s behaviour; and• movement towards improvement in family functioning

– all families perceived there was significant change to the presenting issues due to intervention;

– they reported the influence of intervention on goal achievement;– they perceived that progress towards goals was due to intervention;– all young people maintained or showed improved grades in at least 3 out

of 4 school subjects after intervention, yet their school attendance showed little change between initial and closure measurements; and

– all young people scored lower (behaviour improvement) after intervention on at least one of the oriented scales scores (Child Behaviour Checklist and the Youth Self Report).

Red Cross’s Intensive Family Support program Client Outcomes Study

• the children’s school attendance and behaviour; and

• educational achievement.

– improvements in school attendance and positive improvements in behavior for children both at home and at school;

– positive feedback from the children’s teachers regarding educational achievement and attitude; and

– most families made perceived and made positive progress towards their goals.

The Client Outcomes study for the Munujali Jymbi Centre

• improvement in family and community resilience and well being (abbreviated);• establishment of sustainable self determining family and cultural support systems and

structures • reduction in the levels of representation of Aboriginal people and Torres Strait Islander

people in statutory systems and crisis services.

• That there is increased community acceptance of the Centre and a high level of involvement in their programs;

• 5 local people have been trained and are now employed by the centre;• The service has provided mentors for young people, children and their families; and• The community reports that they have increased self confidence and direction and

have a broader vision of the future.

The Eaglehawk Indigenous Family Support Program’s Client Outcomes

• family resilience and self-esteem;

• household budgeting and management skills; and

• community engagement and contact with support networks.

• both families exhibited improvements in family resilience;

• the social and emotional wellbeing of the families improved;

• a direct correlation between budgeting / household management and secure housing / food supply, with both families showing improvement in each;

• external relationships and health management improved. Improvements in positive child/parent relationships and interactions

Prevention and Early Intervention Pilot program Client Outcomes Studies Australasian Evaluation...

Documents

Transcript of Prevention and Early Intervention Pilot program Client Outcomes Studies Australasian Evaluation...