Intensive Community and Home Services (ICHS) A Preliminary · PDF file ·...

Intensive Community and Home Services (ICHS) A Preliminary Evaluation, 2011

A preliminary evaluation of the first two years of ICHS, a Community Based Alternative to Residential Services Program for high risk latency age children at Child Development Institute, Toronto, ON

Margaret Walsh, Manager, Evaluation and Research, Child Development Institute Reet Mae, Research Coordinator, ICHS Evaluation

January 13, 2012

This evaluation project was funded by Ontario Centre of Excellence for Child and Youth Mental Health through Doing Evaluation Grant.

Confidential 13/01/12

Acknowledgements Ontario Centre of Excellence for Child and Youth Mental Health: Funding for ICHS Evaluation Project Latency Age Advisory Committee (LAAC): Community Stakeholders Parents and Children in IHCS ICHS staff and managers Contributors Angela Ball, Paul Kotchapaw, Nynke Annema, Kathy Levene, Tamara Mizrachi, Ron Stoddard, Paul Henry, Emily Eisnor, January 13, 2012


Executive Summary Organization: Child Development Institute (CDI), Intensive Community and Home Services (ICHS) Project Lead: Margaret Walsh, Manager of Research and Evaluation Project Title: Intensive Community and Home Services (ICHS): A Preliminary Evaluation. The ICHS preliminary evaluation is designed to describe the process and program strategies and build capacity for ongoing research, evaluation and knowledge translation. The primary focus is to pilot and test measures and strengthen the agency scientist-practitioner model within the ICHS program. Purpose:

• To build capacity for ongoing program evaluation; • To describe processes within an emerging program; • To inform program development and knowledge exchange protocols for working with high-risk

latency age children. Program: Child Development Institute (CDI) is a multi-service Children’s Mental Health Centre within the city of Toronto. CDI is committed to service and research excellence based on the scientist-practitioner model. CDI’s ICHS program is an innovative home-based community model for some of our most at risk children. Formerly the oldest Residential Services Program (RSP) for high-risk, behaviourally troubled boys under 12 years of age, and their families, ICHS has been transformed to a home-community intensive based model for this significantly vulnerable group of children. It offers additional service benefits that include: increased service capacity, increase in preventative activities, support for families in home and community settings and serving both genders. It is a multi-modal intervention that incorporates SNAP® model cognitive-behavioural strategies within parent and child skills-focused groups, engages in home, school and community supports and offers a Section 23 onsite classroom. Specific treatment plans are based on thorough ecosystemic family assessments. Plan: The evaluation is multi-modal, including both qualitative and quantitative methodologies to provide a comprehensive overview of the efficacy, clinical processes and clients’ experience of the program. The study includes process and outcome measures (e.g., presenting problems, standardized pre-post measures, risk assessment, staff survey, case studies, and parent focus groups). The study sample includes 27 children and their families engaged in various stages of service. Three groups (n=9 each) are defined for the purposes of this project: 1) new admissions during the study protocols; 2) clients ongoing at the time of the study; and 3) children who had been in the program but were effectively discharged. Product: The findings from this evaluation, while preliminary, are promising, and help to identify service strengths of the ICHS program, such as flexibility in responding to clients’ emerging needs. Clients served experience similar presenting problems as previous residential clients; however the average age is lower. Preliminary outcomes indicate that the ICHS program is decreasing risk and improving behaviours such that children who may have previously required residential services are able to remain in their parents’ care. With the program now servicing both genders, comparisons show that girls are experiencing higher emotional, behavioural and family difficulties than boys at admission. Outcome measures indicate that severe externalizing behaviours show improvement through treatment. Parents and children report positive feedback about the program, improved child behaviours, family relationships, and increased social activities. The staff also recognize client responsiveness to treatment and report feeling supported during the initial program development. Gender-specific Early Risk Assessment Lists (EARL-20B; EARL-21G) show decreases in risk for the individual case examples examined. Although the preliminary results are promising, families report having struggled with the number and complexity of the measures. This ambitious project highlights the need to simplify evaluation measures and timelines and the importance of flexibility and transitional support to these high-risk families. Providing adequate follow up services while freeing up staff to take on new clients is emerging as one of the key challenges of the ICHS program. Amount awarded: $33,934.57 Final report submitted: January 13, 2012 Region - MCYS Toronto Region


Table of Contents Table of Contents ................................................................................................................ 4

Introduction ........................................................................................................................ 6 Rationale and Goals of the ICHS Evaluation.............................................................................................................. 6 Agency Overview .................................................................................................................................................................. 7 ICHS Program......................................................................................................................................................................... 8 ICHS Program Development .............................................................................................................................................8 ICHS Referral and Intake....................................................................................................................................................8 Program Description............................................................................................................................................................9 Training of Staff during transition to new program........................................................................................... 10 Collaborative treatment .................................................................................................................................................. 10 ICHS Service Implementation........................................................................................................................................ 11

Purpose of the Evaluation...............................................................................................................................................12 Stakeholders .........................................................................................................................................................................13

Literature Review.............................................................................................................. 13 Prevalence and Need.........................................................................................................................................................13 Residential Care Overview .............................................................................................................................................13

Methodology .................................................................................................................... 15 Characteristics of participants......................................................................................................................................15 Study Groups......................................................................................................................................................................... 15 ICHS Demographic information.................................................................................................................................. 15

Data Collection Framework ...........................................................................................................................................19 Timing of data collection ................................................................................................................................................ 21 Additional Data ................................................................................................................................................................... 23

Outcomes and indicators ................................................................................................................................................24 Measures Selection Process ............................................................................................................................................ 25

Sources of Information.....................................................................................................................................................26 Measures Selected for the Evaluation Study........................................................................................................... 26 BCFPI. The Brief Child and Family Phone Interview (BCFPI ........................................................................... 26 Child Behaviour Checklist (CBCL)................................................................................................................................ 26 Assessing Risk Level: The Early Assessment Risk Lists (EARL) ....................................................................... 27 Parent Measures ................................................................................................................................................................. 27 Child Measures ..................................................................................................................................................................... 28 The SelfReport Antisocial Behaviour Scale (SRA)............................................................................................... 28 Therapeutic Process Measures ..................................................................................................................................... 28 Measures of Family Functioning.................................................................................................................................. 29 Teacher Report .................................................................................................................................................................... 29 Staff Survey............................................................................................................................................................................ 30 Focus Group........................................................................................................................................................................... 30 Case Histories/Single Case Design .............................................................................................................................. 30 Measurement of Treatment Fidelity........................................................................................................................... 31 Quality Assurance............................................................................................................................................................... 31

Evaluation Limitations .....................................................................................................................................................31 Time frame ............................................................................................................................................................................ 31 Case Studies........................................................................................................................................................................... 31 Overall ..................................................................................................................................................................................... 31

Results .............................................................................................................................. 33 Evaluation Findings...........................................................................................................................................................33 Residence and ICHS Comparison.................................................................................................................................33 ICHS Program Results.......................................................................................................................................................34 Heritage .................................................................................................................................................................................. 34


Behaviour problems at Intake ...................................................................................................................................... 35 Initial Assessment: CBCL.................................................................................................................................................. 36 Risk Assesment (Early Assessment Risk List).......................................................................................................... 38 Additional Stressors/Risks.............................................................................................................................................. 39 Parent Depression at Intake .......................................................................................................................................... 40 Parenting Stress at Intake .............................................................................................................................................. 41

Outcome measures ............................................................................................................................................................42 Child Behaviour Outcomes.............................................................................................................................................. 42 Evaluation of Clinical Significance.............................................................................................................................. 45 EARL Outcome ..................................................................................................................................................................... 47

Additional Outcome Measures......................................................................................................................................47 BCPFI Family Measures ................................................................................................................................................... 47

NLSCY Measures .................................................................................................................................................................50 Child Measures ....................................................................................................................................................................50 Process measures ...............................................................................................................................................................50 Single Case Design.............................................................................................................................................................. 50 Case Histories ....................................................................................................................................................................... 51 Child Risk and Outcome ................................................................................................................................................... 53

Focus Group ..........................................................................................................................................................................53 Staff Survey ...........................................................................................................................................................................54 Treatment Intensity........................................................................................................................................................... 61 Quality Assurance............................................................................................................................................................... 63

Objective discussion of findings...................................................................................................................................63 Outcome Summary............................................................................................................................................................. 63 Follow up ................................................................................................................................................................................ 63 Challenges .............................................................................................................................................................................. 64 Emerging Needs .................................................................................................................................................................. 64 School Placements.............................................................................................................................................................. 65 Stakeholder Involvement and Knowledge Exchange.......................................................................................... 65

Conclusion and Recommendations.................................................................................... 67 Discussion and interpretation of findings ...............................................................................................................67 Lessons Learned .................................................................................................................................................................67 Recommendations .............................................................................................................................................................68 Ongoing Evaluation of ICHS program....................................................................................................................... 68 Planned Transitions........................................................................................................................................................... 68 Follow up ................................................................................................................................................................................ 68 Staff Training ....................................................................................................................................................................... 69 Systems Issues ...................................................................................................................................................................... 69

Next Steps ..............................................................................................................................................................................69 Ongoing Evaluation of ICHS program....................................................................................................................... 69 Planned Transitions........................................................................................................................................................... 69 Treatment Components ................................................................................................................................................... 70 Knowledge Translation.................................................................................................................................................... 70 Staff Training ....................................................................................................................................................................... 70 Systems Issues ...................................................................................................................................................................... 70

References ........................................................................................................................ 71

Appendices ....................................................................................................................... 76 Appendix A: Program Logic Model ............................................................................................................................77 Appendix B: Evaluation Framework.........................................................................................................................78 Appendix C: Stakeholders...............................................................................................................................................79 Appendix D: CDI consent................................................................................................................................................80 Appendix E: Staff Survey .................................................................................................................................................81 Appendix F: Quality Assurance Report .....................................................................................................................82


Introduction Rationale and Goals of the ICHS Evaluation. The evaluation activities were designed to build capacity and refine the policy and procedures for the development of the model, tools, and methodologies used for ongoing evaluation and research activities within the ICHS program. The evaluation also emerged from the need of staff, stakeholders at CDI and community partners (e.g. Latency Age Advisory Committee (LAAC), MCYS, and Child Welfare Agencies) to examine the effectiveness of this innovative alternative (ICHS) to residential programming in reducing aggressive behaviours in children. The goals of the research were to provide insight and direction for program development, to examine mechanisms of change for aggressive children in intensive treatment; and to develop best practices for intensive community and home programming for the children and families utilizing such services. Please refer to the process and measures flowchart below for an overview of the program and the evaluation components.


Agency Overview

Child Development Institute (CDI) is a multi-service children’s mental health centre, located the city of Toronto. As an agency, CDI is dedicated to generating new knowledge that will increase our understanding of child development, contribute to service excellence in all program areas, and build on CDI’s foundation of caring, knowledge and innovation which will lead to positive changes for children and families. CDI’s scientist-practitioner philosophy (whereby science informs practice and practice informs science) helps build strong evidence based programs and translate knowledge into practice so that the positive impact on children’s lives can be increased exponentially. CDI is committed to building knowledge, understanding change processes, and developing interventions that work. Over the past 25 years, the Centre for Children Committing Offences (CCCO) at CDI has been involved in developing and implementing stringent evaluation designs, including randomized control trials (RCT) and long-term follow-up studies that help identify and manage risk in children under the age of 12 experiencing disruptive behaviour problems. This work is done in the context of a broader assessment, including issues of attachment, risk and trauma, and incorporates evolving interventions informed by both practice and research. As part of CDI’s commitment to a scientist practitioner model that has evolved since the inception of the SNAP® evidence-based model in 1985, CDI has built partnerships with professionals from a variety of disciplines to assist with ongoing program evaluation, research and dissemination activities which are all designed to enhance treatment effectiveness and program development. The Evaluation Framework for CDI’s new Intensive Community and Home Services program (ICHS) program, was developed with the extensive program evaluation expertise of the CCCO, using a collaborative approach, whereby CCCO’s research team worked with ICHS staff to identify and select measures and build a timeframe for data collection to support ICHS achieving agency goals of excellence and continual program development. The research staff were able to address potential barriers, such as difficulty in getting parents to respond/submit measures and staff buy in, by collaboratively reviewing the potential measures, providing orientation/training to staff on selected measures, clarifying usefulness for treatment planning/outcome evaluation and reviewing how to present measures to families. The research team worked in partnership with ICHS staff to coordinate timely measure distribution/data collection and to support staff in utilizing data for clinical treatment planning.


ICHS Program

ICHS Program Development

For over a decade, CDI advocated for a transformation from its’ oldest service the Residential Services Program (RSP) for high-risk, behaviourally troubled boys, under 12 years of age, and their families to a home-community intensive based model for this significantly vulnerable group of children. In 2007/2008, an extensive needs assessment review, conducted by a GTA latency-age, residential treatment planning group (Centralized Admission and Referral Services [CARS], CDI, Aisling-Discoveries, and Hincks-Dellcrest), resulted in the restructuring of CDI’s RSP in 2009 to an intensive community-based model. The new program was named Intensive Community and Home Services (ICHS). The ICHS program has important additional benefits which include: increased annual service capacity, increase in preventative activities, supporting families in the home setting and serving both boys and girls. Given the concerning risk levels and engagement in disruptive behaviour from an early age, research indicates these children have the potential to continue on an antisocial persistent life course trajectory (Cóté et al., 2006). Although some level of aggressive behaviour is common for most children, for some, persistent levels of aggressive behavior can lead to harmful dysfunction across several contextual domains such as family, school and community (Connor, 2002). Interventions aimed at identifying and managing risk levels that improve young children’s adaptive functioning across multiple settings are recognized as the most promising in helping to reduce future contact with the justice and health care systems (Connor, 2002). A shift toward multimodal, multi-component interventions to target the complex individual and systems contributors to these behaviours are seen as necessary for effective treatment response (Kazdin, 1996; 1997). With the ICHS program, CDI created an innovative home-based community model for some of the children most at risk in the Greater Toronto Area (GTA)

ICHS Referral and Intake

ICHS clients are referred by professionals such as school social workers, teachers, children’s aid societies or other agencies as well as families. The ICHS program and other residential services intake is done through Centralized Access to Residential Services (CARS), a joint assessment between the 3 latency age centres (Aisling Discoveries Child and Family Centre, Child Development Institute, and Hincks Dellcrest) who work in concert to determine the resource(s) best suited to meet the individual needs of the child and family referred for residential treatment. Referrals are assigned at a weekly telephone consultation. Once a child is matched to an agency the family proceeds directly through the agency admission/assessment process to determine placement options within the agency for stabilization and treatment interventions that meet the child and family needs. Inclusionary criteria for ICHS are City of Toronto (including Etobicoke, North York and Scarborough) catchment area, age (6-11.5 years of age), disruptive behaviours (presence of externalizing behaviour, aggression, and conduct issues with scores above a cut off score of 70 on the overall externalizing scale of the BCFPI), and involved caregiver (parents must indicate willingness to be involved in treatment). The ICHS program serves families of both boys and girls, who are exhibiting severe behavioural disorders such as aggression and defiance. Exclusion criteria for ICHS are diagnosis of a developmental disability, autism, or an intellectual delay.


Program Description

Removal from and reintegration into one’s family and community are particularly important factors to consider over the early developmental phases of childhood. CDI’s shift to an intensive program for some of the most troubled and vulnerable children in our community is supported both by research and clinical findings (Bernstein, 1987, Leichtman and Leichtman, 2001). The ICHS model provides an intensive service for these children and their families on an expanded and flexible hours basis, allowing better community and home support throughout the daytime, evening and/or weekend hours, while retaining the child in the family setting. Access to specialized assessments such as psychiatric and psychological assessments is still made through the program, while building a stronger connection with family doctors if the child remains directly in his/her care. A specialized Day Treatment classroom located onsite can accommodate 8 children who are admitted into service and are struggling with severe behavioural within school setting and are therefore often unable to attend regular classes. Support in community-based schools is offered to participants who are not in the onsite Section 23 classroom. The ICHS model provides a comprehensive, cognitive behavioural treatment approach, which incorporates CDI’s evidence-based SNAP® (Stop Now And Plan) and SNAP® Parenting (SNAP-P) strategies. The ICHS program is informed by research and incorporates a risk, needs and responsivity approach. CDI’s internally developed, evidence-based and gender sensitive early risk assessment tools, the Early Assessment Risk Lists, EARL-20B, for boys and EARL-21G, for girls, (Augimeri et al., 2001; Levene et al., 2001) helps counselors to identify areas of greatest child and family risk and inform treatment planning. The SNAP® programs are the internationally recognized evidence based interventions that were developed within the CCCO. The ICHS program from its inception has included SNAP® strategies for behaviour management, self-control and to address cognitive distortions of both parents and children in the program. Therapeutic work, conducted by ICHS, onsite, after school, and within the home setting addresses the needs of the children and families by:

• creating community connections (e.g., programs related to the child’s interests, talents and level of social skill development);

• doing home based family work (e.g., supporting family to facilitate the child ‘s completion of expected routines such as homework);

• coaching in SNAP® strategies(e.g., problem-solving and self-control skills); • engaging in school and community advocacy (e.g., gangs, school suspension); • facilitating ongoing SNAP® based parent and child groups

Fundamental to the ICHS program initiative are community and social development principles. These principles are evidenced in sustained efforts to connect family members with resources that they view as facilitating their individual growth (e.g., ethno-cultural group, religious affiliation, community youth recreational activities, tutoring). Comprehensive assessments, which may include psychiatric consultation, guide multifaceted treatment planning with regard to key factors that influence the development and remediation of family dysfunction. The ICHS program can be regarded as one treatment modality within a comprehensive service system for at risk latency age children and their families. Intensive multi-facetted home and community based services for the highly troubled children and families, such as those typically referred to the CDI residential program are expected to contribute to improved long term outcomes in the primary domains (family, school, peers and community) in


which a child needs to function well to proceed through a healthy developmental course to adolescence and adulthood.

Training of Staff during transition to new program

Recognizing that ICHS staff was well versed in child behaviour management within a residential setting, there was an identified need for additional training in family work the home and community setting. Training components for staff include:

• Cognitive Behavioural Training (CBT) for family work and family counseling sessions; • Ongoing consultation with a CBT trainer once a month; • SNAP® training conducted by CCCO that included SNAP® parenting (SNAP-P); • EARL training on completing risk assessment and clinical treatment planning; • Child and Adolescent Functional Assessment Scale (CAFAS) training occurred during the

time frame of the study; • Regular annual training continued;

o Crisis Prevention Intervention (CPI) - Formal training in crisis prevention/intervention, was expanded to include how to transfer those skills to a community/home environment,

o First aid and CPR annual training, o Online trauma-focused CBT course

Additional safety protocols and training workshops were conducted. ICHS managers consulted with Crossroad’s children’s centre in Ottawa about their protocols to address staff safety needs for doing community-based work. ICHS staff worked collaboratively to develop internal safety protocols. An external trainer from Child Welfare was hired to conduct their “Working Alone in the Community” workshop for all CDI staff. As well, staff can request attendance at external trainings that are relevant to the program on an annual basis. Individual staff requests have included CBT workshop for families who are hoarders, culturally competent services and trauma-focused CBT. These trainings allow individual staff to bring increased skills in specific areas of program need back to the team and to the work with families.

Collaborative treatment

Treatment is a partnership of family members and ICHS staff, who jointly set and evaluate service goals. The program focuses on each family member learning and practicing more effective ways of functioning and interacting at home and in his/her community. Each family is assigned a primary ICHS staff who works with them in the home and community by providing intensive in-home coaching and support (e.g. developing routines, learning behavior management, improving family communication); building connections with community resources; collaborating with schools to support student success, as well as offering advocacy and community support. At the CDI facility ICHS offers weekly parent and child groups, family and individual counselling, weekday summer activity program and a Section 23 classroom for students who are unable to function in their local community schools. Specific treatment components include; anger management, cognitive-behavioral therapy, social skill development, SNAP® approach (self control and problem solving strategies), SNAP® parenting (parent training and behaviour management techniques) and using community resources to develop positive recreation and creative outlets/skills. The intensive home-based intervention, which is a key difference from the former residential program, allows ICHS staff to work with caregivers in the environments in which the problems are


experienced, thus providing the best support for developing realistic and caregiver-directed approaches to improving the child’s behavior and meeting each family’s needs. Ongoing treatment evaluation is done in partnership with both child and caregiver in order to promote positive relationship building that can maximize the probability of positive long-term outcomes. Additional follow-up service component(s) evolved during the implementation of the new ICHS program. This includes extended transition support, if needed, to other programs and school placements, follow-up visits, phone calls, and continued access to family counselling and parenting groups. This less intensive, but highly individualized transitional service ensures that children and families discharged from ICHS are supported to either make a successful transition to their own home and community or to access additional services if these are still required. The length of treatment averages 6-12 months, but is determined on an individual client basis.

ICHS Service Implementation

As the ICHS program has been implemented and the need for individualized follow-up services emerged, it became clear that the length of service for each client was longer than previously anticipated (closer to 12 months vs. 6 months). This has resulted in fewer new admissions than projected during the time frame of this evaluation. Children and families who complete the intensive phase of treatment, frequently have required additional follow up care for a period from 6-12 months. Each ICHS worker is assigned 2-3 primary children at one time, depending on the intensity of the treatment plan for each child and each may carry another 1-2 families for follow up. The treatment begins with an extensive intake, including an eco-systemic assessment and the development of collaborative treatment goals. Once the goals are established, parents/caregivers and ICHS staff develops and contract for an initial multidisciplinary intervention plan. The plan is informed by shared identification and prioritization of needs, strengths and areas for development across 3 domains (family, school, community) and integration of CDI’s SNAP® (Child and Parenting) intervention strategies.


Purpose of the Evaluation

Given that the ICHS program is a newly implemented alternative to Residential treatment for high risk children in the Toronto area and that generalizability of residential treatment improvements into the home and community may be difficult (Burns, et al, 1999) the evaluation, description and program development of ICHS becomes an important tool for capacity building for both the program and the service system. This evaluation was developed to examine the effectiveness of CDI’s ICHS program during its second year of implementation. A multi-modal evaluation was designed to assess effectiveness and inform the continued development of this innovative service for at risk latency aged children (and their families) typically referred to residential treatment. The methodology of this evaluation was broad based and included standardized measures, focus groups, case studies and monitoring process. The evaluation was designed to pilot and review some pre, post and follow-up outcome data in order to make recommendations for future data collection/methodologies. With a Program Logic Model (Appendix A) and Evaluation Framework (Appendix B) in place, ICHS was able to conduct both a Process and Outcome Evaluation. Under the ICHS Evaluation Framework, a Process and Measures Flowchart and accompanying Intake/Evaluation Measures Dictionary were developed. The flowchart allowed ICHS to monitor the processes and procedures undertaken from the point of referral to discharge. The measures dictionary outlined the measures collected at intake, pre, post and follow-up periods and was revised upon completion of this evaluation. With such a framework in place the evaluation was designed to address the following questions:

• Process: o Who are the clients and families served; o What is the level of behavioural issues; o What is the level of risk for the child and family; o What is the level of family functioning and parental skills; o What is the average degree of treatment dosage;

• Outcome: o What is the average length of service; o Does risk identification and management relate to treatment outcome; o Are there decreases in aggressive or antisocial behaviour; o Are there increases in pro-social behaviour; o Does parental stress and/or mood affect treatment response; o Does family functioning show improvement; o Do parents show increases in parenting skills; o Does treatment duration and dosage predict outcome; o Are treatment effects stable?

The evaluation provides direction in identifying potential program and service patterns and trends on the families and children that ICHS serves. According to the CDI scientist-practitioner model, the results of these analyses (positive or negative) will be utilized to help the ICHS program manager and team make informed decisions on program revisions and service planning in order to better serve children and families.


Stakeholders

Through a well-regarded consensual process, the Latency-Age Residential Planning Group which included representatives from the centralized access to residential services (CARS) housed at Delisle, and the Latency Age Residential Service providers in the Toronto catchment area, created an intensive treatment continuum model that includes: Aisling and Hincks-Dellcrest Residential Programs and CDI’s shift from residential to an intensive community program as well as the above mentioned joint screening telephone conferences. Operating under this new model, a Latency Age Advisory Committee (LAAC) was established, which included members from CARS, Latency Age Service Providers in the Toronto area, MCYS, and Child Welfare Agencies. The LAAC meet regularly to monitor the changes to the treatment continuum model, and provides a dissemination point for sharing evaluation findings and information to stakeholders. The Group builds on the collaborative approach that has effectively supported this initiative to date, and remains committed to its further development. As well, a joint assessment committee consisting of designated representatives from CARS continues to centralize and initiate all referrals for CDI and the two latency age residential programs and has developed well-defined guiding criteria. These criteria were incorporated in the evaluation planning exercises. While The LAAC oversees the general performance of the continuum, building capacity was integral to the project as its key objectives were to draw on the experience of others as well as our own to overcome cross-sector barriers, articulate best practices, policies and procedures, and to eventually incorporate them in a community-reviewed protocol. The broader community stakeholders are outlined in Appendix C.

Literature Review Prevalence and Need

Children and youth are at a significant disadvantage when compared to other demographic groups affected by mental illness, in that the failings of the mental health system affect them more acutely and severely than adults. (Canadian Paediatric Society, 2011). Potentially as many as 10-18% of children in Canada and the USA are impacted by mental health concerns and only a small percentage of these children receive needed treatment (Offord et al, 1990, Surgeon General’s Report on Mental Health, 1999). While early intervention can reduce the potential for negative impact during adulthood (McEwan, 2007, Raos and Janus, 2010), families face many challenges in caring for young people with mental health problems (Uta, 2011). Adolescents who suffer with mental health issues compared to adolescents who do not, come into contact with the criminal justice system twice as often as adolescents who do not (Smith, et al, 2008). The cost to the family and to society is large when children’s needs fail to be met at an early age. Early aggression among children as early as age 8 has been found to predict future contact with the criminal justice system. (Loeber and Farrington, 2001). There is a strong relationship between childhood disruptive behavior and continuous chronic offending (Smith et al, 2008) making good outcome research, both short and long-term a critical priority for the children’s mental health system.

Residential Care Overview

The history of residential/out of home placements for children/adolescents began with programs in the 1920’s providing safe environments for adolescents in need of care. Out of home placement services have changed over the past century, as residential care/hospitalization became the most intensive service provision for children facing serious mental health issues (Hair, 2005, Magellan Health


Services, Children’s Services Task Force, 2008) and residential programs continue to utilize a large part of available funding for children’s mental health (Surgeon General’s Report, 1999). Concerns about residential treatment include failure to maintain gains after treatment which may lead to repeat admissions (Burns, et al, 1999); being exposed to negative peers during residential care (Dishion, McCord & Poulin, 1999; Loeber & Farrington, 1998) and lack of improvement in functioning/adjustment in less restrictive environments such as children’s home communities (Hair, 2005; Bickman, Lambert, Andrade & Peñaloza, 2000 ). Recent reports have evaluated the benefits of residential based treatments vs. community-based treatments for children with mental health issues. Having family and community involvement are important contributing factors in positive outcomes for child mental health treatment (McLendon, 2008). Further, research is showing that community based services decrease removal of the child from the family home, and strengthen family relationships, coping skills and positive parenting (McLendon, 2008, Surgeon General’s Report, 1999) Many of the factors contributing to successful residential-based treatments (e.g. family involvement, focus on precipitating problems, discharge planning and community involvement, (Leichtman et al, 2001; Preyde et al, 2011) can be provided within a community based treatment such as ICHS. Given the need to direct resources to effective interventions and to target interventions to specific needs of children and families this evaluation of the ICHS model will provide the first step to improved capacity building and ongoing evaluation of the ICHS program and to placing ICHS within the continuum of services for children and families with mental health needs in Ontario.


Methodology The overall study methodology was based on a collaborative scientist-practitioner that included a multi-modal strategy that was designed to:

• pilot potential outcome measures for future use • examine the process of treatment implementation in this new program • assess the impact of the first two years of the program and • build capacity with staff and managers for ongoing evaluation of the program.

A standard across all the CDI programs is for clients to be asked to sign consents (Appendix D) for program evaluation and research. Characteristics of participants

Study Groups

The study included all clients admitted into the ICHS program from its inception (n=27). The clients were divided into three groups (new intakes, ongoing clients and discharged clients) and for each group a set of measures were identified to allow comparison of data from intake, with post data. Process measures were introduced for all active clients. Measures were selected to provide best possible comparisons matched to potentially available data. Group 1: included all new clients who entered ICHS during the study period (n=9) Group 2: ongoing clients, included clients who were already in ICHS service prior to implementation of study protocols (n=9) Group 3: discharged clients, included clients who had been in ICHS as the first clients in the transitional program, but had been discharged by the start of the evaluation (n= 9). Measures for the ongoing group included all process measures and the EARL, BCFPI family measures and the CBCL as outcome measures. The discharged group were approached and asked to complete Quality Assurance measures, Child Goal Rating, CBCLs and BCFPI family items as post treatment measures.Discharged families were offered a $10 Loblaws’ gift card for completing parent measures, while children were offered a Mad Libs workbook upon completion of child measures. For some of the analyses, data from different groups were collapsed where the same measures were available for all groups. For some descriptors, groups two and three (n= 18) were collapsed. A brief comparison of the last group of residential clients (15) with the first intake of ICHS clients (15) was also included, based on the families having completed an intake measure, the Brief Child and Family Phone Interview (BCFPI). That comparison included previously discharged clients in group 3, and some of the earlier admissions in the ongoing client group, so that a sample of 15 from each group (Residence and ICHS) could be compared. The details of these comparisons are in the results section.

ICHS Demographic information

The children coming into ICHS in the first two years experienced a range of difficulties across multiple settings and complex family issues. Figure 1 below highlights the range of presenting problems reported by parents at intake.


Figure 1: Parent Reported Presenting Problems for Children Referred to ICHS

The majority of children coming into ICHS were reported as having aggressive behavioural issues across multiple settings, along with other externalizing behaviours such as temper tantrums and oppositional-defiant behaviours. Added to that are poor social skills, difficulties at home and within the school setting and school suspensions. Although the program deals with aggressive and acting out behaviours primarily, a majority of children also came into the program experiencing anxiety. Some children have learning disabilities and a number come to the program with a history of witnessing violence. Most children experience complex and multi-layered problems across different settings. This highlights the complexity of the needs of these children and families at admission to ICHS. Figure 1 does not include presenting problems experienced by less than 4 children, although it is worth noting that at least one child admitted to the program was in imminent danger of removal from home. Figure 2: Mean age of children admitted to the ICHS program


The range of ages of children in ICHS was from 6 yrs to 11 years. As seen in Figure 2, the mean age of children in the ICHS program is lower than the mean age of the children who were in the last residential group, although the difference is not statistically significant. The average age of the children in the last residential group (n=15) at time of admission was 9.27 years and the ICHS comparison group (n=15) had an average age of 8.27 years. The average age of the discharged group (n=9), which was the transitional group beginning the new program was very similar to the residential group (9.8), however, the average age of the new admissions (n=9) to ICHS was 8. One significant change from residence to ICHS, is the admission of girls to the program. There were no girls in the residential program, or the discharged group (first admits to ICHS). The group already in service at the time of the evaluation had 2 girls, while girls made up 5 of the 9 in the new admits to the program. The demographic data on the ICHS study samples (all three groups) includes a total of 27 children and their families. Figure 3 Parent Education Levels (N = 27)

As seen in the Figure 3, parents’ educational levels ranged from some high school to completed university, with the majority who responded reporting some level of post secondary education. Family incomes ranged from <10K to >60K. A majority of families (56%, Figure 5) reported income from wages. However, some of the families indicated income levels (Figure 4), well below the poverty line ($22,012 for single adult with one child, Metropolitan Toronto, National Council on Welfare, 2011).


Figure 4: Family income for IHCHS clients (n=27)

Figure 5: ICHS Family Income Source

Family status in the residential and ICHS comparison groups included single parents, divorced, child living with grandparent, and married/living with partner, for those families who provided family status. For all ICHS clients the primary language spoken was English, while the secondary languages identified were Ukrainian (1), French (1) and Spanish (2). These data should be interpreted with caution, due to changes in data collection and incomplete data. Heritage charts for the ICHS group as well as the ICHS and Residential comparison are included in the results sections and provide a better picture of the diversity of the families across programs.


Data Collection Framework

Two sets of measures were included: outcome and process measures, and timeframes for these were drafted based on clinical work timelines and staff input. Although staff initially selected 2 month intervals for process measures, they later requested switching these to every three months for the duration of the study. The measures dictionary tables are included below. Table 1: Measures Dictionary 1


Table 2: Measures Dictionary 2

Measures for the ongoing group included all process measures and the EARL, BCFPI family measures and the CBCL as outcome measures. The discharged group were approached and asked to complete Quality Assurance measures, Child Goal Rating, CBCLs and BCFPI family items as post treatment measures. Discharged families were offered a $10 Loblaw’s gift card for completing parent measures, while children were offered a Mad Libs workbook upon completion of child measures. For some of the analyses, data from different groups were collapsed where the same measures were available for all groups.


Timing of data collection Table 3: Outcome measures timelines (for new admits)

Although staff initially selected 2 month timelines based on the calendar year for the process measures outlined below, they subsequently requested that these timelines be changed to 3 month intervals. The chart also outlines the other evaluation components that were planned for the study. Due to time constraints, the individual interviews were not able to be conducted, however, an additional components (staff survey) was added subsequently.


Table 4: Process Measures and Study Components Timelines (for all clients in service)

The three groups did not all receive all of the above measures. Group 1 received the comprehensive measures package outlined above. Group 2 received a package of measures that related to initial intake measures but included process measures for all active clients. Group 3 were asked to complete quality assurance measures, CBCLs and BCFPI family items. After 2 months process measures (e.g. alliance, parenting practices, family comfort) were put in place based on the calendar year. This timing ensured that each client in service at that time point would have the same process measures collected. The staff originally selected two-month intervals for these, however, later indicated that data collection was too frequent and impacted therapeutic relationship


and trust. Subsequently, these process measures were collected at 3-month intervals based on the calendar year (February, May and August) for the duration of the study. The duration of the evaluation period was 12 months as outlined in Table 5 below, with a further two-month extension being granted to finalize data collection and report writing. During the first 3 months (October to December, 2011), the study measures were selected and refined in collaboration with the ICHS staff. The extension allowed data collection to take place over a slightly longer period (January through September, 2011). Data was collected based on two assessment strategies, evaluating outcomes and examining treatment process and program development. Table 5: Study timelines

Additional Data

In addition to the standardized measures, three cases studies were evaluated. A single case design was piloted with one family and two case studies were conducted, which included file review, parent and clinician interviews. These three cases will be collapsed into future case history summaries. One parent focus group was held during the year and the year-end parenting group discussion about the parent group component of treatment was included in the evaluation. As noted above, time constraints and difficulties in contacting families previously served did not allow for conducting individual parent interviews. An online staff survey was added to the evaluation in the last few months (Appendix E), to allow staff input and expertise to guide the report, particularly around the process of transition and the benefits and challenges of both the earlier residential service model and the new community based model. Two managers and 10 staff completed the online survey and the results for the two groups were collapsed.


An initial meeting was held with the key LAAC stakeholder group to outline the evaluation strategy, and two subsequent meetings were planned. One meeting was held in early December to review the findings and discuss recommendations, and the second meeting is planned for 2012 to present the final report.

Outcomes and indicators

As noted earlier, outcomes and activities are included in the chart in Appendix B. Indicators are outlined in the Process Evaluation Matrices, which are included below. Table 6: Process Evaluation Matrix 1

Table 7: Process Evaluation Matrix 2


Table 8: Process Evaluation Matrix 3

Measures Selection Process For the ICHS evaluation, the researchers collaborated with ICHS managers and staff, to select a comprehensive set of measures that would provide the best possible match with program/treatment goals (e.g. enhancing family interactions, improving parenting skills, decreasing negative parenting strategies). The measures were selected to provide the broadest range of informants and to ensure that all voices were heard in the final report. Both qualitative and quantitative data were included. The criteria for selection of measures are outlined below. Criteria for reviewing and selecting measures:

• Consistency with measures used within broader agency where applicable (e.g. Child Behaviour Checklist [CBCL], Self Report on Antisocial Behaviour [SRA], Moods and Feelings, CDI Quality Assurance)

• Specific to program outcomes and goals (improved family functioning, child behavior, parent-child relationships, development of child outcome measure based on child intake assessment)

• Utilize mandated measures (BCFPI) where suitable • Potential for comparison with Canadian data (National Longitudinal Survey on Children and

Youth [NLSCY] scales and BCFPI) • Best match to any already existing pre-measures for earlier clients • Ensure child and parent perspective are included as well as clinicians. • Align measurement timing with clinical practice and treatment needs (alliance measures

across time, pilot process – session and group rating scales as potentially valuable to informing clinical practice).

• Ensure that both process and outcomes considered in selection of measures • Multi-modal evaluation tools using a variety of inputs and data sources • Develop mixed-methods measurement strategy – including both qualitative and quantitative

data


• Measure selection based on collaborative process with staff, program managers, as capacity building for ongoing assessment, research and evaluation activities upon completion of preliminary evaluation

• Ensure that measures and evaluation process remain useful for informing clinical practice • Ensure that clinical practice and treatment needs are considered in recommendations for

future/ongoing research practice • Ensure that clinician understanding/buy in occurs during measure selection process • Promote CDI scientist-practitioner model for ICHS clinical staff and managers • Mandated measures (BCPFI)

Sources of Information

Measures Selected for the Evaluation Study

Several measures were selected as these were utilized across the other CDI programs and some may become required measures for all early intervention services. These measures included the Child Behaviour Checklist (CBCL) and the Early Assessment Risk List (EARL), which was developed through the Centre for Children Committing Offenses (CCCO), a research, training and dissemination division within the Child Development Institute (CDI). Based on the above criteria, the following measures were selected.

BCFPI. The Brief Child and Family Phone Interview (BCFPI) The BCFPI (Cunningham et al., 2009) is a mandated telephone intake measure for children aged 3-18 accessing treatment at a children’s mental health centre in Ontario. The BCFPI norms are based on Ontario child and youth samples. The BCFPI provides standardized scores on norm-referenced data in the areas of family functioning, child functioning (internalizing, externalizing, total mental health), and caregiver mood scores. The BCPFI also provides background and demographic data on families seeking service. Content validity was based on mapping selected items onto DSM-IV criteria. (Cunningham, Pettingil and Boyle, 2004), The BCFPI is conducted as the second of two telephone intakes for all children requesting service at CDI, however, due to the severity of need of families seeking ICHS services, the BCPFI occasionally was done in person during the initial evaluation process. For the purposes of the study, the family subscales of the BCPFI were selected and collected as both process and outcome measures, as a potential measure of improvement on family functioning across time and groups.

Child Behaviour Checklist (CBCL) The Child Behaviour Checklist (CBCL) is utilized across SNAP® programs within CDI, for the Outreach Project (ORP) and the Girls Connection Program (GC), and was implemented for the ICHS Evaluation as well. The Child Behaviour Checklist (Achenbach, 1991) is a 113-item parent report on their child’s behavioural functioning with acceptable to strong reliability and validity. The CBCL is a standardized, widely used measure that assesses many different aspects of child behaviour, mood, and emotional adjustment. Items are rated as “not true,” “true,” or “very true,” as pertaining to the child. Scale scores in the Clinical range suggest a high level of maladjustment (top 2% of the population). Scores in the Borderline Clinical range may identify a significant problem that may not be severe enough to warrant formal treatment or may identify the potential of developing a problem that requires careful monitoring. In addition to behavioural information, the CBCL also measures children’s strengths and competencies in the area of school and community activity. Both the BCFPI and the CBCL are parent reports, with the BCPFI usually conducted as a prescreening telephone interview, while the CBCL is part of the subsequent intake package of measures for SNAP® clients.


Assessing Risk Level: The Early Assessment Risk Lists (EARL) The Early Assessment Risk Lists were developed through the work of the CCCO at CDI as part of a strategy for early identification of children at risk and are clinician-completed measures. The Early Assessment Risk List for Boys (EARL-20B) and the Early Assessment Risk List for Girls (EARL-21G) are part of a comprehensive psychosocial risk assessment framework and focus specifically on young children 6-11 years of age with significant behavioural issues (Augimeri et al., 1998; Augimeri et al, 2001; Levene et al., 2001). The purpose of these tools are: to provide a platform for increasing clinicians’ and researchers’ general understanding of early childhood risk factors for aggression and antisocial behaviour; to help identify risk levels for particular children, according to acceptable structured format; and to assist in the creation of effective clinical risk management plans for high risk children and their families. The EARL-20B for boys appeared as a “consultation edition” in 1998, and has since been updated in 2001 to a Version 2 (Augimeri et al., 2001). A parallel, professional tool for girls was issued as a “consultation edition” in 2001 (Levene et al., 2001). These tools, the EARL-20B and the EARL-21G have received international acclaim and are being used extensively in Canada, Scandinavia, New Zealand and the United States. Clinicians complete a summary sheet to identify a child’s risk in three areas, family, child and responsivity. The rating scale for risk is measured from 0-2, with 0 being no risk and 2 being high risk. The child’s total score indicates the potential future risk of these children engaging in antisocial behaviors (Hart, 1999). Completion of EARLs during the early phase of ICHS was not consistent. During the study period, staff were asked to complete EARLs for clients in service at intake and at the identified 6 month time periods.

Parent Measures

Parent Depression: Beck Depression Inventory (BDI) The Beck Depression Inventory (BDI) is used to assess parental depression. The Beck Depression Inventory, version 2 is a 21 item self-report measure to assess the severity of depression symptoms in clinical and normal populations (Beck, Steer, & Brown, 1996). The BDI-II is a widely used measure with high internal consistency and construct and discriminant validity (Steer & Clark, 1997).

Parent Stress: The Parenting Stress Index (PSI) The Parenting Stress Index-Short Form (PSI-SF) was used to assess parenting stress. The Parental Stress Index-short form (PSI-SF; Abidin, 1995) is a 36-item parent report comprised of three scales measuring parental distress (from various aspects of their experience), difficult child characteristics (items relating to child temperament), and dysfunctional parent-child interactions (how much satisfaction parent derives from interactions with child). There are 12 items scored between 1 and 5 in each of the sub-scales to a total of 60, with high scores indicating less stress. The total score is out of 180. Test-retest reliabilities are high and vary from .84 for the total score and .78 for the difficult child scale. Concurrent validity has been established by comparing the short form with the long form of the scale that is well validated. The correlation was .94 (Haskett et al, 2006).

Social Support: NLSYC Social Support Scale The Social Support Scale used within the NLSCY framework, (Statistics Canada, 2010 (a)) was a shortened version of the Social Provisions Scale (Cutrona & Russell, 1987) and is used to measures components of social relationships: guidance, reliable alliances and attachment. The social support scale has a total of 6 items, with 4 possible response categories. The Cronbach alpha was 0.82

Parenting Measures The Parent Practices Scale (Strayhorn & Weidman, 1988) measures how parents interact with and parent their children. It is a 34-item questionnaire that measures frequencies of particular parenting


behaviours, reactions, routines, and consequences. Responses are rated on a 7- point likert scale. A total score is derived which measures how favourable the parent practices are. This scale was used in the National Longitudinal Survey of Children and Youth (Statistics Canada, 2010 (a)) and was also included in a longitudinal follow up assessment of the girls program (GC) at CDI. Hence the data had the potential to be compared to the GC program data and/or national data for Canada. The Rational Parenting Scale. Within the Parenting Practices Scale is a subscale identified as the Rational Parenting Scale of which 4 items were found to load onto a coercive parenting practices factor (Statistics Canada, 2010 (a)) , 2 of which relate to coercive parenting practices and the 2 others relate to supportive parenting practices. The Rational Parenting scale was piloted as a potential process evaluation measure to determine whether it could discriminate in expected parenting changes (fewer aversive and increased positive interactions).

Child Measures

The Self‐Report Antisocial Behaviour Scale (SRA)

The SRA (Loeber et al., 1989) is a measure of self-reported delinquent activities (past 6 months), perceptions of problem behaviours, self-reported peer delinquent activities, attitudes toward delinquent behaviour, and mood/feelings questions (past 2 weeks). The SRA is an instrument that has been used with other SNAP® programs (ORP and GC).

The Short Moods and Feelings Questionnaire (SMFQ) The SMFQ (Angold et al., 1995, ) is a short, easy to administer depression inventory for children, that is included in the SRA, which has been shown provide a brief and reliable measure of a core depression construct in children and adolescents (Messer et al, 1995). This measure was used in the follow up evaluation of the SNAP® Girls Connection.

Parenting (child perception) The “My Parents and Me Scale” assesses the child's perception of his/her relationships with his/her parents and their parental supervision and was utilized in the NLSCY (Statistics Canada, 2010 (b)). The questions were based on the Western Australia Child Health Survey developed by Lempers et al. (1989) and measures parental nurturance, rejection and monitoring. These complement the constructs measured in the parent-completed parenting questionnaire (positive interaction, hostile/ineffective parenting, consistent parenting, aversive and non-aversive parent management techniques).

Therapeutic Process Measures

Outcome Rating Scales During the first 2 months of the evaluation data collection, measures of therapeutic outcomes were piloted, to determine if they had clinical value. These measures were the Outcome Rating Scale (ORS),(Miller and Duncan, 2000), Session Rating Scale (SRS V. 3.0), (Miller, Duncan and Johnson, 2002), Group Session Rating Scale (GSRS) and Child Outcome Rating Scale (CORS), (Duncan, Miller and Sparks, 2003). These are a complementary set of measures, which focus on the therapeutic process itself. These measures were piloted for 2 months, to determine their clinical utility and measurement value. At the end of two months, these measures were withdrawn and alliance measures put in place as both families and staff found this difficult to implement on a regular session basis.


Alliance Measures As alliance is an important component of building a therapeutic relationship, two alliance measures were included in the process measures, the Working Alliance Inventory for the parent and therapist and the Therapeutic Alliance Scale for Children, for the worker and the child. Both measures have a client and worker version. Short forms were used for both sets of measures and only total score was utilized.

Working Alliance Inventory The Working Alliance Inventory, WAI (Horvath & Greenberg, 1989) measures the parent-therapist alliance. It includes 36 items, assessing agreement of the task and their relevance in therapy, and mutual endorsement of the goals and outcomes of treatment. As well, it measures the extent to which there is a positive personal attachment, acceptance, and confidence in the therapeutic relationship. The short form- revised (WAI-SR) of the WAI has 12 items for client and therapist.

Therapeutic Alliance Scale Therapeutic Alliance Scale for Children (TAI) (Shirk & Russells, 1996) measures the child’s alliance with therapist. The short form (TAI-S) for both therapist and child version includes 12 items. This version assesses if the child considers the therapist to be an ally, likes spending time with the therapist and feels she/she can tell the therapist problems.

Measures of Family Functioning

The Family Assessment Device (FAD ‐ Global Scale) The FAD is based on the McMaster Model of Family Functioning. The measure contains 6 scales (problem solving, communication, roles, affective responsiveness, affective involvement, behavioural control) and one global rating of functioning, which assesses overall health, and pathology of the family. Both the internal reliability and validity of the FAD have been demonstrated in prior research (Miller, Epstein, Bishop, & Keitner, 1985) with Cronbach's alpha for the Global Scale being 0.92 (Epstein, Baldwin, & Bishop, 1983). The global scale was selected for this study, for ease of use and as an overall measure of the quality of family interaction. The FAD Global Scale is a short, 12-item self-report measure of family functioning that describes emotional relationships. The items are ranked on a 4-point likert scale with alternate questions having negative loadings.

Teacher Report

Child’s behavioural functioning. The Teacher Report Form (TRF; Achenbach, 1991) is a standardized teacher-report questionnaire with acceptable to strong reliability and validity. The TRF is similar in design and format to the CBCL, but omits items unlikely to be observed in the school setting and includes new items appropriate to the school milieu. It can be used to make comparisons between parent and teacher perceptions across the home and school environments. The TRF produces scores on a variety of scales (e.g., aggression, depression) as well as reports on school achievement, engagement, and teacher reported concerns/areas of strength for each child. Due to the tight time frame of the evaluation and the focus on collaboratively developing and selecting appropriate outcome measures, the Teacher Report form was only collected for a few students in the onsite classroom and results will not be reported in this paper. In future, at least in the Section 28 classroom located at CDI, the teacher reports will be collected.


Staff Survey

A late addition to the evaluation study was an online survey, to allow staff to anonymously provide insights, feedback based on their clinical expertise. The online survey was developed in collaboration with the program managers and the research manager. Two sets of data were collected, one from staff and the second from the two managers of the ICHS program and the clinical director at CDI. The two groups were collapsed for the analysis, however, staff feedback will be utilized to inform clinical practice, supervision and ongoing program development.

Focus Group

A semi-structured interview was developed for a focus group of parents attending the ICHS parent group. The focus group was transcribed and a qualitative review of themes conducted. A second less structured evaluation that was conducted by staff at the end of parent groups was also taped, transcribed and included in the focus group outcome. Themes from the focus group are included in this report, but a separate internal agency document outlining the focus group process and themes has been developed in addition to this report, which is available upon request.

Case Histories/Single Case Design

Initially, a single case design was developed, based on treatment goals and parenting strategies. A description of the process is included in the results description. Because of the complexity of working with the selected family, the single case design was adjusted and became a case history. The case history story of this particular case was augmented by interviews with the worker, supervisor and parent group supervisor. Two case histories were selected based on the following criteria:

• one case of each gender (one boy, one girl) • availability of worker to provide insights and input into the assessment and treatment

planning and implementation • currently in treatment • complexity of the case (de-selected very complex cases)

As both of the selected cases were ongoing at the time of the evaluation they provided a good overview of the challenges and successes of working with families with complex needs. Parents were invited to contribute to the case histories and both parents consented and were interviewed. Their interviews were taped and recorded. Parents have also been given an opportunity to update their family situation, and to review the specific case report of their child. A summary of the common elements of the various cases considered is included in the evaluation report. The case studies will be presented as part of a symposium on children at Risk at the 12th Annual Conference of the International Association of Forensic Mental Health Services (IAFMHS). It is also anticipated that these case histories will be submitted for publication to a case history journal, in that they do provide extensive clinical knowledge about treatment implementation, family burden and outcomes.


Measurement of Treatment Fidelity

One of the strengths of the current ICHS program is the adaptability and flexibility of its services, which ensure that families are individually and specifically supported in their areas of need. In order to track treatment fidelity and dosage, clinicians note services and service hours in the agency wide information system. The information collected includes description of service administered, length of sessions, frequency, who was targeted, and length of time in active service, etc. For group sessions, group facilitators track the topics discussed in the session and the activities organized for the group. Service records are tracked for each client and these were reviewed and summarized for this report.

Quality Assurance In the pre-treatment interviews, parents and children are asked to provide their goals for treatment, as well as where they see challenges and what they view as effective aspects of the program. Once the intensive services are completed, respondents complete the agency quality assurance questionnaire (for adults). For the purposes of the ICHS evaluation, a similar quality assurance questionnaire (for children) was developed. The families in the discharged group were also asked to complete the Quality Assurance measures. For the purposes of this report, all responses were collapsed.

Evaluation Limitations

Time frame This evaluation was limited by a number of factors. There were actually 3 groups for whom we sought to obtain consistent within group data. One set of clients who had been discharged from treatment at the time the study began, were approached by their workers and asked to provide input regarding their outcomes. While we were able to obtain data from several families, we were not able to obtain feedback from all in the discharged group. For the second group, ongoing clients, we sought to match data collection to what was potentially on file from intake, to allow comparisons to be made, however there were missing data points. For the newly admitted clients the data collection period was less than 10 months and limited pre-post measures obtainable during this time. In an effort to capture data to present a comprehensive picture of the ICSH program we selected a broad range of measures with staff input, however, due to the small numbers this did not allow us to do the planned level of analysis.

Case Studies As outlined above, the case history and single case designs may be merged into our case history future presentations. The single case design brought to light the difficulty of compliance with certain families, as well as the flexibility that the ICHS program provides in structuring and changing focus to meet family needs. Even for these families, at the time of projected discharge from ICHS, complex situations arose, which required the workers to continue to provide follow up care and transitional support. The complexity and high needs of the families in ICHS meant that at times, families were not prepared to, or able to work on filling out measures, facing too many life stressors or difficulties to be able to comply with data collection time frames.

Overall Given the complexity of the transition from a residential program to a community based program and that these two programs overlapped, there were gaps in the data during this change over period. While there appear to be increasing numbers of girls admitted to the program, the sample sizes did not allow us to conduct gender analysis.


Due to time constraints and the difficulty in reaching families in the discharged group, the planned follow up interviews could not be scheduled, however, some follow up measures were obtained from 5 families and these have been incorporated into the results section


Results Evaluation Findings

As noted above, for some of the analyses, the groups identified in this report were combined where measures were available across groups. For other comparisons, such as new admissions, descriptive reports on the measures piloted during this evaluation are included. These results are to be considered preliminary only and the measures evaluated, based on not only the results, but also the ease of completion by clients and the clinical utility to the staff providing services. The number of measures in this preliminary evaluation was burdensome to some clients, and the recommendations for ongoing evaluation and research reflect a much more tightly designed evaluation process. Residence and ICHS Comparison

Although this evaluation was not designed to compare residential and ICHS clients, a brief comparison was made. The last 15 clients in Residence and the first 15 clients in ICHS were used to examine where there were any differences in the presenting problems, levels of behaviour and demographic characteristics of children referred to either program. The 15 clients for each of the groups mentioned above were selected based on their sequential intake (last group in residential and first group in ICHS) selecting only those with a completed BCPFI on file. Mean scores of each of the BCPFI scales were analyzed using a one-way ANOVA. There were no significant differences between the scale scores of the two groups. A graph of the scores is attached below. Although the difference was not significant, it is interesting to note that the mean age for ICHS clients was one year younger (8.2) than the residence group (9.2) as outlined above and this is consistent with the perception of some staff (as noted in the staff survey). Figure 6: Comparison of Residential and ICHS mean scale scores on the BCFPI.


Family status in both groups included single parents, child living with grandparent, and married/living with partner, for those families who provided family status. Educational levels ranged from completed high school to completed community college for the Residential group, while for the ICHS group, educational levels ranged from some high school to completed university. For all clients in both groups, the primary language spoken was English, while the secondary languages identified by the Residential families were Ukrainian and Italian, and Spanish (2 families) for the ICHS group. These comparisons should be interpreted with caution, due to changes in data collected and incomplete data. Heritage identified by the two groups (Figure 7) provided more information than the secondary language, with both groups having Caribbean/ Caribbean-Canadian families (10%), and Canadian (40%). Figure 7: Heritage of Residential and ICHS comparison groups

ICHS Program Results

Heritage

For all ICHS clients (n=27) the primary language spoken was English, while the secondary languages identified were Ukrainian (1), French (1) and Spanish (2). As with the above comparison, heritage provides more information than the secondary language and Figure 8 below does provide an overview of the diverse heritage of the children in the first two years of the IHCS program.


Figure 8: Heritage of ICHS clients

Behaviour problems at Intake

Initial intake behaviour problems are assessed using the BCPFI. As indicated in Figure 9 below, both girls and boys experience difficulties in the areas of conduct and externalizing behaviours, but also in relationships, within the family and school functioning, with girls’ scores slightly more elevated than the scores of the boys at intake. Figure 9: ICHS Intake - BCFPI by gender (girls, N = 4; boys, N = 13)


Initial Assessment: CBCL

The CBCL is another measure that is used during the initial assessment. There are three sets of scores that the CBCL provides, individual scale scores, composite scale scores and strengths scale scores. Figure 10 below provides an overview of the mean CBCL scores for ICHS children at intake (within the first 3 months). Figure 10: ICHS Intake - CBCL Scale Scores (N = 20)

As would be anticipated, for children referred to the ICHS programs, problems in the externalizing scales (rule breaking and aggressive behaviours) are in the clincial range (i.e., 70 or greater). In Figure 11 below, the composite scale scores reflect that in the externalizing scale, which is in the clinical range (i.e., 64 or greater). When the scores are plotted onto DSM criteria, of Oppositional Defiant Disorder and Conduct Disorder, the mean scores for children are referred to ICHS are also in the clincal range (i.e., 70 or greater). Finally, in looking at the scales that would reflect strengths, Figure 12 below, in which higher scores indicate less distress, it is clear that these children also struggle in the areas of social interactions, school functioning and total competence (31-35 borderline, < 31 clinical for individual scales, for total competence scale, <37 clinical range, 37-40 borderline). Only in activities are these childen in the normal range. The areas children struggle in are all areas that community based treatment can address and help improve, along with the behaviour management and cognitive behavioural strategies of the SNAP® approach that both parents and children learn in the program. So there appears to be a good fit between the goals and strategies of the program and the intake profile of the children admitted to ICHS.


Figure 11: ICHS Intake – CBCL Composite Scale Scores (N = 20)

Figure 12: ICHS Intake - CBCL Strengths based scales (N = 20)


Risk Assesment (Early Assessment Risk List)

The EARL risk assessment measures have been developed by the CCCO as described earlier and these are risk identification and clinical case management tools that are being implemented across various programs. EARLs were completed at intake and at 6-month follow up time points. Figure 13 below shows the mean scores for girls (N = 4) and boys (N = 16) at intake. It should be noted that for the EARL-21G (gender-specific risk assessment for girls) has a total score of 42 versus the EARL-20B (boy’s assessment tool) total score of 40. The EARL-21G has one item more in the family items than the boy’s EARL-20B, so the comparison of the girls’ and boys’ family items is based on an available 7 items for the girls (9 out of possible 14), and 6 for the boys (6 out of possible 12). The other scales have comparable number of items. It is interesting to note that the girls score higher on the child risk items, however the boys score higher on the number of items identified as critical risks, at the time of ICHS intake. Overall, except for critical risk, the girls score higher on identified risk. Figure 13: Intake EARL scores


Figure 14: Clinical Risk Judgment

Upon completion of the EARL forms, clinicians are asked for their clinical judgment about whether the risk for the child is low, moderate or high. Figure 14 shows the mean of these judgments for 6 girls and 16 boys. It is interesting that percentage wise, girls are grouped equally between the three categories, while a smaller percentage of boys are seen as low risk.

Critical Risk at admission For the girls, the items identified as critical risk items were mother-daughter interaction, early onset of behavioural difficulties, peer socialization, sexual development, school functioning, anti-social behaviour, and coping ability. For the boys, more items were identified as critical risk, with the key items including early onset of behavioural difficulties, hyperactivity/impulsivity/attention, abuse/neglect/trauma, parenting style, antisocial values and behaviour, authority contact and coping ability.

Additional Stressors/Risks

Other risk related items. Although the families are not diagnosed according to DSM IV criteria, as was seen in the CBCL composite scores, their behaviours map onto several DSM IV diagnoses. In addition, many of these families face circumstances that map onto the DSM IV Axis IV index of stressors, Figure 15 below. Clinicians reviewed the stressors of the DSM IV, Axis IV and noted how many children/families experienced stressors that could be mapped onto that tool. These are not to be considered diagnostic, but simply plotting child and family circumstances and stressors on these axes by how many families were impacted. Please note that in spite of Ontario Health Insurance (OHIP) coverage, there were a number of families who needed support to access medication or locate a primary health care provider or specialist for assessment. This is not a clinical DSM IV diagnosis, but rather an overall indicator of how many areas of difficulty these families experience. All children in Groups 1(new admits) and 2 (ongoing clients) were plotted on Axis IV, by an experienced clinical manager. All children in both groups experienced disruptions in their primary support group, which could be separation from one or both


parents, caregiving by grandparent, father absence, times in care and other similar issues. All children also were experiencing difficulties with school, which may reflect learning disabilities, and/or behaviour management issues in the classroom. Many children had multiple suspensions and were referred to ICHS because they could no longer be managed within the regular school system and therefore needed specialized educational care in the on site classroom. In each group there is at least one child who had faced serious violence (one child saw someone shot in his backyard). This highlights the high risks of these children and families and the complexity of care required to address multi-layered needs. Figure 15: ICHS child/family stressors on DSM IV, Axis IV (n=9 each group) * indicate taking the whole family into consideration

Parent Depression at Intake

Although the families experience multiple complex life stressors, the majority of parents did not appear depressed at intake, Figure 16 below, as measured by the BDI (n=7). For 57% of the parents, the BDI score was in the normal range (1), with an equal percentage in the mild (2), moderate (3) and severe (4) categories.


Figure 16: Beck Depression Inventory at Admission:

Parenting Stress at Intake These ICHS parents who were experiencing multiple stressors and difficulties with parenting a child with behaviour problems, did reflect these stressors as measured by the Parenting Stress Index. Figure 17 below, highlights 4 scales of the PSI, by client (5 clients) for whom there were intake PSI measures. While not all of these parents indicated parental stress, all scored high on the difficult child scale, and most scored high on the other scales. Figure 17: PSI at intake by client


Outcome measures

As there were three different groups and the actual measures available across groups varied depending on which time period the clients were from, the number of clients with reported results on outcome measures differ also, depending on the available data. The new measures that were selected for this evaluation were implemented as a trial phase for Group 1, those coming into service during the study period. As some were still in process when data collection finished, the numbers are too small to analyze, but provide some interesting descriptive data.

Child Behaviour Outcomes

Pre and Post CBCL The Child Behaviour Checklist is a standard across children’s mental health services and was also used for the ICHS evaluation. We sought to compare children for whom we had initial (intake) CBCL, with their follow out data. There were 6 children, for whom there were pre-post (intake T1 and follow out T2) data on the CBCL. The follow out time period ranged from 12-24 months. Although the numbers are small and do not allow for statistical analysis, the results are promising. Figure 18 below indicates the drop seen in the total CBCL scores from intake to follow out, while the subsequent. As can be seen, there is a large drop (10 points) on the CBCL total score. This brings the mean children’s score from the clinical range to below the clinical range. Figure 18: CBCL Total scores, T1 and T2

Figure 19 shows the two key composite categories in the CBCL, Externalizing and Internalizing behaviours, showing a drop in both Externalizing and Internalizing scores, although the difference is much larger for the externalizing scores. This would be expected, given that these behaviours were in the high clinical range for children at admission to the program, and the program is designed to help bring down levels of aggression and acting out, by assisting parents with behaviour management and children with self-regulation. Figure 20 shows the reduction in the rule-breaking and aggression scales. It is interesting that there is also some improvement on internalizing behaviours.


Figure 19: CBCL Externalizing and Internalizing Scores, T1 and T2

Figure 20: CBCL Rule Breaking and Aggression, T1 and T2

The steepest drop was in aggression, although both rule breaking and aggression dropped to below the clinical range (>70).


As can be seen when reviewing the change in total score by individual client, Figure 21, the majority (5) of the children showed decreased scores. This was also true for externalizing behaviours, Figure 22. When examined individually, there was one child for whom this pattern did not appear and for him, this may not have been the best fit as this child was diagnosed with autism-spectrum disorder during ICHS. So although the staff noted that were improvements in the family functioning, the impact of the program on the child’s functioning as measured by the CBCL was minimal. This does point out the importance of using multiple measurement sources where possible. Figure 21: CBCL Total score by child at T1 and T2

Figure 22: CBCL Externalizing child at T1 and T2


Figure 23: CBCL Internalizing by child at T1 and T2

As can be seen when examined by individual child outcome the decrease in externalizing holds true for most children, however, the change in internalizing is more mixed across the group.

Evaluation of Clinical Significance

One further analysis was conducted, which allowed the clinical significance of child changes (with small sample sizes) to be plotted on to the chart in Figure 24 below. The chart is in Portuguese, because the program that created it was developed in Brazil, however the titles are fairly self-explanatory – pre-treatment at the top and post treatment along the right. In essence the children are plotted on 4 quadrants on a specific measure, in this case, the CBCL, to determine the clinical significance of change (Atkins et al 2005, Beutler & Moleiro, 2001). If they are plotted to the left of the vertical line, they were an appropriate referral for service. If they are plotted below the horizontal, they are in need of continuing service. If they are above the horizontal line, they are no longer in need of service, having change that is clinically significant. This was interesting, in that one child was in the upper right quadrant (appearing to not require service) was in the program. Case examination determined that although this was a child with very aggressive behaviours in a school setting (e.g. throwing desks, refusing to work), in the home setting there were no limits set on his behaviour (e.g. if he refused to complete his schoolwork, the mother would complete it for him or allow him not to do it) and hence, he was not triggered at home as he was at school. Consequently, the mother’s report on the CBCL did not indicate behaviour problems. Upon home observation and school contact, the child was determined to fit admission criteria and was admitted to the program. The placement of the other 6 children was reviewed with the clinical managers and the children in the lower left quadrant are all still in service, but moving towards discharge, while the children in the upper left quadrant are discharged. Clinical managers indicated that this was an approach that accurately placed the children and could have clinical utility in discharge planning. This does highlight again, the value of multiple assessment strategies to determine appropriate placement and discharge.


Figure 24: Clinical significance of change


EARL Outcome

Time one and Time three, (which was 12 months from intake) were plotted for 4 boys in the ICHS program and as can be seen in Figure 25 below, their scores dropped in each of the areas of risk. The EARLs were developed as tools for early risk assessment and clinical treatment planning, but these results show promise in terms of being sensitive to risk reduction through the treatment process. Due to the limited number of girls, a similar comparison could not be made for girls. Figure 25: EARL20-B (boys) T1 –T3 (n=4)

Additional Outcome Measures

BCPFI Family Measures

The BCPFI family measures were piloted to determine whether these would be sensitive to change over time, without having families repeat the entire BCFPI, which can take between 30-60 minutes on the phone. In reviewing the intake BCPFI, family activities, family comfort and global family situation were problematic for the families. Time 1 was intake BCPFI, while time 2 was the initial collection at 6 months, and time 3 was 12 months, however, for some families, their initial intake was longer than 12 months prior as it had occurred prior to the evaluation being implemented. So the timeframes are not consistent, however, it is interesting to note that over time, change does get reflected in these measures. For these scales there were 5 families for whom there were T1 and T3 scale scores (each series is single family). Family Activities are plotted in Figure 26 below (case 1 and 4 overlap, having identical scores at both time points). Higher numbers indicates higher distress, so the change indicates positive changes in each family’s ability to do family activities together without disruption. Family Comfort at T1 and T3 and Family Functioning at T1 and T3 are plotted below.


Figure 26: BCPFI Family Activities T1 and T3 by child.

Figure 27 below shows a positive change in family comfort from intake to T3. Figure 27: BCFPI Family Comfort T1-T3


Figure 28: Family Functioning T1-T3

Family functioning, which may reflect the most difficult of these areas to impact, changed positively for several of the families, but not necessarily for others, however these scales are easy for families to complete and may be worth continuing to collect to see whether they do discriminate change over time. Families have ups and downs during the process of treatment, so this scale may prove useful when comparing treatment process over time. We also examined the three positive questions about discipline style (reason with, send to room, take away privileges) to see whether there was an increase in the use of appropriate discipline over time (where 1=never use and 4= always use) Figure 29 below shows that the frequency of use of positive strategies increased for these families (4 and 5 overlap having identical scores at both time points). Figure 29: BCFPI: Positive Parenting Strategies T1 and T3


NLSCY Measures

Although we sought to implement the NLSCY, Parenting, Social Support and Child Parent measures as outcome measures, and the Rational Parenting scale as a process measure, the timing and data collection issues relating to these did not allow us to analyze any outcome data for these.

Child Measures

Child Measure: SRA Overall children reported a variety of delinquent behaviours (e.g., violence, offence status, damage). For the four children that completed pre-assessment on the SRA, there was a mean delinquent score of 5.75 (range 2-11). No post test assessment was completed.

Child Moods and Feelings For the Moods and Feelings questionnaire scores that are 8 or higher indicate a serious level of depressive symptoms. For the same four children, it was found that two of the four showed some level of concern with regard to their mood (8, 15) with the scores of the other two children being 4 and 7. No post assessment was completed.

Process Alliance measures

We sought to utilize a broad range of alliance measures, based on 3-month intervals on the calendar year, however, this turned out to create problems for analysis. The results for the BCFPI family measures are included in the results section above. One of the difficulties with the calendar year schedule was that measures did not actually map onto each child’s individual status (e.g. intake, 6 month review, 12 month review). Hence these measures were very difficult to interpret and the results for the other measures piloted are not included in this report. Also, as noted earlier, this was a very ambitious evaluation project, during which we had an opportunity to receive both staff and client feedback about the value and timing of various measures, and the discussion section clarifies the outcomes and plans for future evaluation.

Single Case Design

A client for the single case design was selected based on what appeared to be treatment goals that were compatible with a single case design format. The client’s presenting problems were described as “tantruming” in the community. Upon initial assessment, the clinician noted that family lacked structure and routines, including structure and routines about bedtime, and the child getting adequate rest. Initial goals of setting bedtime routines and structures around evening meals were collaboratively developed with the family. The family was given simple charting and tracking tools for recording and rewarding following routines. The family was resistant to the charting and to the establishment of bedtime routines. When the charting and tracking appeared to be negatively impacting the ongoing therapeutic alliance, the recording was withdrawn after a trial period of several months. At that point, it became clear that it was not only the recording that the client found problematic, but the whole set of goals around structured evening and bedtime routines that were not something the client wished to pursue. Subsequently, a new set of goals was developed collaboratively with the client, which focused on community outings and how to plan for outings and deal with tantrums in the community. The flexibility of the ICHS program allowed the transition to goals that the client was willing to actively work towards. The single case design was withdrawn. Interviews with the worker and program managers were conducted to debrief the single case design. Some aspects of this case may be included in the case histories, when these are written up for publication.


Case Histories

Two case histories were selected based on the selection criteria as described earlier in this report. The case histories were based on clinical file reviews and interviews with the workers and the parents of the children. Parents gave their consent to having their child’s case discussed and presented. The case histories will be presented as part of a symposium on children at risk. Parents will be given an opportunity to review the case summaries prior to the conference presentation. For the purposes of this evaluation, a summary of the case histories, common factors, child risk and outcomes are described below, focusing on the EARL risk lists and notes from the child’s ecosystemic assessment. One girl (age 9) and one boy (age 11) were selected for the case history based on availability of worker, current cases and parent willingness to participate in telling the story of their child’s treatment. At the time of calculating treatment intensity (Sept. 15, 2011), the girl had received 500.5 hours of service through ICHS, with earlier service including SNAP® Girls Connection and Family Counselling for total service hours of 583. The boy had received 437.25 hours of service, which was ICHS only. Both were female led single parent families, and both children had siblings. Although one parent was on social assistance and the other was employed, both families had incomes below the poverty line for their household compositions. Parents and workers were invited to contribute to telling the story of the child needs and the treatment program.

Common factors Both cases were single parent households, with early onset of behavioural difficulties, sibling conflict and academic non-achievement, and frequent difficulties within the school system (frequent suspensions [girl], severe enough aggression to warrant police involvement and mental health assessment within a hospital setting [boy]). Both children had little or no contact with their biological father. Child welfare involvement in one case had resulted in early history of foster care, after which mom described the child as aggressive and “suicidal”. There were allegations of abuse during foster care for the female client, however, these allegations had not been proven. Subsequent to ICHS treatment, the child began to disclose abuse during foster care and is now receiving trauma counseling with the ICHS worker. Previous external trauma treatment was ineffective as the child was unable to build trust, thus it was essential that trauma treatment occur with a counselor with whom she has a trusting relationship. As a result of disclosure during the trauma treatment process, the investigation has been reopened and this appears to be serving as a form of empowerment for the child. The child in the other case, has been taken into care by child welfare and concerns exist that progress made during ICHS treatment may be lost if the child is not returned home in a timely fashion.

Presenting Problems The assessment at intake using the BCFPI, showed that this female client was experiencing significant difficulties in externalizing behaviours and managing mood as well as school difficulties (clinical). These are areas that the ICHS program could provide treatment and support towards improving, so the referral and placement was a good match of treatment for the child and family needs. The male client presenting problems on the BCPFI at intake did not necessarily reflect the high level of externalizing behaviours that were later described by the parent (fights at school that required police intervention), although externalizing behaviours and difficulties at school were reflected.

Strengths and Resilience Child and family strengths for both families were identified from the eco-systemic assessment conducted at intake. Parental commitment to the child was a positive factor in both cases as was their willingness to engage in the services, in spite of wait times and previous negative experiences with services and/or child welfare agencies. Child likeability was another positive factor in both instances.


Specific strengths for the girl were that she was friendly, talkative, artistic, smart, affectionate, and likeable. The boy was also likeable, engaging, thoughtful, talkative and intelligent. However, building trust was an issue for both children and the workers spent time relationship building in order to be able to provide treatment.

Treatment Treatment in both cases consisted of identifying and working towards individualized treatment goals, training in SNAP® strategies for parent and child in behaviour management, emotional regulation and cognitive behavioural techniques. ICHS workers built trust with the family, made home visits, worked with the child in local school (m) and the child within the onsite classroom (f) and supported sibling relationships and relationship building.

ICHS Discharge and Transitions When the intensive phase of services was completed and behavioural issues could be successfully managed, the goal was to discharge both clients from the ICHS program. At that point, for each of the clients, additional issues came up that required the worker to remain involved. In the case of the female client, once the child felt safe and home and school settings were stabilized, earlier abuse while in foster care was disclosed. The child is now receiving follow up trauma treatment with the ICHS worker. In the case of the male client, an unfortunate occurrence resulted in the child being taken into care. Again, this resulted in the ICHS worker continuing to provide support to the family in seeking to return the child home. Both these cases highlight the difficulties faced by ICHS staff in transitioning families into additional needed supports once the intensive cognitive behavioural and community and family support treatment components are completed. While treatment was successful, prior to discharge other issues arose, highlighting the complexity of these cases and the importance of a continuing care model of service delivery for high risk, high needs families. Child risk at intake and at discharge was assessed as outlined in the EARLs below. Figure 30: EARL-21G (f) at Intake and 18 months.


Figure 31: EARL-20B (m) at Intake and 24 months.

Child Risk and Outcome The child outcomes as rated on the EARL risk factors are seen in Figure 30 and Figure 31 above. It is interesting to note that for the girl, the child and family risk items both were reduced after the treatment phase, however, at 18 months, the overall clinical judgment was still of a child at moderate risk, reflecting the disclosure of abuse while in foster care. For the boy, the initial high level of family and child risk items was greatly reduced by 24 months; there was an increase of child risk at time 2, which reflected disruption in caregiving, when the child was taken into care. The incident that resulted in this was based on the mother leaving the child in the care of family friends when she went to her own father’s funeral in the Caribbean. This may reflect a lack of understanding of culturally appropriate childcare arrangements within the family’s culture of origin, as the mother’s comments were “we are all family. Your child is my child and my child is yours.” When an incident arose during the time the mother was away, Child Welfare services intervened and the child was taken into care. At the time of writing this report, the child remains in care and the matter is yet to be resolved, although the EARL-20B, indicates the worker’s perception that the impact on the child is to put the child at risk (caregiver continuity) where there was no previous related risk factor.

Focus Group

The focus group has been written up as a separate internal ICHS document as mentioned previously. A session of the regular Tuesday night parents group was set aside to hold a focus group with parents and the final year-end parent group evaluation included an evaluation component. Four parents attended the focus group. Highlights of their comments are included below as they related to:

Early onset of child difficulties “He was very aggressive at 18 months”; Difficulties across multiple settings “He doesn’t function well in society, he doesn’t function well at my front door, he doesn’t function well in my living room”; Systemic issues they faced in seeking help for their children “I have a 10 year old and he has been in 8 schools”; Complexity of child needs “Behavioural and learning difficulties”; Positive sense of the value of the program “The program is new and I have great hopes for it” Strong sense of alliance with the program and the workers “It feels like coming home. I can breathe” “They don’t judge you”;


Treatment being consistent and respectful and applied across multiple settings “Here it is always respect, calmness and consistency. That is very important, because sometimes when I am going to lose it, they are keeping it (calm).” “The way a worker goes in to the home, and the kid gets integrated into school, the worker keeps involved when the child is in day care” Seeing change in their own child and other’s children as well “I couldn’t believe it, but he really improved.” “The aggressive behaviour used to be about 23-28 times a day, very serious aggression. We are down to maybe 5 times a week now.”

Staff Survey

The ICHS Staff survey was a final addition to the set of measures collected, as it was felt that while staff had been busy implementing all the measures for the study, they also needed an opportunity to share their experiences of the transition, the expertise they had in dealing with these complex families and to present their own needs for going forward. The staff survey was designed to allow staff a confidential, anonymous means to contribute to this evaluation as well as to provide information to the managers about the needs of staff moving forward. The full staff survey report is being completed as an internal ICHS document, however the following highlights of the staff survey have been included in this report.

Staff Background and Training The ICHS staff was all experienced in the area of children’s mental health as seen in Figure 32 below. All staff had more than 5 years of experience in Child Mental Health, and a majority of staff had been with CDI for a longer period than the transition to ICHS as seen in Figure 33 below. Staff background and training included CYW/Child and Youth counselor, BA/BSc/BSW and MA/MSW or a combination of these. There were some with more varied backgrounds such as gerontology, HR, biology or sociology, but all were committed to both the agency and the program. Staff expressed a number of reasons for doing the work they were with children but their commitment was clear. Comments reflected belief in the program “I felt the program worked”; their own commitment to the work “passion for helping children with social, emotional and behavioural problems”; a desire to give back “to help clients struggling with similar issues to those I have struggled with in the past”; seeing change in families “I enjoy working with people and developing relationships and seeing families make positive changes” and a commitment to the CDI treatment approach “scientist-practitioner model”.


Figure 32: Staff Years of Experience in Child Mental Health

Figure 33: Staff Years of Service at CDI.


Strengths and Challenges of Residential and Community Based Services The staff was asked what they viewed as the three key strengths and three key challenges of each of the types of programs (Figures 34-37). Staff was asked to respond only if they had worked in the residential program previously. The qualitative responses were grouped according to themes and the following charts indicate the themes that arose, with the percentage of respondents whose comments reflected the theme. Figure 34: Themes: Strengths of Residential Programs

Figure 35: Challenges of Residential Programs


Figure 36: Strengths of the ICHS Program

Figure 37: Challenges of the ICHS Program

As can be seen from Figures 34-37 above, the ICHS staff is realistic in their appraisal of both the previous residential and the new ICHS program, recognizing that both types of services provide inherent challenges and strengths. The online survey was confidential enough that staff felt free to indicate a need for training, which had been addressed to a certain degree at the time of the survey and remains one of the priorities for the ICHS managers. The commitment of staff to the children and to the transition is reflected by the value they place on being able to work with the child in the home and community setting. When asked what types of families would benefit from the ICHS program, (Figure 38), the themes were focused on both the needs of the children and the families as well as the commitment required by parents for the program to be successful.


Figure 38: Families Who Would Benefit

When asked to rank key treatment components for the ICHS program, staff saw the intensive work at home with families as the most important component, followed by the increased number of hours of service that clients received (Table 9 below). Table 9: Key treatment components of the ICHS program.

Key Treatment Median ranking

Intensive work in the home with the family 1.5

Increased number of hours of service for clients each week 2.5

Intensive work in the community with the family 3

Flexibility in terms of time spent with family 3.5

Ability to provide extensive follow up for community integration

5

Ability to access other professional services if needed 6

Other questions that were asked in the staff survey included what changes staff saw in both parents and children in ICHS and these are laid out in Tables 10 and 11 below


Table 10: Key Changes in Parenting Key Change % of Respondents

who Checked Change

Parents better able to set limits and follow through 83

More praise directed towards child 83

Parents tell me they see changes in the child 83

Reduced aggressive/acting out behaviours 83

Parents learn more effective parenting strategies (e.g. time out, rewards, consistency

75

Improved parent-child relationships 75

Parents tell me that there is less conflict in the home 75

Parents feel home life is happier for family 67

Reduced use of coercive parenting strategies 50

Improved emotional connection with child 42

Parents no longer see child as problem 35

Parents tell me they like their child better 25

Clearly staff perceptions are that the parenting strategies and skills that are being taught in the ICHS program are having an impact in improving positive parenting, reducing negative and coercive interactions and improving the quality of life for the parents and children. Staff responses to the changes they see in children reflect similar positive improvements as shown in Table 11 below.


Table 11: Key Changes in Children Key Change % of Respondents

who Checked This Change

Reduced aggressive/acting out behaviours 83

Improved parent child relationships 83

Use of strategies (Stop, count to 10) 83

Fewer discipline problems at home 83

Better communication about emotions 83

Doing better in school with respect to behaviour (fewer suspensions, detentions etc.)

83

Improved peer relationships (school/community) 75

Improved self esteem 75

Child is happier 67

Reduced tantrums (frequency, length, severity)

Improved sibling relationships (fewer fights/arguments)

67

Doing better in school with respect to academics (better able to focus, complete work etc.)

67

Child has gained self-control 67

Child feels valued and supported 58

Staff was also asked what parents tell them about the program and the key theme was positive, including both parents’ appreciation of the program and the effectiveness of the program, Figure 39. This feedback was also consistent with the parents’ focus group.


Figure 39: Themes, parental feedback to staff.

Treatment Intensity

As the ICHS program provides individualized services, duration of treatment and dates of data collection were specific to each family and hence variable. The average length of ICHS treatment has been 9 months, however, for some children, the follow up/transitional support or the need to remain in the classroom while waiting for community school placement has extended this time to as long as 24 months.

Previous CDI services For children in ICHS, more than half (67%) had received some type of previous service through CDI. The range of services included SNAP® group for parent (SNAP-P) and SNAP® child Group (SNAP GRP), Girls Growing up Healthy (GGUH), Family Counselling, Individual Befriending (IB) and School Support as seen in Figure 40 below. The data on service utilization are based on all 27 children in the sample. Figure 40: Previous services received by ICHS children.


Intake Process There are several steps for admission to the ICHS program, which include a full assessment to ensure that treatment goals can be developed and that the program is a good match for the child and family’s needs. This process can take place over a number of months and Figure 41 below shows the mean number of sessions for each part of this process. Figure 41: ICHS Intake Process

Key treatment components include home community appointments, parent group and child group, family counseling and telephone support. In addition, school advocacy and attendance in the on-site classroom, are important aspects of the ICHS program. Figure 42: ICHS Key Treatment Components


Quality Assurance Several measures of quality assurance were piloted with the ICHS program. The agency adult quality assurance measure was used and a version developed that was adapted for children. In addition, a form was developed to track child goals as outcomes, based on child report, however, we were only able to obtain a couple of these, because the form became too cumbersome for the child to complete, so that it was discontinued and not included in the analysis. The quality assurance report is appended as a separate document (Appendix F).

Objective discussion of findings

Outcome Summary

The findings from this evaluation, while preliminary, are promising, pointing out strengths of the ICHS program, such as flexible responding to client emerging needs, as well as individualized family based care, that can provide a strong component of a continuum of care for children and families experiencing complex difficulties. Outcome measures, while based on small samples, indicate that aggressive acting out behaviours are generally improving through treatment. Parents and children are generally positive about the value of the program and of the treatment improving child behaviours and family relationships.

Follow up

The case studies in particular highlight that while stabilization of home and family life are key outcomes of the ICHS program, other traumas or needs may emerge when the presenting behavioural issues have been addressed. The flexibility and ongoing transitional support to families exiting the intensive part of the services have proven to be a strong, emerging aspect of the intensive home and community services program. At the same time, the need for case management by ICHS staff after completion of the intensive phase of services, is creating a waiting list for admission to the program. Providing adequate case management while freeing up staff to take on new clients is emerging as one of the key challenges in moving forward with the program. This also points to systems issues and service gaps. For example, children being ready to transition out of ICHS but not having other appropriate services available to meet their follow up needs. This is something the agency is currently trying to coordinate internally (collaboration between ICHS / SNAP® Girl’s Connection Program / the SNAP® Under 12 Outreach Project) as well as externally (children being transferred to Section 23 classrooms at other agencies, to ensure that ICHS Section 23 class remains for children who require intensive services and more than day treatment alone). The dilemma lies again in the idea of therapeutic alliance and relationship between worker and client. Clients still need some support, but may not want to go through the process of getting to know an entirely new worker and/or program just for “follow up”.

Comparison of ICHS and residential clients. The comparison of ICHS and residential clients was based on a small sample – the last 15 in residence and the first 15 coming into the ICHS program. While there were no statistically significant differences between the two groups, which was consistent with clinicians observations, the clinicians noted that the ICHS group appeared to be slightly younger, which has implications for earlier interventions made possible by the home based approach to services. Clinicians in the staff survey reported that families might be more open to treatment that did not involve removing the child from the home. Ultimately, the outcome of this evaluation is to help ICHS build a scientist-practitioner model of both intervention and knowledge translation, Figure 43 below, in which each component is balanced with the others and all are providing collaborative feedback to ensure the best possible services for children and families.


Figure 43: Balanced Components of Scientist-Practitioner Model

Challenges

One of the biggest challenges for this research was the difficulty of completing the proposed statistical analyses of outcome data, due to sample size limitations and the tight timeframes of the project. While the projected numbers were higher than the actual intakes, all clients who were in the ICHS program from inception were included (with the exceptions noted previously), leaving a sample size of 27 overall. However, for each group (discharged, ongoing, new admits), the available measures differed and in some instances, were incomplete or missing. The range of measures tested provided descriptive data that was augmented by the in depth understanding obtained through the case studies and staff survey. The project increased commitment to the scientist-practitioner model and built capacity to do evaluation and research with the ICHS management and team. The process itself, been part of staff and team development and training and formed a springboard for future evaluations and more complete integration between research and clinical staff. As mentioned previously, one of the challenges that the ICHS program is currently facing, is developing a planned discharge process, that will provide supported transition back to the community and home school for those children able to do so, or to additional treatment and services if these are required.

Emerging Needs

There are two key reasons for the difficulty in helping families making transitions out of the ICHS program once the intensive phase of service is completed and the family is stabilized. Emerging needs that may not have been identified at intake but add to the complexity of care required may become apparent during the course of treatment, or even as discharge is planned. As children and families reach a point of feeling more comfort with each other and have the ability to feel secure and


manage day to day living again, other issues frequently emerge. One of the case histories provides a good example of this, in that allegations of abuse during foster care, that had never been proven or dealt with, were brought forward by the child once home life and acting out behaviours were stabilized. As noted previously, that child is now continuing in trauma treatment with the ICHS worker, and hence the family is not discharged. Disclosure occurred at 4 months into treatment, and projected 6-month discharge changed because of this. Another child in ICHS saw someone shot in his backyard, which again adds a level of complexity to treatment that must address changing inappropriate behaviours and dealing with underlying or multi-level difficulties experienced by the child. Child trauma can be difficult to identify until the child has built enough trust to be able to share such experiences. Once such issues emerge, the ICHS staff must either locate alternative resources, or continue with treatment if additional resources are unavailable or difficult for the family to access.

Transition Support In residence, goals were very specific. These were to reduce aggression sufficiently to enable children to live safely in their home and neighbourhood. Residential staff worked with children. A family worker, who was not part of residential staff, conducted family work. Thus, when children left the residence, the family worker could continue to help with community integration and school support. The family worker position was replaced by a clinical training and supervision position for ICHS staff as staff was doing the follow up care with families. While this provides the children with continuity with one worker who is familiar with child and family needs and progress, it also requires that ICHS workers carry not only clients who are in the intensive phase of services, but also may be doing follow up with families who are in the process of transitioning out of services. So ICHS staff is unable to take on a new client, because they are still doing follow up with families. At any one time with 7 staff, the program may have 17 clients, with each ICHS worker carrying 2 intensive cases and 1 or 2 follow-up cases. This has created a backlog in admitting new children into the program and a wait list has developed. The program at any one time does serve twice as many children (16-17), as the residential program did, however, the cumulative numbers have not increased exponentially as may have been anticipated.

School Placements

The other difficulty with discharging children from the program is finding appropriate school placements within the public or catholic school system, for children who are in the on-site classroom. Children are being retained in the classroom, pending a safe and effective school placement for them. Even once a school placement is found, schools will not place a child until a natural transition in the school year (September or January), when an IPRC is required, so even children who have a placement and are ready to leave the classroom may end up staying in the classroom until the school is ready to receive them. This again, has created a waiting list for children who need to be admitted into ICHS and the on-site classroom. Sometimes the placement does not exist, so the work is around finding the best possible placement for a child, and this can be time consuming for staff and generate wait times for everyone involved.

Stakeholder Involvement and Knowledge Exchange

This study had a broad stakeholder community, as noted in the stakeholder chart. Staff, managers and parents of the families in the ICHS program have been involved in the piloting of the measures. The focus groups allowed parental input; the staff survey ensured that staff had an opportunity to confidentially provide their insights and concerns about this evolving program. The LAAC was intimately involved in the original development and discussions of the role of the ICHS program in the continuum of care required to service latency age children whose behaviours place them in the top few percentiles of the population. An initial meeting was held with the key LAAC stakeholder group to outline the evaluation strategy, and two subsequent meetings were planned. One meeting was held in early December to review the findings and discuss recommendations, and the second meeting is planned for 2012 to present the final report and to


continue the discussion about ICHS needs and the fit of this community based program in the continuum of services for latency age children. Staff of ICHS collaboratively selected measures and timelines for data collection, reviewed the results and will be discussing the recommendations and implementation plan. Parents of the children selected for the case history stories, provided insight and will review the presentations of the cases as prepared for the conference. The report will also be disseminated through the CDI website and a brief summary of the Quality Assurance results will be made available to ICHS clients. Aggregated results of the evaluation will be disseminated to stakeholders and community partners via presentations, bulletins and publications. The results of this study have been presented to the LAAC for discussion as noted above and at the CDI quarterly all staff meeting in December 2011. The case histories will be presented as part of a symposium on children at Risk at the 12th Annual Conference of the International Association of Forensic Mental Health Services (IAFMHS) in Miami, Florida, in April, 2012. Further knowledge exchange activities are planned, including conference presentations and the submission of the case histories for publication.


Conclusion and Recommendations Discussion and interpretation of findings

The ICHS evaluation allowed for a closer examination of the process of intervention in working with complex families as well as helped to identify treatment needs, guidelines for follow up and training of staff. Unfortunately, the time frame and number of measures collected only allowed for descriptors, rather than analysis.however the impact of selecting and piloting measures collaboratively with staff did provide support for the scientist-practitioner model and, for some of whom, this was a new way of looking at research involvement. Results did verify that the families admitted to the program were meeting criteria. One difference noted between the ICHS and residential admissions was the age of the children appears to be decreasing, which is consistent with the perception of some staff (in the staff survey) that the ICHS program is taking in younger children. This may reflect a trend towards younger intakes, as in home services can be more readily accessed by families with younger children, who may have concerns about placing a child into residential care. Further comparisons between Residential and ICHS clients would require more detailed and longer studies to determine whether differences exist between the two client groups. Although, there was limited ability to look across time or to complete statistical analyses, the evaluation was able to verify that many of the targeted treatment goals were being met and/or on track. This is particularly evidenced when reviewing several individual case results which indicated that families were responding to treatment. Overall, these families decreased in risk level, reported fewer behavioural issues with their children and showed increases in positive parenting strategies and family activities. All of these results are key indicators of progressive treatment response and targeted outcomes. Lessons Learned

The period of the evaluation was also a period of change within the ICHS program. Demands on staff were many, including participation in Cognitive-Behaviour Therapy for Parents training; preparation for the upcoming Children’s Mental Health Ontario accreditation process, adaptation to departmental changes such as the introduction of a new ICHS Coordinator and the cut of the ICHS Family Worker position, increased requirements during supervision; and learning more formalized clinical record keeping procedures. These demands and changes occurred in tandem with the selection and implementation of a barrage of measures to pilot. One of the key learnings from the project was that the assessment process needs to be kept simple and digestible in order to deal with these high-risk families. The evaluation was designed to measure process, risk, and treatment response in a comprehensive manner , however, the approach created difficulties for staff to implement. In addition, the realities of literacy, comprehension and stressful life circumstances made it difficult for parents to complete forms without support. While research could offer some support during the evaluation period, this is not feasible as an ongoing strategy. Given the nature of what these families require, the assessment has to be simplified. Follow up and debriefing around the process, will occur when we review and discuss the completed report with the staff and finalize the approach to treatment and assessment for the upcoming year.


Recommendations

Ongoing Evaluation of ICHS program

• Consistent limited data collection, based on meaningful output and clinical utility. • Measures selection to include outcome and process measures. • Use standardized measures where possible • Recommended measures: BCPFI, CBCL, PSI (short or long form or selected scales to be

determined), Alliance measures for parent and child, BCPFI family items. • Consistent data collection methods and tracking • Select, review and continue to evaluate with staff input • Single time frame for data collection based on client intake, and 6-month clinical evaluation

points, rather than some based on calendar year and others on client intake dates • The data collection for the ICHS program will be integrated into the ongoing work of the

research department. • Timely feedback to staff from research to enable staff to use data from measures in clinical

intervention planning • Continue to track whether number of girls in the program increases over time • Continue to track whether younger children are being admitted, and implications for earlier

intervention • Consider Wait List control group for future research, if wait list continues

Planned Transitions

One of the key recommendations arising from this evaluation study is to create a process for planned discharge and transitions. This is based on the recognition that discharge from the ICHS program requires further case management, which to date has been done by the ICHS workers, creating a wait list for admission until cases can be fully closed. There are several possible solutions currently under consideration. These include but are not limited to the following:

• Reinstating the family worker position for discharge case management (if funding permits); • Seeking to obtain a Whatever it Takes (WIT) worker for discharge and community transition

planning; • Continuing to work with the school systems to identify ways in which to expedite access to

appropriate school placements for children who are ready to leave the on site classroom; • Working with other Children’s Mental Health agencies within the City of Toronto to

determine how to support clients through long wait list periods for Section 23 classroom placements within CMHO;

• Developing guidelines for identifying the point at which discharge planning from the ICHS program should begin (e.g. identify when direct service hrs drop below a certain threshold such as 60/40 split then identify timing for next stage with respect to discharge and getting less intensive follow up)

Follow up

Currently, discharged ICHS clients are welcome to continue participation on the Tuesday evening dinner followed by Parent Group, with the child attending Activity Group and 2 families are doing this on a regular basis.

To provide planned opportunity for discharged clients to check in to ICHS, consider developing a plan that includes a bi-annual discharge client groups meeting, which may be linked to long-term


outcome measures (e.g. hold a pizza party for discharged clients, invite children back, say bring your folks, allowing an informal check in with both children and parents)

Staff Training

Clarify Training Requirements and Protocols • Continue to identify and address unique training needs for staff in ICHS (e.g. how to do

family work one on one in home setting) • Safety issues relating to home based interventions • Continue to facilitate staff training on individual by request basis • Continue with CBT consultation for staff regarding issues that arise with clients • Develop ongoing and new intake training requirements and annual schedules (e.g. build in

regular reviews of SNAP-P parenting components in team meetings, which occurred across 6 team meetings in spring/summer 2011), consider whether in future could be reviewed quarterly in team meetings

• Incorporate CCCO SNAP® competency pieces into ongoing training as these are developed. • Regular ongoing training will continue • Develop a formal staff orientation and training package for new hires • Facilitate EARL and SNAP® training as soon as it is offered for new hires.

Systems Issues Identify and work with LAAC around systems issues relating to clients in service at ICHS and within the broader CMH system

• Case management for clients being discharged who need additional services • How to access and/or provide services to clients for emerging needs (once intensive service

phase is completed (e.g. trauma work with client, or retaining child in on-site classroom, when no appropriate school placement is available)

Next Steps

Ongoing Evaluation of ICHS program

The evaluation of the program was productive in implementing the scientist-practitioner model as part of the core practice of the ICHS program. The next steps to continue with this model are to meet with the staff, discuss the outcomes and findings of the evaluation study and to review the recommended measures and timelines for data collection and to implement a system for ensuring that complete data for each client is collected. A standard set of measures and data collection timelines will be developed and implemented as quickly as possible upon completion of this evaluation, to ensure that consistent data collection continues.

Planned Transitions

The study also brought out the need to clarify and structure discharge planning for clients completing the intensive ICHS program. A set of guidelines for identifying the point in treatment at which discharge planning should begin, is currently being developed. In addition, plans for the case management piece of transitioning clients to home/community are under discussion within the agency.


Treatment Components

Focus groups identified the need for consistency in training in parenting skills across program delivery. As the groups are ongoing over the course of the school year, a new schedule of annual parent program content is being developed and will be piloted in the coming year (Sept. 2012-May 2013). This program is currently in development.

Knowledge Translation

More knowledge translation activities are planned for 2012 (see above for activities during the evaluation period). Key among these is presenting the completed evaluation report to the LAAC and to continue to discuss the report and implementation with ICHS staff. The case histories are being presented at an upcoming conference as indicated earlier in this report (see Case Histories above) and will be written up for publication, possibly including the single case design. Prior to the conference presentation, the case histories will be reviewed with the parents of the children. As well, we will be uploading the report to CDI’s website.

Staff Training

Training of new staff will include the scientist practitioner model and implementation within the agency and the ICHS program. Ongoing training will continue, and new training needs will be identified and addressed. Ways to fund the development of a formal new-hire training package are under consideration. Program managers will utilize the staff survey results to ensure that staff opinions and needs are addressed in the continued development of the program. Systems Issues Structured discharge planning will be developed in consultation with staff and the LAAC to ensure that new admissions can come into the program while children who have made the requisite gains can be successfully supported in returning to their community and local schools where feasible, or in accessing additional supports or services within the latency age children’s mental health system in Toronto when further services are required. The fit of the ICHS into the overall latency age CMH system will continue to be discussed with all agencies and parties involved to ensure the best care possible for children and families in need of treatment.


References Abidin, R. R. (1995). Parenting Stress Index, Third Edition: Professional Manual. Odessa, FL: Psychological Assessment Resources, Inc. Achenbach, T. M. (1991) Integrative Guide to the 1991 CBCL/4-18, YSR, and TRF Profiles. Burlington, VT: University of Vermont, Department of Psychology. Angold A, Costello EJ, Messer SC, Pickles A, Winder F, Silver D. (1995). Development of a short questionnaire for use in epidemiological studies of depression in children and adolescents. International Journal of Methods in Psychiatric Research, 5:237-249. Augimeri, L.K. Koegl, C.J., Webster, C.D., & Levene, K.S., (2001). Early Assessment Risk List for Boys: EARL-20B, Version 2. Toronto: Earlscourt Child and Family Centre. Augimeri, L.K., Webster, C.D., Koegl, C.J., & Levene, K.S. (1998). Early Assessment Risk List for Boys: EARL-20B, Version 1, Consultation Edition, Toronto, Earlscourt Child and Family Centre. Atkins, D.C., Bedics, J.D., McGlinchey, J.B. & Beauchaine, T.P. (2005). Assessing clinical significance: Does it matter which method we use? Journal of Consulting and Clinical Psychology, 73(5), 982-989. Beck, A.T., Steer, R.A., & Brown, G.K. (1996). Manual for Beck Depression Inventory-II. San Antonia, TX: Psychological Corporation. Beutler, L.E. & Moleiro, C. (2001). Clinical versus reliable and significant change. Clinical Psychology: Science and Practice, 8, 441-445. Bickman, L., Lambert, E. W., Andrade, A. R., & Penaloza, R. V. (2000). The Fort Bragg continuum of care for children and adolescents: Mental health outcomes over 5 years. Journal of Consulting and Clinical Psychology, Vol 68(4), Aug 2000,: 710 -716. Bernstein, J. (1987) Residential Treatment and Aftercare: Protecting Our Investment. Residential Treatment for Children & Youth. Vol. 4(3) 1987. Burns, B. J., Hoagwood, K., Mrazek, P. J., (1999) Effective Treatment for Mental Disorders in Children and Adolescents. Clinical Child and Family Psychology Review, 12-01, 2 (4): 199 – 254. Burns, B.J., Hoagwood, K., & Mrazek, P. (1999). Effective treatment for mental disorders in children and adolescents. Clinical Child and Family Psychology Review, 2: 199-254. Canadian Paediatric Society. (2001). Retrieved from http://wwww.cps.ca/english/advocacy/mentalhealth.htm Accessed March 25, 2011. Connor, D.F. (2002) Aggression and Antisocial Behaviour in Children and Adolescents: research and treatment. Guilford Press, New York.


Cóté, S. M., Vaillancourt, T., LeBlanc J., Nagin, D.S., Tremblay, R. E. (2006). The Development of Physical Aggression from Toddlerhood to Pre-Adolescence: A Nation Wide Longitudinal Study of Canadian Children. Journal of Abnormal Child Psychology, Vol 34, No 1: 71-85. Cunningham, C.E., Boyle, M. H., Hong, S., Pettingill, P., & Bohaychuk, D. (2009). The Brief Child and Family Phone Interview (BCFPI): 1. Rationale, development, and description of a computerized children's mental health intake and outcome assessment tool. Journal of Child Psychology and Psychiatry and Allied Disciplines, 50(4), 416-423. Cunningham, C.E., Pettingill, P., and Boyle, M.H., (2004). The Brief Child and Family Phone Interview (BCFPI-3): a computerized intake and assessment tool. Interviewer’s manual. Cutrona, C.E., & Russell, D. (1987). The provisions of social relationships and adaptation to stress. In W.H. Jones & D. Perlman (Eds.), Advances in personal relationships (Vol. 1, pp. 37-67). Greenwich, Conn.: JAI Press. Dishion T. J., McCord J., & Poulin F. (1999). When Interventions Harm: Peer Groups and Problem Behavior. American Psychologist, 54: 755 -764. Duncan, B. L., Miller, S. D. Sparks, J. (2007). Common Factors and the Uncommon Heroism of Youth. Psychotherapy in Australia, Vol. 13, No. 2, February, 2007: 34-43 Duncan, B. L., Miller, S. D. Sparks, J. (2003). The Outcome Rating Scale: A Preliminary Study of the Reliability, Validity, and Feasibility of a Brief Visual Analog Measure. Journal of Brief Therapy. Volume2, Number2 – Spring/Summer 2003,: 91 - 100 Epstein, N.B., Baldwin, L.M., & Bishop, D.S. (1983). The McMaster family assessment device. Journal of Marital and Family Therapy, 9: 171-180. Source URL: http://chipts.cch.ucla.edu/assessment/IB/List_Scales/McMaster_Family-Assessment.htm accessed Sept 20, 2010 Hair, H. (2005). Outcomes for children and adolescents after residential treatment: A review of research from 1993-2003. Journal of child and family studies, 14(4) : 551-575. Retrieved from http://www.springerlink.com/content/u6273h94t830216j. Accessed Nov. 4, 2011. Hart, S.D. (1999). Assessing violence risk: Thoughts and second thoughts. [Review of V.L Quinsey, G.T. Harris, M.E. Rice, & C.A. Cormier. Violent offenders: Appraising and managing risk]. Contemporary Psychology, 44 : 486-488. Haskett, M.E., Ahern, L.S., Ward, C.S., and Allaire, J.C., (2006). Factor structure and Validity of the Parenting Stress Index-Short Form. Journal of Clinical Child and Adolescent Psychology, 35(2): 302-312. Horvath, A.O., & Greenberg, L.S.(1989). Development and validation of the Working Alliance Inventory. Journal of Counseling Psychology, 36: 223-233. Kazdin, A. E. (1996). Conduct Disorder in Childhood and Adolescents (2nd Ed.) Thousand Oaks, CA. Sage


Kazdin , A. E. (1997) Practitioners Review: Psychosocial Treatments for Conduct Disorder in Children. Journal of Child Psychology and Psychiatry, 38: 161-178. Leichtman, M. and Leichtman, M.L. (2001) Facilitating the Transition from Residential Treatment into the Community: I. The Problem. Residential Treatment for Children & Youth. Vol. 19(1). Leichtman, M., Leichtman, M. L., Barber, C. C., & Neese, D. T. (2001). Effectiveness of Intensive Short-term residential treatment with severely disturbed adolescents. American Journal of Orthopsychiatry, 71 (2): 227-235. Lempers, J. D., Clark-Lempers, D., Simons, R. L. (1989). Economic Hardship, Parenting, and Distress in Adolescence. Child Development 1989, 60,: 25 – 39. Society for Research in Child Development, Inc. Levene, K.S., Augimeri, L.K., Pepler, D., Walsh, M., Webster, C.D., & Koegl, C.J. (2001), Early Assessment Risk Lists for Girls: EARL-21G, Version 1, Consultation edition, Toronto, Earlscourt Child and Family Centre. Loeber, R., Stouthamer-Loeber, M., Van Kammen, W. B., & Farrington, D. P. (1989). Development of a new measure of self-reported antisocial behavior for young children: Prevalence and reliability. In M. Klein (Ed.), /Cross-National Research in Self-Reported Crime and Delinquency.: 203-225. Boston, MA: Kluwer-Nijhoff. Loeber, R., Farrington, D. P. (1998). Serious and Violent Juvenile Offenders: Risk Factors and Successful Interventions. Thousand Oaks, CA. Sage Publications Inc. Loeber, R., Farrington, D. P. (2001). Child Delinquents: Development, Intervention, and Service Needs Thousand Oaks, CA. Sage Publications Inc. Magellan Health Services Children’s Services Task Force (2008). Perspectives on residential and community-based treatment for youth and families. Retrieved from http://www.magellanhealth.com/media/2718/CommunityResidentialTreatment_White_Paper.pdf Accessed Oct. 15, 2011 McEwan, K., Waddell, C., Barker, J. (2007). Commentary “Bringing Children’s Mental Health out of the shadows”. Canadian Medical Association Journal (CMAJ) Vol 176, No 4. February 13, 2007.: 471-472 Source URL: http://www.cmaj.ca/cgi/reprint/176/4/471Accessed April 7, 2011 McLendon, T. (2008, January). Report 22 best practices to engage parents of children receiving mental health services. Retrieved from http://www.handle.net/1808/3866. Accessed Sept.15, 2011 Messer, S. C., Angold, A., Costello, J. (1995). Development of a Short Questionnaire for use in Epidemiological Studies of Depression in Children and Adolescents: Factor Composition and Structure Across Development. International Journal of Methods in Psychiatric Research, Vol. 5: 251 - 262


Miller, S. D., Duncan, B. L. (2000). The Outcome Rating Scale: A Preliminary Study of the Reliability, Validity, and Feasibility of a Brief Visual Analog Measure. Institute for the Study of Therapeutic Change. Miller, I.W., Epstein, N.B., Bishop, D.S., & Keitner, G.I. (1985). The McMaster family assessment device: Reliability and validity. Journal of Marital and Family Therapy, 11: 345-356. Source URL: http://chipts.cch.ucla.edu/assessment/IB/List_Scales/McMaster_Family-Assessment.htm accessed Sept 20, 2010. National Council of Welfare. (2011). Welfare Incomes 2009-update as of March 2011 Source URL: http://www.ncw.gc.ca/[email protected]?lid=386&fid=1 Accessed Jan. 10, 2011. Offord, D.R., Boyle, M.H., & Racine, Y.A. (1990). Ontario child Health Study: Children at Risk. Toronto, Ministry of Community and Social Services. Preyde, M.C., Cameron, G., Frensch, K. and Adams, G. (2011). 'Parent-Child Relationships and Family Functioning of Children and Youth Discharged from Residential Mental Health Treatment or a Home-Based Alternative'. Residential Treatment For Children & Youth, 28: 1: 55 –74. Source URL: http://dx.doi.org/10.1080/0886571X.2011.550171 Accessed April 7, 2011. Raos, R., Janus, M. (2010). Examining spatial variations in the prevalence of mental health problems among 5 year old children in Canada. Social Science and Medicine, 72,: 383-388. Shirk, R. S., Russell, R. L., (1996). Change processes in child psychotherapy: Revitalizing treatment and research. New York: Guilford Press Smith, B., Duffee, D., Steinke, C., Huang, Y., & Larkin, H. (2008, April 23). Outcomes in residential treatment fro youth: The role of early engagement. Retrieved from http://journals1.scholarspotal.info.uproxy.library.docuoit.ca/tmp/15285387372796903751.pdf Accessed Oct. 7, 2011. Steer, R. A., & Clark, D. A., (1997). Psychometric characteristics of the Beck Depression Inventory – II with college students. Measurement and Evaluation in Counseling and Development, 30,: 128-136 Strayhorn, J.M., Weidman, C.S.(1988). A parent practices scale, and its relation to parent and child mental health. Journal of the American Academy of Child and Adolescent Psychiatry, 27,: 613-618. Statistics Canada, (2010) (a). National Longitudinal Survey of Children and Youth, NLSYC, Cycle 8, Survey Instruments, 2008/09, Book 1: Parent, Child and Youth. Source URL: http://www.statcan.gc.ca/imdb-bmdi/instrument/4450_Q2_V7-eng.pdf Accessed (October 10, 2010). Statistics Canada, (2010) (b). National Longitudinal Survey of Children and Youth, NLSYC, Cycle 8, Survey Instruments, 2008/09, Book 2: Youth Questionnaires. Source URL: http://www.statcan.gc.ca/imdb-bmdi/instrument/4450_Q3_V7-eng.pdf (October 10, 2010). U.S. Department of Health and Human Services. (1999) Mental Health: A Report of the Surgeon


General—Chapter3, Children and Mental Health. Rockville, MD: U.S. Department of Health and Human Services; Substance Abuse and Mental Health Services Administration, Center for Mental Health Services, National Institutes of Health, National Institute of Mental Health, 1999. Uta, W. (2007). Report 20: residential treatment best practices in children’s mental health. Retrieved from http://hdl.handle.net/1808/3879. Accessed Sept. 15, 2011.


Appendices


Appendix A: Program Logic Model


Appendix B: Evaluation Framework


Appendix C: Stakeholders


Appendix D: CDI consent


Appendix E: Staff Survey

ICHS Staff Survey, 2011ICHS Staff Survey, 2011ICHS Staff Survey, 2011ICHS Staff Survey, 2011

Thanks for taking the time to fill out this survey. As you know, your participation in the design and development of the current evaluation is important. But even more important are your feedback and thoughts on the evolution of this program and the strengths and challenges of implementing the intensive home and community services program. The survey should take you between 3045 minutes, but your insights and understandings based on your clinical experience will provdie critical direction to the ongoing evaluation of this program. We have set this survey up online to make it flexible for you to complete. You can log on and do one section at a time. Your answers will remain confidential, as there are no identifiers attached to any person's answer, so please be honest and open about your experiences.

1. Please tell us something about yourself. Why did you choose to do this work?

2. What is your background and training? Please check all that apply.

3. How many years have you worked in the field of child mental health?

Section 1: Introduction and Background

CYW (college level)

SSW (college level)

BSW

BA (Psych)

BA (Sociology)

BA (other)

MSW

MA (psych)

MA (Sociology)

MA (other)

Other (please specify)

less than 1 year

12 years

35 years

610 years

more than 10 years

ICHS Staff Survey, 2011ICHS Staff Survey, 2011ICHS Staff Survey, 2011ICHS Staff Survey, 20114. How many years have you worked at CDI?

less than 1 year

12 years

35 years

610 years

more than 10 years


Please answer these questions if you worked on the residence before ICHS. If you did not work with children/families in the residential program, please move on to the next page.

5. What do you see as 3 strengths of providing services for children and families in a residential program (e.g. shift work, having children under care 24/7, ease of access to deliver services etc.)?

6. What were the challenges of working in a residential program (e.g. shift work, felt more like a caregiver and less like a therapist etc.)?

7. Thinking about the clients you saw in the residential program, what do you see as the differences in the needs of the families or children in residential services as compared to ICHS?

8. What do you see as the key changes in the transition from residential to ICHS services (e.g. family work in the home)?

Section 2: Residential Services


Please share your perceptions of the ICHS program. Please answer the questions below based on your experience and understanding in working with clients over the 2 years of the ICHS program.

9. What are three things you like, or see as the strengths/positives of working with children/families in the ICHS program?

10. What are three challenges of working in the ICHS program?

11. What do parents tell you about their participation in the ICHS program?

12. In your opinion, what types of families/children benefit from the ICHS program?

13. What do you see as the key strengths of the ICHS program? Please rank them in order of importance, based on your experience.

Section 3: ICHS Services

1 2 3 4 5 6

Flexibility in terms of time spent with family

Increased number of hours of service for clients each week

Ability to provide extensive follow up for community integration

Intensive work in the home with the family

Intensive work in the community with the family

Ability to access other professional services if needed


ICHS Staff Survey, 2011ICHS Staff Survey, 2011ICHS Staff Survey, 2011ICHS Staff Survey, 201114. What are some of the key changes you see in parents? Please check all that apply.

15. What are some of the key changes you see in the children in the ICHS program? Please check all that apply.

Parents learn more effective parenting strategies (e.g. time out, rewards, consistency

Parents no longer see child as problem

Parents tell me that there is less conflict in the home

Parents better able to set limits and follow through

Parents feel home life is happier for family

Parents tell me they like their child better

Parents tell me they see changes in the child

Improved parentchild relationships

Reduced use of coercive parenting strategies

More praise directed towards child

Reduced aggressive/acting out behaviours

Improved emotional connection with child


Reduced aggressive/acting out behaviours

Child is happier

Child feels valued and supported

Fewer discipline problems at home

Improved self esteem

Improved sibling relationships (fewer fights/arguments)

Improved peer relationships (school/community)

Better communication about emotions

Doing better in school with respect to behaviour (fewer suspensions, detentions etc.)

Doing better in school with respect to academics (better able to focus, complete work etc.)

Improved parent child relationships

Child has gained selfcontrol strategies (Stop, count to 10)

Reduced tantrums (frequency, length, severity)



This section is about the various treatment components of the ICHS program. First, we will ask you about the overall components for parents and children, then about specific aspects of treatment. Please answer based on your experience and understanding, even if you were not specifically responsible for some of these components.

16. What do you see as the key important treatment components for children in the ICHS program? Please rank the top 6 components

17. What do you see as the key important treatment components for families in the ICHS program? Please rank the top 6 components.

Section 4: ICHS Treatment Components

1 2 3 4 5 6

ICHS Children’s Group

School Support/ Coaching

PsychoEducational Assessment

TAPP – C

Day Treatment (St. Clair)

School Advocacy

Community Support

Individual Counselling

Psychiatric Assessment

Community HookUp

Leader in Training

1 2 3 4 5 6

ICHS SNAP® Parent Group

Family Counselling

Home based coaching

Parent Support/Counselling

Community Advocacy

Home/ Community Visits

School Advocacy

Stopping Stealing Program

Parent Mentor



ICHS Staff Survey, 2011ICHS Staff Survey, 2011ICHS Staff Survey, 2011ICHS Staff Survey, 201118. Parent's Group: Please rate the strengths of the parent group for families, based on the importance of each to the success of the parents in the program, where 5 is very important and 1 is not important.

19. Children"s Group: Please rate the following components of the children's groups based on the importance of each to the success of the child, where 5 is very important and 1 is not important.

5 (very important) 4 3 2 1 (not important)

Sharing with other families

Learning new skills and strategies for improved parenting.

Understanding key SNAP® parenting strategies

Knowing they are not alone

Learning skills from other families

1 (not important) 2 3 4 5 (very important)

Learning SNAP strategies (self control, count to 10)

Anger management

Social skills

Dealing with bullying

Being in a group with other children

Practicing/role play skills



ICHS Staff Survey, 2011ICHS Staff Survey, 2011ICHS Staff Survey, 2011ICHS Staff Survey, 201120. In Home Visits: Please rate the following components of the in home visits based on their importance to the success of child/family in the ICHS program, where 5 is very important and 1 is not important.

5 (very important) 4 3 2 1 (not important)

Reducing aversive interactions between parent and child

Increasing positive interactions between parent and child

Improving communication

Building parentchild relationships

Helping with sibling interaction

Improving behaviour management in the home

Applying SNAP for Parents in a real life setting

Setting routines in place

Coaching parents in real life settings

Modelling behaviour management

Coaching parents regarding community participation/outings


This section is about what is helpful to you as staff in the ICHS program. Please answer based on your current experience.

21. Please rank the top 5 elements of staff support and supervision in terms of helpfulness to you in carrying out your work in ICHS.

22. What would help you as a worker in your continued work with the families in the ICHS program?

23. Is there anything else we should be aware of, that would help the agency/supervisors support you in providing services through the ICHS program?

Section 5 ICHS Staff Needs

1 2 3 4 5 N/A

Ongoing supervision

Team meetings

Case conferences

Sharing skills, strategies with other staff

Outside professional consultation if required

Training opportunities (internal)

Training opportunities (external)

Participating in research

Hearing about research outcomes for ICHS families

In terms of ICHS program components or structure?

In terms of community work with families?

In terms of work at home with families?

In terms of follow up with families?

In terms of supervision?

Other?



In this section, we would like you to reflect on both the changes from Residence to ICHS and your thoughts about improving the program and services in the future.

24. Thinking back on the first two years of the ICHS program, and transitioning from residential services, what do you think made the transition successful?

25. What do you think were some of the challenges of transitioning to the ICHS program?

26. What successes have you seen that you can attribute to the ICHS program? Please give 3 specific examples.

27. What challenges have you faced with clients in the ICHS program? Please give 3 specific examples. How were these resolved?

28. What would be your three key recommendations for changes over the next year or two to improve the program?

Section 6 ICHS Looking to the future of the program


Thanks for all your work on reviewing. selecting and collecting measures for this evaluation. At the end of this evaluation, we will be looking for your input in terms of the value of the measures utilized and the time frames for data collection. Some of the measures will be a given because those are part of mandated measures or agency ongoing data collection. Agency measures that will remain in place are CBCL and the EARLS. Some of the others will be up for discussion with you nearer the end of the evaluation. We will likely debrief these in a staff meeting in the late summer/early fall. At this point, we would welcome your thoughts about how best to schedule ongoing evaluation measures into the program once the evaluation study is completed. This will enable us to have longterm data on the outcomes of the children/families in the ICHS program. Timing of data collection. At the moment, we are thinking that we will recommend the following, which we have tested with this evaluation. Tying outcome measures to clinical time points and process measures to the calendar year.

29. For outcome measures (such as CBCL and EARL) We will recommend tying the collection of these measures to the clinical time points Intake, 6 month review, 12 month review and discharge Would tying this data collection to clinical points enable you to easily track the data collection as part of your ongoing work with clients? Data collection for each client would be individualized, but in synch with your existing clinical work.

30. If no, please provide your suggestion for alternative data collection points

31. As you know, with your input, we tested Alliance Measures at 2 month intervals, but changed to 3 month points. The last one during the evaluation will be in August. These we would like to just connect to 4 collection points based on the calendar year, in which case everyone in ICHS will do them at the same time regardless of when families enter the program. Would this seem like reasonable way to track collection of these measures?

Section 7 ICHS and Ongoing Program Evaluation

yes

no

yes

no

ICHS Staff Survey, 2011ICHS Staff Survey, 2011ICHS Staff Survey, 2011ICHS Staff Survey, 201132. If no, please provide your suggestion for alternative data collection points for process measures.


Thank you for taking the time to complete the questions in this survey. As staff, you remain the experts in both understanding the families and providing treatment that is effective. Your comments, suggestions and insights will be added to the ICHS Evaluation Study and will provide direction as we plan for the next several years of this innovative service.

33. Please let us know if there anything else we haven't asked you about, or that you think we should be aware of, as we begin to write up the evaluation of the ICHS program?

Section 8: Thank you

Confidential Page 82 13/01/12

Appendix F: Quality Assurance Report

ICHS Quality Assurance Report Appendix to the ICHS Evaluation Report

Margaret Walsh, Manager Research and Evaluation Reet Mae, Coordinator, ICHS Evaluation 2010/2011

ICHS Quality Assurance 2011

2

As part of agency standard accreditation procedures, Child Development Institute (CDI)

Research analysis and summarizes the client satisfaction questionnaires (CSQ) administered to clients discharged from their programs. CDI recognizes the need to maintain high quality of standards of service delivery in all aspects of the agency’s operation. Our goal is that all programs will participate in some form of quality assurance activities. CDI ensures the confidentiality of all information collected for the purpose of quality assurance. Information is summarized in an aggregate report and does not include any identifying information regarding individual respondents. Managers are expected to review the Quality Assurance summaries that apply to the programs they supervise with their teams. As well, QA sections of the report are posted in prominent locations throughout the agency and may be shared with extended service groups, as ways of making the findings accessible to clients.

The ICHS Evaluation 2010/2011 included Quality Assurance reports from parents, based on

the agency overall quality assurance form, as well as a newly developed Quality Assurance form for children in the ICHS program, that was piloted during the 2010/2011 phase.

The first section of this report provides the results of the Parent Quality Assurance and the

second reflects the Child Quality Assurance form.


3

Parents Quality Assurance Questionnaire Responses

How would you rate the quality of service you received from Child Development Institute?

Figure 1. Quality of Service Received

All clients were positive about the service received with over half (57%) rating the quality of service as excellent. The remaining 43 % rated it as good.

When asked what types of programs they had participated in the parents responded by

indicating residential, ICHS , parenting group, individual counseling, SNAP child / parent group, ICHS parent group and parenting program. These were all parents with children in the ICHS program indicating that they were aware the program had multiple components, even if they were not entirely sure of the correct names for each of these.

57%

43%

excellent good


4

Did you get the kind of service you wanted?

Figure 2. Whether Services Received Were What they Wanted

One third of clients (29%) responded “Definitely” to the question whether they got the kind

of services they wanted, while over half (71%) identified “Yes, Generally. Overall, “Yes, Generally” was the median response category to whether clients got the kind of services they wanted.

71%

29%

yes, generally definitely


5

If a friend was in need of similar help, would you recommend CDI to him or her?

Figure 3. Likelihood to Recommend CDI to a Friend

Over half of clients (86%) “Definitely, would” recommend CDI to their friends if their friends

were in need of similar help. This is followed by 14% responding “Yes, I think so”. Overall, “Definitely, I would” was the median response category on whether the clients would recommend CDI to friends that needed similar help.

14%

86%

yes, I think so definitely, I would


6

How satisfied are you with the amount of help you have received?

Figure 4. Satisfaction Towards Amount of Services Received

Over half of the clients (71%) were “Very satisfied” with the amount of help they received at

CDI. This is followed by 29% stating that they were “Mostly satisfied. Overall, “Very satisfied” was the median response category on how satisfied the clients were on the amount of help they received at CDI.

29%

71%

mostly sadsfied very sadsfied


7

Have the services you received helped you to deal effectively with your child’s problems?

Figure 5. Whether Services Helped Them Deal Effectively with their Children

Nearly half of all clients (43%) responded “Yes, a great deal” to the question whether they felt the services they received helped them deal effectively with their child’s problems. Another 43% responded “Yes, somewhat”, and 14% indicated “No, not really”. No client reported that the services they received made things worse.

43%

43%

14%

yes, a great deal yes, somewhat no, not really


8

To what extent has CDI met the needs of your family? Figure 6. Whether Services Helped Meet the Families’ Needs

Nearly half of all clients (43%) responded “Almost all met” to the question to what extent

has CDI met the needs of their families. Another 43% responded that CDI services met “most” of their family needs. While 14% of families identified that CDI services met “a few” of their family needs

43%

43%

14%

almost all met most a few met


9

If you were to seek help again for your family, would you return to CDI?

Figure 7. Likelihood to Seek Services with CDI Again If Needed

Over half of all clients (57%) “definitely, would” return to CDI if they sought help again for

their family. Nearly one third of all clients (29%) “think” they would return to CDI if they sought help again for their families. Fourteen percent of clients responded that they “Definitely, would not” seek help again for their family at CDI. Overall, “Definitely I would” was the median response category on whether clients would return to CDI again if they seek help for their family.

57% 29%

14%

definitely, I would yes, I think so Definitely, I would not


10

Children’s Quality Assurance Questionnaire Responses

Did coming here help with the problems you were having?

Figure 8. Whether Services Helped with the Problems Clients Were Having

Over half of clients (60%) responded “Quite a lot” when asked if the services at CDI helped

with their problems. This is followed by 20% stating that services were “Not much” of a help, and another 20% responding “Not at all”. Overall, “Quite a lot” was the median response category for whether services helped with client’s problems

60% 20%

20%

quite a lot not much not at all


11

If you had a friend who had problems like you were having, would you tell them to come here?

Figure 9. Whether Clients Would Refer a Friend with Similar Problems

.

Forty percent of clients stated “Yes, for sure” when asked if they would refer a friend with similar problems to CDI. This is followed by 20% responding “Maybe”, another 20% stating “Maybe not”, and another 20% stating “No, would not”. Overall, “Yes, for sure” was the median response category for whether clients would refer a friend with similar problems to CDI.

40%

20%

20%

20%

yes, for sure maybe maybe not no, would not


12

Has coming here helped you make your problems smaller?

Figure 10. Whether Services Helped to Reduce Client Problems

Forty percent of clients responded “Quite a lot” when asked if the services received at CDI helped to reduce their problems. This was followed by another 40% responding “Not much”, and 20% stating “Sort of”. No client responded “Not at all”.

40%

20%

40%

quite a lot sort of not much


13

Has coming here helped make things better in your family?

Figure 11. Whether Services Helped Better the Family

Over half of clients (60%) responded with “Quite a lot better” when asked if coming to CDI has made things better for their family. This is followed by 40% stating “Not much better”. No client responded with “Not at all”. Overall, “Quite a lot better” was the median response category for whether coming to CDI has made things better for clients’ families.

60%

40%

quite a lot beeer not much beeer


14

Are you happy about the help you got when you were here?

Figure 12. Whether Clients are Happy about the Quality of Help Received

Over half of the clients (80%) responded “Pretty much” when asked if they were happy

about the quality of help they received. This is followed by 20% stating “Not at all”. Overall, “Pretty much” was the median response category for whether clients were happy about the help they received at CDI.

80%

20%

preey much not at all


15

If you needed to go somewhere for help again, would you come here again? Figure 13. Likelihood to Seek Services again with CDI if Needed

Over half of the clients (60%) responded “Yes, for sure” when asked if they would return

to CDI if they needed help again. This was followed with 20% stating “Maybe not” and another 20% stating that they “Would not”. Overall, “Yes, for sure” was the median response category for whether clients would return to CDI should they need help again.

60% 20%

20%

yes, for sure maybe not would not


16

Did you like your worker?

Figure 13. Whether Clients Liked their Worker

Over half of clients (80%) responded “Yes, for sure” when asked if they liked their worker. This was followed with 20% stating “Maybe”. No client responded with “Maybe not” or “No, did not”. Overall, “Yes, for sure” was the median response category for whether clients liked their worker.

80%

20%

yes, for sure maybe


17

Do you think that your worker liked you? Figure 14. Whether Clients Thought their Worker Liked Them

Over half of clients (80%) responded “Yes, for sure” when asked if they thought their

worker liked them. This was followed with 20% responding “Maybe”. No client responded with “Maybe not” or “No, did not”. Overall, “Yes, for sure” was the median response category for whether clients thought their worker liked them.

80%

20%

yes, for sure maybe


18

RESPONSE SUMMARIES

The following section summarized the responses by clients about aspects of the program treatments/services they liked and disliked and suggestions on improving CDI services. The response summaries are broken down by question. First comments are what they liked most about the program.

• Parents comments re program – 7 parents total

• Staff were helpful without being judgmental

• home services rather than residential

• The in home counselling program, and the 4 week/2 night couple counselling

• I liked the CYW working with my child one on one. I also appreciated the CYW coming into our home, it was very helpful

• Workers Che + Ron

• I truly enjoyed the comradery with the other parents. I felt I finally found others who understood.

• The SNAP program helped learn different ways to solve issues and ways to deal with your child. Worker being very involved with the family

Suggestions for improvement

Making facilities wheel chair accessible was a suggestion put forward by one respondent.

The location was not convenient for another. It was suggested by one parent that there were tensions around the transition from residential to ICHS services.

Improved communication between the CDI workers and classroom teachers is preferred so that parents have knowledge about the progress of their children in the program.


19

Children’s Comments about what they liked and what they would tell others

Children’s comments (5 children)

Being able to leave I liked the staff and I liked being able to go home each night. Wonderland Tell them to follow their goals and get back on track or else they will go some where else

even worse like a FARM!!! ya, thanks CDI...

Summary of Quality Assurance (QA) Results

The overwhelming response by CDI clients in the ICHS program was positive, with a majority

of clients indicating they were satisfied with the quality and content of programs and services offered. Clients felt that they received the type of services they needed and that the services they were in, helped them address problems they and their children were experiencing.

When clients were asked to rate the quality of services they received from ICHS, 100% rated the services as good or excellent. When asked about service match, 100% of clients responded positively. All clients (100%) would recommend CDI to their friends if their friends were in need of similar help. Overall, ICHS clients find the services beneficial and a match for their perceived needs.

The majority of clients (86%) felt services helped them deal effectively with the problems their children were experiencing with the remaining 14% saying no not, really, but none reported that the services they received made things worse. For the children, a majority (80%) indicated that the services did help them with the problems they were having.

Overall, the clients, both children and parents expressed positive outcomes from their involvement with the ICHS program.

Intensive Community and Home Services (ICHS) A Preliminary · PDF file ·...

Documents

Transcript of Intensive Community and Home Services (ICHS) A Preliminary · PDF file ·...