Evaluation of the New York State Collocation Program:Findings from the Implementation Study

Eunju Lee, Rose Greene, and Bud LePage Center for Human Services Research

School of Social Welfare, University at Albany and

New York State Office of Children and Family Services

2

Presentation Overview

Description of NYS Collocation ProgramEvaluation MethodologyFindings from the Process StudyNext Step

New York State Collocation Program

4

Description: Historical Background

RFP in Fall 2000 using TANF prevention funds Support of new community-based services for

vulnerable children and families Focus on prevention Collocation programs began in July 2001

5

Description: Program Design

Place AOD specialist (CASAC) in local child welfare offices

Assist child welfare staff with cases involving parental alcohol and/or substance abuse

Early identification, joint CW/AOD assessments, joint family service planning, timely referrals, and case management services

6

Description: Program Goals

Decreased foster care placements Reduced repeat indicated child abuse or

maltreatment reports of families Reduced length of stay in out-of-home care Increased number of family reunifications

from foster care placements

7

Description:Lead Agencies

9 program sites run by 5 agencies Upstate:

Finger Lakes Addictions Counseling & Referral Agency, Inc. (FLACRA)

Otsego Chemical Dependencies Clinic St Mary's Hospital

NYC: Women in Need, Inc. (WIN) Veritas Therapeutic Community, Inc.

8

Description: Map of Program Sites

9

Description: Map of Program Sites

10

Description: Map of Program Sites

Other Studies

12

Literature Review

Delaware’s Title IV-E Waiver Project Multidisciplinary Team Treatment Project Substance abuse counselor located in DFS

unit Program in place 5 years Reduced foster care days and costs Source: Dillard, D. (March, 2002) Final

Evaluation Report

13

Literature Review

Collocation of Mental Health and Health in MI Randomized controlled trials studying

patients with major depression receiving broad-based collaborative treatment by both PCPs and MHPs

Collocation of MHP and PCP practices in same building was strongly associated with increased interaction and collaboration

Source: Valenstein et al.(1999) Journal of Family Practice

14

Literature Review

NJ’s School-Based Youth Services Program Provides range of services to children and

adolescents in schools including mental health and substance abuse counseling, health care, employment preparation, crisis intervention, and summer programs

Improved educational aspirations and psycho-social attributes

Source: K. Hooper-Briar & H. Lawson (Eds.) (1996) Expanding Partnerships for Vulnerable Children, Youth and Families

Methodology

16

Methodology: Logic Model

Program Implementation

Activities Intermediate Outcomes Long Term Outcomes

Joint screening process/joint case investigations

Earlier identification of substance abuse issues

Decreased foster care placements

Assessment by Substance Abuse Specialist

More timely access to appropriate, comprehensive treatment services

Reduced repeat indicated CPS reports of families with substance abuse issues

Jointly developed treatment plan

Better compliance and retention in treatment

Reduced length of stay in out of home care

Case follow-up Reduction in substance

abuse Increased number of

family reunifications from foster care placements

Coordinated service delivery

Planning

Organizational Factors

Contextual Factors

17

Methodology:Overview

Funded by Children’s Bureau, DHHS in 2003 Three year evaluation project Both implementation and outcome studies Implementation study

7 Program Sites Focus Group, Interviews and Reports

Outcome Study 1 NYC site 1 upstate site Case Record review

18

Methodology:Implementation vs. Outcome

Study the implementation at the system level Identify factors for successful implementation

and barriers to success Provide insights on the design of the outcome

study Examine the effects on parents and children

Assess program effectiveness on substance abuse and child welfare outcomes

Identify factors that facilitate or hinder in achieving outcomes

19

Methodology:Outcome Study

A quasi-experimental design using case record reviews Retrospective longitudinal design Follow the two groups of families in the

CPS system over 1 year Treatment group -- families who came in

contact with CPS after collocation Control group – families who came in

contact with CPS before collocation

Findings from Implementation Study

21

Implementation Study: Data Collection

State level interviews Site visits (7 sites)

Focus groups of child welfare workers Focus groups of child welfare supervisors Interviews with Program Coordinator Interviews with administrators of the

treatment agency Interviews with child welfare administrators

22

Implementation Study:Analytical Approach

Utilized evaluation framework to describe

and assess program processes Compared program operations to the

logic model Compared practices among the sites

23

Implementation Findings:Design and Service Initiation

The program framework was identified in the RFP but was left up to the localities to design the program mechanics

Involving the right people in the program design has important implications

Funding uncertainties resulted in early start-up difficulties

Hiring the right person to be AOD Specialist was important

The newly hired AOD Specialists experienced a slow start

24

Implementation Findings:Reaching the Target Population

Two Questions: Are certain targeted groups not receiving

services? Are services delivered to individuals

outside of the targeted group? The crux of the program is the identification of

the target population

25

Implementation Findings:Reaching the Target Population

To reach the goal of increasing the identification of substance abusers presumes there are clients with AOD issues that are unknown to child welfare staff

The program is also designed to serve substance abusers known to the child welfare system

26

Implementation Findings:Reaching the Target Population

Client identification occurred in four ways (each with their own limitations) Through the initial hotline call On the initial investigation Following the initial investigation Families in the system

27

Implementation Findings:Reaching the Target Population

Not all targets are being served There are individuals being served who are not part of the targeted group This precludes obtaining some program goals (e.g., family reunification) This also leads to outcomes in other areas

28

Implementation Findings:Other issues in client identification

Must consider motivation of child welfare staff to refer clients. Caseworkers and their supervisors are gatekeepersWhat are the motivations?Decreased workloadEvidence to indicate caseMoving client through the systemHelping families

29

Implementation Findings:Client Assessment

Child welfare investigation provides “window of

opportunity.” “Good Cop/Bad Cop” approach Value of home visits in assessment process

30

Implementation Findings:Treatment Referral

Factors in deciding where clients are sent for treatmentSponsoring treatment agencyMatching client to appropriate programPurchased service agreementGetting clients engaged in servicesMonitoring compliance

31

Implementation Findings:Follow Up and Other Services

How long AOD Specialist remains with the case differed by locationNYC was exclusively a program to identify clients and refer them to treatmentUpstate programs provided case management

32

Implementation Findings:Relationships Between System

Collocation is a particular form of collaborationThis type of collaborative arrangement is not an equal partnershipSubstance abuse is entering another systemChanges have resulted at the systems levelChanges have resulted at the worker level

33

Implementation Findings:Effects of Collocation

Can collocation achieve the intended outcomes?The potential effects can be determined by the population served and activities provided.Intermediate (substance abuse) outcomesLong term (child welfare) outcomes

34

Implementation Findings:Lessons Learned

Involve front line workers in the design Consider front line motivations Provide structure for early operations Collocation matters Treatment agency involvement makes a difference

Next Step

36

Next Step: Challenges

High expectations on positive outcomes from stakeholders

Selecting sites for outcome study: 1 upstate site and 1 NYC site

Drawing the sample for outcome study Outcome measures Access to case records Developing Case Record Extraction Form

37

Next Step:Study Sample Issues

Over-sampling of the cases served by AOD specialists

How to select Comparison Group Sample Size (tentative N=400) NYC and Upstate comparison

38

Next Step:Outcome Measures

Rates of SA identification Rates of treatment engagement Rates of treatment completion % of CPS re-reports over one year % of placement after one year