Chapin Hall Center for Children Measuring Changes in Reclaiming Futures Communities: National...
-
Upload
brett-west -
Category
Documents
-
view
216 -
download
0
Transcript of Chapin Hall Center for Children Measuring Changes in Reclaiming Futures Communities: National...
Chapin Hall Center for Children
Measuring Changes in Reclaiming Futures Communities: National Evaluation Results
Annual Meeting of the Coalition for Juvenile JusticeWashington, DCJune 11, 2007
Jeffrey A. ButtsChapin Hall Center for Children
University of Chicago
Views expressed are those of the author and should not be attributed to the University of Chicago, Chapin Hall Center for Children, their trustees or funders..
Chapin Hall Center for Children
Reclaiming Futures
An initiative of Robert Wood Johnson Foundation
Focused on substance abuse interventions in juvenile justice
Principal strategy was organizational change/system reform
Evaluation challenge: Tracking change in 10 completely different service systems with 10 different change strategies
Chapin Hall Center for Children
Surveys of Reclaiming Futures Communities
National evaluation tracked change with bi-annual surveys
Surveys measured quality of systems as reported by 20 to 40 expert informants in each community
Six surveys conducted in:2003 – December 2004 – June and December 2005 – June and December 2006 – June
Response rate averaged 70 percent and ranged from 63% (December 2003) to 73% (December 2004)
Chapin Hall Center for Children
Surveys of Reclaiming Futures Communities
58 questions about quality and effectiveness
All questions were brief statements
Respondents indicated whether they strongly disagreed, disagreed, were neutral, agreed, or strongly agreed
Answers coded so higher scores indicate positive opinions
Responses scored: -10 strongly negative -5 somewhat negative 0 neutral +5 somewhat positive +10 strongly positive
Chapin Hall Center for Children
Surveys of Reclaiming Futures Communities
3 General Areas 13 Indices
ADMINISTRATION Access to ServicesData SharingSystems IntegrationResource Management
COLLABORATION Client InformationPartner InvolvementAgency Collaboration
QUALITY AOD AssessmentsTreatment EffectivenessTargeted TreatmentCultural IntegrationFamily InvolvementPro-social Activities
Chapin Hall Center for Children
Surveys of Reclaiming Futures Communities
3 General Areas 13 Indices
ADMINISTRATION Access to ServicesData SharingSystems IntegrationResource Management
COLLABORATION Client InformationPartner InvolvementAgency Collaboration
QUALITY AOD AssessmentsTreatment EffectivenessTargeted TreatmentCultural IntegrationFamily InvolvementPro-social Activities
The overall effectiveness of treatment was measured with five statements:
In the past three months, …
1. The substance abuse treatment needs of youth in my community were adequately met.
2. The mental health needs of youth in my community were adequately met.
3. Graduated sanctions were used effectively to support treatment goals for youth.
4. Youth-serving agencies in my community generally did a good job serving youth.
5. Youth-serving agencies in my community were usually able to provide youth with the range of services they needed.
Chapin Hall Center for Children
Surveys of Reclaiming Futures Communities
3 General Areas 13 Indices
ADMINISTRATION Access to ServicesData SharingSystems IntegrationResource Management
COLLABORATION Client InformationPartner InvolvementAgency Collaboration
QUALITY AOD AssessmentsTreatment EffectivenessTargeted TreatmentCultural IntegrationFamily InvolvementPro-social Activities
Chapin Hall Center for Children
1 AOD Assessment 28% Yes2 Treatment Effectiveness 26% Yes
3 Client Information 24% Yes4 Family Involvement 22% Yes
5 Systems Integration 19% Yes6 Pro-social Activities 18% Yes
7 Data Sharing 18% Yes8 Resource Management 16% Yes
9 Agency Collaboration 14% Yes10 Targeted Treatment 12% Yes
11 Cultural Integration 11% Yes12 Access to Services 9% Yes
13 Partner Involvement -2% No
Change in scores across all Reclaiming Futures communities between December 2003 and June 2006, ranked by percentage of possible improvement realized
Survey IndexRankPercent of Possible Improvement Realized Statistically
Significant?
Note: The total improvement possible in any index is limited by the score in Survey 1. Indices with high values in Survey 1 have less room to improve, given that +10 is the maximum possible score. “Possible improvement” realized was calculated by taking the total difference in survey scores (Survey 6 – Survey 1), and dividing by the maximum possible improvement, or 10 minus the first survey score. Change Indices are rank ordered from largest to smallest change, based on percentage of possible improvement realized. A t-test was used to determine each if change was significantly different from zero.
Chapin Hall Center for Children
Significant increases in survey indices by RF community, 2003-2006
Note: Significant increases occurred when the difference in mean index scores on the first survey (December 2003) and last survey (June 2006) was statistically significant ( p < .05). An increase was considered to be linear when at least 4 of 5 possible score changes between the six surveys were in a positive direction.
Chapin Hall Center for Children
Community Significant and Linear
Increases: 2003-2006 Community Significant and Linear
Increases: 2003-2006
Anchorage Access to Services AOD Assessment Treatment Effectiveness Cultural Integration
New Hampshire Access to Services Data Sharing AOD Assessment Treatment Effectiveness Targeted Treatment
Santa Cruz None Dayton Access to Services Data Sharing Resource Management Client Information AOD Assessment Pro-social Activities
Chicago Data Sharing Systems Integration Resource Management AOD Assessment Pro-social Activities
Portland Data Sharing Cultural Integration
SE Kentucky Data Sharing Systems Integration Client Information Agency Collaboration AOD Assessment Family Involvement
Rosebud Data Sharing Client Information
Marquette Data Sharing Pro-social Activities
Seattle Treatment Effectiveness Targeted Treatment
None of the changes reported in Santa Cruz were significant and linear, but scores in Santa Cruz started out very high and thus had less room to improve.
Chapin Hall Center for Children
What About Bias?
Answers to survey questions could be measures of self-interest or wishful thinking
Regression analyses were performed to check on this possibility
Analyses tested the influence of several factors on the trajectory of changes in individual survey scores
Main variable of interest: Did survey scores change more for RF “Insiders” -- the 4 or 5 people in each community who were involved in planning the RF initiative and who were closer to RWJF?
Chapin Hall Center for Children
Treatment Effectiveness
The values differ, but the trajectories are positive for both groups, indicating growing confidence in the effectiveness of treatment.
Chapin Hall Center for Children
Community Significant and Linear
Increases: 2003-2006 Community Significant and Linear
Increases: 2003-2006
Anchorage Access to Services AOD Assessment Treatment Effectiveness Cultural Integration
New Hampshire Access to Services Data Sharing AOD Assessment Treatment Effectiveness Targeted Treatment
Santa Cruz None Dayton Access to Services Data Sharing Resource Management Client Information AOD Assessment Pro-social Activities
Chicago Data Sharing Systems Integration Resource Management AOD Assessment Pro-social Activities
Portland Data Sharing Cultural Integration
SE Kentucky Data Sharing Systems Integration Client Information Agency Collaboration AOD Assessment Family Involvement
Rosebud Data Sharing Client Information
Marquette Data Sharing Pro-social Activities
Seattle Treatment Effectiveness Targeted Treatment
None of the changes reported in Santa Cruz were significant and linear, but scores in Santa Cruz started out very high and thus had less room to improve.
If we cross out the indices that were possibly affected by bias…
Chapin Hall Center for Children
Community Significant and Linear
Increases: 2003-2006 Community Significant and Linear
Increases: 2003-2006
Anchorage Access to Services AOD Assessment Treatment Effectiveness Cultural Integration
New Hampshire Access to Services Data Sharing AOD Assessment Treatment Effectiveness Targeted Treatment
Santa Cruz None Dayton Access to Services Data Sharing Resource Management Client Information AOD Assessment Pro-social Activities
Chicago Data Sharing Systems Integration Resource Management AOD Assessment Pro-social Activities
Portland Data Sharing Cultural Integration
SE Kentucky Data Sharing Systems Integration Client Information Agency Collaboration AOD Assessment Family Involvement
Rosebud Data Sharing Client Information
Marquette Data Sharing Pro-social Activities
Seattle Treatment Effectiveness Targeted Treatment
None of the changes reported in Santa Cruz were significant and linear, but scores in Santa Cruz started out very high and thus had less room to improve.
Chapin Hall Center for Children
Community Significant and Linear
Increases: 2003-2006 Community Significant and Linear
Increases: 2003-2006
Anchorage Access to Services AOD Assessment Treatment Effectiveness Cultural Integration
New Hampshire Access to Services Data Sharing AOD Assessment Treatment Effectiveness Targeted Treatment
Santa Cruz None Dayton Access to Services Data Sharing Resource Management Client Information AOD Assessment Pro-social Activities
Chicago Data Sharing Systems Integration Resource Management AOD Assessment Pro-social Activities
Portland Data Sharing Cultural Integration
SE Kentucky Data Sharing Systems Integration Client Information Agency Collaboration AOD Assessment Family Involvement
Rosebud Data Sharing Client Information
Marquette Data Sharing Pro-social Activities
Seattle Treatment Effectiveness Targeted Treatment
None of the changes reported in Santa Cruz were significant and linear, but scores in Santa Cruz started out very high and thus had less room to improve.
Chapin Hall Center for Children
Community Significant and Linear
Increases: 2003-2006 Community Significant and Linear
Increases: 2003-2006
Anchorage Access to Services AOD Assessment Treatment Effectiveness Cultural Integration
New Hampshire Access to Services Data Sharing AOD Assessment Treatment Effectiveness Targeted Treatment
Santa Cruz None Dayton Access to Services Data Sharing Resource Management Client Information AOD Assessment Pro-social Activities
Chicago Data Sharing Systems Integration Resource Management AOD Assessment Pro-social Activities
Portland Data Sharing Cultural Integration
SE Kentucky Data Sharing Systems Integration Client Information Agency Collaboration AOD Assessment Family Involvement
Rosebud Data Sharing Client Information
Marquette Data Sharing Pro-social Activities
Seattle Treatment Effectiveness Targeted Treatment
None of the changes reported in Santa Cruz were significant and linear, but scores in Santa Cruz started out very high and thus had less room to improve.
Chapin Hall Center for Children
Community Significant and Linear
Increases: 2003-2006 Community Significant and Linear
Increases: 2003-2006
Anchorage Access to Services AOD Assessment Treatment Effectiveness Cultural Integration
New Hampshire Access to Services Data Sharing AOD Assessment Treatment Effectiveness Targeted Treatment
Santa Cruz None Dayton Access to Services Data Sharing Resource Management Client Information AOD Assessment Pro-social Activities
Chicago Data Sharing Systems Integration Resource Management AOD Assessment Pro-social Activities
Portland Data Sharing Cultural Integration
SE Kentucky Data Sharing Systems Integration Client Information Agency Collaboration AOD Assessment Family Involvement
Rosebud Data Sharing Client Information
Marquette Data Sharing Pro-social Activities
Seattle Treatment Effectiveness Targeted Treatment
None of the changes reported in Santa Cruz were significant and linear, but scores in Santa Cruz started out very high and thus had less room to improve.
Chapin Hall Center for Children
Conclusions• Respondent ratings of local juvenile justice systems improved significantly between 2003 and 2006.
• Increases were statistically significant in 12 of the 13 survey indices.
• Improvements were especially dramatic in the ratings for: - treatment effectiveness, - the use of client information in support of treatment, - the use of screening and assessment tools, and - overall systems integration.
• These findings suggests that the coordination and quality of substance abuse interventions for youthful offenders improved during the RWJF Reclaiming Futures initiative.
Chapin Hall Center for Children
Jeffrey A. ButtsResearch FellowChapin Hall Center for ChildrenUniversity of Chicago1313 East Sixtieth StreetChicago, IL 60637(773) [email protected]
This presentation can be accessed at:www.jeffreybutts.net
For more information: