Post on 07-May-2019
1
New Path Youth & Family ServicesNew Path Youth & Family Services
Program Evaluation Program Evaluation Activities and Counselling for Teens (ACT) ProgramActivities and Counselling for Teens (ACT) Program
September 1, 2003 September 1, 2003 –– March 31, 2006March 31, 2006
April 2006
Prepared by:
Arleigh Luckett, M.Ed App. Psych.
Larry Shirk, B.Sc CIS
ShirLuck Consultants
2
ACTACT Activities and Counselling for TeensActivities and Counselling for Teens
Activity basedFor youth age 9 –1512 session group programParent program (currently Triple P)Individual and family therapy
3
DemographicsDemographics --||--
Age at AdmissionAge at Admission
0
5
10
15
20
(ACT n=83)
Perc
ent o
f Clie
nts
ACT 0 0 3.6 12 8.4 19.3 13.3 21.7 12 7 1.2 0 0
6yrs 7yrs 8yrs 9yrs 10yrs 11yrs 12yrs 13yrs 14yrs 15yrs 16yrs 17yrs 18yrs
4
DemographicsDemographics --||--
Age GroupsAge Groups
56.6
43.4
0
10
20
30
40
50
60
Preadolescent (<= 12yrs) Adolescent (13-18yrs)
Perc
ent o
f Clie
nts
5
DemographicsDemographics --||--
Gender Gender
66%
34%
Boys Girls
6
Brief Child & Family Phone Brief Child & Family Phone InterviewInterview
Intake Screening
7
BCFPI Question/CalculationBCFPI Question/Calculation
Code Question/Calculation
RAIAp Regulation of Attention, Impulsivity and Activity, population normCOp Cooperativeness - population normsCDp Conduct - population norms.EXp Externalizing - population norms.SPp Separation from parents - population normsMAp Managing Anxiety - population normsMMp Managing mood- 6 item population normsSHp 6 mood + 3 self harm indicators, population normsINp Internalizing, population norms.TMHP Total. 6 mental health domains (5 for Teacher form), Population norms.SocPartP Social participation - population normsQRelP Quality of relationships - Population normsSchoolP School participation and achievement - population normsChFp Global functioning... pop normsFActP Family activities... pop normsFcFp Family comfort- population normsGFsP Global family situation - Population normsPMMp Informant - depression -population normsFADp FAD -population norms
8
010203040506070
Perc
ent o
f Clie
nts
IY 31 44 34 44 19 24 44 51 36 51 60 31 35 58 44 60 61 25 8.9
RAIAp
COp
CDp
EXp
SPp
MAp
MMp
SHp INp TM
HP
SocPart
QRelP
SchoolP
ChFp
FActP
FcFp
GFsP
PMMp
FADp
BCFPI Problem PrevalenceBCFPI Problem Prevalence Percentage of Clients Scoring Above CutoffPercentage of Clients Scoring Above Cutoff
n = 101n = 101
9
BCFPI BCFPI Average ScoresAverage Scores
n = 102n = 102
0.0
20.0
40.0
60.0
80.0
100.0
COp Avg Scr CDp Avg Scr EXp Avg ScrSPp Avg Scr MAp Avg Scr MMp Avg ScrSHp Avg Scr INp Avg Scr TMHP Avg ScrSocPartP Avg Scr QRelP Avg Scr SchoolP Avg ScrChFp Avg Scr FActP Avg Scr FcFp Avg ScrGFsP Avg Scr PMMp Avg Scr FADp Avg Scr
10
CAFAS AssessmentCAFAS Assessment
Completed by staff after second session
11
What are the Types and Extent of What are the Types and Extent of YouthsYouths’’ Impairments at Entry CAFAS?Impairments at Entry CAFAS?
:
n sd (standard deviation)
Youths’ Status at Entry CAFAS: 83 26.84
• Mean: average score
• Median: the middle score
• Mode: most frequently occurring score
12
Rating Total at Entry to TreatmentRating Total at Entry to Treatment
49.64
40 40
0
10
20
30
40
50
60
Mean Median Mode
Scor
e on
CA
FAS
13
CAFAS Total ScoreCAFAS Total Score
• 0-30 likely referred to qualified health professional• 40-70 likely receives outpatient services• 80-100 likely receives outpatient care with additional
services of a supportive or intensive nature• 110 -130 likely receives intensive, community-based
services, although some youths may need acute residential services at some point
• > 140 very intensive services would be required; may be in residential or inpatient settings at some point
14
Severity intervals at Entry to Treatment Severity intervals at Entry to Treatment (ACT n=83)(ACT n=83)
30.1
54.2
12
2.4 1.2 00
10
20
30
40
50
60
0-30 40-70 80-100 110-130 >140 not scored
% o
f clie
ntss
15
CAFAS SubscalesCAFAS Subscales
• School• Home• Community• Behavior Towards Others• Moods / Emotions• Self Harm• Substance Abuse• Thinking
16
Percentage of Youths With Severe or Moderate Impairment on Percentage of Youths With Severe or Moderate Impairment on CAFAS Subscales at Entry to TreatmentCAFAS Subscales at Entry to Treatment
(ACT n = 83)(ACT n = 83)
0
5
10
15
20
25
30
35
40
Per
cent
of C
lient
s
ACT 34.9 16.9 12 25.3 33.7 3.6 2.4 2.4
school home communitybehavior towards others
moods/ emotions self-harm
substance use thinking
17
Percentage of Youths Who Were Rated as Severely or Moderately Percentage of Youths Who Were Rated as Severely or Moderately Impaired on None to 8 of the 8 CAFAS Subscales at Entry to Impaired on None to 8 of the 8 CAFAS Subscales at Entry to
TreatmentTreatment with at least some subscale data, not necessarily on all subscalwith at least some subscale data, not necessarily on all subscaleses
0
5
10
15
20
25
30
35
40
Perc
ent o
f Clie
nts
ACT 36.1 31.3 12 12 3.6 3.6 1.2 0 0
None 1 2 3 4 5 6 7 8
18
Percentage of Youths Who Were Rated as Severely Impaired Percentage of Youths Who Were Rated as Severely Impaired on None to 8 of the 8 CAFAS Subscales at Entry to Treatmenton None to 8 of the 8 CAFAS Subscales at Entry to Treatment
with at least some subscale data, not necessarily on all subscawith at least some subscale data, not necessarily on all subscalesles
0
10
20
30
40
50
60
70
80
% o
f clie
nts
ACT 75 4 3 1 0 0 0 0 0
None 1 2 3 4 5 6 7 8
19
Percentage of Youths With Severe Impairment on CAFAS Percentage of Youths With Severe Impairment on CAFAS Subscales at Entry to TreatmentSubscales at Entry to Treatment
0
1
2
3
4
5
6
7
Perc
ent o
f Clie
nts
ACT 6 6 0 1.2 2.4 3.6 2.4 0
school home communitybehavior towards others
moods/ emotions self-harm substance
use thinking
20
What are the Outcomes for Youth?What are the Outcomes for Youth?
• Outcomes from Entry to Last CAFAS– Includes open and closed cases, and those with an Exit CAFAS
(T14)– n= # of cases with two or more CAFAS evaluations, excluding
CAFAS where it was indicated that no treatment was delivered; • Three Ways of Viewing Outcome
– Change in average scores– Proportion of youth improved 20 points or more_ Proportion of youth improved from above cutoff to below cutoff
21
Outcome: Change in Average CAFAS from Outcome: Change in Average CAFAS from Entry to Treatment to Exit CAFASEntry to Treatment to Exit CAFAS
40
10
49.74
20.39
40
20
0
10
20
30
40
50
60
Entry Exit
Scor
e on
CA
FAS
Mean Median Mode
22
Outcome: Change in Average CAFAS Score from Outcome: Change in Average CAFAS Score from Entry to Treatment to Exit CAFAS Entry to Treatment to Exit CAFAS -- Other StatisticsOther Statistics
Statistics for cases having both Entry and ExitStatistical significance .00
n 76Correlation .592
Standard Deviation 23.056
t Score 11.095
23
Outcome: Change in Average Score on CAFAS SubscalesOutcome: Change in Average Score on CAFAS Subscales from Entry to Treatment to Exit CAFASfrom Entry to Treatment to Exit CAFAS
(cases having both Entry to Treatment and Exit CAFAS completed)(cases having both Entry to Treatment and Exit CAFAS completed)
0
5
10
15
Ave
rage
Sco
re o
n C
AFA
S
ACT Entry 10.39 11.18 2.63 10.92 11.45 1.05 0.66 1.45Exit 5 4.47 1.05 5.53 3.42 0.26 0.39 0.26
school home community behavior towards
moods/ emotions self-harm substance
use thinking
24
Degree of Absolute Change in CAFASDegree of Absolute Change in CAFAS (positive value means improvement) (positive value means improvement)
0
5
10
15
20
25
Increase or Decrease in Score by Percent of Clients
Perc
ent o
f Clie
nts
ACT 0 0 0 0 0 0 0 1.3 13 13 16 24 17 3.9 2.6 3.9 2.6 1.3 0 1.3 0 0
-80 -70 -60 -50 -40 -30 -20 -10 0 10 20 30 40 50 60 70 80 90 100 110 120 130
Worse Improved
25
Youths Improved on Various Outcome MethodsYouths Improved on Various Outcome Methods
Outcome method Improved/PossibleImproved on at Least One of Three Outcome Indicators 56/72
Clinically meaningful reduction in total CAFAS Score (Restricted to youths with Entry to Treatment CAFAS >= 20)
55/72
No Severe impairments at exit (Restricted to youth with one or more severe impairments at Entry to Treatment)
4/8
Improving from SED Region no SED (Restricted to youths with Entry to Treatment CAFAS <=40)
30/37
26
Results on Outcome Measures Percent of Clients Not Improved, Improved or Excluded on Each Outcome
Measure
21.1
73.7
5.3
23.6
76.4
05.3 5.3
89.5
9.2
39.5
51.3
0102030405060708090
100
Not Improved Improved % CasesExcluded
Perc
ent o
f Clie
nts
Improved on One of Three IndicatorsClinically Meaningful ReductionNo Severe Impairments at ExitImproving from SED Region to no SED
27
Caregiver InformationCaregiver Information
NFamily Social Support subscale Entry Exit
Primary Family 74 76Non-Custodial Caregiver 13 7Surrogate Caregiver 0 0
Family Material Needs subscalePrimary Family 74 76Non-Custodial Caregiver 13 7Surrogate Caregiver 0 0
28
61.8
28.9
17.16.6
2.62.6 0
80.3
0%
20%
40%
60%
80%
100%
Entry ExitScore on CAFAS
Perc
ent o
f Car
egiv
ers
0 10 20 Missing
Family/Social Support SubscaleFamily/Social Support Subscale Primary Family Score at Entry and ExitPrimary Family Score at Entry and Exit
29
67.1
23.7
6.62.6
0
10
20
30
40
50
60
70
80
No Change Improved Worse Missing Data
Perc
ent o
f Clie
nts
Family/Social Support SubscaleFamily/Social Support Subscale Change in Care Giving Environment (Primary Family)Change in Care Giving Environment (Primary Family)
30
73.7
23.7
0
10
20
30
40
50
60
70
80
No Reliable Change Reduction by min 10 pts
Perc
ent o
f Clie
nts
Family/Social Support SubscaleFamily/Social Support Subscale CaregivingCaregiving Environment (Primary Family)Environment (Primary Family)
Reliable Change (decrease by at least 10 points)Reliable Change (decrease by at least 10 points)
31
Outcome Status for Closed CasesOutcome Status for Closed Cases
Outcome Status n
Treatment Not Needed: Only one CAFAS evaluation was done and there was a good reason for not providing treatment (i.e., evaluation only, no treatment needed, or referred to other service). This information is provided when case is closed
0
Not Improved: Did not improve on any of 3 outcome indicators 16
Improved: Improved on 1 or more of 3 outcome indicators 56
Entry Was 0 OR 10: Outcome can not be evaluated because the entry CAFAS was a 0 or 10. Since the least “ambitious” outcome indicator requires a reduction of 20 points or more, it is not possible to evaluate outcome if the entry score is less than 20
4
Likely Drop-Outs. Needed Treatment: There was an entry CAFAS but no subsequent CAFAS evaluations, and the case appeared to be appropriate for treatment (i.e., case was not coded as “evaluation only,” “no services needed,” or “referred to other services” when case was closed)
4
32
Three Outcome IndicatorsThree Outcome Indicators
Outcome Indicators nReliable Improvement(20 Pts) on CAFAS – Entry to Exit
55
No Severe Impairments at Exit 4Improving from SED Region non SED 30
33
BCFPI BCFPI Pre and Post ResultsPre and Post Results
September 1, 2005 September 1, 2005 –– March 31, 2006March 31, 2006
34
BCFPI BCFPI Average Pre and Post ScoresAverage Pre and Post Scores
Before n = 17 / After n = 15 FormsBefore n = 17 / After n = 15 Forms
0102030405060708090
100110
Stage: Before 68.2 70.8 72.7 73.6 61.2 63.9 74.7 80.9 69.7 74.7 76.2 69.2 75.3 78.4 69.5 72.6 75.4 61.6 57.8Stage: After 66.7 64.9 60.3 67.9 56.2 57.7 60.7 60 66 69.5 59.3 67.9 68.9 70.5 67.1 71.1 59.3 54.2
RAIAp COp CDp EXp SPp Map MM
p SHp INp TMHP
SocPart
P
QRelP
SchoolP
ChFp
FActP
FcFp
GFsP
PMMp
FADp
35
ClientsClients’’ and Staffand Staff’’s s Perception of ChangePerception of Change
Client Information SystemClient Information System
36
Change Reported by Clients Change Reported by Clients At ClosureAt Closure
n = 85n = 85
4%
53%
43%
No Change
SomeImprovementConsiderableImprovement
37
Change Reported by StaffChange Reported by Staff At ClosureAt Closure
n = 99n = 99
3%
56%
41%No Change
Some Improvement
ConsiderableImprovement
38
RecommendationsRecommendations
• Post CAFAS is done at the end of the program. Delay post BCFPI to measure change 3 to 6 months after exit. This could be done by phone, mail or through a follow-up session.
• The parenting aspect of the program has evolved over the period of this study. Caregiver data could be further analyzed to look for effect of changes in the parenting component over time.
• Recognize counselling and support staff for the ACT program for exellent compliance with data entry protocols.
39
AcknowledgementsAcknowledgements
Analysis and presentation of CAFAS data was modeled on:“Child and Adolescent Functional Assessment Scale Service Outcome for Children Receiving Treatment - December, 2005 “, Melanie Barwick, Cristina Vlad & Liling Xiang, CAFAS in Ontario.
Funding for this project was provided by:The Provincial Centre of Excellence for Child and Youth Mental Health at CHEO (PEG 162606-054)