Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and...
-
Upload
felicia-ginger-mcdaniel -
Category
Documents
-
view
215 -
download
1
Transcript of Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and...
![Page 1: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/1.jpg)
Using the Wraparound Fidelity Assessment System to Support High-Quality WraparoundOverview and Lessons Learnedfrom California Development Teams
Eric J. Bruns, Ph.D.April Sather, MPHUniversity of Washington School of MedicineWraparound Evaluation and Research TeamNational Wraparound [email protected]
Presented at theCalifornia Wraparound
Symposium
Sacramento, CaliforniaDecember 3, 2008
![Page 2: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/2.jpg)
Overview of the session Quick recap: The importance of
maintaining (and measuring) fidelity Overview of the Wraparound Fidelity
Assessment System Using the WFAS tools in California’s
Development Team communities Results from California sites Implications and possible next steps
![Page 3: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/3.jpg)
What is Wraparound? Wraparound is a family-driven, team-based process
for planning and implementing services and supports.
Through the wraparound process, teams create plans that are geared toward meeting the unique and holistic needs of children and youth with complex needs and their families.
The wraparound team members (e.g., the identified youth, his or her parents/caregivers, other family members and community members, mental health professionals, educators, and others) meet regularly to implement and monitor the plan to ensure its success.
![Page 4: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/4.jpg)
When wraparound is implemented as intended… High-quality teamwork and flexible funds leads to enhanced
creativity, better plans, and better fit between family needs and services/supports
This in turn leads to greater relevance for families, less dropout Strengths, needs, and culture discovery and planning process
leads to more complete engagement of families As family works with a team to solve its own problems,
develops family members’ self-efficacy Individualization and strengths focus enhances cultural
competence, relevance, and acceptability Focus on setting goals and measuring outcomes leads to more
frequent problem-solving and more effective plans
![Page 5: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/5.jpg)
Ten principles of the wraparound process
Model adherent wraparound
•Youth/Family drives goal setting
•Single, collaboratively designed service plan
•Active integration of natural supports and peer support
•Respect for family’s culture/expertise
•Opportunities for choice
•Active evaluation of strategies/outcomes
•Celebration of success
Phases and Activities of the Wraparound Process
Short term outcomes:
•Better engagement in service delivery
•Creative plans that fit the needs of youth/family
•Improved service coordination
•Follow-through on team decisions
•Family regularly experiences success/support
Theory of change for wraparound process
Intermediate outcomes:
•Participation in services
•Services that “work” for family
Intermediate outcomes:
•Achievement of team goals
•Increased social support and community integration
•Improved coping and problem solving
•Enhanced empowerment
•Enhanced optimism/self-esteem
Long term outcomes:
•Stable, home-like placements
•Improved mental health outcomes (youth and caregiver)
•Improved functioning in school/ vocation and community
•Improved resilience and quality of life
From Walker (2008)
![Page 6: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/6.jpg)
Outcomes of WraparoundDoes wraparound work?
For whom?
What leads to positive outcomes?
![Page 7: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/7.jpg)
Outcomes from Wraparound Milwaukee
After Wraparound Milwaukee assumed responsibility for youth at residential level of care (approx. 700-1000 per year)… Average daily Residential Treatment population reduced
from 375 placements to 70 placements Psychiatric Inpatient Utilization reduced from 5000 days
per year to under 200 days (average LOS of 2.1 days) Reduction in Juvenile Correctional Commitments from
325 per year to 150 (over last 3 years)
(Kamradt et al., 2008)
![Page 8: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/8.jpg)
Results from Nevada:Impact on Child Functioning
0
20
40
60
80
100
120
Intake 6 months 12 months 18 monthsAv
era
ge
Fu
nc
tio
na
l Im
pa
irm
en
t o
n t
he
CA
FA
S
Traditional Svcs Wraparound
Bruns et al. (2006)
![Page 9: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/9.jpg)
Results from Clark County, WAImpact on juvenile justice outcomes
Connections (wraparound) group (N=110) 3 times less likely to commit felony offense than comparison group (N=98)
Connections group took 3 times longer on average to commit first offense after baseline
Connections youth showed “significant improvement in behavioral and emotional problems, increases in behavioral and emotional strengths, and improved functioning at home at school, and in the community”
Pullman et al. (2006)
![Page 10: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/10.jpg)
Other outcomes of wraparound Greater/more rapid achievement of
permanency when implemented in child welfare (Oklahoma)
More successful integration of adult prisoners into the community (Oklahoma)
Reduction in costs associated with residential placements (LA County, Washington State, Kansas, many other jurisdictions)
![Page 11: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/11.jpg)
There have been Seven Published Controlled Studies of Wraparound
StudyTarget
populationControl Group Design N
1. Bickman et al. (2003) Mental health Non-equivalent comparison 111
2. Carney et al. (2003) Juvenile justice Randomized control 141
3. Clark et al. (1998) Child welfare Randomized control 132
4. Evans et al. (1998) Mental health Randomized control 42
5. Hyde et al. (1996) Mental health Non-equivalent comparison 69
6. Pullman et al. (2006) Juvenile justice Historical comparison 204
7. Rast et al. (2007) Child welfare Matched comparison 67
![Page 12: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/12.jpg)
Effect Sizes(Cohen, 1988)
Small = 0.2 Large = 0.8Medium = 0.5
![Page 13: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/13.jpg)
Mean Effect Sizes & 95% Confidence Intervals
![Page 14: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/14.jpg)
Findings from the seven controlled studies Strong results in favor of wraparound found for
Living Situation outcomes (placement stability and restrictiveness)
A small to medium sized effect found for: Mental health (behaviors and functioning) School (attendance/GPA), and Community (e.g., JJ, re-offending) outcomes
The overall effect size of all outcomes in the 7 studies is about the same (.35) as for “evidence-based” treatments, when compared to services as usual (Weisz et al., 2005)
Suter & Bruns (2008)
![Page 15: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/15.jpg)
Positive Outcomes are Not Guaranteed!
Studies indicate that Wraparound teams often fail to: Incorporate full complement of key individuals on the
Wraparound team; Engage youth in community activities, things they do well,
or activities to help develop friendships; Use family/community strengths to plan/implement services; Engage natural supports, such as extended family members
and community members; Use flexible funds to help implement strategies Consistently assess outcomes and satisfaction.
![Page 16: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/16.jpg)
Fidelity is critical to outcomes Higher levels of fidelity to organizational level assessment for
ACT was associated with greater reductions in days spent in psychiatric hospitals (McGrew, Bond, Dietzen & Salyers, 1994)
Improved youth delinquency outcomes for higher fidelity Teaching Family model (Kirigin et. al. 1982)
Improved youth delinquency outcomes for higher fidelity MST (Henggler, Melton, Browndino, Scherer and Hanley, 1997)
Better overall outcomes for youth receiving model adherent FFT (Alexander, Pugh, Parsons and Sexton, 2000)
Better outcomes for school-wide behavioral management when implemented with fidelity (Felner et. al. 2001)
![Page 17: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/17.jpg)
What is the connection between fidelity and outcomes with wraparound?
Provider staff whose families experience better outcomes were found to score higher on fidelity tools (Bruns, Rast et al., 2006)
Wraparound initiatives with positive fidelity assessments demonstrate more positive outcomes (Bruns, Leverentz-Brady, & Suter, 2008)
![Page 18: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/18.jpg)
What does it take to get high fidelity scores? Communities with better developed supports
for wraparound show higher fidelity scores Training and coaching found to be associated
with gains in fidelity and higher fidelity
![Page 19: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/19.jpg)
Stages of Implementation(from Fixsen et al., 2005)
Context and Readiness
StaffSelection
Training
Supervision and
Coaching
Performance Management
ProgramEvaluation
OrganizationalSupports
![Page 20: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/20.jpg)
Team
Organization(lead and partner agencies)
System (Policy and Funding Context)
Effective
Supportive
Hospitable
Levels Of Support For Wraparound
![Page 21: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/21.jpg)
Types of program and system support for Wraparound1. Community partnership: Do we have collaboration across
our key systems and stakeholders?2. Collaborative action: Do the stakeholders take concrete
steps to translate the wraparound philosophy into concrete policies, practices and achievements?
3. Fiscal policies: Do we have the funding and fiscal strategies to meet the needs of children participating in wraparound?
4. Service array: Do teams have access to the services and supports they need to meet families’ needs?
5. Human resource development: Do we have the right jobs, caseloads, and working conditions? Are people supported with coaching, training, and supervision?
6. Accountability: Do we use tools that help us make sure we’re doing a good job?
![Page 22: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/22.jpg)
Data Supporting Wraparound
Implementation
WFAS
WFI-4
TO
M
CSWI
Doc
Rev
iew
Observation Tools
Document Review Tools
21 43
21 63 4 5
VVDBTraining and Coaching
Tools
WFASResearch and CQI
ToolsNew Staff Credentialed Staff
![Page 23: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/23.jpg)
Evaluating the quality and fidelity of wraparound implementation
The Wraparound Fidelity Assessment System
![Page 24: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/24.jpg)
Wraparound Fidelity Assessment System
WFAS
WFI-4
TO
M
CSWI
Doc
Rev
iew
WFI-4 – Wraparound
Fidelity Index
CSWI – Community Supports for
Wraparound Inventory
DRM – Document
Review Measure
TOM – Team Observation
Measure
![Page 25: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/25.jpg)
The Wraparound Fidelity Index, version 4
Assesses implementation of the wraparound process through brief interviews with multiple respondents Caregivers Youths Wraparound Facilitators Team Members
Found to possess good psychometric characteristics Test-retest reliability Inter-rater agreement Internal consistency
Used in research on wraparound Even more widely as a quality assurance mechanism by
wrap programs
![Page 26: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/26.jpg)
Wraparound Fidelity Index, v.4
Items on the principles and core activities, organized by the 4 phases of wraparound Engagement: Did you select the people who would be on your
youth and family team? Principle = Team based
Planning: Does the plan include strategies for helping your child get involved with activities in the community? Principle = Community based
Implementation: Does the team evaluate progress toward the goals of the plan at every team meeting? Principle = Outcome based
Transition: Will some members of your team be there to support you when formal wraparound is complete? Principle = Persistence
![Page 27: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/27.jpg)
WFI Items: Engagement and Team Preparation Phase
Phase 1: Engagement Yes SometimesSomewhat No
1.CC
When you first met your wraparound facilitator, were you given time to talk about your family's strengths, beliefs, and traditions? Circle one: YES NODid this process help you appreciate what is special about your family? Circle one: YES NO
YES to both
questions
YES to only the first question
NO to the first
question
2 1 0
2.FVC
Before your first team meeting, did your wraparound facilitator fully explain the wraparound process and the choices you could make?
2 1 0
3.SB
At the beginning of the wraparound process, did you have a chance to tell your wraparound facilitator what things have worked in the past for your child and family?
2 1 0
4.TB
Did you select the people who would be on your wraparound team?
2 1 0
5.TB
Is it difficult to get agency representatives and other team members to attend team meetings when they are needed?
0 1 2
6.OB
Before your first wraparound team meeting, did you go through a process of identifying what leads to crises or dangerous situations for your child and your family?
2 1 0
![Page 28: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/28.jpg)
Scoring the WFI:Individual items (Planning phase)
Q1. Were you given time to talk about your family's strengths, beliefs, and traditions?True - 10 Partly True - 3 Not True - 2
Q2. Did your facilitator fully explain wraparound & the choices you could make? True - 9 Partly True - 4 Not True - 2
Q3. Did you have a chance to tell your wraparound facilitator what has worked in the past for your child and family? True - 7 Partly True - 4 Not True - 4
Q4. Did you select the people who would be on your wraparound team? True - 7 Partly True - 4 Not True – 4
Q5. Is it difficult to get team members to meetings when they are needed? True – 9 Partly True – 3 Not True - 3
Q6. Did you go through a process of identifying what leads to crises for yr family? True – 8 Partly True – 3 Not True - 4
1.541.46
1.2 1.2
1.4
1.26
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
Q1 Q2 Q3 Q4 Q5 Q6
![Page 29: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/29.jpg)
Example WFI report:Comparing scores across sites and to nat’l mean
1.54 1.46
1.2 1.21.4
1.261.13
1.241.12
1.32 1.261.17
1.4 1.5
1.2 1.3
1.6
1.3
0
0.5
1
1.5
2
Site A 1.54 1.46 1.2 1.2 1.4 1.26
Site B 1.13 1.24 1.12 1.32 1.26 1.17
Natl Mean 1.4 1.5 1.2 1.3 1.6 1.3
Q1 Q2 Q3 Q4 Q5 Q6
![Page 30: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/30.jpg)
WFI-4: Discriminant Validity
50
60
70
80
90
100P
erce
nt
of
tota
l fid
elit
y
Facilitator 83.8 74.4
Caregiver 77.1 62.1
Youth 73.6 61.9
Team Member 78.4 70.1
Wrap sites (n=12)Non-wrap sites
(n=3)
**P<.0001; *p<.01
*
****
![Page 31: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/31.jpg)
What is “high fidelity?”An empirical question
Non-wraparound study samples and “pre-training” sites (N=4)60
50
80
70
100
90
Standard wraparound conditions in published studies (N=5)
Wrap sites assessed with greatest supports (N=2)
Wrap sites with fewest supports (N=2)
Wrap staff/sites achieving best outcomes (N=2)
Wrap staff achieving less positive outcomes (N=1)
“High”
Above avg.
Average
Below avg.
Not Wrap
WFI Total scores
![Page 32: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/32.jpg)
The Team Observation MeasureVersion 1
![Page 33: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/33.jpg)
Team Observation Measure
The Team Observation Measure (TOM) is employed by external evaluators to assess adherence to standards of high-quality wraparound during team meeting sessions.
It consists of 20 items, with two items dedicated to each of the 10 principles of wraparound.
Each item consists of 3-5 indicators of high-quality wraparound practice as expressed during a child and family team meeting.
![Page 34: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/34.jpg)
S T R E N G T H S
Areas for improvement
Rank Item Name Avg. score
1 20 Least Restrictive Environment 3.5
2 15 Youth & Family Voice 3.2
3 18 Positive Team Culture 3.2
4 10 Shared Responsibility 3.0
5 2 Effective Team Process 2.9
6 3 Facilitator Preparation 2.9
7 12 Cultural & Linguistic Competence 2.9
8 16 Youth & Family Choice 2.9
9 1 Team Membership & Attendance 2.5
10 6 Individualized Process 2.4
11 11 Facilitation Skills 2.4
12 4 Effective Decision Making 2.4
13 17 Focus on Strengths 2.0
14 5 Creative Brainstorming Options 1.9
15 9 Team Mission and Plans 1.8
16 13 Outcomes Based Process 1.8
17 19 Community Focus 1.4
18 7 Natural & Community Supports 1.4
19 8 Natural Support Plans 1.3
20 14 Evaluating Progress & Success 1.3
![Page 35: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/35.jpg)
Sample TOM report:Most frequently observed TOM indicators
# Item Pct. SD
20aThe team's mission and/or needs support the youth's integration into the least restrictive residential and educational environments possible 96% .208
1a Parent/caregiver is a team member and present at meeting 92% .266
12e Members of the team use language the family can understand 92% .271
18dSerious challenges are discussed in terms of finding solutions, not termination of services or sanctions for the family. 91% .288
3aThere is a written agenda or outline for the meeting, which provides an understanding of the overall purpose of meeting 89% .320
11eTalk is well distributed across team members and each team member makes an extended or important contribution 89% .320
18e There is a sense of openness and trust among team members 89% .320
20dSerious behavioral challenges are discussed in terms of finding solutions, not placement in more restrictive residential or educational environments
89% .332
![Page 36: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/36.jpg)
Sample TOM report:Least frequently observed TOM indicators
# Item Pct SD
8aIn designing strategies, team members consider and build on strengths of the youth and family
28% .458
13b The team assesses goals/strategies using measures of progress 26% .446
5dThe facilitator leads a robust brainstorming process to develop multiple options to meet priority needs.
23% .429
7cCommunity team members and natural supports have a clear role on the team
23% .429
14aThe team conducts a systematic review of members' progress on assigned action steps
23% .429
19aThe team is actively brainstorming and facilitating community activities for the youth and family
23% .429
8bThe plan of care represents a balance between formal services and informal supports
17% .380
1c Key natural supports for the family are team members and present 11% .362
![Page 37: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/37.jpg)
TOM pilot testing
Now being implemented in nearly 20 sites, including three statewide implementation efforts
Results showed good variability in scores across teams Internal consistency (alpha) = .862
Alpha coefficients also good for individual items
Initial inter-rater agreement for indicators = 79% More rigorous inter-rater reliability study now underway
Significant positive correlation found with CG and WF forms of the WFI-4
![Page 38: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/38.jpg)
Distribution of TOM scores (from initial pilot)
Distribution of Total TOM scores
![Page 39: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/39.jpg)
The CSWICommunity Supports for Wraparound Inventory
![Page 40: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/40.jpg)
The Community Supports for Wraparound Inventory (CSWI)
The Community Supports for Wraparound Inventory (CSWI) is intended for use as both a research and quality improvement tool to measure how well a local system supports the implementation of high quality wraparound.
The CSWI is based on the Necessary Conditions for Wraparound described by Walker & Koroloff (2007)*
Further refined through collaborative work undertaken by the National Wraparound Initiative
Includes 42 community or system variables that support wraparound implementation.
Requires ~45 minutes to complete
*Walker, J. S., & Koroloff, N. (2007). Grounded theory and backward mapping: Exploring the implementation context for wraparound. Journal of Behavioral Health Services & Research.
![Page 41: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/41.jpg)
CSWI
The 42 items are grouped within the 6 core themes for wraparound support
1. Community partnership2. Collaborative action3. Fiscal policies and sustainability4. Service array5. Human resource development, and6. Accountability
Respondents complete the 42 items by rating the development of supports in their community or program on a 5 point scale 0 = “least developed” and 4 = “fully developed”
![Page 42: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/42.jpg)
Community Procedures
Stakeholders are identified by a local coordinator and invited by email to complete the CSWI via a link to a web survey version Stakeholder groups– family, youth, admin, providers, etc. “key respondents” Project employees
Local coordinator builds support for participation Emails that bounce are removed from the sample Reminders sent until research team and local
coordinators decide to close the survey
![Page 43: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/43.jpg)
CSWI Total Scores(Maximum possible = 160)
0
20
40
60
80
100
120
140
160
Site 1 Site 2 Site 3 Site 4 Site 5 Site 6 Site 7 AllSites
![Page 44: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/44.jpg)
Sample Site Feedback: All Themes
0
2
4
6
8
10
12
0.00 1.00 2.00 3.00 4.00
NWI MeanSite 2 MeanTheme 1: Community
Partnerships
Least Developed
Midway Fully Developed
Theme 2: Collaborative Action
Theme 3: Fiscal Policies and Sustainability
Theme 4: Availability of Services and Supports
Theme 5: Human Resource Development
Theme 6: Accountability
Theme Means: Site and National Comparison
![Page 45: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/45.jpg)
Sample Site Feedback: Theme 1
0
2
4
6
8
10
12
14
0 1 2 3 4
NWI MeanSite 2 Mean
1.1: Community Team
Least Developed
Midway Fully Developed
1.2: Empowered Community Team
1.3: Family Voice
1.4: Youth Voice
1.5: Agency Support
1.6: Community Stakeholders
1.7: Community Representativeness
Theme 1: Site and National Item Means
![Page 46: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/46.jpg)
Another Site’s Feedback: Theme 1
0
2
4
6
8
10
12
14
0 1 2 3 4
NWI MeanSite 7 Mean
1.1: Community Team
Least Developed
Midway Fully Developed
1.2: Empowered Community Team
1.3: Family Voice
1.4: Youth Voice
1.5: Agency Support
1.6: Community Stakeholders
1.7: Community Representativeness
Theme 1: Site and National Item Means
![Page 47: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/47.jpg)
Implementing the WFAS tools in California Development Team communities
![Page 48: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/48.jpg)
Conducting a fidelity evaluation: Things to consider
Practice model Does yours align with the NWI model?
Target population Is the full wraparound model implemented for all youth
or just a specific subpopulation?
Sampling frame At what levels do you want to assess quality and
fidelity Whole Community or program? Individual sites or provider organizations? Individual Staff or supervisors?
![Page 49: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/49.jpg)
Conducting a fidelity evaluation: Things to consider Sampling
What percent or number of families do you have the resources to include in the sample? Representativeness of sample (e.g., random
sampling) and completion rate more important than assessing all families served
Which instruments will you use? Will you systematically collect data on a fourth
team member for the WFI-4? E.g., if there are consistent team members (case
worker, family support worker)
![Page 50: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/50.jpg)
Conducting a fidelity evaluation:Things to consider
Data collection considerations Who will collect data? Who will oversee data collection? Who will train interviewers, reviewers, and observers to criteria?
How will you use the data? Is there a state or community oversight entity to review results? Will data be used to hold individual organizations or supervisors
accountable? How will you use the data to construct a quality improvement plan?
![Page 51: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/51.jpg)
Progress report Nine California sites successfully implementing the
WFI Five sites successfully employing the TOM Two sites collecting CSWI data, with additional
counties now coming on board California sites leading the way in using the Online
Data Entry and Reporting System (WONDERS) More formal reports also provided by WERT to each site
Initial efforts at training raters to criteria on the Document Review Measure (DRM) not successful Sites using VVDB doc review tools to serve this purpose Feedback from California sites key to revising the DRM
![Page 52: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/52.jpg)
Data from CaliforniaInitial results from WFI, TOM
![Page 53: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/53.jpg)
CIMH Wraparound ProjectWFI-4 Fidelity Scores
50%
60%
70%
80%
90%
100%
Combined 84% 75% 77% 84% 71% 79% 69% 71% 76%
Facilitator 87% 76% 82% 88% 70% 83% 68% 71% 77%
Caregiver 75% 72% 73% 80% 75% 75% 71% 75%
Youth 80% 77% 75% 70% 76% 73%
Site 1 Site 2 Site 3 Site 4 Site 5 Site 6 Site 7 Site 8 Site 9
![Page 54: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/54.jpg)
CIMH Wraparound ProjectWFI-4 and Team Observation Scores
50%
60%
70%
80%
90%
100%
WFI Combined 84% 77% 71% 79% 69%
Team Observation 84% 81% 76% 68% 66%
Site 1 Site 3 Site 5 Site 6 Site 7
![Page 55: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/55.jpg)
CIMH Wraparound ProjectAverage scores for sites
50%
60%
70%
80%
90%
100%
CDT Project 78% 76% 75% 75%
Natl Mean 80% 72% 71% 69%
Facilitator Caregiver Youth Team Observation
![Page 56: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/56.jpg)
TOM scores by itemItem National
MeanN=106
Site 503 Mean
N=14
Site 507 Mean
N=10
Item 1 Team Membership & Attendance 2.42 2.10 2.08
Item 2 Effective Team Process 3.14 3.70 3.69
Item 3 Facilitator Preparation 3.17 3.00 3.69
Item 4 Effective Decision Making 2.94 3.50 3.08
Item 5 Creative Brainstorming and Options 2.09 2.25 3.00
Item 6 Individualized process 2.98 3.50 2.77
Item 7 Natural and Community Supports 1.96 2.43 1.42
Item 8 Natural Support Plans 2.05 2.30 2.08
Item 9 Team Mission and Plans 2.90 3.60 3.69
Item 10 Shared Responsibility 3.25 3.89 3.31
Item 11 Facilitation Skills 2.69 3.50 3.23
Item 12 Cultural and Linguistic Competence 3.33 3.80 3.77
Item 13 Outcomes Based Process 2.67 3.33 3.92
Item 14 Evaluating Progress and Success 2.27 3.00 3.38
Item 15 Youth and Family Voice 3.35 3.90 4.00
Item 16 Youth and Family Choice 3.14 3.80 2.62
Item 17 Focus on strengths 2.58 3.20 3.77
Item 18 Positive team culture 3.22 3.70 3.77
Item 19 Community focus 2.86 2.30 1.50
Item 20 Least Restrictive Environment 3.52 3.70 3.46
TOTAL TOM Mean Score
2.82 3.24 3.11
![Page 57: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/57.jpg)
California fidelity dataMost frequently observed TOM indicators
# Item Pct. SD
17a Team members acknowledge/list youth/family strengths 98% .158
20aThe team's mission and/or needs support the youth's integration into the least restrictive residential and educational environments possible 97% .118
12e Members of the team use language the family can understand 97% .200
2c Tasks and strategies are explicitly linked to intermediate goals 97% 151
13c Team revises plan if progress toward goals is not evident 96% .200
12dTeam demonstrates clear respect for the family’s values, beliefs, and traditions 95% .211
2aTeam members and attendees are oriented to the wraparound process and understand the purpose of the meeting 95% .221
15a 15b
Team provides extra opportunity for CGs/parents AND youth to speak and offer opinions, especially during decision making 95%
.221/.226
18a Team focuses on improvements or accomplishments in team mtg 95% .211
![Page 58: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/58.jpg)
California fidelity data:Least frequently observed TOM indicators
# Item Pct SD
14b The team reviews progress on measurable outcomes in the plan 68% .450
13b The team assesses goals/strategies using measures of progress 65% .463
14aFacilitator and team members draw from knowledge about the community to generate strategies and action steps
60% .467
4d The plan of care is agreed upon by all present at the meeting 59% .474
7cCommunity team members and natural supports have a clear role on the team
50% .510
5dThe facilitator leads a robust brainstorming process to develop multiple options to meet priority needs.
47% .509
8bThe plan of care represents a balance between formal services and informal supports
47% .505
1d Key natural supports for the family are team members and present 37% .489
1cKey school and agency representatives are team members and present
21% .412
![Page 59: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/59.jpg)
Overall findings from California Fidelity Assessments Scores for California sites above national average – Except
for WFI-Facilitator form Particular strengths in adherence to wraparound
implementation steps and procedures Strength = training/coaching/fidelity infrastructure? Need = additional system-level development for implementation?
WFI and TOM scores correlate at site and family level Evidence for validity of the tools
Outcomes data still being collected Will be able to look at fidelity-outcomes trends at site level
Data used by local sites to support program improvement Data can be used in future in supervision and coaching
![Page 60: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/60.jpg)
What is next?
![Page 61: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/61.jpg)
On-line data entry and reporting
![Page 62: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/62.jpg)
![Page 63: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/63.jpg)
![Page 64: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/64.jpg)
![Page 65: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/65.jpg)
![Page 66: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/66.jpg)
![Page 67: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/67.jpg)
![Page 68: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/68.jpg)
![Page 69: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/69.jpg)
Using data from the WFAS statewideAn example from Maryland
![Page 70: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/70.jpg)
Maryland WFAS Data CollectionWFAS Data Collection Timeline
Intake
1 mo 12 mo
CME Collects Consent To Be Contacted
6 mo
CCs/FSPs notified of upcoming teams eligible for WFI & TOMs
Research Staff conducts WFI & TOMs
CCs/FSPs notified of teams eligible for WFI follow-up
Research Staff conducts WFI Follow-Up
Data Goals – 1) Collect >70% of possible WFIs for Care Coordinators, Caregivers, Youth & Team Members; 2) Collect 1 TOM per care coordinator/6 months
*
*
* Fiscal Year 09 Additions to Data Collection
![Page 71: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/71.jpg)
Maryland Wraparound Sites:Fidelity By Respondent Statewide
87 89
6970
77 7579 78
73
89
80 7881
8580
0
10
20
30
40
50
60
70
80
90
100
CombinedRespondents
CareCoordinators
Care Givers Team Members Youth
Per
cent
Fid
elity
BaltimoreCity
MontgomeryCounty
WicomicoCounty
![Page 72: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/72.jpg)
Use of WFAS Data at the Local Level (from Zabel & Mettrick, 2008)
Provides standards of quality for Wraparound implementation
Encourages Best Practices for Care Coordination & Family Support
Encourages Families to “be part of the solution” and reinforces Family Voice
Directly impacts training and coaching strategies Influences procedural change within Care
Management Entities
![Page 73: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/73.jpg)
Use of WFAS Data at the State Level (from Zabel & Mettrick, 2008)
State and Local Jurisdiction Reports to the Governor's Office for Children (bi-annually)
Supports Wraparound Model to the State Legislature Helped the state obtain approval for the 1915c
Medicaid Demonstration Waiver (1 of nine states) Helped to ensure family peer to peer support was
included in the 1915c waiver
![Page 74: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/74.jpg)
Wraparound Fidelity Index ScoresBenchmarks and real-world reality
50
60
70
80
90
Pe
rce
nt
of
tota
l fi
de
lity
Non-wraparoundcomparisons
58
State No.1 64
State No.2 69
National Average 75
Maryland 80
"High fidelity" 85
Total WFI scores
![Page 75: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/75.jpg)
Getting to “high fidelity”Maryland vs. 2 comparison states
Statewide training and TA center Consistent availability of family partners (+ youth advocates) Certification program for facilitators/FPs Referrals from and fiscal responsibility shared by multiple
agencies Care management entity (CME) that maintains MIS,
develops service array, holds some risk for overall costs Allows for flexible funding of team strategies
1915b Waiver Professional development at SSW and in provider agencies
![Page 76: Using the Wraparound Fidelity Assessment System to Support High-Quality Wraparound Overview and Lessons Learned from California Development Teams Eric.](https://reader030.fdocuments.in/reader030/viewer/2022032707/56649e1a5503460f94b07575/html5/thumbnails/76.jpg)
Fidelity’s Impact on Outcomes at a state level?
0
20
40
60
80
100
120
140
Intake 6 months 12 monthsAv
era
ge
Fu
nc
tio
na
l Im
pa
irm
en
t o
n t
he
CA
FA
S
State 1 State 2 Maryland