CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley CFSR2 Data Indicators: The...

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

CFSR2 Data Indicators:The Good, the Bad, and the Ugly

Center for Social Services ResearchUniversity of California at Berkeley

March 2008

The Performance Indicators Project at CSSR is supported by the California Department of Social Services and the Stuart Foundation


Overview of Key Changes…

• Move from individual measures to data composites for evaluating permanency (no such change for evaluating safety)

• Adjustment of National Standards based on most recent data available

• Measures reversed (where necessary) so that higher score equals better performance (easier to interpret)

• Some changes in time periods for given measures


CFSR: Seven Outcomes

Safety• Children are, first and foremost, protected from abuse and neglect.• Children are safely maintained in their homes whenever possible and

appropriate.

Permanency• Children have permanency and stability in their living arrangements.• The continuity of family relationships and connections is preserved for

children.

Child and Family Well-Being• Families have enhanced capacity to provide for their children's needs.• Children receive appropriate services to meet their educational needs. • Children receive adequate services to meet their physical and mental

health needs.


CFSR: Seven Systemic Factors

• Statewide information system• Case review system• Quality assurance program• Staff and provider training• Service array• Agency responsiveness to the community• Foster and adoptive parent licensing, recruitment

and retention


CA CWS Outcomes System

• Round 1 of the CFSRs– 2 of the “outcomes” = 6 items (2 for safety, 4 for

permanency)– National Standards attached: based on the 75th %tile of

reporting states – States failing to meet a given standard had to include that

item in their Performance Improvement Plans (PIPs)

• Round 2 of the CFSRs– Also comprised of 6 items with standards attached– BUT…this time the permanency standards are comprised of

15 different measures distilled into four composites– TOTAL of 17 FEDERAL MEASURES


Safety

Indicator 1

Indicator 2

Permanency

Composite 1

Composite 2

Composite 3

Composite 4

Component A

Component B

Component A

Component B

Component CComponent A

Component B

Measure 1

Measure 2

Measure 3

Measure 4Measure 1

Measure 2

Measure 3

Measure 4

Measure 5

Measure 1

Measure 2

Measure 3Measure 1 Measure 2 Measure 3


Safety

S1.1

S2.1

Permanency

Composite 1:

Reunification

Composite 2: Adoption

Composite 3: Long-

Term

Composite 4:

Placement

C1.1 C1.2 C1.3 C1.4

C2.1 C2.2 C2.3 C2.4

C2.5

C3.1 C3.2

C3.3

C4.1 C4.2 C4.3


National Standards

• National standards for both the safety indicators and permanency composites are based on State performance in 2004, 75th percentile

• In California, we at CSSR will attempt to replicate each of the measures and composite scores, break them out by child welfare and probation agencies, and report/update quarterly.

• Although national standards have been set for the composites rather than individual measures…– The goal is to improve State performance on all measures

(every improvement reflects a better outcome for children)– Improvement on any given measure will result in an increase in

the overall composite score

• Analogous to Academic Achievement Test Scoring…


Evaluating Safety:

S1.1(Safety Indicator 1)

Nat’l Std=94.6%

Of all children who were victims of a substantiated maltreatment allegation during the first 6 months of a year, what percent were not victims of another substantiated maltreatment allegation within the next 6-month period?

Of all children served in foster care during the year, what percent were not victims of a substantiated maltreatment allegation by a foster parent or facility staff member?

S2.1(Safety Indicator 2)

Nat’l Std=99.68%


Permanency Indicators

Reunification(Composite 1)

Nat’l Std=122.6

Adoption(Composite 2)

Nat’l Std=106.4

Long Term Care (Composite 3)

Nat’l Std=121.7

Placement Stability(Composite 4)

Nat’l Std=101.5

Evaluating Permanency:


Principal Components Analysis (PCA)(the “black box” version)

black box of fancy

statistical tools

Timeliness of Reunification

Timeliness of AdoptionPermanency of

ReunificationPlacement Stability

Median Time in CareRecurrence of Maltreatment

Abuse in Foster Care

Emancipating from Care Component #1Component #2

Component #3

A bunch of measures…

Three components based on related

measures!


Z-Scores?

• Before dumping all of the measures into the PCA “Black Box”, they were transformed into standard scores (z-scores)

• A z-score serves two purposes:Puts measures in the

same “range”Sets measures to the

same “system”


And an Example…

– A researcher interested in measuring “success” in high school.

– Collects the following measures for each student:• Athletic Ability• Good Grades • Physical Attractiveness• Interest in Sports• Chess Club Membership • Science Club Membership• Social Life

Principal Components Analysis…


Interest in Sports Athletic Ability

Good Grades Chess Club Member Science Club Member

Physical Attractiveness Active Social Life

Reduces the number of individual measures:

VERY HIGHLY ASSOCIATED!!

Explores the contribution of each part to the whole:

Jock Component

=

Brainiac

Component =

Popular Kids

Component =

Structures the data into independent components:

Athletic Ability

Interest in Sports

Good Grades

Chess Club Member

Physical Attractiveness

Active Social Life







Composite 1:Timeliness and Permanency of Reunification

C1.2

C1.3

C1.4

Reunification within 12 months (exit cohort)

Median time to reunification (exit cohort)

Reunification within 12 months (entry cohort)

Reentry following reunification

C1.1

Composite 1:Timeliness and Permanency of Reunification






Timeliness(Component A)

Permanency(Component B)







Composite 2:Timeliness of Adoption







Timeliness(Component A)

Progress(Component B)

“Legally Free”(Component C)

Composite 2:Timeliness of Adoption

C2.1

C2.2

C2.4

C2.5

Adoption within 24 months (exit cohort)

Median time to adoption (exit cohort)

Adoption Within 12 Months (17 Months In Care)

Legally Free Within 6 Months (17 Months In Care)

Adoption Within 12 Months (Legally Free)

C2.3







Composite 3:Permanency for Children in Long Term Care







Permanency(Component A)

“Growing-up”(Component B)

Composite 3:Permanency for Children in Long Term Care

C3.1

C3.3

C3.2 Exits To Permanency (Legally Free At Exit)

Exits To Permanency (24 Months In Care)

In Care 3 Years Or Longer (Emancipated or Age 18 In Care)







Composite 4:Placement Stability







Composite 4:Placement Stability

C4.1

C4.2

C4.3

Placement Stability (8 Days To 12 Months In Care)

Placement Stability (12 To 24 Months In Care)

Placement Stability (At Least 24 Months In Care)

Std Measure1 x Weight (0.462)




Std Measure1 x Weight (0.085)Std Measure2 x Weight (0.070)

Std Measure3 x Weight (-0.005)


Component A Timeliness of

Reunification (Sum of Weighted Measures)

Component B Permanency of

Reunification (Sum of Weighted Measures)

(Un-Weighted) County Composite Score (Component A + Component B / 2)

(Weighted) County Composite Score (# of Children in Foster Care x Score)

(Un-Scaled) State Composite Score(County weighted scores summed and divided by total # of children served)

Timeliness & Permanency of Reunification

(Scaled) State Composite Score(Transformed to position on national 50-150 scale)


What else is new?

• Move from static to dynamic website

• Possible to still review change over time

• 75th percentile for measures posted (national goal)

• Several 636 Measures have been dropped– ‘Old’ to ‘New’ Document (posted with county spreadsheets)


CSSR.BERKELEY.EDU/UCB_CHILDWELFARE

Needell, B., Webster, D., Armijo, M., Lee, S., Cuccaro-Alamin, S., Shaw, T., Dawson, W., Piccus, W., Magruder, J., Exel, M., Conley, A., Smith, J. , Dunn, A., Frerer, K., & Putnam Hornstein, E., (2007). Child Welfare Services Reports for California. Retrieved [month day, year], from University of California at Berkeley Center for Social Services Research website. URL: <http://cssr.berkeley.edu/UCB_CHILDWELFARE/>

Emily [email protected]

Barbara [email protected]

CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley CFSR2 Data Indicators: The...

Documents

Transcript of CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley CFSR2 Data Indicators: The...