Module 2 Slide deck: Lessons in using (and misusing) California’s Child Welfare data

Using Publicly Available Data to Engage IV-E Students in Research and Statistics: Instructional Modules

MODULE 2 SLIDE DECK:LESSONS IN USING (AND MISUSING) CALIFORNIA’S CHILD WELFARE DATA

2Module 2: Lessons in Using Data 2

Instructor Notes for Module 2 This module exposes students to data

concerning California’s child welfare system, its purpose is to: Provide a broad overview of California’s child

welfare system through visual displays of data Introduce state and federal child welfare

indicators for tracking agency performance (with a more technical module for optional use)

Promote critical thinking in the context of basic statistical concepts through the review of popular press examples based on actual child welfare data

Module 2: Lessons in Using Data 3

Understanding California’s Child Welfare System through Data

Module 2, Section 1


The “big” picture in 2011… 9,992,333 children under the age of 18 471,790 children reported for maltreatment

(47.2 per 1,000 children) 90,472 children with a substantiated

allegation(9.1 per 1,000 children)

31,431 children entered foster care(3.1 per 1,000 children)

On any given day, roughly 59,484 children in foster care

(6.0 per 1,000 children)


The Iceberg Analogy

Maltreated children not known to child protective services

Maltreated children known to child protective services


Tracking child welfare performance through federal and state outcome measures

Module 2, Section 2


Trends over the last two decades Increased (and improved) data collection Increased emphasis on accountability

Observed across government agencies Shift from measuring processes, to

performance outcomes What matters is where you end-up…

promotes innovation But what “outcomes” should we measure?

And how can we best “measure” these outcomes?

1717

Lesson #1: Any One Measure Will Not be Enough…

counterbalancedindicators of system

performance

permanencythrough reunification,

adoption, orguardianship

lengthof stay

stability of care

rate of referrals/substantiated referrals

home-based services vs.

out of home care

positive attachments to family, friends, and neighbors

use of leastrestrictive

form of care

Slide Source: Usher, C.L., Wildfire, J.B., Gogan, H.C. & Brown, E.L. (2002). Measuring Outcomes in Child Welfare. Chapel Hill: Jordan Institute for Families,

reentry to care

1818

Federal and State Outcome Measures

Federal Child and Family Services Review (CFSR)

State Accountability Act AB 636

Went into effect in California on January 1, 2004. This new system holds the state and counties

accountable for improving outcomes for children through the establishment of improvement goals, public reporting of outcomes and county-specific improvement plans that must be approved by county boards of supervisors and submitted to the state

No goals or standards. Rather, objective is continuous, quality improvement within each county.


Lesson #2: Measuring Outcomes Can Get Complicated (quickly)…


What you will find reported for California

2121

Website view example…

Reunification composite


Optional/additional performance outcome information for instructor use

Module 2, Section 2.1


Children and Families Service Reviews(more details than most will want, but truly useful to understand!)

Federal Child and Families Service Reviews (CFSR) Transition from individual “measures” to safety

indicators and composite measures or permanency and stability

National standards for both the indicators and composites are based on the 75th percentile of state performance in 2004

Although national standards have been set for the composites rather than individual measures… The goal is to improve State performance on all

measures (every improvement reflects a better outcome for children)

Improvement on any given measure will result in an increase in the overall composite score

Analogous to Academic Achievement Test Scoring…


Principal Components Analysis (PCA)(the “black box” version)

black box of fancy statistical tools

Timeliness of Reunification

Timeliness of AdoptionPermanency of

ReunificationPlacement Stability

Median Time in CareRecurrence of Maltreatment

Abuse in Foster Care

Emancipating from Care Component #1Component #2Component #3

A bunch of measures… Three components based on related measures!

25

Z-Scores? Before dumping all of the measures into the PCA “Black

Box”, they were transformed into standard scores (z-scores)

A z-score serves two purposes:Puts measures in the same “range”

Sets measures to the same “system”

2626

And an Example… A researcher interested in measuring

“success” in high school. Collects the following measures for each

student:Athletic AbilityGood Grades Physical AttractivenessInterest in SportsChess Club Membership Science Club MembershipSocial LifePrincipal Components Analysis…


Interest in Sports Athletic Ability

Good Grades Chess Club Member Science Club Member

Physical Attractiveness Active Social Life

Reduces the number of individual measures:

VERY HIGHLY ASSOCIATED!!

Explores the contribution of each part to the whole:

Jock Component =

Brainiac Component =

Popular Kids Component =

Structures the data into independent components: Athletic

AbilityInterest in Sports

Good Grades

Chess Club Member

Physical Attractiveness

Active Social Life


Measure Contributions to Composites

C1.1 (22%)

C1.2 (21%)

C1.3 (12%)

C1.4 (46%)

0%

20%

40%

60%

80%

100%

Composite 1

Reunification Within 12 Months (Exit Cohort)

Median Time To Reunification (Exit Cohort)

Reentry Following Reunification (Exit Cohort)

Reunification Within 12 Months (Entry Cohort)

Note: Measures may not sum to exactly 100% due to rounding.


C2.1 (15%)

C2.2 (19%)

C2.3 (22%)

C2.4 (18%)

C2.5 (26%)

0%

20%

40%

60%

80%

100%

Composite 2

Adoption Within 24 Months (Exit Cohort)

Median Time To Adoption (Exit Cohort)

Adoption Within 12 Months (17 Months In Care)

Legally Free Within 6 Months (17 Months In Care)

Adoption Within 12 Months (Legally Free)




C3.1 (33%)

C3.2 (25%)

C3.3 (42%)

0%

20%

40%

60%

80%

100%

Composite 3

Exits to Permanency (24 Months In Care)

Exits to Permanency (Legally Free At Exit)

In Care 3 Years Or Longer (Emancipated/Age 18)




C4.1 (33%)

C4.2 (34%)

C4.3 (33%)

0%

20%

40%

60%

80%

100%

Composite 4

Placement Stability (8 Days To 12 Months In Care)

Placement Stability (12 To 24 Months In Care)

Placement Stability (At Least 24 Months In Care)




C4.1 (33%)C3.1 (33%)

C2.1 (15%)C1.1 (22%)

C4.2 (34%)C3.2 (25%)

C2.2 (19%)C1.2 (21%)

C4.3 (33%)C3.3 (42%)

C2.3 (22%)C1.3 (12%)

C2.4 (18%)

C1.4 (46%)

C2.5 (26%)

0%

20%

40%

60%

80%

100%

Composite 1 Composite 2 Composite 3 Composite 4Note: Measures may not sum to exactly 100% due to rounding.



Popular press examples of data use/misuse (aka, numbers gone wild)

Module 2, Section 3


Public Data: Putting It All Out There

PROS: Greater performance accountability Community awareness and involvement,

encourages public-private partnerships Ability to track improvement over time,

identify areas where programmatic adjustments are needed

County/county and county/state collaboration

Transparency Dialogue


Public Data: Putting It All Out There

CONS: Potential for misuse, misinterpretation,

and misrepresentation Available to those with agendas or looking

to create a sensational headline Misunderstood data can lead to the wrong

policy decisions “Torture numbers, and they’ll confess

to anything”(Gregg Easterbrook)


There are three kinds of lies:

Lies, Damned Lies and StatisticsMisused

Statistics

^


1)Compare Apples and Oranges2)Use ‘snapshots’ of Small Samples3)Rely on Unrepresentative Findings4)Logically ‘flip’ Statistics 5)Falsely Assume an Association to be

Causal

Five Ways to Misuse Data (without actually lying!):


Two doctors in Anytown, CA… Doctor #1 Doctor #2

What if the populations served by each doctor were very different?

2/1000 20/1000

1) Compare Apples and Oranges

Doctor of the Year?


“Foster Children in Fresno County are three times more likely to remain in foster care for more than a year than in Sacramento.”

SF Chronicle, “Accidents of Geography”, March 8, 2006

43

Different families and children served?

Different related outcomes?First entry rates in Fresno are consistently lower

Re-entries in Fresnoare also lower…

“Foster Children in Fresno County are three times more likely to remain in foster care for more than a year than in Sacramento.”


Number of Crimes Period 1: 76Period 2: 51Period 3: 91Period 4: 76

Crime jumped by 49%!!No change.Crime dropped by 16%Average = 73.5

Crime in Anytown, CA…

2) Data Snapshots…


“A foster child living in Napa County is in greater danger of being abused in foster care than anywhere else in the Bay area...”

SF Chronicle, “No refuge. For foster youth, it’s a state of chance”, November 15, 2005


Abuse in Care RatePeriod 1: 1.80% Period 2: 1.64% Period 3: 0.84% Period 4: 0.00%

Responsible use of the data prevents us from making any of these claims (positive or

negative). The sample is too small; the time frame too limited.

100% improvement! 0 Children Abused!

= 2/111

= 0

= 2/122= 1/119

“A foster child living in Napa County is in greater danger of being abused in foster care than anywhere else in the Bay Area…”


Survey of people in Anytown, CA…

90% of respondents stated that they support using tax dollars to build a new football stadium.

The implication of the above finding is that there is overwhelming support for the stadium…

But what if you were then told that respondents had been sampled from a list of season football ticket

holders?

3) Unrepresentative Findings…


“Some reports indicate that maltreatment of children in foster care is a serious problem, and in one recent large-scale study, about one-third of respondents reported maltreatment at the hands of their caregivers.”

“My Word”, Oakland Tribune, May 25, 2006

49Module 2: Lessons in Using Data

“…in one recent large-scale study, about one-third of respondents reported maltreatment at the hands of their caregivers.” Oakland Tribune

Factually true? Yes.

Misleading? Yes.

This was a survey of emancipated foster youth

Emancipated youth represent a distinct subset of the foster care population

This “accurate” statistic misleads the reader to conclude that one-third of foster children have been maltreated in care…


4) Logical “flipping”…Headline in The Anytown Chronicle:

60% of violent crimes are committed by men who did not graduate from high school.

“Flip” 60% of male high school drop-outs

commit violent crimes?


“One study in Washington State found that 75 percent of a sample of neglect cases involved families with incomes under $10,000.”

Bath and Haapala, 1993 as cited in “Shattered bonds: The color of child welfare” by Dorothy Roberts


In reading statistics such as the above, there is a tendency to want to directionally “Flip” the interpretation

But the original and flipped statements have very different meanings!

75% of neglect cases involved families with incomes under $10,000

DOES NOT MEAN

75% of families with incomes under $10,000 have open neglect cases

Put more simply, just because most neglected children are poor does not mean that most poor children are neglected

“One study in Washington State found that 75 percent of a sample of neglect cases involved families with incomes under $10,000.”


A study of Anytown residents makes the following claim:

Adults with short hair are, on average, more than 3 inches taller than those with long hair.

Finding an association between two factors does not mean that one causes the other…

Hair Length Height

Gender

X

5) False Causality…


“The study, conducted by researchers at the University of California at Berkeley, shows that foster children consistently scored lower in state English and math tests, even when factors such as income, race and learning disabilities were taken into account. ” As reported in USA Today, September 24, 2010


“Foster children struggle to learn…”


Response to Data Misuse? Continued efforts to frame the data,

educate interested media, policymakers, and others what do these findings mean? how can these data be used to gain

insight into where improvements are needed?

Agencies/child welfare workers must be proactive in discussing both the “good” and the “bad” (be first, but be right). be transparent if not playing offense…playing defense

Using Publicly Available Data to Engage IV-E Students in Research and Statistics: Instructional Modules

QUESTIONS? PLEASE CONTACT:

[email protected]@[email protected]

mailto:[email protected]



Module 2 Slide deck: Lessons in using (and misusing) California’s Child Welfare data

Documents

Transcript of Module 2 Slide deck: Lessons in using (and misusing) California’s Child Welfare data