Using Publicly Available Data to Engage IV-E Students in Research and Statistics: Instructional Modules

MODULE 2 SLIDE DECK:LESSONS IN USING (AND MISUSING) CALIFORNIA’S CHILD WELFARE DATA

2Module 2: Lessons in Using Data 2

Instructor Notes for Module 2 This module exposes students to data

concerning California’s child welfare system, its purpose is to: Provide a broad overview of California’s child

welfare system through visual displays of data Introduce state and federal child welfare

indicators for tracking agency performance (with a more technical module for optional use)

Promote critical thinking in the context of basic statistical concepts through the review of popular press examples based on actual child welfare data

Module 2: Lessons in Using Data 3

Understanding California’s Child Welfare System through Data

Module 2, Section 1

4Module 2: Lessons in Using Data 4

The “big” picture in 2011… 9,992,333 children under the age of 18 471,790 children reported for maltreatment

(47.2 per 1,000 children) 90,472 children with a substantiated

allegation(9.1 per 1,000 children)

31,431 children entered foster care(3.1 per 1,000 children)

On any given day, roughly 59,484 children in foster care

(6.0 per 1,000 children)

5Module 2: Lessons in Using Data 5

The Iceberg Analogy

Maltreated children not known to child protective services

Maltreated children known to child protective services

Module 2: Lessons in Using Data 6

Module 2: Lessons in Using Data 7

Module 2: Lessons in Using Data 8

Module 2: Lessons in Using Data 9

Module 2: Lessons in Using Data 10

Module 2: Lessons in Using Data 11

Module 2: Lessons in Using Data 12

Module 2: Lessons in Using Data 13

Module 2: Lessons in Using Data 14

Module 2: Lessons in Using Data 15

Tracking child welfare performance through federal and state outcome measures

Module 2, Section 2

16Module 2: Lessons in Using Data 16

Trends over the last two decades Increased (and improved) data collection Increased emphasis on accountability

Observed across government agencies Shift from measuring processes, to

performance outcomes What matters is where you end-up…

promotes innovation But what “outcomes” should we measure?

And how can we best “measure” these outcomes?

1717

Lesson #1: Any One Measure Will Not be Enough…

counterbalancedindicators of system

performance

permanencythrough reunification,

adoption, orguardianship

lengthof stay

stability of care

rate of referrals/substantiated referrals

home-based services vs.

out of home care

positive attachments to family, friends, and neighbors

use of leastrestrictive

form of care

Slide Source: Usher, C.L., Wildfire, J.B., Gogan, H.C. & Brown, E.L. (2002). Measuring Outcomes in Child Welfare. Chapel Hill: Jordan Institute for Families,

reentry to care

1818

Federal and State Outcome Measures

Federal Child and Family Services Review (CFSR)

State Accountability Act AB 636

Went into effect in California on January 1, 2004. This new system holds the state and counties

accountable for improving outcomes for children through the establishment of improvement goals, public reporting of outcomes and county-specific improvement plans that must be approved by county boards of supervisors and submitted to the state

No goals or standards. Rather, objective is continuous, quality improvement within each county.

19Module 2: Lessons in Using Data 19

Lesson #2: Measuring Outcomes Can Get Complicated (quickly)…

20Module 2: Lessons in Using Data 20

What you will find reported for California

2121

Website view example…

Reunification composite

Module 2: Lessons in Using Data 22

Optional/additional performance outcome information for instructor use

Module 2, Section 2.1

23Module 2: Lessons in Using Data 23

Children and Families Service Reviews(more details than most will want, but truly useful to understand!)

Federal Child and Families Service Reviews (CFSR) Transition from individual “measures” to safety

indicators and composite measures or permanency and stability

National standards for both the indicators and composites are based on the 75th percentile of state performance in 2004

Although national standards have been set for the composites rather than individual measures… The goal is to improve State performance on all

measures (every improvement reflects a better outcome for children)

Improvement on any given measure will result in an increase in the overall composite score

Analogous to Academic Achievement Test Scoring…

24Module 2: Lessons in Using Data 24

Principal Components Analysis (PCA)(the “black box” version)

black box of fancy statistical tools

Timeliness of Reunification

Timeliness of AdoptionPermanency of

ReunificationPlacement Stability

Median Time in CareRecurrence of Maltreatment

Abuse in Foster Care

Emancipating from Care Component #1Component #2Component #3

A bunch of measures… Three components based on related measures!

25

Z-Scores? Before dumping all of the measures into the PCA “Black

Box”, they were transformed into standard scores (z-scores)

A z-score serves two purposes:Puts measures in the same “range”

Sets measures to the same “system”

2626

And an Example… A researcher interested in measuring

“success” in high school. Collects the following measures for each

student:Athletic AbilityGood Grades Physical AttractivenessInterest in SportsChess Club Membership Science Club MembershipSocial LifePrincipal Components Analysis…

Module 2: Lessons in Using Data 27

Interest in Sports Athletic Ability

Good Grades Chess Club Member Science Club Member

Physical Attractiveness Active Social Life

Reduces the number of individual measures:

VERY HIGHLY ASSOCIATED!!

Explores the contribution of each part to the whole:

Jock Component =

Brainiac Component =

Popular Kids Component =

Structures the data into independent components: Athletic

AbilityInterest in Sports

Good Grades

Chess Club Member

Physical Attractiveness

Active Social Life

28Module 2: Lessons in Using Data 28

Measure Contributions to Composites

C1.1 (22%)

C1.2 (21%)

C1.3 (12%)

C1.4 (46%)

0%

20%

40%

60%

80%

100%

Composite 1

Reunification Within 12 Months (Exit Cohort)

Median Time To Reunification (Exit Cohort)

Reentry Following Reunification (Exit Cohort)

Reunification Within 12 Months (Entry Cohort)

Note: Measures may not sum to exactly 100% due to rounding.

29Module 2: Lessons in Using Data 29

C2.1 (15%)

C2.2 (19%)

C2.3 (22%)

C2.4 (18%)

C2.5 (26%)

0%

20%

40%

60%

80%

100%

Composite 2

Adoption Within 24 Months (Exit Cohort)

Median Time To Adoption (Exit Cohort)

Adoption Within 12 Months (17 Months In Care)

Legally Free Within 6 Months (17 Months In Care)

Adoption Within 12 Months (Legally Free)

Note: Measures may not sum to exactly 100% due to rounding.

Measure Contributions to Composites

30Module 2: Lessons in Using Data 30

C3.1 (33%)

C3.2 (25%)

C3.3 (42%)

0%

20%

40%

60%

80%

100%

Composite 3

Exits to Permanency (24 Months In Care)

Exits to Permanency (Legally Free At Exit)

In Care 3 Years Or Longer (Emancipated/Age 18)

Note: Measures may not sum to exactly 100% due to rounding.

Measure Contributions to Composites

31Module 2: Lessons in Using Data 31

C4.1 (33%)

C4.2 (34%)

C4.3 (33%)

0%

20%

40%

60%

80%

100%

Composite 4

Placement Stability (8 Days To 12 Months In Care)

Placement Stability (12 To 24 Months In Care)

Placement Stability (At Least 24 Months In Care)

Note: Measures may not sum to exactly 100% due to rounding.

Measure Contributions to Composites

32Module 2: Lessons in Using Data 32

C4.1 (33%)C3.1 (33%)

C2.1 (15%)C1.1 (22%)

C4.2 (34%)C3.2 (25%)

C2.2 (19%)C1.2 (21%)

C4.3 (33%)C3.3 (42%)

C2.3 (22%)C1.3 (12%)

C2.4 (18%)

C1.4 (46%)

C2.5 (26%)

0%

20%

40%

60%

80%

100%

Composite 1 Composite 2 Composite 3 Composite 4Note: Measures may not sum to exactly 100% due to rounding.

Measure Contributions to Composites

Module 2: Lessons in Using Data 33

Module 2: Lessons in Using Data 34

Module 2: Lessons in Using Data 35

Module 2: Lessons in Using Data 36

Popular press examples of data use/misuse (aka, numbers gone wild)

Module 2, Section 3

37Module 2: Lessons in Using Data 37

Public Data: Putting It All Out There

PROS: Greater performance accountability Community awareness and involvement,

encourages public-private partnerships Ability to track improvement over time,

identify areas where programmatic adjustments are needed

County/county and county/state collaboration

Transparency Dialogue

38Module 2: Lessons in Using Data 38

Public Data: Putting It All Out There

CONS: Potential for misuse, misinterpretation,

and misrepresentation Available to those with agendas or looking

to create a sensational headline Misunderstood data can lead to the wrong

policy decisions “Torture numbers, and they’ll confess

to anything”(Gregg Easterbrook)

Module 2: Lessons in Using Data 39

There are three kinds of lies:

Lies, Damned Lies and StatisticsMisused

Statistics

^

40Module 2: Lessons in Using Data 40

1)Compare Apples and Oranges2)Use ‘snapshots’ of Small Samples3)Rely on Unrepresentative Findings4)Logically ‘flip’ Statistics 5)Falsely Assume an Association to be

Causal

Five Ways to Misuse Data (without actually lying!):

41Module 2: Lessons in Using Data 41

Two doctors in Anytown, CA… Doctor #1 Doctor #2

What if the populations served by each doctor were very different?

2/1000 20/1000

1) Compare Apples and Oranges

Doctor of the Year?

Module 2: Lessons in Using Data 42

“Foster Children in Fresno County are three times more likely to remain in foster care for more than a year than in Sacramento.”

SF Chronicle, “Accidents of Geography”, March 8, 2006

43

Different families and children served?

Different related outcomes?First entry rates in Fresno are consistently lower

Re-entries in Fresnoare also lower…

“Foster Children in Fresno County are three times more likely to remain in foster care for more than a year than in Sacramento.”

44Module 2: Lessons in Using Data 44

Number of Crimes Period 1: 76Period 2: 51Period 3: 91Period 4: 76

Crime jumped by 49%!!No change.Crime dropped by 16%Average = 73.5

Crime in Anytown, CA…

2) Data Snapshots…

Module 2: Lessons in Using Data 45

“A foster child living in Napa County is in greater danger of being abused in foster care than anywhere else in the Bay area...”

SF Chronicle, “No refuge. For foster youth, it’s a state of chance”, November 15, 2005

46Module 2: Lessons in Using Data 46

Abuse in Care RatePeriod 1: 1.80% Period 2: 1.64% Period 3: 0.84% Period 4: 0.00%

Responsible use of the data prevents us from making any of these claims (positive or

negative). The sample is too small; the time frame too limited.

100% improvement! 0 Children Abused!

= 2/111

= 0

= 2/122= 1/119

“A foster child living in Napa County is in greater danger of being abused in foster care than anywhere else in the Bay Area…”

47Module 2: Lessons in Using Data 47

Survey of people in Anytown, CA…

90% of respondents stated that they support using tax dollars to build a new football stadium.

The implication of the above finding is that there is overwhelming support for the stadium…

But what if you were then told that respondents had been sampled from a list of season football ticket

holders?

3) Unrepresentative Findings…

Module 2: Lessons in Using Data 48

“Some reports indicate that maltreatment of children in foster care is a serious problem, and in one recent large-scale study, about one-third of respondents reported maltreatment at the hands of their caregivers.”

“My Word”, Oakland Tribune, May 25, 2006

49Module 2: Lessons in Using Data

“…in one recent large-scale study, about one-third of respondents reported maltreatment at the hands of their caregivers.” Oakland Tribune

Factually true? Yes.

Misleading? Yes.

This was a survey of emancipated foster youth

Emancipated youth represent a distinct subset of the foster care population

This “accurate” statistic misleads the reader to conclude that one-third of foster children have been maltreated in care…

50Module 2: Lessons in Using Data

4) Logical “flipping”…Headline in The Anytown Chronicle:

60% of violent crimes are committed by men who did not graduate from high school.

“Flip” 60% of male high school drop-outs

commit violent crimes?

Module 2: Lessons in Using Data 51

“One study in Washington State found that 75 percent of a sample of neglect cases involved families with incomes under $10,000.”

Bath and Haapala, 1993 as cited in “Shattered bonds: The color of child welfare” by Dorothy Roberts

52Module 2: Lessons in Using Data 52

In reading statistics such as the above, there is a tendency to want to directionally “Flip” the interpretation

But the original and flipped statements have very different meanings!

75% of neglect cases involved families with incomes under $10,000

DOES NOT MEAN

75% of families with incomes under $10,000 have open neglect cases

Put more simply, just because most neglected children are poor does not mean that most poor children are neglected

“One study in Washington State found that 75 percent of a sample of neglect cases involved families with incomes under $10,000.”

53Module 2: Lessons in Using Data 53

A study of Anytown residents makes the following claim:

Adults with short hair are, on average, more than 3 inches taller than those with long hair.

Finding an association between two factors does not mean that one causes the other…

Hair Length Height

Gender

X

5) False Causality…

Module 2: Lessons in Using Data 54

“The study, conducted by researchers at the University of California at Berkeley, shows that foster children consistently scored lower in state English and math tests, even when factors such as income, race and learning disabilities were taken into account. ” As reported in USA Today, September 24, 2010

55Module 2: Lessons in Using Data 55

“Foster children struggle to learn…”

56Module 2: Lessons in Using Data

Response to Data Misuse? Continued efforts to frame the data,

educate interested media, policymakers, and others what do these findings mean? how can these data be used to gain

insight into where improvements are needed?

Agencies/child welfare workers must be proactive in discussing both the “good” and the “bad” (be first, but be right). be transparent if not playing offense…playing defense

Using Publicly Available Data to Engage IV-E Students in Research and Statistics: Instructional Modules

QUESTIONS? PLEASE CONTACT:

ehornste@usc.edubneedell@berkeley.edubrynking@berkeley.edu