Using Publicly Available Data to Engage IV-E Students in Research and Statistics: Instructional Modules

MODULE 1 SLIDE DECK:CHILD WELFARE DATA 101

2Module 1: Child Welfare Data 101 2

Instructor Notes about this module This module serves as an entry point for the

entire course, it’s purpose is to: Help reduce student anxiety and resistance towards

engaging in research and completing the course Provide an overview of California's child welfare data Introduce basic statistical concepts that are

informative, accessible, and relatively easy to generate

Increase students’ comfort level with data, thereby instilling confidence in their capacity as informed consumers of data and research

3Module 1: Child Welfare Data 101 3

Agenda Introduction and purpose/goals of the course Data basics:

Overview of sources and uses of data Sampled vs. population-level data Aggregate vs. micro-level data Longitudinal vs. cross-sectional data

Introduction to accessing and using California’s child welfare data

Understanding and computing basic descriptive statistics with child welfare data (e.g. variable descriptions, measures of central tendency, computing percentages)

Engaging Students in Research, possible slides for instructor use to introduce course

Opening Slides

5Module 1: Child Welfare Data 101 5

The Problem MSW students rank research courses as one

of their least favorite classes in the program (anecdotal…) Coursework feels disconnected from practice Most students do not enter with strong math or

stats backgrounds – high anxiety Timeline allows only a superficial coverage of

analytic methods Need to develop “statistical literacy”

Knowledge which enables people to think for themselves, judge independently, and discriminate between good and bad information (Dewey, 1930)

6Module 1: Child Welfare Data 101 6

Competing Course Models

informed consumers of data & research

junior social scientists

difficult (if not impossible) to do well even successful training is lost as not used in post-grad work

technology = data are everywhere!field is increasingly oriented around continuous improvement, outcomesstatistical literacy & critical thinking necessary for EBP and EIP

burdens agencies tasked with helping students access data

7Module 1: Child Welfare Data 101 7

Why we must be skilled consumers of research… Ethical obligation to our clients to be up to date on

the most recent research and skilled in its critical appraisal Cannot only rely on researchers to tell us the

relevant results and findings Necessary for effective advocacy and practice Helps with the efficient translation of research to

practice Critical that practitioners/researchers can speak

the same “language” in order to ensure future research efforts are relevant and that findings are understood in context and translated appropriately

8Module 1: Child Welfare Data 101 8

Managing by DataProvides social workers with the ability to: Compare metrics with agency mission

and practice model Connect to evidence-based practice and

desired outcomes Strategize on what work needs to be

done Focus on what is being achieved Identify what needs attention

9Module 1: Child Welfare Data 101 9

Connecting Data to Practice

Observe

Explain

Strategy

Outcome

We have noted that…

And believe

it is becaus

e..

So we plan to…

Which will

result in…

HYPOTHESIS: A HIGH LEVEL CAUSE AND EFFECT STATEMENT

IN OTHER WORDS…

Slide Developed by NY OCFS

1010

This Course One of the most important you will take here at X

(in my humble opinion) Focuses on practical skills, understanding data,

statistical literacy, consuming research Very connected to your current field placement –

and work post-graduation – yet helps you acquire empirical skills you may not otherwise have the opportunity to develop outside of your graduate studies

Expects that you read, ask questions, think critically, and engage with the material

Requires that you produce a relevant, readable, empirical research report based on publicly available administrative (secondary) child welfare data

11Module 1: Child Welfare Data 101 11

Administrative Data

Collected during the normal course of agency operations

Tabulated/aggregate data are publicly available Full coverage of populations served Free of “reactivity”(data problems are usually

transparent) Analysis of trends over time Performance indicators Social indicators Particularly salient to social work…

12Module 1: Child Welfare Data 101 12

Why a Secondary Analysis of Administrative Data? Data directly support agencies and capture information for

the clients we are working with Increasing emphasis on making these data available to

researchers Technological advances – data storage, frequent refreshes,

web-based/online access and analysis tools Non-intrusive (we work with vulnerable populations and

busy co-workers!) More transferrable to a post-graduate career in social work

Efficient, cheap, available Useful for advocacy efforts, needs assessment, proposals for

new programs (substantiate a service need) These are needed skills in both public and private agencies

13Module 1: Child Welfare Data 101 13

Realities of Administrative Data Analysis

Measures are often crude Sometimes limited documentation Missing data Do you trust what has been entered? Often much more difficult to analyze

than expected…requires careful thinking May be dated Definitions may have changed over time

Sources and Uses of Data

Module 1, Section 1

15Module 1: Child Welfare Data 101 15

Data Sources1. Census data2. Longitudinal surveys of a subsample of

a population3. Cross sectional surveys4. Longitudinal (multi-wave) surveys of a

single sample5. Administrative data6. Multisource data systems

1616

Uses of Data Descriptive

Demographic characteristics of a population, place, office, etc. Trends over time (one period compared to another) Differences/similarities between groups, counties, placement

settings, interventions, etc. Exploratory

Often conducted as pilot studies, attempt to examine feasibility issues (e.g., recruitment), preliminary data to develop fuller hypotheses and research proposals

Explanatory Analysis of the relationship between two events (or two

variables) Looking at the contributions of various factors to some outcomes

(y=a+bX) Evaluation

To evaluate social policies, programs, and interventions The evaluation process encompasses all three uses of data listed

above

17Module 1: Child Welfare Data 101 17

Data Terminology (Review) Qualitative vs. Quantitative

why, how vs. who, what, where, when “All quantitative data is based upon qualitative judgments; and all

qualitative data can be described and manipulated numerically.” (Research Methods Knowledge Base)

Longitudinal vs. Cross-Sectional repeated observations over time vs. a slice

Primary vs. Secondary you collect it vs. someone else collected it

Aggregate vs. Micro-level group tabulations vs. individual units

Deidentified vs. Identified Joe Smith vs. 987334

Sample vs. Census/Population partial vs. full coverage

Sample vs. Census/Population

Module1, Section 2

19Module 1: Child Welfare Data 101 19

Samples vs. Population Can we select a few people or things

for observation and then apply what we observe to a much larger group? Often impractical to gather data from the whole

population, so samples are drawn (this is not relevant to the administrative data we will be using in this course)

Key is the researcher’s ability to generalize findings from the sample to the whole population

Sample=a finite part of a larger population whose properties are studied to gain information about the whole

If all members of a population were identical in all respects – would we need careful sampling procedures?

population

sample

20

Sampling The act, process, or technique of selecting a suitable

sample, or a representative part of a population for determining the parameters (or characteristics) of the whole population

For a sample to provide useful information, it must reflect the same general variations as the overall population

NSCAW data = sample; CWS/CMS data = populationpopulation sample

Use characteristics/observations of sample, to draw conclusions (inferences) about the larger

population

Module 1: Child Welfare Data 101

Aggregate vs. Microdata

Module 1, Section 3

2222

Aggregated data

2323

Micro-data (individuals as units of analysis)

24Module 1: Child Welfare Data 101 24

Working with Aggregated Data…Disaggregate One of the most powerful ways to work with

data… Disaggregation involves dismantling or

separating out groups within a population to better understand the dynamics

Useful for identifying critical issues that were previously undetectedAggregate Permanency Outcomes

Race/Ethnicity

AgeCounty

Placement Type

25Module 1: Child Welfare Data 101 25

The Problem with Summary Statistics:

The average human has one breast and one testicle. *

* ~Des McHale www.quotegarden.com/statistics.html

2000 July-December First Entries California:

Percent Exited to Permanency 132 Months From Entry, by race and placement

Module 1, Section 4Longitudinal vs. Cross-Sectional (Point in Time) Views of Data

30Module 1: Child Welfare Data 101 30

Time Dynamics Cross-Sectional Studies

Examines a phenomenon by collecting/examining a “cross-section” of data at one time (one observation at a point in time) BIG problem: many questions we seek to answer aim to

understand causal processes that occur over time (e.g., children in foster care and mental health)

Longitudinal Studies Based on repeated observations of a given unit over

multiple points in time Trend Studies Cohort /Panel Studies

31Module 1: Child Welfare Data 101 31

Longitudinal Data1988 2006Birth 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 Exit

X X X X X X X X X XX X X X X X X X X X X X X

X X X X X X X X X X X X X X X X

Age

Longitudinal Analysis

32Module 1: Child Welfare Data 101 32

3 Key Data Views in Child Welfare

Data

Entry

Cohorts

Exit Cohorts

Point in

Time

33

January 1, 2005 January 1, 2006July 1, 2005Child 1

Child 2

Child 3

Child 4

Child 5

Child 6

Child 7

Child 8

Child 9Child 10

The “View” Matters!

34

Longitudinal vs. Point in Time

Module 1: Child Welfare Data 101

35Module 1: Child Welfare Data 101

36Module 1: Child Welfare Data 101

California’s Child Welfare Data

Module 1, Section 5

38Module 1: Child Welfare Data 101 38

Why do we have these data? In 2001, the California Legislature passed the

Child Welfare System Improvement and Accountability Act (AB 636)

Designed to improve outcomes for children in the child welfare system while holding county and state agencies accountable for the outcomes achieved

The statewide accountability system went into effect January 1, 2004 It is an enhanced version of the federal oversight

system mandated by Congress to monitor states’ performance and provides the legal framework for the California Child and Family Services Reviews

39Module 1: Child Welfare Data 101 39

How are these data used? The foundation for this new oversight system comes

from data obtained from the Child Welfare Services/Case Management System (CWS/CMS)

Each quarter, the state provides county child welfare agencies with county-specific data on 14 outcome measures related to safety, permanency and well-being The baseline performance data is gathered for each county

and also made available to the public Quarterly reports provide counties with quantitative data

and serve as a management tool to track performance over time

Public data is also refreshed quarterly (data available on website is 3-6 months old)

4040

Where are the Data from?

4141

How are the public data configured?

unique characteristics and path through the child welfare system

personal info

aggregate data

tabulations

ageracegenderallegationplacementdispositioncountyyear

42Module 1: Child Welfare Data 101 42

What “type” of information is available?

first allegation of

maltreatment

allegation evaluated out

second allegation of

maltreatment

allegation substantiated

pre-placement family

maintenance services provided

child placed in out-of-home foster care

child reunified

third allegation of

maltreatment

child re-enters foster

care

43

Where can I find these data?

http://cssr.berkeley.edu/ucb_childwelfare

Univariate Statistics, child welfare examples

Module 1, Section 6

45

Variablesvariable

categorical

nominal

ordinal

continuous

exhaustive

mutually exclusive

variability in magnitude

(quantitative in nature)

finite number of values or categories (qualitative in nature)

46Module 1: Child Welfare Data 101

Continuous vs. Categorical The average foster child has 2.6 placements while in

foster care This number makes little sense because the underlying

dimension is discrete (i.e., categorical, discontinuous) 1 2 4 5 6

placements

x2.6 3

Continuous Data Discrete DataAge Days in Care Percentages / Rates

Race/Ethnicity Placement Type Referral Reason

47Module 1: Child Welfare Data 101 47

Descriptive Statistics “Summary” statistics, used to describe what’s

“going on” in our data Describe a situation or condition numerically

by quantifying phenomena In California, there are far fewer children in foster care today

than was true a decade ago. In California, the number of children in foster care today is

47,729 less than was true a decade ago, translating into a 49% decline.

Frequency tables (the distribution), measures of central tendency, measures of variability (the dispersion)

48

Computing a Percent

PERCENT: A proportion in relation to a whole expressed as a fraction of 100.

100totalpart100)(per percent %

100totalpart

100440290

100659.0

%9.65

100reunified # total

12m w/in reunified #

Raw Numbers (counts)

# Reunified w/in 12m# Reunified (total)

= 290= 440

What Percentage of Children reunified in 2005 were reunified within 12 months of entering care?

49

Computing a Rate per 1,000RATE: A proportion in relation to a whole, can be expressed as a fraction of 100, 1000, 100,000, etc.

1000totalpart

1000363,3761,333

100000366.

7.3

1000population child #

care entered #

Raw Numbers (counts)

# Entered Care

# Child Population

= 1,333= 363,376

What was the foster care entry rate in 2005? (i.e., how many children entered care out of all possible children in the population?)

1000totalpart1000per rate

Scales for a meaningful interpretation…

5050

Measures of Central TendencyMean: the average value for a range of data Median: the value of the middle item when the data

are arranged from smallest to largestMode: the value that occurs most frequently within the data

4.168

631715129744 Mean

5.102129 Median

4 Mode

7

= 9

= 9.7

12 4 15 63 7 9 4 17 4 4 7 9 12 15 17 63

51

Measures of Variability

Minimum: the smallest value within the dataMaximum: the largest value within the dataRange: the overall span of the data

4 Minimum

63 Maximum

59463 Range

4 4 7 9 12 15 17 63

51

52Module 1: Child Welfare Data 101 52

The Relationship between Mean and Variability Standard Deviation (represented by the

symbol σ) shows how much variation there is from the mean (average or expected value) Low standard deviation indicates that most

of the data points are close to the mean (less variation)

High standard deviation indicates that data points are spread out over a large range of values (more variation)

53Module 1: Child Welfare Data 101

Standard Deviation

symbol for standard deviation

Sum the difference between each “score” (xi ) and the overall mean (m)

Square the sum

Divide by the count of observations/scores minus 1

Take the square root

54Module 1: Child Welfare Data 101

Standard DeviationScores/Observations: 4 4 7 9 12 15 17 63Mean: 16.375

1. Find the distance between each value and the mean• 4-16.375 = -12.375• 4-16.375 = -12.375• 7-16.375 = -9.375• 9-16.375 = -7.375• 12-16.375 = -4.375• 15-16.375 = -1.375• 17-16.375 = 0.625• 63-16.375 = 46.625

2. Square all Values• -12.375 x -12.375 =

153.1• -12.375 x -12.375 =

153.1• -9.375 x -9.375 = 87.9• -7.375 x -7.375 = 54.4• -4.375 x -4.375 = 19.1• -1.375 x -1.375 = 1.89• 0.625 x 0.625 = 0.39• 46.625 x 46.625 =

2173.9

55Module 1: Child Welfare Data 101

Standard DeviationScores/Observations: 4 4 7 9 12 15 17 63Mean: 16.375

3. Sum the squared values (Sum of Squares: SS)

153.1+153.1+87.9+54.4+ 19.1+1.89+0.39+2173.9 = 2643.9

4. Divide the SS by the count of scores minus 1 (this gives you the Variance)

= 2643.9 / 7= 377.574. Take the square root of the variance (this gives you the

standard deviation)= = 19.4

56Module 1: Child Welfare Data 101 56

Percentage Points vs. Percent Change

Percentage point difference Absolute increase or decrease from one percentage value

to another (calculated by addition or subtraction) The percentage of children in foster family agency (FFA)

care increased by 12 percentage points between 1998 and 2012, from 15% to 27%

Percent change Relative change from one value to another (as a fraction

of the original amount) The proportion of children in foster family agency (FFA)

care increased by 80% between 1998 and 2012, from 15% to 27%

57Module 1: Child Welfare Data 101

Percent Change

Time Period 1

Time Period 2

10 children

12 children

10011 Period2 PeriodChange %

10011.2

1000.2

20%

1001kids 10kids 12

58Module 1: Child Welfare Data 101

Percent ChangeTime Period 1 Time Period 2

10% 12%

% %% %% %% %% %

% %% %% %% %% %

%%

100110%12%Change %

%20

59

Percent Change Calculation

Baseline Referral Rate (time period 1):

7.50100005067.963,637,9

419,488

Comparison Referral Rate (time period 2):

3.4810000483.199,988,9

706,482

Percent Change:

1001Rate Baseline

Rate Comparison

100150.748.3

%7.4100047.0

1001)-.9526(

50.7 48.3 -4.7%

12.0 10.8 -10%

Min

or D

iffer

ence

s du

e to

Ro

undi

ng…

Using Publicly Available Data to Engage IV-E Students in Research and Statistics: Instructional Modules

QUESTIONS? PLEASE CONTACT:

ehornste@usc.edubneedell@berkeley.edu