What the data can tell us:

Evidence, Inference, Action!

1Early Childhood Outcomes Center

2

Using data for program improvement

EvidenceInference

Action

3

Evidence

• Evidence refers to the numbers, such as“45% of children in

category b”

• The numbers are not debatable

4

Inference

• How do you interpret the #s?• What can you conclude from the #s?• Does evidence mean good news?

Bad news? News we can’t interpret?

• To reach an inference, sometimes we analyze data in other ways (ask for more evidence)

5

Inference

• Inference is debatable -- even reasonable people can reach different conclusions from the same set of numbers

• Stakeholder involvement can be helpful in making sense of the evidence

6

Action

• Given the inference from the numbers, what should be done?

• Recommendations or action steps

• Action can be debatable – and often is

• Another role for stakeholders

7

Data Quality Checks

• Missing Data• Pattern Checking

8

Missing Data - Overall

• How many children should the state be reporting to OSEP in the SPP/APR table?– i.e., how many children [had entry data,] exited in the year, and

stayed in the program 6 months?– Do you have a way to know?

• What percentage of those children do you have in the table?

• These questions apply whether or not you are sampling.

9

Are you missing data selectively?

• By local program• By child characteristic

– Disability?– Type of exit? (children who exit before 3)

• By family characteristic– Families who are hard to reach (and may leave

unexpectedly)

***Which of these can you check on?***

10

Pattern Checking

Checking outcome data for quality: Looking for patterns

(see pattern checking document)

11

Looking for Sensible Patterns in the Data

• Putting together your “validity argument.”• You can make a case that your data are valid

if …..they show certain patterns.• The quality of your data is not established by

one or two numbers. • The quality of the data is established by a

series of analyses that demonstrate the data are showing predictable patterns.

12

Drilling down: Looking at data by local program

• All analyses that can be run with the state data can be run with the local data

• The same patterns should hold and the same predictions apply.

• Need to be careful about the size of N with small programs.

13

If Data are High Quality

Assuming: pattern checking and review of individual child records/data show high quality data

Now what?

Explaining variation

Who has good outcomes = Do outcomes vary by

• Region of the state?• Amount of services received?• Type of services received?• Age at entry to service?• Level of functioning at entry?• Family outcomes?• Education level of parent?

14

15

Data Analysis to identify where differences exist but shouldn’t

Data Analysis to identify where program improvements might be made

16

Analysis by Local Area

If local areas are serving similar kinds of children, progress data should be similar.

Analysis: Distributions by local area

Are children making more/less progress in some local programs? Why?

17

Analysis by Outcome

Analysis: Distribution across the outcomes

Are children making progress in some outcome areas more than others? Why?

18

Analysis by certain characteristics

Progress data should not be related to certain characteristics (e.g., race/ ethnicity, primary language, gender).

Analyses: Distributions by characteristics

Are children with some characteristics making more progress than others? Why?