Post on 06-Jan-2018
description
What the data can tell us:
Evidence, Inference, Action!
1Early Childhood Outcomes Center
2
Using data for program improvement
EvidenceInference
Action
3
Evidence
• Evidence refers to the numbers, such as“45% of children in
category b”
• The numbers are not debatable
4
Inference
• How do you interpret the #s?• What can you conclude from the #s?• Does evidence mean good news?
Bad news? News we can’t interpret?
• To reach an inference, sometimes we analyze data in other ways (ask for more evidence)
5
Inference
• Inference is debatable -- even reasonable people can reach different conclusions from the same set of numbers
• Stakeholder involvement can be helpful in making sense of the evidence
6
Action
• Given the inference from the numbers, what should be done?
• Recommendations or action steps
• Action can be debatable – and often is
• Another role for stakeholders
7
Data Quality Checks
• Missing Data• Pattern Checking
8
Missing Data - Overall
• How many children should the state be reporting to OSEP in the SPP/APR table?– i.e., how many children [had entry data,] exited in the year, and
stayed in the program 6 months?– Do you have a way to know?
• What percentage of those children do you have in the table?
• These questions apply whether or not you are sampling.
9
Are you missing data selectively?
• By local program• By child characteristic
– Disability?– Type of exit? (children who exit before 3)
• By family characteristic– Families who are hard to reach (and may leave
unexpectedly)
***Which of these can you check on?***
10
Pattern Checking
Checking outcome data for quality: Looking for patterns
(see pattern checking document)
11
Looking for Sensible Patterns in the Data
• Putting together your “validity argument.”• You can make a case that your data are valid
if …..they show certain patterns.• The quality of your data is not established by
one or two numbers. • The quality of the data is established by a
series of analyses that demonstrate the data are showing predictable patterns.
12
Drilling down: Looking at data by local program
• All analyses that can be run with the state data can be run with the local data
• The same patterns should hold and the same predictions apply.
• Need to be careful about the size of N with small programs.
13
If Data are High Quality
Assuming: pattern checking and review of individual child records/data show high quality data
Now what?
Explaining variation
Who has good outcomes = Do outcomes vary by
• Region of the state?• Amount of services received?• Type of services received?• Age at entry to service?• Level of functioning at entry?• Family outcomes?• Education level of parent?
14
15
Data Analysis to identify where differences exist but shouldn’t
Data Analysis to identify where program improvements might be made
16
Analysis by Local Area
If local areas are serving similar kinds of children, progress data should be similar.
Analysis: Distributions by local area
Are children making more/less progress in some local programs? Why?
17
Analysis by Outcome
Analysis: Distribution across the outcomes
Are children making progress in some outcome areas more than others? Why?
18
Analysis by certain characteristics
Progress data should not be related to certain characteristics (e.g., race/ ethnicity, primary language, gender).
Analyses: Distributions by characteristics
Are children with some characteristics making more progress than others? Why?