Welcome to the Child Outcomes Data Workshop!

Welcome to

the Child Outcomes Data Workshop!

Early Childhood Outcomes CenterEarly Childhood Outcomes Center 2

Child Outcomes Data WorkshopChild Outcomes Data Workshop

Pre-Meeting Workshop at the OSEP National Early Childhood Conference

Washington, DC December 2007


Today’s Cast:Today’s Cast:

SRI International

Kathy Hebbeler

Donna Spiker

The State of Connecticut

Alice Ridgway

The State of Minnesota

Lisa Backer

Frank Porter Graham Child Development Institute

Lynne Kahn

Robin Rooney

Christina Kasprzak

Courtney Valdes


Objectives Objectives

Participants will begin to understand:

1. How to examine the validity of state child outcomes data

2. How to interpret and use valid data

3. How to talk about early and future data with the media


The Layout of the DayThe Layout of the Day

Morning: Early Data Are the data valid? What can we say to the media (and how do we

say it) about less than ideal data? Lunch On Your Own (12:30 – 2:00)

During which all data become valid. Remember what you ate…

Afternoon: Using Quality Data What can we learn from outcomes data?

Data Analysis Data Analysis as a Tool to as a Tool to

Promote Data Promote Data QualityQuality

Data Analysis Data Analysis as a Tool to as a Tool to

Promote Data Promote Data QualityQuality


Keeping our eye on the prize:

High quality services for children and

families that will lead to good outcomes.


High Quality Data on OutcomesHigh Quality Data on Outcomes

Data are a piece of a system that helps to achieve overarching goals for children and families

Data yield Findings that can be interpreted as

having a particular meaning that leads to specific actions to improve the system.


Prof’l Development•Preservice•Inservice

System for Producing Good Child and System for Producing Good Child and Family OutcomesFamily Outcomes

Good Federal policies and programs

Good State policies and programs

High quality services and supports for children 0-5 and their families

Good outcomes for children and families

Good Local policies and programs

Adequate funding

Strong Leadership


The Vision: Using Data as a Tool for The Vision: Using Data as a Tool for Program ImprovementProgram Improvement

State will have quality data available on an ongoing basis about multiple components of the system Child and family outcomes Services provided Personnel (types, qualifications, etc.) Etc.


FMAFMA

Findings

Meanings

Action


FindingsFindings Findings are the numbers

10% of families responded …… 45% of children were in OSEP category b

The numbers are

not debatable

(assuming the numbers

are correct…)


MeaningMeaning

The interpretation put on the numbers Is this finding

Credible? (Based on valid data?) Good news? Bad news? News we can’t

interpret?


MeaningMeaning

Meaning is debatable. Reasonable people can reach different conclusions from the same set of numbers

Stakeholder involvement can be helpful in making sense of findings

To interpret meaning, sometimes we analyze data in other ways (ask for more findings)


Putting Meaning on the DataPutting Meaning on the Data What are alternative explanations for

the finding? Are there other ways of looking at the data that might provide insight into a possible explanation?

(i.e., should we run more analyses?)


ActionAction Given the meaning put on the findings,

what should be done? Possible actions:

Continue quality assurance activities to improve the quality of the data

Accept the data as credible and develop recommendations based on the findings

Action is always debatable – and often is debated

Another role for stakeholders


Building quality into the state system Building quality into the state system

Keep errors from occurring in the first place

Develop mechanisms to identify weaknesses that are leading to data collection errors

Provide ongoing feedback including reports of the data to programs and providers


Procedures to Promote QualityProcedures to Promote Quality Preparing for data collection

Adequate training and communication

During data collection Commitment to the data collection System of supports for the “data providers”

After data collection Data entry Data follow up Data analysis


Quality of the process: Quality of the process: Preparing for data collectionPreparing for data collection

Training and Communication

Is there a process for checking whether all of the

[data] providers understand what they are to do?

Is there a process for checking whether they do it?

Do they know why they are doing it?

**What do we know about one shot trainings??**


Quality of the Process:Quality of the Process:During Data CollectionDuring Data Collection

Commitment to the data collection Do providers understand the importance of the activity? Has the system been designed so providers (and families)

will receive benefit from collecting and providing data? Do providers know someone will be checking on what they

are doing?Supports

Has the process been designed to make it as easy and to take as little time as possible? (Can any part be streamlined?)

Is a knowledgeable person observing or tracking data collection activities and providing feedback in a timely manner?

Is there a way for providers to get ongoing questions addressed?


Quality of the Process: After Data Quality of the Process: After Data CollectionCollection

Data entry

Are there safeguards to minimize data entry errors?

Data follow up

Verification: Is there a process in place for checking [a sample of] records for accuracy and completeness?

Is there a process for providing timely feedback when errors are discovered?


Quality of the Process: After Data Quality of the Process: After Data CollectionCollection

Data analysis

Cleaning individual data: Are there procedures for identifying out of range values, anomalies, incomplete data?

Is there a plan for looking at the aggregated data in various ways to identify unexplainable variations, strange patterns, etc.?

Is there a process for providing timely feedback when errors are discovered?


ValidityValidity

Validity refers to the use of the information Does evidence and theory support the

interpretation of the data for the proposed use?

Or Are you justified in reaching the conclusion

you are reaching based on the data? Standards for Educational and Psychological Testing (1999) by

American Educational Research Association, American Psychological Association, National Council on Measurement in Education


Validity ArgumentValidity Argument

Accumulation of evidence from a series of “if-then” propositions about the data

If the data are valid, then……, e.g., Data should not vary wildly across programs

serving the same kinds of children Data for children with certain kinds of disabilities

should look different than data for other children Etc.

Are there sensible patterns in the data?


In Search of In Search of ValidityValidity


Take Home MessageTake Home Message

If you conclude the data are not (yet) valid, they cannot be used for program effectiveness, program improvement or anything else.

Meaning = Data not yet valid

Action = Continue to improve data collection and quality assurance


The validity of your data is The validity of your data is questionable if:questionable if:

?

Validity Exercise 1Validity Exercise 1

Acquisition and use of knowledge and skills (including early language/communication and early literacy):

Number of children

% of children

a. Percent of preschool children who did not improve functioning 1 4

b. Percent of preschool children who improved functioning but not sufficient to move nearer to functioning comparable to same-aged peers 5 22

c. Percent of preschool children who improved functioning to a level nearer to same-aged peers but did not reach 7 30

d. Percent of preschool children who improved functioning to reach a level comparable to same-aged peers 6 26

e. Percent of preschool children who maintained functioning at a level comparable to same-aged peers 4 17

Total N= 23 100%

Validity Exercise 1Validity Exercise 1Acquisition and use of knowledge and skills (including early language/communication and early literacy):

% of children

a. Percent of preschool children who did not improve functioning 4

b. Percent of preschool children who improved functioning but not sufficient to move nearer to functioning comparable to same-aged peers 22

c. Percent of preschool children who improved functioning to a level nearer to same-aged peers but did not reach 30

d. Percent of preschool children who improved functioning to reach a level comparable to same-aged peers 26

e. Percent of preschool children who maintained functioning at a level comparable to same-aged peers 17

Total 100%



1. The n is too small.

If your n is small, make sure all of your tables and text show the n prominently.



Number of children

% of children






Total N= 1753 100%




2. The overall pattern in the data looks “strange.”

Strange = Unexplainable VariationStrange = Unexplainable Variation

But how do you know?But how do you know?

Compared to what?Compared to what?





Compared to what you would expect.



Number of children

% of children


b. Percent of preschool children who improved functioning but not sufficient to move nearer to functioning comparable to same-aged peers

210 12




Total N= 1753 100%



Number of children

% of children






Total N= 1753 100%





Compared to what you would expect. Compared to other data you have

available.


What else do you know (or can find What else do you know (or can find out?)out?)

Child outcomes: D + E = 73% of children exiting meeting

age expectations (reasonable?)

Part C 618 Exit Data: Do the exit data support this?





Compared to what you would expect. Compared to other data you have

available. Compared to other states (that you

would expect to be similar).

Validity Exercise 4Validity Exercise 4Acquisition and use of knowledge and skills (including early language/communication and early literacy):

State 1(n= 1,753)

%

State 2(n = 6,543)

%

State 3 (n=2,451)

%

State 4(n=487)

%

a. Percent of preschool children who did not improve functioning 3 2 1 2

b. Percent of preschool children who improved functioning but not sufficient to move nearer to functioning comparable to same-aged peers

28 21 10 24

c. Percent of preschool children who improved functioning to a level nearer to same-aged peers but did not reach

20 24 46 26

d. Percent of preschool children who improved functioning to reach a level comparable to same-aged peers

27 25 19 20

e. Percent of preschool children who maintained functioning at a level comparable to same-aged peers

22 28 24 28


State Data SharingState Data Sharing

(AKA Looking (AKA Looking

for Red Flags)for Red Flags)


Outcome ___

State 1(n=)

%

State 2(n =)

%

State 3 (n=)

%

State 4(n=)

%

State 5(n =)

%

State 6 (n=)

%

State 7(n=)

%

State 8(n=)

%

a. Did not improve functioning

b. Improved but not sufficient to move nearer to functioning comparable to same-aged peers

c. Moved nearer to same-aged peers but did not reach

d. Reached a level comparable to same-aged peers

e. Maintained functioning comparable to same-aged peers


Validity and GeneralizabilityValidity and Generalizability

Which group do these findings apply to? Is the group with data representative of

children served in the program statewide? By geography By demographics By types of disabilities and

delays By length of time in service


Non-representative dataNon-representative data

If data does not include: All areas of state

Dallas ≠ Texas All kinds of families in state

No minority families All kinds of children served in program

Only children with severe disabilities



If data does not include: Children who have been in program

the maximum length of time, .e.g., 36 months for Part C.

All states have non-representative child outcomes data in 2008 (and 2009…).

Percentage of Infants & Toddlers Entering Percentage of Infants & Toddlers Entering Services by Age at Entry *Services by Age at Entry *

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36Age in Months

Per

cent

* Age at development of the Individualized Family Service Plan (IFSP).

Eligibility by Age at EntryEligibility by Age at EntryEligibility by Age at EntryEligibility by Age at Entry

32

75

91

36

16

6

32

93

0

10

20

30

40

50

60

70

80

90

100

<12 months 12-24 months > 24 months

Developmental Delay Diagnosed Condition At-risk



If data does not include: Children who have been in program

the maximum length of time, .e.g., 36 months for Part C.

This applies to every state’s data. Complete the “Earliest possible

date….” worksheet.


Implications of Implications of non-representative datanon-representative data

The findings may be valid

BUT

only for the group represented in the data

The findings are not valid for your state overall.

Validity is related to use of the data.





3. The data are not representative of the state and the conclusions being drawn suggest they are.




2. The overall pattern in the data looks “strange” (= unexplainable variation)

3. The data are not representative of the state and the conclusions being drawn suggest they are.

4. The pattern for subgroups looks “strange” (=unexplainable variation).


Data ExplorationData Exploration

Examine the data to look for inconsistencies

If and when you find something strange, look for some other data you have that might help explain it. Is the variation caused by something other than bad data?


Data ExplorationData Exploration

If the variation can be explained, lower the red flag and consider the data valid. Proceed to analyze the data for program improvement (come back after lunch…)

If you conclude the variation is caused by poor quality data (M), develop a targeted plan to improve the data collection (A).


So when can you trust your data?So when can you trust your data?

When you can’t find any more red flags.

When the errors that remain will not lead to incorrect conclusions.

Improving data collection is a continuous process.


The data can be considered valid for The data can be considered valid for conclusions related to program effectiveness conclusions related to program effectiveness and program improvement when:and program improvement when:

1. The n is sufficiently large.

2. The overall pattern in the data looks reasonable (no unexplainable variation)

3. The data are representative of the state.

4. The pattern for various subgroups in the data looks reasonable (no unexplainable variation)

By locality By disability By ?, ?, ?


How far along is your state?How far along is your state?

Quality of Child Outcomes Data

Time


Take Home MessageTake Home Message

If you conclude the data are not (yet) valid, they cannot be used for program effectiveness, program improvement or anything else.

Meaning = Data not yet valid

Action = Continue to improve data collection and quality assurance

Welcome to the Child Outcomes Data Workshop!

Documents

Transcript of Welcome to the Child Outcomes Data Workshop!