Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the...
-
Upload
chloe-joseph -
Category
Documents
-
view
216 -
download
1
Transcript of Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the...
Approaches to Measuring Approaches to Measuring Child OutcomesChild Outcomes
Approaches to Measuring Approaches to Measuring Child OutcomesChild Outcomes
Kathy HebbelerECO at SRI International
Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes, Albuquerque, NM April 2006
2
What is happeningWhat is happening
Outcomes measurement is difficult and very complex
No one group has all the answers – or even most of the answers
There are some exciting things going on around the country
3
Purpose of the meetingPurpose of the meeting
Share our challenges and what we have learned so far
Contribute to the collective knowledge base
Advance the discussion incrementally moving closer to producing outcomes data
*** for the ultimate good of children and families ***
4
Critical eventsCritical events Spring 2005 – ECO submitted
recommendations to OSEP on what should be collected with regard to child and family outcomes
Summer 2005 – OSEP released the reporting requirements
December 2005 – States submitted their plans for outcome data collection in their State Performance Plan
Spring 2006 – States are collecting data
5
The Number One Question*:The Number One Question*:
What are other states doing?
Have other states done X?
*In two forms
6
Purpose: The Overriding Purpose: The Overriding QuestionQuestion
Why is a state collecting data on child outcomes?
Context, resources, values, etc. enter into the answer
7
To respond to federal
reporting requirements
To meet provider/teacher, local and/or state need for outcome
information and
to respond to federal reporting
requirements
Purpose
WHY?
Context & Context & Values Drive Values Drive
DecisionsDecisions
Resources?
Stakeholder input?
Burden on locals?
Standardized assessment?
Authentic assessment?
Interagency issues?
Local control?
Minimize change?Early learning guidelines?
Multiple sources of information?
Policymakers want data?
Other early childhood initiatives?
9
Who is included in the outcomes Who is included in the outcomes system?system?
Pt. C system; Pt. B system Some blending of C and B
Same assessment Data sharing Data linking
Early Childhood System that includes C and B
10
How does the state get data on How does the state get data on outcomes?outcomes?
Who provides? What assessments are used? How often is data collected? When is data collected? (When is it
reported?) Dealing with multiple sources? Dealing with different assessments?
How Outcomes Data Get to the State AgencyHow Outcomes Data Get to the State Agency
12
Analysis of SPPsAnalysis of SPPs
Analyses are based on SPP reports submitted in December 2005
Pt. C N=56; Pt B N= 58 Limitations
Variation in level of detail provided Landscape keeps changing
Analysis done by Lynne Kahn and staff at UNC/FPG
13
““Camera” Issue: Capturing Child Camera” Issue: Capturing Child FunctioningFunctioning
What are the sources of the information on child functioning?
What kind of assessment tools are states planning to use?
14
Capturing Child Functioning: Capturing Child Functioning: How many sources?How many sources?
Multiple sources Pt. C: 50% (28 states) Pt. B: 16% (16 states)
One data source Pt C: 39% (22 states)
Assessment instrument (21 states) Pt B: 55% (32 states)
Assessment instrument (31 states)
15
Issues RaisedIssues Raised Data needs to reflect a child’s
functioning in each broad outcome area Functional outcomes summarize each
child’s current functioning across settings and situations
Best practice for assessing young children recommends the use of multiple measures
Will single sources (= assessment tool) Will single sources (= assessment tool) produce valid data on functional produce valid data on functional outcomes? How good is the camera?outcomes? How good is the camera?
16
The Child OutcomesThe Child Outcomes
Children have positive social relationships
Children acquire and use knowledge and skills
Children take appropriate action to meet their needs
17
Part C Outcomes Data SourcesPart C Outcomes Data Sources
Data Source # %
Formal assessment instruments
45 80%
Parent report 25 45%
Observation 14 25%
Clinical opinion 10 18%
IFSP goals & objectives 6 11%
Record review 4 7%
Not reported 6 11%
18
Preschool Outcomes Data SourcesPreschool Outcomes Data Sources
Data Source # %
Formal assessment instruments
45 80%
Observation 12 21%
Parent report 11 19%
Teacher/provider report 8 14%
IEP goals & objectives 1 2%
Clinical opinion 1 2%
Not reported 10 17%
19
Role of FamiliesRole of Families Impossible to understand how a child is
functioning across a variety of everyday settings and situations without family input
Options Incorporated into the assessment tool Collected through a parent-completed tool Incorporated into a summary rating
Issue: How is information from families Issue: How is information from families being included?being included?
20
Capturing Child Functioning: Capturing Child Functioning: Approaches to identifying Approaches to identifying assessment toolsassessment tools
One assessment selected by state
List of assessments developed by state; programs pick
Programs can use whatever they have been using
21
Capturing Child Functioning: Capturing Child Functioning: Assessment Tools Being UsedAssessment Tools Being Used
Part C – 20 different assessment tools identified 3 states using state developed tool
Part B – 43 different assessment tools identified 7 states using state-developed tool
22
Commonly Reported Commonly Reported Assessment Instruments: Part CAssessment Instruments: Part C
Of 28 states who listed specific assessment instruments: HELP - 15 states BDI/BDI2 - 13 states AEPS - 11 states Creative Curriculum - 6 states ELAP- 6 states
Not reported – 30 states Not yet determined - 23 states
23
Commonly Reported Commonly Reported Assessment Instruments: PreschoolAssessment Instruments: Preschool
Of 31 states who listed specific assessment instruments: BDI/BDI2 - 9 states Creative Curriculum - 8 states Brigance- 7 states High Scope COR - 6 states AEPS - 5 states State developed assessments - 7 states
Not reported - 27 states Not yet determined – 21 states
24
Capturing Child Functioning: Capturing Child Functioning: Combining Information from Multiple Combining Information from Multiple SourcesSources
Part C : Using ECO Summary Form – 52% (29
states) Developing own summary tools – 7%
(4 states) Part B:
Using ECO Summary Form – 29% (17 states)
Developing own summary tools – 10% (6 states)
25
Capturing Child Functioning: Capturing Child Functioning: TimingTiming
When and how often outcome information is being collected is related to why state is collecting data
What assessment is also related to why
OSEP requirement is entry and exit
26
When will data be “collected”?When will data be “collected”?
Aligned around the naturally occurring data review points in programs
“Collected” may mean Data reviewed/summarized to determine
a functional level for each of the outcomes
Summary rating or other data reported to state or OSEP
Some states did not report anything beside at entry and exit; (C – 28 states; B - 15 states)
27
When entry data will be collectedWhen entry data will be collected(three general patterns)(three general patterns)
Referral Eligibility Initial IFSP- e.g. goals, services, settings
IFSP 6 month review- intervention planning
Around eligibility (based on evaluation data)
Initial IFSP (based on eval and assessment data)
After services begin (based on eval, assess, and/or ongoing progress monitoring data
28
Part C examples of when data Part C examples of when data will be collectedwill be collected
W/in 45 days of referral- 7 states W/in 1 month of IFSP- 2 states W/in 6 months of enrollment- 1 At initial IFSP and 6 month and
annual reviews- 21 W/in 2 months, 45 days, 3 months,
6 months of exit
29
Preschool examples of when Preschool examples of when data will be collecteddata will be collected
Initial evaluations/eligibility - 7 states Initial IEP development -4 states Annual IEP reviews - 11 states Time periods prescribed by curriculum
referenced tools (2 or 3 times a year)- 8 states
Annually at the end of the school year- 6 states
30
Which Children Will Be Included: Which Children Will Be Included: Part C Part C
All children – 40 states After a pilot or phase in period- 16 states
Sampling – 7 states 1 sampling at exit (all children will have
entry data) 1 sampling at entry (will only collect entry
and exit data on children in sample) Other 5 - could not tell from SPP
Not reported or undecided - 9 states
31
Which Children will be Included: Which Children will be Included: PreschoolPreschool
All children – 42 states After a pilot or phase in period- 15 states
Sampling - 8 states 3 will collect data on ALL children, but select
a sample to report to OSEP 1 sampling at entry (will only collect entry
and exit data on children in sample) Other 4 - could not tell from SPP
Not reported or undecided- 8 states
32
Collaboration between C and B Collaboration between C and B
25 states reported in Part C SPP collaborating with Part B on outcomes
21 states reported in Part B SPP collaborating with Part C on outcomes
33
Collaboration with Other Early Collaboration with Other Early Childhood InitiativesChildhood Initiatives
Collaborate with or align outcome efforts with broader early childhood accountability initiatives in their state Part C: 3 states Part B: 18 states
Issue: What are the outcomes being Issue: What are the outcomes being assessed in the broader initiatives?assessed in the broader initiatives?
34
Role of the Early Learning Role of the Early Learning GuidelinesGuidelines
May change or add to the outcomes questions
Are children meeting the ELGs?Are children meeting the ELGs? May mean mapping the ELGs to
the 3 OSEP outcomes Aligning with ELGs:
Part C – 8 states; Part B – 18 states
How Outcomes Data Get to the State AgencyHow Outcomes Data Get to the State Agency
36
Transfer Issues: How does Transfer Issues: How does information move?information move?
In what form? At what level of
detail? With what level of
identification?
37
In what form?In what form?
Online In an electronic
file On paper
38
At what level of detail?At what level of detail?
Child Level Data Item level data on the child (from an online
assessment system) Scores on assessment tool ECO Summary Rating OSEP Categories (a, b, c) Other?
Aggregated Data Scores, Rating, OSEP categories, etc.
39
With what level of With what level of identification?identification?
Only relevant for child-level data Can state link outcome data be linked to
other information though an ID? Does it enter the system already linked?
Linkage to other data has major Linkage to other data has major implications for analysis and implications for analysis and questions state will be able to questions state will be able to answeranswer
40
Training Training
Focused on various topics Training in assessment tools Training in use of the ECO Summary Form
Various approaches Various levels of investment ECO is developing materials and
compiling training materials for web site (including materials designed for parents)
Contact NECTAC or ECO for help
41
ConclusionsConclusions
States are building many different kinds of outcomes measurement systems
Features of the system reflect the contexts and values of the state
We know some things about what states are doing but the landscape keeps changing