Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting

How to Explain the Numbers:Helping Staff, Parents, and Other

Stakeholders Understand the Results of the NCSEAM Surveys for

Part C and 619

Batya Elbaum, NCSEAMPam Roush, West Virginia Part COSEP National Early Childhood

MeetingArlington, VA, December 2007

Purpose of the session To provide participants with

strategies for explaining the measures and percentages that come from the NCSEAM rating scales addressing Indicators C4 and B8.

Part C Indicator #4

“Percent of families participating in Part C who report that early intervention services have helped the family a) know their rights, b) effectively communicate their children’s needs, and c) help their children develop and learn.”

Part B Indicator #8

“Percent of parents with a child receiving special education services who report that schools facilitated parent involvement as a means of improving services and results for children with disabilities.”

WV Part C Indicator 4 4A – Know rights

2006 76.5% 2007 73.8%

4B – Communicate 2006 72.8% 2007 71.0%

4C – Help child 2006 86.3% 2007 78.7%

Change in mean measure by region

Region 2006

N 2006 Mean

2007 N

2007 Mean

Change in Mean

1000 Part C Impact on Family Measures 62 716.41 44 705.32 -11.09

1100 Part C Impact on Family Measures 64 702.91 44 715.36 +12.45







Percent on 4A by Region in 2006 and 2007

2006 2007

Region N 4A % N 4A %

1000 62 72.58 44 70.45

1100 64 71.87 44 86.36

1200 50 72.00 47 74.47

1300 93 .78.49 58 77.59

1400 28 92.86 24 58.33

1500 25 60.00 25 64.00

1600 48 79.17 33 63.64

1700 90 81.11 53 79.25

Why use a measurement analysis?


Use of a measurement framework ensures that measures will mean the same thing, regardless of how many items, or which specific items, are administered.


We can’t assume that all survey items are equally agreeable.

A measurement analysis gives us a measure of each item’s overall agreeability.

Order of Impact on Family items from least to most agreeable

Item Calibration

Item Stem: Over the past year, Early Intervention services have helped me and/or my family:

678 Participate in typical activities for children and families in my community.

656 Know about services in the community.

640 Know where to go for support to meet my family's needs.

625 Keep up friendships for my child and family.

609 Know where to go for support to meet my child's needs.

584 Be more effective in managing my child's behavior.

576 Make changes in family routines that will benefit my child with special needs.

576 Do activities that are good for my child even in times of stress

570 Improve my family's quality of life.

565 Feel that I can get the services and supports that my child and family need.

563 Get the services that my child and family need.

562 Feel that my family will be accepted and welcomed in the community

559 Feel more confident in my skills as a parent.

559 Feel that my child will be accepted and welcomed in the community.

556 Communicate more effectively with the people who work with my child and family.

553 Understand how the Early Intervention system works.

546 Understand the roles of the people who work with my child and family.

539 Know about my child's and family's rights concerning Early Intervention services.

534 Be able to evaluate how much progress my child is making.

516 Understand my child's special needs.

498 Feel that my efforts are helping my child.

498 Do things with and for my child that are good for my child's development.


Use of a measurement framework allows us to test whether all the items are measuring the same thing.

Measuring the 3 subindicators

NCSEAM’s measurement analysis of pilot data from thousands of families showed that items that are related to the three OSEP outcomes all fit into a single scale of family outcomes.

Measuring the 3 subindicators

Families appear to achieve these outcomes in a very consistent order. Families who report that EI helped them

know their rights also report that EI also helped them help their child develop and learn.

Families who report that EI helped them effectively communicate their children’s needs also report that EI also helped them know their rights and help their child develop and learn.

Location of key items related to Indicator C4

Indicator 4c: Help their children develop and learn. IFS items: “Understand my child's special needs.” [516] “Do things with and for my child that are good for my child's development.” [498]

Indicator 4b: Effectively communicate their children’s needs. IFS Item: “Communicate more effectively with the people who work with my child and family.” [556]

Indicator 4a: Know their rights.IFS Item: “Know about my child's and family's rights concerning Early Intervention services.” [539]

556

539

516


We can’t assume that the response choices (very strongly disagree, strongly disagree, disagree, agree, strongly agree, very strongly agree) indicated by equidistant circles or numbers on a page are really equidistant.

Spacing of response choices

◘ ◘ ◘ ◘ ◘ ◘ 1 2 3 4 5 6

Spacing of response choices

◘ ◘ ◘ ◘ ◘ ◘ 1 2 3 4 5 6 1K 5K 10K ½ Mar. Marathon Ultra

The logic of distributions

Example: Distribution of height

Distribution of Measures of Reading Achievement - 1

X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X >5’ 5’0 5’2 5’4 5’6 5’8 5’10 6’0 6’2 6’4 6’6 6’8 6’10 7’ >7’

Distribution of Measures of Reading Achievement - 2

X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X >5’ 5’0 5’2 5’4 5’6 5’8 5’10 6’0 6’2 6’4 6’6 6’8 6’10 7’ >7’

West Virginia results 2006

West Virginia results 2007

WV without extreme cases 2006

WV without extreme cases 2007

Instrumentation and approach to data analysis make a difference

Choice of instruments: Part B NCSEAM K-12 and 619 9 NCSEAM K-12 21 Customized NCSEAM survey 11 State-developed or adapted 18 ECO Family Outcomes Survey 1

States’ reported baseline data onIndicator 8B

Choice of instruments: Part C NCSEAM Survey 25 ECO Family Outcomes Survey 18 State-developed or adapted 10 Combination 1

Analysis of states’ baseline data:Indicator C4

Sub-Indicator a. Know their

rights b. Communicate children’s needs

c. Help children develop and learn

Mean 79% of families 78% of families 85% of families

Range 45% - 99% 42% - 99% 53% - 99%

Analysis of states’ baseline data:Indicator C4

Figure 2: Baseline by Survey Tool and Criteria for Positive Response

0

10

20

30

40

50

60

70

80

90

100

ECO 3 & above

N=2

ECO5 & above

N=16

NCSEAM alternate analysis

N=6

NCSEAM standard analysis

N=17

State Survey

N=11

Tool and Analysis

% a

% b

% c

APR Family Outcome Results – Part C States Using NCSEAM Survey and RASCH analysisAPRs submitted February 2007

4A 4B 4C Response Rate

#

California 48% 42% 71% * 5,413 Connecticut 73% 71% 84% 27% 712 Florida 56% 52% 57% 25% 402 Idaho 56% 52% 68% 14.74% 355 Kentucky 82% 73% 89% 22% 786 Mass. 80% 77% 86% 30% 290 Michigan 56% 51% 73% 40.8% 2,439 Nebraska 76% 73% 86% 48% 921 New Mexico 78% 72% 92% 23.8% 183 New York 71% 65% 82% 25.6% 580 Pennsylvania 67% 63% 79% 23.81% 3,018 Utah 74% 71% 79% 25.4% 605 Wyoming 78% 85% 86% 30% 230 South Carolina 74% 70% 86% ? ?

Reference to a standard

Percent of measures above the adopted standards

standards

What is the relationship between % at or above the NCSEAM standard and % agreement on the “threshold item?”

Sample descriptive interpretation related to Indicator 4A Approximately 90-95% of families agreed,

with approximately two-thirds of families expressing strong or very strong agreement, that early intervention helped them:

Understand their child’s special needs. Do things with and for their child that

are good for their child’s development. Feel that they can handle the

challenges of parenting a child with special needs.

Part C Example: WV data

Q44 - know about my child's and family's rights

AKnow Response Category Frequency Percent

% Agree, SA, VSA

1 5 6.0 2 1 1.2 3 1 1.2 4 65 78.3 5 9 10.8

6 2 2.4 91.5 Total 83 100.0

Below cut-score on this standard

Missing 3

Total 86 4 15 6.4 5 57 24.5

6 161 69.1 100.0 Total 233 100.0 Missing 9

At or above cut-score on this standard Total 242

Q42 - communicate more effectively

BComm Response Category Frequency Percent % Agree, SA,

VSA 1 4 4.8 2 3 3.6 3 6 7.1 4 67 79.8

5 4 4.8 84.6 Total 84 100.0 Missing 11

Below cut-score on this standard Total 95

3 1 .5 4 15 7.0 5 62 29.0

6 136 63.6 99.5 Total 214 100.0 Missing 19

At or above cut-score on this standard

Total 233

Q46 - understand my child's special needs

CHelp Response Category Frequency Percent

% Agree, SA, VSA

1 6 9.2 2 1 1.5 3 4 6.2 4 50 76.9 5 4 6.2 83.1 Total 65 100.0 Missing 5

Below cut-score on this standard

Total 70

3 1 .4 4 24 10.4 5 52 22.5

6 154 66.7 99.6 Total 231 100.0 Missing 27

At or above cut-score on this standard Total 258

Part B ExampleItem: “The school explains what options parents have it they disagree with a decision of the school.”

SEPPS Measure

% VSD

% SD

% D

% A

% SA

% VSA

Total % A/SA/VSA

>=350<400 21.9 8.8 47.4 20.2 0.9 0.9 22.0 >=400<450 7.6 6.7 51.4 33.3 0.0 1.0 34.3 >=450<500 4.1 2.0 27.6 57.1 7.1 2.0 66.2 >=500<550 3.3 3.3 18.0 62.0 8.7 4.7 75.4 >=550<600 0.0 0.0 16.0 54.0 16.0 14.0 84.0 >=600<650 0.0 3.1 4.6 26.2 44.6 21.5 92.3 >=650<700 0.0 0.0 3.3 30.0 20.0 46.7 96.7 <600 (n=704) 704/903 = 78%

22.2 6.3 28.4 35.9 4.4 2.8 43.1

>=600 (n=199) 199/903 = 22%

0.5 1.0 4.5 15.6 24.1 54.3 94.0

ALL 17.4 5.1 23.1 31.5 8.7 14.2 54.4

The NCSEAM standard is a stringent standard Cut score for B at 600 Cut scores for C at 539, 516, 556 The standard is set so as to ensure

that approximately 95% or more of the responses will be in one of the agree categories on the item designated as the “threshold item”

Simple agreement with the “threshold item” is not enough

How precise are the percentages that states are reporting?

Error in estimation Whenever data are based on a

sample (whether owing to sampling or to a less than 100% return rate), there is some amount of error in generalizing to the population.

“Error,” or imprecision, can be represented by a confidence interval.

As sample size decreases, error of estimate - and its representation as a confidence interval - increases.

The decrease is not linear!

Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting

Documents

Transcript of Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting