Post on 19-Dec-2015
Communicating through Data Displays
October 10, 2006
© 2006 Public Consulting Group, Inc.
Key Terms
Aggregated Data: Data that are presented in summary (as opposed to student-level data)
Alignment: The quality that allows you to compare one test to another test (A vertically aligned test represents real gains or losses from one year to the next)
Disaggregation: Summary data split into different subgroups (e.g. gender, race, ethnicity, lunch status, SPED)
Error: Impacts the validity of an assessment; Includes measurement error and sampling error
Inference: Conclusions that are drawn from a data set Sample: Group of students included in a data set Validity: The statistical term used to determine how much inference can be
made
The Framework
Measures Multiple measures allow for a more complete
picture of student performance Explorations
Explorations enable looking at the data through different lenses to answer essential questions
Disaggregators Disaggregators help reveal the various
factors that impact educational outcomes
Types of Measures
Measures are the “yardstick” that is used to measure student performance. The more measures that are used, the more robust and complete the
picture State Assessments (MCAS)
Usually taken in spring and reported the following fall Not vertically aligned Tests vary from year to year
National Assessments (Terra Nova, ITBS, etc.) Some districts choose to supplement the state assessment with a national
assessment Many of these are vertically aligned and are aligned from year to year National assessments are not aligned to the state curriculum framework
Diagnostic Assessments (DRA, DIBELS) Diagnostic assessments help identify students who need interventions and
supports Diagnostic assessments may not be vertically aligned
Types of Measures (cont’d) Subject Area and Course Grades
Grades are subjective and can tend to be inflated Grades can be compared to performance on state and other assessments to
identify disparities Disciplinary Records
Discipline data can be used to monitor high-risk students and explore the impact of behavior on performance
Discipline consequences provide important information for identifying inequities among groups of students (e.g., students with disabilities, ethnic groups)
Attendance Rates Attendance data can be used to identify students who are at risk Attendance data can be used to explore the relationship between attendance and
performance Graduation Rates
Graduation rates can be used to evaluate the effectiveness of curriculum and instruction
Graduation rates can be analyzed to identify inequities based on student characteristics
Types of Explorations
The type of exploration you choose depends on the question that you want to answer. Types of exploration include: Snapshot Cross-Sectional Longitudinal Gains Item-Level Student Listing Correlation
Snapshot
Shows how a group of students performed against a given measure at a certain point in time.
Limitations: This analysis only presents one point in time.
(Graph Type: Histogram / Bar)
How did students perform at a certain point in time?
State Assessment Performance5th Grade Math, 2005-2006
Washington Elementary School
26%
42%
21%
11%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
Warning (n=90) Needs Improvement(n=144)
Proficient (n=72) Advanced (n=36)
Performance Level
Per
cent
of S
tude
nts
Historical
Looks at how students at a particular grade level performed on a given measure across multiple years.
This is what NCLB uses to calculate AYP.
Limitations: This analysis does not take into account differences in the group of students from year to year.
(Graph Type: Floating Column)
How did students at a certain grade-level perform historically?
Analysis of Proficiency over Time5th Grade State Math Assessment
Washington Elementary School
38% 36% 33% 30% 28% 31%
41%36%
30%24% 23%
26%
15%21%
27%
30% 30%28%
6% 6% 9%16% 19% 15%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2000 (n=34) 2001 (n=33) 2002 (n=33) 2003 (n=37) 2004 (n=43) 2005 (n=39)
Year
Pe
rce
nta
ge
of S
tud
en
ts
Warning Needs Improvement Proficient Advanced
Longitudinal
Looks at a cohort of students over time.
Shows “real gains” Limitations:
Comparisons of a group of students from one year to another are only valid using a vertically-aligned test.
(Graph Type: Line)
How did a cohort of students perform over time?
Terra Nova Math PerformanceClass of 2010, Disaggregated by Gender
Washington Elementary School
63 6568
71 6972 71
6259 61 63 65
69 6868 69 7175 77 77 76
0
10
20
30
40
50
60
70
80
90
2000(n=34)
2001(n=34)
2002(n=34)
2003(n=34)
2004(n=34)
2005(n=34)
2006(n=34)
Ave
rag
e N
CE
Sco
re
My District Male Female
Gains
Looks at the extent to which students are improving over time or losing ground based on a particular measure.
Limitations: Caution must be used when drawing conclusions about a given student based upon performance on two tests.
(Graph Type: Stacked Column)
How did students who performed at each level on a prior assessment perform on
subsequent assessments?
State Math Assessment Gains AnalysisGrade 5 (2006) Performance Grouped by Grade 3 (2004) Performance
Washington Elementary School
32
167
0
10 38
18
3
3
11
18
4
0
0
5
9
0
10
20
30
40
50
60
70
Warning (n=45) Needs Improvement(n=65)
Proficient (n=48) Advanced (n=16)
Grade 5 (2006) Performance Level
Num
ber
of S
tude
nts
3rd Grade Warning 3rd Grade NI 3rd Grade Proficient 3rd Grade Advanced
Student Listing
Allows the analysis of students in a group in relation to each other.
Conditional formatting can be added to highlight outliers.
Limitations: Student listings can be difficult to interpret when too many data elements are included.
Student Listing - 6th Grade, 2006-2007Performance on the 5th Grade and 3rd Grade Math Assessment
Washington Elementary School
Last Name First Name3rd Grd Perf Level
5th Grd Perf Level
Move-ment Gender Ethnicity Race
Lunch Status
SPED Status
LnameA FnameA Basic Basic 0 M Hispanic White Full YesLnameB FnameB Basic Basic 0 M Hispanic Black F/R NoLnameC FnameC Proficient Basic -1 M Hispanic Black F/R NoLnameD FnameD Below Basic Basic 1 F Not Hispanic White F/R NoLnameE FnameE Proficient Proficient 0 M Hispanic White Full YesLnameF FnameF Basic Below Basic -1 M Hispanic Black Full NoLnameG FnameG Basic Proficient 1 F Not Hispanic Black Full NoLnameH FnameH Basic Proficient 1 F Not Hispanic Black F/R YesLnameI FnameI Below Basic Basic 1 F Not Hispanic Asian F/R YesLnameJ FnameJ Proficient Basic -1 M Hispanic White Full NoLnameK FnameK Below Basic Basic 1 F Not Hispanic Asian F/R NoLnameL FnameL Below Basic Below Basic 0 M Not Hispanic Asian Full NoLnameM FnameM Advanced Proficient -1 M Hispanic White F/R No
Advanced one or more levelsStayed the sameDropped one or more levels
What are the characteristics of specific students?
Item Analysis
Displays how students did on each item or within a particular standard or strand.
Providing reference groups is important for tests that are not aligned from year to year because that is the only way to determine relative performance.
Limitations: Smaller sample sizes (e.g. classroom-level) limit the inferences that can be made
(Graph Type: Scatter)
How did a group of students perform on an item or on a set
of items on a specific assessment?
Percent Correct by Objective 5th Grade Math Assessment, 2005-2006
Washington Elementary School
40%
45%
50%
55%
60%
65%
70%
75%
80%
85%
90%
Ap
pro
x.
Me
as
ure
s
Wh
ole
Nm
brs
&D
ec
ima
ls
Co
mp
os
ing
/Re
vis
ing
Alg
eb
raic
Co
nc
ep
ts
Me
tric
Me
as
ure
s
Fra
cti
on
s
Inte
rpre
tati
on
Pe
rce
nts
Cla
ss
. &
Lo
g.
Re
as
on
ing
Ba
sic
Fa
cts
Ma
th.
Ap
pli
ca
tio
ns
Objective
Per
cent
Cor
rect
My School
District
State
Correlation
Looks at the relationship between performance in one measure to performance in another measure.
Correlation does not equal causation.
Limitations: Correlations should not be done with small groups of students.
How is performance in one measure related to performance
in another?
(Graph Type: Scatter)
State Assesment5th Grade Reading to 10th Grade Math
Anytown School District
200
300
400
500
600
700
800
0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150
Grade 5 Reading Scale Score
Gra
de 1
0 M
ath
Sca
le S
core
Pearson Coefficient = 0.80
Advanced / Proficient
Needs Improvement /Warning
Cut Scores
Disaggregators
Disaggregators are used to reveal how performance between one group of students differs from another group.
Disaggregators include the following: Race Ethnicity Gender Special Education Status Lunch Status (Income Level) English Proficiency District Grade School
Limitations: Disaggregating small groups of students can lead to subgroups with only a few students. Caution must be used when making inferences from disaggregated data.
Teacher and Teacher Qualifications Program information Mobility Attendance Rates Discipline Infractions and
Consequences Course-taking Patterns Years in the School/District Retention
NC
LBS
ubgr
oups
How does performance differ from one group of students to
another?
Disaggregation (cont’d)State Assessment Performance
5th Grade Math, 2005-2006Washington Elementary School
16% 26% 35%11%
18%12%
-32% -31% -29%
-42%-26% -24%
-100%
-80%
-60%
-40%
-20%
0%
20%
40%
60%
80%
100%
Pe
rce
nt
of
Stu
de
nts
Proficient Advanced Needs Improvement Warning
My School(n=343)
My District(n=23,413)
My State(n=126,845)
(Graph Type: Floating Column)
Disaggregation (cont’d)
(Graph Type: Bar of Pie)
Proficiency by Years in District5th Grade Math Assessment, 2005-2006
Washington Elementary School
Advanced7%
Proficient32%
2 years16%
3 years13%
4 years7%
> 4 years7%
Warning orNeeds Improvement
78%
<= 1 year18%
Disaggregation (cont’d)
(Graph Type: Bar of Pie)
Terra Nova Math PerformanceClass of 2010, Disaggregated by Gender
Washington Elementary School
63 6568
71 6972 71
6259 61 63 65
69 6868 69 7175
77 77 76
0
10
20
30
40
50
60
70
80
90
2000(n=34)
2001(n=34)
2002(n=34)
2003(n=34)
2004(n=34)
2005(n=34)
2006(n=34)
Ave
rage
NC
E S
core
My District Male Female
Data Don’ts
Data can be dangerous! You should avoid:
Comparing performance on tests that have not been aligned; for example: Don’t compare 3rd grade scale scores to 5th grade scale scores Don’t compare 3rd grade Math scale scores to 3rd grade ELA scale scores
Making large inferences from a few data points; for example: Be wary of conclusions about a subject area based on one item on a test Be wary of conclusions about a student’s overall level based on
performance on one test Be wary of conclusions about a student’s strengths or weaknesses based
on performance on one item on one test
Questions?