1

New England Common Assessment Program

Guide to Using the October, 2006 NECAP

Reports:

Companion PowerPoint

Presentation February 2007

2

Welcome and Introductions

Mary Ann Snider

Director of Assessment and Accountability

RI Department of Education

3

Measured Progress Service Center: 1-877-632-7774

Welcome and Introductions

Tim Crockett

Assistant Vice-President

1-800-431-8901 x2106

tcrockett@measuredprogress.org

Harold Stephens

NECAP Program Director

1-800-431-8901 x2235

hstephens@measuredprogress.org

Amanda Smith

NECAP Program Manager – Grade 11

1-800-431-8901 x2259

asmith@measuredprogress.org

Elliot Scharff

NECAP Program Manager – Science

1-800-431-8901 x2126

escharff@measuredprogress.org

Josh Evans

NECAP Program Manager – Grades 3-8

1-800-431-8901 x2244

jmevans@measuredprogress.org

Tina Haley

NECAP Program Assistant

1-800-431-8901 x2427

chaley@measuredprogress.org

Jennifer Varney

NECAP Program Assistant

1-800-431-8901 x2115

jvarney@measuredprogress.org

Mellicent Friddell

NECAP Program Assistant

1-800-431-8901 x2355

mfriddell@measuredprogress.org

4

• To assist schools and districts in using the variety of reports and support material issued by the NECAP states in conjunction with the release of results

• To provide school and district personnel an opportunity to ask questions regarding the reports and support materials

Purpose of the Workshop

5

• Development of Grade Level Expectations

• Test Item Review Committees• Bias and Sensitivity Review Committees• Classroom Teacher Judgment data• Standard Setting Panelists• Technical Advisory Committee

Involvement of Local Educators

6

• Fall test – previous year’s GLEs• Reading and Mathematics – grades 3-8• Writing – grades 5 and 8• Variety of Item Types

• Reading: multiple choice and constructed response• Mathematics: multiple choice, short answer (one point), short answer

(two points), and constructed response• Writing: multiple choice, constructed response, and extended

response

• Common and Matrix Items

Basics of Test Design

7

Two Basic Types of Educational Tests

• Norm-Referenced: What is the student’s status or rank in a designated group?

• Criterion or Standards-Referenced: Has the student learned the skills or achieved the standard?

NECAP is a Standards-Referenced Test

8

The Family Educational Rights and Privacy Act (FERPA)

• Access to individual student results is restricted to:• the student• the student’s parents/guardians• authorized school personnel

• Superintendents and principals are responsible for maintaining the privacy and security of all student records.

• Authorized school personnel shall have access to the records of students to whom they are providing services when such access is required in the performance of their official duties.

• FERPA website: http://www.ed.gov/policy/gen/guid/fpco/ferpa/index.html

Family Educational Rights and Privacy Act

9

National Council on Measurement in Education

Educating Others

Evaluation and Research

Interpretation and Use

Scoring

Development

Marketing and Selling

Selection

Administration

www.ncme.org

10

Questions about the purpose of theworkshop and/or NECAP background?

Questions and Answers

11

Rhode Island Results

Suburban

Urban Fringe

Urban

12

Rhode Island Results

READING MATHEMATICS

2005 2006 2005 2006

All Students

58% 62% 50% 53%

Native Americans

46 49 33 38*

Asian 56 64* 52 59*

Black 35 41* 24 30*

Hispanic 31 38* 24 31*

White 68 71 59 62

Female 63 67 50 53

Male 54 58 50 54*

IEP 24 27 20 23

ELL 17 23* 14 20*

Poverty 38 44* 30 36*

* Indicates that gain exceeds the gain made by all students

13

• Student Report• Item Analysis Report• School/District Results Report• School/District Summary Report• District Student Level Data Files

Types of Reports

14

Student Report

• Two copies mailed to the school:• color copy for parents/guardians• black and white copy for local use

• Dual sided: text side and data side

• Spanish version of the student report available on the RIDE website for the Office of Assessment and Accountability

15

16

17

18

19

• Review to be sure you have a student report for every student tested in October, 2006.

• Prepare a letter to provide your interpretation of the results for your school and/or district.

• Be sure teachers are prepared and comfortable discussing individual student results with families; including any instructional interventions.

• Use your district data file to analyze growth between students you had last year and this year.

Fun with Student Level Reports!

20

Questions about the Student Report?

Questions and Answers

21

• Rationale• Inclusion of students in “Teaching

Year” reports

Teaching School vs. Testing School

22

Schools Can View Reports for Testing Year (2006-07)

Teaching School vs. Testing School

23

Teaching School vs. Testing School

Or Teaching Year(2005-06)

24

Testing Year(released this month)

Teaching Year (accountability)

25

Item Analysis Report

• Available to schools and districts on-line• These reports are confidential and the

website is password-protected. • One per content area• Use in conjunction with Released Items

documents

26

27

28

Released Item Documents

29

Grade 5 2006 Released Mathematics Item

DOK: 3

GLE: DSP 4.1

30

Grade 5 Practice Test Item (page 4)

DOK: 2

GLE: DSP 4.1

31

Fun with Released Support Materials!

• Build practice tests

• Share writing samples as models

• Mock scoring sessions to calibrate teacher judgments

• Compare released items (GLE and DOK) with classroom instruction and assessments and student work.

• Map instruction and scaffolding practices to items based on student performance.

32

Released Item Documents

• RIDE website at: http://www.ride.ri.gov/assessment/NECAP.aspx

• New information provided in the Released Item Support Materials documents includes:• For all content areas, providing the multiple-choice items

and the GLE code and text that each item is linked to• For math, indicating which items were included in session 1

of the math test by adding a “no calculator” symbol above the items

• For reading, listing the passage name and type of text• For writing, listing passage name and mode of writing

33

Questions about the Item Analysis Report or the Released Item documents?

Questions and Answers

34

• Available on-line at the Measured Progress website

• Separate report for each grade level tested

• District report contains no individual school data

School/District Results Report

35

http://iservices.measuredprogress.org

36

New thisyear

Please note:

You must always log in through this page to view confidential

reports and data

37

38

39

40

41

42

• Page 2 of the NECAP School/District Results Report

• Summary of participation and results

Grade Level Summary Report

43

44

• Review last year’s grade level summary report with this year’s

• If gains were made, identify what strategies, interventions, and programs may have worked

• If no progress was made, hypothesize why and what can be done at the school or district level to help students make progress

• Identify where your results differ from the district and state results. Hypothesize why they may differ

Fun with Grade Level Summaries!

45

Questions about the Grade LevelSummary Page?

Questions and Answers

46

Historical Data

• Data is included for two years and is combined in the Cumulative Total row

• In the Cumulative Total row for the achievement level columns:• The N columns are summed• The % columns are found by dividing the total

number of students in the level by the total number of students tested

• The Mean Scaled Score column is a weighted average

47

48

Comparisons of NECAP Scores Across Years

• School and District-Level Scaled Scores and Achievement Levels

• Student-Level Scaled Scores and Achievement Levels

• Content Area Subscores

49

• Pages 3, 5, and 7 of the NECAP School/District Results Report

• Provides information about achievement in subtopics of tested content areas

Content Area Results

50

51

Number of Scaled Score Points Denoting Minimally Statistically Significant Difference for Average Group Results*

Grade SubjectNumber of Students Tested in Group (Class, School etc.)

10 25 50 100 200

3Reading 6 4 3 2 1

Mathematics 5 3 2 2 1

4Reading 6 4 2 2 1

Mathematics 5 3 2 2 1

5

Reading 5 3 2 2 1

Mathematics 5 3 2 2 1

Writing 7 4 3 2 2

6Reading 6 4 3 2 1

Mathematics 6 4 3 2 1

7Reading 6 4 3 2 1

Mathematics 5 3 2 2 1

8

Reading 6 4 3 2 1

Mathematics 5 3 2 2 1

Writing 6 4 3 2 1

*Standard error of the mean difference with assumption n1=n2 and s1=s2

52

Percentage Difference in Student Achievement Level Classification Denoting Minimally Statistically Significant

Differences for Group Results*

Percentages of Students in Achievement Level(s)

Number of Students Tested in Group (Class, School etc.)

10 25 50 100 200

10 9 6 4 3 2

20 13 8 6 4 3

30 14 9 6 5 3

40 15 10 7 5 3

50 16 10 7 5 4

60 15 10 7 5 3

70 14 9 6 5 3

80 13 8 6 4 3

90 9 6 4 3 2

*0.05 level of statistical significance

53

Questions About the Content AreaResults Page?

Questions and Answers

54

• Pages 4, 6, and 8 of the NECAP School/District Results Report

• Provides information about achievement for different groups in a school/district

Disaggregated Content Area Results

55

56

• Compare last year’s Disaggregated Report with this year• Highlight in yellow, groups that had significant positive

increases• Highlight in blue, groups that remained stable or had decreases

• Establish cross-grade and content groups to hypothesize what the school can do to close gaps and maintain increases• Access to rigorous curriculum?• Professional development in a specific area?• Adequate support programs?• Adequate family involvement?

Fun with Disaggregated Results!

57

Questions About the Disaggregated Content Area Results Page?

Questions and Answers

58

• Separate report for each grade level tested

• Provides details about student achievement by content area

School/District Summary Report

59

School/District Summary Report

• The first line designates the year the test was administered

• The second line is the name of the report• The third line differentiates between

“teaching year” and “testing year”

60

61

• Are there consistent patterns of achievement across grades? Across content areas?

• Compare results to last year’s school summary• Highlight in yellow, areas where there has been

significant growth. Highlight in blue, areas where there has been no growth or decline.

• Establish cross-grade and content groups to hypothesize what the school can do to close gaps and maintain increases

• Develop a school/district plan to address areas of weakness and continue areas of growth

Fun with School Level Summary Data!

62

Student Level Data Files

• All demographic information for each student that was provided by the state

• The scaled score, achievement level, and subscores earned by each student in all content areas tested

• Also contain:• Performance on released items• Student questionnaire responses• Optional reports fields data

63

64

Supporting Materials

• Guide to Using the 2006 NECAP Reports and companion PowerPoint

• Grade Level Expectations• Test Specifications documents• Release Item documents• Preparing Students for NECAP: Tips for

Teachers to Share with Students• Technical Report