Presented at the December 2010 meeting of theMichigan State Board of Education

December 7, 2010

MEAP Writing Update

2

Moved from grades 3-8 to grades 4 & 7 Added more points to improve reliability

15, 20, and 23 points (in 2006, 2007, and 2008, respectively) 2009 = Field tested new assessment 2010 = 50 points (grades 4 & 7 only)

Added analytic rubrics Scoring multiple facets of student writing samples

independently Gives more and targeted feedback to schools and students

Standardized the question asked of students on the “Peer Response to a Student Writing Sample” essay

Added Informational Writing to the assessment Maintained Narrative Writing on the assessment Divided the New Writing assessment over two days

December 7, 2010

MEAP Writing Update

3

Essay in Narrative Style 15 points Essay in Informational Style 15 points Multiple Choice Response to Student Writing Sample #18

points Multiple Choice Response to Student Writing Sample #2 8

points Essay Response to Student Writing Sample 4 points Total 50 points

Total from essays 34 points

Percent of total from essays 68

December 7, 2010

MEAP Writing Update

4

Standardized Question “What could the writer do to improve this sample?”

Improves consistency of scoring from year to year Responsive to grade level content expectations

regarding student understanding of the writing process, including such aspects as revision for sequence, flow, style, clarity, coherence, and consistency; proofing; editing; etcetera.

New Holistic Rubric 4 points Scored on whether the student adequately, and in an integrated

manner, gave recommendations for improvement using relevant and specific details from the sample to support the recommendations, and an explanation of why their recommendations improve the sample

December 7, 2010

MEAP Writing Update

5

Write about change:

Your life changes as you get older. For example, you can now do things you could not do before. Think about things that have changed in your life. Write about change.

Do only one of the following:

Write a story about a time when you had to make a changeOR

Write a story about someone you know that had to make a change

ORWrite a story in your own way about change

Includes a checklist for the writers to check their work against

December 7, 2010

MEAP Writing Update

6

Responsive to grade level content expectations regarding Creation of a narrative Use of ideas consistent with narrative Organization consistent with narrative Personal style Grammar and usage Spelling

December 7, 2010

MEAP Writing Update

7

Analytic Rubric (15 points total—ideas doubled)

December 7, 2010

MEAP Writing Update

8

Example prompts Grade 4

“Animals are interesting. Think about two animals. What do they look like? Where do they live? What do they eat? What else do you know about them? Compare two animals.”

Grade 7 “What effect does watching television have on study

habits?”

The prompts include checklists for high quality writing for the writers to check their work against

December 7, 2010

MEAP Writing Update

9

Responsive to grade level content expectations regarding Creation of an informational text Use of ideas consistent with informational

text Organization consistent with informational

text Personal style Grammar and usage Spelling

December 7, 2010

MEAP Writing Update

10

Analytic Rubric (15 points total—ideas doubled)

December 7, 2010

MEAP Writing Update

11

Goals for New Reports Represent best practices in writing

instruction and assessment Provide rich feedback regarding student

writing skills in order to… Aid teachers in designing effective instruction

for individual students Assess curriculum needs Help parents and guardians understand the

writing progress of students

December 7, 2010

MEAP Writing Update

12

New Overall Writing Scale Score Overall Writing Performance Level Item scores for 16 Multiple Choice

responses to student writing samples Revised holistic (0-4) score for Essay

response to student writing sample Analytical score for Essay response to

narrative writing task (15 points total) (0-3) Ideas, doubled (0-3) Organization (0-3) Style (0-3) Conventions

Analytical score for Essay response to informational writing task (15 points total)

(0-3) Ideas, doubled (0-3) Organization (0-3) Style (0-3) Conventions

Total of 27 reported elements per student

Total of 50 points

Old Overall Writing Scale Score Overall Writing Performance Level Item scores for 13 Multiple Choice

responses to student writing samples Holistic (0-4) score for Essay

response to student writing sample

Total of 17 reported elements per student

Total of 23 points

December 7, 2010

MEAP Writing Update

13

Students have taken the MEAP Writing Assessment Assessments have been returned to the contractor

(Measurement Inc.) for processing Processing, scoring, and data file creation is nearly

complete All MEAP results (minus Writing) will be released in data

files to schools as soon as data file creation is complete and validated (released last year in mid December)

Standard setting activity will occur for the new writing assessment in January, 2011

Results of standard setting will be reviewed and validated by our Technical Advisory Committee of national measurement experts

Results of standard setting will be presented to the State Board of Education at the February, 2011 meeting

Pending SBE action at the February meeting, all reports of MEAP results will be finalized and released to schools and the public as soon as possible

December 7, 2010

MEAP Writing Update

14

A “Modified Body of Work” procedure will be used for standard setting to identify cut scores on the writing test

Three cut scores will be set to identify four ranges on the score scale representing student achievement in writing as follows:

The process will cover three days with training on day 1, and multiple rounds of standard setting activities on days 2 and 3

Roughly 20-25 panelists will be recruited to serve in each of grades 4 and 7

Panelists will be representative of Michigan education stakeholders and will include teachers, administrators, parents, and other business or community leaders

NotProficient

PartiallyProficient

Proficient Advanced

December 7, 2010

MEAP Writing Update

15

Before Round 1 Panelists will take the assessment Panelists will review “Performance Level

Descriptors” (PLDs) Descriptions of what students in each of the four

performance levels should know and be able to do Panelists will become intimately familiar with the

PLDs All subsequent sorting of student work samples

will be based on their match to specific PLDs Panelists will receive training on the standard

setting process

December 7, 2010

MEAP Writing Update

16

Panelists will review a large set of student work samples Each work sample will include responses to all tasks on the

assessment (or the students’ bodies of work) The complete set of bodies of work will span the full range of

scores observed on the assessment The complete set of bodies of work will be strategically

selected to have desirable statistical and qualitative features

Particular attention will be given to the students’ essays

Panelists will examine the student work samples, the scores that students received on those work samples, and their compatibility with the Performance Level Descriptors (PLDs) to provide recommendations for cut scores.

December 7, 2010

MEAP Writing Update

17

Statistical analyses will be run to identify the three group cut scores from the cut scores that each panelist recommended in round 1

Feedback information will be provided that shows the cut score recommendations for each panelist and the group cut scores Panelists will have the opportunity to discuss

the feedback information prior to round 2

December 7, 2010

MEAP Writing Update

18

Panelists will again examine student work samples and the scores that student received of those work samples to provide cut score recommendations. However, in this round, the selected bodies of work… Will not span the entire range of student achievement on the

assessment Will be targeted at, just below, and just above the three

recommended cut scores resulting from round 1

Panelists will also be provided with information on relationship of the scores on these new work samples to the work samples that they rated in round 1.

The targeted selection allows for fine tuning of the three cut scores

December 7, 2010

MEAP Writing Update

19

Statistical analyses will be run to identify the refined (round 2) cut score recommendations

Feedback information will again be provided that shows the cut score recommendations for each panelist and the group cut scores

Statistical analyses will also be run to provide impact data to the panelists (e.g., the percentage of students in each of the four categories and that would be at or above each of the three cut scores)

December 7, 2010

MEAP Writing Update

20

Panelists will receive impact information and get to see how the impact information is related to the student work samples that they that examined in rounds 1 and 2

Vertical Articulation Grade 4 and 7 panelists will meet together to make sure that

the recommended cut scores are appropriately articulated across the two grades

Information on high school writing cut scores will also be incorporated to assure coherence between grade 4, 7, and 11 writing cut scores

Grade 11 writing cut score is already very similar in difficulty to the ACT College Readiness Benchmark in English

Panelists will be reminded that cut scores should be tied to the Performance Level Descriptors

Panelists will make their final recommendations regarding cut scores

December 7, 2010

MEAP Writing Update

21

We look forward to bringing for your approval the recommended cut scores on this new and much more informative MEAP writing assessment

December 7, 2010

MEAP Writing Update

22

Joseph Martineau Director Office of Educational Assessment &

Accountability Michigan Department of Education

martineauj@michigan.gov 517-241-4710