Scoring Open-Ended Literacy Items .
-
Upload
alban-riley -
Category
Documents
-
view
217 -
download
3
Transcript of Scoring Open-Ended Literacy Items .
Goals for the SessionGoals for the Session
Understand the importance of open-ended literacy items
Analyze prompts and rubrics for these itemsScoring tipsRelating Scoring to Classroom InstructionPractice scoring open-response reading
questionsPractice scoring writing samplesShare best practices
Importance of Open-Ended Importance of Open-Ended Literacy ItemsLiteracy Items
Reading Open-ended Questions◦1 Open Response question for each passage
◦Total of 3 for ACTAAP tests grades 3-8◦Total of 6 for Grade 11 Literacy Exam◦Account for 50% of score (24 points out of 48 for grades 3-8)
Importance of Open-Ended Importance of Open-Ended Literacy ItemsLiteracy Items
Writing Open-ended Tests◦2 questions (prompts) requiring essay-length responses
◦Account for 40 out of 48 points◦That is 83% of total writing score in grades 3-8
Importance of Open-ended Importance of Open-ended Literacy ItemsLiteracy Items
Open-ended questions account for 67% of the total literacy score
Students need to earn significant points in these items in order to achieve a proficient score
It’s critical that we score these items accurately so that the testing experience and data will be relevant in helping students improve
Few SimilaritiesFew Similarities
Reading open response questions and writing prompts have little in common
Frustrating because teachers want students to express themselves well in all written responses
Difficult for teachers and students to accommodate the differences
Reasons to ScoreReasons to Score
The two reasons to score open-ended items are often at cross-purposes1. To upload scores quickly to get data back
Scoring should be as accurate to the ACTAAP scoring as possible
With practice this can be done with a minimum of marks and time
2. To give meaningful feedback to students and parents
This scoring is more detailed, in-depth, and takes much longer
Scoring DecisionsScoring Decisions
District, or at least building, must decide whether to score interim assessments for quick, accurate scores to be uploaded for data purposes only or for meaningful in-depth scoring for feedback and reteaching
To do both will require delayed uploading, double scoring, or some creative time use
Scoring RequirementsScoring Requirements
Writing:5 domains on the state
scoring rubric are used to assess writing.
ContentStyleSentence FormationUsageMechanics Can the student
express himself in written form?
Reading: Content-specific
rubric is used to assess the reading open response question.
Scored for content only
Did the student read and understand?
Open Response Reading Open Response Reading QuestionsQuestions
Each reading passage has 1 open response question
The question will generally require more than one thinking skill, even at the lower grades:◦Analysis◦Inference◦Evaluation◦Comparison
Open Response Reading Open Response Reading QuestionsQuestions
Question is tied to the passage and requires information from the passage as support
It is a more in-depth question than can be covered by a multiple-choice format question.
Short answerScored on a 0 to 4 scaleTLI continues to use a B for Blank
Examples of Open Response Examples of Open Response QuestionsQuestions
Based on the information that you read in the passage, describe how Molly and her mother used Molly’s old crayons to make new crayons. Then, describe Molly’s new crayons. Include details from the passage.
Grade 3, 2006
Examples of Open Response Examples of Open Response QuestionsQuestions
Use information from the passage to describe two ways in which Jesse Owens influenced people. Be sure to use details from the passage to support each of your ideas or conclusions.
Grade 5, 2006
Examples of Open Response Examples of Open Response QuestionsQuestions
Compare and contrast Sylamore Horse Trail and Ozark Bicycle Trail. Use specific examples from the passage to support your response.
Grade 11, 2006
RubricsRubrics
Each open response reading question is scored by a specific rubric◦Can be subjective
Was the response thorough or only adequate?
◦A variety of tasks requested
Analysis of Open Response Analysis of Open Response Reading QuestionsReading Questions
Based on the information that you read in the passage, describe how Molly and her mother used Molly’s old crayons to make new crayons. Then, describe Molly’s new crayons. Include details from the passage.
What does “describe” really mean?How much description is enough?How many details do we include?
Analysis of Open Response Analysis of Open Response Reading QuestionsReading Questions
Use information from the passage to describe two ways in which Jesse Owens influenced people. Be sure to use details from the passage to support each of your ideas or conclusions.
Describe or tell?How many details should support each
idea?What conclusions?
Sample Rubric Sample Rubric
Score 4: The response clearly and thoroughly describes at least two ways in which Jesse Owens influenced people and supports each with relevant information from the passage. The response demonstrates comprehensive understanding of the task.
Sample Rubric Sample Rubric
Score 4: The response clearly and thoroughly describes at least two ways in which Jesse Owens influenced people and supports each with relevant information from the passage. The response demonstrates comprehensive understanding of the task.
New Open Response FormatNew Open Response Format
With the 2007 test a new format was begun by the state
Old format is still in useMost questions in 2007 and 2008 did
follow the new formatContinue to use old format in practiceTLI now uses the 4 point format
New 4-point Open Response New 4-point Open Response FormatFormat
These questions are written so that there are 4 tasks to perform
The rubrics show score points equal to the number of tasks correctly performed
4—all 4 tasks are completed correctly3—any 3 of the tasks are completed
correctly2—any 2 of the tasks are completed
Analysis of New Format ExampleAnalysis of New Format Example
Explain why it is important to guard against germs in the kitchen. Identify three ways to fight against infectious diseases in relation to food and its preparation.◦Explain why◦Identify way #1◦Identify way #2◦Identify way #3
Variations in RubricsVariations in Rubrics
Most rubrics seem to accept any 3 tasks to score a 3, any 2 tasks for a 2
Occasionally a rubric will be specific about the tasks.◦For a 4, the response explains and gives 3
examples◦For a 3, the response must explain and give 2
examples (3 examples without explanation may not be acceptable)
OR Short Answers EncouragedOR Short Answers Encouraged
Fragments OKCharts or Diagrams OKLists OK“Don’t be creative.” --ADEJust answer the questionsSome questions have parts that should be
labeled. Generally at lower grades.Some questions have charts. The missing
information should be listed in the answer box.
Respond to Match the QuestionRespond to Match the Question
Some questions are difficult to answer with lists
If the question suggests discussion, sentences may be a better approach
Complete paragraph construction is never necessary
Scoring should never take anything into consideration except what is specifically called for in the rubric (either positively or negatively)
Scoring Tips for Open Response Scoring Tips for Open Response ItemsItems
Scorers should be very familiar with the passage and the question
Scorers should work the “problem” themselves, preferably more than 1 way
TLI provides possible answer informationFollow the rubric exactly, even if the
response is unexpected. Give credit if tasks are accurately performed.
Scoring Tips for Open Response Scoring Tips for Open Response ItemsItems
Scan the response, looking for answer information
Put a check-mark in the margin for each task performed
Tasks do not have to be placed in the order requested
If question divides answer and requests labeling, no points should be removed if student does not label
Scoring Tips for Open Response Scoring Tips for Open Response ItemsItems
Score of B—blank, no attempt to answerPull these from the stack and score
◦Analyze why Did student run out of time? Did student waste time? Was student unaware of question? Did student refuse to attempt?
Does this reveal a hidden problem?
Score 0—Student attempted an answer but earned no points
Analyze why◦Ran out of time and jotted something down◦Did not understand the question◦Was uncertain of format
For all responses that earn points, look quickly at why points were not earned◦Only 3 tasks performed? Why not the other?◦4 tasks performed, but some incorrectly?
◦What could student do to earn just one more point?
More OR Scoring Tips More OR Scoring Tips
Do you know that open response question and rubric templates are on the TLI Web site?
The Scoring-Teaching ConnectionThe Scoring-Teaching Connection
DO teach students to analyze the Open Response prompt
DO teach the 4 types of Open Response SkillsDo expect “analysis” to be the underlying skillDO expect 4 tasks on each questionDO teach students to score their own work—
great practiceDON’T score for mechanics, sentence
formation, and the like
The Scoring-Teaching ConnectionThe Scoring-Teaching Connection
DO teach students to complete tasks in order
DO encourage labeling/numbering tasksDO work with students to find ways to
increase the score by 1 pointDO encourage students to look for clues
within the question DO encourage students to mark the
passage to locate answer information for the question
The Scoring-Teaching ConnectionThe Scoring-Teaching Connection
DO teach students to anticipate an OR question
DO teach students to write their own OR questions and rubrics
DO help students understand the difference between the reading open response and a writing assignment:
Short box= short answer= score for content only
The Scoring-Teaching ConnectionThe Scoring-Teaching Connection
DO teach students to consider their OR answers to be “prewriting;” do not need to develop the answer past this stage
DO teach students to use information from the passage; prior knowledge is rarely requested
DO encourage reading the questions first with students for whom that might work◦Search for answers as they read the text