Center for Applied Linguistics 1 The Item-Writing Process for ACCESS for ELLs Online Involvement by...
-
Upload
jordan-skinner -
Category
Documents
-
view
217 -
download
0
Transcript of Center for Applied Linguistics 1 The Item-Writing Process for ACCESS for ELLs Online Involvement by...
1 Center for Applied Linguistics
The Item-Writing Process forACCESS for ELLs™ Online Involvement by ESL Teachers
January 2005
Illinois 28th Annual Statewide Conference for Teachers Serving Linguistically and Culturally Diverse Students
Jim BaumanCenter for Applied LinguisticsWashington, DC
2IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
Presentation Outline
The WIDA Assessment Framework
Structure of ACCESS for ELLs™
The Item Writing Course
3IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
WIDA’s Foundation
Standards
Clear expectations for students and schools
Motivation towork hard
ProfessionalDevelopment
Improvedteaching
Higher levelsof learning
Adapted from:
National Research Council, 1999 Testing, Teaching, and Learning:A Guide for States and School Districts
Assessment
Accountability
4IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
WIDA Enhanced Assessment System
Specifications
StandardsTask Blueprint
ELP StandardsClassroom
Assessment
State ContentStandards
Lesson Plan
AssessmentFramework
ELP StandardsLarge-scaleAssessment
Specifications
Test Blueprint
English LanguageProficiency Test
(ACCESS for ELLS)
Alternate Assessmentof Academic Achievement
(Alternate ACCESS)Classroom Assessments
5IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
Organization of the WIDA Standards
Frameworks for Classroom & Large-Scale Assessment (2)
English Language Proficiency Standards (5)
Language Domains (4)
Grade Level Clusters (4)
Language Proficiency Levels (5)
Model PIs are the lowest level ofexpression of the standards
Model Performance Indicators (>800)
SI
LA
MA
SC
SS
L
R
W
S
K-2
3-5
6-8
9-12
1
2
3
4
5
6IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
WIDA Standards ExamplesSocial Studies, Listening, 6-8
English language learners process, understand, interpret, and evaluate spoken language in a variety of situations.
Level 1: Identify icons on maps or graphs from oral statements.
Level 3: Categorize resources or products of regions from oral descriptions.
Level 5: Draw conclusions about resources or products in various regions based on oral descriptions.
Social & Instructional, Writing, 6-8English language learners engage in written communication in a variety of forms for a variety of purposes and audiences.
Level 4: Suggest ideas for making changes in school, such as rearranging a schedule or adding subjects.
Language Arts, Reading, 6-8
English language learners process, interpret, and evaluate written language, symbols, and text with understanding and fluency.
Level 5: Draw conclusions from explicit and implicit text.
7IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
Aligning State Content Standards with WIDA ELP Standards
MIDDLE/JUNIOR HIGH SCHOOL Illinois State Social Science Standards
STATE GOAL 17: Understand world geography and the effects of geography on society, with an emphasis on the United States.
17.B.3a Explain how physical processes including climate, plate tectonics, erosion, soil formation, water cycle, and circulation patterns in the ocean shape patterns in the environment and influence availability and quality of natural resources.
17.C.3b Explain how patterns of resources are used throughout the world.
17.A.3b Explain how to make and use geographic representations to provide and enhance spatial information.
WIDA Social Studies Standards — Listening
English language learners process, understand, interpret, and evaluate spoken language in a variety of situations.
Level 1: Identify icons on maps or graphs from oral statements.
Level 3: Categorize resources or products of regions from oral descriptions.
Level 5: Draw conclusions about resources or products in various regions based on oral descriptions.
8IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
Presentation Outline
The WIDA Assessment Framework and Standards
Structure of ACCESS for ELLs™
The Item Writing Course
9IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
Structure of ACCESS for ELLs
Grade Level and Tier
K
1-2
3-5
6-8
9-12
A (adaptive)
A B C
A B C
A B C
A B C
100 (roll-out Spring 2005)
999 (used to produce screener)
200 (roll-out Spring 2006)
Listening — group admin, machine scored
Reading — group admin, machine scored
Speaking — individual admin, adaptive, TA scored
Writing — group admin, rater scored
Domains
Forms
10IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
Theme and Item Structure
Theme
Item Model
Item Stimulus Question and Response Options
11IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
Expression of the Thematic Folder
• Orientation• Contextualizes the items• Engages background knowledge
• Stimulus — Text and/or Graphic• Presents new information• Organizes information visually• Reduces memory load
• 3 or more items at same or scaffolded proficiency levels
• Item levels determined by specs• Address single strand of PIs
12IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
Sample Listening ThemeScript: Look at the map. It shows some of the important economic activities in different regions of modern Russia. Each symbol in the legend shows a different economic activity. A fish indicates fishing, a factory heavy industry, logs forestry, wheat agriculture, a pick and shovel mining, and an oil rig petroleum.
The Russian Economy
13IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
Sample Listening ItemsScript: Find the symbol for oil. 1
Western Siberia Eastern Siberia Far EastUrals
3
Script: Russia has many kinds of natural resources. These include coal and iron ores from mining, oil from drilling, and wood from logging. Which region in Russia has the greatest variety of natural resources?
Script: The part of Russia from Siberia east is extremely cold for much of the year and is sparsely populated. The western part of the country, particularly in the south, has a milder climate and a longer growing season. Almost 80 percent of the population lives in the western part.
Since factories require a large labor force to run them but they also need raw materials and energy, where would you probably find the greatest concentration of Russia’s heavy industry?
In the southern part of the country where food is plentiful
In the northern part of the country where the population is large
In the eastern part of the country where raw materials are abundant
In the middle part of the country where raw materials and people are abundant
5
14IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
Sample Writing Theme
Changes in Cafeteria MenuItem Old Price New Price
Hamburger 1.25 1.50
Soda .45 .60
Milk .50 .25
Apple .50 .40
Banana .40 .30
Pie .90 1.00
Ice Cream .80 1.10
Yogurt 1.00 .80
The principal at your school has decided to change the prices of some foods in the school cafeteria. Some students like the changes and some do not.
15IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
Sample Writing Item
Look at the new cafeteria price list. Suggest changes you would make to the prices for these or other foods.
Write a short paragraph of at least 8 sentences. Write about changes you would make to cafeteria prices.
16IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
Goals of an Item Specification
• State the assessment objective
• By grade• By PI
• Describe the item stimulus• Use of graphics/text• Mode of presentation to
test taker
• Describe the test taker’s task• Method of selecting a
correct response• Method of constructing a
scorable response• Properties of response
options
Assessment Objective
Assessment tasks will allow ELLs in the 6th to 8th grade to demonstrate the ability to identify icons on maps or graphs from oral statements by selecting the corresponding picture.
Stimulus Description
Stimulus is a simple oral statement or command (no more than one sentence) that requires the students to identify an icon (e.g., stars, trees, compass rose, ) on a map. The teacher will read the stimulus aloud from the test administration book. The stimulus will not be printed in the student's test booklet.
Task Description
Description:Given four pictures on a map, the student selects the picture that matches the oral statement. The map can be of a fictitious country.Response Options:There are four options.Options consist of pictures of icons on a map. There is a single, correct answer. One picture is the key; the others are distractors representing other icons commonly confused with the key.
Social Studies, Listening, 6-8, Level 1
17IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
Life Cycle of a Test Item
1. Receive theme & items from item writer
2. Submit items to external review
3. Commission illustrations
4. Check thematic folder for internal consistency
5. Submit folder and items to bias review
6. Layout folder in test form
7. Field test to establish item properties
8. Operationalize for one year
9. Retire the folder and items
18IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
What the Item Writer Does
Read the scopeof the item
in the specsstatement Read the stimulus
description & identify an appropriate content area
Write ordescribe
the themestimulus and/or
modelWrite the item
stimuli orprompts
Read the taskdescription &
write responseoptions
for the items
R e v i e w &RevIse
19IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
Item Creation & Review
Item Writers
InternalReviewers
Bias & ContentReviewers
Specifications
Raw Items
Cleaned Items
ApprovedItems
(ready for field testing)
20IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
Raw Item
Selected-Response Item
Language Arts, Reading, Grade 6-8, Level 5
In your test booklet are some sentences that tell why something happened in the story I will read.
Orientation
Leo and his father went to the grocery store. They bought food for a party they are having. At the grocery store, they picked out a cake and some ice cream. When they got home, Leo’s father put the cake on the table in the kitchen and walked away. Then, Leo’s dog, who had not eaten all day, came into the kitchen and ate the whole cake!
Theme Stimulus
A. Because he was hungry
B. Because he was a bad dog
C. Because Leo told him to eat it
D. Because he was thirsty
Options
A. Because he was hungryKey
Why did Leo’s dog eat the cake? Question
None Stimulus
Performance Indicator
Draw conclusions from explicit and implicit text
Assessment Objective
Assessment tasks will allow ELLs in the 6th to 8th grade to draw a text supported conclusion from a reading passage by selecting the corresponding text option.
Stimulus Description
Stimulus is a fictional story of 4 to 6 sentences, some of which may show complex sentence structures. Vocabulary is familiar.
Specification (in part)
21IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
Cleaned Item
Item is reconceived as part of a thematic folder with items at levels 3, 4, and 5 (only level 5 is shown here)
The Brown family is going to have a party for their friends and neighbors to celebrate the Fourth of July.
Orientation
Generic picture of a family making preparations for a party.
Theme Stimulus
A. Because he was hungry
B. Because he was happy
C. Because Leo took him to the grocery store
D. Because Leo did not buy him dog food
Options
A. Because he was hungryKey
Why did Rambo eat the cake? Question
Leo and his father went to the grocery store to buy food for the party, but they forgot to feed their dog Rambo before they left. At the grocery store, they picked out a cake, some ice cream, and soda. When they got home, Leo put the food on table in the kitchen and went to help his father. When he got back, he was horrified to see that Rambo had his nose in the cake, happily eating away.
Item Stimulus(Level 5)
22IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
Approved Item
Item after review by content and bias committee.
The Rodriquez family is going to have a party for their friends and neighbors to celebrate Cinco de Mayo, an important holiday in Mexico.
Orientation
Generic picture of a family making preparations for a party. Family members should look Hispanic.
Theme Stimulus
A. Because he was hungry
B. Because he was happy
C. Because Leo took him to the grocery store
D. Because Leo did not buy him dog food
Options
A. Because he was hungryKey
Why did Cabo eat the cake? Question
Leo and his father went to the grocery store to buy food for the party, but they forgot to feed their dog Cabo before they left. At the grocery store, they picked out a cake, some ice cream, and soda. When they got home, Leo put the food on table in the kitchen and went to help his father. When he got back, he was horrified to see that Cabo had his nose in the cake, happily eating away.
Item Stimulus(Level 5)
23IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
The Review Process
• Check for fit to PIs• Does length of stimulus match spec?• Does linguistic complexity of stimulus match spec?• Is vocabulary appropriate to spec?
• Check response options properties• Is there only one correct response option?• Are distractors motivated and equally attractive?• Are response options balanced in length & complexity?
• Check for inter-item balance & integrity• Are items consistent with theme?• Are items independent?
• Check for content accuracy
• Check for language, culture, and gender bias
24IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
Presentation Outline
The WIDA Assessment Framework and Standards
Structure of ACCESS for ELLs™
The Item Writing Course
25IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
Course Goals
• Understand and apply established principles of language test construction, including notions of test validity and reliability
• Develop test items which accord with guidelines established for determining item difficulty, bias and authenticity
• Develop test items which accurately address a set of predetermined item specifications (the WIDA specifications), including writing test prompts and scoring rubrics
• Develop test items across the range of item formats designated by the WIDA specifications
• Develop test items that target specific levels of English language proficiency
• Develop test items that are aligned with the functional language requirements of social and classroom language and of the content areas: English language arts, mathematics, science and social studies.
26IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
Course Work Requirements
• Study and apply the specifications for the participant’s designated grade level
• Write test items in each of the four language domains for the designated grade level
• Communicate with the class on substantive issues via discussion boards
• Communicate with the class on procedural issues via chat room
• Interact with other members of designated grade level
• Take and pass quizzes and final exam
27IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
Course Participants
Lead Instructor: Jim Bauman
Associate Instructors: Jessica Motz & Ellen Parkhurst
Teaching Assistants: George Motz (1-2), Lee Gough (3-5), Jennifer Himmel (6-8), Ellen Daniels (9-12)
Active Participants (28) From IL:
Marion Flaman—SD54Michael Safina—CCSD59Diane Wilkin—SD99Delia Rodriguez—CUSD300Irene Pelc—CCSD15Deb Grant—HSC220
Auditors (5)
28IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
Course Grading Structure
Item submission 60%
Participation in chat sessions and discussions 20%
Quizzes 20%
Successful completion of course requirements at an 80% level earns
2 university credits from the University of Wisconsin
29IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
Course Home Page
CourseMenu
30IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
Structure of the Course
31IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
Structure of a Module
Each module contains:
• Readings
• Self-Paced Exercises
• Interactive Exercises• Discussions• Chats
• Assignments
• Quizzes
32IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
Documents
33IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
Discussion Boards
34IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
Discussion Board Results
Discussion Forum Responses Pages Words
Exploring the item specifications
41 15 3,000
Sources of difficulty in listening items
106 36 10,700
How to evaluate constructed responses
115 43 13,700
Assessing language vs assessing content
85 34 11,725
Sources of reading errors 97 35 9,225
Total 444 163 48,350
35IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
Completing an Assignment
1. Download appropriate set of item specs from Documents area
2. Download data entry form from Documents
3. Conceive & research folder and items
4. Complete item data entry form
5. Submit completed form to Dropbox
6. Revise per instructor’s feedback
7. Resubmit to Dropbox
Specification
BlankData Entry
Form
Content Standards
Dropbox
CompletedData Entry
Form
36IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
The Dropbox
37IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
Test Item Results
Number of Items Produced
Standard
Language Domain GradeLevel LA MA SC SI SS
Grand Total
Listening 1-2 39 32 38 53 21 183
3-5 49 56 51 108 33 297
6-8 60 39 44 83 42 268
9-12 43 34 29 65 15 186
Listening Total 191 161 162 309 111 934
Reading 1-2 28 25 20 51 11 135
3-5 48 44 45 89 21 247
6-8 39 34 25 71 19 188
9-12 26 13 19 38 12 108
Reading Total 141 116 109 249 63 678
Grand Total 332 277 271 558 174 1612
38IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
What Teachers Got as Item Writers
• University credit or CEUs
• Direct experience with on-line learning
• Professional development on the WIDA standards and writing items for large-scale assessments
• Informed discussion on how to apply WIDA standards to the curriculum
• Professional contacts with other teachers across the consortium
• The deep gratitude of the ACCESS for ELLs™ development staff!!
39IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
What WIDA Got from Teacher Item Writers
• Approximately 1600 raw items
• A nucleus in each state of WIDA informed staff
• A pool of talent to serve as test coordinators, administrators, and future item reviewers
• Valuable insight on teachers’ perspectives on standardized testing
• Important clues on how to build professional development to apply the WIDA framework in the classroom
40IRC Conference Jan 24-27, 2005 ACCESS for ELLs™ Item Writing Jim Bauman [email protected]
Questions, Discussion, Feedback?