Post on 20-Jan-2016
description
The Application of ECD to the Redesign of Advanced Placement Exams
CCSSOJune 2012
Maureen Ewing, Kristen Huff, Amy Hendrickson, Pamela Kaliski
Why ECD for AP?
• Need targets of measurement and curricular requirements
• No guesswork on part of the test user, test maker or test taker about what is valued in the domain
• Lack of ambiguity and use of task models helps ensure stable measurement
• Enhances validity argument
Three Critical ECD Phases
Experts with Disciplinary Knowledge Unpacked the Content omains
•BIG IDEAS(LEVEL 1)
•ENDURING UNDERSTANDINGS (LEVEL 2)
•SUPPORTING UNDERSTANDINGS(LEVEL 3)
1
2 2 2
3
Experts with Disciplinary Knowledge Unpacked the Domains
•IDENTIFY SKILLS OR REASONING PROCESSES
•ARTICULATE DEFINITION OF SKILL (I.E., OBSERVABLE EVIDENCE)
1
2 2 2
3
Inputs to Domain Analysis
• National and state standards
• Latest research on student learning and assessment
• Curriculum Surveys
• Other relevant documents
• Expert judgment of panelists
Three Critical (re)Design Phases
Writing Claims and Evidence
• Claims start with the phrase “The student can” to reinforce the focus on what the student knows and can do
• Claims are comprised of content and/or skills that were identified in the domain analysis
• Evidence starts with the phrase “the work is characterized by…”
• Evidence includes nouns and adjectives (no verbs)
If it’s unobservable, it isn’t ‘evidence’
Examples of non-evidence:
• “a deep awareness of the importance of theorems”
• “understanding of differences and similarities between derivatives and integrals”
• What would a student have to do, say, or write in order to prove to you that he/she has a deep awareness of the importance of theorems? The answer to this question is the evidence.
• As evidence is written, necessary edits to the claims may be uncovered; iteration is key!
Achievement Level Descriptions – ALDs• Describe what students in each performance
category should reasonably know and be able to do
• Relate directly to the knowledge, skills, and abilities a student should know and be able to demonstrate to be classified into a particular score category
• Distinguish clearly from one performance category (e.g., AP Grade 2) to the next (e.g., AP Grade 3)
Three Critical (re)Design Phases
Achievement Level 3
Task Models: Take-home messages1) Claims and evidence are “fodder” for
item• Stimuli• Prompt• Response options• Rubrics
2) Many “like” items can be produced from same claim / evidence pair
Construct relevance as the target of measurement is
the claim/evidence
Item generation; Comparability
Form Assembly Specifications - Considerations• Multiple inputs
• Domain Model
• Experts’ ratings of importance of content and skills
• Psychometric criteria
• Structure of the domain
• Claims: skills-based versus integration of skills and content
• Content relationships
• Skill relationships
• Content and skill relationships
Challenges of ECD
• Iterative nature of work • Sometimes viewed as “doing it wrong” the first time, an
annoyance, but it IS the process• Resource and time intensive (6 years and counting),
especially development of task models based on iterative pilots
• Steep learning curve – difficult to think in terms of observable evidence
• Defining appropriate grain size for each piece
Lessons Learned and Ongoing Work
• We need to develop an evaluation process that specifically addresses the iterations between C&E, ALDs, etc
• Documents, captures, thinks about the iteration - this is the real work of figuring out the construct about how students learn as well as ECD
• We need to figure out the grain size issues and how everything fits together