Fast-tracking New Item Types into an · Fast-tracking New Item Types into an Existing Exam: Using...
Transcript of Fast-tracking New Item Types into an · Fast-tracking New Item Types into an Existing Exam: Using...
![Page 1: Fast-tracking New Item Types into an · Fast-tracking New Item Types into an Existing Exam: Using Cognitive Labs to Evaluate Item Types and Inform Decisions. ... A quantitative fill-in-the-blank.](https://reader035.fdocuments.in/reader035/viewer/2022070816/5f1040007e708231d4482e2a/html5/thumbnails/1.jpg)
![Page 2: Fast-tracking New Item Types into an · Fast-tracking New Item Types into an Existing Exam: Using Cognitive Labs to Evaluate Item Types and Inform Decisions. ... A quantitative fill-in-the-blank.](https://reader035.fdocuments.in/reader035/viewer/2022070816/5f1040007e708231d4482e2a/html5/thumbnails/2.jpg)
Fast-tracking New Item Types into an Existing Exam: Using Cognitive Labs to Evaluate Item Types and Inform Decisions
![Page 3: Fast-tracking New Item Types into an · Fast-tracking New Item Types into an Existing Exam: Using Cognitive Labs to Evaluate Item Types and Inform Decisions. ... A quantitative fill-in-the-blank.](https://reader035.fdocuments.in/reader035/viewer/2022070816/5f1040007e708231d4482e2a/html5/thumbnails/3.jpg)
Your Presenters
3
Jared Zurn, AIANational Council of Architectural Registration Boards
Susan Davis-Becker, PhD.Alpine Testing Solutions
![Page 4: Fast-tracking New Item Types into an · Fast-tracking New Item Types into an Existing Exam: Using Cognitive Labs to Evaluate Item Types and Inform Decisions. ... A quantitative fill-in-the-blank.](https://reader035.fdocuments.in/reader035/viewer/2022070816/5f1040007e708231d4482e2a/html5/thumbnails/4.jpg)
Session Objectives
4
• Understand the challenges associated with pretesting new item types and presentation formats on an existing exam.
• Gain insight into how a more agile approach can allow for an iterative approach to test design.
• Understand how cognitive labs can be designed and carried out to predict performance aspects of new item types and formats.
• Be able to interpret the results of a cognitive lab to inform examination decisions and predict how new item types will achieve identified measurement goals.
![Page 5: Fast-tracking New Item Types into an · Fast-tracking New Item Types into an Existing Exam: Using Cognitive Labs to Evaluate Item Types and Inform Decisions. ... A quantitative fill-in-the-blank.](https://reader035.fdocuments.in/reader035/viewer/2022070816/5f1040007e708231d4482e2a/html5/thumbnails/5.jpg)
NCARB’s Exam Change
5
“No one to tell us noOr where to goOr say we're only dreaming”
![Page 6: Fast-tracking New Item Types into an · Fast-tracking New Item Types into an Existing Exam: Using Cognitive Labs to Evaluate Item Types and Inform Decisions. ... A quantitative fill-in-the-blank.](https://reader035.fdocuments.in/reader035/viewer/2022070816/5f1040007e708231d4482e2a/html5/thumbnails/6.jpg)
Challenges & Solutions
6
• Urgency to “Get it Done”
• Lack of details early
• Understanding new items
• Unbiased pretesting
• Agile Approach
• Open Communication
• Cognitive lab
• Post-delivery analysis
![Page 7: Fast-tracking New Item Types into an · Fast-tracking New Item Types into an Existing Exam: Using Cognitive Labs to Evaluate Item Types and Inform Decisions. ... A quantitative fill-in-the-blank.](https://reader035.fdocuments.in/reader035/viewer/2022070816/5f1040007e708231d4482e2a/html5/thumbnails/7.jpg)
Cognitive Lab Design7
Think Aloud Study
Timing Study
![Page 8: Fast-tracking New Item Types into an · Fast-tracking New Item Types into an Existing Exam: Using Cognitive Labs to Evaluate Item Types and Inform Decisions. ... A quantitative fill-in-the-blank.](https://reader035.fdocuments.in/reader035/viewer/2022070816/5f1040007e708231d4482e2a/html5/thumbnails/8.jpg)
Part 1: Think Aloud Study
8
Who & What• Mock candidates (RLAs)• Exam “form” included 1 case study
(+ items) and new item types• Candidates were asked to verbally
report (think aloud) as they worked through each item
HowObservers record process, thoughts, points of confusion, sources of challenge
![Page 9: Fast-tracking New Item Types into an · Fast-tracking New Item Types into an Existing Exam: Using Cognitive Labs to Evaluate Item Types and Inform Decisions. ... A quantitative fill-in-the-blank.](https://reader035.fdocuments.in/reader035/viewer/2022070816/5f1040007e708231d4482e2a/html5/thumbnails/9.jpg)
A Volunteer Opportunity• A VERY BRIEF Think Aloud Session
• FYI: It’s a math question
![Page 10: Fast-tracking New Item Types into an · Fast-tracking New Item Types into an Existing Exam: Using Cognitive Labs to Evaluate Item Types and Inform Decisions. ... A quantitative fill-in-the-blank.](https://reader035.fdocuments.in/reader035/viewer/2022070816/5f1040007e708231d4482e2a/html5/thumbnails/10.jpg)
Think Aloud: Example
10
A machine packs boxes at one box every 45 seconds (3/4 of a minute). How many boxes can this machine pack in ten minutes?
Boxes
A quantitative fill-in-the-blank
![Page 11: Fast-tracking New Item Types into an · Fast-tracking New Item Types into an Existing Exam: Using Cognitive Labs to Evaluate Item Types and Inform Decisions. ... A quantitative fill-in-the-blank.](https://reader035.fdocuments.in/reader035/viewer/2022070816/5f1040007e708231d4482e2a/html5/thumbnails/11.jpg)
Think Aloud Findings
11
• Preparation is key! Topics and content for advance materials
• Interface: Image display, response functionality, tools
• Case Studies: Use of case resources, ideas for training materials, relationship of items to cases
• New item types: Sources of challenge/confusion
![Page 12: Fast-tracking New Item Types into an · Fast-tracking New Item Types into an Existing Exam: Using Cognitive Labs to Evaluate Item Types and Inform Decisions. ... A quantitative fill-in-the-blank.](https://reader035.fdocuments.in/reader035/viewer/2022070816/5f1040007e708231d4482e2a/html5/thumbnails/12.jpg)
Part II: Timing Study
12
Who & What• Mock candidates (RLAs)• Exam “form” included multiple items
with known parameters and a collection of new items
• Candidates were introduced to the exam and allowed to test interrupted
HowTiming data was captured for timing analysis and candidates completed a post-event survey.
![Page 13: Fast-tracking New Item Types into an · Fast-tracking New Item Types into an Existing Exam: Using Cognitive Labs to Evaluate Item Types and Inform Decisions. ... A quantitative fill-in-the-blank.](https://reader035.fdocuments.in/reader035/viewer/2022070816/5f1040007e708231d4482e2a/html5/thumbnails/13.jpg)
Timing Study Findings
13
• Candidates spent longer than in the operational exam
• Estimates of case study review time
• Refined our estimates of average time by item type
![Page 14: Fast-tracking New Item Types into an · Fast-tracking New Item Types into an Existing Exam: Using Cognitive Labs to Evaluate Item Types and Inform Decisions. ... A quantitative fill-in-the-blank.](https://reader035.fdocuments.in/reader035/viewer/2022070816/5f1040007e708231d4482e2a/html5/thumbnails/14.jpg)
Challenges Encountered
14
• Study Design» Putting together the right selection of content» Coordinating work
• Execution» Recruiting candidates» Interpretation of findings, generalization
![Page 15: Fast-tracking New Item Types into an · Fast-tracking New Item Types into an Existing Exam: Using Cognitive Labs to Evaluate Item Types and Inform Decisions. ... A quantitative fill-in-the-blank.](https://reader035.fdocuments.in/reader035/viewer/2022070816/5f1040007e708231d4482e2a/html5/thumbnails/15.jpg)
Summary and Recommendations
15
• Cognitive labs» Plausible option to evaluate format and content» Provide value resources to the development team and
program leaders
• Guidance for implementation» Make this part of your overall test development plan» Conduct when you feel comfortable with content
development progress but also have time to implement changes
![Page 16: Fast-tracking New Item Types into an · Fast-tracking New Item Types into an Existing Exam: Using Cognitive Labs to Evaluate Item Types and Inform Decisions. ... A quantitative fill-in-the-blank.](https://reader035.fdocuments.in/reader035/viewer/2022070816/5f1040007e708231d4482e2a/html5/thumbnails/16.jpg)
Questions?