Optimal Test Design With Rule-Based Item Generation
description
Transcript of Optimal Test Design With Rule-Based Item Generation
![Page 1: Optimal Test Design With Rule-Based Item Generation](https://reader034.fdocuments.in/reader034/viewer/2022051316/56814532550346895db1fa6f/html5/thumbnails/1.jpg)
Optimal Test Design With Rule-Based Item Generation
Hanneke Geerlings
Wim J. van der Linden
Cees A.W. Glas
![Page 2: Optimal Test Design With Rule-Based Item Generation](https://reader034.fdocuments.in/reader034/viewer/2022051316/56814532550346895db1fa6f/html5/thumbnails/2.jpg)
Item generation and automated test assembly
• Hierarchical item response theory– Second-level for each family of item parameters.
• Family calibration– New item generated from a family does not need
to be calibrated– Only family parameters are known, “family-
information function” replaces “item-information”– Depend on the degree of within-family item-
parameter variability across all families.
![Page 3: Optimal Test Design With Rule-Based Item Generation](https://reader034.fdocuments.in/reader034/viewer/2022051316/56814532550346895db1fa6f/html5/thumbnails/3.jpg)
Three different cases of item generation
• Test assembly from a pool of pregenerated individually calibrated items.– Baseline
• Test generation on the fly from pools with calibrated item families.
• Test generation on the fly using calibrated radicals that define the item families.
![Page 4: Optimal Test Design With Rule-Based Item Generation](https://reader034.fdocuments.in/reader034/viewer/2022051316/56814532550346895db1fa6f/html5/thumbnails/4.jpg)
Rule-Based Item Generation
• Radicals– Systematic use can help to ensure the content
validity of the items• Incidentals– Do not have any systematic effect on item
difficulty• Item cloning– Item with identical radicals but different
incidentals
![Page 5: Optimal Test Design With Rule-Based Item Generation](https://reader034.fdocuments.in/reader034/viewer/2022051316/56814532550346895db1fa6f/html5/thumbnails/5.jpg)
The features are radicals or incidentals?
• Coding existing items on the presence or absence of the specified radicals and performing– Exploratory
• Developers define their radicals and incidentals. Model checking is using.– Confirmatory
![Page 6: Optimal Test Design With Rule-Based Item Generation](https://reader034.fdocuments.in/reader034/viewer/2022051316/56814532550346895db1fa6f/html5/thumbnails/6.jpg)
Modeling Rule-Based Item Generation• Between family variation – radicals• Within family variation – incidentals• Multilevel response model– Item cloning model (Glas & van der Linden, 2003)– Level 1
– Level 2
• Common Σ : all families are generated by the same set of incidentals, and no interaction between radicals and incidentals
![Page 7: Optimal Test Design With Rule-Based Item Generation](https://reader034.fdocuments.in/reader034/viewer/2022051316/56814532550346895db1fa6f/html5/thumbnails/7.jpg)
Modeling Rule-Based Item Generation
• The mean family difficulty as a wieghted sum of the effects of radicals–
– Linear Item Cloning Model (LICM)– with Σf : LICM-F– with Σ : LICM-C
RFRFFbF
RRb
RRb
ddd
ddd
ddd
2211
22221212
12121111
![Page 8: Optimal Test Design With Rule-Based Item Generation](https://reader034.fdocuments.in/reader034/viewer/2022051316/56814532550346895db1fa6f/html5/thumbnails/8.jpg)
Family-Information Funciton
• Item-information
• Family- information
– Expected about θ in the response to a random item from family f.
![Page 9: Optimal Test Design With Rule-Based Item Generation](https://reader034.fdocuments.in/reader034/viewer/2022051316/56814532550346895db1fa6f/html5/thumbnails/9.jpg)
Three Cases of Automated Test Design
• Test Assembly From a Pregenerated Item Pool
![Page 10: Optimal Test Design With Rule-Based Item Generation](https://reader034.fdocuments.in/reader034/viewer/2022051316/56814532550346895db1fa6f/html5/thumbnails/10.jpg)
Three Cases of Automated Test Design
• The Generation on the Fly From Calibrated Families
![Page 11: Optimal Test Design With Rule-Based Item Generation](https://reader034.fdocuments.in/reader034/viewer/2022051316/56814532550346895db1fa6f/html5/thumbnails/11.jpg)
Three Cases of Automated Test Design
• Test Generation on the Fly Using Calibrated Radicals Only
![Page 12: Optimal Test Design With Rule-Based Item Generation](https://reader034.fdocuments.in/reader034/viewer/2022051316/56814532550346895db1fa6f/html5/thumbnails/12.jpg)
Effect of Within-Family Item-Parameter Variability on Family Information
•
• σ2=0 means all item parameters are equal to their respective family means.
• ρ = .0, .5 or -.5
Family-Parameter Mean (μ) Variance(σ2)
a 1 0 or 0.05
b-0.258
(optimal to θ = 0) 0 or 0.5
c 0.2 0 or 0.2
![Page 13: Optimal Test Design With Rule-Based Item Generation](https://reader034.fdocuments.in/reader034/viewer/2022051316/56814532550346895db1fa6f/html5/thumbnails/13.jpg)
• σ2b : decrease in the family information
• σ2a : decrease at ability away from optimal
difficult value• σ2
c : no effect
![Page 14: Optimal Test Design With Rule-Based Item Generation](https://reader034.fdocuments.in/reader034/viewer/2022051316/56814532550346895db1fa6f/html5/thumbnails/14.jpg)
• σ2a =0.05, σ2
b =0.5, σ2c =0.00
• ρab =.5 : – θ> μb, Family information increase
θ< μb, Family information decrease
Because a↑ b↑ information ↑
![Page 15: Optimal Test Design With Rule-Based Item Generation](https://reader034.fdocuments.in/reader034/viewer/2022051316/56814532550346895db1fa6f/html5/thumbnails/15.jpg)
• σ2a =0.00, σ2
b =0.5, σ2c =0.2
• ρbc =.5 : small shift to left– Large guessing result in less information
![Page 16: Optimal Test Design With Rule-Based Item Generation](https://reader034.fdocuments.in/reader034/viewer/2022051316/56814532550346895db1fa6f/html5/thumbnails/16.jpg)
• σ2a =0.05, σ2
b =0.00, σ2c =0.2
• ρac =.5 : counterbalance effect, a↑ c↑
• ρac =-.5 : higher family information
![Page 17: Optimal Test Design With Rule-Based Item Generation](https://reader034.fdocuments.in/reader034/viewer/2022051316/56814532550346895db1fa6f/html5/thumbnails/17.jpg)
• If σ2 ≠ 0 (ρ ≠.0 ), use of item information lead to overestimation information on θ in a random item from the family.
![Page 18: Optimal Test Design With Rule-Based Item Generation](https://reader034.fdocuments.in/reader034/viewer/2022051316/56814532550346895db1fa6f/html5/thumbnails/18.jpg)
Simulation Study
• Illustrate Case 1 and Case 2• The use of family information instead of item
information– Effects of test assembly based on knowledge of
item families only.
![Page 19: Optimal Test Design With Rule-Based Item Generation](https://reader034.fdocuments.in/reader034/viewer/2022051316/56814532550346895db1fa6f/html5/thumbnails/19.jpg)
• β=(-2.0, 1.0, 0.3, 0.9, 0.6, 1.2)• μa~0.8(0.01)1.7, μc~0.1(0.01)0.2• Common Σ, W-B ratio: 0.01, 0.05, 0.1 and 0.2• 10 or 20 items were sampled from the family
distribution
![Page 20: Optimal Test Design With Rule-Based Item Generation](https://reader034.fdocuments.in/reader034/viewer/2022051316/56814532550346895db1fa6f/html5/thumbnails/20.jpg)
• M1: without any constraints on radicalsM2: with constraints on radicals
• θp= -1, 0, 1. Rp= (1, 1, 1) or (1, 2, 1)• Number of families l= 10 or 20 (one item in
each family)• In M2, Each of the five radicals occur– 5~6 times (l= 10) or 10~12times(l= 20)
• Case 1 (PIP) vs. Case 2 (CF): function (8) vs. (17)
![Page 21: Optimal Test Design With Rule-Based Item Generation](https://reader034.fdocuments.in/reader034/viewer/2022051316/56814532550346895db1fa6f/html5/thumbnails/21.jpg)
• 10 families and 10 items per family in poolRp= (1, 1, 1)– PIP: W-B↑, information↑, most informative item
are selected– Family information↓, large uncertainty– Fam Inf < True Item Inf (large W-B)– M2 (constraint radical) < M1 (unconstraint radical)
![Page 22: Optimal Test Design With Rule-Based Item Generation](https://reader034.fdocuments.in/reader034/viewer/2022051316/56814532550346895db1fa6f/html5/thumbnails/22.jpg)
• 10 families and 20 items per family in poolRp= (1, 1, 1)– Doubling pool per family, Slight increase
![Page 23: Optimal Test Design With Rule-Based Item Generation](https://reader034.fdocuments.in/reader034/viewer/2022051316/56814532550346895db1fa6f/html5/thumbnails/23.jpg)
• 20 families and 10 items per family in poolRp= (1, 1, 1)– The different (PIP & CF), increase– Shape is the same
![Page 24: Optimal Test Design With Rule-Based Item Generation](https://reader034.fdocuments.in/reader034/viewer/2022051316/56814532550346895db1fa6f/html5/thumbnails/24.jpg)
• Figure 4• Rp= (1, 2, 1)– The value
smaller than uniform target
![Page 25: Optimal Test Design With Rule-Based Item Generation](https://reader034.fdocuments.in/reader034/viewer/2022051316/56814532550346895db1fa6f/html5/thumbnails/25.jpg)
• Figure 5. uniform• l=10, lf=10, M1– W-B increase,
• CF decrease• PIP tend to target
![Page 26: Optimal Test Design With Rule-Based Item Generation](https://reader034.fdocuments.in/reader034/viewer/2022051316/56814532550346895db1fa6f/html5/thumbnails/26.jpg)
• Figure 6. Rp= (1, 2, 1)
![Page 27: Optimal Test Design With Rule-Based Item Generation](https://reader034.fdocuments.in/reader034/viewer/2022051316/56814532550346895db1fa6f/html5/thumbnails/27.jpg)
Discussion
• Model fit• Exposure control• Item uncertainty – capitalization-on-cnance– Small calibration samples– Variability of the true item parameters– Bank size – test length ratio
• Content constraint can mitigate it
![Page 28: Optimal Test Design With Rule-Based Item Generation](https://reader034.fdocuments.in/reader034/viewer/2022051316/56814532550346895db1fa6f/html5/thumbnails/28.jpg)
Questions
• radicals and incidentals are a bit of abstract• certain combinations of radicals and
incidentals may result in invalid item. How do we know its in validness?
• if the cognitive processes involved in solving the test items are known", which is neally impossible
![Page 29: Optimal Test Design With Rule-Based Item Generation](https://reader034.fdocuments.in/reader034/viewer/2022051316/56814532550346895db1fa6f/html5/thumbnails/29.jpg)
Questions
• μb=0.258 (Wolfe, 1981)