Understanding Standard Setting · Standard Setting The purpose of standard setting is to determine...
Transcript of Understanding Standard Setting · Standard Setting The purpose of standard setting is to determine...
![Page 2: Understanding Standard Setting · Standard Setting The purpose of standard setting is to determine the pass mark for a test. Friedman Ben-David (2000) Meaning varies with assessment](https://reader030.fdocuments.in/reader030/viewer/2022021702/5cec100288c99319498c6b59/html5/thumbnails/2.jpg)
Standard Setting
• Who standard sets?
• What do you standard set?
• How do you standard set?
• Why do you standard set?
![Page 3: Understanding Standard Setting · Standard Setting The purpose of standard setting is to determine the pass mark for a test. Friedman Ben-David (2000) Meaning varies with assessment](https://reader030.fdocuments.in/reader030/viewer/2022021702/5cec100288c99319498c6b59/html5/thumbnails/3.jpg)
Standard Setting
The purpose of standard setting is to determine the pass mark for a test.
Friedman Ben-David (2000)
Meaning varies with assessment aims, e.g.
• Progression• Competence• Accreditation
![Page 4: Understanding Standard Setting · Standard Setting The purpose of standard setting is to determine the pass mark for a test. Friedman Ben-David (2000) Meaning varies with assessment](https://reader030.fdocuments.in/reader030/viewer/2022021702/5cec100288c99319498c6b59/html5/thumbnails/4.jpg)
Aims
• Overview of standard setting methods
• Advantages and Disadvantages
• Consideration of application
• Discussion of applicability
![Page 5: Understanding Standard Setting · Standard Setting The purpose of standard setting is to determine the pass mark for a test. Friedman Ben-David (2000) Meaning varies with assessment](https://reader030.fdocuments.in/reader030/viewer/2022021702/5cec100288c99319498c6b59/html5/thumbnails/5.jpg)
Reference Points
Norm Referenced (Relative)Relative to the performance of the group
Criterion Referenced (Absolute)Relative to an external standard
CompromiseCombination of norm- and criterion-reference
![Page 6: Understanding Standard Setting · Standard Setting The purpose of standard setting is to determine the pass mark for a test. Friedman Ben-David (2000) Meaning varies with assessment](https://reader030.fdocuments.in/reader030/viewer/2022021702/5cec100288c99319498c6b59/html5/thumbnails/6.jpg)
Norm-Referenced
Based on group performance.
Does not evaluate competence with respect to external benchmarks/criteria.
For example, the pass-mark may be set at the mean score, or only the top 10% awarded a distinction.
![Page 7: Understanding Standard Setting · Standard Setting The purpose of standard setting is to determine the pass mark for a test. Friedman Ben-David (2000) Meaning varies with assessment](https://reader030.fdocuments.in/reader030/viewer/2022021702/5cec100288c99319498c6b59/html5/thumbnails/7.jpg)
Criterion-Referenced
Independent of the group taking the test.
Standard is based on predetermined level of competency.
For example, the pass-mark may be set at 60% ahead of the exam.
![Page 8: Understanding Standard Setting · Standard Setting The purpose of standard setting is to determine the pass mark for a test. Friedman Ben-David (2000) Meaning varies with assessment](https://reader030.fdocuments.in/reader030/viewer/2022021702/5cec100288c99319498c6b59/html5/thumbnails/8.jpg)
Compromise Methods
These combine both norm- and criterion-referenced standards.
For example, we might set a standard at 40%, but adjust it by mean difference across tests.
![Page 9: Understanding Standard Setting · Standard Setting The purpose of standard setting is to determine the pass mark for a test. Friedman Ben-David (2000) Meaning varies with assessment](https://reader030.fdocuments.in/reader030/viewer/2022021702/5cec100288c99319498c6b59/html5/thumbnails/9.jpg)
Methods
Norm ReferencedSet Proportions, SD from Mean
Criterion ReferencedFixed standard, Angoff, Ebel
CompromiseHofstee, Hofstee-Angoff, Contrasting GroupsBorderline Groups, Borderline Regression
![Page 10: Understanding Standard Setting · Standard Setting The purpose of standard setting is to determine the pass mark for a test. Friedman Ben-David (2000) Meaning varies with assessment](https://reader030.fdocuments.in/reader030/viewer/2022021702/5cec100288c99319498c6b59/html5/thumbnails/10.jpg)
Driving Theory Test
Different methods of standard setting an
assessment…
![Page 11: Understanding Standard Setting · Standard Setting The purpose of standard setting is to determine the pass mark for a test. Friedman Ben-David (2000) Meaning varies with assessment](https://reader030.fdocuments.in/reader030/viewer/2022021702/5cec100288c99319498c6b59/html5/thumbnails/11.jpg)
Fixed Standard
If you were to set a single, unchanging pass-mark for the test, what would it be?
![Page 12: Understanding Standard Setting · Standard Setting The purpose of standard setting is to determine the pass mark for a test. Friedman Ben-David (2000) Meaning varies with assessment](https://reader030.fdocuments.in/reader030/viewer/2022021702/5cec100288c99319498c6b59/html5/thumbnails/12.jpg)
Fixed Standard
![Page 13: Understanding Standard Setting · Standard Setting The purpose of standard setting is to determine the pass mark for a test. Friedman Ben-David (2000) Meaning varies with assessment](https://reader030.fdocuments.in/reader030/viewer/2022021702/5cec100288c99319498c6b59/html5/thumbnails/13.jpg)
Fixed Standard
![Page 14: Understanding Standard Setting · Standard Setting The purpose of standard setting is to determine the pass mark for a test. Friedman Ben-David (2000) Meaning varies with assessment](https://reader030.fdocuments.in/reader030/viewer/2022021702/5cec100288c99319498c6b59/html5/thumbnails/14.jpg)
Fixed Standard
![Page 15: Understanding Standard Setting · Standard Setting The purpose of standard setting is to determine the pass mark for a test. Friedman Ben-David (2000) Meaning varies with assessment](https://reader030.fdocuments.in/reader030/viewer/2022021702/5cec100288c99319498c6b59/html5/thumbnails/15.jpg)
Angoff
Expert judge provides estimates of performance• Proportion correct• Yes-No
Borderlines / minimally competent / similar
The standard is then the average item estimate.
![Page 16: Understanding Standard Setting · Standard Setting The purpose of standard setting is to determine the pass mark for a test. Friedman Ben-David (2000) Meaning varies with assessment](https://reader030.fdocuments.in/reader030/viewer/2022021702/5cec100288c99319498c6b59/html5/thumbnails/16.jpg)
Angoff
For each item, imagine a minimally competent candidate and ask yourself whether they would answer the item correctly.
Record you decision next to each item with a Yes or No.
![Page 17: Understanding Standard Setting · Standard Setting The purpose of standard setting is to determine the pass mark for a test. Friedman Ben-David (2000) Meaning varies with assessment](https://reader030.fdocuments.in/reader030/viewer/2022021702/5cec100288c99319498c6b59/html5/thumbnails/17.jpg)
Angoff
Discuss your responses as a group and review individual judgements.
![Page 18: Understanding Standard Setting · Standard Setting The purpose of standard setting is to determine the pass mark for a test. Friedman Ben-David (2000) Meaning varies with assessment](https://reader030.fdocuments.in/reader030/viewer/2022021702/5cec100288c99319498c6b59/html5/thumbnails/18.jpg)
Angoff
Averaging across your ratings,we set a pass-mark of…
![Page 19: Understanding Standard Setting · Standard Setting The purpose of standard setting is to determine the pass mark for a test. Friedman Ben-David (2000) Meaning varies with assessment](https://reader030.fdocuments.in/reader030/viewer/2022021702/5cec100288c99319498c6b59/html5/thumbnails/19.jpg)
Hofstee
Requires a min/max pass mark
Requires a min/max failure rate
The pass mark is located where the student data intersects these parameters on a cumulative plot.
![Page 20: Understanding Standard Setting · Standard Setting The purpose of standard setting is to determine the pass mark for a test. Friedman Ben-David (2000) Meaning varies with assessment](https://reader030.fdocuments.in/reader030/viewer/2022021702/5cec100288c99319498c6b59/html5/thumbnails/20.jpg)
Hofstee
![Page 21: Understanding Standard Setting · Standard Setting The purpose of standard setting is to determine the pass mark for a test. Friedman Ben-David (2000) Meaning varies with assessment](https://reader030.fdocuments.in/reader030/viewer/2022021702/5cec100288c99319498c6b59/html5/thumbnails/21.jpg)
Hofstee
What would you accept as the:
• Minimum Pass-mark
• Maximum Pass-mark
• Minimum Failure Rate
• Maximum Failure Rate
![Page 22: Understanding Standard Setting · Standard Setting The purpose of standard setting is to determine the pass mark for a test. Friedman Ben-David (2000) Meaning varies with assessment](https://reader030.fdocuments.in/reader030/viewer/2022021702/5cec100288c99319498c6b59/html5/thumbnails/22.jpg)
Grade Boundaries
What happens if you want to add grade boundaries?
• Fixed pass-mark (e.g. >=75% = Distinction)
• Error/Variability (e.g. Cut-score+/-SEM)
• Set Proportions (e.g. Top 5% = Excellent)
• Relative boundaries (e.g. Mean+/-Value)
![Page 23: Understanding Standard Setting · Standard Setting The purpose of standard setting is to determine the pass mark for a test. Friedman Ben-David (2000) Meaning varies with assessment](https://reader030.fdocuments.in/reader030/viewer/2022021702/5cec100288c99319498c6b59/html5/thumbnails/23.jpg)
Grade Information
What happens if you also have global grade*
judgements being made alongside numerical scores?
“This student scored 68%, and I thought their overall performance was Satisfactory”
*Global-grades which provide an overall, more subjective/eclectic/holistic judgement of performance; not necessarily a grade derived from the score (e.g. 75% = A) or otherwise tied to the criteria used to derive the scores.
![Page 24: Understanding Standard Setting · Standard Setting The purpose of standard setting is to determine the pass mark for a test. Friedman Ben-David (2000) Meaning varies with assessment](https://reader030.fdocuments.in/reader030/viewer/2022021702/5cec100288c99319498c6b59/html5/thumbnails/24.jpg)
Borderline Groups
Borderline Groups uses the average of the B graded scoresContrasting Groups sets the pass-mark between U and LS grades
![Page 25: Understanding Standard Setting · Standard Setting The purpose of standard setting is to determine the pass mark for a test. Friedman Ben-David (2000) Meaning varies with assessment](https://reader030.fdocuments.in/reader030/viewer/2022021702/5cec100288c99319498c6b59/html5/thumbnails/25.jpg)
Borderline Regression
Borderline Regression uses the scores and grades to derive a linear model, and sets the pass-mark at the intersection of the borderline group.
![Page 26: Understanding Standard Setting · Standard Setting The purpose of standard setting is to determine the pass mark for a test. Friedman Ben-David (2000) Meaning varies with assessment](https://reader030.fdocuments.in/reader030/viewer/2022021702/5cec100288c99319498c6b59/html5/thumbnails/26.jpg)
Appropriateness
There is no gold standard.
The method must be:• Appropriate• Feasible• Credible• Acceptable to stakeholders• Evidence-based• Defensible (academically and legally)
![Page 27: Understanding Standard Setting · Standard Setting The purpose of standard setting is to determine the pass mark for a test. Friedman Ben-David (2000) Meaning varies with assessment](https://reader030.fdocuments.in/reader030/viewer/2022021702/5cec100288c99319498c6b59/html5/thumbnails/27.jpg)
Recap
Norm-Referenced• Provides standardised pass/fail rate• Easy to implement• Does not adjust for ability• Might not be acceptable to all
stakeholders
![Page 28: Understanding Standard Setting · Standard Setting The purpose of standard setting is to determine the pass mark for a test. Friedman Ben-David (2000) Meaning varies with assessment](https://reader030.fdocuments.in/reader030/viewer/2022021702/5cec100288c99319498c6b59/html5/thumbnails/28.jpg)
Recap
Criterion-Referenced
• Focus on individual items
• Defensible for high stakes
• Borderline can be difficult to define
• Time-consuming
• Where do the numbers come from?
![Page 29: Understanding Standard Setting · Standard Setting The purpose of standard setting is to determine the pass mark for a test. Friedman Ben-David (2000) Meaning varies with assessment](https://reader030.fdocuments.in/reader030/viewer/2022021702/5cec100288c99319498c6b59/html5/thumbnails/29.jpg)
Recap
Compromise Methods
• Suitable for overall pass/fail
• Evidence based
• Simple standard setting
• Can ‘miss the mark’, prone to outliers
• Not first choice for high-stakes
![Page 30: Understanding Standard Setting · Standard Setting The purpose of standard setting is to determine the pass mark for a test. Friedman Ben-David (2000) Meaning varies with assessment](https://reader030.fdocuments.in/reader030/viewer/2022021702/5cec100288c99319498c6b59/html5/thumbnails/30.jpg)
Conclusions
There is no perfect method.
A wide range of methods exist .
Methods must be fit for purpose.
Can be a question of policy.
![Page 31: Understanding Standard Setting · Standard Setting The purpose of standard setting is to determine the pass mark for a test. Friedman Ben-David (2000) Meaning varies with assessment](https://reader030.fdocuments.in/reader030/viewer/2022021702/5cec100288c99319498c6b59/html5/thumbnails/31.jpg)
Conclusions
Choice depends on:• Credibility• Available resources• High stakes level of exam
Method is important, process is critical• Suitable judges• Due diligence applied• Defensible rationale
![Page 32: Understanding Standard Setting · Standard Setting The purpose of standard setting is to determine the pass mark for a test. Friedman Ben-David (2000) Meaning varies with assessment](https://reader030.fdocuments.in/reader030/viewer/2022021702/5cec100288c99319498c6b59/html5/thumbnails/32.jpg)
[email protected]@plymouth.ac.uk