Education | Elections | Census Controlling marking quality in e-Marker ®
-
Upload
silvia-parks -
Category
Documents
-
view
216 -
download
0
Transcript of Education | Elections | Census Controlling marking quality in e-Marker ®
![Page 1: Education | Elections | Census Controlling marking quality in e-Marker ®](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649e9e5503460f94ba0a0f/html5/thumbnails/1.jpg)
Education | Elections | Census
Controlling marking quality
in e-Marker®
![Page 2: Education | Elections | Census Controlling marking quality in e-Marker ®](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649e9e5503460f94ba0a0f/html5/thumbnails/2.jpg)
Education | Elections | Census
Commercial in confidence 2012
Marking quality control
• Marking quality can be delivered by:– e-Standardisation– Seeding and qualification– Percentage double marking– S-Process
![Page 3: Education | Elections | Census Controlling marking quality in e-Marker ®](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649e9e5503460f94ba0a0f/html5/thumbnails/3.jpg)
Education | Elections | Census
Commercial in confidence 2012
Seeding or percentage double marking?
• If the marking guidance is detailed and objective and you would expect complete agreement among markers – use seeding
• If the marking guidance is subjective – or gives guidance for the approach to marking, rather than what is a right or wrong answer – use percentage double marking
• You can use seeding and percentage double marking on the same component
• S-Process can be added to both quality methods for further quality control if desired
![Page 4: Education | Elections | Census Controlling marking quality in e-Marker ®](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649e9e5503460f94ba0a0f/html5/thumbnails/4.jpg)
Education | Elections | Census
Commercial in confidence 2012
Seeding and qualification
![Page 5: Education | Elections | Census Controlling marking quality in e-Marker ®](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649e9e5503460f94ba0a0f/html5/thumbnails/5.jpg)
Education | Elections | Census
Commercial in confidence 2012
Seeds
• Seeds provide an in-built quality check• The administrators define and maintain the levels and
frequency of the quality checks• Senior markers pre-mark a selection of responses - seeds• Quality control applies at individual question level• Quality is checked at the start of each day (qualification)
and while the markers mark (seeding)• Markers can see the seeds they have failed so that they can
improve their marking
![Page 6: Education | Elections | Census Controlling marking quality in e-Marker ®](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649e9e5503460f94ba0a0f/html5/thumbnails/6.jpg)
Education | Elections | Census
Commercial in confidence 2012
Seeding
Qualify at start of day
Mark seeds
Mark batch of
questions
Pass FailStop
MarkingQuestion
Quality check: pass 9 out of 10
Mark batch of
questions
Start marking after discussions with Senior or mark other questions
Quality check: must not fail both seeds
Quality check: must not fail both seeds
![Page 7: Education | Elections | Census Controlling marking quality in e-Marker ®](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649e9e5503460f94ba0a0f/html5/thumbnails/7.jpg)
Education | Elections | Census
Commercial in confidence 2012
Seed window
Quality check:7 out of 10 within tolerance
![Page 8: Education | Elections | Census Controlling marking quality in e-Marker ®](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649e9e5503460f94ba0a0f/html5/thumbnails/8.jpg)
Education | Elections | Census
Commercial in confidence 2012
Seed window
Quality check:7 out of 10 within tolerance
![Page 9: Education | Elections | Census Controlling marking quality in e-Marker ®](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649e9e5503460f94ba0a0f/html5/thumbnails/9.jpg)
Education | Elections | Census
Commercial in confidence 2012
Seeding is set by senior markers
• Seeding needs to be done before the markers start marking• Ensure there are enough seeds – the minimum seed bank
size should be reached• If seeds are retired, they will need replacing• On a seeded question, the senior markers’ mark will be
awarded to the candidate
![Page 10: Education | Elections | Census Controlling marking quality in e-Marker ®](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649e9e5503460f94ba0a0f/html5/thumbnails/10.jpg)
Education | Elections | Census
Commercial in confidence 2012
Seeding for real
![Page 11: Education | Elections | Census Controlling marking quality in e-Marker ®](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649e9e5503460f94ba0a0f/html5/thumbnails/11.jpg)
Education | Elections | Census
Commercial in confidence 2012
What makes a good seed?
• Legible• Unambiguous and complete• Reflects the standards agreed in standardisation• No tricks!• Does not have to be the right answer• Don’t set “not attempted” responses as seeds
![Page 12: Education | Elections | Census Controlling marking quality in e-Marker ®](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649e9e5503460f94ba0a0f/html5/thumbnails/12.jpg)
Education | Elections | Census
Commercial in confidence 2012
Percentage double marking
![Page 13: Education | Elections | Census Controlling marking quality in e-Marker ®](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649e9e5503460f94ba0a0f/html5/thumbnails/13.jpg)
Education | Elections | Census
Commercial in confidence 2012
Percentage double marking
• Percentage double marking involves:– Comparing two marking opinions in real time at question
level;– Where differences in marking exceed a set tolerance,
automated quality checks are used to invoke adjudication by a senior marker or peer
![Page 14: Education | Elections | Census Controlling marking quality in e-Marker ®](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649e9e5503460f94ba0a0f/html5/thumbnails/14.jpg)
Education | Elections | Census
Commercial in confidence 2012
Percentage double marking
1
2
The Senior Marker adjudicates. In this case he/she agrees with Marker 2, so Marker 1 receives a penalty
Marker 2 marks the same clip, not knowing it has been marked before. If Markers 1 and 2 agree, no further action is taken and the mark is awarded to the candidate. In this example however, it is marked outside tolerance of Marker 1, so the marking of both markers is forwarded for adjudication by the Senior Marker
Marker 1 receives a portion to mark. It is selected by e-Marker® as a quality control item (benchmark item) and is forwarded to the next available marker
![Page 15: Education | Elections | Census Controlling marking quality in e-Marker ®](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649e9e5503460f94ba0a0f/html5/thumbnails/15.jpg)
Education | Elections | Census
Commercial in confidence 2012
Percentage double marking
• There are controls in e-Marker® administration to:– Ensure that markers mark at a similar pace;– Don’t collude in their marking;– Identify when discrepancies are forwarded to peer or
senior; and– Identify when adjudication goes against a marker
![Page 16: Education | Elections | Census Controlling marking quality in e-Marker ®](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649e9e5503460f94ba0a0f/html5/thumbnails/16.jpg)
Education | Elections | Census
Commercial in confidence 2012
Pioneer cap
Too high?
• First marker could mark their entire quota without being checked against other markers
• Be particularly careful in small marker pools
Too low?
• Could slow down marking progress with markers being halted too often
• Could make the frequency of the quality checks overt to markers
The pioneer cap defines how many benchmarks the fastest marker can be ahead of the next fastest. The prevents one marker marking too much without any quality control
![Page 17: Education | Elections | Census Controlling marking quality in e-Marker ®](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649e9e5503460f94ba0a0f/html5/thumbnails/17.jpg)
Education | Elections | Census
Commercial in confidence 2012
Partnering gap
Too high?
• Marking is likely to be slowed down as new markers are needed
• Could leave difficult, outstanding parts to mark at the end of marking
Too low?
• Could have markers colluding, or markers with similar opinions remarking each other’s work
The partnering gap defines how frequently a marker’s benchmarks can be marked by each other marker
![Page 18: Education | Elections | Census Controlling marking quality in e-Marker ®](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649e9e5503460f94ba0a0f/html5/thumbnails/18.jpg)
Education | Elections | Census
Commercial in confidence 2012
Suspect cap
Too high?
• Markers could potentially carry on marking when their marking is eventually shown to be substandard
Too low?
• Markers could be stopped from marking too frequently and, potentially, unnecessarily
The suspect cap defines how many adjudications can be outstanding for a marker. This prevents markers from getting too far ahead without the senior checking their work
![Page 19: Education | Elections | Census Controlling marking quality in e-Marker ®](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649e9e5503460f94ba0a0f/html5/thumbnails/19.jpg)
Education | Elections | Census
Commercial in confidence 2012
Penalty cap
Too high?
• A marker below the standards required will be allowed to continue marking
Too low?
• Markers stopped too often, and potentially, unnecessarily
• Additional work for senior markers in stopped marker management
The penalty cap defines how many adjudications can be delivered against a marker before the stopped marker management process is invoked
![Page 20: Education | Elections | Census Controlling marking quality in e-Marker ®](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649e9e5503460f94ba0a0f/html5/thumbnails/20.jpg)
Education | Elections | Census
Commercial in confidence 2012
Marker quality review
• Marker review process is used during percentage double marking– Can be used if there are concerns about a particular
marker or about marking standards at a particular time– Senior markers can request that a greater proportion of
marking is checked– Can be done retrospectively, for example, work done in
the past can be checked
![Page 21: Education | Elections | Census Controlling marking quality in e-Marker ®](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649e9e5503460f94ba0a0f/html5/thumbnails/21.jpg)
Education | Elections | Census
Commercial in confidence 2012
Marker review process
Marker 1
Marker 2Marker 3
![Page 22: Education | Elections | Census Controlling marking quality in e-Marker ®](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649e9e5503460f94ba0a0f/html5/thumbnails/22.jpg)
Education | Elections | Census
Commercial in confidence 2012
Stopped markers
![Page 23: Education | Elections | Census Controlling marking quality in e-Marker ®](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649e9e5503460f94ba0a0f/html5/thumbnails/23.jpg)
Education | Elections | Census
Commercial in confidence 2012
Stopped markers
• Senior markers can view quality failures of markers within stopped marker management. They can:– Permanently stop markers– Restart markers– For seeding and S-Process, can identify if quality check is
not of a good standard
![Page 24: Education | Elections | Census Controlling marking quality in e-Marker ®](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649e9e5503460f94ba0a0f/html5/thumbnails/24.jpg)
Education | Elections | Census
Commercial in confidence 2012
Setting the parameters
![Page 25: Education | Elections | Census Controlling marking quality in e-Marker ®](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649e9e5503460f94ba0a0f/html5/thumbnails/25.jpg)
Education | Elections | Census
Commercial in confidence 2012
Seeding parameters
• The function and application of seeding is governed by parameters set in the e-Marker® Administration system
• These parameters define:– The number of seeds in qualification– The frequency at which seeds are presented in marking– The acceptable number of seeds that can be failed
before a marker is stopped– The minimum seed bank level required
![Page 26: Education | Elections | Census Controlling marking quality in e-Marker ®](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649e9e5503460f94ba0a0f/html5/thumbnails/26.jpg)
Education | Elections | Census
Commercial in confidence 2012
Percentage double marking parameters
• The function and application of percentage double marking is governed by parameters set in the e-Marker® Administration system
• These parameters define:– The frequency with which marking is benchmarked– The pioneer cap– The partnering gap– The suspect cap– The penalty cap
![Page 27: Education | Elections | Census Controlling marking quality in e-Marker ®](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649e9e5503460f94ba0a0f/html5/thumbnails/27.jpg)
Education | Elections | Census
Commercial in confidence 2012
Parameter modelling
![Page 28: Education | Elections | Census Controlling marking quality in e-Marker ®](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649e9e5503460f94ba0a0f/html5/thumbnails/28.jpg)
Education | Elections | Census
Controlling Marking Quality
in e-Marker®