National Standardisation Group Meeting 6 · 2020. 10. 5. · 1 National Standardisation Group...
Transcript of National Standardisation Group Meeting 6 · 2020. 10. 5. · 1 National Standardisation Group...
1
National Standardisation Group Meeting 6
Thursday 6th August
2.00 – 4.40 p.m. via TEAMS.
Attendees:
Andrea Feeney Director CGEO, Chairperson
Elaine Sheridan Asst. Director CGEO
Hugh McManus Asst. Director CGEO
Dr David Millar Assessment Manager, CGEO
Aidan Farrell CEO, State Examinations Commission
Dr Jude Cosgrove Director, Educational Research Centre
Dr Harold Hislop Chief Inspector, Inspectorate, Dept of Education
Orlaith O’Connor Asst Chief Inspector, Inspectorate, Dept of Education
Fernando Cartwright President and CEO Polymetrika International Incorporated.
Dr Kentaro Yamamoto, Technical Advisor.
Paddy Quinn Asst Principal, CGEO, Secretary to the Group
1. At the outset the Chair informed the meeting that the Calculated Grades Programme
Board had been notified on 30th July of the NSG’s change to way of working and
accepted the rationale behind the change.
2. It was also noted that the Scottish Qualification Authority had released their A Level and
GCSE. There had been a high degree of media attention especially on the adjustments to
teacher estimates focused on socio-economic status. . The CGEO will be watching
developments in Scotland and also England, Wales and Norther Ireland when their
results are released on 13/8.
3. A results test file has now been returned form PII for testing on the CGEO database.
4. With the ongoing legal case, any forthcoming announcements re calculated grades will
require careful thought.
5. A number of draft documents had been circulated in advance of the meeting, The Chair
noted with thanks the contribution of CGEO colleagues in the development of these
documents.
Minutes from meeting 5 and matters arising
6. The draft minutes of meeting 5 were agreed and adopted. All matters arising form part
of the agenda.
Schools v Class level Conditioning Distributions
7. Following the last meeting, CGEO had prepared a short document setting out the
approach to conditioning at school v class level, some detail of which had been provided
in the Decision-Making Framework document. This document had been circulated very
2
shortly before the meeting and was not considered in detail. Group members were
invited to provide any feedback on the document via e-mail after the meeting. The
Chairperson noted that while the approach was for approval the overall approach to
standardisation at school level was implicit both in the model design and in the
information published about calculated grades.
Review of Data Models - Overview of outcomes from Pass Through and Estimation Models
to date
8. It was noted that, while there are models up to and including 13 (d), the focus of the
NSG should remain on version 7 as the subsequent versions are test versions. The group
will be updated when later models are available for analysis.
9. An overview was provided of model development to date and progress towards
achieving the objectives. This was a recap of the issues considered by the group already
to date, and extended onwards to the features that were evident in model 7.
10. By way of clarification with regard to the question of whether different models might
conceivably apply across different subjects and/or levels, it was noted and agreed that
the objective is to develop one high quality, generic model with global application,
which blends the four model components to best effect. The application of the model is
context-specific and the particular conditioning data that arises in different schools and
different subjects will give rise to school-level and cross-subject differences as regards
how the different components influence the calculated scores at the different points in
the mark distribution. The difficulty of communicating the model – particularly given
the level of sophistication required in the model in order to maximise fairness in the
standardisation process – was acknowledged.
11. There was some discussion of the degree to which ‘cohort shift’ is taken account of in
the model and/or will need to be addressed more explicitly later, given the lower
propensity of students to drop levels this year, leading to a higher proportion of
students at higher level and lower proportions at Ordinary level and Foundation levels.
This is more evident in some subjects than others. The cohort change arising from this
behavioural change on the part of students is greatest in the Foundation levels in Irish
and Maths, in which the normal candidature has been halved.
12. The NSG noted the focus that there has been on differential effects by demographic
characteristics following the release of the results in Scotland, particularly socio-
economic characteristics. It was noted that examination of such differential effects is
already part of the model validation process.
Review of Data Models – Overview by Polymetrika
13. Overview of strategy
To combine
National distribution National performance by subject from previous years
Use inverse variance weighting to combine these elements in the School-level prior attainment
distribution Prediction of LC results by subject based on prior JC performance of the
3
students in the group aggregated to the school level
“most trustworthy” way.
School historical distribution LC performance in the school from previous years
Teacher estimates Including both rank order and interval-level information contained within the estimated percentage marks
14. Model 5 v Model 7
The national and school regression based estimates were weighted too highly in the early
models. Model 7 changed the weight of how the elements fed in.
15. Interim Model 13 variants
The credibility of the national results can be most readily observed in the scatterplots of
school averages in a “normal” year with the school averages from the model. The goal is to
achieve a similar pattern of results without forcing the model distributions to match
historical patterns. Regression toward the mean will always cause some mismatch – with
the scatterplots not falling exactly on the 45° diagonal. It will be difficult to reconcile the
bottom tail of the distribution as there are few candidates there in the teacher estimates.
There will likely be heavy tails at the top and bottom of the distribution even with the best
model arising from the strategy in play for the current phase of development.
Dr Yamamoto has suggested explicitly building in variance decomposition into the model but
this has not yet been done.
16. Coming up
Polymetrika has generalised the app to allow for the easier implementation of modelling of
the LCA results.
Work has begun on the composite measure for Junior Certificate prior attainment, intended
to improve the quality of the regression-based predictor.
Polymetrika is working on further models 14, 15 and 16 (working labels which may change).
Model 14 involved a restructuring to optimise the running of the algorithm. Model 15 will
test the balance of the variance components and will not be run on many subjects. Model
16 will include the Junior Certificate composite in the predictor and is likely to be the next
milepost for broad checking of calculated marks. It is also anticipated that Model 16 will
allow the running of LCA estimations, although it was also noted the most efficient strategy
in the long run is to have the model fully or almost fully refined before running it on LCA
data.
17. Points noted during discussion
It was noted that one question that would continue to be asked by stakeholders is: what is
the relative weighting of the different elements that feed into the calculated marks? The
difficulty with answering this question in light of the fact that the weighting varies according
to the relative quality of the information from each source at each point in the distribution
4
for each subject and level was acknowledged. Nevertheless, although the contribution of
the four elements will vary in these ways, it was also noted that it will be possible to report
national averages for the relative contributions.
It was again noted that poor performance in the Junior Certificate exams does not
“condemn” a candidate under these models to a poor calculated mark. Junior Certificate
information is fed into the model at a school level rather than an individual one, and that
they are furthermore always based on a distribution and not a point estimate. If there is
little school information, population information is given greater weight. If there is plenty of
information on the school distribution (stability), the population information is given less
weight.
It was noted that the model is run for each subject separately. That is, the ‘school historical’
distribution involved is the one for the subject and level concerned, not one that aggregates
historical performance across subjects.
It was also noted that the better use of prior attainment data may help deal with the much
reduced numbers at Foundation level and the non-random missingness in the NCLs, but this
will need to be kept under review.
Treatment of Small Classes and Subjects Studied Outside of School
The CGEO paper, Treatment of Students in Small Classes and of Subjects Studied Outside of
School (including Out-of-School Learners) in the National Standardisation Process, was
discussed.
18. This treatment of these cohorts within the statistical model is a matter for decision of
the NSG. At the outset, the technical and detailed nature of the documents circulated
in advance of the meeting were acknowledged. The degree of technicality is needed to
give the group full visibility of the proposal. In discussion of the document the following
issues were noted.
The paper was considered very helpful but the addition of the data about the
number of students affected by the decision would assist in understanding
the context
The method and rationale for imputing data from elsewhere for small class
groups was discussed
The rationale for allowing the use of individual prior attainment data and
concurrent other-subject data in the estimation model – both of which are
excluded from use in the generality of cases – for students studying a subject
which is not taught in (the school outside of school) and for out-of-school
learners was discussed.
It was generally acknowledged that it would be unfair to the generality of
students to have any particular cohort not subject to some form of
standardisation. Without some intervention, there is a risk that the
teacher/tutor estimate will become the final mark in cases where this is not
appropriate for these groups of students.
5
Based on the discussion at the meeting the NSG approved the approach, as documented in
the draft paper, to proceed. An updated version of the document will be circulated that will
take account of some observations made at the meeting.
Treatment of Non-Curricular Language Subjects
The CGEO paper National Standardisation and the Non-Curricular EU Languages was
discussed.
19. The following points were noted:
A credible methodology is required for dealing with NCL’s.
The impact of the cohort change needs to be taken account of in the model in a way
which is fair to these students.
That there are some subjects with larger cohorts which could be treated as individual
subjects was noted.
The arguments presented in the paper are very persuasive and strong and the logic
for treating all NCLs as a single subject for conditioning was accepted on the grounds
of fairness.
It was noted that students studying these subjects will, in light of the decision re
subjects taken outside of school, have their individual prior attainment data and
concurrent other-subject data taken into account in the estimation model.
The potential need for further intervention in respect of the macro-level distribution as set
out in the paper was noted. If it is necessary to use soft evidence to change the distribution
(on the grounds that the application of the normal national distribution may disadvantage
this year’s cohort) then this will need to documented at that stage in the process if this
becomes necessary.
20. The Chair drew the attention of the Group to the approach noting the potential for a
similar intervention to be necessary for Foundation Level Maths and Irish given the
degree of cohort shift experienced in those subjects.
Junior Cycle Composite Measure
Following on from the last meeting, the CGEO paper Junior Certificate/Cycle composite input
measure was discussed. This is a matter for noting.
21. The paper describes the creation of a common scale and subsequent composite measure to
improve the predictive power of prior performance at Junior Cycle
22. This has been developed in order to address a technical problem that arises from the
sparseness of the data (and the particular patterns of sparseness) in the student–
subject matrix of JC prior attainment data.
23. The composite approach identified should improve the accuracy of the model and hence the
accuracy of the calculated marks and, ultimately, calculated grades. This is in the interest of
fairness and accuracy of the model.
6
24. The common scale is based on grade threshold alignments implied by the ERC’s JOPS
score which has been used for various studies in the past and which has reputational
value.
25. The proposed approach is to use performance in Irish, English, Maths and the
student’s two best other subjects, excluding CSPE.
Update on Preliminary Report to the Independent Steering Committee.
26. The interim report has been presented to the ISC. While no formal response has been
received, Dr Áine Lawlor, Chair of the ISC, complimented the group on their work to date
and wished them well going forward.
Update on Leaving Certificate Applied.
27. The LCA file has transferred to Polymetrika but as the Leaving Certificate is being
prioritised at the moment, it has not yet been put through any modelling.
28. The LCA should provide a solid basis for the application of the model due to having more
data on which to base the modelling.
29. The work of the CGEO team in cleaning up the LCA data was acknowledged.
30. The hope is that there will be a model to review in the coming week. However, the
greatest efficiency in progressing the overall work in the long run will arise from getting
the model to as final a state as possible before running it on LCA data.
31. The Chair acknowledged that LCA is extremely complex in its make-up and complimented
all on the work to date.
AOB
32. Nothing arose under AOB
In closing the meeting, the Chair outlined that as the group progresses in the standardisation
phase towards sign-off that the expectation is that the meetings would be longer.
The next meeting will be held on Thursday 13th August.
Analysis should continue on version 7 of the model with members to be notified if a later
version becomes available.
The meeting concluded at 4.40 p.m.
Decisions
The Group agreed that, while the latest run of the model had made good progress in
aligning outcomes at a macro level, further iterations of the model are required.
7
School V Class Level Conditioning Distribution document to be reviewed by members with
comments returned by e-mail.
Documents on treatment of
a) Small Classes and Subjects Outside of School and
b) NCL’s
agreed in principle but to be updated to reflect discussion.
Actions
1. Comments to be returned on note describing the process of conditioning at school
level and retaining rank order.
2. Note on an approach to an aggregate for Junior Cycle to be updated.
3. Note on the treatment of small class sizes including class sizes of one to be updated.
4. Note on the treatment of the NCL’s to be updated.
5. Group to continue to analyse version 7 of the model with notification to issue when
an updated model becomes available.