Post on 02-Jan-2016
description
Portfolio Committee on Basic Education
Report on the Quality Assurance of the NSC
12 February 2013
Dr Mafu S Rakometsi
WHAT CAN WE LEARN FROM THE NSC RESULTS?
Introduction to context, principles, approaches and processes
Dr Mafu Rakometsi - CEO of Umalusi
Regulatory Framework
Quality Assurance of Assessment• NQF Act Section 27 (h)
The QC must develop and implement policy and criteria for assessment for the qualifications on its sub-framework.
Section17 of the GENFETQA Act(5) The Council must, with the concurrence of the Director-General and
after consultation with the relevant assessment body or education
institution, approve the publication of the results of learners if the
Council is satisfied that the assessment body or education institution
has—
(i) conducted the assessment free from any irregularity that may
jeopardise the integrity of the assessment or its outcomes;
(ii) complied with the requirements prescribed by the Council for
conducting assessments;
(iii) applied the standards prescribed by the Council which a learner is
required to comply with in order to obtain a certificate; and
(iv) complied with every other condition determined by the Council.
Framework for QA of Learner Achievement Based on established and existing practices in
assessment for certification
• Prescribed components of External assessment (examinations) and Site based/ internal / continuous assessment
• Use of systems, processes, and procedures to evaluate, inspect, monitor and report on examination systems, processes and procedures of public and private assessment bodies.
Framework for Quality Assurance of Assessment Evaluation and /or accreditation of assessment bodies Periodic inspection of assessment systems Ongoing monitoring of assessment systems Quality assurance of external examinations through: Moderation of examination question papers Monitoring and moderation of SBA Monitoring the conduct of examinations Moderation of marking Standardization of assessment outcomes
Approval for the release of ResultsApproval is based on the following requirements:
• The examinations are conducted compliant to the applicable policies regulating the conduct and administration of the examinations
• At the time of approval, there is no serious irregularity which could undermine the credibility of the examinations.
Quality Assurance of the DBE 2012 National Senior Certificate Examination
Vijayen Naidoo – Sen. Manager : Quality Assurance of Assessments
PURPOSE
To ensure that the question papers are of the required standard
(standard captured in the NCS and SAG’s)
To ensure that the question papers are relatively:
- fair
- reliable
- representative of an adequate sample of the curriculum
- representative of relevant conceptual domains
- representative of relevant levels of cognitive challenge External moderators
Moderation of question papers
Moderation of the question papers Approach
Question papers set by panel of examiners – DBE
Internally moderated by DBE
Externally moderated by Umalusi
Subsequent moderations and approval.
Moderation of Question papers
Number of NSC 2012 question papers moderate
Number of subjects
Number of papers
Approved/ Conditionally approved at 1st moderation
Approved/ Conditionally approved at 2nd moderation
Approved/ Conditionally approved at 3rd moderation
4th moderation and beyond
62 (including the non-official languages and LO)
Nov 2012 -132 27.3% (36) 56.8% (75) 15.2%(20) 0.7% (1)
Mar 2013 -130 27,7% (36) 54.6 %(71) 13.9% (18) 3.8% (5)
Moderation of the question papersCriteria
Technical Criteria
Internal moderation – quality of internal moderator reports.
Content coverage
Cognitive Demand
Marking Guidelines
Language and Bias
Adherence to Policies and guidelines
Predictability
Moderation of the question papersFindings: Areas of Good Practice
Percentage of question papers and memoranda approved after first and second moderation ( Nov 2012- 79,6% ; Mar 13 – 72,3%)
Simultaneous moderation of final and supplementary question papers.
Standard of Internal moderation
Moderation of the question papersFindings: Areas of Concern
Adherence to timeframes and impact on quality of setting and moderation.
Question papers requiring more than four moderations.
Role of the external moderator.
Internal assessment refers to any assessment conducted by the provider, the outcomes of which count towards the achievement of the qualification
panels of moderators / subject specialists
Moderation of internal /continuous assessment
Purpose of Umalusi’s verification:
To verify the rigour and appropriateness of the DBE moderation process – linked to DBE plans
Ascertain the degree to which assessment bodies/provinces are attempting to ensure standardisation across
Ascertain the standard and quality of the tasks
Determine the extent and quality of internal moderation and feedback.
Determine the reliability and validity of the assessment outcomes
Moderation of internal assessment (cont.)
Moderation of Internal AssessmentPhase 1 (June/July 2012)- focus on assessment
instruments (teacher files)
PROVINCIAL EDUCATION DEPARTMENT
Subjects
Eastern CapeKwaZulu- NatalLimpopoMpumalanga
Mathematics, Life Sciences, Physical SciencesAccounting, English FAL, History, Geography, Life Orientation,
Moderation of Internal AssessmentPhase 1: Findings
• Quality and Standard of Assessed tasks
Content Coverage – mostly appropriate
Cognitive demand and difficulty levels –standards variable mainly at lower cognitive levels,
Over reliance on previous question papers.
Projects and practical investigations- not cognitively balanced, more theory than practice.
Marking tools – not applied consistently
Teachers subject knowledge often inadequate
Physical Education Task (PET) inappropriately assessed.
Moderation of Internal AssessmentPhase 1: Findings
• Internal Moderation
Moderation instruments – compliance vs. quality
School moderation – seriously neglected
Monitoring vs. Moderation
• Feedback and Support – general support by subject advisors. Very little developmental feedback.
Moderation of Internal AssessmentPhase 1: Areas of Good Practice
• Use of Common Tasks
• Diagnostic Analysis to improve performance
• Compliance with Internal Assessment protocols.
Moderation of Internal AssessmentPhase 2 (October/November 2012)- focus on learner
evidence and teacher files.
PROVINCIAL EDUCATION DEPARTMENT
Subjects
All 9 PED’s Mathematics, Life Sciences, Physical SciencesAccounting, English FAL, History, Geography, Life Orientation,
Moderation of Internal AssessmentPhase 2: Findings
• Quality and Standard of Moderation
Brief unclear – focus on marking only and not on quality of task.
Shadow marking vs. moderation
• Marking of the Tasks
Marking tools – not applied consistently
Teachers subject knowledge often inadequate
Challenges with marking extended writing and research projects.
Marking too lenient
Moderation of Internal AssessmentPhase 2: Findings
• Learner performance
Difficulty with higher-order questions
Ability to express/respond in English a determining factor
Extremely poor performance in certain subjects e.g. Maths, LO Common Assessment Task,
Evidence suggests that very little effective teaching and learning is taking place
Moderation of Internal AssessmentPhase 2: Findings
• Internal Moderation at School, Cluster and District Levels.
Varying quality of Moderation
Limited moderation of candidates work – shadow marking
Little evidence of moderation of tasks.
Lack of mark sheets or incorrect recording of marks.
Moderation of Internal AssessmentPhase 2: Areas of Good Practice
• Use of Common Tasks
• Diagnostic Analysis to improve performance
• Compliance with Internal Assessment protocols.
• Evidence of moderation at the various levels.
• Introduction of National Moderation
Monitoring of Examinations
• State of readiness
• Conduct of examinations
• Marking
State of Readiness
Monitored the DBE audit visits (March/April) and evaluation visits to all
Comprehensive approach that monitors exam systems (including SBA)
Monitoring of ExaminationsState of readiness
Findings
• All provinces have examination systems in place
• Vacant posts and use of contract staff is a major concern.
Monitoring of the writing phase.
Findings
• General Management of the examinations
Generally examinations conducted in line with policy
Isolated instances of non-compliance (suitability of venue, noise, security, contingency plans)
Shortage of question papers and answer books
Provincial Monitoring
No of exam centres No of exam centers monitored by Umalusi
No of Umalusi monitors per province
6623 207 36
Monitoring of the marking phase
No of marking centers(2010)
No of marking centers(2011)
No of marking centers(2012)
No of marking centers monitored by Umalusi
Total 127 123 118 87
Monitoring of the marking phase
Findings
• In general marking processes in line with policy
• Concern with lack of security at some marking centres
• Concern with the appointment and training of markers
• Concern with the quality and standard of marking and internal moderation.
• Number of marking centres – security risk
PURPOSE
• Moderation of marking determines the standard and quality of marking and ensures that marking is conducted in accordance with agreed practices
• Umalusi engages the following during the moderation of marking
1. Pre-marking/memorandum discussion: centralised memo discussions recommended - this will ensure consistency across marking centres
2. Moderation of marking (off-site and on-site)
Verification of marking
Verification of marking
Memoranda discussion meetings Approved and signed off finalized memoranda/
marking guidelines for all the NSC subjects Concern with non attendance and pre- memo
discussion preparations Concern with timing and scheduling of memo
discussions. Practice of marking and training in the memo
discussions impacts positively on standard of marking.
Verification of marking
Findings
Findings
• Adherence to Marking memo
• Marking generally accurate – problem with marking extended writing and essays
Provinces Subjects
Centralised All 9 Gateway subjects + 4 others
On-site Eastern Cape; Gauteng; KZN; Limpopo; Northern Cape
13 in total
Verification of marking• Quality of marking varied. Question marking improved
consistency.
Competency of markers to mark interpretation type questions.
Internal moderation
No unauthorised changes to marking guidelines reported
Training of chief markers and internal moderators at memo discussions having a positive impact on marking.
Standardisation and Verification of Resulting
• Provision of GENFETQA – Council may adjust raw marks.
• International practice – large scale assessment systems
• Standardisation – process used to mitigate the effect of factors other than learners knowledge and aptitude on the learners performance.
• Sources of variability – difficulty in question paper, undetected errors, learner interpretation of questions
Statistical moderation
Standardisation
58 subjects standardised
Raw marks accepted: 41 subjects
Moderated Upward : 5 subjects
Moderated Downward : 12 subjects
Verification of the Resulting Process
Planned Status
Subject structures Subject structures verified.
Candidate registration System verified during state of readiness visits.
Generation of mark Sheets Monitored
Capturing of Marks Monitored
Standardisation data & Booklets
Data sets received and verified
Capturing of adjustments Verified
Statistical moderation & resulting
Verified
Examination Irregularities The majority of irregularities were of a technical nature and
these were reported to Umalusi according to the established channels.
Some irregularities were as a result of registration-related problems, e.g. candidates nor appearing on mark sheets, some registered for incorrect subjects.
Umalusi represented on NEIC
Major irregularities:
Alleged leakage of Question Papers in KZN
WHAT CAN WE LEARN FROM THE NSC RESULTS?
STANDARDISATION DECISIONS
DBE NSC 2012
SUBJECT ADJUSTMENT
Life Sciences
CA( 0 at 0 scale to -8 at 23. scale to 0 at 70, scale to +3 at 119, scale to 0 at 171, scale to -7 at 239, scale to -1 at 291, scale to -7 at 300)
Physical Sciences0 at 0, scale to -12 at 102, block -12 from 102 to 252, scale to 0 at 300
Mathematics Raw
Mathematical Literacy
0 at 0, raw to 114, scale 0 to +9 from 114 to 240, scale to 0 at 300
Mathematics: Probability; Data Handling
Raw
SUBJECT ADJUSTMENT
Geography
0 at 0, scale to -6 at 96, block -6 from 96 to 180, scale to 0 at 240, raw to 300
Life OrientationRaw
Hospitality Studies Raw
Tourism½ CA( 0 at 0, scale to -12 at 121, scale to 0 at 300)
Agricultural ScienceRaw
Consumer Studies Raw
SUBJECT ADJUSTMENTAgricultural Management Practices Raw
Agricultural Technology Raw
Music Raw
Dance Studies Raw
Design
0 at 0, scale to -12 at 102, scale to -6 at 246, scale to 0 at 300
Dramatic Arts½ CA(o at o, scale to -15 at 66, scale to 0 at 300)
SUBJECT ADJUSTMENTVisual Arts Raw
Religion Studies Raw
Information Technology Raw
Computer Applications Technology
Raw
Civil Technology 0 at 0, scale to -9 at 90, block -9 from 90 to 180, scale to 0 at 300
Electrical Technology½ CA(0 at 0, scale to -15 at 211, scale to 0 at 300)
SUBJECT ADJUSTMENT
Mechanical Technology Raw
Engineering Graphics and Design
Raw
Accounting
0 at 0, scale to -9 at 78, block -9 from 78 to 222, scale to 0 at 300
Business StudiesRaw
EconomicsRaw
SUBJECT ADJUSTMENTHistory Raw
Afrikaans HL
0 at 0, scale to +4 at 116, block of +4 from 116 to 247, scale to 0 at 300.
Afrikaans FAL RawAfrikaans SAL Raw
English HL
0 at 0, scale to +8 at 112, block of +8 from 112 to 240, scale to 0 at 300.
SUBJECT ADJUSTMENT
English FAL
0 at 0, scale to -6 at 51, block of -6 from 51 to 189, scale to 0 at 300.
English SAL Raw
IsiNdebele HL Raw
IsiNdebele FAL Raw
IsiZulu HL
0 at 0, scale to +8 at 112, block of +8 from 112 to 232, scale to 0 at 300.
IsiZulu FAL Raw
IsiZulu SAL Raw
SUBJECT ADJUSTMENT
Setswana HL Raw
Setswana FAL Raw
Setswana SAL Raw
Siswati HL Raw
Siswati FAL Raw
Siswati SAL Raw
IsiXhosa HL Raw
IsiXhosa FAL Raw
IsiXhosa SAL Raw
SUBJECT ADJUSTMENT
Xitsonga HL½ CA (0 at 0, scale to -13 at 74, scale to 0 at 300)
Xitsonga FAL Raw
Tshivenda HL
½ CA(0 at 0, scale to -12 at 115, block -12 from 115 to 153, scale to 0 at 300)
Tshivenda FAL Raw
SUBJECT ADJUSTMENT
Sepedi HL RawSepedi FAL Raw
Sesotho HL Raw
Sesotho FAL½ CA(0 at 0, scale to -15 at 300)
Sesotho SAL Raw
Quality Assurance of the IEB 2012 National Senior Certificate Examination
Vijayen Naidoo – Sen. Manager : Quality Assurance of Assessments
Moderation of Question papers
Number of IEB NSC 2012 question papers moderated
Number of subjects
Number of papers
Approved/ Conditionally approved at 1st moderation
Approved/ Conditionally approved at 2nd moderation
Approved/ Conditionally approved at 3rd moderation
4th moderation and beyond
39 Nov 2012 -69 39.7% (28) 54.4% (37) 4.4%(3) 1.5% (1)
Moderation of the question papersFindings: Areas of Good Practice
Percentage of question papers and memoranda approved after first and second moderation ( Nov 2012- 94.1% )
Standard of Internal moderation
Moderation of Internal AssessmentPhase 2 (November 2012)- focus on learner evidence
and teacher files.
IEB Subjects
Mathematics, Maths Lit, Life Sciences, Physical SciencesAccounting, English HL, History, Life Orientation, Business Studies, Economics
Moderation of Internal AssessmentFindings: Areas of Good Practice • Regional and national moderation is in place and well
executed
• The marking of tasks of a highly acceptable standard
Findings: Areas of Concern•Feedback to the learners
•Assessment of Physical Education Tasks in LO – distance education learners
Monitoring of the writing phase.Findings
• General Management of the examinations
Generally examinations conducted in line with policy
Isolated instances of non-compliance (candidate identification)
Lack of Irregularity Registers
Assessment Body Monitoring
Verification of marking8 Subjects
Findings
• marking and moderation of scripts is extremely thorough
• Marking generally accurate
• practice of double marking is highly commendable
• Moderation doesn’t always involve full moderation of a script
Statistical moderation
Standardisation
58 subjects standardised
Raw marks accepted: 44 subjects
Moderated Upward : 9 subjects
Moderated Downward : 4 subjects
WHAT CAN WE LEARN FROM THE NSC RESULTS?
STANDARDISATION DECISIONS
IEB NSC 2012
SUBJECT ADJUSTMENT
Accounting0 at 0, scale to -6 at 96; block -6 from 96 to 212, scale to -5 at 245, back to 0 at 300)
Business Studies
½ CA(0 at 0, scale to -15 at 139, block of -15 from 139 to 208, scale back to 0 at 300)
Computer Application Technology
½ CA (Raw from 0 to 101, scaled from 0 at 101 to +7 at 219, scaled to 0 at 300)
Consumer Studies Raw
Design Raw
Dance Studies Raw
SUBJECT ADJUSTMENT
Dramatic ArtsRaw
Economics½ CA(0 at 0, scaled to +10 at 45, block +10 from 45 to 102, scaled to 0 at 300)
Engineering Graphics and Design
½ CA ( 0 at 0, scale to +6 at 41, block of +6 from 41 to 79, scale to 0 at 300)
Geography
CA(0 at 0, scale to +12 at 36, scale to -7 at 59,scale to -1 at 121, scale to +8 at 173, block +8 from 173 to 200, scale to 0 at 300)
SUBJECT ADJUSTMENT
HistoryCA (0 at 0, scale to -18 at 56, scaled to -25 at 109, scaled to 0 at 300
Information Technology Raw
Hospitality Studies Raw
Tourism Raw
Visual Arts ½ CA ( 0 at 0, scale to -15 at 89, block -15 from 89 to 222, scale to 0 at 300)
Music Raw
SUBJECT ADJUSTMENT
Religion Studies Raw
Life Orientation Raw
Life SciencesRaw
Mathematical Literacy
Raw
MathematicsRaw
Mathematics P3 Raw
SUBJECT ADJUSTMENT
Physical Science
½ CA ( 0 at 0 scale to -7 at 28, scale to 0 at 68, block +1 from68 to 74, scale to 0 at110, scale to -7 at 241, scale to 0 at 300)
English FAL
½ CA (0 at 0, scale to +15 at64, scale to +6 at 221, scale to +10 at 258, scale to 0 at 300)
English HL Raw
Afrikaans FAL
½ CA (0 at 0, scale to -10 at 147, block of -10 from 147 to 175, scale back to 0 at 300)
SUBJECT ADJUSTMENT
Afrikaans HL Raw
Sepedi FAL Raw
Sesotho FAL Raw
Setswana FAL Raw
Siswati FAL Raw
Siswati HL Raw
IsiXhosa FAL RawIsiZulu HL Raw
SUBJECT ADJUSTMENT
IsiZulu FAL
CA( 0 at 0, scale to -16 at 146, scale to -6 at 166, block -10 from 166 to 242, scale to 0 at 300)
IsiZulu SAL Raw
Arabic SALRaw
French SAL
½ CA( 0 at 0, scale to -13 at 51, scale to -1 at 174, scale to -5 at 206, block -5 from 206 to 279, scale to 0 at 300)
German HL Raw
German SAL Raw
SUBJECT ADJUSTMENT
Gujurati SAL Raw
Hebrew SAL 0 at 0, raw to 180, scale to -6 at 246, scale to 0 at 300
Hindi FAL Raw
Hindi SAL Raw
Italian SAL Raw
Latin SAL Raw
SUBJECT ADJUSTMENT
Modern Greek SAL Raw
Portuguese HL Raw
Portuguese SAL Raw
Spanish SAL Raw
Tamil SAL Raw
Urdu FAL Raw
Urdu SAL Raw
Serbian HL Raw
SUBJECT ADJUSTMENT
Equine Studies* Raw
Sport and Exercise Science*Raw
Maritime Economics* Raw
Nautical Science Raw
Advanced Programme Mathematics ½ CA( Block +5)Advanced Programme English
Raw
Annexure to Approval Letter to DBE
Specific concerns regarding the quality assurance of the NSC examination and assessment:
Non-adherence to timeframes for the submission of question papers must be addressed. It is important to note that for the credibility of the NSC examination it is vitally important that every effort is made to adhere to agreed deadlines for setting and moderation of question papers. Any delays affect Umalusi in its quality assurance exercise.
In addition to what is said above, every effort should be made to set question papers of an appropriate quality and standard so that they can be ideally approved at first moderation.
Low levels of performance in key subjects. This issue need to be looked into and addressed so as not to compromise the standard of the NSC.
• Umalusi is concerned with the problem in Life Orientation with
awarding of marks at the 80% level where there is clear
evidence of tampering with the marks. The DBE is urged to
send a strong message of warning to the provinces indicating
that should this problem persist in the future all learners at the
80% mark will be adjusted downward.
The quality of School-Based Assessment (SBA) continues to
be a concern. There may be several plausible reasons for this
poor quality of SBA, but it is clear that there is too little control
over work that is done.
It is evident from our reports that progress has been made over
the years with regard to the standard of marking. There is
however room for improvement in the selection and training of
markers, chief markers, senior markers and internal
moderators. Competency tests will go a long way in the
identification of competent markers.
Conclusion The findings of the quality assurance processes are a clear
indication of a maturing system that has, on the one hand, made positive strides towards improvement in certain areas of assessment and examination, but, on the other hand, still has a few challenges that need to be addressed.
The quality assurance of each of these processes presented above was conducted based on Umalusi criteria. Umalusi uses criteria that are subjected to constant review and refinement, to ensure that they are in line with current trends in assessment and examinations.
Conclusion… In general Umalusi is pleased with the manner in which the
2012 NSC examination was administered.
Umalusi acknowledges that a number of technical irregularities were reported, but these were addressed in a fitting manner.
Umalusi takes this opportunity to express appreciation to the national & provincial departments of education for their concerted effort in ensuring a credible examination.
Umalusi expresses appreciation also to all the relevant stakeholders for the necessary support given in line with Umalusi quality assurance initiatives.
Thank you!