Portfolio Committee on Higher Education

Report on the Quality Assurance of NC(V) Level 2-4; NATED N1-3 and

GETC: ABET L4

Wednesday, 3 June 2015Dr Mafu S Rakometsi

WHAT CAN WE LEARN FROM THE NSC RESULTS?

Introduction to context, principles, approaches and

processes Dr Mafu S Rakometsi

Chief Executive Officer of Umalusi

Regulatory Framework Established through the promulgation of the General

and Further Education and Training Quality Assurance (GENFETQA) Act, no. 58, 2001, as amended 2008;

Two predecessors, namely the Joint Matriculation Board - JMB (1918) and the South African Certification Council – (SAFCERT) (1986); and

Umalusi started work in 2001 having taken over from SAFCERT.

Regulatory Framework

GENFETQA ACT, 2001 amended 2008

Section 17A.

(1) The Council must assure the quality of assessment at exit points.

(3) The Council must perform the external moderation of assessment of all

assessment bodies and education institutions.

(4) The Council may adjust raw marks during the standardisation process.

Section17 of the GENFETQA Act(5) The Council must, with the concurrence of the Director-General

and after consultation with the relevant assessment body or education institution, approve the publication of the results

of learners if the Council is satisfied that the assessment body or education institution has—

(i) conducted the assessment free from any irregularity that may

jeopardise the integrity of the assessment or its outcomes;

(ii) complied with the requirements prescribed by the Council for conducting assessments;

(iii) applied the standards prescribed by the Council which a learner is required to comply with in order to obtain a certificate; and

(iv) complied with every other condition determined by the Council.

Regulatory FrameworkNational Qualifications Framework (NQF) Act, 2008 (Act No. 67 of 2008):

The (3) Quality Councils were established.

Umalusi - Quality Council for the General and Further Education and Training.

Development, management and promotion of the GENFETQS-f.

The capacity of private providers to offer qualifications that are registered on the GENFETQS-f.

NQF Act Section 27 (h)

The QC must develop and implement policy and criteria for assessment for the qualifications on its sub-framework

Framework for Quality Assurance of Assessment

Evaluation and /or accreditation of assessment bodies;

Periodic inspection of assessment systems;

Ongoing monitoring of assessment systems;

Quality assurance of external examinations through:

Moderation of examination question papers; Monitoring and moderation of SBA; Monitoring the conduct of examinations; Moderation of marking; and Standardization of assessment outcomes.

Quality Assurance of theDHET, IEB and BENCHMARK

November 2014 examination

Ms Faith Ramotlhale

Senior Manager: Quality Assurance of Assessment

Moderation of question papers

PURPOSE

To ensure that the question papers are:-

of the required standard; and

are relatively: fair reliable representative of an adequate sample of the curriculum representative of relevant conceptual domains representative of relevant levels of cognitive challenge

Approach

DHET:NATED/NC(V) DHET (GETC) IEB (GETC) BENCHMARK (GETC)

Offsite Centralized Offsite Offsite

Moderation Process

•Question paper development - examiners (assessment body)•Internal moderation - Internal moderators (assessment body)•External moderation - External moderators (Umalusi)•Subsequent moderations and approval

Moderation of the question papers

Technical criteria

Internal moderation

Content coverage

Text selection, types and quality of questions

Cognitive demands

Marking memorandum/ guidelines

Language and bias

Predictability

Criteria

Question Paper Approval

Assessment body

No of papersModerated

Approval of Papers

DHET: GETC 26/26 54% first moderation46% second and third moderation

IEB:GETC 8/8 75% first moderation25% second moderation

BM:GETC 4/4 75% first moderation25% second moderation

DHET: NATED N2-3

N2: 15/15N3: 23/23

Different model used

DHET: NC(V) L2-4 L2: 25/25L3: 24/25L4: 80/80

Different model used

Moderation of the question papers

Areas of Good Practice

No question paper leakages( all GETC- BM,IEB & DHET)

Questions were not predictable- Creativity and innovation (GETC- BM,DHET & IEB)

The setting of an extra set of papers, first step towards eighteen month cycle, was a step in the right direction (DHET-NC(V))

To ensure equivalence of standards, the November, March and back-up examination question papers were moderated simultaneously, (DHET-NC(V) and NATED)

Moderation of Question PapersAreas of concern

Improve the quality of internal moderation, (DHET, IEB, BM).

Training of examiners and internal moderators to enhance understanding of Subject and Assessment Guidelines, (GETC- DHET & Benchmark)

Focus areas of training: depth and breadth (ECD4, INCT4, LCZU4, LCSO4, LCXI4, TECH4, TRVT4 and WHRT4), (GETC-DHET)

Poor quality memoranda - over reliance on Memorandum Discussion process to improve quality of marking guidelines, (GETC-DHET)

Moderation of Question Papers

Areas of concern

The question papers for A4TECH & A4HSSC did not meet the cognitive demands at first moderation (GETC-IEB)

Correct versions question papers, marking guidelines, internal moderator reports and assessment frameworks to be made available for external moderation, (DHET -NC(V))

Resubmission of rejected question papers to Umalusi – turn around time to be improved, (DHET - NC(V)& NATED)

Standard of back-up papers to be the same as November and supplementary papers, (DHET-NC(V)

Quality and standard of internal moderation – a rigorous process should be evident, (DHET-NATED and NC(V))

Moderation of Question Papers

Areas of concern

Outdated and/or underspecified curricula requires urgent revision (DHET-NATED);

Predictability of papers due to repetition of questions from previous question papers should be avoided; (DHET-NATED), and

Quality of assessment grids and marking guidelines should be improved (DHET-NATED, NC(V)).

Internal Assessment (SBA)Definition and Purpose

Internal assessment refers to any assessment conducted by the provider , the outcome of which count towards the achievement of the qualifications

GETC: 50%; NC(V) fundamental subjects 25%; NC(V) vocational subjects 50%; and NATED subjects 40%

Ascertain the degree to which the assessment body is attempting to ensure standardisation across providers

Ascertain the standard and quality of the tasks

Establish the extent and quality of internal moderation and feedback, and

Verify the reliability and validity of the assessment outcomes

Site-Based Assessment: GETCAreas of Good Practice

Year-on-year improvement regarding the administration compliance requirements, (DHET)

The SBA user guides developed by IEB were helpful in structuring the content of Common Assessment Tasks

Moderation of Internal Assessment - GETC

Assessment Body

Common Assessment Tasks (CATs) moderated and approved

CASS

DHET 5 sets x 29 learning areas 492 portfolios moderated for 29 LA

IEB - 120 portfolios moderated for 6 LA

Benchmark 2 x 2 learning areas 107 portfolios moderated for 2 LA

Moderation of Internal Assessment - GETC

Areas of concernNo system to monitor the implementation of the CATs, (IEB)The CATs did not adequately cover the specific assessment criteria as specified in the unit standards at first moderation, (BM)The weightings of the CATs also did not comply with the SAG requirements, (BM)Centres do not submit educator portfolios with sample requested – no evidence of assessment plans (IEB, Benchmark, DHET)Assessors (educators) do not understand the use of rubrics, and marking guidelines (Benchmark, DHET)

Moderation of Internal Assessment -

GETC Areas of concern

Poor compilation of SBA portfolios and insufficient supporting documents provided, (Benchmark)

Very poor internal moderation system and processes (DHET)

Authentication is ad-hoc and sporadic (considering 50% weighting) (DHET)

Substantial differences between SBA and examination marks (DHET).

Moderation of Internal Assessment NC(V)

Areas of concern: ISAT

The quality of some tasks remains a cause for concernEnrolments must be in line with available resources to effectively assess candidates’ acquisition of the practical skills required in the workplaceISATs should be distributed to colleges in time for effective procurement and implementationCare should be taken to prevent subjective and overly lenient markingModeration should be improvedAbsenteeism among students affected the completion of ISATs,

Internal Assessment : ICASS

Directives for Improvement (NC(V)

Improving and deepening the understanding of the latest ICASS Guidelines (NCV)

Development of assignments and tasks

Implementation of practical assessments is still problematic,

Implementation of analysis grids and Bloom’s taxonomy

Development of marking tools and rubrics

Assessment and moderation practices need improvement

Moderation of Internal Assessment ICASS

Areas of good practice: (NATED)

There were pockets of excellence at some of the sites moderated. dedication of innovative lecturers  

Areas of concern

To ensure the availability and implementation of the latest version of ICASS guideline

Training of lecturers - limited experience in the industry

Teaching aids e.g. models are needed to improve learners’ understanding of the subject content

To address class attendance and retention rates at some colleges

Monitoring of Examinations: GETC

Areas of Concern (DHET) Lack of storage rooms and inadequate security for question

papers

Transportation of exam material using public transport (EC)

Candidates allowed to write without proper identification

Some centres keep answer scripts overnight - poor security

Assessment body DHET IEB Benchmark

No exam centres 125 7 2

No of monitors 68 4 2

Monitoring of Examinations: GETC

Areas of concern

Low number of candidates who wrote the exam, (IEB, BM)

Poor compliance with exam regulations (BM, IEB)

Inadequate storage facilities for QPs with poor security arrangements

Poor administration regarding exam regulations

Chief invigilators lack expertise to manage exams

Private exam centres often apply to the IEB to conduct the exam on alternative dates (IEB)

Monitoring of Examinations: TVET

Areas of good practice General management of the examinations was good in

terms of security of papers, appointment and training of invigilators

Areas of Concern

Irregularities during the writing phase of the examination, candidates were found with cell phones

Others entered the examination without identity documents or examination

Some Colleges monitored, papers were not sent to the marking centre within 24 hours, nor securely sealed

Monitoring Of Marking Phase: GETC

Areas of good practice Marking centres were generally well organised and suitable for the task, (DHET,IEB,BM)

Adequate and experienced security personnel at the gates and entrances to the venues, (DHET,IEB, BM)

Training of marking personnel was generally thorough and effective, (DHET,IEB,BM)

DHET IEB Benchmark

No. of marking centres 8 1 1

No. centres monitored 8 1 1

Monitoring of Marking Phase: GETC

Areas of Concern

Shortage of markers in NC and KZN due to appointing markers for more than one learning area, (DHET)

Internal moderation model different and inconsistent across marking centres,(DHET)

Markers in LP disturbed the training process – apparently because of outstanding remuneration issues, (DHET)

The IEB marking centre management plan was not well documented – its based on experience and practice, (IEB)

There was no security in the building where the actual marking was done to control the flow of scripts, (BM)

Monitoring of Marking Phase: TVET

Areas of good practice

Safety and security of marking personnel and material was in general very good

Markers were aware of what constituted an irregularity and how it should be dealt with.

Monitoring of Marking Phase: TVET

Areas of concern

Distribution incorrect versions of marking guidelines - English and Afrikaans

Quality of marking and internal moderation of low enrolment subjects

Inappropriate allocation of marking venues according to the nature of subject, and number of markers - drawing subjects

PURPOSE

Moderation of marking determines the standard and quality of marking and ensures that marking is conducted in accordance with agreed practices

Umalusi engages with the following processes during the moderation of marking:

Centralised memorandum discussions

Verification of marking (centralised and on-site)

Verification of Marking

Areas of good practice Good procedure observed: Panel discussions followed by marking of exemplar scripts – marking commences after markers have demonstrate an acceptable level of competence, (IEB,BM)

Analysis of evidence bound in educator portfolios were meticulous and contained all relevant documents as per policy requirements

Areas of concernPoor practice of pre-marking in 15/26 LAs – discussions very difficult and time-consuming, (DHET)

No pre-marking or preparing alternative responses in advance, (IEB)

Memorandum Discussions

Memorandum Discussion TVET

Areas of good practice

Sample marking improved

Areas of concern

Appointments should be finalised well in advance

Marking personnel must attend marking guideline discussions,

Preparation of marking personnel for the marking process must be improved

Correct version of marking guidelines must be made available

Areas of Good Practice

Good internal moderation practices (DHET, IEB)

Good quality and standard of marking, (DHET, IEB)

Verification of Marking: GETC

DHET IEB Benchmark

Centralised 9 LA 702

Scripts

- -

On-site 5 LA 542

scripts

4 LA 215

Scripts

2 LA 70 scripts

Standardisation and Resulting Provision of GENFETQA – Council may adjust raw

marks

International practice – large scale assessment systems

Standardisation – process used to mitigate the effect of factors other than learners knowledge and aptitude on the learners performance

Sources of variability – difficulty in question paper, undetected errors, learner interpretation of questions

Verification of the Resulting Process

Planned Status

Subject structures Subject structures verified

Candidate registration System verified during state of readiness visits

Generation of mark Sheets Monitored

Capturing of Marks Monitored

Standardisation data & Booklets

Data sets received and verified

Capturing of adjustments Verified

Statistical moderation & resulting

Verified

Statistical Moderation

Scope of standardisation 2014: GETCDecision Number of subjects

DHET IEB BM

Subjects standardised

26 8 2

Raw marks accepted

11 8 2

Moderated Upwards 10 - -

Moderated Downwards

05 - -

Statistical Moderation

Areas of good practice

Booklets and data sets were submitted according to the prescribed timelines (IEB, BM)

Standardisation and approval meetings went smoothly

Statistical ModerationAreas for Concern

Late submission and inaccurate standardisation data delays the approval of results, (DHET)

Late submission of qualitative input reports, (DHET)

Poor performance of learners,30-40% candidates do not write the exam, (DHET)

Non-adherence to the prescribed colours to draw the graphs (IEB)

Very low enrolment numbers (BM)

Statistical ModerationScope of standardisation 2014: NATED / NCV

Decision Number of subjects

NATED NC(V)

Subjects standardised 83 241

Raw marks accepted 46 125

Moderated Upwards 15 38

Moderated Downwards

22 78

Statistical Moderation: TVET

Areas of good practice

Standardisation booklets complied to the prescribed colours

Areas of concern

Late submission of standardisation booklets delayed standardisation

Inaccurate data sets and low capture rate of data delayed the approval of results

Non submission of consolidated irregularity report to Umalusi for standardisation and approval of results

Conclusion The quality assurance processes presented above were

conducted based on Umalusi criteria. Umalusi uses criteria that are subjected to constant review and refinement, to ensure that they are in line with current trends in assessment and examinations

From the findings of the quality assurance processes many good practices were evident but on the other hand, there are still many challenges that need to be addressed

Significant interventions are required to improve the curriculum, teaching and learning, exposure of learners to a compulsory practical component and assessment for the NATED programmes to make it a meaningful qualification

Conclusion The outdated and underspecified curriculum and lack of

subject assessment guidelines for the NATED programmes creates problems for appropriate quality assurance to be conducted

Umalusi takes this opportunity to express appreciation to the Department of Higher Education and Training for their concerted efforts in delivering credible examinations in 2014

Thank you!