Grading Review Sampling
description
Transcript of Grading Review Sampling
Grading Review Sampling
Report and analysis
3
Introduction
Our responsibility
It is the duty of the Awarding Organisation to put in place measures to safeguard the quality, integrity and value of the qualifications it offers.
4
Background
The drivers
• BTEC benchmarking data shows an increase in higher grades awarded at Level 3 over recent years
• ALPs data shows higher grades awarded compared to “predicted grades” for individual learners– www.alps-va.co.uk
• There are several factors which may have influenced this, but we need to ensure that we have all the facts
• We also need to ensure that our quality assurance processes are supporting accurate and standardised assessment, particularly at the higher grades
• Vocational education in general is subject to increasing scrutiny
5
Claims at “DDD” grade: 2010-11
Sector>=
20%>=
30%>=
40%>=
50%>=
60%>=
70%>=
80%>=
90%=
100%
Applied Science 138 105 66 38 19 9 6 3 1
Business 308 200 132 80 53 41 25 15 13
Construction 84 65 37 22 8 5 5 2 0
Engineering 126 83 51 31 17 6 5 2 2
Health & Social Care 285 234 180 131 79 45 23 14 10
Hospitality 26 17 13 10 7 3 2 2 2
IT 231 156 102 61 36 19 10 4 3
Landbased 84 64 42 23 9 6 5 3 2
Learning & Development 137 116 77 53 38 24 11 4 1
Media , Arts & Design 238 137 67 42 23 17 13 8 8
Performing Arts 311 233 171 119 82 62 45 30 27
Sports 372 284 206 140 90 59 46 31 22
Public Services 147 86 47 25 15 9 6 3 2
Travel & Tourism 94 58 39 27 18 9 5 3 3
Overall 2581 1838 1230 802 494 314 207 124 96
6
Our response
Our response
• Grading Review Sampling formed part of our wider response
• New role of Grade Reviewer, recruited from Grade A and B Standards Verifiers
• A modified version of Confirmation Sampling:– Standards Verification visit, but with more focused sampling
strategy– Audit of assessment records with reference to certificate claims– Involved programmes where 30% or more learners achieved
“DDD” at BTEC Nationals in 2010-11
• Note: cohorts of 5 or fewer were not included– They were subject to normal Confirmation Sampling
7
How it went
Centres
• Mainly FE colleges involved
• Several colleges had a high number of allocations
• Significant concerns expressed by many centres
• More positive after context was explained
• Positive feedback following visits
Resource implications
• Manual allocations
• Increased workload for Deployment, Quality Standards, RQMs, SVs
• Old NQF Sectors – QCF Principal Subject Areas
8
Statistics
Results:
Reasons why reports were blocked:
Number of issues:
Programmes reviewed
Programmes released
1st blocks % BlockedReleased
after action & resample
2nd blocks
2065 1495 570 27.6% 539 31
Certification Claims Procedures 189
Assessment Decisions 489
Internal Verification 402
One issue 182
Two issues 314
All three issues 74
9
Statistics
Sector Blocked Released % Blocked
AEROSPACE NQF1-3 0 1 0.0%
AGRICULTURE NQF1-3 3 13 18.8%
ANIMAL CARE NQF1-3 1 45 2.2%
APPLIED SCIENCE NQF1-3 28 23 54.9%
ART AND DESIGN NQF1-3 32 183 14.9%
BUILDING SERVICES NQF1-3 2 0 100.0%
BUSINESS NQF1-3 39 119 24.7%
CCLD NQF1-3 26 52 33.3%
CIVIL ENGINEERING NQF1-3 0 1 0.0%
CONSTRUCTION NQF1-3 30 22 57.7%
COUNTRYSIDE/FISH NQF1-3 6 24 20.0%
DANCE NQF1-3 8 32 20.0%
10
Statistics
Sector Blocked Released % Blocked
DENTAL TECHNOLOGY NQF1-3 1 0 100.0%
E-BUSINESS NQF1-3 0 2 0.0%
ELEC/ELECTRONIC NQF1-3 7 17 29.2%
E-MEDIA NQF1-3 7 9 43.8%
ENGINEERING NQF1-3 9 27 25.0%
FLORISTRY/HORTICULTURE NQF1-3 2 12 14.3%
FORENSIC SCIENCE NQF1-3 16 26 38.1%
HAIRDRESSING/BEAUTY NQF1-3 6 11 35.3%
HEALTH & SOCIAL CARE NQF1-3 58 179 24.5%
HOSPITALITY NQF1-3 2 10 16.7%
IT PRACTITIONERS NQF1-3 51 129 28.3%
LAW AND LEGAL NQF1-3 0 1 0.0%
11
Statistics
Sector Blocked Released % Blocked
MECH + MAN NQF1-3 10 16 38.5%
MUSIC PRACTICE NQF1-3 22 44 33.3%
MUSIC TECHNOLOGY NQF1-3 11 43 20.4%
OPS + MAIN NQF1-3 0 1 0.0%
PERF. ARTS PERFORMANCE NQF1-3 63 116 35.2%
PERF. ARTS PRODUCTION NQF1-3 12 22 35.3%
PUBLIC SERVICES NQF1-3 22 46 32.4%
SPORT + EX. SCIENCE NQF1-3 8 40 16.7%
SPORT NQF1-3 38 155 19.7%
TRADITIONAL MEDIA NQF1-3 32 48 40.0%
TRAVEL + TOURISM NQF1-3 10 24 29.4%
VEHICLE REPAIR NQF1-3 8 6 57.1%
12
Conclusions
• Plenty of good practice observed
• No deliberate assessment malpractice found
• Sampling strategy identified more issues, but blocks rates on assessment decisions are similar to SV
• Centre issues highlighted:○ Management of staff turnover○ Retention of current learner evidence○ Retention of learner records○ Individual assessors, rather than entire teams assessing
inaccurately
Next steps
• An initial report from Regulation, Standards & Research has been sent to the regulator
• No repeat of Grading Review Sampling, but it will inform future QA processes○ Upgrading of “Registration & Certification” Objective in Quality
Review & Development○ Overall grade boundaries being looked at○ Standards Verification must sample a range of grades where
possible○ Guidance and requirements being made more explicit○ Stronger recommendations on assessment practice (e.g.
resubmissions)
• Development of next generation BTEC Nationals from 2014○ Consulting FE sector on potential QA models
13