Grading Review Sampling

Report and analysis

3

Introduction

Our responsibility

It is the duty of the Awarding Organisation to put in place measures to safeguard the quality, integrity and value of the qualifications it offers.

4

Background

The drivers

• BTEC benchmarking data shows an increase in higher grades awarded at Level 3 over recent years

• ALPs data shows higher grades awarded compared to “predicted grades” for individual learners– www.alps-va.co.uk

• There are several factors which may have influenced this, but we need to ensure that we have all the facts

• We also need to ensure that our quality assurance processes are supporting accurate and standardised assessment, particularly at the higher grades

• Vocational education in general is subject to increasing scrutiny

5

Claims at “DDD” grade: 2010-11

Sector>=

20%>=

30%>=

40%>=

50%>=

60%>=

70%>=

80%>=

90%=

100%

Applied Science 138 105 66 38 19 9 6 3 1

Business 308 200 132 80 53 41 25 15 13

Construction 84 65 37 22 8 5 5 2 0

Engineering 126 83 51 31 17 6 5 2 2

Health & Social Care 285 234 180 131 79 45 23 14 10

Hospitality 26 17 13 10 7 3 2 2 2

IT 231 156 102 61 36 19 10 4 3

Landbased 84 64 42 23 9 6 5 3 2

Learning & Development 137 116 77 53 38 24 11 4 1

Media , Arts & Design 238 137 67 42 23 17 13 8 8

Performing Arts 311 233 171 119 82 62 45 30 27

Sports 372 284 206 140 90 59 46 31 22

Public Services 147 86 47 25 15 9 6 3 2

Travel & Tourism 94 58 39 27 18 9 5 3 3

Overall 2581 1838 1230 802 494 314 207 124 96

6

Our response

Our response

• Grading Review Sampling formed part of our wider response

• New role of Grade Reviewer, recruited from Grade A and B Standards Verifiers

• A modified version of Confirmation Sampling:– Standards Verification visit, but with more focused sampling

strategy– Audit of assessment records with reference to certificate claims– Involved programmes where 30% or more learners achieved

“DDD” at BTEC Nationals in 2010-11

• Note: cohorts of 5 or fewer were not included– They were subject to normal Confirmation Sampling

7

How it went

Centres

• Mainly FE colleges involved

• Several colleges had a high number of allocations

• Significant concerns expressed by many centres

• More positive after context was explained

• Positive feedback following visits

Resource implications

• Manual allocations

• Increased workload for Deployment, Quality Standards, RQMs, SVs

• Old NQF Sectors – QCF Principal Subject Areas

8

Statistics

Results:

Reasons why reports were blocked:

Number of issues:

Programmes reviewed

Programmes released

1st blocks % BlockedReleased

after action & resample

2nd blocks

2065 1495 570 27.6% 539 31

Certification Claims Procedures 189

Assessment Decisions 489

Internal Verification 402

One issue 182

Two issues 314

All three issues 74

9

Statistics

Sector Blocked Released % Blocked

AEROSPACE NQF1-3 0 1 0.0%

AGRICULTURE NQF1-3 3 13 18.8%

ANIMAL CARE NQF1-3 1 45 2.2%

APPLIED SCIENCE NQF1-3 28 23 54.9%

ART AND DESIGN NQF1-3 32 183 14.9%

BUILDING SERVICES NQF1-3 2 0 100.0%

BUSINESS NQF1-3 39 119 24.7%

CCLD NQF1-3 26 52 33.3%

CIVIL ENGINEERING NQF1-3 0 1 0.0%

CONSTRUCTION NQF1-3 30 22 57.7%

COUNTRYSIDE/FISH NQF1-3 6 24 20.0%

DANCE NQF1-3 8 32 20.0%

10

Statistics

Sector Blocked Released % Blocked

DENTAL TECHNOLOGY NQF1-3 1 0 100.0%

E-BUSINESS NQF1-3 0 2 0.0%

ELEC/ELECTRONIC NQF1-3 7 17 29.2%

E-MEDIA NQF1-3 7 9 43.8%

ENGINEERING NQF1-3 9 27 25.0%

FLORISTRY/HORTICULTURE NQF1-3 2 12 14.3%

FORENSIC SCIENCE NQF1-3 16 26 38.1%

HAIRDRESSING/BEAUTY NQF1-3 6 11 35.3%

HEALTH & SOCIAL CARE NQF1-3 58 179 24.5%

HOSPITALITY NQF1-3 2 10 16.7%

IT PRACTITIONERS NQF1-3 51 129 28.3%

LAW AND LEGAL NQF1-3 0 1 0.0%

11

Statistics

Sector Blocked Released % Blocked

MECH + MAN NQF1-3 10 16 38.5%

MUSIC PRACTICE NQF1-3 22 44 33.3%

MUSIC TECHNOLOGY NQF1-3 11 43 20.4%

OPS + MAIN NQF1-3 0 1 0.0%

PERF. ARTS PERFORMANCE NQF1-3 63 116 35.2%

PERF. ARTS PRODUCTION NQF1-3 12 22 35.3%

PUBLIC SERVICES NQF1-3 22 46 32.4%

SPORT + EX. SCIENCE NQF1-3 8 40 16.7%

SPORT NQF1-3 38 155 19.7%

TRADITIONAL MEDIA NQF1-3 32 48 40.0%

TRAVEL + TOURISM NQF1-3 10 24 29.4%

VEHICLE REPAIR NQF1-3 8 6 57.1%

12

Conclusions

• Plenty of good practice observed

• No deliberate assessment malpractice found

• Sampling strategy identified more issues, but blocks rates on assessment decisions are similar to SV

• Centre issues highlighted:○ Management of staff turnover○ Retention of current learner evidence○ Retention of learner records○ Individual assessors, rather than entire teams assessing

inaccurately

Next steps

• An initial report from Regulation, Standards & Research has been sent to the regulator

• No repeat of Grading Review Sampling, but it will inform future QA processes○ Upgrading of “Registration & Certification” Objective in Quality

Review & Development○ Overall grade boundaries being looked at○ Standards Verification must sample a range of grades where

possible○ Guidance and requirements being made more explicit○ Stronger recommendations on assessment practice (e.g.

resubmissions)

• Development of next generation BTEC Nationals from 2014○ Consulting FE sector on potential QA models

13