DEEP DIVE : MONITORING & GOVERNANCE – MANAGING KPIs, RISK & PERFORMANCE IN APPRENTICESHIPS

May 2018 Funding & Performance Management Rules V6

Rebecca Rhodes, Senior Associate, UVAC

r.rhodes@bolton.ac.uk

Documentation (with hyperlinks)

Agenda Policy and Rules

External Monitoring and KPIs

Key Performance Indicators and Setting Standards

Reporting and Analysing Data

Governance

EXTERNAL MONITORING

QA-09 Describe how:•Leaders and managers in your organisation communicate their plans and expectations in order to attain high standards of training•You will ensure you have appropriate staff in place including trainers with expertise relevant to the apprenticeship training that you will offer•The apprenticeship training delivered by staff will be up-to-date

QA-10 By considering an apprentices training from start to end, describe how you ensure that your apprenticeship delivery:• meets the needs of both apprentices and employers• is relevant to the apprenticeship frameworks and/or standards that you offer• engages, challenges and inspires apprenticesPlease note that you must address all the points above including detail on training from start to end

QA-11 Describe:• How you assure the quality of your apprenticeship training• What staff performance mechanisms you have/will have in place, and how these are managed• How you measure/will measure the success of apprenticeship delivery across your organisation and how you will use this to improve your apprenticeship trainingPlease note that you must address all the points above

RoATP Narrative Questions – HOW do you ..

‘A quality risk identified from any source is escalated to the Pro-Vice Chancellor

Education with monitoring via the Academic Standards & Quality

Enhancement Committee’.

‘The Associate Dean for Education, who sits on Education

Executive Committee , chairs the monthly

Apprenticeship Steering Group

monthly.’

‘We work with appropriate

industry fora to ensure a full

annual review of the programme

annually’

‘Data on; retention, progression and success; student

and employer satisfaction; and

employment outcomes, are all

benchmarked annually by our

Monitoring Group’

‘Feedback from employers is part of the quarterly

client/apprentice review meetings, annual reviews and the course review process’.

Managing KPIs & Risk

Governancethe system

regularly asks questions about risk, exceptions

and progress based on data

Its Someone’s Job -

Accountability

named people are accountable at every level for

action on risk and progress and exceptions

Data and Evidence

is an input to every discussion, and targets are

set where issues arise)

Processes thresholds are set that trigger risk discussions

everyone questions about risk, exceptions

and progress

Monitoring & Review

people measure, discuss and take action on risk, exceptions and

progress

In Brief – Our Survey* Said

A few had

financial / earnings targets for the programme

Most governed by

the current or ex ‘Project Group’

Most felt academic

team monitoring was ‘adhoc’ or not currently sufficient

Most had some

form of satisfaction survey – though no target

Only some regularly produced data and tracking reports to assess risk

50% had one or

more KPIs, though often without a target

Main KPI in most

HEIs was employer pipeline &/or Apprenticeship starts

No-one had a

compliance improvement KPI !!

Very varied and

polarised response to tracking attendance ☺

All conducted module

reviews with apprentices but some not mandatory

About 50 % had

an Executive Board sponsor as Chair/ present

*not very scientific survey

Some still concerned

about tracking the 20 % off the job

NATIONAL APPRENTICESHIP MONITORING

Funding Rules & Compliance

Eligibility

Subcontracting

Evidence Pack

Documents

ILR matching

Learner & Employer Satisfaction Survey

• Purpose = informed consumers and purchasers

• Annual with a minimum sample size

• Providers manage this survey, distributing links to a standard questionnaire

• IPSOS MORI follow up if the minimum sample size is not reached

• Provider’s own results are available

• National Tables provide further data

• Anonymous

• http://dera.ioe.ac.uk/30301/1/Employer_Satisfaction_national_report_19.10.17.pdf

Qualification Achievement Rates • Purpose = informed consumers and purchasers by showing the relative quality

of provision• Ofsted use them as part of their inspection process• QARs are used in the National Achievement Rate Tables NARTS –

– Level 4+ grouped together – No HEI or HEP provider category

• Calculated for individual qualifications or programmes for apprenticeships. No. starts that achieved

• Typically calculated at a qualification or programme level and aggregated across different types of course, or for particular colleges or providers.– https://www.gov.uk/government/collections/sfa-national-success-rates-tables– https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_da

ta/file/677223/Apprenticeships_QAR_Dataset_Production_Spec_2016-17_V3.pdf– https://www.gov.uk/government/statistics/national-achievement-rates-tables-2016-to-2017

• 2018 Specification –– https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_da

ta/file/712365/Apprenticeships_QAR_Dataset_Production_Spec_2017-18_v1.pdf– Exceptions (Break in Learning, Transfers)

Achievement Rates

Achievement Rates by SSA

Funding Monitoring • Structured analysis of all apprenticeship ILR data to look for issues and concerns • Specification published each year • In addition the general monitoring applied to all provision, there are some

points that are for apprenticeships only

MONITORING & KPI FRAMEWORK

Monitoring & KPI Framework

Learner & Employer

Satisfaction

Quality Achievement

Rates

Starts & Achievements

Funding Monitoring

RoATP Compliance

Audit & Evidence Review

Quality Compliance

Success

Can you see anything yet ….?

VISIBILITY Triage

Manage MonitorRemove

KPI* & Governance Framework

University

Faculty

Subcontractor(s)

School

Course

Individual Individual SuccessProgression, completion, satisfaction, destination

Programme Success:1. Financial (forecast earnings,

timely claims and payments ) 2. Compliance (evidence pack and

ILR error rate)3. Quality (achievement, retention

/ success & satisfaction) 4. Growth (business & starts

volumes, new delivery areas, destinations )

Organisational SuccessRemuneration, reputation, impact..

Course Leader

Course Leaders

/ Head of School

Sponsor

Dean / PVC

Academic Mentor

RISK

REGISTER

*Set KPIs where you have a risk or an opportunity and add trigger points to drive

monitoring and review

Subcontract Monitoring

You must carryout a

regular and substantial

programme of QA checks visits at short notice

You must manage and monitor to

ensure high quality delivery that meets the ESFA funding

rules

You must report any

discrepancies

You cannot transfer the

employer relationship lead

You are responsible for the quality of

delivery by your subcontractors

include whether the apprentices exist and are

eligible – i.e. are meeting Funding

Rules

observation of initial guidance,

assessment and delivery of training and/or on-programme

assessment

Monitoring Individual Success/Risks

Risk Assess at Tri-partite review :

Individual KPI Monitoring • Progress Assessment

– Assignment Grades – Contribution – Attendance – Trajectory ..on schedule? *– Timeliness of Submissions

• Action Plans – Development – Stretch– Referrals

• Complaints • Compliance

– 20 % off the job *– Employer Environs e.g. Working

Week or support

• Overall RAG

• The Commitment Statement is their KPI

• Monthly Assessment and review

• Issues Feed

– Monthly / Bimonthly Course Review

– Course Risk Register

Cohort & Course Review

• Monthly Review – Data drives the triage

process – Risk Review based on

trends – Same list as individual, but

aggregated plus action planning trends

• Termly Review – Formal data review

including module and satisfaction reviews

1. Financial (forecast earnings, timely claims and payments )

2. Compliance (evidence pack and ILR error rate)

3. Quality (achievement, retention / success & satisfaction)

4. Growth (business & starts volumes, new delivery areas, destinations )

Cohort & Course Review – University

• Monthly Review –Apprenticeship Hub – Data drives the triage process – Risk Review based on trends– Use cross cutting data analysis

• Termly Review – Steering Group – Formal data review including

module and satisfaction reviews and compliance audits

– Reports from wider teams e.g. finance, Records/Registry, business development

1. Financial (forecast earnings, timely claims and payments)

2. Compliance (evidence pack and ILR error rate)

3. Quality (achievement, retention / success & satisfaction)

4. Growth (business & starts volumes, new delivery areas, destinations )

Satisfaction Monitoring

• End of autumn term

– After first Tri-partite review

– After first module

– Sufficient work place and off the job

• With ESFA Annual survey ?

• At end of Summer Term

• ESFA is anonymous and my not add to the course review

Annual KPI Monitoring

1. Financial (forecast earnings, timely claims and payments ) 2. Compliance (evidence pack and ILR error rate)3. Quality (achievement, retention / success & satisfaction) 4. Growth (business & starts volumes, new delivery areas,

destinations ) • Time series trends• Sector, demographic, size cross cutting trends

University, Faculty, School and Course - same questions, different scope

Starts and Completions Data Analysis

Sector/Subject/School

Employer Size

ApprenticeshipLevel

Demographic Grouping

TRENDS TRENDS

OVER TIME

FORECAST

GOVERNANCE

Common Governance Gaps • Insufficient scope/remit in the group to drive quality and changes

• Split between Teaching and Learning or quality or other programme governance structures and Performance Structures…

• No cross organisational remit on Risk

• Not decision making or taking – made ‘recommendations’

• Did not meet frequently enough to add value

• Focus on ‘Managing the project’ not ‘improving the product’

• Remit Not on addressing Risk

• Isolated not Integrated with wider university governance

• No statistical input to inform performance or risk assessment

• Risk registers focussed on implementation

• Meetings don’t connect as a circle….one does not feed the other

• No-one collating a single picture of the overall programme

• Not triaging sufficiently so very time and resource hungry

Timetable of Monitoring Monthly Termly Annually At Course

Completion

Quality (achievement, retention / success & satisfaction)

X X X X

Growth (business & starts volumes, new delivery areas, destinations )

X X X

Compliance (evidence pack and ILR error rate) X ILR X EP X EP X EP

Financial (forecast earnings, timely claims and payments X X

KPI, Risk & Governance Matrix Meeting Frequency

Remit Data Input Remit

Individual Apprentice KPI

Monthly Progress Triage Action Plan Trends

Take action needed to ensure apprentice is on planned trajectory

Cohort / Course Monthly Take action needed to ensure course meets KPIs:

School Monthly

Faculty Termly

University Steering Group

Termly

University Corporate Annually

School / Faculty Annually

KPI Management Good Practice

Dust off your RoATP and map what you said you would do against a timeline/ schedule

Performance Monitoring delivers Performance Management (‘Unless you know; you don’t know’) - create a data suite and reports

Monitor and model based on actual data and forecast frequently - £££, volumes, achievement/completion forecasts

Set trigger points to automatically drive an action to improve e.g. (£ outstanding, error rate apprentice progression) and ACT QUICKLY

Ensue HESA/ ILR submissions are timely and supported by the paperwork – and check they match on key dates. Use the action plan / triage process to inform file selection of file compliance checks

Rigorously enforce governance arrangements (Internal and with subcontractors ) everyones priority

Questions

Please use the questions facility to add your feedback and comments about the webinar

Any queries or anomalies - please email me r.rhodes@bolton.ac.uk

Webinars To Book : Audit and the Evidence Pack Webinar

Webinars to Review :Preparing for the RoATP Compliance Visit Subcontracting