Top compliance challenges for trainers and assessors

Maree Thorne

Compliance Assist

• As at 30 June 2014, there were 4664 registered training organisations in Australia; ASQA was the regulator of 3938 of these providers (84%)

• Approximately 49% of the providers for which AQSA has regulatory responsibility have been the subject to at least one audit activity.

• 3,200 audits undertaken by ASQA between 1 July 2011 and 30 June 2014• During 2013–14, ASQA took action to cancel (25) or suspend (49) the

registration of 74 providers. ASQA also gave 188 written ‘notices of intention’ to cancel or suspend the registration of a provider. In 35 cases, ASQA also imposed another type of administrative sanction, such as placing a condition on a provider’s registration.

• ASQA also undertook its first formal investigations during 2013–14• Successful prosecution of CEO in Brisbane August 2014

A Context

‘When audited, almost 75 per cent of existing providers were unable to demonstrate compliance with Standard 15. Even after the submission of rectification evidence, 21 per cent of providers remain unable to demonstrate compliance with this Standard. The high level of non-compliance with Standard 15 is a concern for ASQA, as Standard 15 governs training and assessment—the core business of a registered training organisation, and the most critical Standard for ensuring the delivery of quality outcomes.’ [ASQA Annual Report p. 26]

From ASQA FAQ ….Common non-compliances include:• RTO Training and Assessment Strategies (15.2)

do not reflect actual business practices of the RTO do not meet Training Package requirements do not indicate sufficient duration to provide sound training and assessment

• RTOs not able to demonstrate that trainers/assessors can demonstrate industry currency and competence at the required level (15.4)

• training and assessment materials are not appropriate for their target students, or not appropriate to the AQF level of the qualification (15.3)

• RTOs not able to demonstrate that objective validation of learning and assessment materials has been conducted (15.5)

• RTOs putting forward third party evidence as primary evidence for skills assessment (15.5; 16.4)

• RTOs not able to demonstrate how they have managed the transition to new qualifications and Training Packages (25)

• information for students is not consistent with other course documentation (16.3)

• records of assessment are insufficient to substantiate competency judgements and what those judgements are based on. (15.5)

Top challenges !!

The VET environment (including audit)

Assessment! (including work

placement, 3rd parties)

Trainer & assessor Competence and currency

T&A Strategies (including industry

consultation, SNR 15.3 and 16)

Pre-enrol engagement

Continuous improvement

Managing Transitions

Source: ASQA Strategic Review 2013 Aged Care Report

‘On too many occasions, in undertaking this review, it was clear to the auditors involved that RTO leadership and staff had a poor knowledge and understanding of the required national standards and of the requirements of training packages, which form a core element of these national

standards’

The VET EnvironmentVET Quality Framework

National Vocational Education and Training

Regulator Act 2011

Standards for RTOs 2015

Australian Qualifications Framework (AQF) 2013

Fit and Proper Person Requirements 2011

Financial Viability Risk Assessment Requirements 2011

Data Provision Requirements 2011

Legal framework NOT a Quality Standard(ASQA General Directions also enforceable under the Act)

Specifies standards for Australian qualifications including:

•Learning outcomes for each AQF level and qualification

•Issuing AQF qualifications (testamur wording, use of logos, format)

•Linkages and student pathways

•Registers to facilitate the public identification and verification of Australian Qualifications Framework

(AQF) qualifications (www.training.gov.au; NSSC)

•Volume of Learning:

The Australian Qualifications Framework Council (2013b) provides advice regarding volume of

learning: that

‘the generally accepted length of a full time year, used for educational participation, is 1200 hours.

It notes that:

The volume of learning allocated to a qualification should include all teaching, learning and assessment

activities that are required to be undertaken by the typical student to achieve the learning outcomes.

These activities may include some or all of the following: guided learning (such as classes, lectures,

tutorials, online study or self-paced study guides), individual study, research, learning activities in the

workplace and assessment activities’.

Regulates and monitors the Standards for

NVR RTO’s which are developed and

endorsed by NSSC

Regulation includes:

Registration of training organisations and

their ‘scope’

Monitoring of RTO performance against the

Standards and 3 ASQA General Directions

Training Packages

Describe skills & knowledge

needed to perform effectively in

the workplace – Training

Product

Developed through Industry

Skills Councils & endorsed by

the National Skills Standards

Council (NSSC)

CISC (*COAG Industry and Skills Council)

* Dec 2013 replaces SCOTESE

VET Reform Taskforce

Understanding auditsASQA undertakes two distinct

types of audit—registration audits

and compliance audits • Initial registrations, renew

registrations, add qual/units to scope

• Complaints, strategic, risk

Focus for 2014 = Equine, Childcare, Security

Site audit or desk audit?

For more information consider signing up to ACPET webinar Are you ready for Audit?

Participating in a regulatory audit‘Auditors may assess completed student assessments, trainer

records of professional development, student completion rates, whether students are exiting with competency in the relevant skills, whether industry recognises those skills and if students are being employed.’

Standards specify WHAT not HOW…..

Up to you to explain and demonstrate HOW you achieve the WHAT – and show evidence that you do

(You have to show the evidence – auditor won’t look for it)

AssessmentKey issues identified by ASQA in non-compliance included:

• insufficient evidence on which to determine competence. No advice to assessors on how to determine sufficient and valid evidence to make a judgement of competence.

• Informal assessment of practical skills; RTOs complete ‘knowledge’ checks through written assessments, and rely on workplaces to complete ‘skills’ checklists; no assessment of required communication skills, in both classroom and online;

• poor options in multiple-choice questions, and in relation to online assessment, the correct answer was achieved by continued ‘button pressing

• units of competency requiring at least partial assessment in the workplace were not assessed in the workplace; RTOs do not generally integrate work placement into their overall training and assessment strategy;

• a reliance on third-party reports from workplace supervisors who are not qualified VET assessors, and who were using a ‘tick the box’ approach to assessment, often containing lists of performance criteria as well as all required knowledge and skills;

• assessment tools did not include observable behaviours

• assessment tools did not cover the ‘normal range of workplace conditions’;

• observation of skills on only one occasion (whereas the training package for some units specifies ‘over a number of assessment situations’);

• validation and moderation of assessment tools not undertaken;

• 50% of RTOs used commercial material which was not modified for the needs of the student group or to the RTO’s training and assessment strategies or the qualification area. Assessment materials (including commercial) did not meet the requirements of the training package or standards.

• Of the RTOs delivering and assessing partly in a classroom environment, fewer than 50% had adequate resources for a simulated workplace environment.

10

Assessment issues in VET (from Chris Robinson, ASQA presentation at National ACPET conference 2013)

(NCVER 2013 Halliday Wynes & Misko)

● trainers and assessors lack the skills and expertise to

conduct valid assessments

● RTOs not providing repeated practice needed to

demonstrate competence

● lack of work placements

● inadequate rigour in assessment process

● unclear role for industry/employers

● lack of systematic validation and moderation

Assessment - Strategies• Plan effective strategies, then tools/instruments to support

(collect the evidence)

• Read the Training Package!!‘The unit CHCC305C Assist clients with medication was assessed by several RTOs in the classroom, despite it

being mandatory that it be assessed in the workplace’

‘A number of RTOs stated they were unaware of these requirements, although these requirements are clearly stated in the relevant units of competency’

• *Validation – includes ‘mapping’ to the unit of competence – BUT must review principles of assessment and rules of evidence and application

• Assessment instructions for the assessor and ‘learner’ including benchmarking

• Practical assessment of skills – appropriate use of ‘3rd party’ evidence

• Design observation tools with observable behaviours (What are you looking for?)

Interpret units into work tasks and/or observable behaviours.

Assessment - strategies• RPL – show HOW the assessor has used the evidence to

make their judgment that all the requirements (of each unit) had been met and that they have documented that process and decisions.

A quality review process. It involves checking that the assessment tool (s) produce(s) valid, reliable, sufficient, current and authentic evidence to enable reasonable judgements to be made as to whether the requirements of the relevant aspects of the Training Package or accredited course have been met. It includes reviewing a statistically valid sample of the assessments and making recommendations for future improvements to the assessment tool, process and/or outcomes and acting on such recommendations.

What? (tools + completed assessments)When?Examine the strategies AND the tools Critique the tool for its Clarity, Content accuracy, Relevance, Content validity (i.e. match

to unit of competency – ‘mapping’), avoidance of bias and appropriateness of language for the target population

Review the tool to check that it has: Clear instructions for completion by candidates; Clear instructions for administration by assessors

The National Quality Council (NQC) (now superseded by the National Skills Standards Council (NSSC)) defined Validation as:

Work placement and 3rd party evidence

‘While the majority of the sample had organised work placements for their students, most

did not have clear delineation of the roles and responsibilities of the RTO and the employer in relation to training and assessment, or how the placement was appropriate to the qualification. [There were] no overarching agreements (or equivalent), with worksites that demonstrate that there is capability to deliver and that students will be able to demonstrate competence in the workplace’

‘RTOs are required to ensure that they assess using trained assessors, yet the workplace supervisor is often requested to complete what are commonly known as ‘third-party reports’, and these reports are used as the sole ‘evidence’ of skills instead of an assessment of skills being conducted by a trained assessor. These third-party reports are not to be confused with assessor observation tools, which may be completed in a workplace by a trained assessor (as described above), or may be used in a simulated workplace environment where this is allowed by the training package. An assessment approach where workplace supervisor reports are used as an input to the assessment of skills carried out by the RTO’s qualified assessor is a valid use of a third party report’

Training & Assessment StrategiesKey areas of non-compliance included:

• training and assessment strategy not consistent with practice;

• target group for the strategy not identified; for example, existing worker or school leaver, English speaker or person with English as a second language.

• Duration of course insufficient in relation to the target group; very short courses that were not sufficient to ensure the attainment of the required competencies

• no provision for literacy assistance or identification of needs;

• strategy not covering the range of work environments as detailed in the unit of competency;

• the strategy did not provide information to trainers on how to deliver the unit of competency;

• no evidence of effective industry consultation;

• no process for reviewing strategy;

• no training and assessment strategy

• training package requirements were not met e.g. pre requisite

units were not identified, the selection of units not accurate

RTOs must have strategies for training and assessment which meet the requirements of the training package.

These strategies are to guide trainers and assessors who are delivering a unit of competency, skill set, qualification, or accredited course to students. They must be developed through effective consultation with industry.

‘Some RTOs stated that their trainers provide effective industry consultation, or that because students are offered work, the [ ] industry must be happy with the strategy. Others offered a letter of support from an employer as evidence. There was little evidence provided of any industry consultation with independent industry experts. Without any effective, independent advice from industry, it is difficult to conclude that an RTO has a training and assessment strategy that will fully meet industry, and learner needs, and the requirements of the training package’

There is no specific format for describing a strategy. However, it should include key parameters such as whether it is delivered in a classroom or work-based setting (or both); how much time will be involved; what if any pre-requisite learning is required; who the targeted learners are (school leavers, existing workers, et cetera); as well as an outline of the content.

Where a qualification requires assessment in a workplace, the RTO should ensure the timing and quantity of work placement is clearly stated in the training and assessment strategy, given that this is a key parameter in supporting outcomes for students.

Understand T&A Strategies – Why? Who?

Durations (TRAINING /learning and assessment strategies)

p. 19 & 20 Users’ Guide to the Standards for RTOs 2015

RTO is required to comply with AQF in applying volume of learning to programs

NB. volume of learning not same as ‘nominal hours’

Cert I = . 5 – 1 year (600 – 1200 hours)

Cert II = .5 – 1 year (600 – 1200 hours)

Cert III = 1 – 2 years (1200– 2400 hours)

Cert IV = .5 – 2 years (600 – 2400 hours)

Diploma = 1- 2 years (1200 – 2400 hours)

‘your RTO must be able to identify and explain any significant variations from the time periods

described in the AQF’ (p. 19)

VOL = all teaching and learning activities such as guided learning (classes, tutorials, online,

self paced), individual study, research, learning activities in the workplace, and assessment

activities.

A STRATEGY is ….a plan of action designed to achieve a long-term or overall aim!

• Identify learner groups and needs – plan for them!

• Review regularly. What is regularly? Systematically!

‘An RTO can change its training and assessment strategy at any time and there is no requirement that the changes be reviewed by a regulator. Such changes might be made to reflect the needs of a new client group or the availability of different resources. Such flexibility is necessary and desirable. However, the requirement that the strategies are developed with effective consultation with industry should mean that the strategies (both as developed initially and as amended) would meet the needs of learners (and others impacted such as employers and work colleagues) and will continue at all times to meet the requirements of the training package.’

• Include details of ‘training’ - how the course is sequenced and structured, the duration (learning sessions and assessment activities), specify resources and where/ how available;

• Specify how learner needs identified AND managed

• Make them meaningful and not generic (specific not template phrases)

• Read the Training Package for requirements

• Should be a working document or a suite of documents

• Identify appropriate ‘industry’- actually ‘consult’ (not necessarily a ‘form’ - a process)

Industry ‘consultation’ – a process(Consultation = a conference at which advice is given or views are exchanged)

Is NOT: - marketing, sales, Business

Development

- Employer satisfaction feedback

- A checklist

IS: - Planned, a process, structured

- involvement in the development of how, what, to whom, when you will provide training and assessment services

Industry Consultation - Strategies

• Who is industry? Understand what do you want to get from them?

• Identify engagement opportunities. Systematically program (review T&As?)

• Establish a system for recording /evidencing industry engagement; record who you spoke with, why relevant, what you spoke about (qual/unit) and what the outcome was.

• Raise opportunities for improvement that result from engagement with industry

• ‘Report’ to CEO and management on industry opinion(s) –inform business planning?

SNR 15.3 Resources

Key areas of non-compliance included:

• the language used in the learning materials and online courses was too complex;

• learning materials were out of date;

• inaccurate information concerning state regulations was provided

• RTOs using classroom-based assessment did not have access to a sufficiently robust simulated workplace environment, including resources

(NB. 15.3 intersects with some other standards; for example, if the trainers do not have appropriate qualifications

(Standard 15.4), or assessment is not compliant with requirements (Standard 15.5), then the RTO is

generally also not compliant with Standard 15.3)

Trainer/Assessor Competence

Less than half of all RTOs audited were compliant with this standard

Issues included:

• no procedures for ensuring trainers had vocational competence, or for verifying resume, certificates and qualifications;

• there was little evidence of currency in industry experience;

• there was little evidence of professional development as a trainer or assessor; and

• there was a lack of supervision for trainers without training qualifications

For the strategic audits ‘current’ was interpreted as within the past 5 years

Strategies• Develop a consistent format – demonstrate competence

• Must have either TAE40110 (or demonstrate equivalence to TAE40110 or delivery/assessment skills sets)

On 17 June 2013 the National Skills Standards Council issued the Determination for trainer and assessor competencies to reflect changes made to the TAE10 Training and Education Training Package’s Enterprise Trainer Skill Sets in relation to the minimum qualifications or competencies that must be held by trainers and people delivering nationally recognised training under supervision and assessors in the VET sector (NSSC 2013).

‘Evidence used to demonstrate equivalence of vocational competency may include consideration of relevant past training, including consideration of superseded and pre-existing industry qualifications, experience, and professional development. Equivalency is most commonly demonstrated by an individual through mapping his/her past training, experience and ongoing professional development against the unit of competency. This process may or may not identify gaps that require to be addressed.’

http://www.nssc.natese.gov.au/policies/determination_for_trainer_and_assessor_competencies

Strategies• PLAN PD including both Vocational (industry) and VET

• Approve / plan a range of activities and evidence that demonstrates competence and currency – record and validate (‘map’ to specific units)

• Establish an organisational framework / policy

• Systematically update – include requirement in policy. Part of formal performance planning discussion?

• Circulate opportunities to staff /peers

Understand Why? ‘We are preparing students to be able to perform in the workplace

therefore we need to know about these workplaces (their diversity, complexity, specialisations), how they work, trends and organisational changes in the workplace. We might need to know for example the operational procedures in many different businesses and we need to enrich our experiences by gaining knowledge of current business practices. In this way we can colour our training with interesting up-to-date scenarios and real examples. It adds to our credibility’

‘Employment is not just the doing of the technical job, it is also the social, physical and economic climate of the workplace, which varies considerably and may change over time’

Pre-enrolment informationASQA Strategic audits identified specific issues including:

• were not informed that access to organisational/ supervisory roles was required for demonstration of competency in some units

(‘Some of the units of competency require assessment in a workplace, so this requirement must be clearly stated, including that the RTO has responsibility for providing a suitable workplace’ SNR 15.3)

• were not informed that classroom or online only delivery modes would not provide assessment that meets training package requirements for all units;

• were not advised of requirements such as police check was required before any employment or work placement in aged care is allowed.

‘There is a wide variation in the quality and accuracy of information provided to students before enrolment.’

• No specific or detailed information on student rights and obligations (What are a student’s rights and obligations?)

• Insufficient information about the training and assessment services to be provided. What is sufficient? Attendance requirements ?

• Insufficient information about fees, refund arrangements and related charges

[Licensing and legislative requirements) should be made clear to students prior to enrolment, particularly for online programs, given that some regulators do not accept online training and assessment (eg. for the purposes of issuing a White Card)

Provisions for students with special needs, including language and literacy, should also be explained

BEFORE ENROLMENT(Student Handbook + Program specific information, mechanisms in

enrolment for student acknowledgement)

Fee and Refund information‘The standard relevant to RTO fee practices is the financial management standard (Standard 22) which

details the fee information that must be provided to clients including the amount and type of fees to be paid, the payment terms and the refund policy. This is consistent with the concept of consumer protection and of the obligation of a service provider to disclose relevant information of its service provisions.’

‘Before paying a fee for training and assessment services, consumers should have a clear understanding of what they are buying. Testing of these (online shopping carts) payment gateways found that this information was often not available until the point of payment. It was only at the payment point that the full value of the payment in advance was revealed and often with no or limited refund or other course information being provided’

Standard 22.2 does not provide guidance about the nature of refund policies or arrangements. It simply requires that RTOs provide information to clients about their refund policy. Similarly the Australian Consumer Law outlines the rights and obligations of business and consumers with respect to refunds and the circumstances in which they must be available, but does not prescribe the nature of refund arrangements.

While there is no prescription concerning refund requirements, consumer protection law refers to reasonableness. The review found some refund conditions that on the face of it could be seen as ‘unreasonable’.

Transition ‘Training packages are updated regularly through a continuous improvement process that

ensures that training delivery and assessment meets current industry requirements. The

transition standard (Standard 25/ 1.26 -1.27) requires that RTOs manage the transition of

superseded training packages so that they deliver only currently endorsed training

packages. It is very important that people are offered the latest training package

qualifications, rather than the RTOs continuing to offer out-dated programs long after the

new ones have been developed and released.’

2013 Strategic Review of marketing found 47.0% RTOs to have at least one qualification advertised on

their web site that was superseded.

More information: General Direction on transition and teach out at

http://www.asqa.gov.au/about/general-directions.html

Updated 1 January 2014

‘An RTO must provide timely and adequate advice and guidance to current students if the qualification or course in which they are enrolled is superseded/deleted/expired and ensure students are given the opportunity to transfer to replacement training package qualifications and accredited courses or other currently endorsed training packages or accredited courses.

Managing Transitions

Standards for RTOs 2015

• 12 months from release date of earliest version of the replacement qualification (including apply for addition of replacement qual to scope)

• + (max) 6 month teach out if genuine disadvantage to transition. An RTO does not need to apply to ASQA to teach-out a training package qualification in accordance with the general direction

• Teach out applicable to qualifications only NOT units of competency

• If not complete in 18 12 months, issue SOA. Enrol new qualification or new RTO

• May NOT enrol new students in superseded qualification during 12 month transition BUT NOT in teach out period (pending addition of new qualification to scope)

• Accredited courses expire and do not transition to other accredited courses

• If no replacement qualification, 2 years to finalise

Continuous improvement

Continuous improvement• Systematic (Purposefully regular)

• What (meaningful) data can be collected?

• Evidence of ‘data’ being collected, analysed and IMPROVEMENTS implemented

Evidence the actual improvement(s) (a bundle of survey forms are not evidence of

improvement)

Collect data, analyse / analyse root cause, plan / implement solution - DO!

Continuous Improvement?What strategies do you have for improvement –

• Corrective / preventative actions from previous audits?

• Student / employer/ industry / staff feedback? (think beyond surveys –interviews; walk-arounds; phone calls)

• Analyse complaints - recurrent issues?

• Validation / moderation

• How recorded?

• How do you know /show you have improved?

NB. Not all changes are improvements!

Insanity: doing the same thing over and over again and expecting different results.

(Albert Einstein)

Continuous improvement - Strategies

• What data can be collected?

• Apply a systematic approach to CI

• CI to be a standing agenda on team/ management meetings

• Identify and suggest opportunities for improvement

• Contribute to the maintenance of a CI Register

• Analyse. Conduct root cause analysis. DO! Insanity: doing the same thing over and over again and expecting different

results.

(Albert Einstein)

Record KeepingWhat records?

(+ funding contract requirements?)

• Ensure records can be accessed on request

• Version and other document and record controls

• Assessment decisions are recorded appropriately

(sufficiently detailed)

• Participation records (AVETMISS) unit/ educational

content specific

• Retention of assessment materials (General

Direction)

• Electronic data aligns with hard copy records -

controlled

• Establish arrangements for security, disposal,

transfer, retention.

•11% RTOs non compliant with SNR 19 (Interactions with the VET regulator)

•37% RTOs non compliant with SNR 17

Questions

The quality stamp