Enhanced Challenge Pathway Project Completion Report ECP Project... · Enhanced Challenge Pathway...
Transcript of Enhanced Challenge Pathway Project Completion Report ECP Project... · Enhanced Challenge Pathway...
Enhanced Challenge Pathway Project Final Report
March 2015
2
Table of Contents` 1 Introduction .......................................................................................................................................................... 4 2 Project History ...................................................................................................................................................... 6
2.1 Phase 1 ............................................................................................................................................................ 6 2.2 Phase 2 ............................................................................................................................................................ 7
3 Project Methodology ............................................................................................................................................ 8 4 Industry Consultation Update ............................................................................................................................... 9
4.1 Background – Initial Industry Consultation (2011) ............................................................................................ 9 4.2 Enhanced Challenge Pathway Industry Consultation (2013) ............................................................................ 9 4.3 Conclusion ...................................................................................................................................................... 10 4.4 Purpose of 2014 Industry Consultation ........................................................................................................... 11 4.5 Interview Process ........................................................................................................................................... 11 4.6 Interview Topics and Questions ..................................................................................................................... 11 4.7 Interview Participants ..................................................................................................................................... 12 4.8 Carpenter Group Interview ............................................................................................................................. 12 4.9 Construction Electrician Group Interview ........................................................................................................ 13 4.10 Metal Fabricator Group Interview ................................................................................................................... 14 4.11 Millwright Group Interview .............................................................................................................................. 14 4.12 Heavy Duty Equipment Technician Group Interview ...................................................................................... 15 4.13 Summary – Industry Interviews ...................................................................................................................... 15
5 Summary of Registration Practices of Select Non-trades Occupations ............................................................. 16 5.1 Comparison .................................................................................................................................................... 16 5.2 Support Practices for Internationally Trained Professionals (ITPs)................................................................. 17
6 Summary of Practical Assessment Practices for Canadian Apprenticeship Authorities ..................................... 20 7 Practical Assessment Pilots ............................................................................................................................... 22
7.1 ITA Registered Assessor and Assessor Training ........................................................................................... 24 7.2 Enhanced Challenge Operating Models ......................................................................................................... 25
7.2.1 Centralized Model (ITA as Assessment Agency) ....................................................................................... 25 7.2.2 Distributed Model (ITA as Assessment Agency) ........................................................................................ 26 7.2.3 Third Party Assessment Agency Model ...................................................................................................... 26
7.3 Heavy Duty Equipment Technician Practical Assessment Pilot ..................................................................... 27 7.3.1 Purpose ...................................................................................................................................................... 27 7.3.2 Assessment Agency Model ........................................................................................................................ 27 7.3.3 Practical Assessment Tools ....................................................................................................................... 27 7.3.4 Venues ....................................................................................................................................................... 27 7.3.5 Assessor Orientation .................................................................................................................................. 28 7.3.6 Pilot Results ............................................................................................................................................... 28 7.3.7 HDET Moderation Session ......................................................................................................................... 28 7.3.8 Findings ...................................................................................................................................................... 29 7.3.9 Conclusions: ............................................................................................................................................... 29
7.4 Metal Fabricator Practical Assessment Pilot .................................................................................................. 30 7.4.1 Purpose ...................................................................................................................................................... 30 7.4.2 Assessment Agency Model ........................................................................................................................ 30 7.4.3 Assessment Lab, Tools and Equipment, Consumables ............................................................................. 30 7.4.4 Assessment Tools ...................................................................................................................................... 31 7.4.5 Assessor Orientation Session .................................................................................................................... 31 7.4.6 Candidate Assessment Session ................................................................................................................. 32 7.4.7 Moderation Session .................................................................................................................................... 33 7.4.8 Assessment Venue Debrief ........................................................................................................................ 34 7.4.9 Venue ......................................................................................................................................................... 34 7.4.10 Costs .......................................................................................................................................................... 35
Enhanced Challenge Pathway Project Final Report
March 2015
3
7.4.11 Findings ...................................................................................................................................................... 35 7.4.12 Conclusions: ............................................................................................................................................... 36
7.5 Millwright Practical Assessment Pilot ............................................................................................................. 37 7.5.1 Purpose ...................................................................................................................................................... 37 7.5.2 Assessment Agency Model ........................................................................................................................ 37 7.5.3 Venue ......................................................................................................................................................... 37 7.5.4 Practical Assessment Tools ....................................................................................................................... 37 7.5.5 Candidate Assessment Session ................................................................................................................. 38 7.5.6 Findings ...................................................................................................................................................... 38 7.5.7 Conclusions: ............................................................................................................................................... 40
8 Summary of Findings – Enhanced Challenge Pathway Completion Project ...................................................... 41 8.1 Practical Assessment Pilots – Candidate Results .......................................................................................... 42
9 What Worked? What Did Not Work? .................................................................................................................. 44 9.1 What Worked? ................................................................................................................................................ 44 9.2 What Did Not Work? ....................................................................................................................................... 45
10 Conclusion ......................................................................................................................................................... 46 10.1 National FQR Initiative .................................................................................................................................... 46 10.2 National Harmonization Initiative .................................................................................................................... 48 10.3 Findings for National Initiatives ....................................................................................................................... 49 10.3.1 Challenge requirements ............................................................................................................................. 49 10.3.2 Challenge processes, supporting tools and forms ..................................................................................... 49 10.3.3 Costs ......................................................................................................................................................... 50 10.3.4 Implementation .......................................................................................................................................... 50 10.3.5 Candidates................................................................................................................................................. 50 10.3.6 Assessors .................................................................................................................................................. 50 10.3.7 Enhanced Challenge Pathway System ...................................................................................................... 51
Appendix A. Project Methodology ...................................................................................................................................... 54 Appendix B. Industry Interview Questionnaire ................................................................................................................... 55 Appendix C. List of Interviewees ....................................................................................................................................... 57 Appendix D. Carpenter Group Interview ............................................................................................................................ 58 Appendix E. Construction Electrician Group Interview ...................................................................................................... 60 Appendix F. Metal Fabricator Group Interview .................................................................................................................. 62 Appendix G. Millwright Group Interview ............................................................................................................................. 64 Appendix H. Heavy Duty Equipment Technician Group Interview ..................................................................................... 66 Appendix I. Industry Interview Summary ........................................................................................................................... 68 Appendix J. Summary of Registration Practices of Select Non-trades Occupations ......................................................... 70 Appendix K. ITA Registered Assessor Training ................................................................................................................. 73
Enhanced Challenge Pathway Project Final Report
March 2015
4
1 Introduction
British Columbia is on the cusp of a development boom, driven by growth in liquefied natural gas and other key sectors.
This has the potential to create generations of employment for British Columbians, especially in the skilled trades sector.
However, a shortage of skilled tradespeople threatens to undermine the province’s ability to capitalize on these
opportunities.
As the provincial body with authority to issue trade credentials, the Industry Training Authority (ITA) is in a unique position to
influence the supply of skilled workers that British Columbia needs. The ITA's mandate is to both encourage participation in
formal apprenticeship training and to recognize those individuals with existing skills and knowledge that were acquired
outside of a formal Canadian apprenticeship.
ITA oversees two pathways to certification: training (apprenticeship) for new workers and assessment for experienced
workers (challengers). ITA is looking at ways to ensure that experienced workers – especially those who have the skills for
the identified growth sectors – are able to achieve credit for knowledge and skills they already possess; and where required
are appropriately trained and credentialed such that they can enter the workforce as quickly as possible at the appropriate
skill level. ITA is directly positioned to have the potential for significant impact in the area of Foreign Qualification
Recognition (FQR) and assessment in the trades.
In 2008 ITA embarked on a process to establish the best ways to achieve these ends. An in-depth Foreign Qualifications
Recognition (FQR) Review of the pros and cons of various systems and methodologies from around the world was
undertaken including desk research and face to face discussions exploring best practices in the study of New Zealand,
Australia and South Africa practices. Through this process, ITA identified successful practices other jurisdictions had used to
address the issues raised. These were learned from the implementation of systems and methodologies that provided a
variety of ways to assess the competencies of challengers. This enabled ITA to formulate the guiding principles for model
development and testing in BC.
ITA decided on an enhanced assessment approach (The Enhanced Challenge Pathway) for challengers. This approach
uses a broader range of assessment tools to judge the competency of challengers. Challengers are assessed on skills
previously obtained through formal and informal learning and work experience against a set of Occupational Performance
Standards.
The goals of the enhanced challenge pathway are:
• To strengthen assessment and recognition of individuals’ current skills and experience to enable successful continuation in the certification process.
• To address barriers to certification experienced by British Columbians trained within or outside of the Canadian apprenticeship system (with work experience from within or outside Canada)
• To ensure that the challenge pathway is robust, credible, and responsive to the needs of workers and industry to increase employer confidence in those who earn trade certification through the challenge pathway.
Enhanced Challenge Pathway Project Final Report
March 2015
5
The FQR Review completed in August 2012 found that moving away from credential and academic assessment towards
recognition of competencies is an emerging best practice within Foreign Qualifications Recognition. Beginning in 2008, the
ITA researched and piloted new forms of enhanced assessment through its "Multiple Assessment Pathways" (MAP)
initiative.
This project has built on the expertise and lessons learned from the MAP project to develop five occupation specific
competency assessments. The five trades were selected by the Ministry of Jobs, Tourism and Skills Training because they
were high volume, high-demand, and integrated into the Red Seal interprovincial standard across Canada.
The purpose of this project was for the ITA to develop and pilot competency-based assessments for internationally trained
tradespeople in five trades. The trades selected were identified as high priority occupations through employer consultation
done by the Ministry of Jobs, Tourism and Skills Training (JTST), and the FQR Review. The project was also intended for
forming a foundation from which pre-arrival assessments of skills and competencies would be built.
This report summaries the work undertaken on the project and outlines findings that can be used to inform a certification
challenge process to increase employer confidence in workers certified through challenge and apprenticeship.
The report starts with a brief history of the project and summary of work completed. It continues with a summary of
outcomes from three research activities: follow-up, more granular interviews with industry, an environmental scan of the
challenge process for selected non-trades occupations, and a survey conducted by HRSDC of practical assessment
practices in other Canadian jurisdictions. From there it provides an overview of the practical assessments piloted and
findings from each, and concludes with some next steps.
This report represents fully the challenges and successes of this project and provides evidence to direct further investment
and decisions.
Enhanced Challenge Pathway Project Final Report
March 2015
6
2 Project History
In January 2013, ITA undertook a contract with JTST to develop and pilot five competency based assessments for
internationally trained trades’ people, for the purpose of providing a more accurate assessment of skill levels. This was
intended to provide:
• A more granular understanding of skills for preparation for the Red Seal exam • The development of specific modular gap training to address skills shortages • A process that informs employers about the skill levels and employment potential of internationally trained
trades’ people prior to Red Seal certification • Better preparation for the Red Seal exam with greater potential for success
Contract deliverables were:
1. Five competency assessments for the following trades: Carpenter, Construction Electrician, Heavy Duty Equipment Technician, Millwright and Metal Fabricator. Deliverable to Ministry was a detailed framework, operational procedures/policies and methodology for each competency assessment.
2. Advisory Committees for each trade that included stakeholders, subject matter experts and representatives from the appropriate Industry Training Organizations (ITOs).
3. Assessor training program and materials to deliver the competency assessments and sustainability plan for training assessors on an on-going basis.
4. Training materials for assessors to ensure consistent and equitable delivery of the competency assessments. 5. Competency assessment process that described the steps involved in the assessments and what the applicants
could expect to gain from completing an assessment. 6. Competency assessment piloted for each trade at least once by the end of the contract term.
The project underwent several disruptions that affected the project schedule and deliverables and ultimately led the project
to be defined in terms of two phases.
2.1 Phase 1
Phase 1 of this project began in January 2013. In this phase ITA contracted the services of three ITOs1 to develop the
practical assessment tools for the 5 trades. Each ITO received $100,000 per trade to develop these tools over a period of
several months. All deliverables were due by March 31, 2014; however, not all ITOs met this deadline. The last set of
partial deliverables was received on June 11, 2014 from CITO for Construction Electrician.
To ensure that industry had opportunity to provide input throughout the project, Advisory Groups comprised of industry,
training providers and subject matter experts were formed for each trade. These Advisory Groups were scheduled to meet
4 times throughout the life of the project. The objective of those meetings was to:
• Learn about the background of Enhanced Challenge Pathway options for those not trained in the Canadian apprenticeship system
• Learn about the successful Challenge Pathway practices nationally and internationally in like jurisdictions
1 ITOs include: Transportation Career Development Association of BC (TransCDA): Heavy Duty Equipment Technician; BC Construction Industry Training Organization (CITO): Construction Electrician, Carpenter; Resource Training Organization (RTO): Millwright, Metal Fabricator.
Enhanced Challenge Pathway Project Final Report
March 2015
7
• Provide feedback and guidance on the consultation process to ensure a broad spectrum of the industry training system are engaged in the process
• Involve constituent stakeholder group in the consultation process • Provide leadership and strategic guidance to ITA • Build relationships and liaise with community stakeholders as required • Recognize the interests and needs of those seeking entry into industry via the Challenge Pathway process • Review and comment on the summary of the consultation to ensure the input and feedback heard was captured
appropriately • Provide input into the development of scenarios for future Challenge Pathway options and strategy development
as requested.
Due to staff changes at ITA in November 2013, it became clear the original deadline would not be met and an extension
was granted till December 2014.
The ITOs developed the practical assessment tools with the assistance of Subject Matter Experts (SMEs) from each of the
trades. These SMEs were instructors and industry resources with considerable knowledge and experience in the trade.
In April 2014 the BC government made public the ITA mandate review report as part of a larger initiative--B.C.’s Skills for
Jobs Blueprint. Among other recommendations, the Government recommended ITA no longer contract program
development work to ITOs. This precipitated a pause to the project while ITA developed a transition plan to implement the
changes. ITOs discontinued work on the project.
The assessment tools for one trade—Heavy Duty Equipment Technician—were ready for pilot testing and ITA worked with
the ITO to conduct this from May-July 2014. Given partially completed deliverables and discontinued ITO involvement, ITA
requested the project be put on hold while a new completion strategy was developed. This was the end of phase 1.
2.2 Phase 2
In September 2014, ITA and JTST jointly decided to adjust the remaining project deliverables and timeline. The remaining
deliverables would be completed by March 2015:
• Pilot test practical assessments for two more trades—Millwright and Metal Fabricator • Pilot test two different delivery models for practical assessments • Conduct a follow-up survey with industry to determine continued support of the model, now that more details
were understand • Submit a final report with findings on the use of practical assessments as a mandatory component of the
challenge application process
ITA’s new Manager of Assessment became the Project Manager and a consultant was hired to support the project. As part
of Phase 2, ITA also decided to include:
• An environmental scan of the challenge application, verification and certification process for three non-trades-related occupations
• An overview of practical assessments conducted by Apprenticeship Authorities in other Canadian jurisdictions
Enhanced Challenge Pathway Project Final Report
March 2015
8
3 Project Methodology
The project methodology was built on four lines of inquiry:
1. Comparative analysis – Review of Phase I ECP activities, processes and outcomes. 2. Environmental scan – Review of application registration practices. 3. Qualitative research – Industry consultation and key informant interviews. 4. Pilot studies – Practical assessment pilots.
The project benefited from a full review and comparative analysis of Phase I ECP results (1 above). The review provided a
comprehensive analysis of the project results by trade. It was an important step in developing a good understanding of
trade-specific challenge approaches, noting lessons learned and identifying necessary process improvements.
The scan (2 above) provided detailed information on registration/licensure requirements, including specific processes for
internationally trained applicants, from general supports and information provided to applicants to particular policies and
practices. Analyzing the scan brought up several concepts and practices for engaging and supporting internationally trained
professionals that could be adapted and used for the ITA’s trades challenge processes.
The project benefited from rich information collected though industry interviews (3 above). Interviewees provided
suggestions on potential improvements in the following areas: experience validation, practical assessment delivery options,
employer and trades training provider roles in practical assessment.
Two pilot projects – Industrial Mechanic and Metal Fabricator practical assessments – advanced the study further by testing
tools, protocols and other enhanced challenge components in real life situations.
The combination of desktop and primary research provided well-rounded information to inform the Project.
The Table 3.1 ECP Project Methodology (Appendix A) describes project approaches in greater details.
Enhanced Challenge Pathway Project Final Report
March 2015
9
4 Industry Consultation Update
4.1 Background – Initial Industry Consultation (2011)
In 2011, ITA conducted an extensive industry consultation on Enhanced Assessment methods comprising of surveys,
webinars, and face-to-face meetings in Kamloops, Vancouver and Victoria locations.
85 participants attended the face-to-face meetings from across the stakeholder communities, with more than a dozen trades
represented, and excellent representation from unions, educators, associations, industry training organizations and some
employers.
Table 4.1 – Feedback on the Current Challenge Process (2011)
Positive Issues
� Perspective of quick, inexpensive, simple to use, understood by employers and potential challengers
� Lack of testing of the full scope of the trade � Lack of verification of practical experience � Exam questions can be compromised � No approval or sponsor required � Written exam poses barriers to some � Difficult to verify quality of time in trade.
Table 4.2 – Feedback on the Enhanced Assessment Approach (2011)
Positive Issues
� Four assessment tools represent a significant departure from the multiple choice test in the current challenger process – and people are keen to learn more about how it might work.
� Nearly unanimous agreement that practical assessments have great potential as a measure of a candidate’s competency and verification of submitted work experience
� Enhanced assessment options were widely perceived to be a significant improvement over the current challenger process – much more likely to ensure that successful candidates have earned their certification;
� Tools such as the portfolio, competency interview and practical assessments introduce a measure of subjectivity that is not present in the multiple choice exam and a clear and transparent process of assessor accountability and management needs to be in place
� Challengers will need guidance in the preparation of portfolios
� There is going to be an on-going educational challenge, and engagement challenge to insure a new approach is well designed and understood
� Introduction of a new enhanced assessment approach for challengers will require adjustments; this is not going to be a one-size-fits-all proposition
4.2 Enhanced Challenge Pathway Industry Consultation (2013)
The Enhanced Challenge Pathway project team created Industry Advisory Groups for each of the five trades to provide the
industry with an opportunity to share their experiences, challenges and opportunities in the trades. The Industry Advisory
Groups engaged the services of Subject Matter Experts from Trade Schools and the industry to develop the practical
assessment tools.
These Advisory Groups were scheduled to meet 4 times throughout the life of the project. The objectives of the Advisory
Enhanced Challenge Pathway Project Final Report
March 2015
10
Group meetings were to:
� Learn about the background of Enhanced Challenge Pathway options for those not trained in the Canadian apprenticeship system.
� Learn about the successful Challenge Pathway practices nationally and internationally in like jurisdictions. � Provide feedback and guidance on the consultation process to ensure a broad spectrum of the industry training
system is engaged in the process. � Involve constituent stakeholder group in the consultation process. � Provide leadership and strategic guidance to ITA. � Build relationships and liaise with community stakeholders as required. � Recognize the interests and needs of those seeking entry into industry via the Challenge Pathway process. � Review and comment on the summary of the consultation to ensure the input and feedback heard was captured
appropriately. � Provide input into the development of scenarios for future Challenge Pathway options and strategy development
as requested.
The Advisory Group members were engaged in the consultation process and demonstrated high interest in developing and
implementing enhanced challenge pathway methods. In spite of this, the consultation was only partially completed (one or
two meetings per Advisory Group –TBC). The main reasons for the incomplete consultation include ambiguous project
directions, project interruptions, staff and organizational changes.
4.3 Conclusion
The 2011 consultation examined the views of the consultation participants on the Enhanced Challenge approach used in
the Multiple Assessment Pathways (MAP) projects involving Heavy Duty Technicians and Cooks. A MAP trade challenge
process used a variety of methods (portfolio of evidence, interview, written assessment, practical assessment) to determine
a candidate’s competency against industry performance standards. The Candidate is deemed competent if he/she can
demonstrate evidence of competency through the combination of the assessment methods (blended mark).
The 2013-14 Enhanced Challenge Pathway (ECP) project is built on the notion of introducing additional assessment
methods to the existing interprovincial multiple-choice exam. Achieving a 70% mark on the IP exam is a mandatory
requirement. Additional evaluation methods such as practical assessment would be used to provide supplementary
evidence of the Candidate’s practical skills and overall competency in a trade. The Candidate is evaluated on his or her
ability to perform practical assessment tasks against established performance standards. It is a mandatory requirement to
meet these standards during the assessment.
The major difference between the MAP and ECP pilots is the certification achievement criteria. For the MAP project, it is a
cumulative mark from multiple assessment methods. For the ECP project, the criteria require two independent, not
combined, results from the written IP exam and practical assessment against established standards.
It is important to consider these differences while conducting the analysis of the results of the initial consultation and current
industry interview.
Enhanced Challenge Pathway Project Final Report
March 2015
11
4.4 Purpose of 2014 Industry Consultation
The purpose of the Industry Consultation Update (2014) was to consult with industry on the importance of the practical
assessment component in the certification challenge process and obtain recommendations on making the challenge
pathway robust and credible, and responsive to the needs of workers and industry.
The objectives of the consultation were to:
1. Collect industry opinions on current challenge requirements and challenge process; 2. Identify ways to strengthen current challenge process; 3. Identify ways to increase employers’ confidence in the skills of workers who achieved trade certification through
challenge rather than formal apprenticeship; 4. Determine the value of trade certification in selecting and hiring skilled trades workers, including internationally
trained workers.
The consultation was focused on five trades: Carpenter, Construction Electrician, Heavy Duty Equipment Technician,
Industrial Mechanic, and Metal Fabricator.
4.5 Interview Process
A three-step process was used in the organization and completion of the industry interview.
1. Interview preparation, including: • Identifying and approving potential industry contacts for each trade with the Industry Relations Manager and
Project Authority • Determining and approving interview questions
2. Conduct interviews:
• Minimum five interviews per trade
• Collect feedback from a range of employers including large companies and small- and medium-sized enterprises, subject matter experts working in the industry, union representatives and members of industry associations.
3. Summarize and analyze findings
4.6 Interview Topics and Questions
The interview focused on determining whether the current challenge process reflects industry needs and on strengthening
the challenge process to increase employer confidence in Candidates who acquired certification by challenge. The
interview was also a way to discuss the perceived overall value of trade certification and the employers’ confidence in the
skills of certified employees who went through the challenge process compared to those who completed a Canadian
apprenticeship to become certified.
The following four topics were selected for examination:
1. Challenge requirements.
Enhanced Challenge Pathway Project Final Report
March 2015
12
2. Enhanced challenge process – practical assessment, interview, others – and its impact on employers’ confidence in the skills of challenge Candidates;
3. Enhanced challenge costs; 4. Importance of certification for workers and for making hiring decisions.
A total of 12 questions were developed for the interview. Most of the questions were open-ended or combination questions.
These question types allowed for an open discussion and exploration of solutions. Other than the first question that
addressed trade-specific challenge requirements, the questions were identical. A copy of the questionnaire is enclosed in
Appendix B.
4.7 Interview Participants
The interviewees included trade-certified subject matter experts, HR managers, operations managers, supervisors, and
union and industry group representatives. The list of interviewees by trade is available in the Appendix C.
The interviewees were selected from the current ITA sponsors, industry representatives from Industry Task Forces and
workforce committees, unions and employer associations. A consultant created an initial list that was later updated by the
ITA Industry Relations Managers. The final list of potential participants was reviewed and confirmed by the Project Authority.
An e-mail invitation to participate in the interview was sent out to potential participants who had been determined and
confirmed earlier. The e-mail response was low; direct follow up phone calls helped achieve the desired number of interview
participants for each trade group.
Although the timelines for organizing and conducting interviews were constricted, the project exceeded the minimum
number of 25 interviews, achieving 33. The interviews took place over the period of November 14 and December 5, 2014.
4.8 Carpenter Group Interview
The group consisted of five respondents.
The interview revealed the following:
1. There is low confidence in the effectiveness of the current Employer/Experience Declaration Form. Specifically employers are not confident in whether the declared hours cover the full scope of the trade.
2. There is an agreement that experience declaration and verification processes need to be strengthened. 3. Suggestions on how to strengthen these processes include:
a. Improved employer declaration form – include more details on skills, duties, within the scope the trade b. Practical assessment – will allow demonstration and observation of skills c. Verification interview conducted by a subject matter expert.
4. Respondents support having a practical assessment for challengers. However, there are concerns related to assessment design and implementation, specifically:
a. Carpentry trade is not mandatory b. Scope of practical assessment in relation to core competencies c. Assessment administration d. Industry support and employer buy-ins
5. There is strong support for on-site practical assessments conducted by the employer – Employer Sign-Off.
Enhanced Challenge Pathway Project Final Report
March 2015
13
6. If the employer does not conduct the practical assessment, the majority of the respondents agree on an 8-hour maximum assessment. This would cover core competencies and be sufficient for getting evidence of the candidate’s core practical skills.
7. Proposed practical assessment fees are in the range of $300-$500 excluding challenge application fees (currently $120).
8. Although the respondents strongly support having a technical interview by a subject matter expert, they do not think it should substitute the practical assessment
A detailed interview summary is available in Appendix D.
4.9 Construction Electrician Group Interview
Eight industry representatives participated in the interview (two companies provided two representatives each).
The key findings for the Construction Electrician group include:
1. Four out of seven respondents think that the current challenge process should be strengthened. 2. Other three respondents are satisfied with the current challenge requirements; they think that additional
requirements may create barriers to certification and hiring. 3. All agree that there is a need to strengthen candidate screening and experience verification processes. 4. Suggestions for strengthening the overall challenge process include:
a. Improved validation of claimed experience (detailed employer/experience declaration and verification interview conducted by subject matter expert)
b. Mandatory Canadian Electrical Code Course c. Completion of a college-based practicum (4-8 weeks).
5. Two respondents suggested that the challenge requirements for internationally trained professionals should be different. Suggestions include:
a. “Preferred countries” list; b. Provisional certification.
6. Practical assessment as a mandatory part of the challenge process is viewed positively but there are concerns related to assessment design and delivery format.
7. Those who support the practical assessment have more confidence in a college-based practicum. 8. If the practical is not college-based, then the suggested duration is 6-8 hours. 9. The range of the assessment fees is broad – from $300 to $1000 – depending on the assessment design and
delivery format. 10. There is agreement that a technical interview is valuable, but that it should not be a substitute for a practical
assessment. 11. There is an observation that those who deal with selection and hiring are more likely to see practical assessment
as a barrier; the majority of operational representatives do not see it as an obstacle.
Additional comments and suggestions include:
1. Develop strategies for attracting Electricians from the U.S. to meet growing demand for trained and certified electrical workers.
2. Consider a provisional certification for electrical workers. 3. Add college practicum to the certification challenge process as a substitute for the practical assessment. This
would better align apprenticeship and challenge processes than a practical test for challengers. It would also increase employer confidence in Challengers’ skills and knowledge. A practical assessment delivered through a college would be more standardized and may have more value.
A detailed interview summary is available in Appendix E.
Enhanced Challenge Pathway Project Final Report
March 2015
14
4.10 Metal Fabricator Group Interview
Seven industry representatives participated in the interview.
The key findings for the Metal Fabricator group include the following:
1. The majority of respondents (5 out of 7) have low confidence in the results of the current experience declaration and verification processes.
2. There is an agreement that experience verification interviews (with both employer and challenger) and practical assessment should strengthen the current challenge processes.
3. Practical assessment is viewed as a stronger assessment method than an interview. 4. It was mentioned that blueprint reading and interpretation is one of the core skills and it must be incorporated in the
challenge process. 5. Proposed practical assessment duration varies from a 4-8 hour assessment to a 4-6 week college-based
practicum. 6. Proposed fees vary from $300 to $2000; the latter fee is related to a college-based practicum. 7. Respondents believe that the practical assessment business model needs to be built on an effective cost recovery
rather than profit generation basis. 8. Respondents agree that challenge requirements need to be the same for different candidates including those who
gained experience in, across and outside Canada. 9. Employers hire workers with and without certification. For unionized companies, a trade certificate is a requirement
for employment as a journeyperson.
A detailed interview summary is available in Appendix F.
4.11 Millwright Group Interview
Seven industry representatives participated in the Millwright group interview.
The key findings for this group include:
1. All respondents want to see a stronger challenge application process that will include an enhanced Employer and Experience declaration form.
2. Respondents want to ensure that the experience candidates submit is within the scope of the trade. 3. There is an agreement that employers should be accountable for reporting on the working hours of their workers. 4. There is support for making practical assessment a part of the challenge process; it is believed that the practical
assessment will strengthen not only the challenge process and its outcomes but also the initial candidate screening process by attracting the “right” candidate.
5. When asked to define the “right” candidate, respondents said – “someone who has direct industry experience within the scope of the trade, understands the trade, and is able to demonstrate practical skills”.
6. The suggested duration for a practical varies from 4 - 16 hours to a 40-hour college-based practicum. 7. Millwright trade is highly diversified so practical assessment has to cover core competency areas. 8. Assessment fees vary from $500 to $3000 depending on the type of practical assessment. The majority of
responses about assessment fees fall in the $500-$1000 range. 9. Most (6 out of 7) respondents do not hire workers without trades certification for Millwright positions. 10. Employers want to see up to 95-100% of their Millwright workers certified. They also value direct industry
experience; experience has high importance in employee selection and hiring process.
A detailed interview summary is available in Appendix G.
Enhanced Challenge Pathway Project Final Report
March 2015
15
4.12 Heavy Duty Equipment Technician Group Interview
Six industry representatives participated in this interview.
The key findings for the HDET group include:
1. The majority (5 out of 6) of respondents think that the current challenge process needs improvement. Specifically, they want to see improved experience declaration and verification process and a possible addition of practical assessment.
2. Challenge requirements need to be the same for all challenge candidates no matter where they acquired their work experience – in or outside Canada.
3. Most respondents (5 out of 6) support adding a practical assessment as a mandatory component of the challenge process.
4. The majority (4) of respondents suggest a 4-8 hour assessment that covers core skills. 5. Fees vary from $250 to $1000. Assessment fee is viewed as a barrier for some candidates but not a concern for
the industry; industry wants to attract and support the “right” candidate. 6. There is support for having an interview conducted by a subject mater expert as an additional assessment method
but not as a substitute for the practical assessment. 7. The majority of the interviewees (4 out of 6) think that employers have more confidence in the skills of candidates
who achieved certification through formal apprenticeship rather than through the challenge process.
A detailed interview summary is available in Appendix H.
4.13 Summary – Industry Interviews
In summary, seven major themes emerged in the industry interview:
1. Lack of satisfaction with the current challenge process. 2. Lack of satisfaction with experience declaration and experience verification processes. 3. Practical assessment is a preferred method for measuring Candidates practical skills. 4. Different level of support for the practical assessment as a mandatory component of the challenge process across
five trades (non-regulated trades – lower support). 5. Concerns about practical assessment design, development, delivery, costs, long-term maintenance. 6. Interest in a mandatory college-based practicum for challenge candidates. 7. Concerns from some industry representatives in Human Resources roles who see a mandatory practical
assessment as a barrier to certification and, consequently, hiring.
The findings from industry interviews with 5 trades groups (Carpenter, Construction Electrician, HDET, Metal Fabricator,
and Millwright) are provided in the Table 4.3 (Appendix I).
Enhanced Challenge Pathway Project Final Report
March 2015
16
5 Summary of Registration Practices of Select Non-trades Occupations
The review of practices for registration of Engineers, Technologists and Registered Nurses included an examination of
current certification and licensing processes used by regulatory bodies to assess applicants. The review focused on
identifying current processes and practices for internationally trained candidates and evaluating whether these practices can
be adapted for assessing applications and trades equivalency of internationally trained tradespersons.
In addition, it was expected that the review would help develop a model that could streamline the application, verification
and assessment processes for internationally trained tradespersons including those who reside in and outside of Canada.
Further examination of the resources and their application is recommended to evaluate how this approach can be
implemented for internationally trained tradespersons.
A detailed summary of registration practices for three non-trades occupations (Engineers, Technologists and Nurses) is
provided in the Appendix J.
5.1 Comparison
The table below provides a summary of current registration/licensure requirements, categories and processes for three
organizations representing Engineers, Technologists and Registered Nurses.
Table 5.1 – Registration/Licensure Requirements (APEG BC, ASTT BC, CRN BC)
Organization
Candidate Support
APEG BC ASTT BC CRN BC
Pre-registration support • Yes/APEGBC Online Self Assessment
• Yes/ Technology registration Canada (TRC) Online Self Assessment;
• ASTTBC Online Self Assessment;
• ITTP web portal
• Yes/ CRN BC website; • National Nursing
Assessment Service (NNAS) – FQR support across Canada
Application
requirements
• Engineering Degree from recognized university or college;
• 4 years of engineering experience with references;
• Canadian environment experience;
• English language competency
• Diploma of Technology (2-3 years);
• Min 2 years of experience;
• Main requirements: Graduation from a Nursing program;
• Different requirements for 1
st time applicants and
renewal applicants
Enhanced Challenge Pathway Project Final Report
March 2015
17
Organization
Candidate Support
APEG BC ASTT BC CRN BC
Registration
categories
• 14 different categories, e.g.
• Professional Engineer, Engineer-in-Training, Limited Licensee (Eng.L.) Professional Geoscientist, Geoscientist-in-Training, Non-Resident Licensee (NRL) P.Eng., International Mobility P.Eng., other.
• Applied Science Technologist (AScT); Certified Technician;
• Provisional certification for internationally trained (have to complete 1-year work experience requirement within 5-year period after registration)
• Associate member; Other.
• Registered Nurse (RN); • Nurse Practitioner (NP); • Nurse Practitioner
Grandparented (GNP); • Licensed Graduate Nurse
(LGN); • Other
Assessment and
registration
process/requirements
• Application approval; • Completion of the
Law & Ethics Seminar;
• Completion of the Professional Practice Examination (PPE);
• Application approval; • Completion of Professional
Practice and Ethics Examination;
• 1 year of experience in Canada for internationally trained professionals with Provisional Certification
• Application approval; • Written examination; • Objective structured
clinical examination (OSCE);
• Experience/special conditions for members with Provisional certificate/license
Costs • Application fee - $472;
• Academic Examination(s) - $338 per examination;
• Professional Practice - $240 (in Canada)-$340 (outside of Canada);
• Law& Ethics seminar - $400.
• Application fee - $250 • Annual dues – Various
depending on type of certification
• ($365- AScT, $260 – Graduate Technologist)
• Registration fees – Registered Nurse -$450;
• Nurse practitioner - $650; • RN exam-$615;
NP written exam - $700; • NP clinical exam - $2,000 • Other fees – renewal,
reinstatement
5.2 Support Practices for Internationally Trained Professionals (ITPs)
The scan has identified current support practices that help internationally trained professionals (ITPs) prepare for
registration/licensure process in B.C. It is clear that both the industry and the government make significant investments into
the attraction of skilled workers. Both direct their efforts to reducing barriers to achieving professional certification, and to
facilitating successful skilled immigrant labour market integration.
Enhanced Challenge Pathway Project Final Report
March 2015
18
A brief description of these support practices is provided in the table below.
Table 5.2 – Support practices for Internationally Trained Professionals (APEGBC, ASTTBC, CRNBC)
APEG BC
ASTT BC CRN BC
Support Process • APEG BC web info and links to ITPs
• Online self assessment tool
• ASTTBC web info and links to ITPs
• Online Self assessment tools (ASTTBC and Technology Registration Canada- TRC)
• Internationally Trained Technology Professionals web portal (www.itppbc.com)
• CRN BC info and links to ITPs
• National Nursing Assessment Service
• Provisional Certification
Purpose • Information for applicants; • Help Candidates determine
their skill level and decide on certification type;
• Targets internationally trained Candidates; Canadian Candidates can use the tool as well.
• Information for applicants;
• Help Candidates determine their skill level and decide on certification type;
• Targets internationally trained Candidates; Canadian Candidates can use it too.
• Information for l applicants;
• Provide harmonized approach towards assessment of internationally trained nurses across Canada.
• Reduce or eliminate employment barriers associated with a lack of Canadian work experience.
Outcome • Starting point • Self-assessment results
(printout form) as a recommendation to apply to APEG BC or ASTT BC;
• The self-assessment is not an official assessment of qualification for registration or license in B.C
• Starting point • Self-assessment
results (printout form) as a recommendation to apply to APEG BC or ASTT BC.
• Self-assessment is not an official assessment of qualification for registration or license in B.C.
• Starting point • Information and support
systems (bilingual) • Certification as
employment requirement;
• Faster employment.
In summary, these are four models that the regulatory bodies use to guide internationally trained applicants through the
registration and licensure processes:
1. Online self-assessment tools (Technology Registration Canada (TRC), APEGBC, ASTT BC). 2. Internationally Trained Technology Professionals web portal (www.itppbc.com). 3. Provisional certification (technologists and nurses). 4. National Nursing Assessment Service.
Enhanced Challenge Pathway Project Final Report
March 2015
19
The first two models could be adapted to develop online self-assessment tools for foreign trained tradespersons. Such tools
would help foreign trained workers determine their skill set and decide on challenge level. The third model – provisional
certification – could be adapted to help reduce barriers to employment for internationally trained trades persons. The forth
model, NNAS, focuses on aligning national requirements.
Further review is required to determine how these practices fit current ITA priorities.
Enhanced Challenge Pathway Project Final Report
March 2015
20
6 Summary of Practical Assessment Practices for Canadian Apprenticeship Authorities
In October and November 2014, a jurisdictional survey was conducted on the use and structure of practical tests in
Canadian provinces and territories. 10 provinces and 3 territories submitted responses.
Key findings for this survey include:
1. There are no specific or consistent criteria to determine which trade has a practical assessment. a. In Ontario, practical exams were developed as an alternative to the written exam to provide an avenue for
individuals who have been unsuccessful in passing the written exam; b. In Alberta, the provincial or local apprenticeship committee makes a recommendation for a practical and
the Program Development Officer (PDO) and the Provincial Apprenticeship Committee (PAC) develop the assessment together;
c. In Saskatchewan, the criteria is typically established when a trade is being designated; d. In Manitoba, there are no criteria.
2. Three (BC, ON, NL) apprenticeship authorities have been approached to develop a practical assessment for the following trades: Hairstylist, Construction Electrician, Plumber, Automotive Technician, and Heavy Duty Equipment Technician, Machinist (for apprentices in NL).
3. In BC, apprenticeship authority receives regular requests for trade challengers, but employers are discouraged due to cost and complexity to manage. In NB newly designated trades have the option of requiring a practical assessment for a period of time to “grandfather” experienced candidates
4. In most cases, practical assessments are delivered by training providers, apprenticeship authorities, and the department of labour. In BC, practical assessments are also delivered by Third Party Assessment Agencies under contract with ITA.
5. The level of support for challenge Candidates or apprentices to prepare for a practical assessment varies across the provinces; some offer basic online information and notification letters, others offer no support.
6. The duration of a practical assessment across different trades is 1.5-8 hours. 7. All jurisdictions require assessors to be certified in their trade with a minimum amount of experience (varies by
trade and jurisdiction). Ratio of assessors to candidates varies by trade and jurisdiction and range from 1:1 (ON) to 1:6 (BC).
8. There is no consistency in Assessor training. a. In British Columbia, there is an assessor training program comprised of three (3) days of generic training
and one (1) day of trade-specific training; b. In Saskatchewan, markers may also have a “trial marking session” and in some cases, large groups of
markers will have an annual marking seminar; c. In Manitoba, most assessors are college instructors and do regular apprenticeship training. Other
examiners may train examiners, plus they have the occasional professional development session; d. In New Brunswick, new assessors work with a seasoned assessor; e. CWB and Department of Labour train their own assessors.
9. Practical assessment fees vary and are trade specific. At the moment, the lowest fee is $200 (Carpenter/NT), the highest is $750 (Cranes/BC).
10. Practical assessment cost is one of the biggest concerns for all apprenticeship authorities. 11. The biggest challenges apprenticeship authorities face are:
a. Costs associated with the development of practical assessment tools b. Costs associated with assessment administration and management; c. Low number of challenge candidates for some trades; d. No standard practical assessment practices; e. Lack of suitable testing locations; f. Lack of qualified assessors; g. Colleges’ availability to hold practical assessments;
Enhanced Challenge Pathway Project Final Report
March 2015
21
h. Candidates’ readiness to take practical assessment; i. Internal resources and staff to develop and coordinate practical assessment; j. Maintenance of assessment tools.
12. Practical assessment benefits: a. Increased employer confidence; b. Opportunity for Candidates to demonstrate competency in the trade versus ability to write an exam; c. Results help to confirm the written assessment results; d. Industry engages with and supports the process; e. Verification of practical skills – encourages buy-in from industry
Enhanced Challenge Pathway Project Final Report
March 2015
22
7 Practical Assessment Pilots
Three practical assessment pilots were conducted over the period April 2014 to February 2015. Heavy Duty Equipment
Technician pilot was completed over the period April 2014 to June 2014. Practical assessment pilots for the Metal
Fabricator and Industrial Mechanic trades were part of the ECP Phase Two and were completed over the period from April
2014 to January 2015.
The purpose of the pilots was twofold – to test the validity and usability of tools and to determine the effectiveness of the
enhanced challenge models and their compatibility with existing organizational functions.
In order to recruit Candidates to participate, ITA offered a financial incentive: $200 Canadian Tire gift certificate. The profile
of the Candidates varied across the pilots and trades. Because these were only pilots, the Candidates were informed that
their results would not count toward their certification.
The table below provides a brief summary of the pilots’ characteristics, tools under revision and validation, and operational
models under review.
Table 7.1 – Summary: Enhanced Challenge Pathway Practical Assessment Pilots Trade
Criteria
HDET Metal Fabricator Millwright
No. of Candidates
Assessed/Passed
12/10 2/2 2/1
No. of Assessor
Trainees
12 2 2
Assessor to Candidate
Ratio
1:1 1:1 1:1
Assessor Rate
$55/hour $75/hour $500/day
Estimated Cost
(practical only)
$1,500 $1,450 $900
Operational Model
(pilot only)
Decentralized/College Centralized/College Third Party
Enhanced Challenge Pathway Project Final Report
March 2015
23
Trade
Criteria
HDET Metal Fabricator Millwright
Tools Tested • Self-assessment; Checklist tested – found not useful;
• Interview tested – found subjective, requires changes;
• Practical Assessment Tool tested;
• Assessment Venue criteria confirmed;
• Assessor Qualifications confirmed.
• Practical Assessment Tool reviewed and undergone significant changes; Revised version tested;
• Validation with broader industry is required; Assessment Venue criteria confirmed;
• Assessor Qualifications confirmed.
• Practical Assessment Tool reviewed and undergone significant changes.
• Revised version tested; • Candidate Handout
developed and tested; • Validation with broader
industry is required; • Assessment Venue
criteria revised; • Assessor Qualifications
confirmed. • Assessor Toolkit
developed NEW* Level of Efforts Significant Significant Moderate
Successes • Approved practical assessment tool;
• Assessor contracts; • Decentralized model
tested; • Suggestions on
improving tools and processes;
• Candidates were engaged.
• Improved and simplified practical assessment tool;
• Centralized model tested;
• Assessor contracts and scope of work;
• Candidates were engaged.
• Finalized Candidate handout, including work order and drawing;
• Improved and simplified practical assessment tool;
• Developed Assessor Toolkit - flexible delivery;
• Third party model tested; • Assessment time
reduced; • Candidates were
engaged and felt comfortable in worksite setting;
• Employer partner provided support and endorsement;
• Venue costs; • Per diem Assessor rate.
Issues • Additional costs due to travel (location of venues and potential assessors);
• Pilot documentation not completed or completed poorly;
• Standard of performance was missing.
• Small sample size (2 Candidates);
• Lack of time to test other assessment tools;
• Assessor rate ($75/hr) is high; Total cost of $1450 might be a barrier;
• Pilot documentation not completed due to ongoing revisions of the tool.
• Small sample size (2 Candidates);
• Lack of time to test other assessment tools;
• Pilot delay due to unforeseen events;
• Pilot documentation not completed due to ongoing revisions of the tool
Enhanced Challenge Pathway Project Final Report
March 2015
24
In summary, the pilots revealed the following:
1. Conducting small-scale pilots was a cost effective way to test the tools, various operating models, make necessary revisions and adjustments.
2. Practical assessment tools were designed to test both assessment process and outcomes. This format is more suitable for practical assessments in a college lab environment.
3. The pilots determined that the assessment needed to focus more on the performance outcomes. Focusing on performance outcomes allowed for tools to be simplified and be more applicable across various assessment settings (trade training lab, worksite).
4. Practical assessment tools (for three trades) have undergone reviews and revisions to streamline the assessment process.
5. Candidates from all three trades confirmed that practical tasks have a sufficient level of complexity and reflect basic core skills that a qualified tradesperson must possess.
6. Assessors demonstrated relevant expertise for conducting practical assessments and managing required protocols.
7. ITA Registered Assessor training is required to ensure that Assessors have a strong understanding of assessment rules and protocols (including safety, privacy, duty to accommodate).
8. The Third Party model proved to be the most flexible and cost effective. 9. Candidates requested detailed information on the practical assessment (online practical assessment guide,
instructions or checklist) that would help them prepare accordingly. 10. The tools that have undergone changes as result of the pilots require review and validation by Advisory Group
members and /or broader industry. 11. Practical assessment tools (for all three trades) require final amendments prior to implementation. 12. Enhanced challenge forms (e.g. self-assessment checklist, experience declaration form) require final amendments
prior to implementation.
The detailed descriptions of the ITA Assessor training, operating models and three pilots are provided in the following
sections.
7.1 ITA Registered Assessor and Assessor Training
Assessor Training
ITA’s Registered Assessor Course is designed around 5 areas of Assessor performance, specifically:
1. Prepare for Assessment. 2. Demonstrate Effective Communication and Interpersonal Skills. 3. Conduct Assessments and Make Assessment Decisions. 4. Record and Report Assessment Decisions and Recommendations. 5. Apply Continuous Improvement to the Assessment Process.
The course consists of a pre-course study, 4 days of classroom and shop instruction and activities, and an assessment
practicum (2 supervised assessments).
By the end of the course participants will be able to:
1. Explain the certification system and roles of ITA, Assessors, Assessment Agencies, and certification candidates 2. Interpret the competency standard for a specific trade or occupational qualification. 3. Define the qualities and characteristics of a competent assessor. 4. Explain the code of conduct, privacy, and conflict of interest requirements. 5. Identify the parts and steps of the assessment process. 6. Prepare candidates for assessment.
Enhanced Challenge Pathway Project Final Report
March 2015
25
7. Conduct assessments and make decisions in accordance with the principles of assessment and rules of evidence. 8. Complete necessary paperwork. 9. Provide candidates with helpful and objective feedback. 10. Describe service standards and ITA-approved operational procedures.
Upon successful completion of required assessor training, ITA will issue an Individual Identification Number to an Assessor.
ITA will maintain a registry of Assessors in ITA Direct Access.
The detailed information of the ITA Registered Assessor training is available in the Appendix K.
7.2 Enhanced Challenge Operating Models
Enhanced Challenge Operating models determine how the new assessment functions will operate within the established
ITA structure and processes.
It was important to test the models and identify the best match with existing organizational processes and functions. It was
anticipated that the successful model would support, strengthen and accelerate enhanced challenge and be flexible as
economic events and industry demands change over time.
The project included the testing and analysis of three Enhanced Challenge operating models:
1. Distributed Model a. ITA as assessment agency b. Different training institutions as venue
2. Centralized Model a. ITA as assessment agency b. One training institution as venue
3. Third Party Model a. Third party assessors approved by ITA b. Work site as venue
7.2.1 Centralized Model (ITA as Assessment Agency)
In the Centralized Model the ITA serves as the assessment agency. The ITA identifies an independent assessment venue
(e.g. public or private school with trades training lab) that has demonstrated the willingness and capacity to deliver
specialized practical assessments for a trade in a safe manner.
The ITA oversees the following activities:
� Identify and confirm appropriate assessment venue for a trade; � Develop and sign assessment venue agreement (includes scope of activities and expected outcomes); � Identify, train and register Assessors as per ITA policy framework; � Organize and conduct Assessor training; � Develop and manage Assessor contracts; � Manage Candidate application and other required documentation; � Develop and coordinate assessment schedule with the venue. � Manage Assessor performance
Enhanced Challenge Pathway Project Final Report
March 2015
26
� Maintain an Assessor registry � Manage Assessor Moderation Session and Professional Development Activities
The Assessment Venue is responsible for the following:
� Assessment lab; � Assessment materials, tools and equipment; � Access to the facility; � Site security; � First Aid and emergency contacts; � Scheduling required shop, classroom or other space.
* Note: other activities may be included as per specific trades’ needs
7.2.2 Distributed Model (ITA as Assessment Agency)
The Distributed Model has the same characteristics as the Centralized Model except that different training institutions
assume the role of the assessment venue for a given trade. All training institutions/venues involved in the Enhanced
Challenge activities must meet established Assessment Venue Criteria.
7.2.3 Third Party Assessment Agency Model
In the Third Party Assessment Agency Model the ITA outsources the enhanced challenge operation through a bidding
process.
The ITA oversees the following activities:
� Identify successful bidder though an RFP process; � Develop and sign Assessment Agency Agreement; � Assign a staff person to oversee the Assessment Agency Agreement and reporting.
The Assessment Agency provides the following services:
� Work in collaboration with ITA staff responsible for the Assessment Agency agreement; � Identify and confirm assessment venue; � Manage and coordinate ITA-registered Assessors; � Coordinate assessment applications; � Develop and coordinate assessment schedules; � Monitor and record assessments as per established ITA standards; � Submit assessment records and recommendations to the ITA; � Establish safety, First Aid, emergency response procedures in collaboration with assessment site(s); � Confirm all applicable insurances (for Candidates, Assessors and other participating parties).
* Note: other activities may be included as per specific trades’ needs
Enhanced Challenge Pathway Project Final Report
March 2015
27
7.3 Heavy Duty Equipment Technician Practical Assessment Pilot
7.3.1 Purpose
The purpose of the HDET Practical Assessment pilots was to:
� Test the tools specific to the practical assessment � Identify opportunities to improve the practical assessment process � Confirm that the Assessors know and understand their roles and responsibilities � Identify which Venue(s) would be most appropriate to launch the practical assessments � Identify real costs associated with planning, organizing and implementing the practical assessments for the trade � Provide a learning experience for ITA participants who had never been involved in practical assessments.
It should be noted that only practical assessments, and no other sub-processes or tools related to the overall practical
assessment process (such as candidate scope of work verification interviews) were piloted. The project team was directed
to “test” them during the launch.
7.3.2 Assessment Agency Model
This pilot used the Distributed Model.
7.3.3 Practical Assessment Tools
The practical assessment pilot for Heavy Duty Equipment Technicians was conducted during Phase 1 of the project.
The following practical assessment tools were used during the HDET practical assessment pilots:
� Assessor’s Practical Skills Assessment Task Summary � Assessor’s Information Sheet � Candidate’s Work Order � Marking Sheet � Confidentiality Agreement
7.3.4 Venues
Lessons learned regarding venue requirements and issues is summarized in the list below:
� Not all Venues provide First Aid Contacts. Must ensure that 1 person participating during the practical assessment has First Aid Level One training.
� Need to ensure that Assessors have a designated area to conduct initial instructions for the practical assessment participants.
� All participants, not just the Candidates, must participate in Safety Orientation. � Despite the fact that discussions took place with Venues and agreements were put in place for the practical
assessments, it became evident that ITA must be very specific in terms of its expectations as to the services provided by the Venues.
� Because the Venues did not provide the Assessors to conduct the pilots and the Candidates did not attend the trade schools, ITA had to purchase liability and errors and omissions insurance at a cost of $11,000 for all the pilots for all 5 trades. Should the ECP project go live, ITA could negotiate with the trade schools, a fee that would cover the insurance for both the Assessors and the Candidates.
Enhanced Challenge Pathway Project Final Report
March 2015
28
7.3.5 Assessor Orientation
Although all the HDET Assessors participated in the Assessor training/tool development workshop, not all Assessors
demonstrated the skills required to become a Registered Assessor and will require additional training and mentored practice
to administer practical assessments. All will require training and practice using any other assessment tools such as work
verification interviews if ITA adopts these for use in the future. Interview training and successful completion of a mentored
practicum is standard procedure for new assessors.
The cost of HDET Assessor “Training”, or rather Assessor tool development & trial, was approximately $17,600 for 11
Assessors or $1,600 per person. This amount includes a daily rate of $175 and their travel and accommodation expenses.
This training did not include training on conducting or evaluating technical interviews or completing assessment
documentation.
7.3.6 Pilot Results
The BCIT pilot was the first pilot conducted and was the most valuable pilot in terms of learning experience. Six ITA staff
participated in that pilot throughout the weekend; some participated over the 2 day period while other attended only 1 day.
Because these were only pilots, the Candidates were informed that their results would not count toward their certification.
The profile of the Candidates varied across the HDET pilots.
The table below provides more information on the HDET pilots.
Table 7.1 – HDET Pilots: dates, locations, and participants
Location BC Institute of
Technology
Vancouver
Thompson
Rivers
University
Kamloops
Okanagan
College
Kelowna
Northern Lights
College
Fort St. John
Vancouver
Island
University
Nanaimo
Dates May 3-4, 2014 May 31, 2014 June 1, 2014 June 7-8, 2014 June 14, 2014
# of Assessors 5 2 2 3 Pilot Cancelled –
No Candidates # of Candidates 4 2 2 4
Two of the 12 HDET Candidates did not pass the practical assessments. The first Candidate who did not pass was a Truck
and Transport Mechanic and was not familiar with hydraulics. The second Candidate who did not pass had language
challenges as English was not his first language. Both of these Candidates were approved to write the HDET IP Exam but
have not yet written it.
7.3.7 HDET Moderation Session
The HDET Moderation was conducted over two days on July 5 and 6, 2014 at the ITA Richmond Office. The objectives
were to:
� Review assessments performed and discuss each as a case study to evaluate for validity, consistency and fairness
Enhanced Challenge Pathway Project Final Report
March 2015
29
� Discuss challenger performance and assessor marking to reach consensus on consistency in marking to ensure inter-rater reliability of judgments
� Incorporate any required changes into tasks, procedures, and supporting documents.
Eight assessors participated: 4 from the Lower Mainland (2 Instructors and 2 from Industry), 1 from Nanaimo (Associate
Dean), 1 from Kamloops (Industry), 1 from Kelowna (Instructor – now retired), and 1 from Dawson Creek (Instructor).
Seven ITA employees attended: 2 Program Development Officers (PDO) (1 facilitating), Manager of Program Standards,
Manager of Assessment, Manager of Policy & Research, ECP Project Lead, Independent Consultant.
The planned interview role-play on the agenda was scrapped when it was determined that the interview script was not
designed to adequately evaluate scope of work.
7.3.8 Findings
The following are key findings during pilot activities:
Assessors
� The importance of having the Assessors review their pilot documentation and training material prior to the practical assessments was verified.
� The Assessors would like a more robust marking guide with the measures of success for performance of practical tasks. Further investigation is required to determine what this requirement might be.
� There is a need for Assessors to confirm that the Candidate has heard and understood the instructions correctly. � The Assessors would like practical assessment guidelines including:
- Briefing the Candidates on practical assessment tasks and expected performance; - Candidates’ questions and requests for assistance;
Assessment Tools:
� Required simplification/clarification/addition of wording
� Added boxes, columns, headers, partial reformatting
� Confidentiality agreement could be incorporated into the Assessor’s Practical Assessment Task Summary.
Shop Tools and Equipment:
� Tool list for Candidates required improvements. � The need was identified to have the Venue prepare a toolbox containing all the tools for the practical assessment. � Include additional tools required for the practical assessment and tools that are not required for the practical
assessment and incorporate tool selection into the marking guide. � Have the Venue provide the manuals for the practical assessments in both hard copy and electronically and have
them readily available for the practical assessment.
7.3.9 Conclusions:
� A decentralized practical assessment model with multiple colleges across the province requires significant time and resources to coordinate and manage.
� An increase in coordination time increases the total assessment costs, and, therefore a Candidate’s practical assessment fees.
� It is recommended to validate tested tools, forms and operational model with broader industry representation prior to launch.
� The model demonstrates low to moderate level of readiness for full implementation.
Enhanced Challenge Pathway Project Final Report
March 2015
30
7.4 Metal Fabricator Practical Assessment Pilot
7.4.1 Purpose
The goal of the Metal Fabricator Practical Assessment pilot was to:
• Test and analyze the assessment venue model; • Test practical assessment tools; • Conduct a two-day Assessor orientation session including two assessment simulations where trainees acted as
candidates and evaluated one another’s performance; • Conduct two practical assessments on Candidates; • Confirm Assessors’ readiness to conduct assessments to ITA standards; • Record all findings and suggest changes to prepare the tools for full implementation; • Gather feedback from the Assessors and the Candidates.
7.4.2 Assessment Agency Model
The Metal Fabricator practical assessment pilot was built on the Centralized Assessment Venue model.
Below is the description of how the model worked and the roles and responsibilities of participating parties.
ITA:
1. Selected the venue, developed and signed the assessment venue agreement. The agreement outlined all required conditions, including insurance, and deliverables.
2. Identified and selected Metal Fabricator Assessor trainees. Assessor trainee selection was based on the Assessor qualifications established by the industry and approved by the ITA.
3. Determined the Assessors’ pay rate based on industry consultation. A $75/hour rate was established for the Metal Fabricator Assessors.
4. Hired Assessors on contractual terms to conduct assessment and complete all relevant activities. 5. Identified and confirmed pilot candidates and coordinated all assessment preparation and logistics. 6. Prepared and conducted the Assessor, Candidate, Moderation and Venue Debrief sessions.
This model requires the direct involvement of ITA staff and/or consultants in the assessment preparation and coordination.
Piping Industry College of BC (PIC BC):
1. Prepared assessment lab and classroom, including required tools and equipment set up. 2. Ordered and delivered consumables according to the assessment requirements. 3. Arranged on-site First Aid. 4. Assigned First Aid and emergency contact for the time of the assessment. 5. Provided access to the facilities during the assessment. 6. Attended debrief meeting and provided feedback and findings.
7.4.3 Assessment Lab, Tools and Equipment, Consumables
The Assessment lab was set-up as per practical task requirements. Lab set-up and teardown requires between four and six
Enhanced Challenge Pathway Project Final Report
March 2015
31
hours to set up for two candidates. Having a dedicated shop would be more effective for future operations. However, if the
number of the Challengers is low, the shop cannot be dedicated to challenge assessments due to insufficient use of
valuable space and equipment.
The tools and equipment for mandatory and elective tasks were provided by the venue and met assessment requirements.
Consumables included steel beam, plates, and cardboard/paper, cost approximately $175 per Challenger including delivery
cost. Consumables need to be ordered and delivered to the venue one or two days prior to the assessment.
The venue used during the pilot project has a strong internal security system with restricted access to labs and classrooms.
Should the same venue be used in future assessments, UAPIC BC will need to determine a more efficient way to provide
access to Assessors, Candidates and ITA support staff to the operative classroom and labs.
7.4.4 Assessment Tools
The following practical assessment forms were used during the Metal Fabricator practical assessment pilot:
1. Assessor’s Practical Assessment Task Summary: 2. Confidentiality Agreement and Code of Conduct 3. Assessor Information Sheet 4. Marking Guide 5. Candidates’ Work Order. 6. Technical Drawings.
7.4.5 Assessor Orientation Session
The assessor orientation session was conducted on November 17 - 18, 2014 and included an overview of project
background, discussion of project goals, introduction to assessment rules, preliminary review of tasks and forms, shop tour
and safety talk, and practical assessment role-play. The ITA staff observed the practical assessment role-play and an
independent consultant and the findings on process improvements were recorded.
Session attendees included two Assessor trainees, ITA Manager of Assessment, ITA Manager of Policy and Research, ITA
Assessment Coordinator, and an independent consultant.
The two Assessor trainees who participated in the orientation session trialed all assessment tasks – one mandatory in the
shop and three electives in the classroom – and demonstrated strong knowledge of the tools and good understanding of the
protocols. Based on the trials they confirmed that there should be sufficient time for challenge candidates to complete the
tasks, and made the following suggestions for revisions to the tools to clarify and streamline the assessment/evaluation
process:
1. Include the following sections in the Practical Assessment Tool: a. Confidentiality agreement b. Candidate orientation checklist c. Assessment task description d. Mandatory and elective task combination options e. Performance criteria for each task f. Assessment summary sheet
Enhanced Challenge Pathway Project Final Report
March 2015
32
g. Decision whether or not candidate performance met the approved standard h. Suggestions to ITA based on outcomes
2. Revise the mandatory task performance criteria
a. Remove most of the operational process criteria as per findings b. Incorporate safety criteria c. Establish ‘Set up an oxy-acetylene unit’ as a ‘pass/fail’ criterion d. Establish ‘Beam measurements’ as a pass/fail criterion
3. Revise the elective tasks performance criteria
a. Remove the operational process criteria b. Include measurements established during the pilot c. Include photographs of process and product at specified junctures
4. Redesign the Practical Assessment Tool:
a. Use A4 format b. Consider the development of an electronic PDF version for record keeping and reporting efficiency c. Include ITA assessment staff contact information
The Assessor orientation session, as mentioned earlier, included a brief overview of the assessment rules and
requirements. It is recommended to conduct additional Assessor training on the ITA assessment administration and
reporting procedures to ensure that assessment activities are aligned with current ITA policies. If interviews are incorporated
into enhanced challenge protocols in future, training will be provided for conducting interviews and scoring responses.
7.4.6 Candidate Assessment Session
The goals of this session were to:
1. Assess two Candidates using the revised tools. 2. Confirm appropriate time allowed to complete mandatory and elective tasks. 3. Confirm appropriate time allowed for full assessment, including lunch and breaks. 4. Verify if tasks’ complexity reflects requirements (i.e. includes key competency areas) for evaluating candidate
performance against approved performance standards. 5. Monitor assessment processes and record recommended changes. 6. Gather Candidates’ feedback and suggestions.
The Candidate assessment session took place on November 20, 2014. The attendees included two Candidates, two
Assessor trainees, ITA Manager of Assessment, ITA Manager of Policy and Research, ITA Assessment Coordinator, and
an independent consultant.
Of the two Candidates: one holds a Metal Fabricator Certificate of Qualification and is interested in writing the Red Seal
examination, the other wrote but did not pass the IP/Red Seal exam.
Both Candidates completed mandatory and elective tasks within the allowed time. One Candidate completed the mandatory
task in two hours, the other in two and a half hours.
Candidates worked on different elective tasks. One Candidate completed the task within 1 hour and 30 minutes, the other
completed the within 1 hour and 45 minutes. Time allowed was two hours.
Enhanced Challenge Pathway Project Final Report
March 2015
33
The Assessor trainees assessed Candidates’ performance using the revised assessment criteria developed during the
Assessor preparation/training session. The Assessor trainees recorded assessment results in the Assessor’s Practical Task
Summary form; they also took photos of the Candidates’ work to support assessment records. Both Candidates met the
criteria for passing the practical assessment.
The session was completed on time; the estimated time for the practical assessment – seven hours – was accurate.
Candidates participated in the assessment follow-up discussion and provided their feedback and comments on the
assessment process and tools. The conversation included the following questions:
1. How do you feel about the practical assessment component in the challenge process? Both candidates were positive about the practical assessment as part of the challenge process
• “I feel like a real metal fabricator” • “This is what we are asked to do in the workplace”
2. What do you think about the assessment venue (location, lab set up, classroom set up)? • No problem finding the venue; • No problem with parking; • Lab and classroom were good and served the purpose of the assessment.
3. Did you find the tasks too hard/ or too easy?
• The tasks were just right; • Mandatory task is designed to use core fabricating skills; • Elective tasks require refreshment and that is good.
4. What is your overall impression about the assessment? • It was interesting to participate in the process.
5. Other suggestions included: • It would be helpful to have assessment information such as task description, time and fees online; • Including FAQ section would be helpful; • Reference to learning materials will help prepare for the assessment.
7.4.7 Moderation Session
The moderation session took place on November 27, 2014 from 9 a.m. until 1 p.m. at UAPIC BC in Delta, B.C. The
attendees included two Assessors, the ITA Manager of Assessment, ITA Manager of Policy and Research, ITA Assessment
Coordinator and an independent consultant.
The purpose of the moderation session was to:
1. Briefly describe the moderation and its role in the enhanced challenge process. 2. Review and discuss pilot results. 3. Review and validate recommended changes to the tools and forms. 4. Gather feedback and findings on assessment implementation.
One of the Assessors and the consultant met prior to the session to review changes to the practical tasks and revise the
assessment tool accordingly. These changes were incorporated into the Assessor Practical Task Summary form and
presented to all session attendees for review and further comments.
Enhanced Challenge Pathway Project Final Report
March 2015
34
It is important to remember that the moderation session focused on the practical assessment only and did not include a
review of other enhanced challenge tools and forms (e.g. self-assessment, employer declaration).
The moderation session resulted in the following:
1. Confirmed standard of performance for mandatory and elective tasks. 2. Confirmed evaluation criteria for mandatory and elective tasks. 3. Noted suggestions to make changes to the Assessor Practical Task Summary form as per 1 and 2 (detailed
described in the Assessor session section). 4. Established the time to complete full assessment as seven (7) hours:
- Four (4) hours for mandatory task - Two (2) hours for each elective task - 30 minutes for lunch - 30 minutes for breaks - Total time: seven (7) hours;
5. Suggestion to increase the Assessor to Candidate ratio to 1:2 if possible. Further evaluation of this proposition is required;
6. Assessor trainees completed the Assessor Practical Task Summary form but did not complete the tasks evaluation summary and recommendation to certification sections. These sections require further changes. Note, if the IP is still a certification requirement, the Assessors will not be performing a recommendation for certification based on practical assessment results unless the IP has already been written and passed by the candidate.
7. Both Candidates passed the practical assessment as per new criteria. 8. The final version of the tool is to be developed and circulated to the Assessors and other SMEs who participated in
the development process for review and validation.
7.4.8 Assessment Venue Debrief
The venue debrief session took place on December 4, 2014 from 10 a.m. until 12 p.m.
The purpose of the debrief was to:
• Review and evaluate the pilot under the Assessment Agency model; • Gather venue’s (PIC BC) feedback on assessment set up and operations; • Discuss and evaluate cost efficiency; • Discuss venue’s interest in conducting future practical assessments; • Discuss the approximate number of assessments per year and confirm venue’s capacity; • Determine a necessary minimum number of assessments per year; • Discuss the Assessor and the Assessment Venue roles and responsibilities.
7.4.9 Venue
1. The venue met the criteria for conducting the practical assessment. 2. The venue prepared the assessment lab, classroom, tools, equipment and materials as per assessment agency
agreement. There is no dedicated shop for conducting Metal Fabricator practical assessments at this time. Establishing a permanent or dedicated lab could be considered if assessment demand/volume warrants.
3. One of the Assessors was an UAPIC BC instructor. In the future, school instructors might not be involved in the assessment process since under the Assessment Venue model the ITA selects and hires Assessors. There are some conflict of interest restrictions on which candidates instructors can assess. A number of instructors are ITA-registered assessors for other trades programs.
Enhanced Challenge Pathway Project Final Report
March 2015
35
4. Consumables (steel beam, set of steel plates, card board) were ordered and delivered in advance. Consumables are not reusable and need to be recycled after the assessment. Recycling will require additional time.
5. Shop rental rate includes the costs for required tools and equipment. 6. Should the model be used in the future, further discussion on lab set up, roles, responsibilities and costs will be
required.
7.4.10 Costs
1. Assessor rate: a. $75/hour for ten (10) hours = $750 (based on seven (7) hours for the assessment time, and three (3)
hours for preparations and marking). b. The assessor wage rate for the pilot was $75/hour. It became evident that the $75/hour rate and 1:1
Assessor to Candidate ratio results in high assessment costs and associated fees. To address high assessment costs it was recommended to change the Assessor to Candidate ratio from 1:1 to 1:2.
2. Shop rent - $400/day. 3. Classroom rent - $200/day. 4. Consumables - $150/person. 5. Delivery of materials (five (5) sets) - $125. 6. Shop set-up - $250. 7. Shop teardown - $150.
a. As mentioned earlier, a shop could be dedicated if a sufficient number of assessments is scheduled on a monthly basis. If the assessment numbers are low, the shop set up and tear down time will be included in the assessment costs. This will increase assessment fees.
b. Currently, approximately 20 – 30 Candidates apply to challenge the Metal Fabricator certification per year (TBC with ITA). This would not justify a dedicated space.
8. Total estimated cost per Candidate is $1,450 based on 1:1 Assessor to Candidate ratio used during the pilot. 9. The number of challenge Candidates per month is important; it will impact the development of the ITA enhanced
challenge operational and business models.
7.4.11 Findings
The table below summarizes key variables tested in the Metal Fabricator pilot, as well as findings and suggestions. The
suggestions are based on pilot findings and feedback from Candidates, Assessors, and the Venue, collected during the
candidate follow up, moderation and assessment venue debrief sessions respectively.
Table 7.2 – Metal Fabricator Pilot Summary
Criteria Findings Suggestions
Number of
Assessors
2 N/A
Assessor to
Candidate Ratio
• 1:1 ratio did not prove cost effective; The ratio can be reduced to improve Assessor time.
• Change from 1:1 to 1:2; This change will help use Assessor time more effectively and reduce assessment costs and Candidate fees.
Assessor Rate • $75/hour used in the pilot • Review Assessor rate and discuss possible hourly rate reduction or establish per diem rate in the range of $450-$500/day
Enhanced Challenge Pathway Project Final Report
March 2015
36
Criteria Findings Suggestions
Estimated Cost • Cost ($1,450 per Candidate) • is high and may create barriers
to certification.
• Change Assessor to Candidate ratio from 1:1 to 1:2; Determine if Assessor rate can be reduced;
• Determine if group assessment is possible for elective tasks and how it can help reduce Assessor time.
Operational Model • Centralized. • Moderate effort.
• This model is difficult to implement if the number of challenge Candidates is small;
• Look into other models (Third Party, College Practicum) and evaluate cost effectiveness;
Tools and Forms • Practical assessment tool required significant changes and undergone revisions;
• Other tools and forms were not tested.
• Complete final revisions to the practical assessment tool as per pilot findings and validate with a boarder SMEs/industry group;
• Review and validate other assessment tools and forms as per decision to implement Metal Fabricator challenge pathway.
Level of Effort • Substantial - Required significant coordinating time.
• Review other operating models for cost effectiveness.
Assessors • Demonstrated skills to become ITA Registered Assessors;
• Required more in depth knowledge of ITA protocols.
• Assessors to attend ITA Assessor training to learn assessment recording and reporting protocols.
• Develop Assessor guide that outlines assessment process, Assessor roles, and responsibilities and provides samples of assessment forms to be completed.
Candidates • Were engaged in the process; • Supported the enhanced
challenge process; • Confirmed that practical tasks
cover core skills and competencies;
• Required more information on the practical assessment process.
• Develop online challenge guide, including specific information on practical assessment;
• Develop FAQ, resources that help candidates prepare for the challenge process.
7.4.12 Conclusions:
� A centralized practical assessment model (one college) with requires less time and resources to coordinate and manage than decentralized.
� A centralized model requires high number of challengers to be feasible. � Assessor rate ($75/hour) is high; it increases the total assessment costs, and, therefore a Candidate’s practical
assessment fees. Review and adjustment is required to ensure that fees do not create barrier to certification. � It is recommended to validate tested tools, forms and operational model with broader industry representation prior
to launch. � The model demonstrates moderate level of readiness for full implementation.
Enhanced Challenge Pathway Project Final Report
March 2015
37
7.5 Millwright Practical Assessment Pilot
7.5.1 Purpose
The purpose of the Millwright Practical assessment pilot was to:
1. Complete the development of the Candidate Handout (work order and technical drawing) based on the practical assessment tools developed by the industry.
2. Conduct Assessor orientation session including review and revision of the practical assessment tool. 3. Conduct practical assessment for two Candidates, test and revise practical assessment tools and work order as
per pilot findings. 4. Provide suggestions on effective business model, estimate assessment fees, suggest improvements.
7.5.2 Assessment Agency Model
The Industrial Mechanic practical assessment pilot was built on the “Assessment agency” or Third Party model. This model
is not new to the ITA; it has been effectively used for conducting practical assessments for a number of trades and
occupations including Cooks, Shipyard Laborers, Climbing Arborists, Crane Operators and Parks and Public Works
Operations Workers. Although the model was proven successful for some of these trades and occupations, it needed
further testing to ensure its suitability for conducting practical assessments for the five chosen trades.
Five potential assessment agencies were identified and invited to respond to the request for proposal (RFP) to conduct a
practical assessment pilot for the Industrial Mechanic trade. These agencies were:
1. Kwantlen Polytechnic University. 2. Industrial Marine Training and Applied Research Centre (IMTARC). 3. Christian Labour Association of Canada (CLAC). 4. Fulford Certification. 5. Millwrights Union, Local 2736.
One of the agencies, Fulford Certification (who act as an Assessment Agency for ITA Crane and Funeral Director trades),
provided their response at a later date and was awarded the contract.
7.5.3 Venue
Vancouver Pile Driving Ltd’s welding and fabrication shop in North Vancouver was selected as an assessment venue.
The shop provided flexible working space and access, as well as the required materials and tools.
The venue tour and safety orientation were provided to the assessment team and ITA Assessment staff observers prior to
assessment day.
7.5.4 Practical Assessment Tools
The pilot identified that the original assessment tool was more suited for conducting assessments in a standard Millwright
college lab and required significant resources (tools, equipment, materials, lab space).
The suggested assessment tool did not meet the requirement that all practical assessments need to be universally
applicable. This requirement ensures that assessments can be conducted with the same criteria across various work place
Enhanced Challenge Pathway Project Final Report
March 2015
38
settings and are fair for all candidates regardless of their work location or residence in the province.
As the result of the pilot, the Millwright practical assessment tool has undergone significant revisions and resulted in the
following:
1. Modified practical assessment tool that can be used in an industrial setting across the spectrum of users of the trade.
2. Completed Candidate Handout including work order, drawing and task instructions. 3. Portable assessment kit that allows conducting assessments in workplace settings across the province.
The following modification were completed and tested during the pilot:
1. Use Precision Measuring Tools (Mandatory) – 45 minutes were deemed the maximum allowable time for this task.
2. Mount and Remove Bearings (Mandatory) – 90 minutes were deemed the maximum allowable time for this task.
3. Perform Shaft-to-Shaft Rough Alignment (Mandatory) – 45 minutes were deemed the maximum allowable time for this task.
4. Drill and Tap Holes in Steel Plates (Elective) – 45 minutes were deemed the maximum allowable time for this task.
5. Assemble a Hydraulic Circuit (Elective) – 60 minutes were deemed the maximum allowable time for this task while using computer based hydraulic simulators.
6. Total assessment time for three mandatory and one elective task – 3.5 hours. This is far below the estimated 8 hours.
7.5.5 Candidate Assessment Session
The assessment took place in the Welding Shop of Vancouver Pile Driving on January 24, 2015. The services of two Red
Seal Millwright assessors had been secured – one from the original development team (Al Story) and the other, his
colleague (Chris Mellors).
Two Challengers from the provided list volunteered their time to help pilot the practical assessment. One challenger was a
senior maintenance supervisor at a waste processing industry in Vancouver, while the other was in charge of maintenance
for a Lower Mainland municipality aggregate plant.
The assessment proved to be a viable and effective way of determining workplace performance to a standard.
One candidate met standards as per the pilot’s requirements, while the other candidate did not meet the required standards.
7.5.6 Findings
The pilot resulted in an effective and valid assessment process supported by a tested assessment tool and Candidate
handout. In addition, the pilot demonstrated that the assessment could be conducted anywhere in the province by a skilled
Assessor using a portable assessment kit.
The Third party model proved to be cost effective. The model offers flexible assessment delivery and stronger employer
engagement in the assessment process.
The assessment tasks chosen by the development team and modified by the practical assessment pilot team cover core
areas of the Trade and are simple to coordinate.
Enhanced Challenge Pathway Project Final Report
March 2015
39
Having the practical component in the challenge would ensure that a higher level of industry-defined skill is required before
Certification is offered. Finding a cost effective, simple way to ensure a higher skill level for challengers is a strong outcome
of this pilot.
Table 7.3 – Millwright Pilot Summary
Criteria Findings Suggestions
Number of
Assessors
2 N/A
Assessment time 3.5 hours (below est. 8 hours)
Assessor to
Candidate Ratio
1:1 proved to be effective 2 assessments per day
Assessor Rate $500/day -
Estimated Cost $900 (below est. $2,000) Further discussion on pricing model with ITA.
Operational Model • Third Party • This model is flexible. Determine if assessments can be delivered across the province.
• Identify potential designated employers – “ITA Millwright Assessment Employer”
Tools and Forms • Practical assessment tool was modified for use in an industrial setting.
• Portable Assessor toolkit was assembled for the pilot and tested.
• Other tools and forms were not tested.
• Complete final revisions to the practical assessment tool as per pilot findings and validate with a broader industry group;
• Review and validate other assessment forms as per decision to implement Millwright challenge pathway.
• Confirm portable Assessor toolkit efficiency for the challenge process.
Level of Effort • Moderate
Assessors • Demonstrated skills to become ITA Registered Assessors;
• Required more in depth knowledge of ITA protocols.
• Assessors to attend ITA Assessor training to learn assessment recording and reporting protocols.
• Develop Assessor guide that outlines assessment process, Assessor roles, and responsibilities and provides samples of assessment forms to be completed.
Candidates • Were engaged in the process; • Supported the enhanced
challenge process; • Confirmed that practical tasks
cover core skills and competencies;
• Required more information and support on the challenge process.
• Expressed lack of satisfaction with the current challenge process.
• Develop online challenge guide, including specific information on practical assessment;
• Develop resources that help candidates prepare for the challenge process.
• Review gap analysis feedback for the written and practical components.
Enhanced Challenge Pathway Project Final Report
March 2015
40
7.5.7 Conclusions:
� A Third Party practical assessment model required the lowest time and resources to coordinate and manage than college based models (Decentralized and Centralized).
� Venue rate and Assessor rate per diem are more manageable and cost effective. � It is recommended to validate tested tools, Assessor toolkit and operational model with broader industry
representation prior to launch. � The model demonstrates high level of readiness for full implementation.
Enhanced Challenge Pathway Project Final Report
March 2015
41
8 Summary of Findings – Enhanced Challenge Pathway Completion Project
The findings from the enhanced challenge pathway projects are organized into tables (Table 8.1; Table 8.2; Table 8.3).
Each table provides description of critical findings for each stage of the project.
Table 8.1 - Feedback on the Current Challenge Process
Positive Issues
• Sufficient amount of working hours • Written tests required knowledge
• Lack of testing of the full scope of the trade • Lack of verification of practical experience within the scope of the
trade • Written exam poses barriers to some • Written is not sufficient to evaluate practical skills • Lack of feedback on knowledge/experience gaps
Table 8.2 - Feedback on the Practical Assessment Component in the Enhanced Challenge Process
Positive Issues
• Practical is viewed as strong mechanism to evaluate a candidate’s competency and verify submitted work experience
• Practical increases employers’ confidence in the skills of candidates who achieved certification through challenge rather than apprenticeship
• Practical attracts experienced workers, strengthens the challenge process and improves challenge outcomes
• As an enhanced assessment method, practical is considered stronger than technical interview
• Non-regulated trades and lack of support from employers • All five trades require highly diversified skill set that is difficult to
measure in a 1-day assessment • Practical assessment tasks need to include core trade skills • Practical assessment fees and affordability for unemployed
workers • Education and engagement of apprenticeship stakeholders to
ensure practical assessment is well designed and understood • Challenger education and preparation will require additional
resources.
Table 8.3 – Enhanced Challenge Pathway Operational Models
Criteria Decentralized (HDET) Centralized (MFAB) Third Party (IM)
Time to Coordinate
Candidates
High High Moderate
Time to Coordinate
Assessors
Moderate/
Contract with ITA
Moderate/
Contract with ITA
Low/Contract with Third
Party
Venue Cost Lab rate Lab rate Negotiable/Possible
industry in kind contribution
Assessor Rate Hourly - $55/hr Hourly- $75/hr $500 per diem
Assessment Time 8 7 3.5
Venue College Lab College Lab Worksite(s)/ across B.C.
Assessor/Candidate Ratio 1:1 1:2 recommended 1:1/2 candidates per day
Flexible Delivery No No Yes
Enhanced Challenge Pathway Project Final Report
March 2015
42
Readiness for
Implementation
60%
TBC w/ITA
75%
TBC w/ITA
85%
TBC w/ITA
Based on the above-presented analysis the Millwright pilot provides a basis for launching practical assessment for the
Millwright Trade.
Other two trades tested in the pilots require more time to finalize operating model, costing, number of challenge Candidates
per year, and Assessor readiness and availability.
8.1 Practical Assessment Pilots – Candidate Results
The table below describes the qualifications of the pilot Candidates and summarizes the results they achieved during
practical assessments for their trade.
Table 8.4 – Practical Assessment Pilots: Candidate Results
Trade
School
Location Candidate
Name
Employment
Status
At time of HDET
Pilot
Pilot
Results
Comments
BCIT Vancouver LC Truck & Transport
Mechanic
Approved to write
HDET IP Exam; wrote
2 times and failed.
Did Not
Pass
MS Automotive
Service
Technician at
Airport
Approved to write
HDET IP Exam
Pass
RB Unknown Approved to write
HDET IP Exam; wrote
2 times and failed.
Did Not
Pass
English not
first language
SD Automotive
Service
Technician
Approved to write
HDET IP Exam
Pass
NLC Fort St.
John
MH Industrial
Mechanic
(Millwright)-
deregistered
apprentice;
HDET apprentice
Pass Physical
Disability
MS 4th year HDET
apprentice;
certified at the
time of the Pilot
Pass
KH Automotive
Service
Applied to challenge
in May 2014
Pass
Enhanced Challenge Pathway Project Final Report
March 2015
43
Technician
DL HDET apprentice Pass
Okanagan
College
Kelowna AB HDET apprentice
- (deregistered as
an HDET
apprentice)
Approved to write
HDET IP Exam
Pass
KM HDET apprentice Pass
TRU Kamloops CR Truck & Transport
Mechanic -
Certified
Pass
RC Outdoor Power
Equipment
Technician
Pass
PIC BC Vancouver EQ Unknown Approved to write
Metal Fabricator IP
Exam; written and
failed the exam with
58%
Pass English not
first language
PD UAPICBC/
Shop staff
Completed
apprenticeship in
19??(TBC)
Not a challenger
Pass
Van Pile North
Vancouver
EK Journeyman
Machinist. Senior
maintenance role.
Certified Machinist.
Approved to write
Millwright IP Exam;
wrote on July 10,
2014 (53%)
Did not
pass
KW Senior
maintenance
supervisor.
Approved to write
Millwright IP Exam;
wrote and failed on
July 23, 2014 (64%)
Pass
16
Candidates
13 pass/
3 didn’t
pass
Enhanced Challenge Pathway Project Final Report
March 2015
44
9 What Worked? What Did Not Work?
9.1 What Worked?
� Phase II Project Team, comprised of ITA staff and an independent consultant, had good team dynamics, energy, commitment, experience and knowledge.
� Engaging an independent consultant to assist with project management and delivery allowed for more flexibility and efficient use of internal and external resources.
� Small-scale pilots were effective in testing assessment tools and making necessary adjustments. The revised tools will help streamline the assessment process, decrease assessment time and costs.
� The pilots were effective in testing three operational models. � Piloting two trades, not five, at the same time worked well and provided more time to complete required changes
and adjustments to the tools. � Call for Proposals process was useful in identifying organizations that currently have the capacity to deliver
practical assessments as per ITA established standards. � Assessor contracts (used for Decentralized and Centralized models) were useful and helped specify the scope of
work for each pilot. � Focusing Assessor recruitment for the Millwright and Metal Fabricator pilots on both industry and post-secondary
representatives worked well. Millwright Assessors from the industry contributed to the development of the portable Assessor Toolkit.
� Industry interviews, conducted over the phone, had a positive impact on industry engagement and participation. � Industry interviews provided valuable information and feedback on the current challenge process and ways to
improve it. � Industry interviews helped determine specific demands and preferences for practical assessments. � Industry interviews helped prove the Enhanced Challenge Pathway concept. � The Centralized model proved practical and efficient; it needs costs adjustment to make it more cost effective. � The Third Party model proved to be practical and the most cost effective. � Phase II project plan was clearly defined and all project goals and objectives were achieved as planned. � Assessor qualifications that were developed and approved by the industry worked well and helped select qualified
Assessor Trainees who were instrumental during the pilot trial and validation. � The Millwright team felt that having Assessor Candidates from the industry was particularly useful for adapting
practical assessment tools to a shop environment. � Assessor peer assessments were useful in making continuous revisions and modifications to the tools during the
trial process. � A follow up review of the modified tools with Assessors and project consultants who are familiar with the
assessment tools development process helped make final changes to the tools (Metal Fabricator and Millwright pilots).
� The Assessor Marking sheet worked well. However, the format (size, boxes) of the document was not easy to use during practical assessment; marking sheet requires redesign and improvement to reflect pilot modifications to the tools.
� Industry questionnaire covered key areas of the pilot study. The questionnaire could be used for a broader industry survey if further data validation is required.
� One-day Assessor orientation session including a brief overview of practical tools worked well for the pilot. There is no need for detailed training on the tools prior to the pilot since tools are being revised during the pilot. Once assessment tools are tested and confirmed, a more detailed Assessor training session is required.
� Venue for the millwright pilot – medium size industrial shop – worked well. The pilot showed that practical assessment could be conducted in smaller millwright shops as well. Smaller shops will need to meet Venue criteria (e.g. tools, equipment, ventilation, safety).
� Venue for the metal fabricator pilot – training lab at PIC BC – worked well. Standard lab setting, low safety risks, availability of food, washrooms, study space, and staff’s familiarity with practical assessment procedures
Enhanced Challenge Pathway Project Final Report
March 2015
45
contributed to the pilot quality. Approved Venue criteria worked well. It is important to specify venue requirements prior to the pilot and ensure safety and insurance needs. Venue criteria should be reviewed during the pilot and revised if needed.
9.2 What Did Not Work?
� Lack of clarity provided to the ITOs on the enhanced challenge pathway framework, newly introduced assessment methods and their purpose for the challenge process. Lack of clarity provided to the ITOs on requirements for tool development resulted in tools that are not standard and difficult to manage within standard protocol.
� Tools developed by the ITOs and the SMEs are too complex and require significant time and resources to implement.
� Tools’ complexities significantly increase the cost of practical assessments and, therefore, challenge fees. Candidates may simply go to another jurisdiction to become certified.
� Piloting only the practical assessment is not sufficient for the full implementation of enhanced challenge pathway. There is a need to pilot other assessment tools and forms to ensure the quality and efficiency of the enhanced challenge process.
� Making practical assessment part of the challenge process for non-mandatory trades, such as Carpenter, may result in Candidates not seeking certification or getting Certification in other jurisdictions.
� Some tools, such as HDET scope verification interview, require substantial revisions to be used according to their purpose (e.g. scope verification rather than competency validation).
� Assessor trainees who participated in the pilots would need to complete full ITA Registered Assessor Training Course to ensure they have a strong understating of assessment tools and forms and can effectively manage all required protocols.
� Assessor recruitment for the HDET that focused on college instructors did not generate full support for the project, as the objective of the project is to certify Candidates outside the trade school/apprenticeship programs.
� Assessor recruitment for the HDET pilot did not consider Assessors’ and Venues’ location. Travel and accommodation costs increase the costs to the Candidate.
� Unforeseen circumstances (business and personal emergencies) delayed the implementation of the Millwright project. Consequently, the project deadline required extension.
� No guidelines on enhanced challenge pathway development and implementation processes resulted in lack of understanding about project goals and expected outcomes among ITOs. The guidelines would help align and project manage several projects with multiple stakeholders simultaneously.
� Practical assessment tools were designed to evaluate standard work procedures (process) and performance outcomes (product). This made the tools too complex. The need for these types of practical assessment tools was not clearly defined. Therefore, tools needed to be revised and simplified.
� Millwright practical assessment tool did not have a completed Candidate Handout. A Candidate Handout, including clear instructions on practical tasks and achievement criteria, was completed during the pilot.
� Small-scale pilot did not provide sufficient time for Assessors to properly work with revised practical assessment forms (e.g. marking guide, assessment record, assessment report) and learn standard ITA assessment protocols. Additional training is required.
� One elective task for the Millwright pilot – assembling hydraulic circuit – could only be realistically performed on a hydraulic training board in a College Lab. The pilot team suggested using computer based hydraulic simulators. Further examination of this option is required.
� Review and analysis of the enhanced challenge assessment documentation (tools and supporting forms) showed that the documentation’s format varies. That may create administrative inefficiencies in the future. Standard templates (need to be developed) would help with building standard enhanced challenge pathway administrative system and process.
Enhanced Challenge Pathway Project Final Report
March 2015
46
10 Conclusion
Since the inception of this project, a series of initiatives with overlapping scope have commenced. ITA is involved and fully
committed to the success of these initiatives and will ensure they benefit from any lessons learned from this project. As a
consequence, ITA will not be moving forward at this time with any of the recommendations flowing directly out of this
project.
Below is a description of two key national initiatives whose scope overlaps that of this current project:
10.1 National FQR Initiative
In September 2013, CCDA launched the FQR Initiative with the intent of creating an optimal process and standards
framework that is timely, fair, transparent and consistent across all jurisdictions for assessing the eligibility of internationally
trained workers to become certified in any one of the Red Seal trades.
A Project Task Team of representatives from six Canadian jurisdictions has been formed for this initiative. Representatives
are from the provinces of Alberta, British Columbia, Manitoba, New Brunswick, Ontario and Saskatchewan along with an
ESDC staff member. The Government of Alberta has received funding from ESDC to manage the CCDA FQR Initiative.
Ideal Process Model
As one of the first steps in the CCDA FQR Initiative, an Ideal Process Model was developed that assists in determining
eligibility decisions for the Red Seal Exam. The model is broken into five distinct phases:
1. Pre-application 2. Application 3. Application assessment 4. Competency assessment 5. Decision and communication points
Below is a graphical representation of the Ideal Assessment Process
Enhanced Challenge Pathway Project Final Report
March 2015
47
Projects Summaries:
Project #1: Application and Assessment Process project
Application and application assessment were identified as the best place to start strengthening the Ideal Process Model. A
contractor was hired in March 2014 and launched the Application and Assessment Process project by reviewing the
application documents and assessment processes used in each participating jurisdiction. The Application and Assessment
Process project produced a set of documents and processes that will increase consistency, transparency and fairness
across jurisdictions while respecting differences in standards and legislation. The work on the Application and Assessment
Process project wrapped up in December 2014.
Project #2: Application and Assessment Pilot and Implementation Planning Project
The Application and Assessment Process work continued in January 2015. In this project there will be pilot testing of the
revised application package in six jurisdictions across Canada. Once the outcomes of the pilots are evaluated, preparation
will be made for full implementation.
Project #3: Content for Web Based Resources Project
In January 2015 work also began to support the pre-application phase of the Ideal Process Model by identifying, gathering
and analyzing information and materials intended to populate several web based tools that are under development. These
tools include a website, an online self-assessment tool and an inventory of out-of-country credentials and training systems.
Enhanced Challenge Pathway Project Final Report
March 2015
48
The website will help internationally trained workers and employers navigate Canada’s foreign qualification recognition
systems and become better informed about working in the trades before submitting an application. The website will house
information, links and tools including self-assessment tools designed to compare an applicant’s experience against a trade-
specific Red Seal National Occupational Analysis. Content for the website will come from a combination of pre-existing and
new information and resources.
The inventory of out-of-country credentials and training systems will help jurisdictions review applications from
internationally trained workers in a more efficient manner. Details gathered could include descriptions of trades-related
training programs, regulatory bodies, credentials issued, sample documents as well as useful websites and contact
numbers. Content for the inventory will come from a combination of pre-existing and new information and resources.
10.2 National Harmonization Initiative In July 2014, the Forum of Labour Market Ministers (FLMM) recognized the need to enhance and improve apprenticeship in
Canada. Ministers agreed to a pan-Canadian approach to advance apprenticeship initiatives for implementation of the
harmonization of 10 Red Seal Trades in 18 months by Fall 2015.
The Canadian Council of Directors of Apprenticeship (CCDA) identified Harmonization as a strategic priority for 2013
through 2016 and has done extensive research and consultation with Industry and training providers nationally. The FLMM
has charged the CCDA with moving this project forward.
On February 27, 2014, the CCDA approved the following recommendations for the Harmonization project on 10 targeted
trades:
• Sequencing of technical training curriculum content and total training hours (in-school and on-the-job combined)
• Jurisdictional trade names and definitions;
• Use of the most recent national occupational analysis for P/T technical training curriculum development; and
• Specific to the Mobile Crane Operator trades, more consistent weight restrictions and equipment classifications.
These recommendations were subsequently confirmed with national industry and training stakeholders at consultation
meetings help in February and April 2014 as well as the CCDA National Stakeholder meeting in March 2014. Stakeholders
were supportive of work on harmonization and indicated their interest in continued engagement on the Initiative. Industry
also confirmed that sequencing of technical training was the key priority to enhance apprentice mobility.
In conjunction with the Forum of Labour Market Ministers (FLMM) and the Canadian Council for Directors of Apprenticeship
(CCDA), the ITA was tasked with providing a report that represented the consultations with BC’s Industry and Training
Providers. As shown in the National Ideal Assessment Process Model above, the work on competency assessments is
scheduled for after steps 1-3 are completed.
Enhanced Challenge Pathway Project Final Report
March 2015
49
10.3 Findings for National Initiatives
ITA will pass on the following 26 findings to the national initiatives. These are based on the outcomes from the four research
streams of this study: ECP project review, three practical assessment pilots, industry consultation update and environmental
scan.
The findings are organized by topics used in the industry interviews and are not listed in any priority order. Before pursuing
enhanced challenge for the five trades in this study, or other additional trades in the future, prioritization of these findings
would help determine short, medium and long-term priorities in order to build an effective enhanced challenge system.
The 2013 industry interview sample size for each trade group was small (5-8 interviewees). The reader should keep this in
mind when interpreting results and findings. Also, it is recommended that project findings be validated with Advisory Groups
and broader industry representation.
10.3.1 Challenge requirements
According to the industry interviews, the majority of respondents view the current challenge process as lacking rigor.
Findings for strengthening the challenge process include requiring additional proof of experience within the scope of the
trade, increased employer sign-off responsibility, assessment of core practical skills, completion of a college-based
practicum, and subject matter interviews. Additional findings include:
1. Developing an enhanced challenge application package that includes specified employer and statutory declarations. 2. Developing online self-assessment forms to familiarize candidates with challenge requirements and help determine
their skill set and appropriate challenge level. 3. Developing certification challenge guide for employers that outlines employer responsibilities in declaring and
verifying candidates’ working and on-the-job training experiences. 4. Determining trade(s) for practical assessment implementation (consider regulated and non-regulated trades,
number of challenge candidates per year, trade harmonization priorities).
10.3.2 Challenge processes, supporting tools and forms
Per the consultation findings, an interview with a subject matter expert and a practical assessment are viewed as two
additional methods to evaluate experienced trades workers’ competency. The practical assessment, as data shows, has
more value and weight for employers than the interview. The consultation confirmed that adding the practical assessment
component to the challenge process would help increase employers’ confidence in the skills of workers who achieved
trades certification through challenge. As such, findings include:
5. Including practical assessment in the challenge process and develop supporting forms. 6. Reviewing and validate revised practical assessment tools and protocols with Trade Advisory Groups. Finalize
accordingly. 7. Developing standard templates that can be populated with available trades-specific information. Support with
samples that contain instructions. 8. Reviewing and validate operating models with Trade Advisory Groups. Finalize accordingly.
Enhanced Challenge Pathway Project Final Report
March 2015
50
9. Conducting trade specific surveys on different practical assessment formats (college-based practicum, employer sign-off, worksite standardized assessment) to prove the concept with broader industry.
10. Developing new employer declaration forms by including more detailed information on the scope of the trade. 11. Increasing employer responsibility in signing off candidates’ industry experience by introducing “Sign-off Liability
Statement” or other relevant measures. 12. Developing challenge pathway process maps for candidates and employers to help prepare and manage enhanced
challenge processes. 13. Developing enhanced challenge guide for candidates and employers that outlines challenge completion
requirements and processes. 14. Developing online challenge preparation tools and resources (e.g. self-assessment checklist, competency profiles,
challenge guide for employers)
10.3.3 Costs
Findings regarding costing include the need to:
15. Discuss assessment costs determined through the pilots with potential assessment agency partners. 16. Re-assess assessor pay rates used during the pilots and determine a benchmark rate. Test to see if assessors will
agree to perform services at that rate. 17. Confirm assessor competency, training and currency requirements meet any policy guidelines and estimate costs of
providing the training. 18. Determine processes, estimate required internal and external resources for launching assessment for one trade.
10.3.4 Implementation
If implementing practical assessments, findings include:
19. Setting up practical assessment launch criteria and choosing one out of three piloted trades for implementation. 20. Analyzing effectiveness of operational models for each of the three piloted trades and developing ECP operating
manual(s). 21. Exploring industry proposed college-based practicum model and its potential for enhanced challenge pathway.
10.3.5 Candidates
Developing effective mechanisms for attracting challenge candidates is crucial for building skilled workforces in BC. To
ensure that the candidates from different groups of the province’s labour pool can apply, prepare for and complete the
challenge process, findings included:
22. Developing an online self-assessment tool that would help the candidates assess their skills and determine their readiness for certification challenge versus entering the apprenticeship training system or doing gap training.
23. Developing online challenge resources (e.g. process map, FAQ, training programs, courses and learning materials).
10.3.6 Assessors
Registered Assessors are an integral part of a practical assessment process. Assessors would need to complete a robust
Enhanced Challenge Pathway Project Final Report
March 2015
51
assessor training program, maintain currency in their trade and well as assessment methodology. As such, findings include
the need to:
24. Conduct ongoing assessor training, moderation, and professional development to ensure that the Assessors stay current on industry performance standards.
25. Build an assessor candidate pool.
10.3.7 Enhanced Challenge Pathway System
Enhanced challenge is intended to strengthen both the challenge process and the validity of its outcomes. The system
needs to provide an effective framework for evaluating trades equivalency of a broad pool of experienced trades workers
including those who gained their experience and education in and outside of Canada. The system needs to be efficient with
its resources, budget and time requirements to deliver clear benefits which outweigh costs and risks to all parties involved.
As such, findings include:
26. Determining core operational processes (e.g. assessment, training, evaluation, insurance, audit, appeal) and required resources.
Enhanced Challenge Pathway Project Final Report
March 2015
53
1. Appendix A. Project Methodology 54 2. Appendix B. Industry Interview Questionnaire 55 3. Appendix G. List of Interviewees 57 4. Appendix D. Carpenter Group Interview 58 5. Appendix E. Construction Electrician Group Interview 60 6. Appendix F. Metal Fabricator Group Interview 62 7. Appendix G. Millwright Group Interview 64 8. Appendix H. Heavy Duty Equipment Technician Group Interview 66 9. Appendix I. Industry Interview Summary 68 10. Appendix J. Summary of Registration Practices of Select Non-trades Occupations 70 11. Appendix K. ITA Registered Assessor Training 73
Enhanced Challenge Pathway Project Final Report
March 2015
54
Appendix A. Project Methodology
Table 3.1 ECP Project Methodology
Source Information collected
1. Comparative analysis – Review of Phase I ECP activities, processes and outcomes
Assessment tools and forms for five trades, their purpose and practical use, content, alignment with standards, benchmark for developing practical tasks, challenge process, new roles and responsibilities for conducting practical assessment.
2. Environmental scan – Review of application registration practices
Requirements for registration/licensure as engineers (APEGBC), technicians, technologists, (ASTTBC), and registered nurses CRNBC. Provisional certification practices. Registration requirements, registration/licensure processes for internationally trained professionals. Information and resources for internationally trained professionals.
3. Qualitative research - Industry consultation and key informant interviews
Perspectives of employer, labour and industry association representatives, human resources and apprenticeship program representatives on current challenge requirements and challenge process, practical assessment for challengers (Total number of interviewees=33).
4. Pilot study - Practical assessment pilots
Tools and forms design and content. Assessor readiness and training. Operational models, cost effectiveness, usability of assessment tools, relevance of practical tasks in evaluating candidate performance against approved industry standards, estimated assessment fees, continued process improvements.
Enhanced Challenge Pathway Project Final Report
March 2015
55
Appendix B. Industry Interview Questionnaire
Interview: Practical Assessment Component in the Certification Challenge Process
The purpose of this interview is to consult with industry on the importance/value of the practical assessment component in
the certification challenge process. In addition, this interview is intended to clarify how trade certification helps employers to
source, select and hire experienced tradespersons from a diverse pool of job candidates, including those who obtained their
trades work experience in, across and outside Canada.
Currently, in order to achieve trade certification through exam challenge, experienced workers must prove required working
hours within the scope of the trade and pass a multiple-choice exam (passing mark is 70% or higher).
I. Challenge Requirements
1. Challenge candidates for your trade must have completed a certain number of work based hours. In addition, they must have a past (or current) employer declare witnessing the candidate having performed a specific range of tasks (unique for each trade). Do you think this is enough or does it need to be strengthened?
2. Challenge candidates include representatives from the following groups: a. Experienced trades people who gained their work experience in B.C.; b. Experienced trades people who gained their experience in other Canadian provinces; a. Experienced trades people who gained their trades experience and education outside of Canada.
Should the certification challenge requirements for different groups be the same or different?
II. Enhanced Challenge Process and Components
3. Would adding a practical assessment as a mandatory component of the certification challenge process for your trade strengthen the process and/or increase your confidence in the abilities of workers who achieve certification through the challenge process?
4. How long do you think a practical assessment needs to be to determine a worker’s level of competence in your trade (e.g. to give you sufficient confidence in the challenger’s abilities to hire them?)
5. Do you think that requiring a practical assessment for certification challengers would deter appropriate candidates from challenging for certification in your trade?
6. Other than conducting a practical assessment, there may be ways that the challenge process for your trade could be strengthened. One way is by requiring candidates to be interviewed by a subject matter expert about their knowledge and experience. Do you think this could replace a practical assessment for your trade?
III. Challenge costs and value of certification
7. Enhanced assessments cost a great deal more to administer than machine-scored multiple-choice examinations. How much do you think a certification challenge candidate would be willing to pay for assessment (e.g. At what price do you think some or most candidates would decline assessment and not apply for challenge?) Price I think candidates would be willing to pay for enhanced challenge in my trade: _$____________
8. If the fee for assessment was higher than the cost you estimated above, what do you think the impact would be in your industry?
IV. Importance of Certification for your workers and in making hiring decisions
9. Do you hire experienced trades workers without formal Canadian credentials? (C of Q/ Red Seal)?
Enhanced Challenge Pathway Project Final Report
March 2015
56
10. At your firm, what is the optimal percentage of certified tradespersons in Carpentry? (e.g. is it a requirement for all the workers to be certified or are there lower-level jobs, which don’t require certification that make up a significant portion of your skilled trades workforce?)
11. It has been stated that employer confidence in skills and abilities of workers certified by exam challenge is lower than for those of workers who completed apprenticeships. What is your level of agreement with that statement? If you agree, how would the current challenge process have to change to increase your confidence in the knowledge
and abilities of candidates who achieved certification through challenge?
12. Can you please describe the process your organization uses for screening, selecting and hiring trades workers and what, if any, consideration is given to whether or not the applicant is certified in their trade?
Enhanced Challenge Pathway Project Final Report
March 2015
57
Appendix C. List of Interviewees
33 industry representatives total
The Carpenter Group (5)
1. Abigail Fulton, Vice-President BCCA, Victoria, BC 2. Eric Bell, Northern Legendary Construction, Owner, Charlie Lake, BC 3. Larry Richardson, Director of Training CLAC, Langley, BC 4. Randy Callaghan, Personnel Manager, PCL Construction, Vancouver 5. Steve Haab, Haab Home Construction, Owner, Fort St. John, BC
The Construction Electrician Group (8)
6. Andy Cleven, Electrical Training Joint Committee 7. Amy Davidson, HR Manager, CCT Controls 8. Carol Rycrofft/HR Manager, Superpoint Technologies Group 9. Kevin Shillito/Manager of BC Operations Superpoint Technologies Group 10. Abigail Fulton, Vice President, BC Construction Association 11. Cathy Thomson, Payroll Administrator, Status Electrical, Abbotsford, BC 12. Foreman (name TBC) Status Electrical, Abbotsford, BC 13. Ted Pimms, General Manager, Pimm’s Production Equipment, Fort St. John, BC
The Metal Fabricator Group (7)
14. Allen White, Fabrication Manager, Teck, BC 15. Dave Marcinew, Plant Manager, CANRON, Delta, BC 16. John Folkers, Fabrication Foreman, Weldco Beals, Langley, BC 17. Ian Baxter, Apprenticeship Manager, National Defense, Victoria, BC 18. Kevin Leyenhorst, Manager, Intercontinental Truck Body, TB 19. Mike Kuzinuk, President, Fabrite Services Ltd., Cranbrook, BC 20. Trevor Hill, Metal Fabricator Lead/Manager National Defense, Victoria, BC
The Millwright Group (7)
21. Al Storey, Millwright, Harmac Pacific, Nanaimo, BC 22. Chris Babcock, Manager Field Services, Service on Site, Fort St. John, BC 23. David Hiltz, Millwright, Catalyst Paper, Port Alberni, BC 24. Gordon Balfour, Millwright, Training Coordinator, Millwrights Union, Langley, BC 25. Mike Cass, Vice-President HR, Western Forest Products, Vancouver/Nanaimo, BC 26. Vern Phillips, General Manage HR, Catalyst Paper, 27. Richard Turnbull, Millwright and Machinist, FMF, National Defense, Victoria, BC
The Heavy Duty Equipment Technician Group (6)
28. Devon Lock, Mechanic Nels Ostero Sand and Gravel, Taylor, BC 29. Doug Richardson, General Manager, Inland Kenworth, Langley Branch, BC 30. Gordon Armour, BC Mining Task Force. 31. Ian Simpson HSE Manager/Edmonton Parts, Compression Technology Inc., Fort St. John B.C. 32. Richard Demchuk, Manager Human Resources Howe Sound Pulp and Paper Corporation 33. Alejandra Posadas, Apprenticeship Coordinator, Finning, Edmonton, AB
Enhanced Challenge Pathway Project Final Report
March 2015
58
Appendix D. Carpenter Group Interview
The group consisted of five respondents:
1. Abigail Fulton, Vice-President BCCA, Victoria, BC 2. Eric Bell, Northern Legendary Construction, Owner, Charlie Lake, BC 3. Larry Richardson, Director of Training CLAC, Langley, BC 4. Randy Callaghan, Personnel Manger, PCL Construction, Vancouver 5. Steve Haab, Haab Home Construction, Owner, Fort St. John, BC
1. Current challenge requirements. 4 out of 6 respondents think that the current requirements for the challenge are satisfactory.
4 out of 6 said that the challenge process needs to be strengthened by enhancing the experience
verification procedure and possibly including employer based practical assessments.
2. Challenge requirements for different groups. All six (6) participants agree that the challenge requirements for all individuals (with domestic or foreign
experience) should be the same.
3. Practical assessment as a mandatory challenge component. Respondents agree that the practical assessment will strengthen the challenge process and increase
employers’ confidence in Candidates.
There is some support for introducing the practical assessment as a mandatory component of the
challenge process; however, there are concerns about practical assessment scope, delivery format, and
industry and Candidate buy-ins.
A practical assessment on the employer’s site – Employer Sign Off – is preferable.
4. Practical assessment duration. The suggested duration for the practical assessment is from three (3) to eight (8) hours.
5. Practical assessment as a barrier. Some respondents think that the mandatory practical assessment will deter some Candidates. Others think
that the appropriate Candidate will not be discouraged from taking the practical assessment.
6. Interview as a substitute for practical. All view the Candidate Interview useful but not as a substitute for the practical assessment.
7. Practical assessment fee. Respondents think that the fee for the practical assessment should be in the range of $300-$500
8. Fee as a barrier. The fee should be reasonable.
If it is too high it will create barriers to certification.
*Note that current estimates of cost-recovery fee for practical assessment are significantly higher than the
range suggested by interviewees.
9. Trades workforce supply pool. Most (4 out of 5) employers hire Candidates without certification.
These Candidates come from the two groups identified in the interview: 1) with experience across Canada
and 2) with foreign experience and certification;
Enhanced Challenge Pathway Project Final Report
March 2015
59
10. Optimal percentage of trades certified workers. The optimal percentage of certified Carpenters in the companies represented by the interviewees varies
from 50% to 90% and depends on projects and operations.
One respondent was not sure about these numbers and said that the number of certified Carpenters in the
company is market driven – when the demand is high, employers are not looking for a certified Carpenter,
but a Carpenter who can complete a project on time.
11. Employer confidence in skills of Challenge vs. Apprenticeship Candidates. Three (3) out of five (5) disagree that employers have lower confidence in the competencies of those who
challenged the certificate than those who completed formal apprenticeship. One (1) support that statement
and one (1) is not sure.
12. Certification and hiring process. The hiring process for both big and small employers is focused on finding the most experienced Candidate
for the role.
Although trade certification is viewed as valuable, years, breadth and depth of experience outweigh the
value of the certification.
Most of the employers provided various examples of their “star” performers who are not certified and
compared them with a Red Seal certified journeyman who completed a 4-year apprenticeship but knows
only the basics of the trades and the industry.
All employers agreed that the depth and the breadth of work experience is what makes a real journeyman
Carpenter.
A combination of certification and experience is what employers are looking for in the right candidate.
The hiring process for many employers is focused on determining practical skills and the scope of trade
experience.
Certification is valued but preference is often given to strong industry experience.
Canadian experience is generally valued higher.
Enhanced Challenge Pathway Project Final Report
March 2015
60
Appendix E. Construction Electrician Group Interview
Eight industry representatives participated in the interview (two companies provided two representatives each).
1. Andy Cleven, Electrical Training Joint Committee 2. Amy Davidson, HR Manager, CCT Controls 3. Carol Rycrofft/HR Manager, Superpoint Technologies Group 4. Kevin Shillito/Manager of BC Operations Superpoint Technologies Group 5. Abigail Fulton, Vice President, BC Construction Association 6. Cathy Thomson, Payroll Administrator, Status Electrical, Abbotsford, BC 7. Foreman, Status Electrical 8. Ted Pimms, General Manager, Pimm’s Production Equipment, Fort St. John, BC.
Interview summary is provided below:
1. Current challenge requirements. 3 out of 7 interviewees think that the current challenge requirement is enough.
4 think that it should be strengthened.
One suggests that the amount of required working hours could be increased to 10,000 hours.
One suggested five years of industry experience in Canada.
Other suggestions include Grade 12 diploma, completion of the Canadian Electrical Code course, and
completion of college training (e.g. practicum, level 4) similar to apprenticeship.
Great importance was given to the validation of experience within the scope of the trades.
2. Challenge requirements for different groups of candidates. Most (5) employers think that the challenge requirements for different groups need to be the same (hours,
education requirements)
Some (2) suggest that internationally trained professionals may need to go through a different process of
evaluation to ensure that their experience matches Canadian requirements.
A comment about trades training systems in immigrants’ home countries pointed out that while some
systems are similar to Canadian systems, others are significantly different.
One representative suggested developing a “preferred countries” list and streamlining the process for
Candidates from those countries by recognizing education and issuing “provisional certification.” In this
case, the next step could be job placement and employer sign-off. After that the candidate would receive
full certification.
3. Practical assessment as a mandatory challenge component. The idea of practical assessment as a mandatory component of the challenge process is perceived
positively.
Employers agree that this would help determine the Candidate’s level of expertise. Many want to see a
practical assessment that would help identify gaps, and help suggest further training and job placement.
There are a number of concerns about mandatory practical assessments, including the scope of trade,
assessment efficiency, format and costs.
4. Practical assessment duration. The suggested duration of practical assessment is broad – from six (6) to eight (8) hours for an
assessment that targets core competencies, to a two-month continuous assessment on the employer’s site
or at the college.
Enhanced Challenge Pathway Project Final Report
March 2015
61
5. Practical assessment as a barrier. Employers’ opinions on whether the practical assessment is going to deter appropriate Candidates vary.
Two (2) think that this may create unnecessary barriers.
Others think that it will not be a problem for the ‘appropriate’ Candidates.
Generally, those who deal with selection and hiring are more likely to see the practical assessment as a
barrier; the majority of operational representative do not see it as an obstacle.
The respondents prefer a school-based practical assessment such as the completion of a practical course,
practical modules, or projects.
6. Interview. All respondents agree that the interview is a valuable tool but it cannot replace the practical assessment for
this trade.
7. Practical assessment fee. The suggested fee for practical assessment is in the range of $350-$1000.
8. Fee as a barrier. In general, a fee higher than $1000 should not have an impact in the industry. However the fee structure
needs to be reasonable and comparable with trades training tuition fees.
9. Job candidates. As for the job Candidate pool, most respondents hire workers without Canadian credentials; most workers
have Canadian experience, followed by workers with foreign experience and credentials. Two respondents
said that they also hire workers with foreign experience and without credentials.
10. Optimal percentage of trades certified workers in a company. The optimal percentage of certified workers working for firms represented by the interviewees varies from
25% to 100%.
11. Employer confidence in skills of Challenge vs. Apprenticeship Candidates The respondents’ support for the statement regarding a lack of confidence in the skills of Candidates who
challenged their certification differs.
Two out of seven disagree, one is not sure and the rest agree. The latter group suggests that adding a
practical test, interview, or completing training can strengthen the current challenge process.
12. Certification and hiring process. In recruitment, all of the employers focus on previous employment, direct industry experience, certification
and safety training.
Some employers include tests (e.g. math, codes, safety) in their hiring process.
All employers look for the right attitude and a good fit with company culture.
All employers put job candidates on probation (two weeks – three months) prior to offering full-time
employment.
Additional comments and suggestions include:
4. Develop strategies for attracting Electricians from the U.S. to meet growing demand for trained and certified electrical workers.
5. Consider a provisional certification for electrical workers. 6. Add college practicum to the certification challenge process as a substitute to practical assessment. This would
better align apprenticeship and challenge processes than a practical test for challengers. It would also increase employer confidence in Challengers’ skills and knowledge. A practical assessment delivered through a college would be more standardized and may have more value.
Enhanced Challenge Pathway Project Final Report
March 2015
62
Appendix F. Metal Fabricator Group Interview
Seven industry representatives participated in the interview:
1. Allen White, Fabrication Manager, Teck, BC 2. Dave Marcinew, Plant Manager, CANRON, Delta, BC 3. John Folkers, Fabrication Foreman, Weldco Beals, Langley, BC 4. Ian Baxter, Apprenticeship Manager, National Defense, Victoria, BC 5. Kevin Leyenhorst, Manager, Intercontinental Truck Body, TB 6. Mike Kuzinuk, President, Fabrite Services Ltd., Cranbrook, BC 7. Trevor Hill, Metal Fabricator Lead/Manager National Defense, Victoria, BC
The summary below outlines the key findings of the interview:
1. Current challenge requirements. Two (2) out of seven (7) respondents think that the current challenge requirements are enough. Five (5) out of seven (7) think that although the requirements are reasonable, there are many concerns about
experience validation and competency evaluation processes.
There is low trust in the current employer declaration – respondents think that it is too general and requires
more in depth description of the Candidate’s skills and experience.
Many believe that writing an exam is good, but not enough for evaluating or predicting candidate performance
to approved industry standards.
They suggest including a practical assessment, an interview with Challengers, and an interview with
employers to determine Challengers’ competencies and practical skills.
2. Challenge requirements for different Candidate groups. Participants think that the challenge criteria should be the same for all Challenger groups.
3. Practical assessment as a mandatory challenge component. Five (5) out of seven (7) interviewees said that a mandatory practical assessment would increase their
confidence in the skills of workers who challenged their Metal Fabricator trade certification.
4. Practical assessment length. The suggested length of the practical assessment component varies from four (4) hours to a 4-6 weeks
college-based practicum.
The college-based practicum is viewed as an opportunity to bridge the gap between the Challengers and
Apprentices. This type of practicum is viewed as more in-depth standardized industry training.
5. Practical assessment as a barrier. Those who supported the practical assessment as part of the challenge process do not think that it will deter
‘appropriate’ Candidates.
An appropriate Candidate, according to an employer is someone who is serious about his/her career and
supports his/her own professional development.
6. Interview as a substitute for practical assessment. Interview with the Challengers can be helpful but cannot substitute practical assessment.
7. Practical assessment fee. Assessment fee varies from $300 to $2000 where the former is for a 4– 8 hour assessment and the latter is
for a 4-6 weeks college-based practicum.
Enhanced Challenge Pathway Project Final Report
March 2015
63
8. Fee as a barrier. Participants think that the higher fee (more than $300-$500) will not have a significant impact on the industry.
Assessment fees need to be reasonable and built on a specific formula.
There is an understanding that the assessment business model needs to be built on an effective cost recovery
model.
9. Job candidates. Four (4) out of seven (7) interviewees hire workers without certification, mostly with Canadian experience.
One hires all three groups of workers.
Some employers (2) (mostly those who operate under union agreement) require C of Q or Red Seal for
employment.
Others hire non-certified workers in trades and trades related positions.
10. Optimal percentage of trades certified workers in a company. All employers would like to see most of their trades workers certified – they report that the optimal percentage
of trades workforces in their companies is 100% but current percentage varies from 10% to 75% depending
on operational needs.
11. Employer confidence in skills of Challenge vs. Apprenticeship Candidates. Five (5) out of seven (7) interviewees agree that employers have lower confidence in the competencies of
those who challenged the certificate than those who completed formal apprenticeship.
12. Certification and hiring process. Hiring practices have a high degree of similarities and are based on a screening process that includes
verification of qualifications, training, and industry experience. A small number of respondents hire entry-level
workers and provide in-house training and/or apprenticeship.
Enhanced Challenge Pathway Project Final Report
March 2015
64
Appendix G. Millwright Group Interview
Seven industry representatives participated in the interview:
1. Al Storey, Millwright, Harmac Pacific, Nanaimo, BC 2. Chris Babcock, Manager Field Services, Service on Site, Fort St. John, BC 3. David Hiltz, Millwright, Catalyst Paper, Port Alberni, BC 4. Gordon Balfour, Millwright, Training Coordinator, Millwrights Union, Langley, BC 5. Mike Cass, Vice-President HR, Western Forest Products, Vancouver/Nanaimo, BC 6. Vern Phillips, General Manage HR, Catalyst Paper, 7. Richard Turnbull, Millwright and Machinist, FMF, National Defence, Victoria, BC
1. Current challenge requirements The current hours should be the same. Most of the respondents would like the application and verification
processes to be strengthened; the suggestions on how to strengthen the processes are different and
include stronger more detailed employer declaration, increased employer responsibility for signing working
hours, interview and practical assessment.
2. Challenge requirements for different Candidate groups. The majority agrees that the challenge requirements should stay the same for different groups of
challengers.
3. Practical assessment as a mandatory challenge component. The practical assessment is viewed as a strong component that would strengthen the challenge process
and increase employers’ confidence in Challengers’ competencies.
4. Practical assessment length. The suggested duration for a practical assessment is from 4 to 16 hours. There was one suggestion to
have a 40-hour practicum at the college that would cover all major skills of a fully qualified Millwright.
5. Practical assessment as a barrier. Respondents do not think that a mandatory practical assessment would deter appropriate candidates from
challenging certifications.
6. Interview as a substitute for practical assessment. Interview with the Challengers can be helpful but cannot substitute practical assessment.
7. Practical assessment fee Assessment fees vary from $500 to $3000.
8. Fee as a barrier. If the fee were higher than suggested, impact on industry would not be significant. The fee needs to be
reasonable and need not create a barrier for those who are willing to challenge.
9. Job candidates. Most (6 out of 7) respondents do not hire workers without formal credentials.
One employer hires Canadian workers without certification and workers with international experience and
certification; they are hired on a condition to challenge trade certification.
10. Optimal percentage of trades certified workers. The optimal percentage of certified Millwrights for the majority of respondents is 95% -100%, excluding
apprentices.
11. Employer confidence in skills of Challenge vs. Apprenticeship Candidates. Five (5) out of seven (7) interviewees agree that their confidence in the Challengers’ skills and
competencies is lower than in skills of those who completed formal apprenticeship.
Enhanced Challenge Pathway Project Final Report
March 2015
65
12. Certification and hiring process. Hiring practices include screening for Red Seal, relevant industry experience, interview process, some
employment testing, probationary period and onboarding.
Enhanced Challenge Pathway Project Final Report
March 2015
66
Appendix H. Heavy Duty Equipment Technician Group Interview
This group included six interviewees:
1. Devon Lock, Mechanic Nels Ostero Sand and Gravel, Taylor, BC 2. Doug Richardson, General Manager, Inland Kenworth, Langley Branch, BC 3. Gordon Armour, BC Mining Task Force. 4. Ian Simpson HSE Manager/Edmonton Parts, Compression Technology Inc., Fort St. John B.C. 5. Richard Demchuk, Manager Human Resources Howe Sound Pulp and Paper Corporation 6. Alejandra Posadas, Apprenticeship Coordinator, Finning, Edmonton, AB
1. Current challenge requirements One (1) respondent said the challenge requirements are fully sufficient.
Other five (5) respondents agree that the current hours requirements are enough, but they would like to
see some improvements within the challenge process, specifically:
Improved verification process (e.g. ask employers to provide more details on the candidate’s experience
within the scope of the trade, add practical assessment);
Possible practical assessment.
2. Challenge requirements for different Candidate groups The majority (5 out of 6) of respondents think that the challenge requirements should be the same for
different groups of Challengers.
One respondent said that the challenge requirements should be different for the Candidates with non-
Canadian experience and education to ensure that foreign trained candidates meet Canadian standards.
To complete college training would help align the skills of foreign trained mechanics with Canadian
workplace requirements and standards.
3. Practical assessment as a mandatory challenge component Five out of six (5 out of 6) respondents think that adding a practical assessment as a mandatory
component would strengthen the challenge process and increase employers’ confidence in the abilities of
workers who achieved certification through the challenge process. All respondents agree that a practical
assessment would help demonstrate critical skills required for the workplace. In addition, most of the
respondents think that a practical assessment would help showcase their trade related skills.
4. Practical assessment length The suggested assessment duration varies: two hours, one full day, two full days, one month on the
employer’s site.
Those who suggested that the practical assessment be longer than one day think that eight (8) hours is not
enough to conduct an effective and meaningful assessment for this trade.
5. Practical assessment as a barrier Three out of six (3 out of 6) respondents do not think that requiring a practical assessment for certification
challengers would deter appropriate Candidates.
Two (2) respondents said that it might deter some Candidates.
Possible deterrents include: assessment fees, assessment delivery (lab vs. workplace) with the preference
for a workplace based assessment, or a Candidate’s readiness to challenge.
Overall, this should not be an issue as industry is looking for the appropriate candidates that meet industry
standards.
Enhanced Challenge Pathway Project Final Report
March 2015
67
6. Interview as a substitute for practical assessment The interview is seen as a positive component of the challenge process but not as a substitute for the
practical assessment.
One respondent suggested that challengers could go through a two to three hour interview with subject
matter experts to help determine the Candidates’ knowledge of processes and identify any possible gaps.
7. Practical assessment fee Assessment fee varies from $250 to $1000.
8. Fee as a barrier Increased assessment fee should not create any impact on industry. Assessment should be cost effective
and reasonable.
9. Job candidates – three groups One (1) respondent hires all three groups of workers;
One (1) respondent hires workers with Canadian experience and workers with international experience and
certification;
One (1) hires only workers with Canadian experience;
Two (2) do not hire uncertified workers;
One (1) skipped this question.
10. Optimal percentage of trades certified workers in company Three out of six (3 out of 6) respondents were not sure.
Other three responses varied from 65% to 100%.
11. Employer confidence in skills of Challenge vs. Apprenticeship Candidates One (1) disagrees, four (4) agree, one (1) is unsure about the statement.
Suggestions to strengthen the challenge process include incorporating practical assessment, interview and
verification process.
12. Certification and hiring process Respondents’ hiring practices are similar and include the following:
Job advertising;
Internal hiring for unionized companies and mandatory certification;
Screening for certification, industry experience, relevant training;
Interview and tests (math, aptitude, personality);
Probationary period
Enhanced Challenge Pathway Project Final Report
March 2015
68
Appendix I. Industry Interview Summary
The information included in this table is based on the majority of responses for each question.
Table 4.3 – Industry Interview Summary
Trade
Question
Carpenter Construction
Electrician
HDET Metal Fab Millwright
1. Current challenge
requirements
Need to be
strengthened
Need to be
strengthened
Need to be
strengthened
Need to be
strengthened
Need to be
strengthened
2. Challenge requirements
for different groups
Same Same Same Same Same
3. Practical assessment
as a mandatory
challenge component
Support
Support
Support Support Support
4. Practical assessment
duration
4-8 hours or
Employer
Sign Off
8 hours or
College based
practicum
8 hours or
College based
practicum
8 hours
8 hours
College based
practicum
5. Practical assessment
as a barrier
No
No No No
No
6. Interview
Only to support,
not substitute
practical
Only to support,
not substitute
practical
Only to support,
not substitute
practical
Only to support,
not substitute
practical
Only to support,
not substitute
practical
7. Practical assessment
fee
$350-$600 range $750-$1000
range
$500-$1000
range
$500-$750 range
$500-$1000
range
8. Fee as a barrier
No/ only
for some
candidates
No/ only
for some
candidates
No/ only
for some
candidates
No/ only
for some
candidates
No/ only
for some
candidates
9. Job candidates
Hire all 3 groups
plus apprentices
Mostly with
Certification,
experience
across Canada
plus apprentices
Mostly with
experience
across Canada
plus apprentices
Mostly with
Certification,
Experience
across Canada
plus apprentices
Mostly with
experience
across Canada
plus apprentices
10. Optimal % of trades
certified workers in a
journeyperson position
Union – 100%
other depending
on market
demand
Up to 100%
Up to 100%
80%-100% Up to %100
11. 50:50 Lower in Lower in Lower in Lower in
Enhanced Challenge Pathway Project Final Report
March 2015
69
Trade
Question
Carpenter Construction
Electrician
HDET Metal Fab Millwright
Employer Confidence challenge
candidate
challenge
candidate
challenge
candidate
challenge
candidate
12.
Value of Certification in
hiring process
Certification
preferable
Experience
mandatory
Certification
mandatory
Experience
mandatory
Certification
mandatory
Experience
mandatory
Certification
preferable
Experience
mandatory
Certification
mandatory
Experience
mandatory
Enhanced Challenge Pathway Project Final Report
March 2015
70
Appendix J. Summary of Registration Practices of Select Non-trades Occupations
Association of Professional Engineers and Geoscientists of B.C. (APEGBC)
APEGBC is the licensing and regulatory body for B.C.’s professional Engineers and Geoscientists. Only engineers licensed
with APEGBC, or those who work under a direct supervision of P.Eng. licensed with APEGBC, are allowed to practice
engineering in B.C.
Registration Requirements
To register as a Professional Engineer (P.Eng.) with APEGBC, the Candidate needs the following:
� Acceptable academic qualifications; � A minimum of four years of satisfactory engineering experience with references; � Completion of the Law & Ethics Seminar requirement; � Successful completion of the Professional Practice Examination (PPE); � English language competency.
All applicants must demonstrate that they have Canadian Environment experience. The term Canadian Environment is
defined as:
� Work experience obtained in Canada, supervised by a Professional Engineer, registered or licensed in the applicable Canadian jurisdiction; or
� Work experience acquired outside Canada, where applicants demonstrate good knowledge of local Canadian engineering laws, practices, standards, customs, codes, conditions and climates
Membership and licensure include 14 categories, such as Professional Engineer, Engineer-in-Training, Limited Licensee
(Eng.L.) Professional Geoscientist, Geoscientist-in-Training, Non-Resident Licensee (NRL) P.Eng., International Mobility
P.Eng.
Self-Assessment Tool for Internationally Trained Engineers
APEGBC, in collaboration with the Applied Science Technologists and Technicians of BC (ASTT BC) and the Ministry of
Jobs, Tourism and Skills Training, has developed an online self-assessment tool. The tool was designed to help
internationally trained Candidates determine whether they should pursue a career as a Professional Engineer or as an
applied science Technologist or Technician.
The self-assessment tool includes a series of questions on education and work experience and provides information on how
declared professional qualifications are likely to be assessed by APEGBC or ASTTBC. The applicants can print the results
of the self-assessment. The applicants are informed that the self-assessment is not an official assessment of qualification
for registration or license in B.C.
The applicants must submit a formal application for registration or license to the appropriate regulatory body supplemented
by the required documentation and fees.
Prospective applicants that completed training in Canada can also use the self-assessment tool; they may find it helpful in
evaluating their qualifications and learning more about the application process.
Enhanced Challenge Pathway Project Final Report
March 2015
71
Association of Technicians and Technologists of B.C. (ASTTBC)
ASTTBC is responsible for the assessment and certification of Technologists and Technicians in B.C. ASTTBC Certification is not required to practice. Professional designations include Applied Science Technologist (AScT) and Certified Technician (CTech). Other classifications include: Graduate Technologist/Technician (GradTech), Associate and Student.
Registration and Certification Requirements
The minimum education required for AScT certification is a Diploma of Technology (two or three years) from a nationally
accredited program or equivalent. AScT and CTech certifications require a minimum of two years of related work
experience, with at least one year at the level of certification requested.
Provisional Certification for Internationally Trained Professionals
Internationally trained professionals may qualify for provisional membership if their education qualification and work
experience are acceptable. Reclassification from provisional status is possible after one year of work experience that
demonstrates an understanding of Canadian standards.
The provisional members have to complete the one-year work experience requirement within a five-year period after
registration. After the five years, their membership status would automatically be required to be reassessed.
Internationally Trained Technology Professional Portal
ASTTBC, with the help from the BC Government and industry support, has created a web portal for internationally trained
professionals – www.itppbc.com - to guide the internationally trained applicants through the certification process and provide
information on working conditions, language requirements, and employer expectations.
Other available resources include a self-study guide, workshops and webinars, information on codes and standards,
professional practice and ethics; they are designed to help internationally trained professionals navigate the certification
process.
Further examination of these practices and resources is required to determine how the ITTP concept can be used and
applied for engaging and guiding internationally trained tradespersons through the trade certification process in B.C.
College of Registered Nurses of B.C.
In British Columbia, only individuals registered with CRNBC can legally call themselves a Registered Nurse or provide
duties legally restricted to the profession of registered nursing.
Registration requirements
First time applicants include:
• B.C. educated Nurses. Graduates from a nursing program in B.C. must apply for CRNBC registration and apply to write the national nursing exam;
• Canadian Nurses. Registered Nurses with current or previous registration in another Canadian jurisdiction, or recent graduates from a Canadian RN nursing program, are eligible to apply for registration in British Columbia;
• Internationally Trained Nurses. Internationally trained Nurses who have never been registered in Canada may be eligible for registration to
Enhanced Challenge Pathway Project Final Report
March 2015
72
practice in B.C. First, they must complete the National Nursing Assessment Service (NNAS) process. After completing the NNAS process, CRNBC may require internationally trained nurses to undergo a competency assessment.
Provisional registration
Provisional registration is granted to qualified applicants who have to meet outstanding registration requirements. The
provisional registration is assigned specific conditions that must be met to be eligible for the practicing registration. When
these conditions are met, the provisional registration is automatically converted to the practicing registration.
Examples of provisional registration conditions include:
1. For Canadian RN: a. A condition that the applicant writes and passes the national nursing exam b. A condition that the applicant completes supervised practice experience (SPE)
2. For International RN: a. A condition that the applicant writes and passes the national nursing exam b. A condition that the applicant completes coursework in a specific area
(Note – these courses will normally be determined after reviewing the results of a Substantially Equivalent Competency (SEC) Assessment)
c. A condition that the applicant’s employment is limited to specified facilities
Internationally Educated Nurses
National Nursing Assessment Service (NNAS) is a national body of nurse regulatory bodies funded by the Government of
Canada's Foreign Credential Recognition Program (FCRP).
It developed a harmonized approach to the assessment of internationally educated nurses (IENs). NNAS coordinates a
nation-wide approach for IENs seeking registration and licensure to practice in Canadian jurisdictions.
Significant features of the NNAS include:
• A single national web portal;
• A common approach to initial document collections, authentication, validation and fraud screening;
• A consistent competency based assessment of the IEN application file;
• A consistent format for advisory reports to regulatory bodies to inform eligibility assessments;
• A bilingual customer care centre for IEN applicants;
• A national IEN database providing enhanced reporting and analysis capabilities.
The self-selection guide helps applicants decide which nursing profession is best suited to their experience, education and
credentials obtained outside of Canada. The guide describes three regulated nursing groups – Registered Nurse (RN),
Licensed Practical Nurse (LPN), and Registered Psychiatric Nurse (RPN) – and requirements for registration.
The website contains numerous resources to assist applicants with certification and licensure process.
Enhanced Challenge Pathway Project Final Report
March 2015
73
Appendix K. ITA Registered Assessor Training
Pre-course Participant Knowledge Check
For each item below, please rate your current level of knowledge and understanding on a scale of 1 to 5
(1= no Knowledge; 5= very high)
Do not sign the form; your response is anonymous. At the end of the classroom session and again at the end of your
practicum mentored assessments you will be asked to update the rating and indicate where you feel you need additional
information or training.
Assessor Course Content Area My rating of my
knowledge
Knowledge of BC’s Trades and Occupational Certification Framework 1 2 3 4 5
Roles and Responsibilities of ITA Approved Assessment Agencies, Registered Assessors, and
Candidates
1 2 3 4 5
Qualities and Competencies of an Effective Assessor 1 2 3 4 5
Principles of Assessment and Rules of Evidence 1 2 3 4 5
Occupational Performance Standard Against Which Candidates are evaluated in my Trade or
Occupation
1 2 3 4 5
ITA Enhanced Assessment Operational Procedures 1 2 3 4 5
Methods of Assessment for my Trade or Occupation 1 2 3 4 5
How to Use Approved Trade or Occupation-specific Assessment Tools 1 2 3 4 5
Use of Effective Questioning Techniques 1 2 3 4 5
How to Complete ITA Forms, Keep Records and Report Results 1 2 3 4 5
Providing Constructive Written Feedback 1 2 3 4 5
Access to Information, Protection of Privacy, Conflict of Interest, and Assessor Code of Conduct 1 2 3 4 5
Procedures around Requests for Reconsideration & Appeals of Decisions 1 2 3 4 5
TOTAL
Date:________________________________
ITA ASSESSOR
TRAINING
Pre-Course Study
Enhanced Challenge Pathway Project Final Report
March 2015
74
Assessor Definition
An ITA-registered Assessor is authorized to conduct competency-based assessments for specified trades or
occupations and to make professional judgments on whether evidence provided by candidates is sufficient to
meet approved certification standards.
Assessor eligibility and criteria are recommended by industry and approved by ITA. Assessors must be
selected based on the assessor eligibility criteria and, in order to conduct enhanced assessments, must be
registered with ITA and meet currency requirements. Hard copies of assessor files (including registration forms
and performance management documentation) must be kept with the Assessment Agency and must be
available for audit by ITA at any time.
Assessor currency requirements are defined in the relevant ITA Program Profile. The Assessment Agency is
responsible for annual verification of assessor currency. Records of assessor currency must be available for
audit by ITA at any time.
Enhanced Challenge Pathway Project Final Report
March 2015
75
ITA OPERATIONS POLICY PA3008
V.15.01.2013
POLICY: ITA-REGISTERED ASSESSORS
1.0 Scope This policy governs the registration and activities of individuals contracted or employed by ITA or by an ITA-approved
Assessment Agency. These candidates may be either completing apprentices or individuals challenging for certification.
2.0 Enhanced Assessment Enhanced Assessment describes the evaluation of competencies against occupational standards for the trade or
occupation through means other than writing a standardized examination.
3.0 Authorized Administration of Enhanced Assessment
Where completion of an ITA apprenticeship training program or certification challenge requires demonstration of
competency through a formal evaluation of underpinning knowledge and applied skills as well as, or in place of, a written
Certification examination, such evaluation must be conducted by an ITA-registered Assessor who meets requirements
prescribed in the relevant ITA program standard.
4.0 Eligibility for Assessor Registration
4.1 Registered Assessors must:
• have successfully completed ITA-approved Assessor training;
• have submitted, by way of criminal record check, evidence of no record of criminal offences related to the duties which the applicant would be required to perform as an assessor;
• have signed and agreed to follow ITA’s conflict of interest, confidentiality, and code of conduct rules; • have provided contact information for a minimum of 2 industry colleagues who support their application.
4.2 In addition, the relevant program standard defines Assessor requirements for each of the following:
� minimum amount, scope, and currency of experience in the trade or occupation; � whether or not supervisory experience is required, and if so, how much; � prerequisite certifications and endorsements; � number and type of industry references/recommendations; � minimum number of mentored assessments which must be successfully completed; � currency requirements for maintaining Active Assessor status, including:
o minimum annual number of assessments performed (may vary by program) o minimum annual participation in moderations, professional development or upgrading (may vary by
program) o satisfactory annual performance evaluation by Assessment Agency or ITA (mandatory for all
programs) o contract or employment with an Assessment Agency or ITA. (mandatory for all programs)
5.0 Maintenance of Registry
ITA will maintain an up-to-date registry of all Assessors. ITA-approved Assessment Agencies are responsible for
immediately notifying ITA of changes such as termination of employment or cancellation of a contract for an Assessor.
Enhanced Challenge Pathway Project Final Report
March 2015
76
6.0 Cancellation of Registration or Re-assignment from Active to Inactive Status
ITA reserves the right at its sole discretion to cancel an Assessor registration, or to re-assign an active registration to
inactive status with or without conditions.
Approved: 24 July.12
Updated 15.Jan.13
Enhanced Challenge Pathway Project Final Report
March 2015
77
Course Description
ITA’s Registered Assessor Course is built around 5 areas of assessor performance detailed in the Assessor Occupational
Analysis Chart. The course comprises pre-course study materials; 2 days of classroom instruction and activities including a
homework assignment after day 1; and a practicum consisting of the conduct of 2 mentored assessments. Practicum
assessments will be conducted under the supervision of an experienced ITA Registered Assessor and a representative of
ITA.
Course Content
Course content covers principles and rules of assessment and application of the principles and rules using occupation-
specific assessment tools and methods.
In-Class Training Outcomes By the end of the in-class training session you will be able to:
• Explain the certification system and roles of ITA, Assessors, Assessment Agencies, and certification candidates • Interpret the competency standard for a specific trade or occupational qualification • Define the qualities and characteristics of a competent assessor • Explain code of conduct, privacy, and conflict of interest requirements • Identify the parts and steps of the assessment process • Prepare candidates for assessment
Pre-Study Package
• Principles of assessment and rules of evidence
• Roles and attributes of an assessor
• Assessment methods and tools
• Operational Procedures
Day 1 In-class Training
• Demonstrations of Practical Assessment & Classroom Debrief & Moderation
Day 1 - 4 Shop Training & Debrief
• Conduct 2 mentored assessments under the supervision of an ITA experience assesor
Practice Assessment
Registration
Enhanced Challenge Pathway Project Final Report
March 2015
78
• Conduct assessments and make assessment decisions in accordance with the principles of assessment and rules of evidence
• Complete necessary paperwork • Provide candidates with helpful and objective feedback • Describe service standards and ITA-approved operational procedures
Practice Assessment Outcomes At the end of the Assessor Training course you will be able to demonstrate competence in the following assessor activities
required to become an ITA-registered assessor:
• Prepare for assessments
• Demonstrate effective communication and interpersonal skills
• Conduct assessments and making assessment decisions in accordance with ITA Enhanced Assessment
Operational Procedures
• Record and report assessment decisions and recommendations
• Apply continuous improvement to the assessment process.
Assessor Occupational Analysis Chart
The occupational analysis chart (OAC) is a document that identifies the distinct competencies required province-wide to achieve competency in an occupation. An OAC is not a training program or instructor guide; it informs the development of appropriate training and assessment methods for the occupation training program.
Occupation Description: An “Assessor” is authorized to conduct competency-based assessments for specified trades or occupations and to make professional judgments on whether evidence provided by a certification candidate is sufficient to meet approved certification standards. The Assessor documents assessment outcomes, provides feedback on assessment results, and manages assessment records in accordance with approved policies and procedures.
Line A Prepare for Assessment
A1 Prepare assessment resources, logistics, documentation and venue.
A2 Make professional judgments on the validity, authenticity, currency and sufficiency of evidence provided in
portfolio of evidence
A3 Inform and clarify assessment details with candidate
A4 Identify required accommodations for candidates with special needs for assessment activity
A5 Identify real and perceived conflict of interest scenarios
Line B
Demonstrate Effective Communication and Interpersonal Skills
B1 Use effective communication and interpersonal skills to develop and maintain a professional relationship
B2 Demonstrate use of plain language and terminology that is clear and concise
B3 Provide clear and constructive feedback
B4 Demonstrate active participation in moderation sessions using effective communication skills
Enhanced Challenge Pathway Project Final Report
March 2015
79
Line C
Conduct Assessments and Make Assessment Decisions
C1 Make professional judgments on the validity, authenticity, currency and sufficiency of evidence.
C2 Identify requirements for further evidence.
C3 Make judgments that are consistent with judgments made on similar evidence.
C4 Conduct assessment activities in accordance with principles of assessment and Privacy and Confidentiality
legislation.
Line D
Record and Report Assessment Decisions and Recommendations
D1 Prepare and submit records of assessment outcomes and supporting evidence in accordance with Operational
Procedures and Retention Policy
Line E
Apply continuous Improvement to the Assessment Process
E1 Identify strengths and weaknesses of the process and tools by applying the principles of assessment and rules
of evidence.
E2 Identify opportunities and recommendations to improve the assessment process and their own practice
Legislative and Policy Framework for Enhanced Assessment Legislative Level Section 8 of British Columbia’s Industry Training Authority Act [SBC 2003] Chapter 34 grants ITA the legislative authority and commensurate responsibility to develop criteria to award industry training credentials, and to establish standards and procedures for assessments. The foundation for issuance of credentials, therefore, is based on industry subject matter experts performing a thorough job analysis and recommending measurable indicators of what a competent practitioner should know and be able to do in the workplace to meet current occupational standards. These indicators are used to construct assessment tools, including written exams, and, where enhanced assessment forms part of the evaluation, portfolio evaluations, technical interviews, practical demonstrations of skill, and so forth. The summary description of trade or occupational certification requirements is found in the relevant ITA Program Profile. Policy Level ITA Board policy governs administration of certification exams, other types of evaluations of candidate knowledge and skill which fall under the classification of “Enhanced Assessment”. Administration of Enhanced Assessment is governed by ITA Operations Policy PA3008, ITA-Registered Assessors, which restricts delivery of enhanced assessments to ITA-Registered Assessors working under ITA supervision or the supervision of an ITA-Approved Assessment Agency. Operational Level By way of an Approved Assessment Agency Agreement between the Agency and ITA, roles and responsibilities Assessment Agencies are assigned to the parties. These administrative responsibilities must be carried out in accordance with ITA’s Enhanced Assessment Operational Procedures.
Enhanced Challenge Pathway Project Final Report
March 2015
80
ASSESSOR POLICY FRAMEWORK
ITA Operations Policy PA3008
Registered Assessor Policy
Industry Specific Assessor Eligibility Criteria
ITA Assessor Training Course
ITA Registered Assessor Assessor
Enhanced Challenge Pathway Project Final Report
March 2015
81
Declaration of Conflict of Interest and Agreement to Adhere to the Code of Conduct for ITA Registered Assessors
A: CONFLICT OF INTEREST
A conflict of interest exists in any situation in which you have a private or personal interest sufficient to influence or appear to
influence the objective exercise of your official duties as an assessor.
If you identify a real or perceived conflict of interest, ITA will re-assign the file. The sooner this issue is identified, the less
disruptive it is for the candidate. If you or the candidate discover or declare a conflict of interest part way through the
process (e.g. you didn’t recognize the name or information in a portfolio, but upon meeting the candidate recognize the
person and identify a potential conflict of interest), you must terminate the assessment at the time the conflict of interest is
identified and have the file re-assigned without prejudice.
Typical situations which create real or perceived conflicts of interest and are not allowed include assessing:
• Family members or close friends
• Current co-workers, supervisors, subordinates, business associates and contractors you work with
• Your current students or apprentices
• Persons to whom you owe an obligation
• Competitors
When in doubt, err on the side of caution. If you have a question about whether or not something constitutes a conflict of
interest, contact ITA for advice.
B: ASSESSOR’S CODE OF CONDUCT
The assessor code of conduct detailed below supports professionally responsible assessment practice and is a guide to all
assessors on the responsibilities of their work in line with best practice.
• You are expected to treat all challengers with courtesy and respect.
• It is important to maintain a calm demeanour during all communication with the challenger, whether this is in
person, in writing or over the phone. You should not become involved in any altercation or disagreement with
a challenger. Any issues that arise in relation to the challenger should be referred to the credentialing
authority.
• You must comply with any specific instructions from the credentialing authority in relation to the assessment
process.
• You must maintain strict confidentiality at all times relating to information that is disclosed to you or that you
Enhanced Challenge Pathway Project Final Report
March 2015
82
discover in the performance of your work. Definitive information about an assessment outcome should not be
given to applicants. This will be provided in writing to the applicant after the assessment event. Under no
circumstances should you talk about the assessment content to anyone other than those responsible for
administering the assessment process.
• Consumption of alcohol or improper use of drugs or other substances must not occur while undertaking your
assessment duties or in any activity that relates to your work as an assessor and must not affect your work
performance.
• You must not accept gifts from applicants, agents or other representatives. If you are personally offered any
inducements, you should refuse the offer promptly and report the matter to the credentialing authority.
• You should not seek or accept favours or gifts for services performed in connection with your work. Included in
this category are gifts in kind, such as free accommodation, travel or entertainment vouchers, whether for you
or members of your family. The general principle to be followed is that you should not seek or accept favours
or gifts from anyone who could benefit by influencing you.
• You must not unlawfully or unjustifiably discriminate against any individual in your practice as an assessor.
• You must not discriminate against any individual on the basis of gender, race or disability or allow their views
of the lifestyle, culture, belief, colour, gender, sexuality or age of others to prejudice your decisions as an
assessor.
• You should maintain your professional knowledge and skill and observe and keep up to date with laws and
professional codes of practice that affect your work.
INDUSTRY TRAINING AUTHORITY (ITA)
___________________________________
Signature
___________________________
Print Title and Name
______________________________
Date
)
)
)
)
)
)
)
)
)
)
__________________________
Signature
__________________________
Print Title and Name
___________________________
Date
Enhanced Challenge Pathway Project Final Report
March 2015
83
Enhanced Assessment Operational Procedures
March 2014
Enhanced Challenge Pathway Project Final Report
March 2015
84
Table of Contents
Overview ..................................................................................................................................................................... 84 1. Introduction ..................................................................................................................................................... 84 2. Process Overview ........................................................................................................................................... 85 3. Systems Roles and Responsibilities ............................................................................................................... 87
Administrative Process ................................................................................................................................................ 89 1. Information on the Enhanced Assessment Challenge Process ....................................................................... 89 2. Candidate Application, Registration and Approval to Challenge ..................................................................... 90 3. Assessment Activity ........................................................................................................................................ 91 4. Resource Requirements ................................................................................................................................. 95
Quality Assurance Process.......................................................................................................................................... 97 1. Administration ................................................................................................................................................. 97 2. Occupational Performance Standards and Assessment Tools ....................................................................... 97 3. Candidate Documentation and Record Keeping ............................................................................................. 98 4. Assessor Eligibility and Currency .................................................................................................................... 98 6. System Compliance ........................................................................................................................................ 99
Assessor Management Responsibilities ...................................................................................................................... 99 1. Eligibility Criteria .............................................................................................................................................. 99 2. Training ......................................................................................................................................................... 100 3. Registration ................................................................................................................................................... 101 4. Moderation .................................................................................................................................................... 101 5. Performance Review ..................................................................................................................................... 103
Appendix 1 – Glossary of Terms ..................................................................................................................................... Appendix 2 – Audit Checklist ........................................................................................................................................... Appendix 3 – Complaints and Appeals Procedure …………………………………………………………………………………
Overview
1. Introduction
Enhanced assessment refers to a structured evaluation process of certification candidates against program
standards that involves different assessment methodologies beyond the written multiple-choice examination.
This could include review of a portfolio of evidence, written knowledge assessment, technical interview and
practical assessment. It is recognized that each trade/occupation encompasses a specific set of skills and
competencies and therefore the assessment model will be tailored to meet the specific standard requirements
of each trade.
An enhanced assessment system involves performance criteria development, operational processes and quality
assurance. This document outlines the roles, responsibilities and processes associated with the management
of enhanced assessment of candidates and it provides the framework to ensure an efficient, high quality and
client-focused service irrespective of the methodology adopted.
These operational procedures are for the use of the ITA Operations department and the Assessment Agency
involved. It details activities that are related to the administration of applications, the management of
assessment activities and the business requirements of both organizations. It also outlines the audit criteria for
each stage of the administrative process. These criteria are detailed in an audit checklist at the end of each
section; a summary can be found in Appendix 2 for use throughout the quality assurance process.
Enhanced Challenge Pathway Project Final Report
March 2015
85
2. Process Overview
The following is a process map that details the operational flow for processing a candidate for certification in a
trade or occupation through means different than the standardized multiple choice written examination. The
reassessment process flow follows the same process map as the initial assessment.
Enhanced Challenge Pathway Project Final Report
March 2015
87
Enhanced Assessment Roles and Responsibilities
Below are the roles and responsibilities of the ITA and the Assessment Agency to operate the enhanced
assessment system. Note that in some cases ITA may also be responsible for the activities of the assessment
agency.
ITA
Roles and Responsibilities
Assessment Agency
Roles and Responsibilities
Application
• Provide general information to the candidate
• Receive and process applications and application fees
• Register candidate into ITA Direct Access (ITADA)
• Forward to Assessment Agency
• Receive and process reassessment applications and fees
• Schedule written exam
Results
• Process results submitted by Assessment Agency
• Issue results notification letter to candidates
• Issue Certificates to candidates who meet the certification standard
Records Management
• Store and Maintain electronic candidate files and assessment records.
• Process and respond to complaints and requests for review of assessor decisions
Assessor Training and Registration
• Train assessors
• Register assessors
• Facilitate moderation sessions
• Maintain updated assessor registry
Application
• Provide specific information to the candidate
• Provide advice to candidates on how to collect evidence to complete application forms and build portfolios, where applicable
• Perform initial review of eligibility, where applicable
• Inform ITA of eligibility to challenge, where applicable
• Set and collect fees for assessment and reassessment activities
• Support ITA in managing complaints in accordance with ITA’s complaint management procedure
• Coordinate reassessment
Assessment Activity
• Assign candidate files to assessors
• Schedule assessment activities and notify relevant parties
• Advise ITA of assessment results including feedback and gap training strategies, where applicable
• Manage required venues, including facilities, equipment and supplies as required
Records Management
• Securely store files and assessment results
• Input, catalogue and archive records
Assessor Management
Enhanced Challenge Pathway Project Final Report
March 2015
88
ITA
Roles and Responsibilities
Assessment Agency
Roles and Responsibilities
Quality Assurance
• Quality assure and audit administrative processes and assessor documentation
• Store, update and publish assessment tools with version control
• Maintain audit criteria for assessors
• Maintain audit standards for venues
• Approve Assessment Agencies who sign Assessment Agency Agreements
Strategy and Policy
• Manage disputes, complaints, requests for reconsideration and appeals
• Recruit and recommend assessors for assessor training and registration
• Assist with design and delivery of moderation sessions
• Manage ongoing performance of assessors
• Submit annual performance evaluations to ITA to confirm currency and verify active assessor status
Quality Assurance
• Recommend to ITA industry-specific criteria for assessment venues
• Recommend to ITA industry-specific eligibility criteria for assessors
• Manage audit requirements
Finance/Contracting
• Manage own financing and contractual arrangements
Enhanced Challenge Pathway Project Final Report
March 2015
89
Administrative Process
Candidate administration consists of a number of activities that are detailed in this section of the procedures. Highlighted
within the text of each section are the roles and responsibilities of each entity involved in the process. The following
stakeholders are directly involved in the administration process:
• ITA
• Assessment Agency
• Registered assessors
• Candidates
1. Information on the Enhanced Assessment Process
Detailed information about the purpose of the assessment, the assessment standards and the assessment
process are provided by the ITA for the candidates to review. General information is provided and kept up-to-
date by both ITA and the Assessment Agency via the website, including:
• The candidate information kit
• Candidate self-assessment tool
• ITA challenge application form
• Other promotional materials
Audit Checklist
Assessment Agency
� Materials in use are current and approved versions
a. Published candidate information kit
b. Published assessor guide
c. Assessment tools (version matched with ITA)
� Advise candidates how to collect evidence for applications and building portfolios, where applicable
a. Provide staff (including backup) and resources to aid potential candidates throughout the
application and portfolio-building process, where applicable
ITA
� Version control protocols are in place to ensure currency of:
a. Published Approved Performance Standards, where applicable
b. Program Outline/NOA
c. Assessment tools
d. Assessment Agency Agreements
e. Assessor registry
Enhanced Challenge Pathway Project Final Report
March 2015
90
2. Application, Registration and Eligibility approval
Application
An application package, which includes the application form, and if applicable, a self-assessment tool, and a
guideline for preparing the portfolio of evidence, is provided for the candidate. The candidate completes the
package and submits it, along with the challenger application fee, to the ITA. Fees are payable by cheque
(made payable to “Industry Training Authority”) or through the ITA PayPal site.
Registration
When an application is received, the ITA Customer Service creates an individual file in ITADA, following current
registration procedures. Once the candidate has been registered and the Individual Identification Number has
been issued, ITA will forward the application package to the Assessment Agency with any relevant supporting
documentation.
Candidate Approval
The Assessment Agency will evaluate the application and, when applicable, portfolio of evidence for eligibility
against the program standards. If this initial review finds that the individual is not eligible for assessment the
Assessment Agency will notify ITA. If the individual is eligible for assessment the Assessment Agency will
inform ITA and begin the assessment process, which includes assigning an assessor to the file and, scheduling
assessment activities and venues as required.
Audit Checklist
ITA
� Process for receiving and registering application (date stamp) and opening candidate file
a. Candidate information management system to track intake of applications (i.e., Excel spreadsheet,
ITADA notes)
b. Process for registering candidate in ITADA
c. Service standard of 5 business days from receipt of application to registration in ITADA
� Process for collecting application fess
� Communication of eligibility to candidate
a. Letters communicating acceptance and notification of assessment process next steps
Assessment Agency
� Process for receiving application (date stamp) and opening candidate file
Enhanced Challenge Pathway Project Final Report
March 2015
91
a. Candidate information management system to track intake of applications (i.e., Excel spreadsheet,
ITADA notes)
� Process followed for reviewing application for completeness and relevancy to the eligibility criteria for the
specific trade or occupation.
a. Identify the individual who is conducting the review and methodology
� Communication of eligibility to ITA, and notification to candidate of venue and potential assessment dates
a. Notifications of eligibility and scheduling to all parties
3. Assessment Activity
Approved assessment activities vary by trade/occupation and may involve one or more assessment
methods.
a. Assessment venues2
Assessment venues are identified by the Assessment Agency in accordance with industry-specific criteria.
Venues should be geographically distributed where possible across the province to meet the needs of the
industry and candidates.
Each assessment venue must:
• Meet industry requirements as defined in the program outline
• Facilitate the efficient use of assessment resources and tools
• Meet Occupational Health and Safety (OHS) requirements
b. Scheduling assessment activities
The assessment process involves one or more of the following assessment activities:
• Desktop review of challenge application
• Technical Interview
• Written Knowledge Assessment
• Practical Assessment
The scheduling of assessment activities is conducted by the Assessment Agency. For trades or
occupations in which the assessment process involves all of the above-mentioned activities; written
assessments and Technical Interviews should be scheduled a minimum of two weeks prior to practical
assessments to allow for the planning of the practical assessment.
The Assessment Agency will send a scheduling notification to the candidate a minimum of 10 business
days prior to the assessment activity.
c. Assessor conducts assessment activity
2 For program-specific venue requirements, refer to Program Outline Section 3. Facility Requirements
Enhanced Challenge Pathway Project Final Report
March 2015
92
The first evaluation is a desktop review of the candidate application and portfolio if one is submitted.
• The flow of the assessment activity is specific to each trade or occupation and detailed in the Assessor
Guide or approved assessment tools.
• The screening interview will be the first face-to-face contact with the candidate and must be conducted
in accordance with approved assessment tools. Audio recordings of technical interviews must be retained
on each candidate’s file.
Based on responses to screening interview questions the assessor may determine that the candidate is
attempting a certification for which he or she does not have the required experience and scope of trade or
occupation. The assessor may recommend challenge of a level or levels of the trade where the applicant is
ineligible to challenge the full certification. Alternatively, further assessment activity may be postponed until
the identified gaps are addressed. In this case, the assessor should advise the Assessment Agency of the
results and recommend an assessment plan. Assessment Agency will notify ITA and the candidate of
assessment options.
Practical assessments will comprise a pre-approved set of tasks determined by the standards set out in the
relevant Program Outline or OPS. Published program standards and any materials necessary for planning
or preparing for the assessment tasks are to be made available to all candidates.
Assessors must use tools provided by the Assessment Agency to conduct the assessment and document
results to ensure that the required performance standard has been demonstrated. Photographs and/or
video of the assessment may be taken to provide a clear record of the practical tasks and finished products
presented.
d. Results of assessment activity
Following the completion of the assessment activity, the assessor has 10 business days to complete and
electronically submit the assessment results package to the Assessment Agency, followed by submission
of hard copies of the documents and, as applicable, audio/video recordings.
The results package will vary depending on the methods of assessment applied, as appropriate may
include:
• Written Assessment Result
• Technical Interview Responses documented with notes by the assessor
• Practical Assessment results forms
• Assessment Summary
• Complete portfolio of evidence, including the self-assessment form and any additional evidence
gathered during the assessment activity, such as pictures, audio/video recordings, etc. as applicable
The information collected will be used to provide the notification of assessment outcomes.
Following receipt of the results package, the Assessment Agency will review the forms to ensure that they
accurately articulate the outcome of the assessment in sufficient detail to support the assessment decision
and to provide clarity to the candidate regarding the outcome.
Enhanced Challenge Pathway Project Final Report
March 2015
93
If during the assessment process the candidate provided insufficient evidence to meet the standards, the
Assessment Summary must include detailed information on the areas in which the candidate must provide
further evidence or gain further skills and/or knowledge to meet the certification standard. Assessors are
required to provide constructive feedback to the candidate on their performance and information on further
training, work experience or other ways the candidate can address his or her gaps in skills and knowledge.
4. i. Reporting and cataloguing results
Once the Assessment Agency confirms that the assessment activity reporting tool has been
completed accurately and the assessment results package is received from the assessors, they will
send a reply confirming receipt and the assessor should destroy any files on their personal
computer or in their possession that are related to the candidate.
All documentation submitted in the results package from the assessor must be catalogued and
submitted to ITA to be stored in ITA DirectAccess.
These documents provide details of the assessment and why a decision was reached The
Assessment Summary has the same evidentiary value as an examination answer sheet and must
be signed by the assessor, dated and stored according to ITA’s record retention policies and
procedures, along with any original evidence collected during the assessment process.
5. ii. Communication to candidate of outcomes
The Assessment Agency will submit the results to ITA to be stored in ITA DirectAccess.
Notification of Credential Not Achieved
Candidates who are not successful in meeting all of the standards requirements will not receive a
credential. ITA will issue a notification letter detailing the assessment outcome. This notification
will identify the gaps in evidence to meet the standards to achieve certification and will outline the
reassessment process.
The notification letter should include:
• Assessment summary
• Assessor comments and recommendations
Candidates will be advised that they may contact the Assessment Agency for advice on how to
address the gaps identified during the assessment process, including potential gap training options.
It is the candidate’s responsibility to address the identified gaps and to provide new evidence prior
to reassessment.
Notification of Credential Achieved
Candidates who met the certification standard will be issued the relevant credential in their
notification of results.
e. Complaints, requests for reconsideration, and appeals
Any complaint received from a candidate must be submitted in writing and all parties must be notified in
accordance with ITA’s Complaints and Appeals Management Procedure (See attached appendix 3.).
Enhanced Challenge Pathway Project Final Report
March 2015
94
Complaints should be acknowledge promptly by telephone, within 2 business days of the date of receipt
and followed up by email to advise the candidate how the complaint will be dealt with.
A candidate who believes that the assessment process was not followed correctly or that they were unfairly
treated may request reconsideration by the ITA CEO or appeal the assessment decision to the Industry
Training Authority Appeal Board. A request must be made in writing to ITA in accordance to ITA policy
PA3005 – Appeals/Dispute Resolution.
If a candidate requests reconsideration by the ITA CEO, a formal review of the assessment result, which
will be managed by the ITA Manager of Program Standards, will be performed only for competencies not
achieved. In no case will a unit of competency previously granted be revoked as a result of the review. The
review will be conducted by a panel of a minimum of three ITA registered assessors (reviewers), one of
whom will be nominated as chair by the review panel. If an assessor has been a part of the assessment
under review, he or she cannot be a member of the review panel. Each of the three reviewers will
independently review the documentation. The assessors will then jointly determine the outcome of the
review, either upholding or amending the results. Each of the three reviewers must submit an independent
review document. The chair will prepare and submit the final joint panel decision document
f. Reassessments
A reassessment may take place when a candidate is unsuccessful in providing sufficient evidence to meet the
program certification standards. Fees for Reassessment activities are set and collected by the Assessment
Agency.
Reassessment will follow the same administrative processes described in Sections 1, 2 and 3.
6. ii. Procedures for reassessment
All applicants for reassessment must follow the procedures listed below:
• There is a 30-day waiting period from the date of the results notification letter before applying
for reassessment
• Candidates applying for reassessment must submit new evidence. New evidence is defined as
evidence collected and dated after the previous assessment.
• The evidence provided must apply to gaps identified in the previous assessment result.
• An assessor will evaluate the new evidence to determine candidate readiness/eligibility for
reassessment and determine the appropriate assessment method (e.g. technical interview,
practical assessment, etc.)
• Reassessment must be conducted in accordance with the plan developed by the assessor. Audit Checklist
Assessment Agency
� Venue meets the industry standard for assessment activity
a. Venue meets specified industry criteria and OHS requirements
Enhanced Challenge Pathway Project Final Report
March 2015
95
b. Completed checklist for each venue to ensure they meet the requirements (with dates and venue
evaluation process identified)
c. Any formal venue complaints investigated in accordance with ITA Complaints Management
Procedures and documented with proposed improvement and evaluation plan where required
� Assessment agency makes any required contractual arrangements with venues which meet program-
specific venue criteria
a. Staff (including backup) and resources are in place to successfully execute the activity
b. Ensure all required contracts are in place and enforced
� Assessment Agency assigns candidate file to an assessor, schedules assessment activity and , secures
venue
a. Member of staff/backup and resources are in place to successfully execute the activity. NOTE:
Assessment date is based on venue schedules, candidate preferences and assessor availability.
b. Service standard of 10 business days from ITADA registration to scheduling of assessment
� Receive and review documentation of assessment results, upload results to ITADA and report results to
ITA
a. Member of staff /backup and other resources are in place to accept and review assessment results
b. Information management system to track receipt of completed assessment tools and any further
evidence gathered throughout the assessment activity
c. Service standard of 15 business days from assessment completion to reporting results to ITA
� Respond to Complaints in Accordance with ITA Procedures
a. Service standard of 2 business days for responding to complaints by phone and following up via
ITA
� ITA process assessment results and issue notifications; and where the certification standard is met, issue
the credential
a. Information management system to track results (i.e., Excel spreadsheet, ITADA notes)
b. Service standard of 5 business days from receipt of results to entering the competencies in ITADA
7. Resource Requirements
Assessment Agencies set and collect fees for Assessment and Reassessment activities; ITA does not set or
collect these fees. The following are costs and resources the Assessment Agency should consider when setting
assessment activity fees:
Enhanced Challenge Pathway Project Final Report
March 2015
96
Candidate Support, Application and Eligibility Reviews
• Assessment Agency Administrative staff time to aid in completion of application and portfolio
development where applicable
• Assessment Agency Administrative staff time to review application and process registration
• Assessor time to review candidate eligibility for assessment.
Arranging Assessments
• Assessment Agency administrative time and overhead to liaise and coordinate assessment, schedule
assessors, candidates and assessment venues.
• Assessment Agency administrative staff time and overhead to implement appropriate contractual
arrangements with assessors and assessment venues (including insurance issues)
• Assessment Agency administrative staff time and overhead to answer candidate queries on the
process
Conducting Assessments and Processing Results
• Assessor remuneration to perform all assessment activities (portfolio review, assessment plan
development, technical interview, written knowledge assessment, and practical assessment as
applicable) for both first-time candidates and reassessments
• Reimbursement of approved Assessor expenses (e.g. travel, accommodation, etc.)
• Applicable assessment venue costs to conduct technical interview, written assessment and practical
assessment (including facility rental, supplies, consumables, tools and equipment)
• Assessment Agency Administrative staff time and overhead to process results
Annual Compliance Audit
• Administrative staff time and overhead to plan and prepare for audit
• Administrative staff time and overhead to support audit while in progress
• Management staff time to review audit findings and agree on a plan of action for improvement Assessor Management
• Administrative staff time and overhead to complete and submit to ITA annual Assessor Performance
Evaluations and attestation of currency
• Assessor recruitment activities
• Participation in assessor moderation and professional development activities
Enhanced Challenge Pathway Project Final Report
March 2015
97
Quality Assurance Process
To ensure that enhanced assessment operates effectively and that all applicants are treated in a fair, valid and
consistent manner throughout the assessment process, a number of quality assurance requirements have been
established. The ITA Director of Assessment will manage the audit of the Assessment Agency to ensure compliance at
all stages of the assessment process. Note that the audit process could involve an external auditor.
Quality assurance arrangements cover:
• Systems administration for a quality assessment process
• Compliance with jurisdictional legislation and regulatory requirements
• Administrative and records management procedures
• Assessment judgments/decisions
• Access and equity
• Complaint procedures
The following stakeholders are directly involved in the quality assurance process:
• ITA Assessment and Program Standards departments
• Assessment Agency
• Compliance Auditor
1. Administration
The Assessment Agency must operate the enhanced assessment system in compliance with all ITA program
administration policies (PA3001 – 3006) and program policies (PP1000 – 1016). These can be found on ITA’s
website via the following link: http://www.itabc.ca/about-ita/policies .
The Assessment Agency is required to administer the enhanced assessment process in compliance with these
procedures, and in compliance with the ITA Approved Assessment Agency Agreement to which they are a
party. An ITA compliance audit will be conducted a minimum of once per fiscal year.
Should the audit show non-compliance with any ITA policy, non-compliance with these procedures or a
substantial breach of the Approved Assessment Agency Agreement the ITA will develop, in collaboration with
Assessment Agency, an action plan for improvement or, in extreme cases, may recommend suspension or
termination of the Agreement by ITA’s Director of Assessment. Action plans will be based on the
recommendations detailed in the audit report. If the recommendations have not been implemented within the
agreed timeline, ITA may, at its sole discretion, suspend or terminate the Approved Assessment Agreement and
cease accepting assessment results from the Agency.
2. Program Outlines, Occupational Performance Standards and Assessment Tools
Program Outlines, Approved Performance Standards and assessment tools for provincially recognized trades or
occupations must be developed according to the specifications outlined in the ITA Program Development
Process (PDP). New Performance Standards will be approved by the ITA Standards Review Committee (SRC).
Enhanced Challenge Pathway Project Final Report
March 2015
98
Approval criteria are detailed in the ITA PDP handbook. Revisions to these will be recommended to ITA SRC in
accordance with the agreed program standards review process.
Moderation sessions and assessor meetings should allow for continuous improvement the natural evolution of
the assessment tools. Assessment tools must be developed to Program Development Process specifications
and are auditable by ITA at any time to ensure compliance with development guidelines and appropriate use of
assessment tools.
3. Candidate Documentation and Record Keeping
Record-keeping systems are critical in the assessment process and for the efficient operation of the enhanced
assessment system.
Efficient record-keeping arrangements will be established to cover:
• Processing and administering candidate applications
• Managing the assessment sessions, e.g., scheduling assessors, candidates, venues etc.
• Recording and managing assessment outcomes and issuing results
• Recording and managing reassessment decisions and outcomes
To ensure that the enhanced assessment system operates efficiently, provincial standards in records retention
apply to any information gathered throughout the assessment process. Records will be maintained and securely
stored in ITA Direct Access.
Assessment records include the assessment tools, assessment outcomes and any communications with
candidates. Records include: notes regarding phone or in-person conversations; copies of electronic exchanges
by fax, e-mail, or text; written communications sent by mail, courier, or hand-delivered.
Records of the following will be kept for quality assurance purposes:
• Information provided to candidates
• Versions of assessment tools used
• Records of assessment times, dates, locations, and names of assessor(s) and candidate
• Documentation relating to the assessment decision/judgment
• Documents relating to the assessment process (e.g., comments, complaints, etc.)
• Assessment outcomes
• Records of feedback to candidate
• Credentials issued
Individual assessors will complete all assessment documentation for each assessment to assist with the
provision of feedback to candidates and to clarify issues that may arise in any appeal or reassessment process.
4. Assessor Eligibility and Currency
Assessor eligibility and criteria are recommended by industry and approved by ITA. Assessors must be selected
based on the assessor eligibility criteria and, in order to conduct enhanced assessments, must be registered
with ITA and meet currency requirements. Hard copies of assessor files (including registration forms and
Enhanced Challenge Pathway Project Final Report
March 2015
99
performance management documentation) must be kept with the Assessment Agency and must be available for
audit by ITA at any time.
Assessor currency requirements are program-specific, and defined in the assessor training application form for
each program. The Assessment Agency is responsible for annual verification of assessor currency. Records of
assessor currency must be available for audit by ITA at any time.
5. System Compliance
The quality assurance strategies of internal and external audit will be used to evaluate compliance with
enhanced assessment system requirements and to promote continuous improvement. Compliance and
continuous improvement will focus on:
• Adherence to policies and procedures
• Record-keeping requirements
• Analysis of candidate feedback and complaints
• Requirements of the agreements established with the assessment venues, the Assessment Agency
and the assessors
Internal Audit
Administration staff involved in the enhanced assessment system will be provided with checklists and guidelines
that enable them to undertake internal audits of relevant processes to ensure compliance against ITA’s
requirements and to identify areas to target improvement. Results of the audits are to be submitted to ITA
annually.
External Audit
To validate the outcomes of internal audits by the Assessment Agency, ITA will undertake its own regularly
scheduled annual audits and spot checks for compliance against enhanced assessment quality assurance
system requirements detailed in Appendix 2.
Assessor Management Responsibilities
Registered assessors who are trained in enhanced assessment methodology play a critical role in competency-based
assessment. After evaluating a range of evidence, they make the professional judgment as to whether the evidence
provided by the candidate meets the standards for certification.
To ensure the consistency and quality of the enhanced assessment process, the Assessment Agency, will recruit
assessors for ITA-approved training and facilitate the ongoing professional development of assessors through
participation in development and delivery of annual moderation activities.
1. Eligibility Criteria3
3 Program-specific eligibility criteria – see Program Outline
Enhanced Challenge Pathway Project Final Report
March 2015
100
Assessors will be eligible for registration once they have been evaluated on their ability to conduct
assessments, and the specific requirements of each of the following areas have been demonstrated:
• Prescribed years of work experience in the occupation or trade
• Prescribed trade or occupational certification(s)
• Prescribed endorsements (where applicable)
• Prescribed licenses (where applicable)
• Other Industry requirements such as supervisory experience
• Satisfactory criminal record check
• Signed code of conduct, conflict of interest and confidentiality agreements
Note: Criteria are program-specific, recommended by industry and approved by ITA through the Program
Development Process.
An eligible assessor must demonstrate competence in the following areas of the Occupational Analysis Chart:
• Communicates effectively
• Works effectively with others
• Makes professional judgments related to a candidate’s competence
• Follows code of conduct and maintains required confidentiality at all times
2. Training
Assessor training comprises ITA-approved in-class instruction and mentored practice assessments. Assessors
within the enhanced assessment system are required to have successfully completed an approved ITA
assessor training program covering the following key areas:
• Planning assessments
• Conducting assessments
• Making and properly documenting assessment decisions
• Providing support and feedback for assessment decisions
• Contributing to assessment quality assurance processes
• Communicating effectively
In addition, assessors must have current knowledge of industry practice, exhibit strong interpersonal and
communication skills, and exhibit knowledge of language, literacy and numeracy issues in the context of
workplace assessment. Assessors’ readiness to perform solo enhanced assessments will be determined based
on evidence of competence in performing tasks in the Occupational Analysis Chart demonstrated during
mentored assessments.
An Assessor Candidate is eligible for Registration as an ITA-registered Assessor when the candidate
demonstrates to the satisfaction of an experienced assessor supervising mentored assessments the ability to:
• Interpret units of competency
• Select tools to confirm evidence requirements where assessor discretion on tool selection is permitted
Enhanced Challenge Pathway Project Final Report
March 2015
101
• Use approved assessment methods, tools and procedures to gather and organize evidence
• Apply rules of evidence and principles of assessment to gather evidence
• Evaluate evidence against program performance standards to make a defensible professional
judgment on whether or not the standard is met
• Consult with peers on assessment outcomes and make professional judgment decisions
• Administer assessments according to approved procedures
• Organize and deliver reassessments (if required)
• Document rationale for standard met or not met, and return documentation in accordance within
approved service standards
• Discuss and implement reasonable adjustment strategies for candidates with special needs as
necessary
• Address OHS risks according to workplace and jurisdictional procedures
3. Registration
Upon successful completion of required assessor training, ITA will issue an Individual Identification Number to
an assessor. ITA will maintain a registry of assessors in ITA Direct Access that will include the following
information:
• Individual personal information
• Evidence of completion of the ITA assessor training program
• Records of conducted assessments
• Annual performance evaluation and verification of currency
4. Moderation
Moderation sessions are a venue for quality assurance of the standards, assessment process, assessment
tools, related documentation, assessment methodology and professional judgments, with the intent of
continuous improvement.
The purpose of a moderation session is to provide a consultative forum to:
• effectively compare and evaluate the assessment process, tools and evidence contributing to the
professional judgments made by a range of assessors against the same approved standards, and to
ensure reliable and consistent assessment outcomes
• effectively provide advice on continuous improvement of assessment tools, documentation, industry
standards, and assessment methodology
• provide advice, support, and quality assurance on the planning, implementation and reporting of
assessment activities, including issues with venues, schedules, tools, etc.
• consider issues arising from the assessment activities that may be relevant to group members and
their constituents. This may include discussion of borderline cases that may have challenged
assessors in their decision-making process.
Enhanced Challenge Pathway Project Final Report
March 2015
102
• maximize opportunities for information sharing about the assessors’ experience of competency-based
assessment
• contribute to the ongoing professional development of assessors and administrators involved in
competency-based assessments. This may include the creation of case studies to facilitate discussion
and assist in reaching consistency in the assessors’ professional judgments.
• develop action plans to improve the quality of assessments and tools
The Assessment Agency will participate in scheduling moderation meeting dates, organizing venues and
related items as required, coordinating with assessors, and helping to plan moderation session agendas and
activities with industry input. Moderation sessions must be conducted at least once per year.
The following roles should be fulfilled at a moderation session:
• Facilitator – ITA Program Standards Manager responsible for assessment standards
• Co-facilitator – Assessment Agency representative responsible for assessment standards
• To be considered an active and current assessor, all registered assessors and mentored assessors (in
training) are required to attend moderation sessions as required by their contractual obligations with
the Assessment Agency. ITA will only accept assessment results where assessments were conducted
by active assessors.
• Other parties may be invited to attend moderation sessionss at the request of the facilitator or the co-
facilitator to provide advice and assistance considered necessary. One of the functions of moderation
is to provide assessors the opportunity to make recommendations to ITA as needed for updates or
changes to assessment processes or standards such as tools. Other than assessors, those attending
meetings are considered to be ex officio and have no voting rights on recommendations for change to
the assessment standards.
ITA is responsible for updating session participants on any proposed or approved changes that relate to
standards, procedures, or documentation, and for updating session participants on the status of previously
submitted recommendations.
The outcomes of a validation process generally result in recommendations for improvements to assessment
tools, processes and outcomes. It is the responsibility of ITA to produce moderation reports documenting
outcomes. This may include recommendations for changes to the:
• Context and condition of the assessment
• Assessment tasks
• Administration processes
• Assessment criteria used for judging the quality of performance (e.g., the decision-making rules,
evidence requirements, etc.)
• Guidelines for making reasonable adjustments to the way that evidence of performance was gathered,
to ensure that the expected standard of performance specified with the unit of competency has not
been altered
• Recording and reporting results
Enhanced Challenge Pathway Project Final Report
March 2015
103
In addition to making recommendations for improvement to the assessment tools (as outlined above),
moderation may also include giving assessor feedback on past judgments to adjust future assessments.
5. Performance Review
One of the critical cornerstones of any competency-based assessment process is the quality and consistency of
assessors in making judgments, providing feedback and completing administrative duties.
The performance and professional judgment of all assessors needs to be monitored closely by the Assessment
Agency to ensure that standards are being met relating to adhering to roles and responsibilities, preparing
documentation and assessment records, and adhering to the code of conduct at all times. As assessors gain
experience and competency, this review is likely to become less frequent and fall into regular quality assurance
cycles and processes. There are many opportunities for the Assessment Agency to provide feedback to
assessors on their performance, both during initial training and while conducting assessment activities. The
most effective assessment of an assessor’s skills and competency is observation. The Assessment Agency is
responsible for recording any observations of assessor performance that are conducted, and is responsible for
sharing this feedback with the assessor and for providing any relevant information to ITA.
Assessment Agencies must submit to ITA an annual performance evaluation and attestation of currency for
every assessor.
Ongoing Quality Assurance and Currency
When assessors have been registered and are conducting assessments unsupervised by another assessor, the
role of the Assessment Agency becomes one of quality assurance. It is critical to ensure that feedback is given
and corrective action taken, should the following apply:
• Approved tools are not used or are used incorrectly
• Documents not completed correctly, such as providing too little or too much detail in the summary
• Use of unclear language, inappropriate language or comments
• Assessor does not complete assessments within the required time frames
• The assessment summary notes and the assessment decision are at odds or would not be
supportable upon request for reconsideration or appeal of a decision
• Complaints are filed about process or results
• Inconsistent quality of decision making among assessors
Assessors need to maintain currency of their industry knowledge as well as their assessment competence.
Currency means awareness of the up-to-date industry practice, the workplace and the job/role of the persons
being assessed against the Approved Performance Standards.
Assessors also need to ensure that their assessment competence is current. Self-assessment is an integral
strategy for an individual assessor’s professional development. It involves assessors reflecting on their own
skills, knowledge and attributes and determining where they are in relation to where they need to be and
participating in assessor moderation activities.
Enhanced Challenge Pathway Project Final Report
March 2015
104
In cases of conflict or complaints about the assessment process, the Assessment Agency and ITA will work
collaboratively to ensure that all concerns are addressed in a timely fashion (see Complaints and Appeals
section).
Audit Checklist
Assessment agency
� Assessors administering enhanced assessments meet program-specific eligibility criteria, as prescribed in
the assessor guide, and have completed the assessor training program
a. Assessor’s file contains registration form demonstrating eligibility requirements
b. Records demonstrating successful completion of the assessor training course and mentored
practicum assessments
� Assessor attendance at a minimum of one assessor moderation session within a fiscal year and record of
conducting a required number of assessments (varies by program)
a. Record of assessments and attendance at moderation sessions
� Feedback on assessors performance delivered to assessors on a regular basis
a. Record of assessor reviews and feedback – particularly following initial assessments (new
assessors)
� Annual performance review and attestation of currency completed and submitted to ITA
Enhanced Challenge Pathway Project Final Report
March 2015
105
Appendix 1 – Glossary of Terms
TERM DESCRIPTION
Assessment The process of reviewing and evaluating the skills, knowledge and qualifications of
a candidate against predetermined workplace performance standards. This process
could include: practical assessments, written assessments, technical interviews and
evaluation of prior work experience, learning or credentials.
Assessment Agency An organization which has entered into an Approved Assessment Agency
Agreement with ITA to manage enhanced assessment activity for specified trades
or occupations in accordance with ITA’s Enhanced Assessment Operational
Procedures.
Assessment method Technique used to gather different types of evidence, which may include
questioning, observation, third-party reports, interviews, simulations and portfolios.
Assessment resources Any materials that assist in any part of the assessment process. These may include
information for the candidate or assessor, assessment tools, or any other resources
for the quality assurance arrangements of the assessment system.
Assessment plan The approach to assessment and evidence gathering used by the assessor,
including the assessment process, methods and assessment tools.
Assessment process The series of steps that the candidate undertakes within the application,
assessment, recording and reporting cycle.
Assessment system A controlled and ordered process designed to ensure that assessment decisions
made in relation to many candidates, by many assessors, in many situations, are
consistent, fair, valid and reliable.
Assessment tool Contains both the instrument and the procedures for gathering and interpreting
evidence:
• Instrument – specific questions or activity developed from the selected
assessment methods
• Procedures – the information given to the candidate and/or assessor on
conditions under which the assessment should be conducted and recorded
Assessor An individual who has successfully completed an ITA Assessor Training program
and is registered with ITA to evaluate competencies of individuals seeking
certification in specified trades. ITA Authorization to perform assessments is
restricted to assessors who meet program-specific currency requirements.
Certificate of Qualification The credential awarded by ITA to individuals who have successfully completed the
requirements of an Accredited Industry Training Program or a Recognized Industry
Training Program.
Enhanced Challenge Pathway Project Final Report
March 2015
106
TERM DESCRIPTION
Candidate Any individual who has amassed sufficient practical work experience to meet the
established ITA trade-specific criteria to attempt the certification in the trade and
who has obtained their skills and knowledge outside the B.C. apprenticeship
system.
Competency A competency is the smallest unit of job activity that results in a meaningful
outcome (product, service or decision). Competencies describe what the individual
is able to do. They must be observable and measurable. Competencies are
standards requiring the approval of ITA’s Standards Review Committee.
Credential Issued by ITA, indicating that the recipient has met completion requirements for the
program and therefore has demonstrated the competencies, skills and knowledge
associated with the occupation. A credential can be an awarded certification or an
advanced credential. Awarded certifications and advanced credentials are
standards requiring the approval of ITA’s Standards Review Committee.
Dimensions of
competence
The concept of competence includes all aspects of work performance, not just
narrow task skills. The four dimensions of competence are task skills, task
management skills, contingency management skills and job/role environment skills.
Element The basic building blocks of the unit of competency that describe the tasks that
make up the broader function or job, described by the unit.
Enhanced Assessment A structured evaluation process of candidates’ knowledge, skills and attributes
against program standards that involves different assessment methodologies
beyond the written multiple choice exam. e.g., practical assessment, technical
interview, or portfolio review.
Evidence Information that is used to demonstrate competence against agreed standards.
Quality evidence is valid, authentic, sufficient and current.
Gap assessment Assessment of areas of competence in which the candidate was unable to provide
sufficient evidence to meet certification standards in a previous assessment.
Holistic assessment An approach to assessment that covers the clustering of multiple units/elements
from relevant Occupational Performance Standards and focuses on the assessment
of a ‘whole of job’ role or function that draws on a number of units of competency.
Individual Identification
Number
The unique 6-digit identifier for an individual registered in ITADA. All apprenticeship
and certification-related records for the individual bear this identifier. Formerly called
Trade Worker Identification Number (TWID) or Registration Identification Number.
Industry Training
Authority (ITA)
The B.C. Crown corporation that manages B.C.’s Industry Training, Apprenticeship
and Trades Credentialing system. ITA is governed by the Industry Training
Authority Act [SBC 2003] Chapter 34.
Enhanced Challenge Pathway Project Final Report
March 2015
107
TERM DESCRIPTION
Industry Training
Organization (ITO)
A not-for-profit legal entity incorporated under the provincial Societies Act that
contracts with ITA under an Enterprise Partnership Agreement to take lead
responsibility for industry training within a particular sector. Established and
operated by industry and with a province-wide mandate, ITOs are financially
supported by industry and through Enterprise Partnership Agreements with ITA.
ITA Direct Access (ITADA) The electronic management information system used to house all of the training
records for individuals completing industry training programs and for individuals who
receive certification.
Moderation A process that involves assessors in discussing and reaching agreement about
assessment processes and outcomes in a particular industry.
National Occupational
Analysis (NOA)
A detailed listing of the competencies required for an occupation for an Accredited
Industry Training Program that is nationally developed under the Interprovincial Red
Seal Program. The NOA organizes competencies in blocks, tasks and sub-tasks.
B.C. Industry Training Programs are designed around the Occupational Analysis
Chart (OAC), often derived from the NOA. The OAC organizes competencies as
general areas of competency (GAC) and competencies. Competencies are
standards requiring the approval of ITA’s Standards Review Committee.
Occupational Performance
Standards (OPS)
Statements developed by Industry and approved by ITA that describe effective
performance in the workplace. These are sometimes expressed as units of
competency that describe an aspect of work in a particular trade, what is to be
assessed and the required level of performance.
Performance criteria Evaluative statements that specify what is to be assessed and the required level of
performance.
Portfolio of evidence A package consisting of primary and third-party evidence from prior experience and
training submitted to demonstrate competence.
Practical Assessment An observation of a candidate’s performance, skills or knowledge to the standard
expected in the workplace, as expressed in the relevant industry program
standards.
Program Outline A communications document that represents the full scope of knowledge and skills
required to competently perform an occupation anywhere in B.C. It defines the
knowledge and skills to be gained through training, but does not contain
instructional procedures.
Program Profile A communications document that describes all of the credentialing pathways,
credentialing model, completion requirements and credentials associated with a
program.
Enhanced Challenge Pathway Project Final Report
March 2015
108
TERM DESCRIPTION
Quality Assurance Quality assurance involves a planned and systematic auditing process of ensuring
that the requirements of the assessment system, Occupational Performance
Standards and any other criteria are applied in a consistent manner. Quality
assurance mechanisms or procedures are an integral part of the assessment
system.
Reconsideration Reconsideration by the CEO of the results of an assessment review is available
upon request by a candidate.
Records of assessment The information of assessment outcomes that is retained by the organization
responsible for issuing the transcript of results or credentials.
Red Seal Program A standardized national endorsement for a specific trade that enables
interprovincial and inter-territorial mobility. On successful assessment, a nationally
recognized Red Seal endorsement is affixed to the provincial Certificate of
Qualification.
Reporting assessment
outcomes
The different ways in which the outcomes of assessment processes are reported to
the person being assessed, employers and other appropriate personnel or
stakeholders.
Review Review of an assessment decision by a different Assessor is available upon request
by a candidate.
Risk management The systematic application of management policies, procedures and practices to
the tasks of identifying, analyzing, evaluating, treating and monitoring risk.
Self-assessment A process that allows candidates to collect and provide evidence on their own
performances against the Occupational Performance Standards.
Simulation A form of evidence gathering that involves the candidate completing or dealing with
a task, activity or problem in an off-the-job situation that replicates the workplace
context.
Stakeholders Assessment Agencies, Industry training organizations, industry employers,
sponsors of apprentices, training institutions, assessors and the Canadian Council
of Directors of Apprenticeship.
Standards ITA is responsible for establishing the standards to which apprentices and
tradespeople must perform in order to receive ITA certification in an occupation or
trade. These standards, often referred to as program standards, include
credentialing, assessment and achievement criteria designed to ensure that B.C.’s
candidates for assessment (whether apprentices or challengers) can demonstrate
their performance at levels established by national and provincial industry groups.
All program standards require the approval of ITA’s Standards Review Committee.
Enhanced Challenge Pathway Project Final Report
March 2015
109
TERM DESCRIPTION
Standards Review
Committee (SRC)
ITA Committee responsible for the oversight of standards, as part of the ITA’s
regulatory responsibility. The Standards Review Committee is also responsible for
ensuring that the ITA processes for establishing, developing and managing new
and existing programs and assessment methods are followed by industry partners.
Unit of competency Describes an aspect of work in a particular occupation or industry and used as a
benchmark for assessment of competence.
Validation Involves reviewing, comparing and evaluating assessment processes, tools and
evidence contributing to judgments made by a range of assessors against the same
standards. Validation strategies may be internal processes with stakeholder
involvement or external validations with other providers and/or stakeholders.
Enhanced Challenge Pathway Project Final Report
March 2015
110
Quality Assurance of Enhanced Assessment for Candidates
Audit Checklist
ASSESSMENT AGENCY
� Materials are kept up to date
a. Published challenger information kit
b. Published assessor guide
c. Assessment tools currently in use (current version matched with ITA)
d. Occupational Performance Standards currently in use (current version matched with ITA)
Administrative Process in place4
� Provide advice to candidates on how to collect evidence and build portfolio
a. Process identifying a staff person/back-up and other resources to aid candidates throughout the application and portfolio building process
� Process for receiving application (date stamp) and candidate file opened
� Candidate information management system to track intake of applications (i.e. excel spreadsheet, ITADA
notes)
� Inform ITA to register candidate in ITADA [ITA receiving initial application and fee, then forwarding to
Assessment Agency]
� Service standard of 5 business days from receipt of application to submission of information to ITA for
registration
� Process for reviewing application for completeness and relevancy to the applied program and assignment to
assessor for portfolio review
o Identify staff person who is conducting the review and methodology
� Communication of eligibility for assessment or request for further action from candidate, and where eligible,
scheduling of assessment dates and venues.
o Letters or assessment voucher communicating acceptance and notification of assessment
Scheduling
� Venue meets the industry standard for assessment activity
o Up to date venue industry requirements and OHS
o Completed checklist for each venue to ensure they meet the requirements (with dates and venue
evaluation process identified)
4 Appropriate notes must be captured in ITADA documenting every step of the process
Enhanced Challenge Pathway Project Final Report
March 2015
111
o Formal venue complaints investigated and documented in accordance with ITA Operational
Procedures
� Assessment Agency schedules assessment activity and secures venue etc. and assigns candidate to an assessor
o staff person/back-up and other resources are in place to successfully execute the activity o NOTE Assessment date is based on venue schedules and candidate availability and where
possible, candidate preferences
o Service standard of 5 business days from ITADA registration to assessment scheduled
� Assessment Agency makes contractual arrangement with venue o staff person/back-up and other resources are in place to successfully execute the activity o Ensure contracts are in place
� Receive and review documentation of assessment results, upload results to ITADA, and report results to ITA
o staff person/back-up and other resources are in place to accept and review assessment results
o Information management system to track receipt of completed assessment tools and any further
evidence gathered throughout the assessment activity
o Service standard of 15 business days from assessment completion to reporting results to ITA
� Document complaints o Service Standard in accordance with ITA Enhanced Assessment Operational Procedures
INDUSTRY TRAINING AUTHORITY
� Materials are kept up to date
b. Published Occupational Performance Standards
c. Assessment tools currently in use
d. Version control protocols are in place
� Process for registering application (date stamp) and candidate file opened
a. Candidate information management system to track intake of applications (i.e. excel spreadsheet,
ITADA notes)
b. Process for registering candidate in ITADA
c. Service standard of 5 business days from notification of eligibility to challenge to registration
� ITA process assessment results and issue notification and/or credential a. Information management system to track results (i.e. excel spreadsheet, ITADA)
b. Service standard of 5 business days from receipt of results to entering the competencies in ITADA
� Document complaints and follow approved ITA procedures a. Service Standard for responding to complaints in accordance with Enhanced Assessment
Operational Procedures
Enhanced Challenge Pathway Project Final Report
March 2015
112
Appendix 3 – ITA Enhanced Assessment Complaints and Appeals Procedure
ITA Enhanced Assessment Complaints and Appeals Procedure
BACKGROUND
Part 4, Section 11 of the Industry Training Authority Act, and ITA Operations Policy PA3005 – Appeals / Dispute Resolution
govern Reconsiderations and Appeals of ITA decisions. Special administrative procedures apply to complaints related to
enhanced assessment and to requests for CEO reconsideration of decisions made by ITA-registered assessors. These
administrative procedures are documented in ITA’s Operational Procedures for the Enhanced Assessment of Challengers.
This Enhanced Assessment Complaints and Appeals Procedure defines the roles and interaction between ITA, the
Approved Assessment Agency or Industry Training Organization (AAA / ITO).
PRINCIPLE
ITA and the AAA / ITO have a mutual commitment to provide professional and timely service to any individual who raises a
concern about service, assessment, or procedures. The goal is to verify and address all issues brought forward by a
complainant in a timely and customer-centric manner, with escalation to senior management as appropriate.
RATIONALE
Information sharing and timely communication amongst all parties to the complaint is critical for fair and timely resolution of
complaints, so it is important that there is an agreed-to protocol and service standard for information sharing,
documentation, response, and resolution of complaints.
SCOPE
This procedure applies to two types of complaints. One relates to assessment processes and fairness issues; and the other
relates to competencies not granted. It is possible for either one or both types of complaint to arise from a single
assessment.
Under these Procedures, if a complaint cannot be resolved, and a challenger requests CEO reconsideration of an assessor
decision not to grant one or more competencies, a formal review of the assessment result is performed only for
competencies not achieved. In no case will a competency previously granted be revoked as a result of the review. The
review is conducted by a panel of a minimum of three ITA-registered or HRSDC-registered assessors (reviewers), one of
whom is nominated as chair by the review panel. No assessor involved in the assessment under review may serve as a
member of the review panel. Each of the three reviewers independently reviews documentation from the contested
assessment. The assessors then jointly determine the outcome of the review, making a recommendation to the CEO to
either uphold or amend the results. Each reviewer submits an independent review document, and the chair prepares and
submits the final joint panel recommendation to the CEO.
Complaints Related to Assessment Process and Fairness Complaints Related to Competencies Not Granted
Process and Fairness complaints include, but are not
limited to issues concerning:
� Venues � Equipment and supplies � Assessor professionalism
Complaints regarding competencies not granted are
subject to formal appeal to the ITA Appeal Board in
accordance with Part 4, Section 11 of the Industry
Training Authority Act, and ITA Operations Policy
Enhanced Challenge Pathway Project Final Report
March 2015
113
� Confidentiality � Discrimination � Safety � Time and scheduling
PA3005, Appeals / Dispute Resolution.
COMPLAINT REVIEW TEAM
The following individuals serve as the Enhanced Assessment Complaint Review Team (CRT):
� ITA Manager of Policy and Research � ITA Director of Assessment � ITA Program Standards Manager responsibility for oversight of the program � Program or Customer Service Manager at the relevant AAA / ITO
All members of the CRT need to be aware of the complaint in order to work together to respond and resolve the complaint in
an appropriate and timely manner. Therefore all complaints or issues raised by a challenger will be logged by the receiving
party and communicated by e-mail or telephone to immediately inform the other parties who make up the CRT) of the
receipt and nature of the complaint. A detailed record of the complaint will be recorded via “Notes” on ITA DirectAccess
within 2 business days of receipt.
Complaints may be able to be resolved relatively quickly by agreement of the parties; but where this is not the case, a
formal procedure for managing escalation is required. From the time a complaint is received until it is resolved, there are
three management levels for resolution. These are:
1) resolution by agreement of the CRT and complainant; 2) review and decision by the COO; and 3) final determination by the CEO.
1
Enhanced Challenge Pathway Project Final Report
March 2015
114
ESCALATION MATRIX
Complaint Received
Level 1 - Initial Complaint
Level 2 - COO Review
Level 3 - CEO Review
Complainant contacts
ITA or AAA / ITO
Complaint immediately
referred to CRT made up of
ITA Manager of Policy and
Research, ITA Program
Standards Manager, ITA
Director of Assessment, and
Program Mgr at relevant AAA
/ ITO for review and decision.
If not resolved, move to level
2.
Complaint referred to ITA COO
for review and decision.
If not resolved, move to level 3.
ITA CEO for final
determination.2
1 NOTE: If complainant is appealing a decision regarding units of competency not granted then the complainant will
be provided with the procedure as to how to file an appeal with the ITA Appeal Board. 2As Above: competency not granted is subject to formal appeal to the ITA Appeal Board.
PROCEDURE
Level 1 – Initial Complaint Management By CRT
1. Complaints received by ITA or the AAA /ITO must be logged in ITA DirectAccess, and must be immediately referred to the CRT. (Documentation received by ITA will be put into ITA Service Requests (aka.Biz) which will automatically populate ITA DirectAccess.)
a. ITA Supervisor, Quality Service to assign a staff member to scan and save all documents into ITA DirectAccess and distribute via email to Complaint Review Team.
b. PSM organize a conference call for joint review and resolution of the complaint. Review includes, but is not limited to:
• Agreeing on method of verifying complaint • Review of all documentation/evidence • Development of a written action plan
c. PSM will contact the complainant within 2 business days, by phone or e-mail to: • acknowledge the complaint has been received and will be actively reviewed by the CRT • gain additional information regarding events leading up to the complaint • confirm what resolution is being sought • provide a single point of contact for any further communication on this issue • describe next steps and approximate timelines for the complaint review process • In accordance with the Operational Procedures for the Enhanced Assessment of
Challengers, If the resolution being sought by the complainant is regarding competency not granted, then an assessment review panel of 3 independent assessors who were not involved with the initial assessment will be engaged to review the file and make a judgment regarding competency for each contested block or unit of competency.
i. All assessment documentation, including the report from the assessment review panel, to be filed and forwarded to the CEO to assist in the CEO’s decision regarding a request for reconsideration.
Enhanced Challenge Pathway Project Final Report
March 2015
115
ii. inform the complainant about the review process and that an initial response will be sent within approximately 30 business days
d. If the contact referenced in section c. above was by telephone, PSM to send follow-up e-mail, courier, or registered letter summarizing the conversation and next steps. This will be copied to the CRTand saved in ITA DirectAccess.
e. CRT to meet by phone or face-to-face within 5 business days of being notified of the complaint to discuss the complaint to determine a plan to communicate back to the complainant and to resolve the issue. The agreed-upon plan must be logged in ITA DirectAccess.
2. The PSM will coordinate all further activity in consultation with the Manager, Policy and Research, to ensure: a. Follow-up actions are completed b. A joint resolution of the CRT is agreed-upon c. A formal response letter is approved by the CRT
3. The PSM will communicate the decision to the complainant verbally, and send the formal response letter by e-mail,
courier, or registered letter.
4. All communications and documentation of any kind must be retained for evidence (including informal e-mails, notes regarding telephone conversations, letters, and records of any kind, including audio tapes and photographs of assessment activities or outcomes.
5. The owner of the Complaint procedure will be the Manager, Policy Research who will be responsible for ensuring the process is adhered to, leading the CRT meetings and coordinating records of complaints received and actions taken.
Level 2 – COO Review
1. If the complainant is not satisfied with the decision of the CRT the complaint will be referred to the ITA COO for review and decision.
2. All documentation will be provided to the COO for review.
Level 3 – CEO Review
1. If the complainant is not satisfied with the decision of the COO, the complaint will be referred to the ITA CEO for reconsideration and decision.
a. All documentation will be provided to the CEO for review The CEO decision is final except as noted above where competencies not granted are subject to formal appeal to
the ITA Appeal Board.