International Vocational Educational Organization (IVETA) Conference - December 3 & 4, 2013, Las...
description
Transcript of International Vocational Educational Organization (IVETA) Conference - December 3 & 4, 2013, Las...
International Vocational Educational Organization (IVETA) Conference - December 3 & 4, 2013, Las Vegas, Nevada, USA
QUALITY ASSURANCE IN TECHNICAL VOCATIONAL EDUCATION & TRAINING (TVET)
HEART TRUST/NTA (JAMAICA)
Approaches toMonitoring &
Evaluating TVET
ProgrammesPresenter: J Wallder
Jamaica • An island nation surrounded by the Caribbean Sea. • Located 90 miles south of Cuba; 118 miles west of Haiti.• 145 miles in length, up to 50 miles in width and 4,243 sq. miles
in area. • Population – approx. 2.7 million.
TVET – CARIBBEAN REGION IMPERATIVES
Dec. 2013 2
United Nations Educational, Scientific, and Cultural Organization (UNESCO) & International Labour Organization (ILO), 2002:
Those aspects of educational process involving the study of technologies and related sciences (including general education) and the acquisition of practical skills, attitudes, understanding and knowledge relating to occupations in various sectors of economic life.
TECHNICAL VOCATIONAL EDUCATION & TRAINING (TVET) – DEFINITION
Dec. 2013 3
Caribbean Community (CARICOM) Council for Human and Social Development (COHSOD), at its 19th Meeting, April 2010, established plans to review the 1990 CARICOM Regional Strategy for TVET to consider new realities of the world of work.
CARICOM Secretariat stated “it has become appropriate at this time to review the strategy primarily because global trends have indicated a need for a knowledge-based and skilled labour market”.
TVET – CARIBBEAN REGION IMPERATIVES
Dec. 2013 4
Implications:
Education & training – more relevant to meet market needs
Emphasis on lifelong learning/continuous retooling
Renewed strategies to be aimed at workers at all levels of operations – operational, managers, professionals, technical experts.
TVET – CARIBBEAN REGION IMPERATIVES
Dec. 2013 5
Established as a statutory body under the HEART Act in 1982 to govern the development and delivery of TVET in Jamaica.
The Human Employment and Resource Training Trust/National Training Agency (HEART Trust/NTA)
THE HEART TRUST/NTA - JAMAICA
VISION STATEMENT: The HEART Trust/NTA is committed to the systematic design, development and delivery of an integrated, flexible and responsive TVET system, enabling a productive workforce for national priorities and global competitiveness
Dec. 2013 6
Key Drivers of the HEART Trust/NTA
UNESCOILO
Board of Directors
Govt. of Jamaica
(2030 Plan)
Key Ministries: Education,
Labour
CARICOM
THE HEART TRUST/NTA
Dec. 2013 7
• Core business - training and certification of Jamaican citizens
for employment, while regulating and setting standards for
the TVET system in Jamaica.
• 30 Training institutions – current focus towards tertiary level
status; integration of levels 1 & 2 programmes into secondary
schools.
• Partnering training institutions (CTIs: 79);
• (Private Providers : 12)
• 2010 – 2013: Approx. 100 skill areas offered (levels 1 – 5)
Average # certified per year: 18,300
THE HEART TRUST/NTA - JAMAICA
Dec. 2013 8
Key Objectives:• Framework to guide M&E activities• Performance standards for TVET design, operations and
delivery• Monitoring and evaluation data collection and feedback
mechanisms/tools to facilitate strengthening of processes• Monitoring and evaluation audits (across the TVET system)
against the relevant standards towards programme accreditation and quality improvement
• Evaluations to ascertain programme effectiveness, impact and customer satisfaction within the TVET system • Provision of timely information to inform strategic decisions.
ORGANIZATIONAL RESTRUCTURING APRIL 2013 - NEW MONITORING & EVALUATION DEPARTMENT ESTABLISHED
Dec. 2013 9
Current Internal Quality Systems; Org. Commitment, Etc.
External Quality Assurance Agencies
Global Trends Re M&E
Best Model/Approach to Monitor & Evaluate Jamaica’s TVET System
SETTING UP THE M&E SYSTEM
FACTORS TAKEN INTO ACCOUNT
Dec. 2013 10
HEART TRUST/NTA – INTERNAL QUALITY SYSTEMS
The National Council for Technical Vocational Education & Training (NCTVET)• Responsible for quality assuring the TVET system • Established quality framework for developing competency standards,
along with guidelines/procedures for delivery and assessment activities• Accredit TVET training providers and training programmes
Training Institutions• Required to establish comprehensive Policy and Procedures Manuals
(PPM)• Management/administrative structures• Internal Quality Assurance Committees – monitors activities/alignment
with PPM
The Organization • Guided by Act of Parliament/sound mandate; other key drivers• Planning system premised on Balanced Scorecard methodology• Comprehensive structures to support TVET programmes
Dec. 2013 11
HEART Trust/NTA – Internal Quality SystemsStandards used by the NCTVET • Quality Assurance & Control/Governance• Staff Resources• Physical Resources• Client Services & Activities• Training• Assessment• Programme Evaluation
Dec. 2013 12
HEART TRUST/NTA – EXTERNAL QUALITY SYSTEMS
Quality standards used by the UCJ:• Governance and mission• Academic programmes being offered• Admission policies and procedures• Staffing• Student assessment• Student support services• Physical accommodation• Library/resource centre• Laboratory/computer facilities• Finances
The University Council of Jamaica (UCJ) • statutory body – established by the University Council of Jamaica
Act, 1987. • Primary focus – tertiary educational institutions; Institutional
registration and programme accreditation
Dec. 2013 13
Globalization - increased pressures on governments and organizations around the world to be more responsive to the demands of internal and external stakeholders for:• Good governance• Accountability • Transparency• Greater development effectiveness• Delivery of tangible results.
Translated into attendant need for results-based monitoring and evaluation of policies, programmes and projects.
MONITORING & EVALUATION – GLOBAL TRENDS
Dec. 2013 14
MONITORING & EVALUATION – GLOBAL TRENDSDefinitions - Organization for European Co-operation and Development (OECD)
A continuous function that uses the systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress, and achievement of objectives in the use of allocated funds.
Monitoring
The systematic and objective assessment of an ongoing or completed project, programme, or policy, including its design, implementation, and results. The aim is to determine the relevance and fulfillment of objectives, development efficiency, effectiveness, impact, and sustainability.
Evaluation
Dec. 2013 15
Monitoring & Evaluation – Global Trends
Monitoring & Evaluation - Two Definitions Distinct but Complementary
Monitoring Gives Information on Where a Policy, Programme, or Project is at Any Given Time:• Clarifies programme
objectives• Routinely collects data on
indicators; compares actual results with targets
• Reports progress to management and alerts them to potential problems
Monitoring
Evaluation Gives Evidence of Why Targets and Outcomes are/not being Achieved:• Analyzes why intended
results were or were not achieved
• Explores unintended results • Provides lessons, highlights
significant accomplishments or programme potential, and offers recommendations for improvement
Evaluation
Dec. 2013 16
Training delivery of TVET programmes; executed through all HEART training institutions, partnering/Community Training Institutions, secondary school system, private providers.
Must ensure standardization of inputs, processes, outputs, outcomes and impact across training entities.
Rationale for a Monitoring and Evaluation Framework for Jamaica’s TVET System
Why Monitor
& Evaluate?
Dec. 2013 17
The NCTVET (and other accreditation/certification bodies) - QA for accreditation.
M&E - critical support function – seeks to monitor/track progress of delivery entities.
Ensuring systems and processes are established against quality/accreditation standards; In state of readiness for accreditation/verification activities.
RATIONALE FOR A MONITORING AND EVALUATION FRAMEWORK FOR JAMAICA’S TVET SYSTEM
Why Monitor
& Evaluate?
Dec. 2013 18
Monitoring and Evaluation Framework established, guided by a Systems Approach, to facilitate effective monitoring and evaluation activities of Jamaica’s
TVET system.
A SYSTEMS APPROACH TO M&EMonitoring and Evaluation Framework – A Process/Results Model
INPUT PROCESS OUTPUT & OUTCOME
IMPACT
QA & CONTROL; M&E
Dec. 2013 19
SYSTEMS APPROACH TO MONITORING AND EVALUATION
Inputs
Financial, human, and material resources used for implementation of TVET programmes
Technical Expertise, Approved Curricula,Equipment, Funds
Processes
Systems established; work performed
Strategic planning, Organization structure, Roles/responsibilities established; quality assurance systems; Training programmes
implemented
Outputs The products and services that result from the
TVET system
Number of persons trained and certified
Dec. 201320
SYSTEMS APPROACH TO MONITORING AND EVALUATION
Outcomes
The likely or achieved short-term and medium-term effects or changes arising from persons
trained and certifiedIncreased skilled labour force
More persons obtaining employment
Impact
The long-term results of the implementation of TVET programmes
Persons enjoying improved standard of living
Improved national productivityPositive impact on GDP
Dec. 201321
M&E aligned with goals/objectives of the Organization Captures context of the TVET system Tracks progress on inputs, processes, outputs and outcomes of the system Conducts impact evaluations to measure actual results Documentation & reporting mechanisms; inform stakeholders & mgtment decisions.
A SYSTEM RESULTS-BASED APPROACH M&E MODEL
Strategic Goals & Objectives of
the HEART Trust/NTA;
TVET System
3. Outcomes & Impact Evaluation: Analysis of Outputs/Results & Impact Assessment
2. Performance Tracking: Appraisal & Adjustment
1. Design of M&E System: Based on FEA – Context of the TVET System
Documentation of Accountability & Lessons Learned
Dec. 2013 22
Monitor and evaluate systems and processes of the
TVET system to facilitate conformance with
educational quality standards
Provide ‘on the ground’ capacity building activities through
guidance/coaching, sharing of best practices, sharing of resources, follow-up M&E
activities
Facilitate preparation for accreditation
activities
Provide timely information to
stakeholders of the TVET system
M &E Framework in accordance with
international standards
developed and implemented
ALIGNMENT OF OBJECTIVES WITH GOALS OF ORGANZATION/TVET SYSTEM Org. Goals: A Robust Quality Management System Implemented & Maintained
Dec. 2013 23
DESIGN OF THE M&E SYSTEM: Context of TVET System, Stakeholders, Resources , Etc.
Dec. 2013 24
The Ministry of Education The HEART Trust Board of
Directors The Senior Executive Team
Provide strategic direction and resources
KEY STAKEHOLDERS OF THE M&E SYSTEM; ENGAGEMENT
Directors of the HEART Trust/NTA
Senior Director (Workforce Development & Employment Division)
Directors/Principals
Instructor/Teachers Employees of training
entities Learners
Accreditation bodies: NCTVET UCJ
Dec. 2013 25
The TVET Development and Support Systems Division Learning Resources Development (LRD) Department TVET Consultancy and Institutional Capacity Building (TCICB)
Department Vocational Training Development Institute (VTDI) Monitoring & Evaluation Department
KEY STAKEHOLDERS OF THE M&E SYSTEM; ENGAGEMENT
The Monitoring and Evaluation Department – Process Owner Its mandate: to effectively implement and maintain the
M&E system Responsible for engaging all stakeholders to facilitate
successful implementationDec. 2013 26
• Sound understanding of organizational and wider TVET objectives.
• Obtain requisite resources (through budgetary system) – staffing, funding for operations, accommodation & equipment.
• Critical areas of competence of M&E staff - working knowledge of quality assurance systems and measures.
• Familiarization with TVET training programmes, including competency-based delivery and assessment methodologies.
CRITICAL SUCCESS FACTORS FOR SUCCESSFUL IMPLEMENTATION
Dec. 2013 27
Development/imp. of macro plan for M&E
• Target training institutions for M&E• Timelines• M&E personnel assigned, including collaborations
with partnering entities (e.g., CTIs and MOE schools)
• Budget to facilitate M&E operations.
f
Educational quality standards (consistent with NCTVET/UCJ) Document review Portfolios of workQuestionnaires InterviewsObservation rating scale inventory check-list status reports
CRITICAL SUCCESS FACTORS FOR SUCCESSFUL IMPLEMENTATION
Rationalize/dev./imp. appropriate M&E tools and strategies
Dec. 2013 28
Documentation Facilities
• Template (incorporating QA standards) for M&E audits & reporting
• Results-based evaluations – standard report structure
• Monitoring - via monthly score card reporting process; information is fed upwards from individual, to dept, to executive/board/Ministry levels.
• Needs assessment currently underway to inform integrated IT system for M&E
CRITICAL SUCCESS FACTORS FOR SUCCESSFUL IMPLEMENTATION
Dec. 2013 29
IMPLEMENTATION OF THE M&E SYSTEM: Performance Tracking; Use of QA Standards; Evaluation & Documentation
Dec. 2013 30
PERFORMANCE TRACKING
Performance Tracking Monitoring the TVET System Checking for consistency between what was intended against what is actually taking place. Tracking progress of the implementation process. Identifying system strengths and weaknesses. Documenting experience and appraising performances, e.g., assessing training performances against objectives; & making recommendations for adjusts where necessary
Dec. 2013 31
IMPLEMENTATION - PERFORMANCE TRACKING
Standards for Monitoring the TVET System
NCTVET – primary QA body for TVET programmes - accreditation standards are adopted Provide general orientation/reinforcement on QM issues Conduct M&E audits to facilitate readiness for accreditation/NCTVET external verification evaluations. Reinforce quality-consciousness among TVET stakeholders.
Dec. 2013 32
IMPLEMENTATION - PERFORMANCE TRACKING
Example of Utilization of Standards in Reporting Template1. Standard: Quality Assurance & Control/GovernancePerformance Indicator/Clause 1.1
Example of Evidence
Findings Statement
The organization has a mission statement that reflects the defined goals/aims and objectives as they relate to TVET and the needs of the target population and is communicated to staff, clients and to the general public effectively.
Mission statement: Documented Clarity; Addresses TVET Articulates outcomes re client
benefitsStatement of goals/aims and objectives: Clarity; Relates to Org. mission
Communication of mission to target group: Copies of advertisements used for
the programmes or organizationOrganization’s brochure, client and staff handbooks, and related docs.Mission statement mounted and placed at strategic locations, such as the administrative office and the library
FindingsConformances xx xx Non-conformances yy yy Recommended Actions for Non-conformances zz zz
Dec. 2013 33
IMPLEMENTATION - PERFORMANCE TRACKING
Types of Tools for M&E1. Standard: Quality Assurance & Control/GovernancePerformance Indicator/Clause 1.1
Example of Evidence
Example of Types of Tools for Data Collection
The organization has a mission statement that reflects the defined goals/aims and objectives as they relate to TVET and the needs of the target population and is communicated to staff, clients and to the general public effectively.
Mission statement: Documented Clarity; Addresses TVET Articulates outcomes re client
benefitsStatement of goals/aims and objectives: Clarity; Relates to Org. mission
Communication of mission to target group: Copies of advertisements used for
the programmes or organizationOrganization’s brochure, client and staff handbooks, and related docs.Mission statement mounted and placed at strategic locations, such as the administrative office and the library
• Document review (policies and procedures, reports, records, etc.)• Checklists• Interviews• Observation
Dec. 2013 34
Completed M&E
Reports
Director, TCICB – for Capacity-buidling Issues
Relevant Senior
Directors
Directors, Principals,
Managers of Training Insts. Ministry
of Education
M&E Department
INFORMATION FROM M&E TRACKING PROCESS INFORMS FINAL REPORTS
Dec. 2013 35
M&E Officer required to follow-up with the training institution to:• Confirm receipt of M&E
report• Establish convenient time
to meet with the wider management team of the training institution
• Review content of report• Develop action plan with
timelines to address gaps identified during the M&E audit
• Agree on schedule to facilitate follow-up reviews ensure action items are
being addressed (inc. recommended capacity building issues)
M&E FOLLOW-UP PROCESS
Dec. 2013 36
EVALUATION
Performance tracking/monitoring – not enough to measure real results.
Evaluative information required to determine whether or not the intended outcome/result/impact is being achieved.
Aims to determine the relevance and fulfillment of objectives, effectiveness, impact, and sustainability of the TVET system.
Dec. 2013 37
EVALUATION
Sub processes to consider to inform Evaluation process:• Diagnosing needs, capabilities and capacities. • Determining baseline/starting-point
performance.• Pilot-testing/trial-run to validate tools and
strategies.• Formative evaluation of inputs, processes,
outputs and impact.• Summative or outcomes evaluation. • Impact assessment (main effects and side
effects).
Dec. 2013 38
EVALUATION
Strategies for Evaluating the TVET System:• Directives from Senior Management – e.g.,
evaluations on the effectiveness of the restructured training system.
• Evaluations on selected training interventions to analyse possible impact that has been made towards quality improvement.
• Evaluations on certification rates.• Tracer studies. • Training impact evaluations.
Dec. 2013 39
• Reports generated from M&E activities appropriately documented and disseminated
• The M&E model - serve to foster/promote learning and enhance knowledge management processes.
stakeholders are able to effectively apply information in their daily work for accountability, improvements in performance, decision making and learning.
Provide critical support for the TVET system’s strategic process thereby strengthening sub-processes to facilitate the achievement of objectives.
Improve the overall performance and quality of results of ongoing and future programmes, projects and strategies.
DOCUMENTATION AND DISSEMINATION OF M&E INFORMATION
Dec. 2013 40
.
Info/updates shared at meetings: dept.; div.; executive; Board; MOE
Existing ICT infrastructure; NA currently being done to inform Integ. M&E system
Reports emanating from M&E activities – to stakeholders
Action plans generated via M&E audits – Follow-ups & Feedback
Quarterly progress reports (M&E audits)
Examples of documentation
and dissemination of M&E information
DOCUMENTATION & DISSEMINATION OF M&E INFORMATION
Updates via Balanced Scorecard reports: monthly/quarterlyannually
Dec. 2013 41
MONITORING THE M&E SYSTEM
• Continuous feedback from TVET system:- Training Institutions- Directors/Principals/Instructors/ Learners- Senior Management, etc.
• Comprehensive questionnaire administered annually
Information gathered - basis for learning/inform improvements in M&E systems/processes
CONTINUOUS FEEDBACK
Dec. 2013 42
JAMAICA’S APPROCH TO MONITORING AND EVALUATION
OF ITS TVET SYSTEM
THANK YOU
Dec. 2013 43