Piloting Monitoring and Evaluation

Jean Luc MARCELLIN

IVETA 03 December 2013

Outline

• What is piloting

• What is monitoring

• What is evaluation

• How do they relate to each other

• M&E before the implementation of a new training

program (piloting phase)

• M&E during the implementation of a new training

program (piloting phase) or during the normal

course of a regular training program

• Observing and tracking carefully the

operation of a system or a process, in

real-time o Driver of a car on the road

o Air plane pilot

o Training manager

o Director of studies

o Monitoring is the process of systematically collecting data

(the key elements of program performance, usually inputs

and outputs) on a regular basis

o To monitor is to check on how the activities are progressing. It is systematic and purposeful observation

What is monitoring?

What is Piloting?

• This is the operation in which the pilot

decides and acts in real time using the

information provided by monitoring

indicators (dashboard / scoreboards).

What is Evaluation? • Assessment of the change in targeted

results (objective judgment) and make decisions for the future, during or after completing the project

• Evaluation should provide a clear picture of the extent to which the intended objectives of the activities of the piloting process have been realized. “Have we achieved our objective?” is the question to be asked

How do they relate to each other?

Evaluation (judgement, appraisal)

are closely

related

Integration

Interdependence

Piloting (to make

decision and act

accordingly)

Monitoring (gathering data,

information system)

Interdependence • Monitoring a training program = to observe (monitoring)

specific indicators in real time.

• Piloting = Based on the observations made through

monitoring, decide in real time what should be done to

achieve the goals, objectives and meet the defined standards

• Given the expected standards and objectives for the program: establish a process for collecting and processing information to monitor states, developments and gaps (real time).

• The information gathered is translated into indicators and feed scoreboards for monitoring the program.

• Depending on the evolution of these indicators, analyzes are made and decisions taken in real time (piloting).

M&E at different levels • In TVET, monitoring, piloting and

evaluation are related to training

programs, but also to the entire TVET

system at four different levels: 1. for each program (criteria and indicators),

2. for each program at national level (ATVET),

3. local (the VTPC),

4. for all of the VT system.

The first three levels feed the fourth.

Inter relations at the different levels

Training program X Local monitoring

and evaluation Indicators 1

Training program X National monitoring and evaluation

Compilation of Indicators 1 + national specifics indicators 2

Monitoring and evaluation ot the VET Systeme at

the local level (VTPC)

Indicators 1 +

Specifics indicators 3

Monitoring and evaluation of the TVET system at the natioonal level

Compilation of Indicators 1+2+3

+

Specific national Indicators 4

Piloting Monitoring and Evaluation at the local

level

Part 2

M&E at the local level • Within the VTPC, monitoring and

evaluation is done: o for each specific training program by program

team members and training manager in

cooperation with high level management

o At the intuitional level (specific indicators for

the VTPCs at the local level)

M&E at the local level What do we want to know?

• Does the VTPC have all the resources it needs to

deliver the training in term of human resources,

equipment, classrooms, workshops, learning and

teaching resources, and also alternative resources

(Public Private Partnership) in compliance with the

curriculum requirements?

• Is the training delivered in conformity with the training

program standards (chronology, numbers of

competencies, number of modules, number of hours per

modules theory /practice, competency assessment,

teaching and learning strategies) and in conformity with

Competency Based Training pedagogical principles and

practices?

M&E at the local level What do we want to know?

• What is the performance of training in terms of

student’s success rate, drop outs?

• What is the satisfaction of the stakeholders

mainly the students, the trainers, the employers

and the students parents?

Always keep in mind the qualitative aspects in terms of the object of the training, the motivation of the trainers,

motivation and learning experience of the students, aspect of time and resources management, the application of

appropriate learning and teaching strategies, governance, transparency, and the quality of the competency assessment

in cooperation with the industry.

We have to distinguish 2 types of situations:

• The monitoring and evaluation for the implementation

of a new training program (the first time this training

program is delivered)

• The monitoring and evaluation of a regular ongoing

training program

M&E at the local level (VTPC) • Concerning the training programs, what actions

should we follow, what must be monitored in order to

react in real time (piloting)?

• What are the indicators and what scoreboard do we

need?

• We need also to clearly define

• who should perform the monitoring and

evaluation,

• what are the data and information needed?

• Finally, for each actor, what is his own real time

of reference (the week, the month, the semester

the full cycle, or many cycle)?

• This will lead to the determination of the frequency of

the observation of the indicators for each actor,

involved.

M&E at the local level (VTPC)

The monitoring and evaluation for the implementation of a

new training program (the first time this training program is

delivered)

• What do we need to monitor during the preparation

phase of the delivery of the new training program?

• What do we need to monitor during the delivery of the

new training program?

For detailed answers to these questions we have designed

the implementation guide tool kit (11 actions) and the

monitoring and evaluation hand book.

M&E at the local level (VTPC)

The implementation of a new training program requires

that the following activities to be done

Action plan for PPP development

Diagnostic of the training center,

feasibility study, decision to go forward

or no, setup of the implementation

team. (real need?, and readiness to

face challenges by leadership team)

Training need analysis for teachers,

management and staff

Elaboration of an detailed implementation action plan

Resource allocation:: instructors, classrooms, labs and workshops,

training equipment, consumables, instructional material, budget,

administrative staff support, external support (PPP)

Carry out the training and coaching

Communication and marketing plan

Administrative organization

of the training, collection of

data for scoreboards,

Pedagogical organization of the training

(registration process, planning tools, assessments,

instructional support, students groups management, pedagogical coordination…).

M&E at the local level (VTPC)

According to the 4 axes of the M&E guidelines and hand book

(resources, conformity, performance, satisfaction), below are

examples of indicators

• The respect of the activities scheduled in the detailed

implementation action plan (Who ? Training manager,

When? weekly)

• The deliverables mentioned in the action plan (action 1 in

the piloting tool kit) (TM, W)

• The quality and conformity of the deliverable

• The training & coaching of the trainers, management and

staff according to the recommendation of the TNA report.

• The allocation of all the resources in compliance with the

training program requirements

What should be monitored during the preparation phase, before the

delivery of the new training program?

M&E at the local level (VTPC)

• Rate of implementation of the action plan based on the

projected schedule (time and resources management).

• Checklist of deliverables with mention about their

quality and conformity

• Rate of realization of the training plan derived from the

TNA

• Number of the beneficiaries compared to projected

number

• % of satisfaction related to the evaluation of the

training received

What should be monitored during the delivery of the training program

M&E at the local level (VTPC)

By who? Director (D), Training Manager (TM), Teachers (T) , Stakeholder (SH), ATVET, Student (S) etc.

Frequency? Weekly (W), Daily (D), Monthly (M) Life cycle of the training program (LC) etc.

• In terms of resources

• % of training equipment, learning material,

instructional resources , etc. in comparison with the

requirements of the training program

• Real costs compared to the initial budget

• % of training spaces provided (school premises,

classroom, workshops, laboratories, on the job

training facilities)

• Number of trainers allocated compared to the number

of trainers required

• Number of students enrolled compared to the

maximum number per group

• Availability of Internet access (bandwidth and speed)

What should be monitored during the delivery of the training program?

M&E at the local level (VTPC)

• Access to work places (student palcement PPP)

• Efficiency of the communication and marketing plan

• Number of visitors enquiring about the new training

program / # of phones calls

• % of increase of PPP conventions with private

sectors

• % evolution of the grants, gifts in kinds, equipment

loans etc.

• % of increase of teachers participation and training

in the workplace

• % of increase of workers from private sector

coming to give training in the VTPC

What should be monitored during the delivery of the training program?

M&E at the local level (VTPC)

• Use the tools provided in the piloting tool kit

At the beginning of the project

• Establish a gant diagram based on the implementation

chronogram provided (implementation tool kit)

• Establish a list of deliverable with the name of the

person responsible, the due date, the conformity and

the quality specifications requirements check list for

each deliverable

• For deliverables examples refer to the implementation

tool kit and the monitoring and evaluation hand book

Recommended best practices (1)

M&E at the local level (VTPC)

• Plan weekly meetings to gather all data required to

feed the indicators

• Plan monthly meeting to elaborate the scoreboards

based on the indicators selected.

• Plan monthly management meetings with the director

to take decision and act based on the scoreboards

• Update the work plan and calculate the ratio of

realization (respect of dates of deliverables should

comply to quality and conformity requirements)

• Automatize all these tasks by using an information

management system

Recommended best practices (2)

M&E at the local level (VTPC)

Questions ?

Thank you

Very much