Evaluating Teacher Training changing classroom practices Richard Lambert, Ph.D.

Evaluating Teacher Evaluating Teacher TrainingTraining changing changing classroom practicesclassroom practices

Richard Lambert, Ph.D. Richard Lambert, Ph.D.

Magic bullets and everyday practiceMagic bullets and everyday practice

In educational evaluation we try to In educational evaluation we try to determine the value of the magic determine the value of the magic bullet.bullet.

In the end what we really end up In the end what we really end up doing is investigating the doing is investigating the interaction between the magic interaction between the magic bullet and the teachers who were bullet and the teachers who were asked to change their practices.asked to change their practices.


We discussed this question as a We discussed this question as a part of Process Evaluationpart of Process Evaluation– Implementation FidelityImplementation Fidelity– Follow-up technical assistanceFollow-up technical assistance– Administrative buy-inAdministrative buy-in– Nothing happens without the teacherNothing happens without the teacher

What process must take place for What process must take place for magic bullets to make it into magic bullets to make it into everyday classroom practice?everyday classroom practice?


Teachers have to see a need for Teachers have to see a need for changechange

Teachers have to believe the magic Teachers have to believe the magic bullets will meet the needbullets will meet the need

Teachers have to be “free” to changeTeachers have to be “free” to changeResources have to be availableResources have to be availableTime to planTime to planTraining has to be effectiveTraining has to be effectiveWhat is effective training?What is effective training?

Effective TrainingEffective Training

Targeted to real needsTargeted to real needsEngaging / ChallengingEngaging / ChallengingPrompts new thinkingPrompts new thinkingModelingModelingTraining with peers / time to processTraining with peers / time to processTime to experiment / try out Time to experiment / try out

methodsmethodsLearning takes place / new Learning takes place / new

knowledgeknowledgeLeads to change in practiceLeads to change in practice

Barriers to ChangeBarriers to Change

Administrative support / expectations for Administrative support / expectations for implementationimplementation

Technical assistance and follow-upTechnical assistance and follow-up Interactions between the new and innovative Interactions between the new and innovative

strategies from training and what was strategies from training and what was already in placealready in place

Implementation fidelityImplementation fidelity Resources for sustainabilityResources for sustainability

The Kirkpatrick ModelThe Kirkpatrick Model

The first task involved in evaluating a The first task involved in evaluating a training or professional development training or professional development program, is to design a process for program, is to design a process for documenting what actually takes documenting what actually takes place during the training. place during the training.

The evaluator needs to conduct a The evaluator needs to conduct a mini-process evaluation about the mini-process evaluation about the training itself.training itself.

The Mini-Process The Mini-Process EvaluationEvaluationDesign an evaluation process to Design an evaluation process to

document the following:document the following:

Who attended?Who attended?What content was delivered?What content was delivered?What mix of activities and teaching What mix of activities and teaching

techniques did the trainer use?techniques did the trainer use?

The Mini-Process The Mini-Process EvaluationEvaluationWhat were the physical characteristics What were the physical characteristics

of the space in which the training took of the space in which the training took place? place?

How long was the actual contact time How long was the actual contact time between the attendees and the trainer, between the attendees and the trainer, considering breaks and time-on-task considering breaks and time-on-task issues?issues?

How engaged were the attendees?How engaged were the attendees?

Process Evaluation Process Evaluation StrategiesStrategiesVideo or audio taping of the trainingVideo or audio taping of the trainingThe tapes can then be translated The tapes can then be translated

into transcripts or other recordsinto transcripts or other recordsThe tapes and / or transcripts can be The tapes and / or transcripts can be

codedcoded

Attendance recordsAttendance recordsObservers / ChecklistsObservers / Checklists InterviewsInterviews

Next StepsNext Steps

Evidence for the effectiveness of Evidence for the effectiveness of Training can take many formsTraining can take many forms

– Simple short term satisfactionSimple short term satisfaction

– Causal inferences about the Causal inferences about the effects of trainingeffects of training

– High levels of organizational High levels of organizational change and long term impact.change and long term impact.

Next StepsNext Steps

The next task is to determine the The next task is to determine the level or levels of evaluation level or levels of evaluation evidence that the stakeholders are evidence that the stakeholders are interested in, and that the interested in, and that the resources are sufficient to address.resources are sufficient to address.

Kirkpatrick’s four levels of Kirkpatrick’s four levels of evaluation can serve as a guide in evaluation can serve as a guide in this process.this process.

The Kirkpatrick ModelThe Kirkpatrick Model

ReactionReactionLearningLearningBehaviorBehaviorResultsResults

The goal is to make The goal is to make INFORMEDINFORMED judgments about the merit or judgments about the merit or value of training.value of training.

The LevelsThe Levels

The four levels from Kirkpatrick’s The four levels from Kirkpatrick’s modelmodel

Level I – ReactionLevel I – Reaction– How well did the attendees like the training?How well did the attendees like the training?

Level II – LearningLevel II – Learning– What concepts, facts, or skills were learned?What concepts, facts, or skills were learned?

Level III – BehaviorLevel III – Behavior– What changes in behavior or attitude did the What changes in behavior or attitude did the

attendees make once back on the job?attendees make once back on the job? Level IV – ResultsLevel IV – Results

– What were the tangible impacts of the What were the tangible impacts of the training program on the organization?training program on the organization?


Each of these four levels of Each of these four levels of evaluation is associated with evaluation is associated with specific evaluation goals and specific evaluation goals and strategies. strategies.

With each increase in level comes With each increase in level comes

an increased level of difficulty for an increased level of difficulty for the evaluator. the evaluator.


It may be relatively easy to It may be relatively easy to document process and gauge document process and gauge participant reaction to a training participant reaction to a training program.program.

It may be very difficult to establish It may be very difficult to establish causal links between a training causal links between a training program and organizational program and organizational changes.changes.


A set of specific steps, in the form of A set of specific steps, in the form of questions and activities, will be outlined questions and activities, will be outlined in the coming weeks to further illustrate in the coming weeks to further illustrate the nature of each level.the nature of each level.

What evaluation activities / strategies What evaluation activities / strategies match the levels?match the levels?– ReactionReaction– LearningLearning– BehaviorBehavior– Results Results

Evaluating Teacher Training changing classroom practices Richard Lambert, Ph.D.

Documents

Transcript of Evaluating Teacher Training changing classroom practices Richard Lambert, Ph.D.