Will Thalheimer, PhD - worklearning.com · Data & Analysis Why is this farmer gathering and...

Will Thalheimer, PhD

Transforming Evaluation: From Hopeless Smile Sheets to Performance‐Focused Data

Slides available at: www.is.gd/will999stuff

Data & Analysis

Why is this farmer gathering and analyzing growing rates,

crop yields, infestation rates, soil quality, amount of fertilizer,

rainfall, sunshine, revenue per sales channel,

et cetera?

To Help HimMake His Most Important Decisions!

Data & Analysis

What criteria should he have for the data he’s collecting?

• It should be ACCURATE/VALID

• It should be RELEVANT

• It should be HIGHLY PREDICTIVE

• It should be COST EFFECTIVE

• Most importantly, it should enable his organization to make its most important DECISIONS!

Data & Analysis

We also need to collect data that is:

• Is ACCURATE/VALID

• Is RELEVANT

• Is HIGHLY PREDICTIVE

• Is COST EFFECTIVE

• Helps us make our most important DECISIONS!

Learning Professionals

What are our most important decisions?

Are we doing enough to get learners support in applying the learning?

Are we sufficiently motivating our learners to inspire them to act?

Is training useful or should we provide other or additional supports?

Is this learning method working or should we use another one?

Is this skill content useful enough to teach?

Consult

Educate

Evaluate

Bridging Gap between Research and Practice


Research Translation


https://WorkLearning.com/catalog/

Quite simply, the BEST book on smile sheet creation and utilization, Period!

Karl M. KappProfessor of Instructional Technology

Bloomsburg University

Thoughtful and sensible advice for feedback tools that will provide valid and actionable data.

Robert O. BrinkerhoffProfessor Emeritus, Western Michigan University

& Director, Brinkerhoff Evaluation Institute

Evidence‐based practice at the master level.

Julie DirksenAuthor of Design For How People Learn

34‐Page Report

QuestionWhen designing learning, should we base our learning

methods on the preferences of our learners?

A. Learners are FAIRLY GOOD JUDGES of their own learning, so we should trust their learning‐design intuitions.

B. Learners are NOT OFTEN GOOD JUDGES of their own learning, so we should utilize other learning factors.

C. Learners are SOMETIMES GOOD JUDGES and SOMETIMES POOR JUDGES of their own learning.

The Kirkpatrick Four-Level Model

Level 1

ReactionLevel 2

LearningLevel 3

BehaviorLevel 4

Results

The Kirkpatrick-Katzell Four-Level Model

Read about Raymond Katzell’s role:

https://is.gd/originator

Level 1

ReactionLevel 2

LearningLevel 3

BehaviorLevel 4

Results

“The Kirkpatrick framework has a number of theoretical and practical shortcomings.”

“[It] is antithetical to nearly 40 years of research on human learning, leads to a checklist approach to evaluation (e.g., ‘we are measuring Levels 1 and 2, so we need to measure Level 3’), and, by ignoring the actual purpose for evaluation, risks providing no information of value to stakeholders…” (p. 91)

https://is.gd/research22review

“Kirkpatrick's framework is not grounded in theory and the assumptions of the model have been repeatedly disproven over the past 25 years…

…(Alliger & Janak, 1989; Alliger, Tannenbaum, Bennett, Traver, & Shotland, 1997; Holton, 1996; Sitzmann, Brown, Casper, Ely, & Zimmerman, 2008; see Kraiger, 2002, pp. 333–335 for a critical review of Kirkpatrick's theory).”

Sitzmann and Weinhardt (in press from 2017)

“The evaluation of training programs in terms of ‘results’ is progressing at a very slow rate.”

Donald Kirkpatrick1960

“With some notable exceptions… relatively little work has been devoted to making evaluation more useful and worthwhile.”

Robert Brinkerhoff1981

Only 20% were able to do the learning measurement they wanted to do…

2007

Dixon, G., & Overton, L (2017). Making an impact: How L&D leaders can demonstrate value. Available at: www.towardsmaturity.org/impact2017

In general, are you able to do the learning measurement you want to?

54 CommonEvaluation Mistakes

www.is.gd/evaluation54mistakes

Hendrick had been CLO for three months and had begun to wrest control of the thrashing octopus that was his company’s learning‐and‐development department. He started by getting learning evaluation under control, based on the dictum “what gets measured, gets managed.”

The first pilot was with the strategically‐important Leadership for High‐Potentials course. The new evaluation strategy involved the following:

• Improving the evaluation questions asked of learners by getting rid of the old questions and only asking questions that target Level 3 (work improvement) and Level 4 (organizational impact).

• Adding a Net Promoter Score (NPS) question which asks learners to rate the likelihood that they would recommend the learning intervention to their coworkers.

A. NOT AT ALL effective.

B. SOMEWHAT effective.

C. MOSTLY effective.

D. VERY effective.

What do you think Sandra, a world‐class learning evaluation expert, will tell Hendrick about the effectiveness of his evaluation design?

Most importantly, whatdo you think are the

strengths and weaknesses of Hendrik’s approach?

There is no perfect measurement tool!

“Our measures are not perfect, but they should instead be thought of as approximations.”

Deborah L. Bandalos, Author of the 2018 book:

Measurement Theory and Applications for the Social Sciences

How would you rate your learner‐feedback questions?

Terrible

1

Bad

2

Okay

3

Good

4Given, that there is NO perfect

measurement…

Smile Sheetsto

Learning

r=.09

Alliger, Tannenbaum, Bennett, Traver, & Shotland (1997).

A meta-analysis of the relations among training criteria.

Personnel Psychology, 50, 341-357.

Smile Sheetsto

Learning

r=.09

Sitzmann, T., Brown, K. G., Casper, W. J., Ely, K., & Zimmerman, R. D.

(2008). A review and meta-analysis of the nomological network of

trainee reactions. Journal of Applied Psychology, 93, 280-295.

1990’s 2000’s

Weak Relationship is below .30 and .09 is VERY WEAK

So…SMILE SHEETS tell us VERY LITTLE about Learning

2017 – University Teaching

r = .20

“Despite more than 75 years of sustained effort, there is presently no evidence supporting the widespread belief that students learn more from professors who receive higher… ratings.”

Likert-like Scales provide Poor DataA. Strongly AgreeB. AgreeC. Neither Agree Nor DisagreeD. DisagreeE. Strongly Disagree

54321

Sharon Shrock and Bill Coscarelli, authors of the classic text, now in its third edition, Criterion‐Referenced Test Development, offer the following wisdom:

On using Likert‐type Descriptive Scales (of the kind that uses response words such as “Agree,” “Strongly Agree,” etc.):

“…the resulting scale is deficient in that the [response words] are open to many interpretations.” (p. 188)

Research shows that learnersdon’t always know their own learning…

Learners are Overly Optimistic Zechmeister & Shaughnessy (1980).

Learners can’t always OvercomeFaulty Prior KnowledgeKendeou & van den Broek (2005).

Learners Fail to Properly Use ExamplesRenkl (1997).

Learners Fail to Give ThemselvesRetrieval PracticeKarpicke, Butler, & Roediger (2009).

Two Recent Reviews Emphasize Learners’ Lack of Knowledge of LearningBrown, Roediger & McDaniel (2014); Kirschner & van Merriënboer (2013).

QuestionWhen designing learning, should we base our learning

methods on the preferences of our learners?

A. Learners are FAIRLY GOOD JUDGES of their own learning, so we should trust their learning‐design intuitions.

B. Learners are NOT OFTEN GOOD JUDGES of their own learning, so we should utilize other learning factors.

C. Learners are SOMETIMES GOOD JUDGES and SOMETIMES POOR JUDGES of their own learning.

My Journey in Tryingto Create a Better

Smile Sheet

My “New” Smile Sheet

Specific Concepts

Helping Learners Calibrate

Best Feedback Comes from Comments

Performance-FocusedSmile Sheets

http://is.gd/TrainingResearch2012

http://is.gd/ddResearch

Ultimate Goal

MAXIMALLY EFFECTIVE

SMILE SHEET

Primary Goals Secondary Goals Tertiary Goals

LEARNINGEFFECTIVE?

Will the learning be effective in supporting

on-the-job performance?

RESULTS ACTIONABLE?

Will the Smile-Sheet results communicate

with clarity and urgency to guide

action?

LearnersUnderstand?

LearnersRemember?

LearnersMotivated to Apply?

Do Smile Sheet Results Distinguish between Different

Levels of Success?

After-Training Supports in Place?

Learners Smile Sheet Decisions

Accurate? Avoiding GIGO?

Are We Measuring the Things that

Matter?

Are We Using the Smile Sheet

Opportunity to Educate Our

Stakeholders?

LearnersEngaged

Cognitive Supports Effective

Realistic Retrieval

SpacingSit-Action Triggers

Belief in Value of Concepts

Self-Efficacy in Skill Area

InoculatedJob Aids

Supervisors Follow-up

Do learners remember enough to answer the questions, are the questions focused on most

important factors, are the answers calibrated to provide granularity, are leading questions avoided, do questions avoid areas of bias?

From the information, can we determine whether a course needs to be maintained, improved, or removed? Are we avoiding

numeric averages that discourage a standards-based decision on success and failure?

While smile sheets are not capable on their own to determine effectiveness, we should at

least try to examine the four goals, (1) understanding, (2) remembering, (3)

motivation to apply, (4) after-training support.

We should use smile sheets to send stealth messages to our stakeholders, including senior decision makers, instructors,

instructional designers.

Three Key Goals:

1. Are the Questions Well-Designed?

2. Are Learners Making Good Smile Sheet Decisions?

3. Are The Data we’re Getting Clear and Actionable?

Ensuring that we are focused on the science-

of-learning factors that matter!

QUATERNARYGOALS

Traditional Smile Sheets

Learners rate instructors as credible and

engaging?

Learners say that classroom environment

was conducive to learning?

Learners satisfied with experience?

Learners think course was well‐organized?

What Most Smile Sheets Provide

1. Red‐flagging training programs that are not sufficiently effective.

2. Gathering ideas for ongoing updates and revision of a learning program.

3. Judging strengths and weaknesses of a pilot program to enable revision.

4. Providing instructors with feedback to aid their development.

5. Helping learners reflect on and reinforce what they learned.

6. Helping learners determine what (if anything) they plan to do with their learning.

7. Capturing learner satisfaction data to understand—and make decisions that relate to—the reputation of the training and/or the instructors.

8. Upholding the spirit of common courtesy by giving learners a chance for feedback.

9. Enabling learner frustrations to be vented—to limit damage from negative back‐channel communications.

Modified based on work by Robert Brinkerhoff

Reasons for Smile Sheets

Effectiveness

of the

Learning

Reputation

of the

Learning

Supporting Learners

Ultimate Goal Primary Goals Secondary Goals Tertiary Goals

HOW ABLE ARE YOU to put what you’ve learned into practice on the job? Choose One.

A. I am NOT AT ALL ready to use the skills taught.

B. I have GENERAL AWARENESS but will NEED MORE GUIDANCE to put the skills into practice.

C. I need MORE HANDS‐ON EXPERIENCE to be GOOD at using these skills.

D. I am FULLY COMPETENT in using these skills.

E. I am CAPABLE at an EXPERT LEVEL in using these skills.

A Better Smile Sheet Question

0 10 20 30 40 50 60

HOW ABLE ARE YOU to put what you’ve learned into practice on the job? Choose One.

Unacceptable

Acceptable

Alarming

Superior

Percentage of Respondents

Superior/Overconfident?

NOT AT ALL READY

HAVE AWARENESS,NEED MORE GUIDANCE

NEED MOREHANDS ON EXPERIENCE

FULLY COMPETENT

COMPETENTAT EXPERT LEVEL


In regard to the concepts taught in the course, how motivated WILL YOU BE to UTILIZE these skills in your work?

A. I will NOT MAKE THIS A PRIORITY when I get back to my day‐to‐day job.

B. I will make this a PRIORITY—BUT A LOW PRIORITY when I get back to my day‐to‐day job.

C. I will make this a MODERATE PRIORITY when I get back to my day‐to‐day job.

D. I will make this a HIGH PRIORITY when I get back to my day‐to‐day job.

E. I will make this ONE OF MY HIGHEST PRIORITIES when I get back to my day‐to‐day job.

A Question About Motivation…


After the course, when you begin to apply your new knowledge at your worksite, which of the following supports are likely to be in place for you? Select as many items as are likely to be true.

A. I will have my PROGRESS MONITORED BY MY SUPERVISOR in applying the learning.

B. I will have someone available TO COACH OR MENTOR ME in applying the learning.

C. I will have easy access to a COURSE INSTRUCTOR to contact for guidance and support.

D. I will have JOB AIDS to guide me in applying the learning to real job tasks.

E. I will be PERIODICALLY REMINDED of key learning concepts/skills over the next few months.

F. I will NOT get much direct support, but will rely on my own initiative.

A Question About Follow-Through…

But won’t learners dislike

these new type of

questions?

Percentage saying NEW QUESTIONS EQUAL OR BETTER:

90%

Percentage saying NEW QUESTIONS

BETTER:

80%


What aspects of the training made it MOST EFFECTIVE FOR YOU? What should WE DEFINITELY KEEP as part of the training?

Three Open-Ended Questions to End Your Survey

What aspects of the training COULD BE IMPROVED?

Remember, your feedback is critical, especially in providing us with constructive ideas for improvement.

Is there anything else we should have asked about? Is there anything you want to tell us?

Smile Sheets should be ONLY ONE PARTof our learning evaluation efforts

Smile Sheets

Understanding

Remembering

Motivation to Apply

After Supports

Meeting Target Goals?

Job Performance

Organizational Results

Learner Expectations

Other Expectations

Supports:

Management Support?

Workplace Obstacles?

Reinforcement?

Reminders?

Learners able to:

Understand?

Remember?

Make Decisions?

Apply the Learning?

What Messages Are Being Sent?

What Messages Are Being Sent to the Online Trainer?

Compared to most webinars (online meetings), how well did the session keep

YOUR attention? Select ONE choice.

‐‐‐

A. I had a HARD TIME STAYING FOCUSED.

B. My attention WANDERED AT A NORMAL LEVEL.

C. My attention RARELY WANDERED.

D. I was very much SPELLBOUND throughout the session.

RESEARCH/EVIDENCE

For each column,

SELECT ONE CHOICE.

HOW MUCH RESEARCH/EVIDENCE

WAS SHARED?

⃝ ZERO OR VERY LITTLE RESEARCH/EVIDENCE

was shared.

⃝ A SMALL AMOUNT OF RESEARCH/EVIDENCE

was shared.

⃝ A SUBSTANTIAL AMOUNT OF RESEARCH/

EVIDENCE was shared.

QUALITY OF RESEARCH/EVIDENCE SHARED?

⃝ I DON’T KNOW THE RESEARCH well enough to

answer this question.

⃝ Too often, the concepts presented SEEMED TO

CONFLICT WITH THE MOST‐TRUSTED RESEARCH.

⃝ The concepts presented WERE GENERALLY

SUPPORTED BY SOLID RESEARCH.

What Message Is Being Sent?

eLearning

Self‐StudyLearning Program

Employee Exit Survey

Performance-Focused Smile Sheets:

What Questions Do You Have?

I am clear about what is expected of me as a result of going through this training.

A. Strongly DisagreeB. DisagreeC. NeutralD.AgreeE. Strongly Agree

Good

Okay

Bad

How READY are you TO WRITE SCENARIO‐BASED QUESTIONS?

CIRCLE ONE OR MORE ANSWERS AND WRITE YOUR REASONING BELOW

A. I’m STILL NOT SURE WHERE TO BEGIN.

B. I KNOW ENOUGH TO GET STARTED.

C. I CAN TELL A GOOD QUESTION FROM A BAD ONE.

D. I CAN WRITE MY OWN QUESTIONS, but I’d LIKE to get SOME FEEDBACK before using them.

E. I CAN WRITE MY OWN QUESTIONS, and I’m CONFIDENT they will be reasonably WELL DESIGNED.

In your own words, how ready do you feel you are in being able to write scenario‐based questions?

Good

Okay

Bad

Things

No Time For

34‐Page Report

Special Thanks:

• Julie Dirksen• Clark Quinn• Roy Pollock• Adam Neaman• Yvon Dalat• Emma Weber• Scott Weersing• Mark Jenkins• Ingrid Guerra‐Lopez• Rob Brinkerhoff• Trudy Mandeville• Mike Rustici

Industry Empathy

Grok Problem

Triggering Event

Generate Quick

Solution

Gather Blind‐Spot Feedback

Improve Solution

Reality‐Test

Solution

Why Do We Need a New Evaluation Model?

54 CommonEvaluation Mistakes

www.is.gd/evaluation54mistakes

Too Many of Our Learning Evaluations Focus on:

Butts in Seats Learner Surveys

One Metric Biased Metrics

A

D

D

I

E

Assessment

Design

Development

Implementation

Evaluation

Messaging of Four-Level Model of Learning Evaluation

Harmful Messages (Sent or Missed)

• Does Not Warn Us Against Ineffective Evaluation Practices

• Ignores the Role of Remembering

• Level 2 Learning is Mashed into One Bucket

Beneficial Messages

• Don’t Just Focus on Learning! Focus on Results too!

• Learner Opinions Are Not Most Important

Understanding‐‐‐‐‐‐‐‐

Remembering

Performance‐Focused Smile Sheets

TraditionalSmile Sheets


Remembering


Remembering

Work Performance

Performance In Learning

Compared to

Kirkpatrick‐Katzell

Four‐Level Model of Learning Evaluation

Level 1 ‐‐ Reaction

Level 2 ‐‐ Learning

Level 3 ‐‐ Behavior

Level 4 ‐‐ Results

Two Ways to Use LTEM1. Assessing Your Evaluations2. Learning Design & Development:

Working Backward from Your Goals

2019

2019

2020

2020

Two Ways to Use LTEM1. Assessing Your Evaluations2. Learning Design & Development:

Working Backward from Your Goals

1. Sales Increase by 5%

2. Managers Coach Better

3. Simulated Coaching

4. Scenario Questions

5. IF‐THEN Decisions

6. Perf‐Focused Questions

Key LTEM Messages1. Goal of learning is to create transfer and positive learning benefits.

2. We should consider the many effects of learning transfer, not just organizational results.

3. Measuring attendance or learner activity is inadequate to validate success.

4. Measuring learner perceptions is inadequate to validate success. Focusing on effectiveness is better than focusing on satisfaction or reputation.

5. We should evaluate our success in supporting remembering, not just comprehension.

6. Measuring knowledge is generally inadequate to validate success.

7. Measuring decision‐making or task competence during learning is better than measuring knowledge.

Same asFour‐Level

What Questions do You have?

What Questions do You have?

Hendrick had been CLO for three months and had begun to wrest control of the thrashing octopus that was his company’s learning‐and‐development department. He started by getting learning evaluation under control, based on the dictum “what gets measured, gets managed.”

The first pilot was with the strategically‐important Leadership for High‐Potentials course. The new evaluation strategy involved the following:

• Improving the evaluation questions asked of learners by getting rid of the old questions and only asking questions that target Level 3 (work improvement) and Level 4 (organizational impact).

• Adding a Net Promoter Score (NPS) question which asks learners to rate the likelihood that they would recommend the learning intervention to their coworkers.

A. NOT AT ALL effective.

B. SOMEWHAT effective.

C. MOSTLY effective.

D. VERY effective.

What do you think Sandra, a world‐class learning evaluation expert, will tell Hendrick about the effectiveness of his evaluation design?

Most importantly, whatdo you think are the

strengths and weaknesses of Hendrik’s approach?

When we ask learners questions!!!

Article:One of the Biggest Lies in

Learning Evaluation

https://is.gd/BiggestLies

Level 1

Reaction

Level 2

Learning

Level 3

Behavior

Level 4

Results

When we ask learners questions!!!

Article:One of the Biggest Lies in

Learning Evaluation

https://is.gd/BiggestLies

Using traditional smile sheets, how much are learner‐feedback questions correlated

with learning outcomes?

A. High marks indicate that the training was likely to be VERY SUCCESSFUL in creating learning.

B. High marks indicate that the training was likely to be at least SOMEWHAT SUCCESSFUL in creating learning.

C. High marks on learner surveys tell us VERY LITTLE about the success of our training programs in creating learning.

Why do we evaluate?

What do Performance‐Focused Smile Sheets and LTEM have in common?

A More Muscular Approach to EvaluationSending Messages to Nudge Improvements

Evaluation that helps us make our most important decisionsand helps us get the resources and support we need!

• LTEM Report and Model — https://is.gd/LTEM999

• Katzell’s Contribution — https://is.gd/Katzell

• 54 Common Evaluation Mistakes — https://is.gd/evaluation54mistakes

• Updated Smile‐Sheet Questions — https://is.gd/Questions2018

• A Better Net Promoter Question — https://is.gd/replaceNPS

• Be Careful When Benchmarking — https://is.gd/DoNotBenchmark

• Biggest Lies in Learning Evaluation — https://is.gd/BiggestLies

• Debate About Kirkpatrick Model — https://is.gd/epicbattle

• Better Responses on Smile Sheets — https://is.gd/betterresponses

Subscribe to my Newsletter: — https://www.worklearning.com/sign‐up/

SmileSheets.com

Will Thalheimer, PhD

Phone: +1‐617‐718‐0767Email: [email protected]: WorkLearning.comBook: SmileSheets.comTwitter: @WillWorkLearn

SLIDES:www.is.gd/will999stuff

Will Thalheimer, PhD - worklearning.com · Data & Analysis Why is this farmer gathering and...

Documents

Transcript of Will Thalheimer, PhD - worklearning.com · Data & Analysis Why is this farmer gathering and...