Steven W. Vil lachica, PhD

SteveVillachica@boisestate.edu

Organizational Performance and Workplace Learning

WHEN PERFORMANCE IS FUZZY: THE CRITICAL INCIDENT TECHNIQUE

(CIT)

Download slides and handout at https://sites.google.com/a/boisestate.edu/ieeci/e2r2p/project-deliverables

Agenda2

Introduction

CIT Examples

Your Turn

Wrap Up

3

Where CPT Fits in Performance Improvement

What do you do when…4

Exemplary performance is fuzzy? No one knows what a “good one”

looks like? Managers and clients don’t know how

work gets done? There are no functional descriptions of

workplace activities? Job descriptions workplace tasks?

What the organization says it values what the organization really values?

5

Jonassen, Tessmer, & Hannum (1999a,

1999b)

Harless (1986)

Hoffman, Coffey, Ford, & Carnot (2001)

Flanagan (1954, 1962)

Critical Incident Technique

6

CIT as Evidence-Based Practice

Data-Driven Decision Making

1. Determine questions you want to answer

2. Collect specific, relevant data from different, triangulated sources in the field

3. Analyze the data

4. Draw conclusions to answer the questions--in ways that improve valued performance

Almost 60 Years of Use In peer-reviewed

journals In a variety of disciplines

and workplace settings To create all sorts of

performance improvement solutions

More stuff that works!

7

CIT in Many Fields

Researcher Used CIT toFlanagan (1954) Create procedures to select and

train WWII aircrews.Thomas & Bostrom (2010)

Identify triggers that virtual teams use to adapt their uses of technology during a project

Korte (2010) Investigate how newly hired engineers socialize themselves within a firm

8

CIT in Many FieldsAbout 20 CIT appearances in PI and PIQ

Researcher Used CIT toCraytor (1968) Create programmed instruction in

therapeutic radiology for nursing students

Smith (2009) Identify areas of expertise associated with ASTD’s competency model

Hale (2011) Create ISPI’s proficiency-based certification for school improvement specialists

9

CIT in Many FieldsAbout 20 CIT appearances in PI and PIQ

Researcher Used CIT toLundberg, Elderman, Ferrell, & Harper (2010)

Conduct a needs assessment investigating a problem with billable hours in a national retailer’s parts and service department

Bacdayan (2002)

Create a test that quality improve-ment teams can use to determine the suitability of a given project

Dean (1998) How to conduct CITMarrelli (2005) How to conduct CIT

A Basic CIT Process10

1. Determine the aim of the CIT

3. Collect incidents• Observations• Interviews• Focus groups• Surveys

4. Analyze data• Frame of reference• Categories• Priorities• Verification

5. Report findings• Categories• Prototypical incidents• Limitations• PI conclusions

2. Plan the CIT• Observers• Observations• Specific behaviors

CIT isn’t a rigid recipe. It’s a set of flexible set of principles. (Flanagan, 1954; Woolsey, 1986)

Handout pp. 1-2

11

Example 1: Making Service Standards Real

The opportunity The incidents

• Service Standards Professional Respectful Compassionate Helpful

• What do the standards REALLY mean?

• How do we operationalize the standards?

• How do we close gaps in service performance? The results

• Client (anonymous)• Stephanie Clark• Amanda Collins• Julie Kwan• Allison Sesnon

12

What the ID Team Did

Use CIT to collect stories about exemplary and non-exemplary performance

Generated competencies Ranked the criticality of the competencies Focused on two key competencies

• Responding to clientele needs

• Communicating with clientele and team Fixed the environment and provided training

• Standards, feedback, process

13

A Service Standard Example: Helpful

Exemplary Performance Non-Exemplary Behavior

An elderly guest, using a cane, came into the kitchen for a yogurt. A volunteer working in the kitchen greeted her and engaged in friendly conversation. The volunteer recognized that the guest was having difficulty going out the door, so the volunteer offered to hold the door. The guest remembered she needed a spoon for her yogurt, and the volunteer fetched a plastic spoon for the guest and assisted the guest out the door.

A family is in a hurry to get on the road. They were just informed unexpectedly that they need to check their child out of the hospital this afternoon. They are frantically trying to get everything done to leave. A volunteer notices the family is leaving and reminds them to be sure to wash the room laundry before they leave.

14

How CIT Helped

Made otherwise abstract standards visible by associating compelling stories with each

Targeted service competencies needing improvement

Provided a mechanism to fix environmental causes of the performance gap

Provided sources of demos and role-play activities for the training

Made the effort “by and for the volunteers and staff”

http://www.performancexpress.org/2012/08/tales-from-the-field-making-service-standards-real-for-families-in-need/

Example 2: Decreasing Time to Competent Engineering Performance

15

This material is based upon work supported by the National Science Foundation under Grant No. 1037808.

Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Engineering Education Research to PracticeE2R2P

16

Improve engineering education in ways that improve workplace performance.

Education engineering for engineering education.

Research-to-PracticeValley of Death

Engineering Education Engineering Workplace

Resea

rch Faculty

Adoption

EngineeringStudents

WorkplaceSkills

Newly Graduated and Hired “Freshout” Engineers

Our Shared OpportunityDecrease Ramp Up Time to Competent Performance

Desired Competency

Per

form

ance

Time

Actual Competency

Promotion!

New Task/Project

Leave University/Enter Workforce

$$$

Company Costs

$ Training$ Errors$ Mentoring$ Salary$ Opportunity$ Other projects$ Others?

{REDUCE CO$T

Increase Starting Skills

Change Learning Curve- OR - }- OR -

Make Boundaries Porous

17

Spanning Gaps between Actual and Desired Engineering Performance

18

Decrease Ramp-up Time to Competent Job Performance in the Engineering Workplace

Research Questions

• What are newly graduated and hired “fresh out” engineers doing/not doing in the workplace that they should?

• What are the consequences of performance/non-performance in the workplace?

• What workplace competencies should fresh outs possess?• In what workplace contexts do fresh outs apply the

competencies?• What are the root causes of workplace nonperformance?

Focus Groups & Surveys

• Engineering managers, engineering leads, HR personnel, and technical scientists who work with fresh out engineers

• Fresh out engineers

Engineering Practice Survey

Shared Opportunity

Root Cause

Analysis

Escape Cause

Analysis

Corrective Action

Escape Corrective

Action

Problem Inspection Failures

Problem Identification

Education Engineering

Method

19

Participants

7 Focus Groups Qualitative design using critical

incident technique (Flanagan, 1954)

16 engineering managers, lead engineers, supervising engineers, technical scientists, and HR personnel that work with freshouts to bring them up to speed in the workplace

10 freshouts

Company Manager Fresh-OutParametrix 5 0

Micron 4 3Motive Power 3 4

CH2MHill 4 3

Total 16 10

MethodProcedure

20

Identify Company Sponsors

Arrange Focus Groups

Collect Data about Workplace

Performance

Collect Data about Causes of Nonperformance

Share Results and

Sensemaking

Work towards Collaborative

Corrective Action

Recruitment

Facilitate Focus Groups

Grow Collaboration

MethodInstrumentation

21

Critical Incident Card

Handoutp. 4

MethodInstrumentation

13

Root Cause Analysis

• Data• Expectations• Feedback• Standard Operating

Procedures

• Resources• Software• Tools• Support

• Incentives• Rewards• Consequences

• Knowledge• Skills

• Physical Capacity• Mental Capacity• Flexibility• Resilience

• Motives• Affect• Work Habits• Drive

EN

VIR

ON

ME

NT

PE

RS

ON

INFORMATION TOOLS MOTIVATION

23

Comparing the Examples

ID team supporting nonprofit service standards

Research team conducting a performance analysis for freshout engineers

Collect instances Group instances into

behaviors or competencies Prioritize the groups Make sense of the data

“It’s the small stories gathered together that made that big ah-ah.”

--Allison Sesnon

24

Your Turn!Create and Classify Incidents for ISPI

ISPI wants to collect real-world stories about performance improvement

1. Instances of successful HPT performance

2. Instances of unsuccessful HPT performance

3. Assign categories

4. Leave completed incidents on the table

You will need a pen or pencil!

25

Ground Rules

Doing this voluntarily. Can stop any time. Feel free to omit your name and email if you

want. ISPI may contact you to learn more about an

incident you provide. ISPI considers your data confidential. ISPI will report data in aggregate form,

without mentioning individual contributors.

26

Name (optional) Email address (optional)

Describe an incident that occurred to a practitioner (you or someone else) trying to improve performance.

What level of performance was the practitioner trying to improve?(Check all that apply) Worker Organization Enterprise Society

Does this incident reflect (check one): Where the practitioner successfully

performed a task related to improving performance?

Where the practitioner was unsuccessful in performing a task related to improving performance?

What were the general circumstances leading up to this incident?

What the practitioner was trying to accomplish?

How did this incident affect the goals of the practitioner’s project?

How often do incidents like this occur? Only once Sometimes Frequently

Discrete Tasks• Deliver a client

presentation• Identify the root

cause of a problem• Analyze a data set

• Keep stories short• Focus on quick

generation

Use Action Verbs• Delayed production• Increased costs• Satisfied customers• Met standards

27

Successful Incident (With apologies to Harold Stolovitch)Describe an incident that occurred to a practitioner (you or someone else) trying to improve performance.

Describing a training request, the client seemed focused on means (schedules, compliance, length of training). Client didn’t mention anything about the ends –the valued business goals that the training should produce.

Used probing questions to:• Frame statements of

actual and desired performance.

• Align the gap with business goals.

What level of performance was the practitioner trying to improve?(Check all that apply) Worker Organization Enterprise Society

Does this incident reflect (check one): Where the practitioner successfully

performed a task related to improving performance?

Where the practitioner was unsuccessful in performing a task related to improving performance?

What were the general circumstances leading up to this incident?

Training request from human resources department.

What the practitioner was trying to accomplish?

Focus on valued performance

How did this incident affect the goals of the practitioner’s project?

Refocused client on delivering a valued success story

How often do incidents like this occur? Only once Sometimes Frequently

28

Unsuccessful Incidents

Generate incidents where a practitioner was unsuccessful in performing a task related to improving performance.

29

Code the Incidents You’ve CreatedPart 1

CPT Standard (1-10)

Code of Ethics (A-F)

Cause Analysis (a-l)

For every instance:

1. Specify at least one relevant CPT standard (1-10)

2. Specify at least one ethical code (A-F)

30

Code the Incidents You’ve CreatedPart 2

For unsuccessful performances,

1. Indicate ONE potential root cause (a-l)

CPT Standard (1-10)

Code of Ethics (A-F)

Cause Analysis (a-l)

--Based on Gilbert (1978)

Initial E2R2P FindingsProblem Identification

31

12%

12%

9%

6%

6%

5%5%3%

3%

3%

3%

3%

2%

2%

2%

2%

Other23%

What Freshouts Do on the Job—Success-fully and Otherwise

Communication and Teamwork

Design

Analysis

Technical fundamentals

Software skills

Problem solving

Motivation

Positive attitude

Leadership

Work Ethic

Circuit debug

Trouble shooting and critical thinking

Real world engineering

Process Knowledge

Programming

Business System Knowledge

Other

35%

29%

11%

11%

8%6%

Initial E2R2P FindingsRoot Cause Analysis

32

17%

4%

45%

15%

19% 18%17%

57%

2%5%

Env. InfoEnv. ToolEnv. MotInd. KnowInd. CapInd. Mot

Managers Freshouts

Dean (1997)

Next E2R2P Steps33

Outreach to professional organizations, new company sponsors, and other universities

Present survey, problem identification, and root cause analysis findings to company sponsors and participants for collaborative sensemaking

Create a community of shared practice and concern

Build to a corrective action forum with all stakeholders (a.k.a. “design solutions”)

https://sites.google.com/a/boisestate.edu/ieeci/e2r2p

34

CIT Wrap Up

CIT has a track record spanning almost 60 years

CIT is an evidence-based practice for performance improvement

CIT is applicable in a wide variety of settings where performance is fuzzy

CIT can be a valuable tool for performance improvement practitionersWhen performance is fuzzy,

consider CIT!

References

35Bacdayan, P. (2002). Preventing stalled quality improvement teams: A written test of project selection ability. Performance Improvement Quarterly, 15(1), 47-66. doi:

10.1111/j.1937-8327.2002.tb00240.x

Butterfield, L.D., Borgen, W.A., Amundson, N.E., & Maglio, A.-S.T. (2005). Fifty years of the critical incident technique: 1954-2004 and beyond. Qualitative Research, 5(4), 475-497. doi: 10.1177/1468794105056924

Clark, S., Collins, A., Kwan, J., & Sesnon, A. (2012). Tales from the field: Making service standards real for families in need. Performance Xpress, (August 1). http://www.performancexpress.org/2012/08/tales-from-the-field-making-service-standards-real-for-families-in-need/

Craytor, J.K. (1968). Critical incident technique, programmed instruction and nursing education. NSPI Journal, 7(6), 12-18. doi: 10.1002/pfi.4180070606

Dean, P.J. (1998). A qualitative method of assessment and analysis for changing the organizational culture. Performance Improvement, 37(2), 14-23. doi: 10.1002/pfi.4140370207

Flanagan, J.C. (1954). The critical incident technique. Psychological Bulletin, 51(4), 327-358. doi: 10.1037/h0061470

Flanagan, J.C. (1962). Measuring human performance. Pittsburgh, PA: The American Institute for Research.

Hale, J.A. (2011). Competencies for professionals in school improvement. Performance Improvement, 50(4), 10-17. doi: 10.1002/pfi.20208

Harless, J.H. (1986). Guiding performance with job aids. In M. Smith (Ed.), Introduction to performance technology (Vol. 1, pp. 106-124). Washington, DC: The National Society for Performance and Instruction.

Hoffman, R.R., Coffey, J.W., Ford, K.M., & Carnot, M.J. (2001). Storm-lk: A human-centered knowledge model for weather forecasting. Paper presented at the Human Factors and Ergonomics Society 45th Annual Meeting, Minneapolis/St. Paul, MN.

Jonassen, D.H., Tessmer, M., & Hannum, W.H. (1999a). Job task analysis. In Task analysis methods for instructional design (pp. 55-62). Mahwah, NJ: Lawrence Erlbaum Associates.

Jonassen, D.H., Tessmer, M., & Hannum, W.H. (1999b). Procedural analysis. In Task analysis methods for instructional design (pp. 45-54). Mahwah, NJ: Lawrence Erlbaum Associates.

Korte, R. (2010). ‘First, get to know them’: A relational view of organizational socialization. Human Resource Development International, 13(1), 27 - 43. doi: 10.1080/13678861003588984

Lundberg, C., Elderman, J.L., Ferrell, P., & Harper, L. (2010). Data gathering and analysis for needs assessment: A case study. Performance Improvement, 49(8), 27-34. doi: 10.1002/pfi.20170

Marrelli, A.F. (2005). The performance technologist's toolbox: Critical incidents. Performance Improvement, 44(10), 40-44. doi: 10.1002/pfi.4140441009

Stone, D.L., Blomberg, S., & Villachica, S. (2009, April). Capturing and leveraging expert decision making and problem solving. Paper presented at the International Society for Performance Improvement, Orlando, FL. http://www.dls.com/1175_CTA.pdf

Thomas, D.M., & Bostrom, R.P. (2010). Vital signs for virtual teams: An empirically developed trigger model for technology adaptation interventions. MIS Quarterly, 34(1), 115-142.

Van Tiem, D.M., Moseley, J.L., & Dessinger, J.C. (2012). Performance improvement/HPT model--an overview. In Fundamentals of performance improvement: A guide to improving people, process, and performance (3rd ed., pp. 41-59). San Francisco, CA: Pfeiffer. http://www.ispi.org/images/HPT-Model-2012.jpg

Woolsey, L.K. (1986). The critical incident technique: An innovative qualitative method of research. Canadian Journal of Counselling, 20(4), 242-254.

Thank You

36

Questions? Comments?

SteveVillachica@boisestate.edu