Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

32
Evidence-Based Management Helping Managers to Make Better Decisions Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior Carnegie Mellon University [email protected] What It Means to be an Evidence-Based Manager

description

What It Means to be an Evidence-Based Manager . Evidence-Based Management Helping Managers to Make Better Decisions . Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior Carnegie Mellon University [email protected]. What is Evidence-Based Management?. - PowerPoint PPT Presentation

Transcript of Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

Page 1: Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

Evidence-Based Management Helping Managers to Make Better Decisions

Denise M. RousseauH.J. Heinz II University Professor of Organizational Behavior

Carnegie Mellon [email protected]

What It Means to be an Evidence-Based Manager

Page 2: Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

EBMgt is the practice of making organizational decisions based upon conscientious use of

1. Science-based principles & knowledge

2. Valid & relevant organizational and business facts

3. Critical thinking aided by decision supports

4. Ethical considerations (i.e., effects on stakeholders)

What is Evidence-Based Management?

Page 3: Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

Better Decisions by Using Practices that Work (and avoiding those that don’t!)

Defensible Decisions that Stand Up to Scrutiny (using best evidence and best process)

Developing Expertise throughout a Career (experience can be a poor teacher--bad habits!)

20 years of valid experience is different than 1 year of experience repeated 20 times!

Why Should We Care about EBMgt?

Page 4: Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

What does EBMgt Look Like?

Page 5: Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

Evidence-based Piloting?

Page 6: Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

Chesley Sullenberger, USAIR pilot, has been a visiting scholar at UC Berkeley’s Collaborative for Catastrophic Risk Management since 2007

Does research on how to make decisions to maintain safety despite technological complexity and crisis conditions

1. Use of Scientific Findings

Page 7: Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

Has written and analyzed aviation accident reports for over 20 years

2. Reliance on Reliable and Valid Organizational Facts

Page 8: Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

Used Decision Aids to Support Good Decision: As Sully considered what decision to make that day, he had his copilot review and follow all checklists on board relevant to crash landings

Formal Education to Prime His Skills: Sully is a graduate of the U.S. Air Force Academy and holds masters degrees from both Purdue University in Industrial Psychology and the University of Northern Colorado in Public Administration

3. Mindful Decision Making: Becoming Decision Aware

Page 9: Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

The last person to leave the plane, Chesley Sullenberger twice walked the plane’s aisle to check all passengers were off

Sully’s last act onboard was to grab the passenger list. Used on-shore to verify rescue of all passengers and crew

4. Ethics and Responsibility to Stakeholders

Page 10: Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

In Sullenberger’s Own Words…

“One way of looking at this might be that for 42 years, I've been making small,

regular deposits in this bank of experience, education and training.

And on January 15, the balance was sufficient so that I could make a very large withdrawal.”

Page 11: Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

How Is Sullenberger’s Example Relevant to Your Own Leadership

Experience?

Page 12: Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

EBMgt is a means to improve decision quality.

It’s a career, not a course.

Page 13: Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

Evidence-based practice movements abound in medicine, education, and public policy

Management research from psychology, engineering, operations research (ETC.) yields 1000s of studies annually

Internet (scholar.google.com) gives ready access Innovative companies now hiring “chief evidence

officers” Public demands accountability (quality decisions that

are defensible)

The Zeitgeist

Page 14: Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

EBMgt Overcomes Limits of Unaided Decisions

Bounded Rationality

The Small Numbers Problem of Individual Experience

Prone to See Patterns Even in Random Data

Critical Thinking

Decision Supports

Research• Large Ns > individual

experience• Controls reduce bias

The “Human” Problem Evidence-Based Practice

Page 15: Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

1. Get Evidence into the Conversation2. Use Relevant Scientific Evidence3. Use Reliable and Valid Business Facts

4. Become “Decision Aware” and Use Appropriate Processes

5. Reflect on Decision’s Ethical and Stakeholder Implications

Five Good EBMgt Habits

Page 16: Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

#1 Get evidence into the conversation Regularly ask “what’s the evidence…?”

Illustration- Discuss with your seatmates…

What’s a practice in your organization that you suspect might not be NOT evidence-based?

Five Good EBMgt Habits

Page 17: Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

Evidence is not the same as ‘proof’ or ‘hard facts’

... can be

- so strong that no one doubts its correctness, or

- so weak that it is hardly convincing at all

What is evidence?

Page 18: Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

Evidence of effect (do!)

Evidence of no effect (don’t!)

No evidence of effect (research!)

Don’t confuse

Page 19: Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

#2 Use Relevant Scientific Evidence

Focus on Action Principles Where Science is Clear

Rely on Science-based Sources Example: Locke’s Handbook of Organizational Behavior

(access electronic copy in blackboard)

Peer-reviewed research, especially meta-analyses

Reduce dysfunctional variations in practiceBuild effective routines, procedures, checklists

Five Good EBMgt Habits

Page 20: Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

Scientific Evidence

Best Scientific Evidence is based on large N (sample size of

people/organizations)

well-controlled studies with comparison groups &/or longitudinal data

peer-reviewed

Five Good EBMgt Habits

Page 21: Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

Peer Reviewed Journals

Page 22: Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

#3 Use Reliable and Valid Business Facts

Best Business Facts are large numbers sampled relative to population

(not single or isolated cases, e.g. sales/# sales calls)

linked to context (season, location, #users, etc.)

provide key indicators for business decisions

Five Good EBMgt Habits

Page 23: Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

Illustration--Discuss with your seatmates…

What indicators does your organization most commonly use to make important decisions?

Are these the “best business facts” you need to make these decisions?

What indicators would be more useful, if you could get them??

Five Good EBMgt Habits

Page 24: Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

# Medication errors in Unit 1 were 200% greater in 2011 than Unit 2’s. Is patient safety worse in Unit 1? Depends on number of unsafe incidents divided by # patients or # procedures—needs a control.

Mike has w/10 subordinates & 20% turnover while Kim has 55 employees & 10% turnover. Is retention better in one? Hard to determine. Small N’s have greater bias and are more variable.

McDonald’s stores average 300+% turnover/year. Does Mickey D. have a problem? Depends on industry comparison and business strategy.

Company A managers focus decisions on monthly cost, downtime and revenues. Company B managers focus on service quality, employee retention and profitability by customer category. So what? B’s more diverse performance criteria can promote attention to longer-term and growth-oriented outcomes. A’s narrower economic focus can promote shorter-term thinking.

Help Learner How to Interpret Business Facts

Page 25: Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

#4 Become “Decision Aware”

Identify different kinds of decisions learners face? What kinds of different approaches are used to them? Why?

How can you determine whether you made a “good decision” when you cannot know the outcome? (The answer to this question is what is known as “decision quality”)

Five Good EBMgt Habits

Page 26: Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

“Decision Awareness” Promotes Decision Quality

To manage decisions, know what decisions must be made.

Map out decisions that affect key outcomes. Who is responsible? (Are they prepared?)

What information is required? (Will it be available when needed?)

Five Good EBMgt Habits

Page 27: Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

Five Good EBMgt Habits

Awareness Calls Attention to Decision Process. Proper Processes Improve Decision Quality

What is the process for making the decision?

Different processes work better…- for routine decisions (create validated checklists and action plans)- for decisions with known unknowns (systematic sequence of

considerations)- for decisions with unknown unknowns (pilot-tests and trial/

experiment)

Decisions have an “aftermath” and a “pre-math” that a good manager actively manages. Is the decision well-managed? Help make it so.

Page 28: Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

Using Evidence Well Requires Your Own Critical Judgment

Page 29: Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

#5 Reflect on Decision’s Ethical Implications

Who are stakeholders for this decision?

Possible effects?

How might the decision be altered to optimize positive stakeholder effects and reduce negative?

Five Good EBMgt Habits

Page 30: Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

Scientific Principles for Effective Teaching

Set learning goals (2-5)

Pre-test: where does learner stand on learning goal before course

Build opportunities for practicing those learnings throughout course (curriculum)

Post-test: Measure progress on each learning goal and provide feedback

Feedback & Redesign: Use feedback to make course more effective over time

Five Good EBMgt Habits

Page 31: Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

Evidence based management:closing the gap between research and practice

Turning Evidence into Practice & Practice into Evidence

Page 32: Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior

J. Ehrlinger, K. Johnson, M. Banner, D. Dunning, J. Kruger. (2008) Why the unskilled are unaware: Further explorations of (absent) self-insight among the incompetent. Organizational Behavior and Human Decision Processes, 105,(1) pg. 98

E.A. Locke (ed.), Handbook of Principles of Organizational Behavior, 2nd edition, 2009. Malden, MA: Blackwell.

D. M. Rousseau (2012) Oxford Handbook of Evidence-Based Management, New York.

D.M. Rousseau, D.M. & E. Barends (2011) Becoming an evidence-based manager. Human Resource Management Journal, 21, 221-235.

D.M. Rousseau, J. Manning & D. Denyer (2008) Evidence in Management and Organizational Science: Assembling the field’s full weight of scientific knowledge through reflective reviews. Annals of the Academy of Management, 2, 475-515.

R.C. Schank, D. Llyras & E. Soloway (2010) The future of decision making. New York: Palmgrave Macmillan.

J.F. Yates. (2003). Decision management. San Francisco: Jossey-Bass.

J.F. Yates & M.D. Tschirhart (2006). Decision making expertise. In K. A. Ericsson, N. Charness, P. J. Feltovich, & R. R. Hoffman. (Eds.). Cambridge handbook of expertise and expert performance (pp. 421-438). New York: Cambridge University Press.

J.F. Yates, E.S. Veinott & A.L. Patalano (2003). Hard decisions, bad decisions: On decision quality and decision aiding. In S. L. Schneider & J. C. Shanteau (Eds.), Emerging perspectives on judgment and decision research (pp. 13-63). New York: Cambridge University Press.

Got Evidence? References