Probabilistic reasoning II: Heuristics and biasesjpayne/ba525_powerpoint... · 2015-09-29 ·...

9/29/2015 John W Payne BA925 1

Probabilistic reasoning II: Heuristics and biases

Review of Heuristics and Biases: Tversky & Kahneman (1974) - 1

• Basic ideas: – People rely on a limited number of heuristic cues and processes that

reduce the complex task of assessing probabilities to one that is manageable. Related to the idea of “Bounded Rationality”.

– The heuristics are generally useful, but can lead to systemic errors (biases). Further, “errors of judgment are often systematic rather than random, manifesting bias rather than confusion” – Kahneman & Tversky (1979)

– The biases in forecasts are due to cognitive factors, not just motivational factors.

– Biases will be found with experts, e.g. doctors and lawyers, dealing with important problems as well as laypersons.

– Note, the idea is that error in judgment is not simply due to motivational effects. It is not an argument that motivational biases do not exist.

2

Heuristics and Biases: Tversky & Kahneman (1974) - 2

• Features of Heuristics:

– The heuristic processes are related to common psychological processes, e.g., similarity judgments. – How is this the same or different from Shah and Oppenheimer (2008) conception?

– The heuristics for probability judgments resemble subjective assessments of physical quantities like distance or size. That is, the processes are generally neither conscious or controllable. More System 1 type of thinking, see class 1 notes.

– The use of heuristics means that judgments may be a) insensitive to factors that should matter from a normative perspective, and b) sensitive to factors that shouldn’t matter. A research approach that uses the presence of biased responses to infer heuristic use. Cognitive illusions and visual illusions.

3 9/29/2015 John W Payne BA925

The Three Classic Heuristics

• Representativeness

• Availability

• Anchoring and (Insufficient) Adjustment

• The newer “Affect” Heuristic

9/29/2015 John W Payne BA925 4

Representativeness Heuristic and Biases: Tversky & Kahneman (1974)

• What is the Representativeness heuristic? – Used to make judgments about whether a target object belongs to a

given class of objects. – Judgment made on the basis of the similarity of the target object to

the prototypical instance of the class. Attribute or judgment substitution of an easier to make judgment, e.g. similarity, for a more difficult judgment such as probability. • Assumed that similarity is an easy-to-access form of information. • Is this another form of cognitive simplification?

• Some common biases

– Under-use of base-rates: The Tom W. example. • High correlation between mean similarity rank and mean

likelihood rank

– Discounting (Ignoring) sample sizes – The hospital problem and the Law of Small Numbers. “Fooled by Randomness”

• Relationship between representativeness and the frequency of events?


Classic Example of Representativeness: The Tom W. Problem

• Tom W. is on high intelligence, although lacking in true creativity. He has a need for order and clarity, and for neat an tidy systems in which every detail finds its appropriate place. His writing is rather dull and mechanical, occasionally enlivened by somewhat corny puns and flashes of imagination of the sci-fi type. He has a strong drive for competence. He seems to have little feel and little sympathy for other people and does not enjoy interacting with others. Self-centered, he nonetheless has a deep moral sense.

• The personality sketch of Tom W. given above was written during Tom’s senior year in high school by a psychologist on the basis of projective tests. Tom W. is currently a graduate student. Please rank the following nine field of graduate specialization in order of the likelihood that Tom W. is now a graduate student in each of these fields. (Use 1 for the most likely and 9 for the least likely.)

• A) Business Administration

• B) Computer Science

• C) Engineering

• D) Humanities and Education

• E) Law

• F) Library Science

• G) Medicine

• H) Physical and Life Sciences

• I) Social Science and Social Work


The Classic Tom W. Results

7

A) Business Administration

B) Computer Science

C) Engineering

D) Humanities and Education

E) Law

F) Library Science

G) Medicine

H) Physical and Life Sciences

I) Social Science and Social Work

Likelihood

Rank

Judged %

Base-rate

Similarity

Rank

4.3

2.5

2.6

7.6

5.2

4.7

5.8

4.3

8.0

3.9

2.1

2.9

7.2

5.9

4.2

5.9

4.5

8.2

15

7

9

20

9

3

8

12

17

An example from finance: How likely is an IPO to make (lose) money? Focus on

the case at hand and neglect of the base-rate of success for IPOs.

9/29/2015 John W Payne BA925

Example of a recent (2012) study: Does the use of base-rate versus case information vary by psychological distance of the

target?

• Task: Judge the likelihood that a person has a disease given some information on the base-rate of the disease, e.g., it is either rare or quite common, and given some information on the person, she has either one or four symptoms such as cough and/or fever.

• Vary the closeness of the target “Chris Chan” or Chris Smith (subjects were all of Chinese ethnicity living in Hong Kong).

• Assess the target’s likelihood of having the disease on a 7-point scale (1 = very unlikely; 7 = very likely).

8

See Yan & Sengupta, JCR, 2012


Results

9

Low Diag. High Diag. Diff. Score due to Diag

BR Low 2.31 3.61 1.30

BR High 2.69 4.07 1.38

Diff. Score due to BR

.38 .46

Low Diag. High Diag. Diff. Score

BR Low 2.40 2.94 .54

BR High 3.42 3.90 .48

Diff. Score due to BR

1.02 .96

Close

Far

Base-rate information had more of an effect when judging people more distant

psychologically. (Relate to “outside” versus “inside” perspective.


Law of Small Numbers?

• A fundamental concept in statistics is the law of large numbers. Loosely speaking, the law of large numbers says that as the size of a sample increases, the salient features of that sample (e.g., its mean) will more and more resemble the salient features of the population from which the sample is drawn.

• People often act as if they follow a “law of small numbers” in which even small samples are thought to be highly representative (resemble) the population from which the sample is drawn.

– Consequently, people will tend to over infer from small samples or short

sequences of information. Is a financial analyst with four good stock picks talented? The ‘hot hand” phenomenon. Will housing prices always go up?

– For events drawn from clearly “random” processes, people will believe that long streaks will need “balancing” if the underlying system is not assumed to have changed. “Gambler’s Fallacy”.

10 9/29/2015 John W Payne BA925

Example of Confidence in Small Samples

•Your firm has two plants, one large and one small, which mass produce a standard computer chip. Other than the amount they produce, the two plants are identical in all essential regards. Both use the same technology to produce the same product. When properly functioning, this particular technology produces one percent (1%) defective items.

•Whenever the number of defective items from one day’s production exceeds two percent (2%), a special note is made in the quality control log to “flag” the problem. At the end of the quarter, which plant would you expect to have more “flagged” days in its quality control log? Please mark one.

•

•A) The small plant

•B) The large plant

•C) The same number on average

11 9/29/2015 John W Payne BA925

Property of System 1: A bias of confidence over doubt

9/29/2015 John W Payne BA925 12

In a telephone poll of 300 seniors, 60% support the president.

“If you had to summarize the message of this sentence in exactly three words, what

would they be? Almost certainly you would choose “elderly support president.” These

words provide the gist of the story. The omitted details of the poll, that it was done on

the phone with a sample of 300, are of no interest in themselves; they provide

background information that attracts little attention.

Your summary would be the same if the sample size had been different. Of course, a

completely absurd number would draw your attention (“ a telephone poll of 6 [or 60

million] elderly voters…”). Unless you are a professional, however, you may not

react very differently to a sample of 150 and to a sample of 3,000. That is the

meaning of the statement that “people are not adequately sensitive to sample size.”

Kahneman, Daniel (2011-10-25). Thinking, Fast and Slow (pp. 113-114). Macmillan.

Kindle Edition.

Roulette Anyone?

9/29/2015 John W Payne BA925 13

Roulette is a casino game where a wheel and a ball are spun in opposite directions and eventually, the ball falls into a space. Bets can be placed on whether the ball will fall into a red, black, or green space.

There are 18 red spaces,

18 black, and 1 green

http://upload.wikimedia.org/wikipedia/en/1/11/Rwheel.jpg

Question:

9/29/2015 John W Payne BA925 14

You’re in Vegas and walk up to two roulette tables with the 5 previous spins posted as:

Which table (1 or 2) would you feel more comfortable placing a bet on “black”?

Table 1 Table 2

2 9

27 36

31 7

26 5

12 9

Representativeness Heuristic - 1

• The Gambler’s Fallacy says:

– 5 “reds” in a row in a fair game…black is due!

– Therefore bet on Table2.

– “We were due for one,” Cook said. “We missed our first nine. It's just the law of averages.” http://www.newsobserver.com/2015/01/31/4521195/decock.html#storylink=cpy

• Representativeness Heuristic

– People also think that the recent past is indicative of the future. Table 2 only produces red.

– Therefore bet on Table 1.

• “Good” probability assessment? It doesn’t matter.

9/29/2015 John W Payne BA925 15

Another example of Representativeness: What Will Google’s Price Be In The Future?

9/29/2015 John W Payne BA925 16

$0

$200

$400

$600

$800

Dec-04 Jun-05 Dec-05 Jun-06 Dec-06 Jun-07 Dec-07

Google Stock Price Dec-04 to Dec-07

Excessive Extrapolation: 3 years of increases…it’s bound to

increase again! Graph from Benartzi 2012

The Hot Hand

17

There is a belief in basketball that a player is more likely to hit than miss

following a streak of successful hits. “The hot hand” belief.

Do you agree?


The illusion of “hot hands” in Basketball

• Gilovich, Vallone, and Tversky (1985) analyzed the shooting of a professional (NBA) team over the course of a season and found that the probability of success on any given shot was essentially independent of the outcomes of previous shots.

• There seems to be no hand hot in reality!

• Much debate about this result.

• The search for the “clutch” player.

• Also, does it impact how players and coaches conduct the game?

• Attali (2013) used an entire NBA season’s worth of data to explore that latter question.

18 9/29/2015 John W Payne BA925

Probability of players’ taking the next team shot following an initial shot as a

function of that shot’s success (hit or miss) and distance from the basket.

19

Attali Y Psychological Science 2013;24:1151-1156

Copyright © by Association for Psychological Science

Effect was even stronger when the outcomes

of the prior two shots were considered.


Probability that the last shooting player was substituted as a function of the shot’s

success (hit or miss) and distance from the basket.

20

Attali Y Psychological Science 2013;24:1151-1156

Copyright © by Association for Psychological Science


More likely to be substituted for

if you shot and missed.

Regression Toward the Mean

• Extreme scores or performances measured at any one point in time will probably be less extreme the next time they are measured.

• But this seems counter intuitive due to the representativeness heuristic.

– Why should similarity (fit) vary across time?

9/29/2015 John W Payne BA925 21

Question:

9/29/2015 John W Payne BA925 22

What would you predict these 3 students’ college GPAs to be based on their high school GPAs?

High School GPA

3.80

3.00

2.20

Predicted College GPA

3.46

2.77

2.03

Actual College GPA

3.30

2.93

2.70

Student

A

B

C

Graph from Benartzi 2012

Conclusion: People do not regress enough to the mean.

A form of overconfidence?

Summary “Sins” of Representativeness and Attribute Substitution (System 1 thinking)

• First, it is important to realize that the intuitive impressions that representativeness produces are often more accurate than chance guesses would be. Thus, the use of this heuristic is NOT completely unreasonable. It may reflect a tradeoff between accuracy and effort (ease of use).

• Second, the use of representativeness leads to predictable traps that can, at least, be mitigated such as the neglect of base rate information, overweighting or small samples, and failure to appreciate regression to the mean.

9/29/2015 John W Payne BA925 23

Simple Keys (tricks) to Bayesian Reasoning: You don’t have to be a statistician- Kahneman 2011.

• Anchor you judgment of probability of an outcome on a plausible base rate. – Remember that base rates matter, even in the presence of vivid

evidence about the case at hand unless the evidence is 100% predictive.

• Always question the diagnosticity of your evidence. – Remember that intuitive impressions of the diagnosticity of

evidence are often exaggerated.

• For example, people give to much importance to the perceived strength of evidence, e.g., a good interview impression, and not enough importance on the weight of the evidence, e.g., the size of a sample or the reliability of a interviews.

Question:

9/29/2015 John W Payne BA925 25

Which is the more likely cause of death?

Falling Plane Part Shark Attack

Death by falling plane parts is 30x MORE likely Source: http://www.learner.org/discoveringpsychology/11/e11expand.html

Graph taken from Benartzi 2012,

UCLA

Availability Heuristic

9/29/2015 John W Payne BA925 26

Readily available (easy to retrieve) events in memory affect the judgment of frequency.

Adapted from Benartzi 2012

This is a heuristic that seems impossible to ignore

Availability Heuristic and Biases: Tversky & Kahneman (1974)

• What is the Availability heuristic?

– Process vs. content of thought

– A process for generating a probability?

• Relationship to the frequency of events?

– Events with higher frequency of occurrence are more likely to be recalled. Thus, again, the heuristic is NOT unreasonable.

– However, what else might impact recall likelihood?

• Examples of biases

– Vividness effects and recent occurrences

– Fault-trees and other presentation effects – WYSIATI. 27 9/29/2015 John W Payne BA925

Another Example of Availability in Memory

28

Company

AMR

Cisco Systems*

Coca Cola*

Ingram Micro

McDonald’s*

Oracle*

Owens Corning*

Southwest Airlines*

Supervalu

TechData

United Technologies

Visteon

Median Estimate

18.0

20.0

24.5

10.0

22.0

18.4

15.0

13.5

9.0

9.6

18.0

10.4

Actual

19.0

22.3

20.1

25.2

14.9

10.9

4.8

5.6

23.2

17.2

27.9

17.8

Estimate the sales revenues of the following Fortune 500 firms for the year 2001. The median value on this list is $18.4 billion and the extremes values are $4.8 billion and $27.9 billion.


Availability – Reasoning by Recall (cont.)

High Name-Recognition

Low Name-Recognition

9/29/2015 29 John W Payne BA925

•I am afraid of spiders.

“73%” “40%”

What’s your best guess of the percentage of your

peers who are afraid?

21% 79%

YES NO

Availability and False Consensus: You know what

YOU believe!

9/29/2015 30 John W Payne BA925

Home Investment Percentages From Around The World

9/29/2015 John W Payne BA925 31

94% 82% 98%

Adapted from Benartzi 2012

Cognitive Ease and Probabilities

• Who perceives a higher personal risk of heart disease? – Man A who is asked to think of three risk-inducing behaviors he

engages in, or – Man B who is asked to think of eight risk-inducing behaviors he

engages in?

• One might assume that Man B who thinks of more risky behaviors would perceive a higher risk, however, it is the opposite.

• Why? The explanation (Schwarz, 2005) is that bringing to mind three risky behaviors is mentally easier than dredging up eights, so the individual thinks, in the former case, “Well, I must be at risk.”

• Implications for assessing risk?

9/29/2015 John W Payne BA925 32

9/29/2015 John W Payne BA925 33

A nice picture from the Visit Michigan website

Failures of System 1 together with System 2

9/29/2015 John W Payne BA925 34

• How many murders occur in the state of Michigan in one year?

• How many murder occur in the city of Detroit in one year?

• Kahneman, Daniel (2011-10-25). Thinking, Fast and Slow (p. 45). Macmillan. Kindle Edition.

9/29/2015 John W Payne BA925 35

• Facts that we know do not always come to mind when we need them. (Kahneman, 2011, p. 46).

9/29/2015 John W Payne BA925 36

“Memory Bias”

Too much weight given to recent history Kahneman and Tversky (1973)

“This time is different”

Carmen Reinhart and Ken Rogoff book

•

A Third Heuristic: Anchoring and Adjustment

• Estimation is a process of anchoring on a salient number and adjusting up or down.

• Very reasonable procedure but the anchor can be very, very heavy.

9/29/2015 John W Payne BA925 37

The Anchoring and Adjustment Heuristic and Biases: Tversky & Kahneman (1974)

• Examples of related Biases – Irrelevant anchors such as your SS number.

– Conjunctive versus disjunctive judgments

– Over tight confidence intervals.

• Applies to value (price) judgments as well as probability judgments.

• What determines the anchor value?

38 9/29/2015 John W Payne BA925

8 x 7 x 6 x 5 x 4 x 3 x 2 x 1

1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 median guess = 512

median guess = 2250

correct answer = 40,320

Anchoring by Task Order

9/29/2015 39 John W Payne BA925

if asked $0.80: $0.94

if asked $2.00: $1.47

What is your best estimate of the

exchange rate two years from now?

Do you believe that two years from now the

US$/Euro exchange rate will be above or

below $0.80 per €? ($2.00 per €)?

Anchoring: Exchange Rates

9/29/2015 40 John W Payne BA925

Anchoring as accessibility and adjustment

• A variety of procedures are used in anchoring studies – Anchors provided by an external source – usually involves a “comparative judgment” –

“was Aristotle born before or after AD 1825?” followed by an absolute estimate “In what year was Aristotle born?”

– Anchors generated by the participants themselves – “When was George Washington elected President of the United States? (Initial answer might be 1776, which is then adjusted. Initial anchor on the strength or extremity of evidence and then adjust for its validity (Griffin & Tversky, 1992).

– Anchors generated randomly.

• Grice (1975) conversational norm – a person might expect a stated anchor to matter. However, note, clearly irrelevant anchors do matter, e.g. your SS#.

• Multiple mechanisms underlying anchoring as an effect – Accessibility of anchor-consistent information – Insufficient adjustment – search for the first satisfactory answer

• Not all biases result from a single process.

• Why a bias occurs has implications for debiasing.

41 9/29/2015 John W Payne BA925

Tversky & Kahneman (1983): Extensional versus intuitive reasoning – 1*

• Basic idea: Judgments of probability vary in the degree to which they follow an extensional (decompositional, frequentistic) logic or a more singular, holistic, intuitive logic.

• Note, the idea of two different processes – System 1 vs. System 2?

• The probability calculus as a standard against which to compare intuitive

probability judgments, e.g., Bayes Theorem or the conjunction rule: If (A B) then P(A) P(B). Note, a coherence principle. Compare later to a correspondence standard used in Lens Model studies.

• Heuristics based on “natural” assessment processes like similarity and memory retrieval.

42 9/29/2015 John W Payne BA925

Tversky & Kahneman (1983): Extensional versus intuitive reasoning - 2

• Which reasoning process will be used to assess probabilities will depend upon -

– the nature of the evidence, e.g., how information is presented, e.g. probabilities, e.g. .10 versus frequencies 1 in 10 or 10 in a 100. Note, sometimes people will prefer to bet on an event that occurs 9 times out of 100 rather than an event that occurs 1 time out of 10. More generally, the formulation of the question matters.

– the transparency of the event structure, see Linda results

– appeal of the heuristic, including prior use

– sophistication of the respondent, see Linda results

– But do you sometimes need both sophistication and transparency?

• The use of intuitive processes, however, will be common, and therefore biases will be frequent and sizable.

43 9/29/2015 John W Payne BA925

“Classic’ Example of Type 1 Thinking and An Error in Judgment: The Famous Linda Problem

9/29/2015 John W. Payne 44

• Linda is 31 years old, single, outspoken and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations.

• Below are eight statements about Linda. Rank all eight statements in terms of their probability of being true. Use 1 for the most probable and 8 for the least probable.

• A) Linda is a teacher in elementary school. • B) Linda works in a bookstore and takes Yoga classes. • C) Linda is active in the feminist movement. • D) Linda is a psychiatric social worker. • E) Linda is a member of the League of Women Voters. • F) Linda is a bank teller. • G) Linda is an insurance salesperson. • H) Linda is a bank teller and is active in the feminist movement.

Impact of Task and Statistical Sophistication on the Linda Problem Solving: Sometimes both task changes and knowledge

are needed!

9/29/2015 John W. Payne 45

• First, Linda is a bank teller

• Next, make the relationship less transparent, e.g., Linda is an insurance salesperson- such filler statements serve to make the relationship less transparent.

• Third, Linda is a bank teller and is active in the feminist movement.

% of

violations

8 items 2 items

Transparent

Naïve

undergrad 89 85

Sophisticat

ed Ph.D.

students

85 36


• Betting on a string of events

– RGRRR vs. GRGRRR (selected) vs. GRRRRR • That is adding an additional uncertain event G to RGRRR makes

that possibility more likely?

– Impact of real payoffs? • Answer – not much.

• Implications for Scenario Thinking - good stories can increase the perceived likelihood of an event by adding extra features. – Politicians know this well.

46 9/29/2015 John W Payne BA925


• Conclusion

– When will a logical principle be applied even when understood?

– Coherence versus correspondence views of good judgment?

– The role of “cognitive illusions” in judgment research?

47 9/29/2015 John W Payne BA925

Heuristic and Biases Reviewed

• Heuristics are highly economical and usually effective methods for dealing with judgments under certainty that are based on “natural” psychological assessments , but they can lead to systematic and predictable errors.

• Heuristics can be thought of as (intelligent and adaptive) strategies for searching through a problem space which simplify the process by disregarding some elements of the problem space.

• Judgments of probability vary in the degree to which they follow an extensional (decompositional, frequentistic, distributional) logic or a more singular, holistic, intuitive logic. Both types of reasoning occur when judging probabilities.

• Which reasoning process will be used to assess probabilities will depend upon the nature of the evidence, the formulation of the question, the transparency of the event structure, appeal of the heuristic, prior use, goals, and the sophistication of the respondent. However, we do not have a good theory of when certain heuristics will be more or less likely to be used.

48 9/29/2015 John W Payne BA925

Summary Flowcharts for the Three Heuristics (Adapted from Hastie & Dawes 2001)

49

Question?

Probe

Memory

Retrieve item(s)

that match or are

associated with probe

Assess ease

of recall

Respond

Availability Heuristic

Question?

Case or instance

description (feature

list).

Similarity Match:

How well does the

case feature list match

the category list?

Retrieve Category

representation (feature

list)

Estimate certainty of

category membership

based on similarity

Respond

Representativeness Heuristic

Question?

Search memory or

environment for evidence

Select most important

(remaining) evidence

Extract

Information

Is this first

item? Anchor on

information Adjust

Estimate

Yes No

Is there more

evidence?

Respond

No

Yes

Anchoring


Criticisms of the “heuristics and biases” viewpoint include:

• Over emphasis on error – – “If we are so dumb how did we reach the moon?” – Effort (Speed) tradeoffs?

• Lack of specification of the heuristics

• More needed on when and who

• Need for new methods such as response times and

task manipulations such as cognitive load that go beyond answers to simple word questions.

50 9/29/2015 John W Payne BA925

A newer area of study:

Emotions and Probabilistic Thought Do Interact

9/29/2015 John W Payne BA925 51

The Affect Heuristic

9/29/2015 John W Payne BA925 52

An emotional evaluation that occurs before conscious reasoning

“The stock market is more likely to go up on sunny than on cloudy days…”

Source: S. Benartzi UCLA

9/29/2015 John W Payne BA925 53

“He likes the project, so he thinks its costs are low

and its benefits are high. Nice example of the

affect heuristic.”

Kahneman, Daniel (2011-10-25). Thinking, Fast and Slow

(p. 104). Macmillan. Kindle Edition.

Emotions and the Use of Probabilities

• Cass Sunstein has argued “that when intense emotions are engaged, people tend to focus on the adverse outcome, not on its likelihood.” That is, they are not closely attuned to the probability that harm will occur. They exhibit “probably neglect.”

• Obvious implications for public policy, what about business decisions?

9/29/2015 John W Payne BA925 54

Questions?

55 9/29/2015 John W Payne BA925

Probabilistic reasoning II: Heuristics and biasesjpayne/ba525_powerpoint... · 2015-09-29 ·...

Documents

Transcript of Probabilistic reasoning II: Heuristics and biasesjpayne/ba525_powerpoint... · 2015-09-29 ·...