Lessons From An Oops At Consumer Reports Consumer Follow Experts; Ignore Invalid Information

Uri SimonsohnThe Wharton School 1

The paper in one slide: Jan 4th 2007: Consumer Reports on carseats Jan 18th: Retraction Unique opportunity: Do consumers continue

using Jan 4th info? Test on 6,000+ eBay auctions for carseats Main finding:

Full return to baseline My interpretation: voluntarily ignored info. Alt explanations

Information ‘depreciates’Post-retraction buyers didn’t knowKind-of alternative: Sellers’ behavior

2

Outline Background New information: release and retraction Auction data Main results Alternative specifications Conclusions

3

Can people voluntarily ignore information they possess?

Existing evidence:Debriefing paradigmHindsight biasAnchoringMock juries and inadmissible evidence

4

Debriefing ParadigmRoss, Lepper & Colleageus (JPSP 1975;1980) Critique of false feedback in PsychParadigm: Give false feedback on personality test Debrief: “feedback was false” Ask their beliefs …still influenced by retracted feedback

5

Anchoring Subjects asked to make numerical

estimateLength of Mississippi riverWTP for keyboard.

Asked first: is the amount larger or smaller than anchor.

Final estimate is correlated with anchor. Even when anchor is roulette or SS#

6

OPIM 690 Write down the last 2 digits of your SS#:__ Would you be willing to pay that amount for yearly access to NYTimes.com?

What is the most you would pay? _____

7

$29.84

$45.07

$0.00

$5.00

$10.00

$15.00

$20.00

$25.00

$30.00

$35.00

$40.00

$45.00

$50.00

Social Security # <50 Social Security # >50

Hindsight Bias People told some outcome Asked to estimate what those without

information would predict. Finding: estimates are biased towards

the to-be-ignored outcome.

Next: results from Fischhoff (1975)

8

9

Predicted ex-ante probabilities of subjects who "knew" outcome & actual probabilities of control

group

57%

38%

48%

27%

34%

21%

32%

12%

0%

10%

20%

30%

40%

50%

60%

70%

1 2 3 4

Outcome

Prob

abili

ty

Knew

Control

Mock Juries & inadmissible evidence Dozens of studies Random assignment across “jurors” Control: baseline evidence T1: control + extra evidence T2: T1 + extra evidence is inadmissible. Decisions by T2 fall between control and

T1.

10


12

January 4th, 2007

13

•Corr( rank 2007,rank 2005) = -.08

Retraction and empirical strategy Jan 18th : Oops! Outsourced, 30 vs 38 vs 70 MPH Unique opportunity to study:

1) Causal effects of expert adviceContributions:○ Individual level measures of WTP○ Simple identification strategy (wrong info)

Compared to- Discontinuities around discrete scores- Differences across sites- Timing

2) Ability of consumers to ignore retracted information.14

How would people learn of a new Consumer Rerports carseat rating?

Important because: 1) Face validity of quick market

reactions 2) Post-retraction awareness.

15

From CR to consumers. CR in print

Subscribers: slow○ Library got it 01/11○ They claim: letter for retraction○ Otherwise, not till May

Newstands: slower○ No retraction till May

cr.org Comscore 100k users15% of carseat buyers visit within 305% same dayNot a direct source of info

16

How about the media?

17

0

10

20

30

40

50

60

1/1/

2007

1/2/

2007

1/3/

2007

1/4/

2007

1/5/

2007

1/6/

2007

1/7/

2007

1/8/

2007

1/9/

2007

1/10

/200

71/

11/2

007

1/12

/200

71/

13/2

007

1/14

/200

71/

15/2

007

1/16

/200

71/

17/2

007

1/18

/200

71/

19/2

007

1/20

/200

71/

21/2

007

1/22

/200

71/

23/2

007

1/24

/200

71/

25/2

007

1/26

/200

71/

27/2

007

1/28

/200

71/

29/2

007

1/30

/200

7

# of

new

spap

er a

rticl

es

0

100

200

300

400

500

600

700

800

# of

TV/

Radi

o st

orie

s

Newspaper coverage

"Television & Radio Coverage"

Number of stories about “Consumer Reports” and “Carseats”sources: newsbank+lexisnexis

18

50+ Newspapers 600+ Stories

Internet coverage Can’t do same search for web-coverage Can use web.archive.org to check specific

sites. All major sites covered it

19

In Sum

CR info indirectly received via mediaFast Retracted information remained available

following retraction

I’d argue: Post-retraction buyers probably read stories before being retracted.

20


21

Why auctions Retailers don’t change prices often Few decision makers behind them Auctions:

1000s of DMs interactingPrices change continuously

Aside: Unexploited side to eBay data: pulse on

demand shocks.

22

Auctions Data 6 months: 3 before & 3 after

Many analyses focus on:○ Before: 3 weeks ○ During: 2 weeks ○ After: 3 weeks

Auctions: 6k Bids: 35k

23

Descriptive statistics

25

Table 1 - Descriptive Statistics by Carseat Model

Britax Companion

Safety 1st Designer

Peg Perego Viaggio

Graco Snug Ride

Baby Trend Adjustable

Back

Evenflo Discovery

Rankings (rating ) in Consumer Reports® In 2005 1st (90) 2nd (88) 3rd (70) 4th (69) 5th (55) 6th (45)

In 2007 [retracted] 7th (20) 9th (16) 4th (27) 2nd (61) 1st (64) 11th (0)

Means (standard deviation ) for key variables

Number of observations 606 243 1327 2682 312 301

Final Price (sold items) $99.5 $33.5 $87.1 $40.5 $56.4 $15.1(28.72) (17.85) (50.06) (27.25) (26.84) (10.03)

Shipping (sold items) $25.3 $21.3 $25.1 $20.3 $21.1 $19.8(8.41) (6.06) (18.11) (10.18) (10.81) (9.77)

Starting Price $64.22 $17.49 $48.14 $26.61 $45.48 $10.59(48.43) (20.76) (44.63) (27.62) (34.22) (9.77)

Percentage Sold 72.1% 62.1% 71.8% 70.9% 62.1% 60.5%

Number of bids (sold items) 13.19 10.48 13.19 9.98 12.11 7.98(8.57) (7.56) (10.21) (7.19) (8.70) (5.68)

Number of (paid) extra features included with listing 1.17 1.06 1.14 1.28 1.10 1.12(0.52) (0.23) (0.84) (0.75) (0.52) (0.74)

Percentage of items known to be new 70.1% 75.3% 33.8% 22.9% 51.6% 25.3%

BrandModel

Annoyance:

Shipping is only observed for sold items. Estimate OLS for sold items

(w/shipping) Estimate Tobit for all (w.o./shipping)

26

Outline or regression specifications Y: (tot.pricei/Avg.Pricei,k)

i:auction, k:carseat model

Time variables (dummies): Primarily: before, during, after. Also: biweekly dummies (next slide) Also: 3-day-dummies

Key predictor Primarily: ΔRanking Also: carseat-model-dummies

e.g. Y=OLS(during*ΔRanking , after* ΔRanking, controls) 27

-4%

-3%

-2%

-1%

0%

1%

2%

-6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7

Fortnights (biweeks) since/from new information

Pric

e ch

ange

per

pos

ition

lost

in ra

nkin

g

First: bird’s eye view Estimate Y=OLS(biweekly*ΔRanking)

1 observation every 14 days. Plot point estimates

28

3.98 SD

Next: more fine grained look

Time: before, during, after

29

30

Table 2. Regressions predicting final price of carseat auctionsRegression type OLS OLS OLS Tobit Tobit Tobit Tobit Tobit

Specification Base specification Adds controls forauction-design

Adds controls for competition

Includes unsold items Adds more weeks Full sample Full sample

New itemsFull sampleUsed items

Sample period (in weeks)3 Before 2 During 3 After

3 Before 2 During 3 After







0.159*** 0.145*** 0.150*** 0.180*** 0.159*** 0.169*** 0.238*** 0.128***

(0.034) (0.030) (0.032) (0.033) (0.033) (0.040) (0.083) (0.034)

0.063** 0.081*** 0.092*** 0.096** 0.081* 0.042 0.056 0.042

(0.029) (0.027) (0.031) (0.043) (0.044) (0.048) (0.077) (0.036)

-0.033*** -0.022* -0.020 -0.016* -0.009 -0.001 0.005 -0.003

(0.010) (0.011) (0.012) (0.009) (0.009) (0.009) (0.012) (0.008)

"During" * ΔRanking -0.028** -0.024** -0.026** -0.031*** -0.035*** -0.034*** -0.047*** -0.011(0.011) (0.011) (0.011) (0.010) (0.010) (0.012) (0.017) (0.014)

"After" * ΔRanking -0.001 0.003 0.001 -0.007 -0.008 -0.006 -0.010 -0.009(0.011) (0.011) (0.011) (0.010) (0.010) (0.011) (0.014) (0.012)

0.515*** 0.373*** 0.390*** 0.376*** 0.325*** 0.320***

(0.083) (0.062) (0.063) (0.086) (0.069) (0.045)

-0.048 -0.033 -0.028 -0.067 -0.098** -0.042

(0.048) (0.027) (0.025) (0.054) (0.041) (0.038)

Includes additional base? (1-yes, 0-no) 0.131** 0.142*** 0.139*** 0.061 0.028 0.022 -0.243** 0.031

(0.056) (0.045) (0.043) (0.047) (0.035) (0.026) (0.111) (0.030)

0.491*** 0.491*** 0.366*** 0.388*** 0.391*** 0.450*** 0.572***

(0.076) (0.076) (0.076) (0.075) (0.073) (0.069) (0.031)

0.036*** 0.036*** 0.033*** 0.032** 0.024* 0.005 0.023

(0.010) (0.009) (0.012) (0.013) (0.015) (0.035) (0.015)

Reserve price present (1-yes, 0-no) 0.243*** 0.243*** 0.104*** 0.147*** 0.154*** 0.282*** 0.122***

(0.025) (0.026) (0.037) (0.031) (0.020) (0.075) (0.016)

Offered buy-it-now (1-yes, 0-no) 0.007 0.006 0.126*** 0.101*** 0.099*** -0.010 0.106***

(0.032) (0.032) (0.031) (0.029) (0.021) (0.032) (0.024)

Numer of same-model auctions that day -0.002 -0.003 -0.001 0.003 0.007 -0.000

(0.002) (0.003) (0.004) (0.003) (0.006) (0.002)

-0.058 0.023 0.020 -0.033 -0.163* 0.130**

(0.080) (0.065) (0.069) (0.079) (0.097) (0.057)

Number of observations 1052 1052 1052 1438 2227 5471 2480 3346R2 / Pseudo-R2 .32 .47 .47 .30 .29 .26 .16 .16

Percentage of same-model auctions that day that are new

Auction's starting price (as % of model's average price)

Is item known to be new? (1-yes, 0-no)

Is item known to be used? (1-yes, 0-no)Excluded category: unknown status

"During" (1-yes, 0-no) Does auction end during the two weeks in which new ranking was valid?"After" (1-yes, 0-no) Does auction end after the retraction?

ΔRanking (2007-2005) Difference in Consumer Reports' safety of auctioned carseat model.

Number of (paid) extra features included with listing

Plotting time*dranking betas

31

-6%

-4%

-2%

0%

2%

4%

6%

baseline xb xb+ tobit tobit 12weeks

tobit 6months

new used

During After

``

So far: just before-after How quick are the reactions?

Y=OLS(3-day-dummies* Δranking)

32

price=OLS(3-day dummies * Δrank)omitted cat.: two previous weeks

-5%

-4%

-3%

-2%

-1%

0%

1%

2%(-6

,-4)

(-3,-1

)

(0,2

)

(3,5

)

(6,8

)

(9,1

1)

(12,

14)

(15,

17)

(18,

20)

(21,

23)

(24,

26)

(27,

29)

(30,

32)

(33,

35)

Days after new ranking (in 3-day intervals)

Poin

t est

imat

e fo

r 3-d

ay ti

me-

dum

my

inte

ract

ed

with

rank

ing-

chan

ge

New ranking released

Retracted

33

How about non-winning bids? Camerer et al (1989) “Curse of

Knowledge”Market forces reduce itRational agents trade more

Same here?Are non winning bidders ‘cursed’?

Unit of observation: auction bid Quantile Regression

34

Specification Bids are unit of observation. If more than one bid by same bidder,

take highest only. Estimate quantile regressions of:

bid $ = f(Time*ΔRanking) With quantiles at 20% ,40% ,60% ,80%.

35

36

Table 3. Quantile regressions with bid-amount as dependent variable

20% 40% 60% 80%

0.052*** 0.064*** 0.086*** 0.089**

(0.012) (0.016) (0.016) (0.035)

0.013 0.031* 0.063*** 0.042

(0.012) (0.016) (0.018) (0.033)

-0.001 -0.002 -0.001 -0.008

(0.003) (0.003) (0.006) (0.005)

"During" * ΔRanking -0.007** -0.013*** -0.021*** -0.022***(0.003) (0.004) (0.006) (0.007)

"After" * ΔRanking 0.002 0.004 0.003 0.006(0.004) (0.003) (0.006) (0.007)

Is item known to be new? (1-yes, 0-no) 0.122*** 0.238*** 0.255*** 0.221***

(0.022) (0.022) (0.036) (0.054)

-0.021 -0.018 -0.041 -0.162***

(0.025) (0.020) (0.038) (0.058)

Does auction include additional carseat-base? (1-yes, 0-no) 0.026 0.068* 0.071** 0.125**

(0.022) (0.036) (0.031) (0.053)

Auction's starting price (as % of model's average price) 0.869*** 0.776*** 0.687*** 0.558***

(0.019) (0.023) (0.022) (0.033)

Auction's duration in days -0.003 -0.002 -0.001 0.002

(0.002) (0.003) (0.004) (0.004)

Number of (paid) extra features included with listing 0.011** 0.019*** 0.021** 0.033*

(0.006) (0.007) (0.010) (0.017)

Reserve price present (1-yes, 0-no) 0.089*** 0.098*** 0.124*** 0.160***

(0.021) (0.032) (0.033) (0.043)

Offered buy-it-now (1-yes, 0-no) -0.024* -0.007 0.015 0.067***

(0.014) (0.017) (0.018) (0.023)

log (seller reputation + 2) -0.008*** -0.008*** -0.013*** -0.020***

(0.003) (0.002) (0.004) (0.005)

# of same-model auctions that day 0.000 0.001** 0.001*** 0.002**

(0.000) (0.000) (0.000) (0.001)

% of same-model new that day -0.006 -0.064*** -0.037 -0.034

(0.022) (0.022) (0.048) (0.047)

Number of observations 3529 3529 3529 3529Pseudo R2 .212 .219 .224 .211

Is item known to be used? (1-yes, 0-no)Note. Excluded category: unknown new/used status.

Quantile

"During" (1-yes, 0-no) Does auction end during the two weeks in which new ranking was valid?

"After" (1-yes, 0-no) Does auction end after the retraction?

ΔRanking (2007-2005) Difference in Consumer Reports' safety ranking of auctioned carseat model.

Plotting the betas

37

Point estimates for time*drank

-0.025

-0.02

-0.015

-0.01

-0.005

0

0.005

0.01

20% 40% 60% 80%

Quantil Regression for all Bids at X%

poin

t est

imat

e

During

After

38

Point estimates for time*drank

-0.03

-0.025-0.02

-0.015

-0.01

-0.0050

0.005

0.01

20% 40% 60% 80%

Quantile Regression for all Bids

poin

t est

imat

e

During

After

Dividing point estimates by average bid % at quantile

From ΔRanking to model-dummies Previous analyses: Impose

Δ%price=b* ΔRanking Don’t allow for heterogeneity in effect Next: estimates by model. Plot avg(OLS,Tobit)

40

41

-8%-10% -10%

13%

35%

-30%

0% 2%6%

0%

-25%

0%

-40%

-30%

-20%

-10%

0%

10%

20%

30%

40%

% C

hang

e in

pric

es w

ith re

spec

t to

Befo

re p

erio

d

DuringAfter

Britax (1st → 7th)

Safety 1st(2nd → 9th)

Evenflo(6th → 11th)

Baby Trend(5th → 1st)

Graco(4th → 2nd)

Peg Perego(3rd → 4th)

Price=f(demand AND supply)

43

Starting Price Number of paid features

45

-4%

-2%

0%

2%

4%

-6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6

Biweeks before/after new information

% c

hang

e in

st

artin

g-pr

ice

-0.10

-0.05

0.00

0.05

0.10

Chan

ge in

# o

f pai

d fe

atur

es

Starting Price Paid features

# of items for sale% New

46

-10%

-5%

0%

5%

10%

-6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6

Biweeks before/after new information

% c

hang

e

Number of carseats % new

Summary of evidence Biweekly: biggest price drop in 6 months During vs. After:

Market responded to information Ceased to once retracted

3-day: Market respond virtually immediately

Quantile regressions Bidders across the full spectrum do so.

Carseat dummies Every carseat (6/6) exhibits the pattern

Supply: No evidence of changes in supply

48

Interpretation Consumers successfully ignored

information they possessed once it was retracted.

49

Alternative Explanations1) Knowledge depreciates

…& coincides w/retractionBut: 3-day graphs

2) Buyers never knewRetracted information still available online- Evenflo

50

Why cursed in the lab but not here? Field, but not lab: credible instruction to

ignore.Mock juries & substantive instructionsDebriefing paradigm & credible instructionShould you really ignore info in

○ Hindsight Bias○ Knowledge curse

Field, but not lab: DM control informationDilution effect goes away when you can scratch

irrelevant infoHindsight and anchoring attenuate when explicitly

consider alternatives.51

Future research

Run lab experiments explicitly manipulating variables that differ in vs. outside the lab.

52

Lessons From An Oops At Consumer Reports Consumer Follow Experts; Ignore Invalid Information

Documents

Transcript of Lessons From An Oops At Consumer Reports Consumer Follow Experts; Ignore Invalid Information