“An Experimental Study of Self-Relevance in Information Processing”
description
Transcript of “An Experimental Study of Self-Relevance in Information Processing”
“An Experimental Study of Self-Relevance in Information
Processing”
Seda Ertaç
University of ChicagoJune 30, 2007
Questions:1. Do individuals process information as they
should? How Bayesian are they? 2. Does this depend on the relevance of the
information to the self?
• Self-serving use of information?
Study Bayesian updating with different types of information
The Experiment
MAIN IDEA: Compare two theoretically equivalent updating problems (within-person)
Processing of information when information is:1. Self-relevant (relative performance feedback)
a) Addition task (11+25+34+40+91=?)b) Verbal task (GRE verbal)
2. Irrelevant to the self (a statistical urn problem)
DESIGN:
Performance Rounds
Task Performance Stage (Piece-rate compensation)
Submit Initial Estimates of Relative Performance (top, middle, or bottom of the distribution)
Receive performance feedback (“top” vs. “not top”, some new sessions with bottom/not bottom)
Submit revised estimates of performance
Accurate beliefs compensated using a quadratic scoring rule.
Belief Elicitation in the Performance Rounds:
Assign probabilities to each of the following three states:
MIDDLE 60%
BOTTOM 20%
TOP 20%
Non-Performance Rounds
Computer randomly picks one of three states (top, middle, bottom) according to a probability distribution->Comes from each subject’s own submitted priors in the task rounds
Subject sees the prior probabilities of each state being picked
Assigns a probability to each of the three states
Feedback is received (top/not top)
Revised probabilities about the three states are submitted.
Beliefs compensated using a quadratic scoring rule.
Procedures:UCLA undergraduates200 participants. 62% female.29% econ/business, 36% natural
sciences+engineering, 35% other social sciences
10 participants in each group. Depending on session: 16 or 24 rounds played1,5 hours
Initial Assessments in the Performance Rounds
1. More positive self-assessments in the addition task than the verbal (individual and group)
2. No difference in “confidence in assessment” across tasks
3. Women have less positive self-assessments than men, especially in the verbal task.
Performance Assessment
Index
Overconfidence
Index
Addition 1.05 (0.44) 0.05 (0.58)
Verbal 0.86 (0.45) -0.14 (0.556)
Accuracy of Judgment
0
20
40
60
80
100
120
140
160
Underconfident Correct Overconfident
Addition
Verbal
A c c u r a c y o f J u d g m e n t , A d d it io n
0
0 . 1
0 . 2
0 . 3
0 . 4
0 . 5
0 . 6
U n d e rc o n fid e n t C o r re c t O ve r c o n fid e n t
Per
cent
age
M a le s
F e m a le s
A c c u r a c y o f J u d g m e n t , V e r b a l
0
0 . 1
0 . 2
0 . 3
0 . 4
0 . 5
0 . 6
U n d e rc o n fid e n t C o r re c t O ve r c o n fid e n t
Per
cent
age
M a le s
F e m a le s
Response to Information in the Performance Rounds
When the information “not top” is received:
Actual-Bayesian Posteriors for the state “MIDDLE” (Bias)Addition Average: -0.026 (0.14)
Average Absolute: 0.10Verbal Average: -0.04 (0.14)
Average Absolute: 0.10
Response to Information in Performance Rounds
The probability assigned to “middle” is lower than it should be (z=-3.88, t=-3.75)
The bias is very significantly different from zero. (p=0.0000)
Relation between initial “self-confidence” and updating:
BIAS IN THE POSTERIOR FOR MIDDLE:When learn “not top”
When learn “not bottom”
Confident -0.005 -0.05
Not Confident
-0.045 0.004
Bias In the Non-Performance Rounds:
Non-performance rounds with objective priors
Bias AbsoluteBias
All “interesting” casesN=376
-0.004 0.075
Same initial priorsN=139
-0.001 0.045
Comparison of Performance and Non-Performance Rounds
Absolute bias is greater in the performance rounds (bias=0.10 vs. bias=0.7)
t=3.46, p=0.0005Restricting attention to “risk-neutral” cases,
we get (bias=0.10 vs. bias=0.045): t=6.15, p=0.000
Conservative View of BU:In the non-performance task, errors occur in either direction at
the same frequency.Performance rounds:
77.5% revised state correct19% revised state reflects overuse
3.5% revised state reflects underuseNon-Performance Rounds
85.5 % revised state correct 7% revised state reflects overuse
7.5 % revised state reflects underuse
Comparison of Performance and Non-Performance Rounds (within person)
• Look at cases where priors are exactly the same and feedback is also the same, also exclude the “trivial” cases of updating.
• Average difference=Pr(mid)NP-Pr(mid)P=0.044
• Both Wilcoxon and t-tests confirm that the probability attributed to “middle” is higher in the non-performance rounds (z=3.36, t=3.81)
• Are subjects better Bayesian updaters with self-relevant information?
The absolute bias is significantly higher in the performance rounds. (t=6.98, p=0.000)
Pr(mid)NP-Pr(mid)P
Absolute bias difference
0.045 0.052
N=109
Task 3—Belief Updating under AmbiguityOnly one of the objective probabilities is revealed.
MIDDLE X %
BOTTOM ?%
TOP ?%
Frequency of Types of Errors
Treatments
Type of Bias in Pr(mid)
Task Non-Performance
Ambiguity
(+) 35% 42% 41%
0 5% 12% 10%
(-) 60% 46% 49%
Does the QSR work? 1440 instances where we know true and submitted priors. 685 of them have the exact same priors for the 3 states.
In 30% of the deviations, the highest submitted probability is higher than the highest true probability, and the lowest submitted probability is lower.
In 9% of the deviations, the reverse is true. Risk-aversion does not seem to hold. (Data on ind.risk preferences->to be analyzed)Experience seems to help in reducing deviations
Summary:Not much support for use of information in a
self-serving way. If anything, individuals pay undue attention to
performance information (may be because they are not confident enough in their priors: ambiguity?).
Confidence and direction of information seems to also matter.
Individuals seem to be better Bayesian updaters when processing information irrelevant to themselves.
Posterior Probability DifferencePerformance vs. Non-Performance Rounds
(ProbNP-ProbP)
Gender and Use of Information • There is no significant difference between the
genders in terms of information processing in the statistical problem.
• No significant difference when they get the information top/not top (still, women more likely to go to “bottom”).
• Significant difference when the information is “not bottom” (men are much more likely to go to top, bias 0.05 versus ~0).
Gender and Use of Information(continued)
Does this come from confidence?
Among the subgroup of self-confident people, there is no significant difference.
A Conservative Measure of BU:
Revised Choice
Bayesian Choice
Middle Middle~Bottom
Bottom
Middle 162 (80%)
33(16%)
8(4%)
Middle~Bottom
10(43.5%)
3(13%)
10(43.5%)
Bottom 1(9%)
2(18%)
8(73%)
77% of the time people choose the correct state.
Initial Assessments in the Performance Rounds
• Overconfidence at the group level?• Overconfidence at the individual level?
States perceived as most likely by the subjects:
Choices Top Top~Middle Middle Middle~Bottom Bottom
Addition 19% 5% 61% 9% 6%
Verbal 10% 6 % 50% 17% 17%
Bias (restricted to RN)
Bias Absolute Bias (restricted to RN)
Absolute Bias
Performance N/A -0.035 N/A 0.10
Non-performance
~0 ~0 0.045 0.07
Ambiguity -0.015 -0.015 0.09 0.09
SessionsFirst Set of Sessions: 1-7, order: Performance+Non-PerformanceFeedback Type: Top/Not TopMore Sessions to Control For Some Issues:Sessions 9-11Perf+Non-Perf+AmbiguityFeedback Type: Bottom/Not BottomSessions 12-15Statistical+Task+Ambiguity Feedback Type: Top/Not TopSession 16: Accurate beliefs not compensated
Quadratic Scoring Rule:If really in the top:Payoff=50+100 pT-50(pT
2+pM2+pB
2)
If really in the middle:Payoff=50+100 pM-50(pT
2+pM2+pB
2)
If really in the bottom:Payoff=50+100 pB-50(pT
2+pM2+pB
2)
----------------------------------------------------- Payoffs are min. if assign 1 to a wrong statement, max. if
assign 1 to a true statement Submitting true beliefs is optimal for a risk-neutral
expected utility maximizer.
TOTAL EARNINGS=EARNINGS FROM PERFORMANCE (piece rate per question solved) +EARNINGS FROM PRE-INFO BELIEFS+
EARNINGS FROM POST-INFO BELIEFS
The Importance of Good Relative Performance in the Verbal Task, Females
The Importance of Good Relative Performance in the Addition Task, Females
Survey Responses and Gender 65% of men, 50% of women think addition is
more reflective of overall ability. 23% of men, 31% of women think addition is
easier. 73% of men, 63% of women say they enjoyed
the addition task more. Men care about the addition task more than
women do. Women and men are not different in their level
of caring about performance in the verbal task.
The Importance of Good Relative Performance in the Verbal Task, Males
The Importance of Good Relative Performance in the Addition Task
Hypotheses:1. Bayesian Updating:
Pr(T|“Top”)=1 Pr(T|“Not Top”)=0 Pr(M|“Not Top”)=PrM/(PrM+PrB) Pr(B|“Not Top”)=PrB/(PrM+PrB)Likewise for bottom/not bottom. 2. Same posteriors in the performance and non-
performance rounds, if the priors and the received information are the same.
Some Notes on Survey Responses
23% of men and 31% of women find the addition task to be more difficult.
65% of men think that addition is more indicative of general intelligence, whereas 50% of women do so.
The Importance of Good Relative Performance in the Verbal Task
The Importance of Good Relative Performance in the Addition Task, Males
Summary of Survey Answers
Men do better in the addition task (z=2.71)No significant difference in the verbal task. Men (women) get negative feedback 74%
(83%) of the time in the addition task, and 80% (79.5%) of the time in the verbal task. (in 74% of the instances where they are faced with feedback).
Things to be Done:Using the Existing Data:Individual-level analysis:• Beliefs• How they use information (do we have
consistently good “Bayesians”?)• Stated importance of relative performance
Design Issues• Order Effects (non-task first?)• Type of feedback (positive vs. negative)• Objective vs. subjective priors (ambiguity?)• Potential issues with the quadratic scoring
rule
=>Additional Sessions
End-of-Experiment Survey• Which task do you think is most reflective of a
person’s overall ability?• Rank tasks in terms of difficulty for you.• Rank tasks in terms of enjoyability for you.• How important was it to you to be better than
others in task X (0 to 10 scale)?• Did the positive(negative) information you
received affect your morale? (-10 to 10 scale)• Did the positive (negative) information you
received affect your subsequent performance? (-10 to 10 scale)
• Gender, major
Taking only the subsample of subjects that say they understood perfectly does not change the results about information processing.
Task Performance
Average # of questions solved:Men Women
Addition 4.95 4.30
Verbal 5.14 4.87
A Conservative Measure of BU, Non-Task:Revised Choice
Bayesian Choice
Middle Middle~Bottom
Bottom
Middle 227 (80%)
24(16%)
1(4%)
Middle~Bottom
7(43.5%)
27(13%)
0(43.5%)
Bottom 3(9%)
9(18%)
31(73%)
87% of the time people choose the correct state. (contrast with 77 in the task case)
Order Effects? Performance in the non-performance updating
task seems to be independent of order, no significant difference (p=0.40)
Performance in the performance updating task is also not significantly different. (p=0.75)
No clear trend in the data as rounds progress
Session with Accuracy Not Compensated
No significant difference in initial confidence levels (people taking seriously)
The incidence of submitting the objective priors is significantly higher.
Belief Updating When Accuracy is not Compensated8% bias with addition task, 13% with verbal task. (Same
direction, higher than in the case of incentives) Highly significant (p=0.004 for verbal, p=0.03 for
addition)On average, 0 bias in non-performance task. Not
significant. Likewise in the ambiguity. This suggests that risk-loving cannot be the whole
reason why people put too much probability to the bottom
Survey ResultsOverwhelming majority indicated high clarity
of instructions.1 person said he/she did not understand at all. 57% thinks addition more important for
ability.74% find the verbal task more difficult
Value of Information Information increases “confidence” in
estimation.
Information is always significantly valuable for payoffs.
No significant difference between task performance and statistical, but value higher for ambiguity rounds.
Related Work:Theoretical: Belief-dependent utility and self-serving use of info: e.g.
Koszegi (2005), Benabou and Tirole (2002), Eliaz (2002).
Experimental:Experiments on Bayesian updating: e.g. Dave and Wolfe
(2005), Jones and Sugden (2001), Ertac (2005)Most similar experimental design:Clark and Friesen (2003)Real effort task, predict future performance, compensated
by a quadratic scoring rule.