Accuracy of Small-Group Estimation
description
Transcript of Accuracy of Small-Group Estimation
Accuracy of Small-Group Estimation and the Wisdom of Crowds
Jenny ShiMichael D. Lee
Department of Cognitive ScienceUniversity of California, Irvine
Cheese tray knife set
Crafted of sustainable bamboo and sleek stainless steel, these five distinct blades carve up your brie and Roquefort with ease.
Price in Dollars ($1-$50):
Real price: $38
Experimental Stimuli
100 everyday items
• Images, description, prices
• Between $5-$45
• Obtained through shopping websites
Two sets of 50
• Items uniformly distributed by price
• Each item = 1 trial
Wisdom of Crowds
Groups of people can be smarter than the best individuals among them in the right conditions (Surowiecki, 2005)
A crowd can be “wise” when four conditions are met:
Diversity: Each individual has own unique view
Independence: Less relying on others
Decentralization: Draw info from different sources
Aggregation: Turn individual to collective decision
Preliminary Analysis: Individual vs. Crowds
• Examined the mean average deviation of the price estimations of 22 participants.
• Looking at each individual serves a lower bound
• Standard Wisdom of Crowd analysis serves as upper bound.
Current Study
What if there are only small groups available?
• groups of three individuals
• Between subjects design
• Priming, cooperative and competitive settings
Research questions:
• Which of these settings lead to better or worse estimation of the true prices?
• How does the best setting compare to the individuals and standard wisdom of crowd analysis (our preliminary analyses)?
Experimental Conditions
Condition type
1. With two primes (drawn from previous data sets)
2. Cooperate by hearing each other’s bids
3. Cooperate by agreeing on an estimate
4. Compete with each other by playing the Price is Right game
Participants cooperating or competing with each other estimated sequentially and systematically alternated between each trial.
The Price is Right
Rules: To win the game, player must bid closest to the retail price without going over.
Players can bid as high as they want, but they cannot bid the same amount as others or bid less than $1.
Price is Right encourages strategic estimation
Item for bid: Ipod
$150$165 $1
Cognitive model for competition estimate
(Lee & Shi, 2010)
Bottom line: Instead of aggregating the “raw” estimates from participants that competed, we used a cognitive model to infer their latent knowledge.
wx (a,b,c,μ,σ) πc (c | a,b,μ,σ)= w3 (a b,c,μ,σ)p (μ,σ | a,b,c)p (a,b,c | μ,σ ) p (μ,σ)
…blah blah blah.
Results for small group estimates
$9.36 $8.82 $8.79
Competitive Results
Competitive MAD: $8.05
Primed MAD: $9.36
Cooperative Average MAD: $8.82
Cooperative Consensus MAD: $8.79
Competitive estimate was better than both primed and cooperative
Summary of our results
Wisdom of crowds performs best
• Four conditions were present
Competitive outperformed both cooperative and priming.
• Competing participants discouraged to mimic other bids because of winning incentive.
• Participants that cooperate or were primed may be dependent on other participants in the group or additional information given.
Conclusion
Wisdom of crowd analysis is superior to any other aggregation method.
• Resourceful in extracting information from people.
Competition > Cooperation > Individual
• Groups perform better than individuals in estimation tasks.
• Cooperation worse than competition possibly because lack of independence.
Using cognitive models is an efficient way of combining knowledge across individuals.
• Helps us understand both the observed behavior and the reasoning behind it.
Thanks!
References
Lee, M.D., & Shi, J. (2010). The accuracy of small-group estimation and the wisdom of crowds. In R. Catrambone, & S. Ohlsson (Eds.), Proceedings of the 32nd Annual Conference of the Cognitive Science Society. Austin, TX: Cognitive Science Society.
Surowiecki, J. (2004). The wisdom of crowds. New York: Random House.
Extra Slides
Starbucks stainless steel tumbler
Enjoy your favorite Starbucks brew in this 10-oz. stainless steel tumbler bottle with convenient handles.
Price in Dollars ($1-$50):
Real price: $34
Optimal Price is Right Bidding
For just 3 players, bidding between $1-$50
1 10 20 30 40 500
0.2
0.4
0.6
0.8
1 Player 1
1 10 20 30 40 500
0.2
0.4
0.6
0.8
1 Player 3
1 10 20 30 40 500
0.2
0.4
0.6
0.8
1 Player 4
1 10 20 30 40 500
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1 Winning Probabilities
Toaster Inference
Participants were shown a $28 toaster
• Bid $31, $28, $1
• Mean of data is $20
• Mean of inferred latent pricedistribution is $29
1 10 20 30 40 50Bid
Pro
babi
lity
1 10 20 30 40 50 1 10 20 30 40 50