DATA COLLECTION IN A FLAT WORLD: THE · PDF filedata collection in a flat world: the strengths...
Transcript of DATA COLLECTION IN A FLAT WORLD: THE · PDF filedata collection in a flat world: the strengths...
![Page 1: DATA COLLECTION IN A FLAT WORLD: THE · PDF filedata collection in a flat world: the strengths and weaknesses of mechanical turk samples goodman, cryder, & cheema . questions about](https://reader031.fdocuments.in/reader031/viewer/2022030503/5ab006c67f8b9a59478e18a0/html5/thumbnails/1.jpg)
Joseph K. Goodman, Washington University in St. Louis
Cynthia E. Cryder, Washington University in St. Louis
Amar Cheema, University of Virginia
DATA COLLECTION IN A FLAT WORLD: THE STRENGTHS AND WEAKNESSES
OF MECHANICAL TURK SAMPLES
GOODMAN, CRYDER, & CHEEMA
![Page 2: DATA COLLECTION IN A FLAT WORLD: THE · PDF filedata collection in a flat world: the strengths and weaknesses of mechanical turk samples goodman, cryder, & cheema . questions about](https://reader031.fdocuments.in/reader031/viewer/2022030503/5ab006c67f8b9a59478e18a0/html5/thumbnails/2.jpg)
QUESTIONS ABOUT MTURK
• Problems/Questions • Do they really pay attention? • Who is doing these surveys?
• Are they different from the rest of us? Are they ‘normal’?
• In other words, do they show the same biases as normal people?
GOODMAN, CRYDER, & CHEEMA
![Page 3: DATA COLLECTION IN A FLAT WORLD: THE · PDF filedata collection in a flat world: the strengths and weaknesses of mechanical turk samples goodman, cryder, & cheema . questions about](https://reader031.fdocuments.in/reader031/viewer/2022030503/5ab006c67f8b9a59478e18a0/html5/thumbnails/3.jpg)
Research questions Do they really pay attention? Who is doing these surveys?
• Motivation & Cognition • Do they pay attention?
• Valuing Money & Consumption • Do they value money in odd ways?
• Big Five and other Individual Differences
• Do they have personality differences? • Judgment and Decision Making
• Are they “normal” (aka, irrational like the rest of us)? Do they show normal biases?
GOODMAN, CRYDER, & CHEEMA
![Page 4: DATA COLLECTION IN A FLAT WORLD: THE · PDF filedata collection in a flat world: the strengths and weaknesses of mechanical turk samples goodman, cryder, & cheema . questions about](https://reader031.fdocuments.in/reader031/viewer/2022030503/5ab006c67f8b9a59478e18a0/html5/thumbnails/4.jpg)
TEST ACROSS TWO STUDIES
• Study 1 • Compare MTurkers to a community sample in Pittsburgh • Mturk: $.10, ~10 min study via Qualtrics
• Study 2 • Compare MTurkers to student sample from Wash U in
St. Louis • MTurk: $.20, ~16 min study via Qualtrics • Students: Paper-and-pencil, 14 min, and web-based 13
min
GOODMAN, CRYDER, & CHEEMA
![Page 5: DATA COLLECTION IN A FLAT WORLD: THE · PDF filedata collection in a flat world: the strengths and weaknesses of mechanical turk samples goodman, cryder, & cheema . questions about](https://reader031.fdocuments.in/reader031/viewer/2022030503/5ab006c67f8b9a59478e18a0/html5/thumbnails/5.jpg)
• Age: 31 MTurk vs. 19.4 students
• Female: 43% MTurk vs. 59% Students
• 100% of students were students and had “some college”!
• 24% MTurk’ers were students
• 92% MTurk’ers “some college”
• Age: 33.5 on average
• Female: 59% MTurk vs. 52% Comm
• Education: Modal and Median 4 year bachelor’s
• 26% non-US (3% Canada, 19% India)
• More ESL: 28% vs. 11%*
DEMOGRAPHICS
Study 1 - Mturk vs. Community
GOODMAN, CRYDER, & CHEEMA
![Page 6: DATA COLLECTION IN A FLAT WORLD: THE · PDF filedata collection in a flat world: the strengths and weaknesses of mechanical turk samples goodman, cryder, & cheema . questions about](https://reader031.fdocuments.in/reader031/viewer/2022030503/5ab006c67f8b9a59478e18a0/html5/thumbnails/6.jpg)
STUDY 1 - COGNITION & MOTIVATION THE IMC Research in decision making shows that people…prefer not to pay attention.... If you are reading this question and have read all the other questions, please select the box marked 'other' and write 'decision making' on the line below. Do not select your own opinions and behaviors. Thank you for participating and taking the time to read through the questions carefully!
What was this study about? • Your own opinions and behaviors
• Lions (Study 2: Political history) • Tigers (Study 2: Friends’ behaviors) • Other ___________________________
(adapted from Oppenheimer, Meyvis, & Davidenko, 2009) GOODMAN, CRYDER, & CHEEMA
![Page 7: DATA COLLECTION IN A FLAT WORLD: THE · PDF filedata collection in a flat world: the strengths and weaknesses of mechanical turk samples goodman, cryder, & cheema . questions about](https://reader031.fdocuments.in/reader031/viewer/2022030503/5ab006c67f8b9a59478e18a0/html5/thumbnails/7.jpg)
STUDY 1 & 2 - COGNITION & MOTIVATION IMC
GOODMAN, CRYDER, & CHEEMA
81.10% 84.50%
66.20%
88.50%
50% 55% 60% 65% 70% 75% 80% 85% 90% 95%
MTurk Community MTurk Student
Study 1 Study 2
Correct Answers
ns (X2 < 1) p < .05
![Page 8: DATA COLLECTION IN A FLAT WORLD: THE · PDF filedata collection in a flat world: the strengths and weaknesses of mechanical turk samples goodman, cryder, & cheema . questions about](https://reader031.fdocuments.in/reader031/viewer/2022030503/5ab006c67f8b9a59478e18a0/html5/thumbnails/8.jpg)
COGNITION & MOTIVATION THE CRT & SYSTEM 2 PROCESSING
• Cognitive Reflection Test • 3 questions
A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?
(Frederick 2005) GOODMAN, CRYDER, & CHEEMA
Other web-studies = 1.10
1.17 0.96
1.23 1.7
0
0.5
1
1.5
2
2.5
3
MTurk Commun. MTurk Student
Study 1 Study 2
# of Correct Answers
p < .05 p > .2
![Page 9: DATA COLLECTION IN A FLAT WORLD: THE · PDF filedata collection in a flat world: the strengths and weaknesses of mechanical turk samples goodman, cryder, & cheema . questions about](https://reader031.fdocuments.in/reader031/viewer/2022030503/5ab006c67f8b9a59478e18a0/html5/thumbnails/9.jpg)
COGNITION & MOTIVATION NFC
GOODMAN, CRYDER, & CHEEMA
• Need for Cognition • 18 questions • “I would prefer complex
to simple problems. • “I like to have the
responsibility of handling a situation that requires a lot of thinking”
63.14
70.85
55
60
65
70
75
MTurk Community
Study 1
p < .001
61.77 62.7
55
60
65
70
75
MTurk Student
Study 2
ns
![Page 10: DATA COLLECTION IN A FLAT WORLD: THE · PDF filedata collection in a flat world: the strengths and weaknesses of mechanical turk samples goodman, cryder, & cheema . questions about](https://reader031.fdocuments.in/reader031/viewer/2022030503/5ab006c67f8b9a59478e18a0/html5/thumbnails/10.jpg)
VALUING MONEY & CONSUMPTION TVM
GOODMAN, CRYDER, & CHEEMA
• Time Value of Money • Would you go an entire day (24
hours) without drinking liquids for • $5? (yes/no) • $25? • $100? • $1000?
• Would you complete a 1-hour survey for
• $5? • $25? • $100? • $1000?
(Cryder and Loewenstein 2010)
5.13 4.46
1 2 3 4 5 6
MTurk Community
Study 1
p < .05
5.49 5.27
1 2 3 4 5 6
MTurk Student
Study 2
ns # o
f “ye
s” an
swer
s
![Page 11: DATA COLLECTION IN A FLAT WORLD: THE · PDF filedata collection in a flat world: the strengths and weaknesses of mechanical turk samples goodman, cryder, & cheema . questions about](https://reader031.fdocuments.in/reader031/viewer/2022030503/5ab006c67f8b9a59478e18a0/html5/thumbnails/11.jpg)
GOODMAN, CRYDER, & CHEEMA
• 4 questions on spending • Spendthrift:
trouble limiting spending. Often spend money when they would do better not to.
• Tightwad: trouble spending money. Spending money makes them anxious, often don't spend money on things they should spend on.
VALUING MONEY & CONSUMPTION TIGHTWAD-SPENDTHRIFT SCALE
13.69
15.86
12 13 14 15 16 17
MTurk Community
Study 1
p < .01
14.56 14.05
12 13 14 15 16 17
MTurk Student
Study 2
ns Sp
endt
hrift
(vs.
Tigh
twad
)
![Page 12: DATA COLLECTION IN A FLAT WORLD: THE · PDF filedata collection in a flat world: the strengths and weaknesses of mechanical turk samples goodman, cryder, & cheema . questions about](https://reader031.fdocuments.in/reader031/viewer/2022030503/5ab006c67f8b9a59478e18a0/html5/thumbnails/12.jpg)
GOODMAN, CRYDER, & CHEEMA
• 9 questions • e.g., My life would be
better if I owned certain things I don't have.
VALUING MONEY & CONSUMPTION MATERIAL VALUES SCALE (MVS)
20 21 22 23 24 25 26 27
MTurk Community
Matierialism
p = .05
Study 2: no difference on Materialism Scale (27.57 vs. 27.69)
![Page 13: DATA COLLECTION IN A FLAT WORLD: THE · PDF filedata collection in a flat world: the strengths and weaknesses of mechanical turk samples goodman, cryder, & cheema . questions about](https://reader031.fdocuments.in/reader031/viewer/2022030503/5ab006c67f8b9a59478e18a0/html5/thumbnails/13.jpg)
STUDY 2 – VALUING MONEY & CONSUMPTION PRESENT BIAS
GOODMAN, CRYDER, & CHEEMA
0
0.5
1
1.5
2
2.5
3
3.5
Delay $15 for 3 months
Delay $15 for 12 months
Expedite $15 for 3 months
Expedite $15 for 12 months
Disc
ount
Rat
e*
Within-subject
MTurk Student
*continuously compounded discount rates (Thaler, 1981; Lowenstein, 1988; Malkoc & Zauberman, 2006)
![Page 14: DATA COLLECTION IN A FLAT WORLD: THE · PDF filedata collection in a flat world: the strengths and weaknesses of mechanical turk samples goodman, cryder, & cheema . questions about](https://reader031.fdocuments.in/reader031/viewer/2022030503/5ab006c67f8b9a59478e18a0/html5/thumbnails/14.jpg)
STUDY 1 - JDM ANCHORING & ADJUSTMENT
• Last 2 digits of phone number • # countries of Africa
• 10% MTurk’ers guess correct! (54)
GOODMAN, CRYDER, & CHEEMA
Community: β = .45, p < .01 MTurk: β = .06, ns
![Page 15: DATA COLLECTION IN A FLAT WORLD: THE · PDF filedata collection in a flat world: the strengths and weaknesses of mechanical turk samples goodman, cryder, & cheema . questions about](https://reader031.fdocuments.in/reader031/viewer/2022030503/5ab006c67f8b9a59478e18a0/html5/thumbnails/15.jpg)
STUDY 2 - JDM ANCH. & ADJ. (FOLLOW-UP STUDY)
GOODMAN, CRYDER, & CHEEMA
• Instructions vs. Compensation • Main effect for instructions
“…do NOT use external sources…”
• Main effect for compensation ($0, $.10, $1)
0%
10%
20%
30%
40%
50%
60%
70%
No Payment $0.10 $1.00
Control
"Do NOT search"
![Page 16: DATA COLLECTION IN A FLAT WORLD: THE · PDF filedata collection in a flat world: the strengths and weaknesses of mechanical turk samples goodman, cryder, & cheema . questions about](https://reader031.fdocuments.in/reader031/viewer/2022030503/5ab006c67f8b9a59478e18a0/html5/thumbnails/16.jpg)
STUDY 2 - JDM RISK
GOODMAN, CRYDER, & CHEEMA
• Four Gambles • Risk Averse for gains, both big and small • Certainty Effect: Preference decreases with small probabilities (students
especially) • Risk seeking for losses
$3 @ 100% (vs. $4 @80%)
$240 @100% (vs. $320 @ 80%)
$3 @ 25% (vs. $4 @ 20%)
-‐$3 @ 100% (vs. -‐4 @ 80%)
MTurk 0.85 0.86~ 0.52* 0.24
Student 0.82 0.77 0.24 0.21
* p < .05 ~ p < .06
![Page 17: DATA COLLECTION IN A FLAT WORLD: THE · PDF filedata collection in a flat world: the strengths and weaknesses of mechanical turk samples goodman, cryder, & cheema . questions about](https://reader031.fdocuments.in/reader031/viewer/2022030503/5ab006c67f8b9a59478e18a0/html5/thumbnails/17.jpg)
• Less extroverted • Less emotionally stable • Lower self-esteem • Less happy
(satisfaction with life)
• No consistent differences: • Agreeableness • Conscientiousness • Openness to
experiences • Self-control • Maximizer-satisficer
INDIVIDUAL DIFFERENCES MTURK’ERS ARE (BOTH STUDIES)..
GOODMAN, CRYDER, & CHEEMA
![Page 18: DATA COLLECTION IN A FLAT WORLD: THE · PDF filedata collection in a flat world: the strengths and weaknesses of mechanical turk samples goodman, cryder, & cheema . questions about](https://reader031.fdocuments.in/reader031/viewer/2022030503/5ab006c67f8b9a59478e18a0/html5/thumbnails/18.jpg)
• Only 1 IMC x MTurk interaction (TW-ST)
• Including Everyone: Emot. Stab. marginal (p=.07)
• ESL and non-US more likely to fail IMC
• MTurk x ESL: CRT • Fitler by ESL: Emot. Stab. (F < 1)
and conscientiousness (p=.06) • Filter by non-US: Emot. Stab. (p>.
15) and conscientiousness (p>.2)
• Only 1 IMC x MTurk interaction (emotional stability)
• Including everyone: Emot. Stab. (p=.15) ns and MVS (p=.15) and
• ESL and non-US more likely to fail IMC
• MTurk x ESL: Emot. Stab. • Filter by ESL: MVS ns (p>.2)
• Filter by non-US: MVS ns (F<1), Emot. Stab. ns (p>.1)
IMC, ESL, AND NON-US PARTICIPANTS FEW DIFFERENCES Study 1 Study 2
GOODMAN, CRYDER, & CHEEMA
![Page 19: DATA COLLECTION IN A FLAT WORLD: THE · PDF filedata collection in a flat world: the strengths and weaknesses of mechanical turk samples goodman, cryder, & cheema . questions about](https://reader031.fdocuments.in/reader031/viewer/2022030503/5ab006c67f8b9a59478e18a0/html5/thumbnails/19.jpg)
• MTurk’ers are Different! • They pay attention, but
students in the lab are best • Views of money and spending
are different than community (but not different than students)
• More tightwad
• Willing to do more tasks for money
• More materialistic
• Some personality differences
• But MTurk’ers show the same biases • Present biased, delay/
expedite asymmetry • Risk seeking for losses, risk
averse for gains • Respond to honestly pleas
CONCLUSIONS
GOODMAN, CRYDER, & CHEEMA
![Page 20: DATA COLLECTION IN A FLAT WORLD: THE · PDF filedata collection in a flat world: the strengths and weaknesses of mechanical turk samples goodman, cryder, & cheema . questions about](https://reader031.fdocuments.in/reader031/viewer/2022030503/5ab006c67f8b9a59478e18a0/html5/thumbnails/20.jpg)
Happy MTurkin'
GOODMAN, CRYDER, & CHEEMA