RepliCHI: Graduate Student Perspectives
-
Upload
michael-bernstein -
Category
Documents
-
view
925 -
download
4
description
Transcript of RepliCHI: Graduate Student Perspectives
![Page 1: RepliCHI: Graduate Student Perspectives](https://reader033.fdocuments.in/reader033/viewer/2022061211/547a4fa6b4af9f92208b457f/html5/thumbnails/1.jpg)
RepliCan’tGraduate Student Perspectives
MIT HUMAN-COMPUTER INTERACTION
Michael BernsteinMIT Computer Science and Artificial Intelligence [email protected] | @msbernst
![Page 2: RepliCHI: Graduate Student Perspectives](https://reader033.fdocuments.in/reader033/viewer/2022061211/547a4fa6b4af9f92208b457f/html5/thumbnails/2.jpg)
Speaking for every graduate student in SIGCHI, I can say one thing:
![Page 3: RepliCHI: Graduate Student Perspectives](https://reader033.fdocuments.in/reader033/viewer/2022061211/547a4fa6b4af9f92208b457f/html5/thumbnails/3.jpg)
I can’t speak for everygraduate studentin SIGCHI.
![Page 4: RepliCHI: Graduate Student Perspectives](https://reader033.fdocuments.in/reader033/viewer/2022061211/547a4fa6b4af9f92208b457f/html5/thumbnails/4.jpg)
My Unassailable, Extremely ScientificData Collection Protocol:Survey of CHI student volunteers, CHI-students ACM listserv, and snowballed recruitment through Facebook and Twitter.
N=93 responses
![Page 5: RepliCHI: Graduate Student Perspectives](https://reader033.fdocuments.in/reader033/viewer/2022061211/547a4fa6b4af9f92208b457f/html5/thumbnails/5.jpg)
Have you ever replicateda study or system?
No
Yes
0 10 20 30 40 50 60 70 80 90
Respondents
17% yes, 83% no
![Page 6: RepliCHI: Graduate Student Perspectives](https://reader033.fdocuments.in/reader033/viewer/2022061211/547a4fa6b4af9f92208b457f/html5/thumbnails/6.jpg)
Do you ever plan to replicatea study or system?
38% “Hell yes”, 62% “Hell no”
Hell no
Hell yes
0 10 20 30 40 50 60 70 80 90
Respondents
![Page 7: RepliCHI: Graduate Student Perspectives](https://reader033.fdocuments.in/reader033/viewer/2022061211/547a4fa6b4af9f92208b457f/html5/thumbnails/7.jpg)
Set Yourself Apart
The point of research is to come up with exciting ideas that solve problems. Not copy others’ work.
“
”
![Page 8: RepliCHI: Graduate Student Perspectives](https://reader033.fdocuments.in/reader033/viewer/2022061211/547a4fa6b4af9f92208b457f/html5/thumbnails/8.jpg)
Set Yourself Apart
I'm more creative than that.“ ”
![Page 9: RepliCHI: Graduate Student Perspectives](https://reader033.fdocuments.in/reader033/viewer/2022061211/547a4fa6b4af9f92208b457f/html5/thumbnails/9.jpg)
There’s No Reward
New studies confirming old studies have no chance of publication.
“”
![Page 10: RepliCHI: Graduate Student Perspectives](https://reader033.fdocuments.in/reader033/viewer/2022061211/547a4fa6b4af9f92208b457f/html5/thumbnails/10.jpg)
There’s No Reward
Reviewers […] didn't feel replication was necessary even though the original study was specific to a single company.
“
”
![Page 11: RepliCHI: Graduate Student Perspectives](https://reader033.fdocuments.in/reader033/viewer/2022061211/547a4fa6b4af9f92208b457f/html5/thumbnails/11.jpg)
There’s No Reward
I very frequently see reviewers criticize submissions for presenting results that are “not novel” or “have already been shown”.
“
”
![Page 12: RepliCHI: Graduate Student Perspectives](https://reader033.fdocuments.in/reader033/viewer/2022061211/547a4fa6b4af9f92208b457f/html5/thumbnails/12.jpg)
There’s No Reward
I very frequently see reviewers criticize submissions for presenting results that are “not novel” or “have already been shown”.
“
”
![Page 13: RepliCHI: Graduate Student Perspectives](https://reader033.fdocuments.in/reader033/viewer/2022061211/547a4fa6b4af9f92208b457f/html5/thumbnails/13.jpg)
Responding to IncentivesOpen access and replication. A true scientist’s ideals, but see:The grad student must conform.
“”
![Page 14: RepliCHI: Graduate Student Perspectives](https://reader033.fdocuments.in/reader033/viewer/2022061211/547a4fa6b4af9f92208b457f/html5/thumbnails/14.jpg)
Haikusstudies should break
ground replication wastes our
timelet's find new
problems
“”
![Page 15: RepliCHI: Graduate Student Perspectives](https://reader033.fdocuments.in/reader033/viewer/2022061211/547a4fa6b4af9f92208b457f/html5/thumbnails/15.jpg)
Haikus
think analyzingCMC is tough? try it
reproducibly!
“”
![Page 16: RepliCHI: Graduate Student Perspectives](https://reader033.fdocuments.in/reader033/viewer/2022061211/547a4fa6b4af9f92208b457f/html5/thumbnails/16.jpg)
Haikus
repeat to be surewe stand on giants’
shouldersbut do so on faith
“”
![Page 17: RepliCHI: Graduate Student Perspectives](https://reader033.fdocuments.in/reader033/viewer/2022061211/547a4fa6b4af9f92208b457f/html5/thumbnails/17.jpg)
83% have not62% will not
(But we’ll need to replicate the study to be sure.)
![Page 18: RepliCHI: Graduate Student Perspectives](https://reader033.fdocuments.in/reader033/viewer/2022061211/547a4fa6b4af9f92208b457f/html5/thumbnails/18.jpg)
![Page 19: RepliCHI: Graduate Student Perspectives](https://reader033.fdocuments.in/reader033/viewer/2022061211/547a4fa6b4af9f92208b457f/html5/thumbnails/19.jpg)
Why?
repeat to be surewe stand on giant's shouldersbut do so on faith
“
”
![Page 20: RepliCHI: Graduate Student Perspectives](https://reader033.fdocuments.in/reader033/viewer/2022061211/547a4fa6b4af9f92208b457f/html5/thumbnails/20.jpg)
Why?
Replication is a critical component of scientific research, and it should be encouraged and rewarded. The lack of it is detrimental to the scientific soundness of our discipline.
“
”
![Page 21: RepliCHI: Graduate Student Perspectives](https://reader033.fdocuments.in/reader033/viewer/2022061211/547a4fa6b4af9f92208b457f/html5/thumbnails/21.jpg)
Why?
we lack the time forreplication of studiesjust review strictly
“
”
![Page 22: RepliCHI: Graduate Student Perspectives](https://reader033.fdocuments.in/reader033/viewer/2022061211/547a4fa6b4af9f92208b457f/html5/thumbnails/22.jpg)
Why?
think analyzingCMC is tough? try it reproducibly!
“
”
![Page 23: RepliCHI: Graduate Student Perspectives](https://reader033.fdocuments.in/reader033/viewer/2022061211/547a4fa6b4af9f92208b457f/html5/thumbnails/23.jpg)
Why?
It seems like the best outcome for a replication, rather than success, is actually a refutation of the original study.
“
”
![Page 24: RepliCHI: Graduate Student Perspectives](https://reader033.fdocuments.in/reader033/viewer/2022061211/547a4fa6b4af9f92208b457f/html5/thumbnails/24.jpg)
Why?
Because its's a waste of time: HCI studies are so small, I know they surely WON'T replicate, so why bother!
“
”
![Page 25: RepliCHI: Graduate Student Perspectives](https://reader033.fdocuments.in/reader033/viewer/2022061211/547a4fa6b4af9f92208b457f/html5/thumbnails/25.jpg)
Why?
CHI is too cutting edge for things like replication, or good science, or careful analysis, or the humility to accept that other topics besides Fitts' Law deserve dozens of nearly-identical studies.
“
”
![Page 26: RepliCHI: Graduate Student Perspectives](https://reader033.fdocuments.in/reader033/viewer/2022061211/547a4fa6b4af9f92208b457f/html5/thumbnails/26.jpg)
![Page 27: RepliCHI: Graduate Student Perspectives](https://reader033.fdocuments.in/reader033/viewer/2022061211/547a4fa6b4af9f92208b457f/html5/thumbnails/27.jpg)
Why?
I do not intend on taking the risk of replicating some of my favorite works unless I see evidence that the CHI community supports such a thing.
“
”
![Page 28: RepliCHI: Graduate Student Perspectives](https://reader033.fdocuments.in/reader033/viewer/2022061211/547a4fa6b4af9f92208b457f/html5/thumbnails/28.jpg)
There’s No RewardCase A. Confirmation of the earlier results (very boring)
Case B. Conflict with earlier results (unpublishable problem)
“
”