Hadas + mitchoweb.mit.edu/hackl/www/lab/turkshop/slides/week4.pdf · Hadas + mitcho {hkotek,...
Transcript of Hadas + mitchoweb.mit.edu/hackl/www/lab/turkshop/slides/week4.pdf · Hadas + mitcho {hkotek,...
![Page 1: Hadas + mitchoweb.mit.edu/hackl/www/lab/turkshop/slides/week4.pdf · Hadas + mitcho {hkotek, mitcho}@mit.edu Hackl Lab Turkshop April 2013 . Today..](https://reader035.fdocuments.in/reader035/viewer/2022071102/5fdc087aa603e24a0f2effe5/html5/thumbnails/1.jpg)
Turk data analysis
Hadas + mitcho
{hkotek, mitcho}@mit.edu
Hackl Lab Turkshop
April 2013
![Page 2: Hadas + mitchoweb.mit.edu/hackl/www/lab/turkshop/slides/week4.pdf · Hadas + mitcho {hkotek, mitcho}@mit.edu Hackl Lab Turkshop April 2013 . Today..](https://reader035.fdocuments.in/reader035/viewer/2022071102/5fdc087aa603e24a0f2effe5/html5/thumbnails/2.jpg)
Today..
Goal: prepare data for analysis, get (basic) results.
The Manage Tab
Approve subjects
Get data
Decode your results
R!
R basics
Prepare data for analysis
Calculate accuracy
Discard subjects
Look at results
2
![Page 3: Hadas + mitchoweb.mit.edu/hackl/www/lab/turkshop/slides/week4.pdf · Hadas + mitcho {hkotek, mitcho}@mit.edu Hackl Lab Turkshop April 2013 . Today..](https://reader035.fdocuments.in/reader035/viewer/2022071102/5fdc087aa603e24a0f2effe5/html5/thumbnails/3.jpg)
NOT Today..
Statistics
Graphics
…. those need to fit your design and data
There is no ‘one size fits all’
Requires more knowledge (take a stats course!)
3
![Page 4: Hadas + mitchoweb.mit.edu/hackl/www/lab/turkshop/slides/week4.pdf · Hadas + mitcho {hkotek, mitcho}@mit.edu Hackl Lab Turkshop April 2013 . Today..](https://reader035.fdocuments.in/reader035/viewer/2022071102/5fdc087aa603e24a0f2effe5/html5/thumbnails/4.jpg)
Materials
These slides and materials on ESSL website:
http://web.mit.edu/hackl/www/lab/turkshop/
Download and unzip “week4-results” in examples.
And if you haven’t done so, install R 3.0 and RStudio 0.97:
http://cran.r-project.org/
http://www.rstudio.com/ide/download/desktop
4
![Page 5: Hadas + mitchoweb.mit.edu/hackl/www/lab/turkshop/slides/week4.pdf · Hadas + mitcho {hkotek, mitcho}@mit.edu Hackl Lab Turkshop April 2013 . Today..](https://reader035.fdocuments.in/reader035/viewer/2022071102/5fdc087aa603e24a0f2effe5/html5/thumbnails/5.jpg)
The Manage tab
5
Monitor the progress of your experiment
Approve/reject subjects
Get your data!
![Page 6: Hadas + mitchoweb.mit.edu/hackl/www/lab/turkshop/slides/week4.pdf · Hadas + mitcho {hkotek, mitcho}@mit.edu Hackl Lab Turkshop April 2013 . Today..](https://reader035.fdocuments.in/reader035/viewer/2022071102/5fdc087aa603e24a0f2effe5/html5/thumbnails/6.jpg)
The Manage tab
6
![Page 7: Hadas + mitchoweb.mit.edu/hackl/www/lab/turkshop/slides/week4.pdf · Hadas + mitcho {hkotek, mitcho}@mit.edu Hackl Lab Turkshop April 2013 . Today..](https://reader035.fdocuments.in/reader035/viewer/2022071102/5fdc087aa603e24a0f2effe5/html5/thumbnails/7.jpg)
Approving/rejecting workers
7
Reject subjects who did not comply with instructions.
Completed more than one survey
(often a requirement for language studies!).
Did not complete survey.
Possibly also:
Failed on ‘catch’ items.
Exhibit guessing behavior.
But NOT: Non-native speakers!
May bias participants into saying they are native speakers.
![Page 8: Hadas + mitchoweb.mit.edu/hackl/www/lab/turkshop/slides/week4.pdf · Hadas + mitcho {hkotek, mitcho}@mit.edu Hackl Lab Turkshop April 2013 . Today..](https://reader035.fdocuments.in/reader035/viewer/2022071102/5fdc087aa603e24a0f2effe5/html5/thumbnails/8.jpg)
Approving/rejecting workers
8
Open your .csv results file in Excel.
Custom sort
by WorkerId
Find
duplicates
Conditional formatting:
highlight unanswered
questions;
decide if anyone
missed too many.
![Page 9: Hadas + mitchoweb.mit.edu/hackl/www/lab/turkshop/slides/week4.pdf · Hadas + mitcho {hkotek, mitcho}@mit.edu Hackl Lab Turkshop April 2013 . Today..](https://reader035.fdocuments.in/reader035/viewer/2022071102/5fdc087aa603e24a0f2effe5/html5/thumbnails/9.jpg)
What we wanted to test: blocking example
9
Effect of crossing animacy of causee and verb type on
acceptability of causative sentences.
# blocking 1 inanimate-v
That’s the ball that the coach bounced on the floor.
# blocking 1 animate-v
That’s the gymnast that the coach bounced on the floor.
# blocking 1 inanimate-make-v
That’s the ball that the coach made bounce on the floor.
# blocking 1 animate-make-v
That’s the gymnast that the coach made bounce on the floor.
![Page 10: Hadas + mitchoweb.mit.edu/hackl/www/lab/turkshop/slides/week4.pdf · Hadas + mitcho {hkotek, mitcho}@mit.edu Hackl Lab Turkshop April 2013 . Today..](https://reader035.fdocuments.in/reader035/viewer/2022071102/5fdc087aa603e24a0f2effe5/html5/thumbnails/10.jpg)
What it looks like from the workers’ end
10
![Page 11: Hadas + mitchoweb.mit.edu/hackl/www/lab/turkshop/slides/week4.pdf · Hadas + mitcho {hkotek, mitcho}@mit.edu Hackl Lab Turkshop April 2013 . Today..](https://reader035.fdocuments.in/reader035/viewer/2022071102/5fdc087aa603e24a0f2effe5/html5/thumbnails/11.jpg)
Our Turk survey
11
Forced choice: natural = 1, unnatural = 0
10 items
5 targets
Four conditions each
5 fillers
Two clearly unnatural, three clearly natural
8 lists
56 total participants
![Page 12: Hadas + mitchoweb.mit.edu/hackl/www/lab/turkshop/slides/week4.pdf · Hadas + mitcho {hkotek, mitcho}@mit.edu Hackl Lab Turkshop April 2013 . Today..](https://reader035.fdocuments.in/reader035/viewer/2022071102/5fdc087aa603e24a0f2effe5/html5/thumbnails/12.jpg)
Understanding and decoding our results
12
Open the .csv results file in Excel.
What information does it give us?
What does it not have?
Open the .decode.csv file in Excel.
What information does it give us?
Combine the two files using decoder.py!
Now open the decoded file in Excel.
What information does it give us?
![Page 13: Hadas + mitchoweb.mit.edu/hackl/www/lab/turkshop/slides/week4.pdf · Hadas + mitcho {hkotek, mitcho}@mit.edu Hackl Lab Turkshop April 2013 . Today..](https://reader035.fdocuments.in/reader035/viewer/2022071102/5fdc087aa603e24a0f2effe5/html5/thumbnails/13.jpg)
Fields in the decoded results
13
WorkerId: Amazon ID for the worker
AssignmentId: unique to that submission
AssignmentStatus: Submitted, Approved, Rejected
WorkTimeInSeconds
ListNumber, PresentationOrder
Section, Item, Condition
These came from the decode file
field_N: your actual stimuli
In our template skeletons:
Choice: forced choice value
Extras: english, foreignlang, numanswered, useragent
![Page 14: Hadas + mitchoweb.mit.edu/hackl/www/lab/turkshop/slides/week4.pdf · Hadas + mitcho {hkotek, mitcho}@mit.edu Hackl Lab Turkshop April 2013 . Today..](https://reader035.fdocuments.in/reader035/viewer/2022071102/5fdc087aa603e24a0f2effe5/html5/thumbnails/14.jpg)
R
14
Open source statistics software
Free! So lots of people use it.
Designed specifically for statistical data analysis
Generally treats everything as “data made up of multiple observations”
as we will see
Really, a programming language of its own
It’s a little weird and annoying
RStudio: a better interface for R
Open RStudio. Let’s play with some R! [Tutorial]
![Page 15: Hadas + mitchoweb.mit.edu/hackl/www/lab/turkshop/slides/week4.pdf · Hadas + mitcho {hkotek, mitcho}@mit.edu Hackl Lab Turkshop April 2013 . Today..](https://reader035.fdocuments.in/reader035/viewer/2022071102/5fdc087aa603e24a0f2effe5/html5/thumbnails/15.jpg)
Writing an analysis script
15
In an analysis script, we (might) want to:
1. Read in the results
2. Filter out parts of them
3. Check filler accuracy and disqualify participants
4. Recode some of the data
5. Look at the results
6. (Compute statistics)
7. (Make pretty charts)
Writing a script forces us to precisify what we are doing.
It also makes the procedure reproducible.
Eliminates Reduces human error
![Page 16: Hadas + mitchoweb.mit.edu/hackl/www/lab/turkshop/slides/week4.pdf · Hadas + mitcho {hkotek, mitcho}@mit.edu Hackl Lab Turkshop April 2013 . Today..](https://reader035.fdocuments.in/reader035/viewer/2022071102/5fdc087aa603e24a0f2effe5/html5/thumbnails/16.jpg)
Writing our analysis script
16
In RStudio:
Start a new script and save it in the same folder as the results.
Set working directory to Source File Location.
[Tutorial]
Pre-baked version in blocking-analysis.R