A comprehensive eye-tracking test battery: Pupil Labs vs ......˛ Researchers have to know the...
Transcript of A comprehensive eye-tracking test battery: Pupil Labs vs ......˛ Researchers have to know the...
SummarySummaryHow can we measure eye-tracker performance?We developed a test battery with multiple tasks and measured different metrics of eye-trackers.Simultaneous data collection with the remote EyeLink 1000 against the mobile Pupil Labs glasses allow for direct and powerful within-subject comparisons.
FreeviewingFreeviewing
−10° 5°
−7.5°
7.5°
Y-po
sitio
n [°
]
X-position [°] EL PL
11.25
20
Fixa
tions
per
imag
e
Fixation duration [s]0 1 1.50
100
400EyeLink
Pupil Labs
Num
ber
of fi
xatio
ns
Pupil dilationPupil dilation
Diff
eren
ce E
yelin
k −
Pup
il La
bsno
rmal
ized
pup
il ar
ea [a
.u]
100 150 200 250
Luminance [%max]
-10%
-5%
0%
5%
pa_n
orm
0 2 4 6
60%
40%
80%
100%
Nor
mal
ized
pup
il ar
ea [a
.u.]
luminance onset luminance offset
monitor luminance
25% 50% 75% 100%
Time [s]
EyelinkPupil Labs
N = 15. For preprocessing we used the same algorithms where possible. For saccade classification we used the Engbert mergenthaler algorithm (λ = 5).
CalibrationCalibration
1 3 7Time [s]
−10
0
10
20
X-po
sitio
n [°
]
−10
0
10
20
2 6 8
binocular fusion artefact
staircase fusion artefact
no blink detected ?
?
EyeLink
Pupil Labs
fixation
saccade
Accuracy & PrecisionAccuracy & Precision
∙ Researchers have to know the strengths and weaknesses of their devices
∙ Eye-tracking data quality cannot be reduced to a single value for all tasks
∙ Concurrent measurements can reveal consistent subject-biases
Take homeTake home
EyeLink Pupil Labs
Spa
tial A
ccur
acy
[°]
Accuracy
0.50
0.75
1
0.2
0.3
0.4
Pre
cisi
on S
tand
ard
Dev
iatio
n [°
]
Precision [SD]
EyeLink Pupil Labs
EyeLink Pupil Labs
0
0.5
1
1.5
Eye
-tra
cker
repo
rted
valid
atio
n ac
cura
cy [°
]
Samples of horizontal gaze movement during the grid task. Note the sample-wiseeye movement classification. Blink samples were removed.Samples of horizontal gaze movement during the grid task. Note the sample-wiseeye movement classification. Blink samples were removed.
large grid pursuit free viewing micro saccades blinks
pupil dilation small grid small gridhead yaw head roll
calibration / validation
Repeat 6x
Psychtoolboxpowered by
0.57°
0.82°
0.19°
0.31°
Get the full paper:A new comprehensive eye-tracking test battery concurrently evaluating the Pupil Labs glasses and the EyeLink 1000doi: 10.7717/peerj.7086
Stimulus & Analysis Code:10.5281/zenodo.2553176
FundingThis work was supported by the EuropeanUnion (H2020-FETPROACT-2014,SEP-210141273, ID: 641321 socSMCs)
Validation accuracy before each block reported by the eye-tracker‘s manufacturer software.
Spa
tial A
ccur
acy
[°]
Small Grid I Small Grid II
EyeLink
Large Grid (subset)
0.5
1
1.5
Pupil Labs
On the group level, both eye-trackers report the same normalized pupil area; on the single subject level, the eye-trackers show a reliable subject-specific bias.
Pupil Labs shows much higher variance in estimated gaze positions and finds fewer and shorter sac-cades than EyeLink and, consequentially, on average longer fixation durations.
EyeLink shows 40% better accuracy and 60% better precision. Pupil Labs exhibits strong calibration decay after ~ 4 minutes of recording.
~4m 40s ~6m 20s
Katharina Groß¹,*, Benedikt V. Ehinger¹,²,*, Inga Ibs¹, and Peter König¹,³ ¹ Institute of Cognitive Science, University of Osnabrück2 Donders Institute for Brain, Cognition and Behaviour, Radboud University3 Department of Neurophysiology and Pathophysiology, University Medical Center Hamburg-Eppendorf
* Authors contributed equally
A comprehensive eye-tracking test battery:Pupil Labs vs. EyeLink 1000
Find 700 GB of eye videos:10.6084/m9.�gshare.c.4379810.v2