Eye-Hand Coordination in Object ManipulationRoland Johansson, Goran Westling, Anders Backstrom, Randall Flanagan

The Journal of Neuroscience, September 1, 2001, 21(17):6917-6932

Premise of paper: eye movements are used to obtain spatial information about a scene important for completing visual-motor tasks

Previous studies: generally found that eye movements played a proactive role in guiding hand movements*Land and colleagues (1999)

- Found that in a particular visual-motor task (making tea), there were between 4 and 6 fixation points gazed was directed to

*Ballard and colleagues (1992, 1995; Smeets et al., 1996) - Found that gaze led the hand in a task involving lifting a block

Open Questions: * When grasping an object, do subjects look at a particular landmark (grasp site, protruding edges, etc)? * When moving a grasped object, do fixations support motion planning? * How does the probability of a fixation occurring on a particular landmark depend on that landmark’s role in the task?* Does the lag between the eye movement and hand movement depend on the amount of spatial information extracted?

- i.e., does the lag vary in different phases of the trial?

Hypothesis: Brain uses gaze fixations to obtain spatial info for controlling manipulatory actions

Materials & Methods

Subjects: * 4 women & 5 men between 22 - 52 y/o;* All right handed; * No ophthalmological or neurological disease

General Procedure:

* While seated behind a table, subjects used tips of right index fingers & thumb to grasp & manipulate a bar (2 x 2 x 8 cm) located on a horizontally supported surface formed by the top of a wooden stand placed on a table.

* Each task involved moving the bar from a resting position to a switch (the “goal”). Some trials also included an obstacle which subjects had to avoid touching while moving the bar to the goal.

* Gaze position was recorded with an infrared camera, while fingertip position was recorded using position sensors.

Apparatus

Tasks

* Subjects first performed 4 consecutive trials without an obstacle in the path of the bar

* Then 4 trials with each of the 2 obstacles that had to be avoided (rectangle & triangle), for a total of 8 trials with obstacles.

* Each trial was initiated by an auditory cue followed by the opening of the shutter to allow vision of object. * To obtain a uniform start position of gaze subjects gazed the fixation light during the inter-trial periods. The fixation light was off when shutter was opened & during the task.

* However there was one test without eye movements: during the rectangular obstacle (4 trials) the fixation light remained on during the task.

8 phases of the trials defined kinematically

From Resting position to target Returning to resting position

* R = Point of regard signal

* P = Pupil position

* G = Measure of gaze position

R & P were combined to determine and obtain a measure of gaze position with adequate temporal resolution & spatial accuracy = G

C - histogram of all saccade durations in the experimentD - histogram of all saccade amplitudesE - scatter plot of saccade amplitude vs. saccade velocityF - scatter plot of saccade amplitude and duration with regression line G - histogram of fixation durations

General characteristics of saccades observed

Results:The pattern of eye & hand movements for a single trial involving the triangular obstacle

* The dashed line represents the position of the index finger

* The solid line represents gaze position

* The numbered circles indicate gaze fixations & their sequence

* Subjects never fixated or tracked the hand or moving bar during the task

Results:The pattern of sequential landmark fixations

within & across trials & across obstacle conditions

* Subjects directed gaze almost exclusively to objects involved in the task

* These included: Grasp site on bar, the left tip of the bar used to contact the target, the protruding points on the obstacle, the target & support surface.

The black circles represent fixations within 3 degrees (2cm in the work plane) of one of the 5 landmarks.

The gray circles represent fixations outside the landmark zones.

The figure shows 1351 which correspond to an average of 12.5 fixations per trial.

Of the total # of fixations, 82% were within the 3 degrees of one of the landmarks.

The duration of fixations within the landmark zones were significantly longer (median= 0.34 sec) than those outside the landmark zones (median = 0.19 sec).

Subjects directed their gaze toward landmark zones 90% of the time

Results:

The dotted-line circles represent the landmark zones (3 degree radius). In this figure they removed fixations outside landmark zones.

The dots represent fixation location

Gaze fixations were not evenly distributed within the 3 degree r. Instead they were clustered, and the degree of clustering varied across landmarks.

Solid line circles represent 90% of captured fixations

Figure C shows the location of the grasp site (empty circle) in bar coordinates & the gaze position of grasp site fixations (filled circle).

Fixation position and grasp site were positively correlated with a slope of almost 1 (r=0.76; p<0.001)

Figure C does not show that fixation precedes grasp, only that the 2 are correlated.

Results:Sequence of Landmarks Fixated

The sequence was clearly linked to the progress of the task

The arrows in this figure show the flow of gaze fixations between landmarks for each obstacle condition

The width of each arrow represents the proportion of all gaze shifts between landmarks during the task.

In the triangle obstacle trials, gaze often shifted from the grasp site directly to the target (and back again) without stopping at the obstacle.

NOTE: these figures are based on data aggregated over all trials and all subjects.

Results:Spatiotemporal coordination of gaze & hand

Figure is based on data aggregated over all trials and all subjects involved in the triangular object

A Solid line = distance of median gaze position from landmarkDotted line = distance of fingertip (or the bar after being picked up) from the landmark

BThick solid line = instantaneous probability of fixation within 3 degrees of landmarkShaded area = instantaneous probability of fixation within 2 degrees of landmarkThin solid line = cumulative probability that fixation within 3 degrees of the landmark had occurred at any previous point in the trial

Results:

A - cumulative probability of fixation on a landmark (within 3 degrees) at any point during the trial (same as figure B of the previous slide but here presented as a bar)

B - Duration of fixations for each phase of the trial

C - Number of fixations for each phase of the trial (excluding results where the landmark was not fixated)

D - Durations of individual fixations in the landmark zone

E - Duration of each phase of the trial

Key Points:* the rectangular obstacle was fixated with a higher probability than the triangular one

* total fixation duration at the target was greater in the no-obstacle condition than in the two obstacle conditions combined (but was due to number of fixations more than the duration of each fixation)

* the number of fixations on obligatory landmarks (grasps site, target, and support surface) was greater than the number of fixations on optional landmarks

Gaze fixation characteristics

Results:

Cumulative frequency of gaze entry / exit of the landmark zone (within 3 degrees of the landmark) for each phase of the task, relative to the kinematic event associated with that task.

Each chart shoes some lag between gaze entry and the kinematic event (time 0).

For all landmarks except the support surface, the gaze was somewhat likely (~50%) to have left the target zone by the time the kinematic event happened.

The point of this figure is to show that gaze arrives at the landmark before the hand or the tip of the bar.

Time relation between kinematic events & landmarks

Results:

Previous studies (Becker, 1991) showed that saccades directed to a target tend to undershoot the target and are followed by smaller corrective saccades.

Figures show saccadic error (distance between the saccade and the average fixation position in the target zone) as a function of saccade amplitude.

In absolute distance, there was a positive correlation between saccadic error and the amplitude of the saccade, indicating that saccades were less accurate, the longer the jump was. (A)

For the x-distance, there was a slight positive correlation and for the Y-distance there was a negative correlation. (B and C)

The negative correlation between Y-distance and saccade amplitude indicates that subjects undershot as they directed their eyes upward which is consistent with Becker.

There was a negative correlation between absolute distance and fixation duration indicating that the farther the fixations were from the target, the faster the eyes were to correct them.

Accuracy of saccades to landmarks

Results:Overall subjects’ performance was only modestly degraded when their gaze was restricted.

Chart shows cumulative probability of the tip of the bar traveling a given distance* within the target zone for the no-obstacle and both obstacle conditions combined.

Thick solid line = trials when the subjects were required to fix their gaze on a light at the top right corner of the work plane.

Thin solid line = trials when the subjects’ vision was occluded by the shutter.

Dotted line = trials where the subjects could freely gaze the scene.

*NOTE: the researchers considered this distance a measure of “target efficiency.” The smaller the distance travelled within the target zone, the greater the efficiency.

The point of this figure is to show that even though subjects were able to complete the tasks without free gaze, they were less efficient in doing so. This is consistent with the theory that gaze supports motion planning.

Performance without eye movements

Possible sources of error in the study

* A potential problem with the experiment is that there was a lot of data manipulation. The position signal & the gaze signal were recorded independently & then later analyzed together. The experimenters gave a somewhat vague indication of how the signals were synchronized (e.g. was the recording of each signal controlled by the same switch?) They gave little info on how the signals were combined too. (The paper just says “time synchronized”) So we don’t know if they were synchronized properly. Given that the time lags they observed were generally less than 1 sec, there was a small margin for error here.

* In the trials with the triangular obstacle, some subjects did not look at the obstacle: Was this a learning effect? Was it a few subjects who were driving the results? They did not do the analysis to explain this effect. The data should have been disaggregated & analyzed separately.

Conclusions:* gaze supports hand movement planning by marking key positions to which the fingertips or grasped object are subsequently directed. In addition to improving the accuracy of hand movement towards target objects, gaze plays a complementary role in the monitoring of task progression & the upholding of sensorimotor correlations important for prediction.

* the salience of gaze targets arises from the functional requirements of the task

* Fixating the target provides extraretinal information about eye position. It has been suggested that eye position (extraretinal gaze signals) serves as target for the hand. In other words: the hand points to where the eye is looking.