Hand trajectories reveal hidden cognitive states

Ken Nakayamaa

Joo-Hyun Songb

Matthew Finkbeinerc

Alfonso Caramazzaa

aHarvard UniversitybSmith-Kettlewell Eye Research Institute

cMacquarie University

today’s talk

Suggest there are new opportunities to study a wide

range of internal cognitive states from the detailed study of motor

behavior(Spivey et al., for language processing)

Remarkable power of psychophysical and behavioral methods

• Some examples• Wertheimer, 1912• Hecht, Shlaer and Pirenne, 1943• Julesz 1960s• Sperling 1960s

Early processing

Later processing

Cognition/decision

Motor plan

input

Early processing

Later processing

Cognition/decision

Motor plan

input

fMRI

EEG/MEGTMS

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

C O G N IT IV E

N E U RO S C IE N C E

Early processing

Later processing

Cognition/decision

Motor plan

input Reaching

Measure RT And accuracy PLUS

Full details of trajectory

Reveals otherwisehidden states

.

Hand movement recording: 120Hz with Polhemus Fastrak

QuickTime™ and aSorenson Video 3 decompressorare needed to see this picture.

Polhemus system: Sample rate (120 Hz)

• Recording 3-D movement of finger tip

Visually-guided pointingSong and Nakayama (VSS 05,06)

• I. Early hand trajectories reflect an underlying number representation – Song and Nakayama, Cognition (in press)

• II. pointing trajectories reveal influence of unseen words– Finkbeiner, Song, Nakayama, and Caramazza,

Visual Cognition (in press)

Two recent studies

Mental number line

IF X>5 respond “greater”Else respond “lesser”

if digital number line

BUT, RT and error rates systematically decrease as the numerical distance between two numbers increases.

Moyer and Landauer (1967)

Supporting “Analog” number line

X

In numeric comparison task 3 vs. 5, 13 vs. 5

Same response to 3 vs. 5 &13 vs. 5

Numeric distance effect

• Characteristics of number representations are mainly examined with discrete responses such as reaction time and accuracy.

• In the current study, to map invisible internal cognitive processes of numeric comparison in spatial domain over time, we measured trajectories.

X-Y trajectories

• Systematic shift of initial trajectories towards a hypothetical position on a number line intermediate between the numeral 1 and 5 positions

Mean trajectory deviation

• ≈ 480 ms after target onset • ≈ 150 ms after the movement onset

Endpoint deviation

• Continuous internal processes for number representations

• Early detection of mental numeric line than previous tasks

• Kinematic measurements: More reliable and robust with small numbers of subjects

• I. Early hand trajectories reflect an underlying number representation – Song and Nakayama, Cognition (in press)

• II. Pointing trajectories reveal influence of unseen words– Finkbeiner, Song, Nakayama, and Caramazza,

Visual Cognition (in press)

linkage between invisible text and actions?

Goodale and Milner

DorsalVs

Ventral

spinach

Point to color of the word

########

boy

KMDKS

spinach

500 ms

30ms

10 ms

Until Response

########

red

KMDKS

spinach

500

30

10

2000

########

green

KMDKS

spinach

500

30

10

2000

neutral incongruent congruent

########

boy

KMDKS

spinach

500 ms

30ms

10 ms

Until Response

########

red

KMDKS

spinach

500

30

10

2000

########

green

KMDKS

spinach

500

30

10

2000

neutral incongruent congruent

-10 0 10

0

5

10

15

20

25

-10 0 10

0

5

10

15

20

25

-10 0 10

0

5

10

15

20

25

Left Right (cm)

Up(cm)

Down

A. B. C.

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

100 msec after movement onset

Time after movement initiation

Hor

izon

tal m

ovem

ent

(in)

incongruent

congruentneutral

Replication with varying prime duration

• Despite the unavailability of the invisible stimulus to visual awareness, the congruity of the prime stimulus with the target stimulus had a significant effect on participants’ pointing trajectories.

• Masked unseen word processing extends down to include the formulation of overt motor responses

Early processing

Later processing

Cognition/decision

Motor plan

inputReaching

Full details of trajectory

LEAKAGE

Suggest there are new opportunities to study a wide

range of internal cognitive states from the detailed study

of motor behavior