Watching the eyes when talking about size: An investigation of message formulation and utterance...
-
Upload
scarlett-cummings -
Category
Documents
-
view
217 -
download
0
Transcript of Watching the eyes when talking about size: An investigation of message formulation and utterance...
Watching the eyes when talking about size: An investigation of message formulation and utterance planning
Sarah Brown-Schmidt, Michael K. Tanenhaus
Presentation by Olatz MendiolaLing 5555November 17, 2010
Introduction
How do speakers plan and update messages that are translated into utterances during language production?
How are message formulation and utterance planning coordinated?
Speakers often update messages on the fly based on new information. Message formulation and utterance planning are
mixed in time and must communicate with one another at a relatively fine temporal grain.
Introduction
Levelt & Maassen (1981) Latency difference evidence that the unit of advance planning includes each
of the lemmas in the subject noun phrase.
Meyer (1996) Dutch speakers perform a naming task with auditory distractors. Naming latencies were delayed by a distractor that was phonologically
related to the first noun in NP but not the second word in the NP. Distractors semantically related to either 1st or 2nd noun decreased
latencies.
Schriefers &Teruel (1999) Presented phonological distractors to German speakers in a naming task. Phonological distractors related to 1st syllable of 1st word facilitated naming
times, no effect found for distractors related to 2nd word and distractors related to 2nd syllable of 1st word had a weak effect.
In some cases, articulation may begin before the 1st phonological word is entirely planned.
Only 1st word is prepared before speech onset
Introduction
The degree to which speakers pre-plan a complex phrase is highly variable. Amount of advance planning varies depending on the task and type of
utterance Planning is partially under control of speaker.
Little work has addressed how messages are prepared
Griffin and Bock (2000) Participants described simple events (woman shooting a man…) The sequence of eye movements reflected the order of constituents in the
utterance. Speakers looked at picture objects about 800ms to 1s before naming them. This eye-voice lag is similar to the time it takes to initiate naming an object in
isolation, suggesting that the eye-voice delay reflects word preparation.
Difficult?
Introduction
Timing of eye movements could provide insight into interplay between message formulation and utterance planning.
Work investigating gaze-speech link may typically conflate message formulation with utterance planning.
How disentangle message formulation form utterance planning?
creating situations in which the speaker, while in the process of planning or producing an utterance, encounters new information that requires revising the message.
If eye movements can be used to infer when the speaker 1st encounters that information, then the timing between the uptake of the new information and the form of the utterance might shed light on the interface between message production and utterance planning.
Brown-Schmidt
Tanenhaus (2006)
Present Study. Experiment 1
18 pairs of participants Seated at different computers. 5 feet separation No eye contact Subject X Eye tracker + headset
microphone Subject Y Headset microphone
Present Study
Hypothesis 1 Fixation to the contrast will predict use or non-use of a size
adjective. Higher rates of modification when contrast has been fixated.
Hypothesis 2 There is a systematic relationship btw timing of 1st look to
the contrast and the form of the utterance. Hypothesis 3
Timing of looks to the contrast and use of a pre-nominal vs. post-noun repair for descriptions of complex and simple shapes should provide insight into the size of message planning units, and how repairs to these units affect utterance planning.
Results
Manipulations successful in eliciting expected forms:
1. Size adj. typically used only when contrast present in display.
2. Pre-nominal adj. used for Simple Shapes.
Adj. in post-nominal PP used for Complex Shapes.
Results
1. Use of size adjectives
Contrast : size adj. on 98% of trials No contrast: size adj. on 27% of trials
Contrast/Simple: size adj. on 99% of trials Contrast/Complex: size adj. on 97% of trials NoContrast/Simple: size adj. on 23% of trials NoContrast/Complex: size adj. on 31 % of trials
Statistically Significant
Not significant
Results
2. Utterance form
Contrast / Size adj. / Simple shape pre-nominal 89% of trials
Contrast / Size adj. /Complex shape post-nominal PP 73% of trials
Size adj. also occurred in post-noun repairs“The triangle, uh the small one”“The square with triangles, uh small ones”
11% of trials for simple shapes 16% of trials for complex shapes
Results.
Hypothesis 1 Fixation to the contrast will predict use or non-use of a size adjective.
Higher rates of modification when contrast has been fixated.
Size contrast / fixated Size adj. on 99% of trials
Size contrast / not fixated Size adj. on 68% of trials
Results.
Time of the first fixation to the contrast relative to onset of utterance.
Form of referring expression is related to timing btw 1st look to size contrast and onset of utterance.
ANOVA: Effect of adjective position
Earlier fixation for pre-nominal adj.
Effect of complexity. Earlier fixation for references to
simple shapes Interaction btw complexity and
adj. position was not significant
New message elements can be added to a planned
referential description duringor immediately before production
Supports Hypothesis 2UNEXPECTED
Results
Simple shapes: Negative relationship between first fixations and adj. onset time.
Complex shape: marginal effect
When need for a size adjective is noticed after speech onset, preparation of the repair is delayed by preparation of the original utterance.
Lag between 1st fixation to contrast and adjective onset Complex: 1220ms Simple: 440ms
Results
Simple shape Prenominal
1546ms after display Post repair
1061ms after display
Complex shape Prenominal
1434ms after display Post repair
1390ms after display
Conclusions
Fixation on size contrast predicts use of size adj.
Timing of fixations predicts the form of the utterance.
When preparing a modified referring expression, messages are incrementally prepared and passed onto utterance planning in units smaller than the size needed for an entire referring expression.
Towards experiment 2
2 unexpected results 600ms lag between fixation to the contrast and onset of
utterance. 200ms faster than literature reports. Explanation? A small number of shapes were repeatedly used
on target trials, which may reduce the demands of lexical encoding
Size adj. were used on more than 25% of trials when there was no contrast. Explanation? 25% of trials had targets with size contrast
which might make inclusion of size in the description somewhat formulaic
Experiment 2 Increasing target type Decreasing proportion of target trials
Results
Speakers rarely used a size adj. when there was no contrast in display = 1 %
Speakers used a size adj. 72% of the references on trials with contrast
Proportions consistent with other studies Of the size contrast trials where adj. was
used: 62% pre-nominal 37% post-noun repairs
Results Analysis was restricted to trials were there was a size contrast on same side
of the river as target. Fluent and disfluent trials analyzed separately.
Fluent trials Looks to contrast predicted whether the size of target would be
mentioned (Hyp 1) = Exp 1
86% of fluent trials during which speakers looked at contrast, the referring expression included a size modifier.
When there was no fixation to the contrast, modifier was only used in 19 % of trials
First fixation to the contrast was systematically related to the form of the utterance. (Hyp 2) = Exp 1
The mean 1st fixation to contrast for pre-nominal modifiers was887ms before onset of referring expression.
The mean 1st fixation to contrast with post-noun repairs was 14ms before onset.
Results
Disfluent trials When speakers fixated the contrast adj was used
on 91%. When contrast was not fixated ajd was used 34%. First fixation to contrast systematically related to
form of utterance. First fixations were on average 528ms before onset of
referring expression for utterance with pre-nominal modifiers.
First fixations were on average +819ms for post-noun repair utterances
Results.
Earliest fixations are associated with fluent prenominal utterance
Latest fixations are associated with a post-noun repair. Consistent with H3: suggests speaker was able to use
disfluency to buy time to include size in the noun phrase as a prenominla adj. rather than as a postnoun repair.
=Exp 1. Relationship btw time speakers 1st fixated the size contrast and the timing of the size adj. for postnoun repairs.
Results
Time to begin speaking Average time between the highlighting of target and NP onset <
2s ≠ Exp1. Utterance onset time was not significantly affected by the
form of the utterance For initially fluent utterances, speakers started pre-nominally
modified expressions 345 ms faster than expressions with post-noun repairs (not reliable)
For disfluent utterances pre-nominally modified expressions were only 120ms faster than utterances with postnoun repairs
Speakers uttered disfluent pre-nominally modified expressions on average 66ms faster than fluent pre-nominally modified utterances (not reliable)
General Discussion
3 results 1. Speakers typically used size adj. in a referential
expression only when there was a similar object in the referential domain that differed in size
2. Use of size adj was more likely if speaker had fixated on the size contrast.
3.Timing of first fixation to contrast was related to the form of referential expression Earlier – prenominal adj Later – postnoun repair
General Discussion
Preliminary support for hypotheses about interplay between message formulation and utterance planning.
When an error is detected in the utterance plan after speaking has begun, initiation of a repair may be delayed by preparation of original utterance.
Messages that will be mapped onto referring expressions can be constructed and passed onto utterance planning incrementally.