Каналы мультимодальной коммуникации:

относительный вклад в понимание дискурса

А.А. Кибрик (ИЯз РАН и МГУ)Н.Б. Молчанова (BearingPoint)

aakibrik@gmail.com

«Мультимодальная коммуникация»15 ноября 2013

2

What is the contribution of different communication channels?

Traditional approach of mainstream linguistics:

the verbal channel is so central that prosody and the visual channel are at best downgraded as “paralinguistics”

Applied psychology It is often stated that (figures go back to Mehrabian

1971):• body language conveys 55% of information• prosody conveys 38% of information• the verbal component conveys 7% of information

Who is right?

3

Relative contribution of three communication channels?

DISCOURSE

Vocal channels Visual channel

Verbal channel Prosodic channel

4

Experimental design

Isolate the three communication channels Present a sample discourse in all possible

variants (23=8) Present each of the eight variants to a

group of subjects Assess the degree of understanding in

each case Such assessment may lead to estimates of

the contributions of communication channels

5

Studies in this line of research

Èl’bert 2006, year paper Èl’bert 2007, diploma thesis

Reinterpreted and refined in Kibrik and Èl’bert 2008

Molchanova 2008, year paper Molchanova 2009, year paper Molchanova 2010, diploma thesis

Reinterpreted and refined in Kibrik and Molchanova 2013

6

Èl’bert 2007, Kibrik and Èl’bert 2008

Russian TV serial “Tajny sledstvija” – “Mysteries of the investigation”

Context excerpt: 8 minutes Experimental excerpt: 3 min. 20 sec.

consisting of conversation alone, to ensure that we are testing the understanding of discourse rather than of the film in general

Two vocal channels have been separated: Verbal: running subtitles Prosodic: superimposed filter creating the “behind a

wall” effect

Participants: Native speakers of Russian Eight groups of 10 to 17 participants

7

Eight experimental groups

Group 0: only the context excerpt Groups 1 (one communication channel)

Verbal: subtitles, temporally aligned Prosodic: filtered sound Visual: video

Groups 2 (two communication channels): Verbal + prosodic = original sound Verbal + visual: subtitles and video Prosodic + visual: filtered sound and video

Group 3: original material

8

Verbal + visual

9

Procedure

The context and the experimental excerpts were shown to a group of subjects on a large screen

Each subject answered 23 multiple-choice questions concerned with the experimental excerpt alone

What Tamara Stepanovna offers Masha before the beginning of the conversation: a. to take off her coat b. to have a cup of tea c. to have a seat d. to have a drink

Percentage of correct answers is used as an assessment of a subject’s degree of understanding

10

Results

All three channels are substantially informative

Verbal > visual > prosodic

Integration of visual and prosodic channels is difficult

Group 0

Verbal

Prosodic

Visual

Ver+Pro

Ver+vis

Vis+pro

Group 3

0,00%

20,00%

40,00%

60,00%

80,00%

100,00%

11

Molchanova 2010Kibrik and Molchanova 2013

Methodological issues The following aspects of the prior

study have been changed (improved) Stimulus material Methods of isolating the channels Questionnaire Participants and interviewing procedure

12

Stimulus material: discourse type

Shortcomings of movies Plot facilitates guessing Possible familiarity with the movie Quasi-natural behavior of actors

Solution: natural dialogue Guessing game

original.avi, 0:19 – 0:57

13

Stimulus material: speakers

Shortcomings of the prior studies Same-sex speakers indistinguishable

in the prosody-only version

Solution: Different sexes: F0 range is different

14

Methods of isolating the channels: Verbal channel

Shortcomings of subtitles Subtitles belong to the visual mode Hard to read without punctuation

• Especially at the rate of speech• And especially in the “verbal + visual”

condition

Solution: spoken prosody-free signal Each word in transcript is recorded

individually from the corresponding person All thus elicited words are glued together

in the right order

15

Visual + verbal (the robot condition)

16

Verbal channel

Remaining problem Unnatural input

• No reduction• No intonation• etc.

17

Methods of isolating the channels: Prosodic channel

Shortcomings of the prosodic material as used in previous studies Excessive noise

Solution: Loudness is decreased radically at all

frequencies except for the speaker’s average F0 frequency

This has led to a more satisfactory “behind the wall” (or “behind the glass”) effect

18

Visual + prosodic (the mermaid condition)

19

Questionnaire

Shortcomings of prior studies Èl’bert 2007: gap between Group 0 (38.3%)

and Group 3 (87.4%) is insufficient

Solution Testing stage

• Identify trivial questions (high Group 0) –5 • Identify unfortunate questions (low Group 3) –2 • 30 23

Group 0: 34.5% correct answers Group 3: 88.0% correct answers

20

Participants and interviewing procedure

Shortcomings of prior studies Uncontrolled social status and geographical origin of

participants Multiple participants in one room may affect each

other’s performance Need for a big screen

Solutions Control for social status and geographical origin;

homogeneous group Comparable, independent, and comfortable conditions

• Detailed guidelines Remote implementation

• Stimulus materials at Youtube.com• Questionnaire at Googledocs

21

Kibrik and Molchanova 2013: Results

Each individual channel is substantially informative and prevails over the null condition (34.5%)

F-test: verbal and visual: p<0.05, prosodic: p=0.127

Verbal (58.8%) > visual (52.2%) > prosodic (40.2%)F-test: verbal > prosodic,

visual > prosodic: p<0.05, verbal > visual: p=0.071

22

Kibrik and Molchanova 2013: Results

Two-channel conditions prevail over the one-channel conditions much more clearly than in the previous experiment (verbal+prosodic – 73.5%, verbal+visual – 88.2%)

F-test: all pairwise comparisons but “visual+prosodic > visual”:

p<0.05; all two-channel conditions > all

one- channel conditions: p<0.0001

A dramatic dip in the visual+prosodic condition is even clearerF-test: significant difference from the two other two-channel conditions, p<0.0001

23

Kibrik and Èl’bert 2008 vs. Kibrik and Molchanova 2013

0,00%

20,00%

40,00%

60,00%

80,00%

100,00%

General picture is remarkably similarIn the new study all effects are clearer

24

Normalized contribution of three channels

Suppose the three channels are independent

Sum up all percentages of individual channel contributions and normalize to 100%

Identify normalized contribution

25

Normalized contribution of three channels

Kibrik and Èl’bert 2008

Kibrik and Molchanova 2013

Summed percentages

72+51+62=185 59+52+40=151

Normalized contributions

Verbal 72%:1.85≈39%

59%:1.51≈39%

Prosodic

51%:1.85≈28%

46%:1.51≈30%

Visual 62%:1.85≈33%

49%:1.51≈32%

26

Gender differences

Molchanova 2010: gender advantages Percentages of correct answers

Condition Men Women Advantage

Verbal only 59.1 69.9 Women: +10.7

Visual + prosodic

66.1 51.6 Men: +14.5

27

Conclusions

All communicatioin channels are highly significant the traditional linguistic viewpoint is incorrect

The verbal channel is the leading one the viewpoint popular in applied psychology is incorrect

Information from the prosodic and the visual channels is primarily used through integration with the verbal channel

Very similar results have been attained in different studies, in spite of very different methodological details

28

Further questions

Auditory or graphic presentation of the “verbal alone” channel?

Explore different discourse types, such as monologic discourse

…and: Other suggestions on this approach?

29

Acknowledgements

Olga FedorovaAnna LaurinavičiuteAndriy Myachykov

RGNF #11-04-00153

30

Thanks for your attention 

verbal channel

visual channel

prosodic channel

language