microscopes, macroscopes, semantic maps

and a good compass

Michael ZOCKCNRS, LIF (University of Marseille)michael.zock@lif.univ-mrs.fr  

Wheels for the mind ���of the language producer

Goals

1.  Present a new way to address unsolved problems;2.  Show the current status of my understanding and

solutions;3.  Stimulate new ideas on your side.

ContextDeal with language production (L1 and L2)

➠  DEFINITION: given some problem (goal), decribe a method for solving it via language, that is conceive, organize and express thoughts in order to solve a given problem (focus on the last task: expression)

Motivation for speaking➠  starting point A: problem (cognition, emotion)

➠  end point B: solution

Build tools and resources helping you to get from A-B➠  microscopes;➠  macroscopes;➠  semantic maps;➠  good compass.

Problems and goals at a very high level of abstraction

(metaphorical level)

Slide 5

What is involved in language production ? Language production requires accessing and assembling information at various levels and the programing of specific motor muscles to produce the corresponding spoken or written form

1.  Conceptual level (idea, message)2.  Linguistic level (formulation : lexicon + grammar)3.  Motor level (articulation)

Conceptualization

Lexicon Grammar

morphology

Formulation Articulation

S

NP

VP

V

<cat><the>

NP

<the> <dog> <chases>

Constraint juggling

!

Spontaneous discourse is a complex process requiring the processing of various kinds of information

at various levels under severe space- and time-constraints

!

ideas

words word order

word forms

Danger of getting drowned or overwhelmed

!

time and space constraints

Goal : keep things under control

Language is a skill. Skillfull people manage to make difficult things look easy.

Everything seems to flow naturally without any effort.

Stay on top of the wave

!

Get into the driver's seat

!

control : direction, speed

Tools

Magnifying glass or microscope

grasp the details

!

Macroscope : get the great picture

abstract away from irrelevant details discover relations and patterns at the various levels (word, sentence, text)

!

Tools for orientationmaps

compass

Map out the territory

See how things are related. Where am I?

How can I get from A to B?

!

There are maps for many things : cities, subways, galaxies, words

Semantic maps

See how words are related. Association thesaurus WordNet

Slide 18

Naviga&on  in  an  associa&ve  network

Since search takes place within a semantic network, i.e. a graph where all words (nodes) are related (via a certain kind of association), search consists in entering this network at any point and follow the links to get from the starting point (source word, SW) to the end (target word, TW). This latter may be directly related to the initial input, i.e. SW (direct association/neighbour; distance 1) or not (indirect association).���Note that the user knows the starting point, but not the end-point (target).

Compass

Where are we now ? How can we reach our goal from here?

!

What about the Compass?

!

The compass is in peoples' minds. While we have to provide them with the semantic map and the signposts (orientational guidelines; categorial tree), the decision where to go is left to the user, as he is the only one to know the target. Even if he is not able to name it, he is still able to recognize it in a list. Hence we have to present this list (in our case, the direct neighbors of the input, query word). In the case of word access authors generally know fairly well where in the map is located the target word and what are its direct neighbors, as this is generally the one they will use as input.

OutlineProducing language, a difficult task

➠  How is it possible and how to make it feasable?

1.  Sentence production in real time➠  The problem of determining syntactic structure

2.  Learning this skill in a foreign language➠  goal: become fluent and reach the survival level➠  means: creation of a multilingual, selfextending phrasebook,

augmented with an exercice generator (drill tutor)

3.  Finding the needle in a haystack (lexical access)➠  Enhance electronic dictionaries by taking a look at the mental

lexicon, i.e. the functioning of the human mind

������

Language production is a very complex task which has to be learned

1

Speech is fast, but not as fast as thought

–  3-5 words per second (speed which can be doubled in case of need)

–  locate words in a store of 30 - 50.000 words

–  problem of manageing I/O constraints (forgetting) : race between conceptual input (ideas) & linguistic output (well formed words and sentences)

NLG or Language production���What’s the challenge ?

2

3

Catch me if you can

1.  We tend to think faster than we can find the corresponding words and convert them into sounds.

2.  Also, the order of ideas does not always correspond to the order of words

Conceptualization

Expression

C 1

W 1

C 2

W 2

C 3

W 3

C 4

Ques%on  

What is involved in speaking and why is it difficult?

24/02/15  

The  3  principal  steps  

The mice are dancing.

idea

form

sound

concepts

abstract words/lemmasyntactic category

morphology

phonemesgraphemes

The three principal steps:���ideation, formulation, expression

Conceptualpreparation

Morphological andphonological encoding

Formulation

Articulation

Semantic andsyntactic encoding

ideas

6

Text plan Unforma-ted text

Sentenceplan

[goals + discourse situation] + [knowledge base + plan library + o ntolo gy + user uodel ] + [dictio nary + grammar] + [discourse histo ry]

Positewords

Determine part-of-speech

Insert functionwords

Inflection + agreement

Surfacerealisation Physical presentation

Articulation

Formated written textSpoken Text

Content determination

Content structuring

Macroplanning

Referring expressions

Grammaticalization

Lexicalisation

Aggregation

Microplanning

Layout

Punctuation

The 4 components���in the Reiter & Dale model

7

Language production, a difficult task

1.  plan what to say (conceptual level)

2.  find the adequate words (linguistic level: lexicalisation)3.  find appropriate sentence frame (linguistic level: syntax)4.  insert words in the right place (linguistic level: syntax)5.  add function words (linguistic level: syntax)6.  morphological adjustments (linguistic level: morphology)���

7.  articulate (phono-acoustic level)���

8.  plan next stretch while speaking (conceptual level)

24/02/15  

9  

The  necessary  informa:on  for  synthesis  is  sca;ered  all  over  all  over  

GIVESubject DirectObject

BOOK

Donnes-MOI ce livre !Give ME the book !

Ne ME le donnes pas !Don't give ME the book !

LISTENER

Indirect Object

PersonSPEAKERNumberTu ME donnes le livre.You give ME the book.

Tu LUI donnes le livre.You give HIM / HER the book.

Tu ME donnes le livre.You give ME the book.

Tu NOUS donnes le livre.You give US the book.

negative

command

present perfect

Polarity

Speech act

TenseTu M'as donné le livre.You have given me the book

Tu Me donnes le livre.You give me the book.

tu ME donne le livre.You give ME the book.

Donne-le MOI !Give it to ME !

10

Input present

SPEAKER LADIES Agent Object HELP

Direct and indirect consequences of a choice

11

Direct and indirect consequences of the verb choice

Direct consequences : syntactic function (DO /IO)

type of auxiliary (être/avoir)

Indirect consequences : form of the personal pronoun (les/leur)

verb ending : agreement (e/ées)

JeJe

Je

CHOICES

les ai aidés.

leur

ai porté du secours.leur

venu en aide.

C O N S E Q U E N C E S

VENIR EN AIDE

PORTER SECOURS

AIDER

suis

PRON AUX AGREEMENTVERB CHOICE

Paul la aide

SYNTACTIC FUNCTIONS & VOICE

LEXICALIZATION

aider = active voicePaul = subjectMarie = direct object

PARTS OF SPEECHaider = verb

Paul = nounMarie = pronoun

MESS AGE

PAUL = PaulMARIE = Marie

AIDER = aider

MORPHOLOGYverb : 3d person, singular, present -> aide

PRAGMATIC CHOICE

WORD ORDER

Paul l' aide.

Paul = topicMarie = givenaider = new

Subject : Noun -> PaulDirect object: pronoun -> la

PAUL Agent ObjetAIDER MARIE

PHONO-GRAPHEMIC ADJUSTMENTS

pré sent

SUBJECT

DIRECT OBJECT VERB

noun pronoun verb

24/02/15  

13

Input present

PRAGMATIC CHOICE Paul = topic Marie = given Aider = new

MORPHOLOGY Verb : 3d person, singular, present ⇒ aide Subject : Noun ⇒ Paul Direct object : pronoun ⇒ la

LEXICALIZATION HELP = aider PAUL = Paul MARY = Marie

PHONO- GRAPH. SYNTH.

Paul l’aide.

PAUL MARY Agent Object

PART OF SPEECH HELP = verb Paul = noun Mary = pronoun

WORD ORDER SUBJECT → noun

DIR. OBJECT → pronoun

VERB → verb

HELP

voice = active Paul = subject Mary = direct object

SYNT. FUNCT. & VOICE

Paul helps her

output

Problems at the articulatory stage:Voiced vs. voiceless fricatives: [ð] (this) vs. [θ] (thing)

How to pronounce the ‘th’ in English [ð] vs. [θ]

1.  Place tip of the tongue behind top teeth2.  Breathe out3.  Retract tongue4.  Vibrate air behind tongue and say5.  “The Smiths wear thin clothes througout the winter months”6.  Please, dont spit!

Problems at the articulatory stage:���tonguetwisters /s/ vs. /sh/ vs. /tch/

1.  We surely shall see the sun shine soon.2.  She sells sea-shells on the sea-shore.

3.  Which witch wished which wicked wish?4.  I thought a thought. But the thought I thought wasn't the

thought I thought I thought.

Challenge at the articulatory level :���tones, similar or unknown phonemes

target : 47 17 74���

47: sì shí qī 17: shí qī74: qī shí sì

24/02/15  

Beware of misunderstandings:���due to incorrect pronunciations���

���The Italian man who went to Malta

Language is a skill

Skillfull people manage to make difficult things look easy. Everything seems to flow naturally without any effort.

The  normal  situa/on  a  cascaded  flow  of  informa/on  

Speaking,  a  process  of  mapping  meanings  on  forms  

LOC

MOVMT.agt

loc

PERSON

SCENE -> ENTITY + PREDICATE ENTITY -> {PERSON, CITY...} PREDICATE -> {REL., ATTR...} ATTRIBUTE -> {SIZE, AGE ...}

S -> NP + VP NP -> {noun, pronoun...} N ->{John, Bali, Paris, ...} V ->{go, be, travel ...}

SNP

V

PRON PP

PREP N

VP

He went to Bali

CONCEPTUALGRAMMAR

LINGUISTIC STRUCTURECONCEPTUAL STRUCTURE

LINGUIST.GRAMMAR

Correspondance rules

Correspondancerules

How to account for the amazing language producer's performance?

Starting hypothesis

Linguistic structures can (at least partially) be predicted on the basis of conceptual structures

1. Systematic correspondances, i.e. default mappings. Pay attention to

➠  the type of nodes: box vs. oval➠  type of role, i.e. linktype: agent vs. attribute➠  direction of the link : incoming vs. outgoing

2. Proficiency grows with➠  the scope or window size: node vs. subgraph➠  the correctness of predictions, which inrease with experience.

Graphs and patterns of various complexity

adverbadjective Relative-clause

infinitive that-clause nominalization

Basic patterns

Basic patterns : ���3 predicates mapping into different parts of speech

agent

MAX

READ

Noun + Verb

attribute

MAX

CLEVER

Noun + Adjective

manner

RUN

FAST

...Verb + Adverb

‘Infinitival-construction’ vs. ‘That-clause’

object

JOHN

FEELMAX exper.

LIEagent

Infinitival construction That-clause

object

LIKEagent

DANCEagent

MAX

MAX

Max  likes  to  dance.   Max  felt  that  John  lied.  

Check  nature  of  the  arguments  (coreference)  and  link  types

‘Infinitival-construction’ vs. ‘That-clause’

Infinitive:She likes to dance

That-clause:She noticed that he lied.

Coreference

‘Nominalisation’ vs. ‘Relative-clause’

object

REGRETMAX exper.

ED LEAVEagent

Nominalisation Relative-clause

Max regretted Ed’s departure. Max knew the man who stole the car.

    2 incoming links pointing towards man.

‘Nominalisation’ vs. ‘Relative-clause’

NOMINALISATION:She regretted his departure.

Relative-clause:She knew the person

who called her.

2 incoming links pointing towards an argument

Potentially multiple mappings:���‘nominalisation’ vs. ‘that-clause’

object

JOHN

FEELMAX exper.

LIEagent

That-clause

Max  felt  that  John  lied.  

object

REGRETMAX exper.

ED LEAVEagent

Nominalisation

Max regretted Ed’s departure.

check ‘nominalizability’