The Computational Nature of Language Learning and...
Transcript of The Computational Nature of Language Learning and...
The Computational Nature ofLanguage Learning and Evolution
Partha Niyogi
The University of Chicago
The Computational Nature ofLanguage Learning and Evolution – p.1
A Linguistic Fact of English
1 (a) He ran from there with his money.
1 (b) He his money with there from ran.
�
��
Linguistic Experience Linguistic Knowledge
The Computational Nature ofLanguage Learning and Evolution – p.2
A Linguistic Fact of English
1 (a) He ran from there with his money.
1 (b) He his money with there from ran.
�
��
Linguistic Experience � � Linguistic Knowledge
The Computational Nature ofLanguage Learning and Evolution – p.2
The Logical Problem of Language Acquisition
� ��� � �
target grammar
��� �ve examples
Learner (Child)
� ���
� �
Learnability
Gold (1967); Valiant (1984)
The Computational Nature ofLanguage Learning and Evolution – p.3
The Logical Problem of Language Acquisition
� ��� � �
target grammar
��� �ve examples
Learner (Child)
� ���
� �
Learnability
��� � ��
Gold (1967); Valiant (1984)
The Computational Nature ofLanguage Learning and Evolution – p.3
Principles and Parameters
SwahiliChinese Hindi
� �
�� �� � � �� �
�English French
Warlpiri
The Computational Nature ofLanguage Learning and Evolution – p.4
X-bar Theory: English
VP
V’
PP
V’
ran
V
Spec
(Comp)
(Comp)
(empty)
XP −−> Spec X’
X’ −−> X’ Comp
P’
NP
N’
P’
with his money
P
Spec
Spec
(Comp)
(empty)
P’
N
PP
Spec P’
P’ NP (Comp)
N’SpecP
N
therefrom
(empty)
(empty)
The Computational Nature ofLanguage Learning and Evolution – p.5
X-bar Theory: Bengali
XP −−> Spec X’
Spec
(empty)
(Comp)PP
VP
V’
V’
V
Spec
(empty)
P’
(Comp)NP
N’Spec
N
P’
P
Spec P’
P’
P
(empty)
NP (Comp)
N’Spec
N(empty)
or niyepaisa
(Comp)PP
X’ −−> Comp X’
shekhan theke douralo
(his) (money) (with) (there) (from) (ran)
The Computational Nature ofLanguage Learning and Evolution – p.6
The (Il)-Logical Problem of Language Change
Langagis, whos reulis ben not writen, as benEnglisch, Frensch and many otheres, ben channgidwithynne yeeris and countrees that oon man of theoon cuntre, and of the oon tyme, myghte not, orschulde not kunne undirstonde a man of the otherekuntre, and of the othere tyme; and al for this, thatthe seid langagis ben not stabili and fondamentaliwriten
Pecock (1454) Book of Feith (from Roberts, 1993)
The Computational Nature ofLanguage Learning and Evolution – p.7
The Evolution of English
Her ... Aelfred cyning ... gefeaht wid ealne, here, and hineHere Alfred king fought against whole army and it
geflymde and him aefter rad od pet geweorc, and paer saetput to flight and it after rode to the fortress and there camped
XIII niht, and pa sealde se here him gislas and mycclefourteen nights and then gave the army him hostages and great
adas, pet he of his rice woldon, and him eac gehetonoaths that they from his kingdom would [go] and him also promised
pet heora cyng fulwihte onfon wolde, and hi paet gelastonthat their king baptism receive would and they that did
The Computational Nature ofLanguage Learning and Evolution – p.8
More Old English
pa Darius geseah paet he oferwunnen beon woldethen Darius saw that [he conquered be would]
(Orosius 128.5)& him aefterfylgende waesand [him following was]
(Orosius 236.29)Nu ic wille eac paes maran Alexandres gemunende beonnow I will also [the great Alexander considering be]
(Orosius 110.10)
The Computational Nature ofLanguage Learning and Evolution – p.9
Complex Constructions
ondraedende paet Laecedemonie ofer hie ricsian mehten swa hie aer dydondreading that Laecedemonians over them rule might as they before did
“dreading that the Laecedemonians might rule over them as they had done in the past”(Orosius 98.17)
peh ne geortriewe ic na Gode paet he us ne maege gescildanalthough not shall-distrust I never to-God, that he us not can shield
“although I shall never distrust God so much as to think he cannot shield us”(Orosius 86.3)
The Computational Nature ofLanguage Learning and Evolution – p.10
Ogden Nash, 1962
Farewell, farewell to my beloved languageOnce English, now a vile orangutanguage
The Computational Nature ofLanguage Learning and Evolution – p.11
Evolution and Learning
...if languages were learnt perfectly by the children ofeach generation, then language would not change:English children would still speak a language as oldatleast as Anglo Saxon and there would be no suchlanguages as French or Italian.
(H. Sweet, 1899)
The Computational Nature ofLanguage Learning and Evolution – p.12
Language Change
ISSUES/QUESTIONS
1. Group Level Description: How does one model populations of linguistic agents?
2. Time Course: How fast do languages change? Can one predict their possible evolutionary patterns?
3. Directionality: When two language types come together, in which direction will the children evolve?
HISTORICAL PHENOMENA
1. Syntax: Change in word order in English, French, Portuguese, etc.
2. Phonology: (a) Change in metrical stress from Proto-IndoEuropean to modern Greek (b) TheGreat Vowel Shift in English...
3. Creoles: Rapid language formation. Do all creoles have similar properties?
4. Language Typology: What are language types? How are they distributed? How do theychange?
The Computational Nature ofLanguage Learning and Evolution – p.13
Language Evolution
1. Origin of Language: How did combinatorial, recursive structures emerge?
2. Communicative Efficiency: What is the role of communicative efficiency and natural selection?
3. Communicative Coherence: How do shared communication systems arise by self organization?
4. Diversity: How did the diversity of natural communicative systems evolve?
Birds, BeesWhales, DolphinsPrimates, Humans
The Computational Nature ofLanguage Learning and Evolution – p.14
Population Linguistics
MICROSCOPIC
LANGUAGE ACQUISITION
INDIVIDUAL #EXAMPLES
MACROSCOPIC
POPULATION LINGUISTICCOMPOSITION
#GENERATION
LANGUAGE CHANGE
GRAMMATICALHYPOTHESIS
The Computational Nature ofLanguage Learning and Evolution – p.15
Timeline of Inquiry
18th century William Jones
(Indo-European Thesis)
19th century Charles Darwin
20th century Linguistic Structure Genetic Code
(Generative Grammar) DNA (Molecular)
The formation of different languages and distinct species, .... are curiously parallel.
(Charles Darwin, Descent of Man, 1871)
The Computational Nature ofLanguage Learning and Evolution – p.16
Timeline of Inquiry
18th century William Jones
(Indo-European Thesis)
19th century Charles Darwin
20th century Linguistic Structure Genetic Code
(Generative Grammar) DNA (Molecular)
The formation of different languages and distinct species, .... are curiously parallel.
(Charles Darwin, Descent of Man, 1871)
The Computational Nature ofLanguage Learning and Evolution – p.16
Evolution in Linguistics and Biology
g1 g1 g1
g2g5
1D D2
A A
g h
grammatical variation in adults
transmission via learning
grammatical variation in children
genetic variation in adults
transmission via inheritance
genetic variation in children
Language Evolution and Biological Evolution
Natural Selection ??
The Computational Nature ofLanguage Learning and Evolution – p.17
Major Insights
1. Different learning algorithms have different evolutionaryconsequences.evolutionary criteria in addition to learnability
learning in heterogeneous populations
2. Phase transition phenomena in linguistic evolution.subtle changes in frequency may lead to dramatic changes in language
3. Natural selection, Social connectivity, and theEmergence of Language.conditions for a shared language to emerge
The Computational Nature ofLanguage Learning and Evolution – p.18
The Basic Framework
1.
� ��� �� � � �
��
�
2.
��� � �� � �� � � �� �
� � � � � �
3.
�� �
� �� � � ��
�
4. � � �� � �
5.
�
: maturation time
6.
� � � � � � � � � �
The Computational Nature ofLanguage Learning and Evolution – p.19
Two Language Models
� � �� �
� � �..that Hans buys the book
Hans buys the book
Hans kauft das Buch
..dass Hans das Buch kauft
where and
The Computational Nature ofLanguage Learning and Evolution – p.20
Two Language Models
� � �� �
� � �..that Hans buys the book
Hans buys the book
Hans kauft das Buch
..dass Hans das Buch kauft
� �� �
� �
where��� �� � � ��� � � and
�� �� � � ��� � �
The Computational Nature ofLanguage Learning and Evolution – p.20
Two Language Models
� � �� �
� � �..that Hans buys the book
Hans buys the book
Hans kauft das Buch
..dass Hans das Buch kauft
� �� �
� �
where��� �� � � ��� � � and
�� �� � � ��� � �
� �
The Computational Nature ofLanguage Learning and Evolution – p.20
Triggering Models
1. Start with arbitrary hypothesis.
2. Receive new example sentence, � �
3. if ( � � parsed)then go to 2else change hypothesis
The Computational Nature ofLanguage Learning and Evolution – p.21
The Logic of Language Evolution
Consider a typical child
� � � � � � �� � � �� �
After
�
examples,
� � � � � � �� � ��� � ��
�� �
Therefore in the next generation,
� �� � � ��� � ��
�� �
The Computational Nature ofLanguage Learning and Evolution – p.22
Population Dynamics
� � � �
� � �� �� � � � �� � � � � �
� � �
� �� � � � �� � � � � � �� � � � � �
no change
The Computational Nature ofLanguage Learning and Evolution – p.23
Population Dynamics
� � � �
� � �� �� � � � �� � � � � �
� � �
� �� � � � �� � � � � � �� � � � � �
� �� �
� �� � � � �
� � � � � � � �� � � � �
� � � � � � � �
� � � �
no change � � � �
The Computational Nature ofLanguage Learning and Evolution – p.23
Appreciating the Bifurcation
Let � ��
� �
and � ��
� �
,
� ��
� �
.
� � � � � �� � � � �
In the experience of the typical child:1. Overwhelmingly many triggers for
unique parseunique parse
2. With high probability the child acquires
Yet....
The Computational Nature ofLanguage Learning and Evolution – p.24
Appreciating the Bifurcation
Let � ��
� �
and � ��
� �
,
� ��
� �
.
� � � � � �� � � � �
In the experience of the typical child:1. Overwhelmingly many triggers for
� �
� � �
unique
� � parse
� � �
unique
��� parse
2. With high probability
���
� � � �
the child acquires
� �
Yet....
The Computational Nature ofLanguage Learning and Evolution – p.24
Cue Based Models
� � � �
� � �
� �
� � �
1. Receive
�
example sentences.
2. Let
�
be
�
of cue sentences.
3. if �� ���
choose
�� .
The Computational Nature ofLanguage Learning and Evolution – p.25
Population Dynamics
� � � �
���� ���
�
�� �� � � � � �� �� � � �
� �
(stable) � � �� (unstable) � � �� � � �� (stable)
� �
not equilibrium
� �� �� ���
� � �� � �� � �� ��� �� �� �� �
0 x x1 2
**
The Computational Nature ofLanguage Learning and Evolution – p.26
The Bifurcation
−0.2 0 0.2 0.4 0.6 0.8 1 1.2−0.2
0
0.2
0.4
0.6
0.8
1
1.2
Parameter p
Fix
ed P
oint
s
The Computational Nature ofLanguage Learning and Evolution – p.27
Middle English
NORTHERN GRAMMAR SOUTHERN GRAMMAR
(Scandinavian) (Saxon)
� �� � ��
SV, SVO SV, SVO
OVS S V O1 O2
S V O1 O2 Adv S V
O1 V S O2 Adv S V O1 O2
O2 V S O1 Adv S V O1 O2
Adv V S O S Aux V
etc. etc.
The Computational Nature ofLanguage Learning and Evolution – p.28
Evolution and Communicative Fitness
In the evolution of languages the dicarding of oldflexions goes hand in hand with the development ofsimpler and more regular expedients that are ratherless liable than the old ones to producemisunderstanding.
Otto Jespersen (1922)
The Computational Nature ofLanguage Learning and Evolution – p.29
The Emergence of Coherence – Fitness
1.
� ��� �
� � �� � � ���
�
2.
� �� � �� � �
– mutual intelligibility
� � � ��� � � � �
and
� � � ��� � � � �
3.
� � � – parent
� � � child
� �
� � � � and
� � � � �� �
The Computational Nature ofLanguage Learning and Evolution – p.30
Population Dynamics
State
� � � �� � � �� �
� � � � � �
Fitness
Dynamics
Continuous
The Computational Nature ofLanguage Learning and Evolution – p.31
Population Dynamics
State
� � � �� � � �� �
� � � � � �
Fitness
� � ��� � �� � �
� �� � �� � � � �
Dynamics
Continuous
The Computational Nature ofLanguage Learning and Evolution – p.31
Population Dynamics
State
� � � �� � � �� �
� � � � � �
Fitness
� � ��� � �� � �
� �� � �� � � � �
Dynamics � � � � � � �
� � � � � � �� � � � �
� � � � � � �� �
Continuous
�� � � � � � � � � � � � � � � � � � � � �
� � � � �
The Computational Nature ofLanguage Learning and Evolution – p.31
The Bifurcation
0.85 0.90 0.95 1.00q
0.0
0.2
0.4
0.6
0.8
1.0X
X-
X+
1/n
q1 q2
The Computational Nature ofLanguage Learning and Evolution – p.32
No fitness, no coherence
� � ���
�� � ���
�
�� � � � � � � ��� � �
�� � � � �
� �
� �� � � �
�
� �
The Computational Nature ofLanguage Learning and Evolution – p.33
Emergence of Coherence – Social Learning
1. No fitness2. Learn from Everybody
� ��� �
� � �� � � ���
�
� � �� �
� � �� � � �
��
�
� � � � �
The Computational Nature ofLanguage Learning and Evolution – p.34
Learning Algorithm
count cues
� �
�� � � ���� �
� if unique
� ��
otherwise
cue
�
with probability � � �
The Computational Nature ofLanguage Learning and Evolution – p.35
Population Dynamics
� � � � � � � � � � � � � �
� � � � � ��
�� ��
� � �
� � ��� �� � � �� � � �� �
� ��
��� � � � ���
�p � � � � 1
�� � �
p� � � � � �
where
� � � � � � � � � � � ��� � � �� � � largest
�
The Computational Nature ofLanguage Learning and Evolution – p.36
Bifurcations
1. small
�
– uniform solution
2. if
� � ��� – new solutions
3. if
� � ��� – uniform solutions become unstable
4. finite number (
� � � � � �
) of fixed points
5. Only are stable.
The Computational Nature ofLanguage Learning and Evolution – p.37
Further Issues
1.
� � �
: variety of linguistic theories
2. variety of learning algorithms
3.
�
4. Generational structure
5. Maturation time: developmental constraints
6. Neighborhood effects and social stratification
7. Finite population effects
8. Bilingual and Multilingual settings
The Computational Nature ofLanguage Learning and Evolution – p.38
Spatial and Network Effects
� ����
��
is influence of � on �
0.33/0.3510 20 30 40 50
10
20
30
40
50
0.31/0.3310 20 30 40 50
10
20
30
40
50
0.2/0.410 20 30 40 50
10
20
30
40
50
0/0.8510 20 30 40 50
10
20
30
40
50
0.1/0.2710 20 30 40 50
10
20
30
40
50
0/010 20 30 40 50
10
20
30
40
50
The Computational Nature ofLanguage Learning and Evolution – p.39
Major Insights
1. Different learning algorithms have different evolutionaryconsequences.memoryless/batchsymmetric/asymmetricmultilingual/monlingual
single teacher/many teachers
2. Phase transition phenomena in linguistic evolution.subtle effects of frequencyexplanations of language change
explanations of dialect formation
3. Natural selection, Social connectivity, and the Emergenceof LanguageCoherence conditions
Social connectivity
The Computational Nature ofLanguage Learning and Evolution – p.40
Conclusions
1. Time scales of Language Evolution(a) Evolutionary - origin
(b) Historical - change
2. Role of Learning
3. Role of Computational Models
4. Empirical Validation
5. There are deep connections to(a) population models in evolutionary biology(b) artificial life and multiagent systems
(c) bounded rationality in social sciences
The Computational Nature ofLanguage Learning and Evolution – p.41