Testing Functional Explanations of Word Order...
Transcript of Testing Functional Explanations of Word Order...
![Page 1: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/1.jpg)
Testing Functional Explanations of Word Order Universals
Michael Hahn Richard FutrellStanford UC Irvine
![Page 2: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/2.jpg)
(Greenberg 1963)
![Page 3: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/3.jpg)
U3: ‘Languages with dominant VSO order are alwaysprepositional.’
![Page 4: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/4.jpg)
U3: ‘Languages with dominant VSO order are alwaysprepositional.’
U4: ‘With overwhelmingly greater than chancefrequency, languages with normal SOV order arepostpositional.’
![Page 5: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/5.jpg)
U3: ‘Languages with dominant VSO order are alwaysprepositional.’
U4: ‘With overwhelmingly greater than chancefrequency, languages with normal SOV order arepostpositional.’
`Relative position of adposition & noun ~relative position ofverb & object’
![Page 6: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/6.jpg)
OV languages with postpositions
VO languages with prepositions
![Page 7: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/7.jpg)
![Page 8: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/8.jpg)
Why do these universals hold?
Innate constraints on language, ‘Universal Grammar’? (Chomsky 1981)
Facilitation of human communication? (Dryer 1992, Hawkins 1994)
Make languages learnable? (Culbertson 2017)
![Page 9: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/9.jpg)
Why do these universals hold?
Innate constraints on language, ‘Universal Grammar’? (Chomsky 1981)
Facilitation of human communication? (Dryer 1992, Hawkins 1994)
Approach: Test functional explanations by implementing efficiency measures, optimizing grammars, and checking whether universals hold in optimized grammars.
Make languages learnable? (Culbertson 2017)
![Page 10: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/10.jpg)
Three Efficiency Measures
Dependency Length Minimization (Rijkhoff, 1986; Hawkins, 1994, 2003)
Surprisal (Gildea and Jaeger, 2015; Ferrer-i Cancho, 2017)
Parsability (Hawkins, 1994, 2003)
![Page 11: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/11.jpg)
Three Efficiency Measures
Dependency Length Minimization (Rijkhoff, 1986; Hawkins, 1994, 2003)
![Page 12: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/12.jpg)
Three Efficiency Measures
Dependency Length Minimization (Rijkhoff, 1986; Hawkins, 1994, 2003)
21 1
![Page 13: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/13.jpg)
Three Efficiency Measures
Dependency Length Minimization (Rijkhoff, 1986; Hawkins, 1994, 2003)
21 1+ + = 4
![Page 14: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/14.jpg)
Three Efficiency MeasuresSurprisal
Surprisal(w1...wi-1) = -Σi log P(wi|w1...wi-1)
![Page 15: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/15.jpg)
Three Efficiency MeasuresSurprisal
Surprisal(w1...wi-1) = -Σi log P(wi|w1...wi-1)
Estimated using recurrent neural networks, the strongest existing methods for estimating surprisal and predicting reading times.
![Page 16: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/16.jpg)
Three Efficiency MeasuresParsability
Mary has two green books.
![Page 17: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/17.jpg)
Three Efficiency MeasuresParsability
Mary has two green books.
Parsability(utterance) := log P(tree | utterance)
![Page 18: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/18.jpg)
Three Efficiency MeasuresParsability
Mary has two green books.
Parsability(utterance) := log P(tree | utterance)
Estimated using a neural network model (Dozat and Manning 2017)
with extremely generic architecture.
![Page 19: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/19.jpg)
Utility Informativity Cost-=
Amount of Meaning that can be extracted from utterance
Cost of processing utterance
λ
Combining Parsability + Surprisal
![Page 20: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/20.jpg)
Utility Informativity Cost-=
Amount of Meaning that can be extracted from utterance
Cost of processing utterance
Long tradition as an explanation of language (Gabelentz 1903, Zipf 1949, Horn 1984, …)
λ
Combining Parsability + Surprisal
![Page 21: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/21.jpg)
Utility Informativity Cost-=
Amount of Meaning that can be extracted from utterance ~ Parsability
Cost of processing utterance
~ Surprisal
λ
Combining Parsability + Surprisal
Long tradition as an explanation of language (Gabelentz 1903, Zipf 1949, Horn 1984, …)
![Page 22: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/22.jpg)
Utility Informativity Cost-=
Amount of Meaning that can be extracted from utterance ~ Parsability
Cost of processing utterance
~ SurprisalLong tradition as an explanation of language (Gabelentz 1903, Zipf 1949, Horn 1984, …)
Formalized in Rational-Speech Acts models (Frank and Goodman 2012)
λ
Combining Parsability + Surprisal
![Page 23: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/23.jpg)
Utility Informativity Cost-=
Long tradition as an explanation of language (Gabelentz 1903, Zipf 1949, Horn 1984, …)
Formalized in Rational-Speech Acts models (Frank and Goodman 2012)
Related to Signal Processing (Rate-Distortion Theory, Information Bottleneck)
λ
Combining Parsability + Surprisal
Amount of Meaning that can be extracted from utterance ~ Parsability
Cost of processing utterance
~ Surprisal
![Page 24: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/24.jpg)
Why do the universals hold?
Innate constraints on language, ‘Universal Grammar’? (Chomsky 1981)
Facilitation of human communication? (Dryer 1992, Hawkins 1994)
Approach: Test processing explanations by implementing efficiency measures, optimizing grammars, and checking whether universals hold in optimized grammars.
Make languages learnable? (Culbertson 2017)
![Page 25: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/25.jpg)
Testing Functional Explanations
Approach: Optimize the word orders of languages for the three objectives, keeping syntactic structures unchanged
![Page 26: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/26.jpg)
Testing Functional Explanations
Approach: Optimize the word orders of languages for the three objectives, keeping syntactic structures unchanged
Languages have word order regularities ⇒ Not sufficient to optimize the word orders of individual sentences
![Page 27: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/27.jpg)
Testing Functional Explanations
Approach: Optimize the word orders of languages for the three objectives, keeping syntactic structures unchanged
Languages have word order regularities ⇒ Not sufficient to optimize the word orders of individual sentences
Instead: optimize word order rules of entire languages
![Page 28: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/28.jpg)
Testing Functional Explanations
Approach: Optimize the word orders of languages for the three objectives, keeping syntactic structures unchanged
Languages have word order regularities ⇒ Not sufficient to optimize the word orders of individual sentences
Instead: optimize word order rules of entire languages
That is: optimized languages have optimized but internally consistent grammatical regularities in word order, and agree with an actual natural language in all other respects.
![Page 29: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/29.jpg)
Mary has two green books
nsubj
dobj
nummod
amod
Dependency Corpus
![Page 30: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/30.jpg)
Mary has two green books
nsubj
dobj
nummod
amod
Mary
hastwo
greenbooks
Tree Topologies
Dependency Corpus
![Page 31: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/31.jpg)
Mary has two green books
nsubj
dobj
nummod
amod
Mary
hastwo
greenbooks
Tree Topologies
Dependency Corpus Ordering GrammarNOUN ADJamod
0.3
NOUN NUMnummod
VERB NOUNnsubj
VERB NOUNdobj
...
0.7
-0.2
0.8
![Page 32: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/32.jpg)
Mary has two green books
nsubj
dobj
nummod
amod
Mary
hastwo
greenbooks
Tree Topologies
Dependency Corpus Ordering GrammarNOUN ADJamod
0.3
NOUN NUMnummod
VERB NOUNnsubj
VERB NOUNdobj
...
0.7
-0.2
0.8
“Object follows verb”
![Page 33: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/33.jpg)
Mary has two green books
nsubj
dobj
nummod
amod
Mary
hastwo
greenbooks
Tree Topologies
Dependency Corpus Ordering GrammarNOUN ADJamod
0.3
NOUN NUMnummod
VERB NOUNnsubj
VERB NOUNdobj
...
0.7
-0.2
0.8
“Adjective precedes noun”
“Object follows verb”
![Page 34: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/34.jpg)
Mary has two green books
nsubj
dobj
nummod
amod
Mary
hastwo
greenbooks
Tree Topologies
Dependency Corpus Ordering GrammarNOUN ADJamod
0.3
NOUN NUMnummod
VERB NOUNnsubj
VERB NOUNdobj
...
0.7
-0.2
0.8
“Adjective precedes noun”
“Object follows verb”
“Numerals follow adjectives & precede nouns”
![Page 35: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/35.jpg)
Mary has two green books
nsubj
dobj
nummod
amod
Mary
hastwo
greenbooks
Tree Topologies
Maryhastwogreenbooks
Counterfactual Corpus
Dependency Corpus Ordering GrammarNOUN ADJamod
0.3
NOUN NUMnummod
VERB NOUNnsubj
VERB NOUNdobj
...
0.7
-0.2
0.8
![Page 36: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/36.jpg)
Mary has two green books
nsubj
dobj
nummod
amod
Mary
hastwo
greenbooks
Tree Topologies
Maryhastwogreenbooks
Counterfactual Corpus
Dependency Corpus Ordering GrammarNOUN ADJamod
0.3
NOUN NUMnummod
VERB NOUNnsubj
VERB NOUNdobj
...
0.7
-0.2
0.8
Each parameter setting generates a different counterfactual corpus.
![Page 37: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/37.jpg)
Mary has two green books
nsubj
dobj
nummod
amod
Mary
hastwo
greenbooks
Tree Topologies
Maryhastwogreen books
Counterfactual Corpus
Dependency Corpus Ordering GrammarNOUN ADJamod
0.9
NOUN NUMnummod
VERB NOUNnsubj
VERB NOUNdobj
...
0.1
0.5
0.2
Each parameter setting generates a different counterfactual corpus.
![Page 38: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/38.jpg)
Mary has two green books
nsubj
dobj
nummod
amod
Mary
hastwo
greenbooks
Tree Topologies
Maryhas twogreenbooks
Counterfactual Corpus
Dependency Corpus Ordering GrammarNOUN ADJamod
0.1
NOUN NUMnummod
VERB NOUNnsubj
VERB NOUNdobj
...
0.95
04.2
0.82
Each parameter setting generates a different counterfactual corpus.
![Page 39: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/39.jpg)
Dependency Length Surprisal
Parsability
2.35.81.8
We compute processing measures on counterfactual corpora.
![Page 40: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/40.jpg)
Dependency Length Surprisal
Parsability
2.35.81.8
Each parameter setting results in different values for the processing measures.
![Page 41: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/41.jpg)
Dependency Length Surprisal
Parsability
2.94.52.9
Each parameter setting results in different values for the processing measures.
![Page 42: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/42.jpg)
Dependency Length Surprisal
Parsability
3.47.81.2
Each parameter setting results in different values for the processing measures.
![Page 43: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/43.jpg)
Dependency Length Surprisal
Parsability
3.47.81.2
Each parameter setting results in different values for the processing measures.
Which settings optimise the measures?
![Page 44: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/44.jpg)
Dependency Length Surprisal
Parsability
3.47.81.2
Each parameter setting results in different values for the processing measures.
Which settings optimise the measures?
Do the optimised settings replicate the Greenberg correlations?
![Page 45: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/45.jpg)
For each objective, find parameters that optimise it.
NOUN ADJamod0.1
NOUN NUMnummod
VERB NOUNnsubj
VERB NOUNdobj
...
0.95
04.2
0.82
NOUN ADJamod0.1
NOUN NUMnummod
VERB NOUNnsubj
VERB NOUNdobj
...
0.85
0.1
0.22
Minimize Dep. Length Minimize Surprisal
NOUN ADJamod0.1
NOUN NUMnummod
VERB NOUNnsubj
VERB NOUNdobj
...
0.7
0.5
0.8
NOUN ADJamod0.21
NOUN NUMnummod
VERB NOUNnsubj
VERB NOUNdobj
...
0.45
0.4
0.32
Maximize Parsability Optimize Pars.+Surp.
![Page 46: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/46.jpg)
For each objective, find parameters that optimise it.
Repeat this for corpora from 51 real languages from Universal Dependencies Project.
NOUN ADJamod0.1
NOUN NUMnummod
VERB NOUNnsubj
VERB NOUNdobj
...
0.95
04.2
0.82
NOUN ADJamod0.1
NOUN NUMnummod
VERB NOUNnsubj
VERB NOUNdobj
...
0.85
0.1
0.22
Minimize Dep. Length Minimize Surprisal
NOUN ADJamod0.1
NOUN NUMnummod
VERB NOUNnsubj
VERB NOUNdobj
...
0.7
0.5
0.8
NOUN ADJamod0.21
NOUN NUMnummod
VERB NOUNnsubj
VERB NOUNdobj
...
0.45
0.4
0.32
Maximize Parsability Optimize Pars.+Surp.
![Page 47: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/47.jpg)
For each objective, find parameters that optimise it.
Repeat this for corpora from 51 real languages from Universal Dependencies Project.
0.1
0.95
04.2
0.82
0.1
0.85
0.1
0.22
Minimize Dep. Length Minimize Surprisal
NOUN ADJamod0.1
NOUN NUMnummod
VERB NOUNnsubj
VERB NOUNdobj
...
NOUN ADJ 0.1
NOUN NUMnummod
VERB NOUNnsubj
VERB NOUNdobj
...
0.7
0.5
0.8
0.7
0.5
0.8
0.21
0.45
NOUN ADJ 0.1
NOUN
NOUN ADJ 0.1
NOUN NUMnummod
VERB NOUNnsubj
VERB NOUNdobj
...
0.7
0.5
0.8
NUMnummod
VERB NOUNnsubj
VERB NOUNdobj
...
0.7
0.5
0.8
0.4
0.32
Maximize Parsability Optimize Pars.+Surp.
1. How do the objectives compare?2. Which universals are predicted?
Minimize Dep. Length Minimize Surprisal
![Page 48: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/48.jpg)
Surprisal and Parsability minimize Dependency Length
![Page 49: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/49.jpg)
Surprisal and Parsability minimize Dependency Length
![Page 50: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/50.jpg)
Surprisal and Parsability minimize Dependency Length
Communicative Utility predicts Dependency Length Minimization.
![Page 51: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/51.jpg)
Better Parsability
Lower Surprisal
z-transformed on the level of languages
Language optimizes Surprisal and Parsability
![Page 52: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/52.jpg)
Better Parsability
Lower Surprisal
Random Grammars
Language optimizes Surprisal and Parsability
![Page 53: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/53.jpg)
Better Parsability
Lower Surprisal
Random Grammars
Grammars fit to Real Orderings
Language optimizes Surprisal and Parsability
Better Parsability
![Page 54: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/54.jpg)
Better Parsability
Lower Surprisal
Random Grammars
Optimized for Surprisal
Optimized for Parsability
Optimized for Parsability+Surprisal
Grammars fit to Real Orderings
Language optimizes Surprisal and Parsability
![Page 55: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/55.jpg)
![Page 56: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/56.jpg)
(Dryer 1992 in Language)
![Page 57: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/57.jpg)
(Dryer 1992 in Language)
`Relative position of adposition & noun ~relative position ofverb & object’
![Page 58: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/58.jpg)
We formalize the correlations in the Universal Dependencies format.
![Page 59: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/59.jpg)
(Dryer 1992 in Language)
![Page 60: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/60.jpg)
X
XX
![Page 61: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/61.jpg)
We formalize the correlations in the Universal Dependencies format.
For any word order grammar, we can then check which correlations it satisfies.
![Page 62: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/62.jpg)
Are the universals satisfied by models fit to the actual orderings for our 50 languages?
%
![Page 63: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/63.jpg)
Are the universals satisfied by models fit to the actual orderings for our 50 languages?
%
![Page 64: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/64.jpg)
Are the universals satisfied by models fit to the actual orderings for our 50 languages?
Prevalence of SVO (Dryer 1992)
Limitation of formalisation
%
![Page 65: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/65.jpg)
![Page 66: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/66.jpg)
Percentage of grammars optimized for each objective satisfying the universal
![Page 67: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/67.jpg)
Percentage of grammars optimized for each objective satisfying the universal
Assessing Significance:X = “Object precedes verb”Y = “Object-patterner precedes verb-patterner”
Logistic model:Y ~ X + (1+X|family) + (1+X|language)
![Page 68: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/68.jpg)
![Page 69: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/69.jpg)
![Page 70: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/70.jpg)
![Page 71: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/71.jpg)
![Page 72: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/72.jpg)
Predictions largely complementary
![Page 73: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/73.jpg)
Predictions mostly agree
![Page 74: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/74.jpg)
Predictions mostly agree
Communicative Utility replicates predictions of Dependency Length Minimization.
![Page 75: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/75.jpg)
Predictions mostly agree
Communicative Utility replicates predictions of Dependency Length Minimization.Both measures predict most of the correlation universals.
![Page 76: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/76.jpg)
Conclusion
● Tested explanations of Greenberg correlation universals in terms of efficiency of human processing and communication
● Using corpora from 50 languages, constructed counterfactual optimized languages
● Most of the correlations can be derived from pressure to shorten dependencies, decrease surprisal, or increase parsability
● Clear evidence for functional explanations of word order universals
![Page 77: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/77.jpg)
![Page 78: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/78.jpg)
Optimized grammars are easier to parse even when sentences are presented in orders very different from natural language
ACEBD ADBEC ACEDBABCDE
![Page 79: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/79.jpg)
Random grammarOptimized grammar
Random grammars remain hard to parse even as training data increases.
![Page 80: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/80.jpg)
![Page 81: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/81.jpg)
Formalizing ParsabilityNeural parser (Dozat and Manning 2017):
Mary met John
R
Mar
y
met
Jo
hn 1. BiLSTM reads the sentence2. Identify heads by
computing score for each pair of words
Generic architecture, no assumptions beyond sequential nature of input.
![Page 82: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/82.jpg)
Formalizing ParsabilityInformation about syntactic tree that can be extracted from sentence:
Mary met John
R
Mar
y
met
Jo
hn
![Page 83: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/83.jpg)
Formalizing Dependency Length
Distance between word and its syntactic head
summing over all words in sentence
sentence w = w1...wn
![Page 84: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/84.jpg)
Formalizing Surprisal
![Page 85: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/85.jpg)
Formalizing Surprisal
summing over all words in sentence
per-word surprisal
![Page 86: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/86.jpg)
Formalizing Surprisal
Surprisal depends on the probability model P.
![Page 87: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/87.jpg)
Formalizing Surprisal
Surprisal depends on the probability model P.
Right choice of P depends on the entire language!
![Page 88: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/88.jpg)
Formalizing Surprisal
Given a word order grammar θd choose the model that minimizes surprisal on the resulting sentences.
![Page 89: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/89.jpg)
Formalizing Surprisal
Use LSTM recurrent neural networks, the SOTA in probabilistic modelling of natural language and predicting reading times.Very general sequence models, arguably minimizing architectural biases.
![Page 90: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/90.jpg)
Formalizing Informativity
Information about the syntactic tree that can be extracted from the sentence:
![Page 91: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/91.jpg)
Formalizing Informativity
Information about the syntactic tree that can be extracted from the sentence:
Use a recent neural model (Dozat and Manning 2017) with generic architecture and SOTA performance on many languages.
![Page 92: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/92.jpg)
Word Order Grammars
For each dependency type, there are two parameters:a. α: probability that whether dependent precede headb. β: determines distance
![Page 93: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/93.jpg)
Mary
has
two green
booksαverb-object = 0.1
αverb-subject = 0.95
αnoun-numeral = 0.99 αnoun-adjective = 0.8
![Page 94: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/94.jpg)
Mary
has
two green
booksαverb-object = 0.1
αverb-subject = 0.95
αnoun-numeral = 0.99 αnoun-adjective = 0.8
![Page 95: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/95.jpg)
Maryhas
two
green
books
![Page 96: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/96.jpg)
Word Order Grammars
For each dependency type, there are two parameters:a. α: probability that dependent precede headb. β: determines distance
![Page 97: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/97.jpg)
Mary
has
two
green
books
βNoun-Adjective = -0.3
βNoun-Numeral = 0.8
![Page 98: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/98.jpg)
Mary
has
two
green
books
βNoun-Adjective = -0.3
βNoun-Numeral = 0.8
softmax(βNoun-Adjective , βNoun-Numeral ) ~ (0.1, 0.9)
adjective first
numeral first
![Page 99: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/99.jpg)
Mary
has
two
green
books
βNoun-Adjective = -0.3
βNoun-Numeral = 0.8
softmax(βNoun-Adjective , βNoun-Numeral ) ~ (0.1, 0.9)
adjective first
numeral first
![Page 100: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/100.jpg)
Maryhas
twogreen
books
![Page 101: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/101.jpg)
Maryhastwogreenbooks
![Page 102: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/102.jpg)
Word Order Grammars
For each dependency type, there are two parameters:a. α: probability that dependent precede headb. β: determines distance
This specifies the space of possible grammars, within which we optimize.
![Page 103: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/103.jpg)
Mary has two green books
nsubj
dobj
nummod
amod
Mary
hastwo
greenbooks
Tree Topologies
Maryhastwogreenbooks
Counterfactual Corpus
Dependency Corpus Ordering GrammarNOUN ADJamod
0.3
NOUN NUMnummod
VERB NOUNnsubj
VERB NOUNdobj
...
0.7
-0.2
0.8
![Page 104: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/104.jpg)
Mary has two green books
nsubj
obj
nummod
amod
Will be working with trees in the Universal Dependencies format:
![Page 105: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/105.jpg)
Mary has two green books
nsubj
obj
nummod
amod
Will be working with trees in the Universal Dependencies format:
To optimize grammars, we need a space of possible grammars.
![Page 106: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/106.jpg)
![Page 107: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/107.jpg)
![Page 108: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/108.jpg)
![Page 109: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/109.jpg)
SOV
SVO
SOV and VSO support correlationSVO does not
VSO
![Page 110: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/110.jpg)
Support SVO(Gibson et al 2013)
![Page 111: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/111.jpg)
Dependency Length MinimizationShort syntactic dependencies ease processing (Gibson, 1998; Grodner and Gibson, 2005; Demberg and Keller, 2008; Bartek et al., 2011)
![Page 112: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/112.jpg)
Dependency Length MinimizationShort syntactic dependencies ease processing (Gibson, 1998; Grodner and Gibson, 2005; Demberg and Keller, 2008; Bartek et al., 2011)
Quantitative corpus evidence from many languages confirms that languages have shorter dependencies than would be expected at random (Futrell et al., 2015).
![Page 113: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/113.jpg)
Dependency Length MinimizationShort syntactic dependencies ease processing (Gibson, 1998; Grodner and Gibson, 2005; Demberg and Keller, 2008; Bartek et al., 2011)
Quantitative corpus evidence from many languages confirms that languages have shorter dependencies than would be expected at random (Futrell et al., 2015).
Argued to explain several of the Greenberg correlations (Rijkhoff, 1986; Hawkins, 1994, 2003)
![Page 114: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/114.jpg)
21 1
![Page 115: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/115.jpg)
Two Objectives for Optimization
Dependency Length Minimization
Communicative Utility
![Page 116: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/116.jpg)
Two Objectives for Optimization
Communicative Utility
Utility Informativity Cost-=
Amount of Meaning that can be extracted from utterance
Cost of processing utterance
λ
Long tradition as an explanation of language (Gabelentz 1903, Zipf 1949, Horn 1984, …)
![Page 117: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/117.jpg)
Two Objectives for Optimization
Communicative Utility
Utility Informativity Cost-= λ
![Page 118: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/118.jpg)
Two Objectives for OptimizationCommunicative Utility
Utility Informativity Cost-= λ
Mary has two green books.
![Page 119: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/119.jpg)
Two Objectives for OptimizationCommunicative Utility
Utility Informativity Cost-= λ
Mary has two green books.
Informativity(utterance) := log P(tree | utterance) - log P(tree)
![Page 120: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/120.jpg)
Two Objectives for OptimizationCommunicative Utility
Utility Informativity Cost-= λ
Mary has two green books.
Informativity(utterance) := log P(tree | utterance) - log P(tree)We use a neural network model (Dozat and Manning 2017) with extremely generic architecture.
![Page 121: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/121.jpg)
Two Objectives for OptimizationCommunicative Utility
Utility Informativity Cost-= λ
![Page 122: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/122.jpg)
Two Objectives for OptimizationCommunicative Utility
Utility Informativity Cost-= λ
Surprisal(wi|w1...wi-1) = -log P(wi|w1...wi-1)
![Page 123: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/123.jpg)
Two Objectives for OptimizationCommunicative Utility
Utility Informativity Cost-= λ
Surprisal(wi|w1...wi-1) = -log P(wi|w1...wi-1)
We use recurrent neural networks, the SOTA in probabilistic modelling of natural language and predicting reading times.
![Page 124: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/124.jpg)
Dependency Length Surprisal
Parsability
2.35.81.8
(1) For each objective, find parameters that optimise it.
(2) Which universals do the resulting counterfactual languages satisfy?
![Page 125: Testing Functional Explanations of Word Order Universalsstanford.edu/~mhahn2/cgi-bin/files/CAMP2018.pdf · 2019. 2. 2. · Dependency Corpus Ordering Grammar NOUN ADJ amod 0.1 NOUN](https://reader036.fdocuments.in/reader036/viewer/2022071103/5fdc80cac2804b232541ae15/html5/thumbnails/125.jpg)
Dependency Length Surprisal
Parsability
2.35.81.8
(1) For each objective, find parameters that optimise it.
(2) Which universals do the resulting counterfactual languages satisfy?