Post on 11-Feb-2022
BackgroundContext Theories
Tools
Language as an Algebra
Daoud Clarke
Department of Computer ScienceUniversity of Hertfordshire
The Categorical Flow of Information in Quantum Physicsand Linguistics, Oxford, 2010
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Overview
1 BackgroundDistributional SemanticsContext-theoretic Semantics
2 Context TheoriesMotivating ExampleExample
3 ToolsQuotient AlgebrasFinite-dimensional AlgebrasAlgebras from Monoids
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Distributional SemanticsContext-theoretic Semantics
Beyond lexical distributional semantics
Distributional semantics:Hypothesis of Harris (1968)LSA, distributional similarity etc.Many applicationsGood for words/short phrases
How can we go beyond the lexical domain?Data sparseness
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Distributional SemanticsContext-theoretic Semantics
Beyond lexical distributional semantics
Distributional semantics:Hypothesis of Harris (1968)LSA, distributional similarity etc.Many applicationsGood for words/short phrases
How can we go beyond the lexical domain?Data sparseness
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Distributional SemanticsContext-theoretic Semantics
Physicists (xkcd.com)
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Distributional SemanticsContext-theoretic Semantics
Physicists (xkcd.com)
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Distributional SemanticsContext-theoretic Semantics
Context-theoretic Semantics
Suggestion: explore the use of unital associative algebrasover the real numbersAn algebra is a vector space together with a bilinearmultiplication:
x · (y + z) = x · y + x · z (x + y) · z = x · z + y · z
Associative algebras form a monoidal category K -Alg
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Distributional SemanticsContext-theoretic Semantics
Context-theoretic Semantics
Suggestion: explore the use of unital associative algebrasover the real numbersAn algebra is a vector space together with a bilinearmultiplication:
x · (y + z) = x · y + x · z (x + y) · z = x · z + y · z
Associative algebras form a monoidal category K -Alg
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Distributional SemanticsContext-theoretic Semantics
Context-theoretic Semantics
Suggestion: explore the use of unital associative algebrasover the real numbersAn algebra is a vector space together with a bilinearmultiplication:
x · (y + z) = x · y + x · z (x + y) · z = x · z + y · z
Associative algebras form a monoidal category K -Alg
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Distributional SemanticsContext-theoretic Semantics
Al-kitab al-mukhtas.ar fı h. isabi-l-jabr wa’l-muqabala
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Distributional SemanticsContext-theoretic Semantics
Algebra over a field
An algebra over a field:abstraction of the space of operators on a vector space
Hilbert space operators→ C∗-algebrasVector lattice operators→ lattice-ordered algebrasMatrices of order n form an algebra under normal matrixmultiplication
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Distributional SemanticsContext-theoretic Semantics
Algebra over a field
An algebra over a field:abstraction of the space of operators on a vector space
Hilbert space operators→ C∗-algebrasVector lattice operators→ lattice-ordered algebrasMatrices of order n form an algebra under normal matrixmultiplication
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Distributional SemanticsContext-theoretic Semantics
Algebra over a field
An algebra over a field:abstraction of the space of operators on a vector space
Hilbert space operators→ C∗-algebrasVector lattice operators→ lattice-ordered algebrasMatrices of order n form an algebra under normal matrixmultiplication
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Distributional SemanticsContext-theoretic Semantics
Algebra over a field
An algebra over a field:abstraction of the space of operators on a vector space
Hilbert space operators→ C∗-algebrasVector lattice operators→ lattice-ordered algebrasMatrices of order n form an algebra under normal matrixmultiplication
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Distributional SemanticsContext-theoretic Semantics
Context-theoretic Framework
A context theory is a tuple 〈A,A, , φ〉The meaning of a string x ∈ A∗ is a vector x ∈ A
A∗ is the free monoid on an alphabet AA is a real associative unital algebra
The mapping ˆ from A∗ to A is a monoid homomorphismthe cat sat = the · cat · sat
Multiplication · is distributive with respect to vector spaceaddition (bilinearity)φ is a linear functional on A indicating probability
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Distributional SemanticsContext-theoretic Semantics
Context-theoretic Framework
A context theory is a tuple 〈A,A, , φ〉The meaning of a string x ∈ A∗ is a vector x ∈ A
A∗ is the free monoid on an alphabet AA is a real associative unital algebra
The mapping ˆ from A∗ to A is a monoid homomorphismthe cat sat = the · cat · sat
Multiplication · is distributive with respect to vector spaceaddition (bilinearity)φ is a linear functional on A indicating probability
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Distributional SemanticsContext-theoretic Semantics
Context-theoretic Framework
A context theory is a tuple 〈A,A, , φ〉The meaning of a string x ∈ A∗ is a vector x ∈ A
A∗ is the free monoid on an alphabet AA is a real associative unital algebra
The mapping ˆ from A∗ to A is a monoid homomorphismthe cat sat = the · cat · sat
Multiplication · is distributive with respect to vector spaceaddition (bilinearity)φ is a linear functional on A indicating probability
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Distributional SemanticsContext-theoretic Semantics
Context-theoretic Framework
A context theory is a tuple 〈A,A, , φ〉The meaning of a string x ∈ A∗ is a vector x ∈ A
A∗ is the free monoid on an alphabet AA is a real associative unital algebra
The mapping ˆ from A∗ to A is a monoid homomorphismthe cat sat = the · cat · sat
Multiplication · is distributive with respect to vector spaceaddition (bilinearity)φ is a linear functional on A indicating probability
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Distributional SemanticsContext-theoretic Semantics
Context-theoretic Framework
A context theory is a tuple 〈A,A, , φ〉The meaning of a string x ∈ A∗ is a vector x ∈ A
A∗ is the free monoid on an alphabet AA is a real associative unital algebra
The mapping ˆ from A∗ to A is a monoid homomorphismthe cat sat = the · cat · sat
Multiplication · is distributive with respect to vector spaceaddition (bilinearity)φ is a linear functional on A indicating probability
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Motivating ExampleExample
Motivating Example: Context Algebras
Meaning as contextFormal language→ syntactic monoidFuzzy language→ context algebraE.g. Latent Dirichlet Allocation→ commutative algebra
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Motivating ExampleExample
Motivating Example: Context Algebras
Meaning as contextFormal language→ syntactic monoidFuzzy language→ context algebraE.g. Latent Dirichlet Allocation→ commutative algebra
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Motivating ExampleExample
Motivating Example: Context Algebras
Meaning as contextFormal language→ syntactic monoidFuzzy language→ context algebraE.g. Latent Dirichlet Allocation→ commutative algebra
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Motivating ExampleExample
Motivating Example: Context Algebras
Meaning as contextFormal language→ syntactic monoidFuzzy language→ context algebraE.g. Latent Dirichlet Allocation→ commutative algebra
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Motivating ExampleExample
Context Algebras: How it works
Fuzzy language C : A∗ → [0,1]
For x ∈ A∗, define x : A∗ × A∗ → [0,1] by
x(y , z) = C(yxz)
Define A as vector space generated by {x : x ∈ A∗}Choose a basis and use multiplication on A∗ to definemultiplication on A
Multiplication is the same, no matter what basis we choose!
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Motivating ExampleExample
Context Algebras: How it works
Fuzzy language C : A∗ → [0,1]
For x ∈ A∗, define x : A∗ × A∗ → [0,1] by
x(y , z) = C(yxz)
Define A as vector space generated by {x : x ∈ A∗}Choose a basis and use multiplication on A∗ to definemultiplication on A
Multiplication is the same, no matter what basis we choose!
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Motivating ExampleExample
Context Algebras: How it works
Fuzzy language C : A∗ → [0,1]
For x ∈ A∗, define x : A∗ × A∗ → [0,1] by
x(y , z) = C(yxz)
Define A as vector space generated by {x : x ∈ A∗}Choose a basis and use multiplication on A∗ to definemultiplication on A
Multiplication is the same, no matter what basis we choose!
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Motivating ExampleExample
Context Algebras: How it works
Fuzzy language C : A∗ → [0,1]
For x ∈ A∗, define x : A∗ × A∗ → [0,1] by
x(y , z) = C(yxz)
Define A as vector space generated by {x : x ∈ A∗}Choose a basis and use multiplication on A∗ to definemultiplication on A
Multiplication is the same, no matter what basis we choose!
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Motivating ExampleExample
Context Algebras: Example
Corpus defined by
C(the cat sat) = 0.8C(the big cat sat) = 0.2
Then
cat = 0.8e(the, sat) + 0.2e(the big, sat)
big · cat = big cat = 0.2e(the, sat)
where e(x , y) is the basis element corresponding to(x , y) ∈ A∗ × A∗.
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Motivating ExampleExample
Context Algebras: Example
Corpus defined by
C(the cat sat) = 0.8C(the big cat sat) = 0.2
Then
cat = 0.8e(the, sat) + 0.2e(the big, sat)
big · cat = big cat = 0.2e(the, sat)
where e(x , y) is the basis element corresponding to(x , y) ∈ A∗ × A∗.
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Motivating ExampleExample
Context-theoretic Semantics: Summary
Map strings to elements of an algebraMotivating example:
Meaning as contextFuzzy language→ context algebra
Lots of other examples in Clarke (2007)
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Quotient AlgebrasFinite-dimensional AlgebrasAlgebras from Monoids
Tools for Constructing Algebras
Quotient Algebras (Clarke, Lutz and Weir 2010)Finite dimensional algebras
with David Weir, Rudi Lutz and Ben Campion
Enveloping Algebras (You?)
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Quotient AlgebrasFinite-dimensional AlgebrasAlgebras from Monoids
Quotient Algebras
Construct a free algebra (tensor algebra)
T (V ) = R⊕ V ⊕ (V ⊗ V )⊕ (V ⊗ V ⊗ V )⊕ · · ·
Choose relations u1 = v1,u2 = v2, . . . we wish to hold
Λ = {u1 − v1,u2 − v2, . . .}
Construct ideal I generated by Λ
Take equivalence classes to get quotient algebra
T (V )/I
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Quotient AlgebrasFinite-dimensional AlgebrasAlgebras from Monoids
Quotient Algebras
Construct a free algebra (tensor algebra)
T (V ) = R⊕ V ⊕ (V ⊗ V )⊕ (V ⊗ V ⊗ V )⊕ · · ·
Choose relations u1 = v1,u2 = v2, . . . we wish to hold
Λ = {u1 − v1,u2 − v2, . . .}
Construct ideal I generated by Λ
Take equivalence classes to get quotient algebra
T (V )/I
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Quotient AlgebrasFinite-dimensional AlgebrasAlgebras from Monoids
Quotient Algebras
Construct a free algebra (tensor algebra)
T (V ) = R⊕ V ⊕ (V ⊗ V )⊕ (V ⊗ V ⊗ V )⊕ · · ·
Choose relations u1 = v1,u2 = v2, . . . we wish to hold
Λ = {u1 − v1,u2 − v2, . . .}
Construct ideal I generated by Λ
Take equivalence classes to get quotient algebra
T (V )/I
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Quotient AlgebrasFinite-dimensional AlgebrasAlgebras from Monoids
Quotient Algebras
Construct a free algebra (tensor algebra)
T (V ) = R⊕ V ⊕ (V ⊗ V )⊕ (V ⊗ V ⊗ V )⊕ · · ·
Choose relations u1 = v1,u2 = v2, . . . we wish to hold
Λ = {u1 − v1,u2 − v2, . . .}
Construct ideal I generated by Λ
Take equivalence classes to get quotient algebra
T (V )/I
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Quotient AlgebrasFinite-dimensional AlgebrasAlgebras from Monoids
Quotient Algebra: Why?
Vectors that were orthogonal in T (V ) can benon-orthogonal in T (V )/IStrings of different lengths can be compared
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Quotient AlgebrasFinite-dimensional AlgebrasAlgebras from Monoids
Quotient Algebra: How?
An ideal I of an algebra A is a sub-vector space of A suchthat xa ∈ I and ax ∈ I for all a ∈ A and all x ∈ ICongruence: x ≡ y if x − y ∈ IQuotient algebra A/I formed from equivalence classes
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Quotient AlgebrasFinite-dimensional AlgebrasAlgebras from Monoids
Data-driven Composition
Use a treebankFor each rule π : X → X1 . . .Xr with head Xh we addvectors
λπ,i = ei − X1 ⊗ . . .⊗ Xh−1 ⊗ ei ⊗ Xh+1 ⊗ . . .⊗ Xr
for each basis element ei of VXh to the generating set.X is the sum over all individual vectors of subtrees rootedwith X
Preserve meaning of head
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Quotient AlgebrasFinite-dimensional AlgebrasAlgebras from Monoids
Example
Example Corpus
see big citymodernise citysee modern citysee red applebuy applevisit big appleread big bookthrow old small red bookbuy large new book
Grammar:
N′ → Adj N′
N′ → N
Generating set Λ:
λi = ei − Adj⊗ ei
Adj = 2eapple +6ebook +2ecity
where ei ranges over basisvectors for noun contexts.
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Quotient AlgebrasFinite-dimensional AlgebrasAlgebras from Monoids
Example: Cosine Similarities
appl
e
big
appl
e
red
appl
e
city
big
city
red
city
book
big
book
red
book
apple 1.0 0.26 0.24 0.52 0.13 0.12 0.33 0.086 0.080big apple 1.0 0.33 0.13 0.52 0.17 0.086 0.33 0.11red apple 1.0 0.12 0.17 0.52 0.080 0.11 0.33city 1.0 0.26 0.24 0.0 0.0 0.0big city 1.0 0.33 0.0 0.0 0.0red city 1.0 0.0 0.0 0.0book 1.0 0.26 0.24big book 1.0 0.33red book 1.0
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Quotient AlgebrasFinite-dimensional AlgebrasAlgebras from Monoids
Example: Cosine Similarities
appl
e
big
appl
e
red
appl
e
city
big
city
red
city
book
big
book
red
book
apple 1.0 0.26 0.24 0.52 0.13 0.12 0.33 0.086 0.080big apple 1.0 0.33 0.13 0.52 0.17 0.086 0.33 0.11red apple 1.0 0.12 0.17 0.52 0.080 0.11 0.33city 1.0 0.26 0.24 0.0 0.0 0.0big city 1.0 0.33 0.0 0.0 0.0red city 1.0 0.0 0.0 0.0book 1.0 0.26 0.24big book 1.0 0.33red book 1.0
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Quotient AlgebrasFinite-dimensional AlgebrasAlgebras from Monoids
Example: Cosine Similarities
appl
e
big
appl
e
red
appl
e
city
big
city
red
city
book
big
book
red
book
apple 1.0 0.26 0.24 0.52 0.13 0.12 0.33 0.086 0.080big apple 1.0 0.33 0.13 0.52 0.17 0.086 0.33 0.11red apple 1.0 0.12 0.17 0.52 0.080 0.11 0.33city 1.0 0.26 0.24 0.0 0.0 0.0big city 1.0 0.33 0.0 0.0 0.0red city 1.0 0.0 0.0 0.0book 1.0 0.26 0.24big book 1.0 0.33red book 1.0
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Quotient AlgebrasFinite-dimensional AlgebrasAlgebras from Monoids
Example: Cosine Similarities
appl
e
big
appl
e
red
appl
e
city
big
city
red
city
book
big
book
red
book
apple 1.0 0.26 0.24 0.52 0.13 0.12 0.33 0.086 0.080big apple 1.0 0.33 0.13 0.52 0.17 0.086 0.33 0.11red apple 1.0 0.12 0.17 0.52 0.080 0.11 0.33city 1.0 0.26 0.24 0.0 0.0 0.0big city 1.0 0.33 0.0 0.0 0.0red city 1.0 0.0 0.0 0.0book 1.0 0.26 0.24big book 1.0 0.33red book 1.0
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Quotient AlgebrasFinite-dimensional AlgebrasAlgebras from Monoids
Finite-dimensional Algebras
Fix dimensionality n of the vector spaceLearn vectors for words and pairs of words using e.g. LSAFind the bilinear product on the vector space which bestfits these vectors
Least squaresLinear optimisation
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Quotient AlgebrasFinite-dimensional AlgebrasAlgebras from Monoids
Finite-dimensional Algebras: Results
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Quotient AlgebrasFinite-dimensional AlgebrasAlgebras from Monoids
Algebras from Monoids
Montague semantics with lambda calculus?Cartesian closed categories?Curry-Howard-Lambek correspondence?Enveloping C∗ algebras?
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Quotient AlgebrasFinite-dimensional AlgebrasAlgebras from Monoids
Algebras from Monoids: the Idea
Given any monoid S, we can construct an algebraPut complex syntactic and semantic information in SThen “vectorize” it using a standard constructionRepresent words as weighted sums of elements of S
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Quotient AlgebrasFinite-dimensional AlgebrasAlgebras from Monoids
From Syntax and Semantics to Monoids
Let S be the set of all pairs (s, σ)
s is a syntactic type (e.g. in Lambek calculus)σ is semantics (e.g. a combination of lambda calculus andfirst order logic)
Multiplication defined by reduction to normal formLambek calculus ∼ residuated latticeLambda calculus is a Cartesian closed category underβη-equivalence (Curry-Howard-Lambek correspondence)
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Quotient AlgebrasFinite-dimensional AlgebrasAlgebras from Monoids
From Syntax and Semantics to Monoids
Let S be the set of all pairs (s, σ)
s is a syntactic type (e.g. in Lambek calculus)σ is semantics (e.g. a combination of lambda calculus andfirst order logic)
Multiplication defined by reduction to normal formLambek calculus ∼ residuated latticeLambda calculus is a Cartesian closed category underβη-equivalence (Curry-Howard-Lambek correspondence)
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Quotient AlgebrasFinite-dimensional AlgebrasAlgebras from Monoids
From Monoids to Algebras
Define multiplication on L1(S) by convolution:
(u · v)(x) =∑
y ,z∈S:yz=x
u(y)v(z)
We want lattice properties of S to carry overC∗ enveloping algebra?Need an involution on SUse right adjoint of Cartesian closed category?
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Quotient AlgebrasFinite-dimensional AlgebrasAlgebras from Monoids
From Monoids to Algebras
Define multiplication on L1(S) by convolution:
(u · v)(x) =∑
y ,z∈S:yz=x
u(y)v(z)
We want lattice properties of S to carry overC∗ enveloping algebra?Need an involution on SUse right adjoint of Cartesian closed category?
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Quotient AlgebrasFinite-dimensional AlgebrasAlgebras from Monoids
Monoid to Algebra Example
Term Context vectorfish (0,0,1)big (1,2,0)
ni = (N, λx nouni(x))
ai = (N/N, λpλy adji(y) ∧ p.y)
Define big = a1 + 2a2 and fish = n3,
Then big · fish = a1n3 + 2a2n3, where
ainj = (N, λx(nounj(x) ∧ adji(x))).
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Quotient AlgebrasFinite-dimensional AlgebrasAlgebras from Monoids
Monoid to Algebra Example
Term Context vectorfish (0,0,1)big (1,2,0)
ni = (N, λx nouni(x))
ai = (N/N, λpλy adji(y) ∧ p.y)
Define big = a1 + 2a2 and fish = n3,
Then big · fish = a1n3 + 2a2n3, where
ainj = (N, λx(nounj(x) ∧ adji(x))).
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Quotient AlgebrasFinite-dimensional AlgebrasAlgebras from Monoids
Monoid to Algebra Example
Term Context vectorfish (0,0,1)big (1,2,0)
ni = (N, λx nouni(x))
ai = (N/N, λpλy adji(y) ∧ p.y)
Define big = a1 + 2a2 and fish = n3,
Then big · fish = a1n3 + 2a2n3, where
ainj = (N, λx(nounj(x) ∧ adji(x))).
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Quotient AlgebrasFinite-dimensional AlgebrasAlgebras from Monoids
Conclusion
Hypothesis: meanings live in a unital associative realalgebraThree ways to construct such algebras:
Quotient algebras — apply relations between vectorsSearch finite-dimensional algebrasWrap up a monoid in an algebra
Daoud Clarke Language as an Algebra
BackgroundContext Theories
Tools
Quotient AlgebrasFinite-dimensional AlgebrasAlgebras from Monoids
Conclusion
Hypothesis: meanings live in a unital associative realalgebraThree ways to construct such algebras:
Quotient algebras — apply relations between vectorsSearch finite-dimensional algebrasWrap up a monoid in an algebra
Daoud Clarke Language as an Algebra