Post on 02-Jan-2016
description
Kevin H. KnuthDepartment of Physics
University at Albany
An Order-Theoretic Approach toward
Entropy
26 Oct 2007 Kevin H KnuthFacets of Entropy
In the Beginning…
26 Oct 2007 Kevin H KnuthFacets of Entropy
Lifting Rocks
This caveman finds it easy to order these rocks in terms of how heavy they are to lift.
26 Oct 2007 Kevin H KnuthFacets of Entropy
Diagram Representing the OrderingH
eavi
er
He uses a binary weight comparison to order his rocks
This configuration is called a chain
26 Oct 2007 Kevin H KnuthFacets of Entropy
IsomorphismsIs
Les
s R
ipe
than
1
2
3
4
5
Is L
ess
than
or
Equ
al t
o
1
2
4
8
16
Div
ides
26 Oct 2007 Kevin H KnuthFacets of Entropy
Antichains
These elements are incomparable under our binary ordering relation
They form a poset called an ANTICHAIN
26 Oct 2007 Kevin H KnuthFacets of Entropy
Partitioning
A set of elements together with a binary ordering relation based onthe notion of containing
26 Oct 2007 Kevin H KnuthFacets of Entropy
Two Posets with Integers
1
2
3
4
Is L
ess
than
or
Equ
al t
o
1
2 3
4D
ivid
es
5
6
7
8
9
26 Oct 2007 Kevin H KnuthFacets of Entropy
The Powerset of {a, b, c}
}{b
}{a }{c
},{ ca},{ ba },{ cb
},,{ cba
,},,{},,{},,{},,{},{},{},{, cbacbcabacbaP
Is a subset of
26 Oct 2007 Kevin H KnuthFacets of Entropy
Posets
A partially ordered set (poset) is a set of elements together with a binary ordering relation .
}{},,{ acba }{},{ aba
1312
includes
covers
26 Oct 2007 Kevin H KnuthFacets of Entropy
Lattices
A lattice is a poset P where every pair of elements x and y has
a least upper bound called the join
a greatest lower bound called the meet
yx
}{b
}{a }{c
},{ ca},{ ba },{ cb
},,{ cba },{}{}{ cbcb
}{},{},{ bcbba Similarly
yx
The greenelements areupper boundsof the blue circledpair. The green circled element is theirleast upper bound ortheir join.
26 Oct 2007 Kevin H KnuthFacets of Entropy
The Lattice Identities
L1. Idempotent
L2. Commutative
L3. Associative
L4. Absorption
If the meet and join follow the Consistency Relations
C1. (x is the greatest lower bound of x and y)
C2 . (y is the least upper bound of x and y)
Lattice Identities
xxxxxx ,
xyyxxyyx ,
zyxzyxzyxzyx )()(,)()(
xyxxyxx )()(
xyx
yyx
yx
26 Oct 2007 Kevin H KnuthFacets of Entropy
Lattices are Algebras
aba
bbaba
StructuralViewpoint
OperationalViewpoint
26 Oct 2007 Kevin H KnuthFacets of Entropy
Lattices are Algebras
aba
bbaba
StructuralViewpoint
OperationalViewpoint
aba
bbaba
Assertions, Implies
aba
bbaba
Sets, Is a subset of
aba
bbaba
),gcd(
),lcm(|
Positive Integers, Divides
aba
bbaba
),min(
),max(Integers, Is less than or equal to
Hypothesis Space(Our States of Knowledge)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Describing a State of Knowledge
These statements describe the fruit:
a = ‘It is an Apple!’
b = ‘It is a Banana!’
c = ‘It is an Citrus Fruit!’
They can also describe what one knows about the fruit.
26 Oct 2007 Kevin H KnuthFacets of Entropy
The Hypothesis Space
a b c
The atoms are mutually exclusive and exhaustive logical statements
a = ‘It is an apple!’
b = ‘It is a banana!’
c = ‘It is a citrus fruit!’
26 Oct 2007 Kevin H KnuthFacets of Entropy
The Boolean Hypothesis Space
a b c
The meet of any two atoms is the absurdity: a b =
We do not allow our state of knowledge to include:
‘The fruit is an apple AND a banana!’
26 Oct 2007 Kevin H KnuthFacets of Entropy
The Boolean Hypothesis Space
a b c
26 Oct 2007 Kevin H KnuthFacets of Entropy
The Boolean Hypothesis Space
a b c
a b
The join of any two elements represents a logical OR: a b
26 Oct 2007 Kevin H KnuthFacets of Entropy
The Boolean Hypothesis Space
a b c
a b a c b c
The join of any two elements represents a logical OR: a b
26 Oct 2007 Kevin H KnuthFacets of Entropy
The Boolean Hypothesis Space
a b c
a b a c b c
The final join gives us the TOP element, often called the TRUISM
= a b c“It is an Apple or a Banana or an Orange!”
26 Oct 2007 Kevin H KnuthFacets of Entropy
The Hypothesis Space
This is a HYPOTHESIS SPACE!!!
It consists of all the statements that can be constructed from a set of mutually exclusive exhaustive statements.
The space is ordered by the ordering relation “implies”
We allow concepts like:‘The fruit is an apple OR a banana!’
while we disallow concepts like:‘The fruit is an apple AND a banana!’
a b c
a b a c b c
26 Oct 2007 Kevin H KnuthFacets of Entropy
Superpositions of States???
This was your initial state of knowledge.
The fruit was never in this state!
This was the state of the fruit.
26 Oct 2007 Kevin H KnuthFacets of Entropy
Two Spaces
Space of statements
describing the fruitSpace of statements describing
a state of knowledge about the fruit
Generalizing Partial Orders to Measures
26 Oct 2007 Kevin H KnuthFacets of Entropy
Inclusion and the Zeta Function
yxif
yxifyx
0
1),(
The Zeta function encodes inclusion on the lattice.
a b c
a b a c b c
26 Oct 2007 Kevin H KnuthFacets of Entropy
Inclusion and the Zeta Function
yxif
yxifyx
0
1),(
a b c
a b a c b c
1),( baa
0),( abasince
since
aba baa
The Zeta function encodes inclusion on the lattice.
26 Oct 2007 Kevin H KnuthFacets of Entropy
The Zeta Function
yxif
yxifyx
0
1),(
a b c
a b a c b c
a b c avb avc bvc T
1 1 1 1 1 1 1 1
a 0 1 0 0 1 1 0 1
b 0 0 1 0 1 0 1 1
c 0 0 0 1 0 1 1 1
avb 0 0 0 0 1 0 0 1
avc 0 0 0 0 0 1 0 1
bvc 0 0 0 0 0 0 1 1
T 0 0 0 0 0 0 0 1
26 Oct 2007 Kevin H KnuthFacets of Entropy
Inclusion and the Zeta Function
yxif
yxifyx
0
1),(
The Zeta function encodes inclusion on the lattice.
We can define its dual by flipping around the ordering relation
yxif
yxifyx
0
1),(
26 Oct 2007 Kevin H KnuthFacets of Entropy
Degrees of Inclusion and Z
yxif
yxifyx
0
1),(
We generalize the dual of the Zeta function
yxif
yxifz
yxif
yxz
0
1
),(
to the function z
26 Oct 2007 Kevin H KnuthFacets of Entropy
Z
yxif
yxifz
yxif
yxz
0
1
),(
The function z
Continues to encode inclusion, but has generalized the concept to degrees of inclusion.
In the lattice of logical statements ordered by implies, this function describes degrees of implication.
26 Oct 2007 Kevin H KnuthFacets of Entropy
How do we Assign Values to z?
yxif
yxifz
yxif
yxz
0
1
),(
Are all of the values of the function z arbitrary?
Or are there constraints?
Here there be monsters…
a b c avb avc bvc T
1 0 0 0 0 0 0 0
a 1 1 0 0 ? ? 0 ?
b 1 0 1 0 ? 0 ? ?
c 1 0 0 1 0 ? ? ?
avb 1 1 1 0 1 ? ? ?
avc 1 1 0 1 ? 1 ? ?
bvc 1 0 1 1 ? ? 1 ?
T 1 1 1 1 1 1 1 1
26 Oct 2007 Kevin H KnuthFacets of Entropy
Lattice Structure Imposes Constraints
I showed that in “general”:
Associativity leads to a Sum Rule…
Distributivity leads to a Product Rule…
Commutivity leads to Bayes Theorem…
zyxzyx )()(
)()()( zxyxzyx
xyyx
),(),(),(),( wyxzwyzwxzwyxz
),(),(),( wxyzwxzwyxz
),(
),(),(),(
wyz
wxyzwxzwyxz
26 Oct 2007 Kevin H KnuthFacets of Entropy
Inclusion-Exclusion (The Sum Rule)
),(),(),(),( wyxzwyzwxzwyxz The Sum Rule for Lattices
26 Oct 2007 Kevin H KnuthFacets of Entropy
Inclusion-Exclusion (The Sum Rule)
)|()|()|()|( iyxpiypixpiyxp
The Sum Rule for Probability
),(),(),(),( wyxzwyzwxzwyxz
26 Oct 2007 Kevin H KnuthFacets of Entropy
Inclusion-Exclusion (The Sum Rule)
),()()();( YXHYHXHYXI
Definition of Mutual Information
),(),(),(),( wyxzwyzwxzwyxz
)|()|()|()|( iyxpiypixpiyxp
26 Oct 2007 Kevin H KnuthFacets of Entropy
Inclusion-Exclusion (The Sum Rule)
),min(),max( yxyxyx
Polya’s Min-Max Rule for Integers
),(),(),(),( wyxzwyzwxzwyxz
)|()|()|()|( iyxpiypixpiyxp
),()()();( YXHYHXHYXI
26 Oct 2007 Kevin H KnuthFacets of Entropy
Inclusion-Exclusion (The Sum Rule)
This is intimately related to the Möbius function for the lattice, which is related to the Zeta function.
),(),(),(),( wyxzwyzwxzwyxz
)|()|()|()|( iyxpiypixpiyxp
),()()();( YXHYHXHYXI
),min(),max( yxyxyx
26 Oct 2007 Kevin H KnuthFacets of Entropy
Probability
yxif
xyifp
xyif
yxp
0
1
)|(
Changing notation
The MEANING of p(x|y) is made explicit via the Zeta function.
These are degrees of implication!
NOT plausibility!NOT degrees of belief!NOT frequencies of occurrences!
yxif
yxifz
yxif
yxz
0
1
),(
26 Oct 2007 Kevin H KnuthFacets of Entropy
Statement – QuestionDuality
26 Oct 2007 Kevin H KnuthFacets of Entropy
Richard T. Cox (1979) defined a question as the set of all possible assertions that answer it. I recast his definition to obtain new insights.
In lattice theory, such a set is called adown-set
In the figure the colored elementsbelong to
I call the set of top elements the irreducible set.
Defining a Question
cba
ba ca cb
ba c
xX
},{
)()(
cbba
cbbaBCAB
26 Oct 2007 Kevin H KnuthFacets of Entropy
When are Questions Equal?
Two questions are equivalent when they are defined by the same set of assertions.
Consider the questions
“Is it raining?”
“Is it not raining?”
They are both answered by the set of assertions generatedby the irreducible set { “It is raining!”, “It is not raining!”} They are therefore equivalent.
26 Oct 2007 Kevin H KnuthFacets of Entropy
Animal, Vegetable, Mineral
Space of statementsdescribing the object
Space of Statementsdescribing a state of knowledge about
the object
a v m
26 Oct 2007 Kevin H KnuthFacets of Entropy
The Central Issue
I = “Is it an Animal, a Vegetable, or a Mineral?”
This question is answered by the following set of statements:
I = { a = “It is an animal!”, v = “It is a vegetable!”, m = “It is a mineral!” }
},,,{ mvaI
26 Oct 2007 Kevin H KnuthFacets of Entropy
Some Questions Answer Others
Now consider the binary question
B = “Is it an animal?”
B = {a = “It is an animal!”, ~a = “It is not an animal!”}
As the defining set of I is exhaustive, mva ~
},,,,{ mvmvaB
26 Oct 2007 Kevin H KnuthFacets of Entropy
Ordering Questions
B = “Is it an animal?”
I = “Is it an Animal, a Vegetable, or a Mineral?”
BI I answers B
B includes I
},,,{ mvaI
},,,,{ mvmvaB
26 Oct 2007 Kevin H KnuthFacets of Entropy
Meets and Joins of Questions
With “is a subset of” as the ordering relation among questions,one can show that:
The meet of two questions is the set intersection of the set of assertions answering the question.
The join of two questions is the set union of the set of assertions answering the question.
YXYX
YXYX
26 Oct 2007 Kevin H KnuthFacets of Entropy
Ideals
An ideal is a nonvoid subset J of a lattice A with the properties (Birkhoff 1967)I1. , where thenI2. , then
I1 is the condition for the set J to be a down-set, or equivalently a question.
I2 assures that there is a unique maximum
Therefore, as ideals are questions, I call them Ideal Questions.
JxJx Jz
Ay xy Jzx
Jy
26 Oct 2007 Kevin H KnuthFacets of Entropy
Ideals and Ideal Questions
26 Oct 2007 Kevin H KnuthFacets of Entropy
There are More Questions!
Questions can be formed from set unions of ideal questions.
which can be written
These questions along with the ideal questions form alattice ordered by set inclusion.
)()( mvaMAV
},,,,{},{},,,{ mvavamvava
26 Oct 2007 Kevin H KnuthFacets of Entropy
The Lattice of Questions
The lattice of questions is the Free Distributive Lattice, Q(N) = FD(N)
No complements!
26 Oct 2007 Kevin H KnuthFacets of Entropy
Real Questions
Real Questions are questions thatcan always be answered by a true assertion.
Real questions must include the exhaustive set of mutually exclusive assertions.
The Real Question lattice R(N)looks almost Boolean for N=3,but this is not true in general.
26 Oct 2007 Kevin H KnuthFacets of Entropy
Real Questions
In Q(3) these are the questions R where
The bottom question is called the Central Issue and as such it results in an unambiguous answer.
RMVA
R(3)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Partition Questions
A partition question P neatly partitions the possible answers.These are the join-irreducible elements of the lattice R(N).
partition latticeR(3)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Valuations on the Question Lattice
26 Oct 2007 Kevin H KnuthFacets of Entropy
Inquiry Calculus
Generalizing the zeta function for the question lattice gives a valuation that describes the degree to which the question Q answers the issue I
I call this quantity the relevance
The relevance follows the Sum Rule
otherwised
QIif
QIif
QId
10
0
1
)|(
),()|( QIzQId
26 Oct 2007 Kevin H KnuthFacets of Entropy
Consistency Principle
Probability assignments in A must be consistent with relevance assignments in Q
26 Oct 2007 Kevin H KnuthFacets of Entropy
Ideal Questions again
Only one assumption:
))|(,),|(),|(()|( 21 TTT nn xpxpxpHPId
The degree to which a partition question resolves the central issue is a function of the probabilities of the statements comprising the partition question’s irreducible set.
The goal is to determine the form of the functions Hn().
Again, the lattice properties constrain the solution!
26 Oct 2007 Kevin H KnuthFacets of Entropy
Consequences of Our Assumption
YXiffYIdXIdYXId )|()|()|(
1. Additivity
(sum rule)
otherwiseYIdXIdYXId )|()|()|(
2. Subadditivity
(sum rule)
)|()|( )()2()1(21 nn XXXIdXXXId
3. Commutativity
(commute)
4. Expansibility
(identifying with bottom)
)|()|( 2121 nn XXXIdFXXXId
26 Oct 2007 Kevin H KnuthFacets of Entropy
Entropy
),,,(),,,(),,,( 212121 nonnn pppHbpppHapppH
n
iiin pppppH
1221 log),,,(
NpppH no 221 log),,,(
Shannon
Hartley
Aczél (Aczél et al., 1974): The function Hm satisfying additivity, subadditivity, commutativity and expansibility is a linear combination of Shannon (1949) and Hartley (1928) entropies
26 Oct 2007 Kevin H KnuthFacets of Entropy
Normalization Conditions
)(
log1 2
IH
Nba
Relevance is maximized when
1)|( IId
This implies that
Relevance is minimized when which gives0)|( TId
I
T
)()(log
)()1()(
12
1
IHNPHN
IHNPH
bP
ii
P
ii
26 Oct 2007 Kevin H KnuthFacets of Entropy
Results
a
a
a a
aa
v v v
v v v
m m m
m m m
AVM VAM MAV
AVAM AVVM AMVM
26 Oct 2007 Kevin H KnuthFacets of Entropy
Relevanceand
Information Theory
26 Oct 2007 Kevin H KnuthFacets of Entropy
Relevance and Entropy
)|( QId
mmvvaamva pppppppppHIH 222 logloglog),,()(
),( mva ppH
26 Oct 2007 Kevin H KnuthFacets of Entropy
Higher-Order Informations
))()(|()|()|()|( VMAAMVIdVMAIdAMVIdVMAMId
);(~)|( VMAAMVIVMAMId
This relevance is related to the mutual information.
In this way one can obtain higher-order informations.
26 Oct 2007 Kevin H KnuthFacets of Entropy
Questions and Geometry
26 Oct 2007 Kevin H KnuthFacets of Entropy
Join-Irreducible Elements
a
a
v
a
v
m
The join-irreducible elements are the simplexes.With the bottom, they form a Boolean Lattice.
26 Oct 2007 Kevin H KnuthFacets of Entropy
Questions ~ Simplicial Complexes
Is it or Is it Not an Animal?
a m
v
mv
26 Oct 2007 Kevin H KnuthFacets of Entropy
Lattice of Simplicial Complices
The lattice of questions is isomorphic to the lattice ofsimplicial complices.
These are alsoisomorphic to the lattice of hypergraphs.
26 Oct 2007 Kevin H KnuthFacets of Entropy
Summary Inclusion can be generalized to degrees of inclusion
The lattice structure places constraints on the values of these degrees of inclusion.
On a Distributive Lattice, these constraints take the form of a Sum Rule, a Product Rule and a Bayes Theorem.
The logical statements ordered by implication gives rise to degrees of implication that follow the rules of probability theory
The lattice of questions is dual to the lattice of assertions in the sense of Birkhoff’s Representation Theorem.
The relevance of questions depends on the Shannon and Hartley entropies.
Special Thanks to:
John SkillingAriel CatichaJanos AczélPhilip GoyalSteve GullJeffrey JewellCarlos Rodriguez
for many insightful discussions and comments
Supported in part by:NASA AIST-QRS-07-0001NASA 05-AISR05-0143