The Combinatorial Basis of Entropy...

The Combinatorial Basis of Entropy (“MaxProb”)

22nd Canberra International Physics Summer School ANU, Canberra

11 December 2008

by Robert K. Niven

Marie Curie Incoming International Fellow, 2007-2008 Niels Bohr Institute, University of Copenhagen, Denmark

School of Aerospace, Civil and Mechanical Engineering

The University of New South Wales at ADFA Canberra, ACT, Australia

R.K. Niven, UNSW 22nd Canberra International Physics Summer School 2

Lectures

1. The Combinatorial Basis of Entropy (“MaxProb”)

2. Jaynes’ MaxEnt, Riemannian Metrics and the Principle of Least Action


Contents • Historical overview

- combinatorics - probability theory

• Combinatorial basis of entropy / MaxProb principle

generalised combinatorial definitions of entropy and cross-entropy

explanation of MaxEnt / MinXEnt

• Applications 1. Multinomial systems (asymptotic vs non-asymptotic) 2. (In)distinguishable particles or categories 3. “Neither independent nor identically distributed” sampling

• Future applications …


(Advertisement)

Courses at UNSW@ADFA, Canberra:

• Short course in “Maximum Entropy Analysis”, 14-15 May 2009

(fee paying $1270).

• Masters course: ZACM8327 Maximum Entropy Analysis, semester 2,

2009 (fee paying or UNSW@ADFA enrolled student)

- 3 hours of lectures + tutorials per week

- based on similar course at Niels Bohr Institute


Historical Overview


Combinatorics Knowledge is very old! (Edwards, 2002)

(a) Number-patterns

Pythagoras (500BC)

Egypt (300BC)

Theon of Smyrna, Nicomachus (100AD)

Higher dimensions: Tartaglia (1523, publ. 1556)

figurate numbers fk


(b) Binomial coefficients

= coefficients of (a + b)N

Al-Karaji (1007); Al-Samawal (1180); Al-Kashi (1429)

Chia Hsien (1100); Yang Hui (1261); Chu Shih-chieh (1303)

Cardano (1570), etc

- applied to solution of equations; finding roots; etc

Chu Shih-chieh (1303)


binomial coefficients

N

k= f

k , where =dimension


(c) Combinations + Permutations

Ancient; e.g.

- Susruta (600BC), Jains (300BC): combinations of 6 tastes

- Pingala (200BC): combinations of syllables

No. of permutations of N things = N !

- Hebrew Book of Creation (700); Bhaskara (1150)

No. of groups of N things, taken k at a time: - Mahavira (850); Bhaskara (1150); ben Gerson (1321)

Without replacement With replacement

Combinations

CkN

=N

k=

N!

k !(N k)!

wC

kN

=N + k 1

k=

(N + k 1)!

k !(N 1)!

Permutations PkN

= N(k) =N!

(N k)!

wPkN

= Nk

with

N

kk=0

N= 2

N


Pascal (1654) - equivalence of figurate numbers AND binomial coefficients AND

numbers of combinations without replacement

Multinomial weight - Bhaskara (1150); Mersenne (1636)

= no. of permutations of N objects, containing ni of each category

i = 1,...,s , is:

W =N !

n1!n

2! ... n

s!= N !

1

ni!

i=1

s


Probability Theory (a) Classical period (e.g. Cardano (1560s), Pascal, Fermat, Huygens, the Bernoullis,

Montmort, de Moivre, Laplace)

Probability =

No. of outcomes of interest

Total no. of outcomes

(b) “Frequentist” school (e.g. Venn, Pearson, Neyman, Fisher, von Mises, Feller)

- probability = measurable frequency, for an infinite number of repetitions of a “random experiment”

- attempt to define probabilities as certainties

- narrow applicability


(c) “Bayesian” or “Plausibility” school (Bayes, Laplace, Jeffreys, Polya, Cox, Jaynes, 1957; 2003)

- probability = “plausibility” = assignment based on what you know

- need not be a measurable frequency - manipulate using sum + product rules (Jaynes, 2003)

- “subjective” = “information-dependent”

- different observers, with different information, can assign different probabilities to the same event

- more useful; encompasses all frequentist situations


Probability Distributions “Measures of Central Tendency”

Continuous parameter x Discrete parameter x p(x) = probability density function (pdf) p(x) = pi = probability mass function


Combinatorial (or Probabilistic) Definition of Entropy


Definitions Entity = a discrete particle, object or agent, or an item in a sequence, which is

separate but not necessarily independent of other entities

Category = possible assignment of an entity

Probabilistic System = a set of entities K assigned to a

set of categories C by a discrete random variable : K C

e.g. physics: particles energy levels

gambling: die throws die sides

communications: signal bits letters of alphabet


Configuration = distinguishable permutation of entities amongst categories e.g.: physics: microstate; information theory: sequence

Realization = aggregated arrangement of entities amongst categories = set of configurations e.g.: physics: macrostate; information theory: type

Commonly define realizations by the no. of entities in each category {n

i}


MaxProb Principle MaxProb Principle (Boltzmann, 1877; Planck, 1901; Vincze, 1974; Grendar & Grendar, 2001)

- “A system can be represented by its most probable realization”

principle for probabilistic inference

- does not depend on asymptotic limits

- does not give certainty

BB GB BG GG

Superset of 2nd Law “A system tends towards its most probable realization”

- not just thermodynamics!


MaxProb Principle MaxProb Principle (Boltzmann, 1877; Planck, 1901; Vincze, 1974; Grendar & Grendar, 2001)

- “A system can be represented by its most probable realization”

principle for probabilistic inference

- does not depend on asymptotic limits

- does not give certainty

BB GB BG GG

Superset of 2nd Law “A system tends towards its most probable realization”

- not just thermodynamics!

Ergodicity

Inference


Multinomial Systems (Boltzmann, 1877)

N distinguishable balls (entities)

s distinguishable boxes (categories)

qi = source (“prior”) probability of ball falling in ith box

= normalised degeneracy gi / gii=1

s

Probability of a given realization {n

i} is given by the multinomial

distribution:

Pmult = N ! qi

ni

ni !i=1

s

qi =1/s

Pmult

=W

mult

sN

; Wmult

= N !1

ni!

i=1

s

If categories equiprobable, can use multinomial weight


MaxProb: want to maximise:

Easier to maximise:

lnPmult = lnN !+ ni lnqii=1

s

lnni !

i=1

s

Asymptotic limit for N (rigorously by Sanov (1957) theorem; crudely by Stirling’s approx. lnm! m lnm m ):

DKL = limN

lnPmult

N= pi ln

pi

qii=1

s

where pi =

ni

N.

If qi = 1/s = constant:

hSh= lim

N

lnWmult

N= pi lnpi

i=1

s

Shannon entropy

Kullback - Leibler cross - entropy

= directed divergence

= negative of relative entropy

Pmult = N ! qi

ni

ni !i=1

s


Summary

• Kullback-Leibler and Shannon functions are asymptotic forms of the

multinomial distribution P

mult

• If minimise D

KL (MinXEnt) or maximise

hSh (MaxEnt) of a multinomial

system, subject to constraints

obtain asymptotic MaxProb realization

Boltzmann principle:

Define entropy and cross-entropy by: h=

lnW

N,

D =

lnP

N

(compare S

total= SN = k lnW )

hence always consistent with MaxProb


Jaynes’ MaxEnt • Jaynes (1957)

- minimise D or maximise h , subject to constraints

“most uncertain” distribution = distribution which contains the least

information

• BUT how do we define uncertainty?

- Jaynes only considers D

KL or

hSh axiomatic basis of Shannon

(1948)

• However, a system: - need not be multinomial ! - need not be asymptotic !

Kullback-Leibler or Shannon functions will not give the MaxProb distribution

If know P P

mult or N , must include this information !


Application to Multinomial Systems (Asymptotic + Non-Asymptotic)


Why the Multinomial? (Niven, AIP Conf. Proc. 954 (2007) 133; Blower, pers. comm.)

Pmult = N ! qi

ni

ni !i=1

s

with pi =

ni

N

1. Frequentist approach

P

mult, {qi } = measurable frequencies

2. Bayesian approach

P

mult, {qi } = Bayesian probabilities

If ignorant about choice of model P , then all models equiprobable

must choose multinomial (“central model theorem”)


Asymptotic Analysis Jaynes’ (1957) Algorithm

Minimise

DKL = pi lnpi

qii=1

s

subject to

pii=1

s= 1 and

pii=1

sfri = fr , r = 1,...,R

Form Lagrangian, differentiate w.r.t. pi

pi*= qi e 0

'

r frir=1

R

=1

Zqi e r frir=1

R

Z = e 0

'

= qi e r frir=1

R

i=1

s

Boltzmann

distribution

with 0

'=

0+1

Jaynes’ (1957, 1963, 2003) analysis many more (generic) relations


Isolated Thermodynamic System:

Isolated from rest of universe

e.g. microcanonical ensemble

Natural:

pii=1

s= 1

Mean energy:

pii=1

si = U

pi

*= qi e 0

'

1 i =1

Zqi e 1 i

D *

h*=

0

'+

1U

compare S* = k lnZ +U

T

or F = kT lnZ = TS * + U

Hence 0

'= lnZ = and

1=

1

kT


Non-Asymptotic Analysis (Niven, Phys. Lett. A, 342(4) (2005) 286; Physica A, 365(1) (2006) 142)

Use raw multinomial:

Minimise

D(N)

=lnP

N=

1

NlnN !+ ni lnqi

i=1

s

lnni !

i=1

s

subject to

nii=1

s= N and

n

ii=1

sfri= F

r, r = 1,...,R

Form Lagrangian, differentiate w.r.t. n

i

pi#=

ni#

N=

1

N

1 lnN !

N+ lnqi 0

(N)r(N)fri

r =1

R

1

where ( )= digamma function

Non - asymptotic

distribution


Example 1:

Multinomial

n1,n2,n3

s=3

q =

1

2,3

8, 1

8

subject to

n

i= N

i=1

s

only


Example 2: Multinomial

n1,n2,n3

s=3

q =

1

14, 4

14, 9

14

subject to

n

i= N

i=1

s

n

i i= E

i=1

s

T

with

= 1,2,4[ ]

U =

ET

N=

5

3


Example 2 (cont’d):

For constant

U =

ET

N=

5

3,

obtain

0(N)

=(N)

1(N)

=1

kT(N)


Thermodynamic Double System:

System 1: N

1 particles; energy levels

i

System 2: N

2 particles; energy levels

j

Probs. of realizations {ni }, {nj } are:

Maximise P1P

2 subject to

n

i= N

1i=1

s,

njj=1

m= N

2 and

most probable distrib.:

pi#=

1

N1

1

lnN1!

N1

+ lnqi 0a

(N1)1 i 1

pj#=

1

N2

1 lnN2 !

N2

+ lnqj 0b

(N2)1 j 1

“Zeroth law” upheld

( 1 in common)

P1 = N1! qi

ni

ni !i=1

s

, P2 = N2 ! qj

nj

nj !j=1

m

ET = ni ii=1

s+ nj jj=1

m


Summary

- combinatorial approach straightforward analysis using system (not

ensemble) parameters

steps towards non-asymptotic thermodynamics

(without thermodynamic limit !)

Application to Information Theory

(Niven, Phys. Lett. A, 342(4) (2005) 286; Physica A, 365(1) (2006) 142)

Adopt Boltzmann principle as definition of information:

I =h

ln2=

log2 W

N or

I =

D

ln2=

log2 P

N (in bits)

non-asymptotic coding ?

non-asymptotic network theory ?


Application: (In)distinguishability


Statistics Consider role of distinguishability:

Disting. balls Indisting. balls

Disting.

boxes

Maxwell-Boltzmann

(Lynden-Bell)*

Bose-Einstein

(Fermi-Dirac)*

Indisting. boxes

? ?

* maximum of 1 ball per box


Statistics Consider role of distinguishability:


Disting.

boxes

Maxwell-Boltzmann

(Lynden-Bell)*

Bose-Einstein

(Fermi-Dirac)*

Indisting. boxes

D:I statistic I:I statistic

* maximum of 1 ball per box


(In)distinguishability Allocate students to PhD supervisors:

1. Disting. students disting. supervisors - consider personal interactions

2. Indisting. students disting. supervisors

- e.g. Dean

3. Disting. students indisting. supervisors

- e.g. student club

4. Indisting. students indisting. supervisors

- e.g. Government department Choice of statistic - and hence entropy - depends on purpose

Tseng & Caticha (2002): “Entropy is not a property of a system … [it] is a property of our description of a system.”


(a) Maxwell-Boltzmann

WMB = N ! gi

ni

ni !i=1

s

(b) Bose-Einstein

WBE =(gi + ni 1)!

ni !(gi 1)!i=1

s

(c) Fermi-Dirac

WFD =gi !

ni !(gi ni )!i=1

s


Entropy functions: Name Asymptotic Entropy Non-asymptotic Entropy

(Niven, 2005, 2006) MB

hMB= pi ln

pi

gii=1

s

hMB(N)

=1

Nln[(piN)!]

i=1

s

+1

Npi ln[N!]+ pi lngi

BE

hBE = ( i + pi )ln( i + pi )

i=1

s

i ln i pi lnpi

hBE(N)

=1

Nln ( iN + piN 1)!{

i=1

s

ln ( iN 1)! ln (piN)! }

FD

hFD = ( i pi )ln( i pi )

i=1

s

+ i ln i pi lnpi

hFD(N)

=1

Ni=1

s

ln ( iN piN)!{

+ ln ( iN)! ln (piN)! }

where i = gi / N = relative degeneracy


MaxProb: maximise h subject to

pii=1

s

= 1,

pi frii=1

s

= fr , r = 1,...,R

Name Asymptotic distribution Non-asymptotic distribution (Niven, 2005, 2006)

MB

pMB,i*

= gi e 0' r frir=1

R

pMB,i#

=1

N

1 ln[N!]

N+ lngi 0 ' r fri

r=1

R

1

BE

pBE,i*

=i

e 0+ r frir=1

R

1

pBE,i#

=

1

N

1( iN + pBE,i

# N) 0 r frir=1

R

1

FD

pFD,i*

=i

e 0+ r frir=1

R

+1

pFD,i#

=

1

N

1( iN pFD,i

# N +1) 0 r frir=1

R

1

where = digamma f’n; -1 =inverse digamma f’n


(d) D:I Statistic

(Niven, CTNEXT07)

- disting. entities

- indisting. categories, each with g indisting. subcategories

Can show

WD:I =N

n1,n2,...,nk ,0,...,0(g)

=N !

ni !

i=1

k

rj !

j=1

N

ni

g=1

min(g,ni )

i=1

k

where rj = no. of occurrences of j in

{n

i}

ni

g = Stirling no. of 2nd kind

Curious behaviour!


(e) I:I Statistic

(Niven, NEXT 07)

- indisting. entities

- indisting. categories, each with g indisting. subcategories

Can show

WI:I(g) =N

n1,n2,...,nk ,0,...,0(g)

= P ( j)

=1

min(g, j)rj

j=1

n1

where rj = no. of occurrences of j in

{n

i}

P ( j) = partition number

a + b + ...( )m

= Wronski aleph = “combinatorial polynomial”


Normal Polynomials Wronski (1811) alephs

(a + b)2 = a2+ 2ab + b

2

(a + b)3 = a3+ 3a

2b + 3ab

2+ b

3

(a + b)m =m

ta

tb

m t

t=0

m

(a + b)2 = a2+ ab + b

2

(a + b)3 = a3+ a

2b + ab

2+ b

3

(a + b)m = atb

m t

t=0

m

Hence

a

=1

m

= a1

t1

t1,t

2,...,t

a2

t2 ...a

t with

t = m

=1


Example: Non-Degenerate MB and BE statistics


Example: Non-Degenerate D:I statistic


Example: Non-Degenerate I:I statistic


Summary (non-degenerate, no moment constraints)


Disting. boxes

MB statistic

MaxProb; MeanProb

Highly symmetric

Strongly asymptotic

uniform distrib.

BE statistic

MeanProb only

Highly symmetric

Strongly asymptotic uniform

distrib.

Indisting. boxes

D:I statistic

MaxProb; MeanProb

Highly asymmetric

Slowly asymptotic, s N

Non-asymptotic, s N ?

I:I statistic

MeanProb only

Highly asymmetric

Non-asymptotic, s N

Monotonic asymptote for s N


Application:

Pólya Distribution


Pólya Distribution (Grendar & Niven, cond-mat/0612697)

- urn: M disting. balls, with mi of each

category,

mi= M

i=1

s

- sample: N balls, with ni of each category

- scheme: draw of ball of category i, return to urn + add c balls of same category to urn

“neither independent nor identically distributed” (ninid) sampling

Prob. of each realization {ni} is:

PPólya =M c + N 1

N

1 mi c + ni 1

nii=1

s

Multivariate

Polya

distribution


Pólya cross-entropy: put qi = mi/M, and =N/M: Name Asymptotic cross-entropy Non-asymptotic cross-entropy

(without Stirling approx.) Pólya (c>0)

DPólyax

=1

Npi ln

(N +1) ( Nc)

( Nc+ N)i=1

s

+ ln(qiN

c+ piN)

(piN +1) (qiN

c)

Pólya (c<0)

DPólyaSt pi

c( c +1)ln( c +1)

i=1

s

+1c

(qi + pi c)ln(qi + pi c)

1c

qi lnqi pi lnpi

DPólyax

=1

Npi ln

(N +1) ( Nc

N +1)

( N

FD

+1)i=1

s

+ ln(

qiN

c+1)

(piN +1) (qiN

cpiN +1)


Pólya MaxProb

Name Asymptotic distribution Non-asymptotic distribution Pólya (c>0)

pPólya,i#

=1

N

1F(N, c) + (

qiN

c+ pPólya,i

# N) 0 r frir=1

R

1

Pólya (c<0)

pPólya,i*

=qi

e 0+ r frir=1R

c

pPólya,i#

=1

N

1K(N, c) (

qiN

cpPólya,i

# N +1) 0 r frir=1

R

1

Compare Acharya-Swamy (1994) ansatz for “anyons”

pi* 1

e 0+

1xi

, with [ 1,1]


Future Applications: Graphs and Networks


Graph Entropy (Körner & Longo, 1973; Körner & Orlitsky, 1998)

- vertices = categories (alphabet)

- lines (edges) connect disting. categories

H(G,P) = limsupN

1

Nlog2 (G

P

N ) +1( )

where P = prob. distrib on vertex set

(GP

N ) = chromatic no. of graph G

P

N , for N-sequence

consider “heterogeneous” distinguishability of categories

BUT is asymptotic (does not consider entities)

Strong connection to networks + coding theory


Conclusions


Conclusions • MaxProb principle: choose realization of highest probability

principle of probabilistic inference

explanation for MaxEnt, MinXEnt

generalised definitions of D and h

• Non-asymptotic theory

- finite N thermodynamics (microcanonical) - other applications!

• Other statistics:

- MB, BE, FD - indisting. categories - Polya sampling (“ninid”)

• Strong connections to graphs, networks + coding


Acknowledgments:

Thanks to:

• The University of New South Wales, Australia

• The European Commission, for Marie Curie Incoming

International Fellowship at University of Copenhagen

• Dr Bjarne Andresen + Dr Flemming Topsøe

• COSNET, ANU and (Prof. R. Dewar)2 for opportunity to present


References Acharya, R., Narayana Swamy, P. (1994) J. Phys. A: Math. Gen., 27: 7247-7263. Boltzmann, L. (1872) Sitzungsberichte Akad. Wiss., Vienna, II, 66: 275-370; English transl.: Brush,

S.G. (1966) Kinetic Theory: Vol. 2 Irreversible Processes, Permagon Press, Oxford, 88-175. Boltzmann, L. (1877), Wien. Ber., 76: 373-435, English transl., Le Roux, J. (2002), 1-63,

http://www.essi.fr/~leroux/. Clausius, R. (1865) Poggendorfs Annalen 125: 335; English transl.: R.B. Lindsay, in J. Kestin (ed.)

(1976) The Second Law of Thermodynamics, Dowden, Hutchinson & Ross, PA, (1976) 162. Clausius, R. (1876) Die Mechanische Wärmetheorie (The Mechanical Theory of Heat), F. Vieweg,

Braunschwieg; English transl.: W.R. Browne (1879), Macmillan & Co., London. Edwards, A.W.F. (2002) Pascal’s Arithmetical Triangle: The Story of a Mathematical Idea, 2nd ed.,

John Hopkins U.P., Baltimore. Grendar, M., Grendar, M. (2001) What is the question that MaxEnt answers? A probabilistic

interpretation, in A. Mohammad-Djafari (ed.) Bayesian Inference and Maximum Entropy Methods in Science and Engineering, AIP (Melville), 83.

Grendar, M., Niven, R.K. (in submission), http://arxiv.org/abs/cond-mat/0612697. Jaynes, E.T. (1957), Physical Review, 106: 620-630. Jaynes, E.T. (Bretthorst, G.L., ed.) (2003) Probability Theory: The Logic of Science, Cambridge

U.P., Cambridge. Körner, J., Longo, G. (1973) IEEE Trans. Information Theory IT-19(6): 778. Körner, J., Orlitsky, A., (1998) IEEE Trans. Information Theory 44(6) 2207. Kullback, S., Leibler, R.A. (1951), Annals Math. Stat., 22: 79-86.


Lilly, S. (2002), A Practical Guide to Runes, Caxton Editions, London. Niven, R.K. (2005), Physics Letters A, 342(4): 286-293. Niven, R.K. (2006), Physica A, 365(1): 142-149. Niven, R.K. (in submission) CTNEXT07, 1-5 July 2007, Catania, Sicily, Italy, http://arxiv.org/

abs/0709.3124. Niven, R.K. (2005-07) Combinatorial information theory: I. Philosophical basis of cross-entropy

and entropy, cond-mat/0512017. Niven, R.K., Suyari, H. (in submission) Combinatorial basis and finite forms of the Tsallis entropy

function. Pascal, B. (1654), Traité du Triangle Arithmétique, Paris. Paxson, D.L. (2005) Taking Up the Runes, Red Wheel/Weiser, York Beach, ME, USA. Pennick, N. (2003) The Complete Illustrated Guide to Runes, HarperCollins, London. Planck, M. (1901) Annalen der Physik 4: 553. Sanov, I.N. (1957) Mat. Sb. 42, 11-44; English transl. Selected Transl. Math. Stat. Prob. 1 (1961),

213-224. Shannon, C.E. (1948), Bell System Technical Journal, 27: 379-423; 623-659. Suyari, H. (2006), Physica A 368(1): 63. Vincze, I, (1974) Progress in Statistics, 2: 869-895. Historical references prior to 1800AD are given in Edwards (2002).


Appendix 1: Runic alphabet: (Refs: Lilly, 2002; Pennick, 2003; Paxson, 2005; Wikipedia)

fuTarkgw hnijIpzs tbemlNod ...

f.u.th.a.r.k.g.w h.n.i.j.eo.p.z.s t.b.e.m.l.ng.o.d

- used across Germanic + central Europe, Britain + Scandinavia, 5th-10th cent.; in Sweden to 17th cent.

- derived from Etruscan alphabet (not Greek or Roman) - each rune has symbolic meaning Anglo-Saxon h (“Haegl”) = old German h (“Hagalaz”) = hail, hailstones - symbolic of destructive force of Nature, but melts and gives

new life - evokes need to accept what is inevitable; to “go with the flow”;

i.e. rune of transformation

The Combinatorial Basis of Entropy...

Documents

Transcript of The Combinatorial Basis of Entropy...