The Combinatorial Basis of Entropy...

The Combinatorial Basis of Entropy (“MaxProb”)

22nd Canberra International Physics Summer School ANU, Canberra

11 December 2008

by Robert K. Niven

Marie Curie Incoming International Fellow, 2007-2008 Niels Bohr Institute, University of Copenhagen, Denmark

School of Aerospace, Civil and Mechanical Engineering

The University of New South Wales at ADFA Canberra, ACT, Australia

R.K. Niven, UNSW 22nd Canberra International Physics Summer School 2

Lectures

1. The Combinatorial Basis of Entropy (“MaxProb”)

2. Jaynes’ MaxEnt, Riemannian Metrics and the Principle of Least Action

Contents • Historical overview

- combinatorics - probability theory

• Combinatorial basis of entropy / MaxProb principle

generalised combinatorial definitions of entropy and cross-entropy

explanation of MaxEnt / MinXEnt

• Applications 1. Multinomial systems (asymptotic vs non-asymptotic) 2. (In)distinguishable particles or categories 3. “Neither independent nor identically distributed” sampling

• Future applications …

(Advertisement)

Courses at UNSW@ADFA, Canberra:

• Short course in “Maximum Entropy Analysis”, 14-15 May 2009

(fee paying $1270).

• Masters course: ZACM8327 Maximum Entropy Analysis, semester 2,

2009 (fee paying or UNSW@ADFA enrolled student)

- 3 hours of lectures + tutorials per week

- based on similar course at Niels Bohr Institute

Historical Overview

Combinatorics Knowledge is very old! (Edwards, 2002)

(a) Number-patterns

Pythagoras (500BC)

Egypt (300BC)

Theon of Smyrna, Nicomachus (100AD)

Higher dimensions: Tartaglia (1523, publ. 1556)

figurate numbers fk

(b) Binomial coefficients

= coefficients of (a + b)N

Al-Karaji (1007); Al-Samawal (1180); Al-Kashi (1429)

Chia Hsien (1100); Yang Hui (1261); Chu Shih-chieh (1303)

Cardano (1570), etc

- applied to solution of equations; finding roots; etc

Chu Shih-chieh (1303)

binomial coefficients

k , where =dimension

(c) Combinations + Permutations

Ancient; e.g.

- Susruta (600BC), Jains (300BC): combinations of 6 tastes

- Pingala (200BC): combinations of syllables

No. of permutations of N things = N !

- Hebrew Book of Creation (700); Bhaskara (1150)

No. of groups of N things, taken k at a time: - Mahavira (850); Bhaskara (1150); ben Gerson (1321)

Without replacement With replacement

Combinations

k !(N k)!

=N + k 1

(N + k 1)!

k !(N 1)!

Permutations PkN

= N(k) =N!

(N k)!

Pascal (1654) - equivalence of figurate numbers AND binomial coefficients AND

numbers of combinations without replacement

Multinomial weight - Bhaskara (1150); Mersenne (1636)

= no. of permutations of N objects, containing ni of each category

i = 1,...,s , is:

W =N !

2! ... n

s!= N !

Probability Theory (a) Classical period (e.g. Cardano (1560s), Pascal, Fermat, Huygens, the Bernoullis,

Montmort, de Moivre, Laplace)

Probability =

No. of outcomes of interest

Total no. of outcomes

(b) “Frequentist” school (e.g. Venn, Pearson, Neyman, Fisher, von Mises, Feller)

- probability = measurable frequency, for an infinite number of repetitions of a “random experiment”

- attempt to define probabilities as certainties

- narrow applicability

(c) “Bayesian” or “Plausibility” school (Bayes, Laplace, Jeffreys, Polya, Cox, Jaynes, 1957; 2003)

- probability = “plausibility” = assignment based on what you know

- need not be a measurable frequency - manipulate using sum + product rules (Jaynes, 2003)

- “subjective” = “information-dependent”

- different observers, with different information, can assign different probabilities to the same event

- more useful; encompasses all frequentist situations

Probability Distributions “Measures of Central Tendency”

Continuous parameter x Discrete parameter x p(x) = probability density function (pdf) p(x) = pi = probability mass function

Combinatorial (or Probabilistic) Definition of Entropy

Definitions Entity = a discrete particle, object or agent, or an item in a sequence, which is

separate but not necessarily independent of other entities

Category = possible assignment of an entity

Probabilistic System = a set of entities K assigned to a

set of categories C by a discrete random variable : K C

e.g. physics: particles energy levels

gambling: die throws die sides

communications: signal bits letters of alphabet

Configuration = distinguishable permutation of entities amongst categories e.g.: physics: microstate; information theory: sequence

Realization = aggregated arrangement of entities amongst categories = set of configurations e.g.: physics: macrostate; information theory: type

Commonly define realizations by the no. of entities in each category {n

MaxProb Principle MaxProb Principle (Boltzmann, 1877; Planck, 1901; Vincze, 1974; Grendar & Grendar, 2001)

- “A system can be represented by its most probable realization”

principle for probabilistic inference

- does not depend on asymptotic limits

- does not give certainty

BB GB BG GG

Superset of 2nd Law “A system tends towards its most probable realization”

- not just thermodynamics!

MaxProb Principle MaxProb Principle (Boltzmann, 1877; Planck, 1901; Vincze, 1974; Grendar & Grendar, 2001)

- “A system can be represented by its most probable realization”

principle for probabilistic inference

- does not depend on asymptotic limits

- does not give certainty

BB GB BG GG

Superset of 2nd Law “A system tends towards its most probable realization”

- not just thermodynamics!

Ergodicity

Inference

Multinomial Systems (Boltzmann, 1877)

N distinguishable balls (entities)

s distinguishable boxes (categories)

qi = source (“prior”) probability of ball falling in ith box

= normalised degeneracy gi / gii=1

Probability of a given realization {n

i} is given by the multinomial

distribution:

Pmult = N ! qi

ni !i=1

qi =1/s

; Wmult

= N !1

If categories equiprobable, can use multinomial weight

MaxProb: want to maximise:

Easier to maximise:

lnPmult = lnN !+ ni lnqii=1

lnni !

Asymptotic limit for N (rigorously by Sanov (1957) theorem; crudely by Stirling’s approx. lnm! m lnm m ):

DKL = limN

lnPmult

N= pi ln

where pi =

If qi = 1/s = constant:

hSh= lim

lnWmult

N= pi lnpi

Shannon entropy

Kullback - Leibler cross - entropy

= directed divergence

= negative of relative entropy

Pmult = N ! qi

ni !i=1

Summary

• Kullback-Leibler and Shannon functions are asymptotic forms of the

multinomial distribution P

• If minimise D

KL (MinXEnt) or maximise

hSh (MaxEnt) of a multinomial

system, subject to constraints

obtain asymptotic MaxProb realization

Boltzmann principle:

Define entropy and cross-entropy by: h=

(compare S

total= SN = k lnW )

hence always consistent with MaxProb

Jaynes’ MaxEnt • Jaynes (1957)

- minimise D or maximise h , subject to constraints

“most uncertain” distribution = distribution which contains the least

information

• BUT how do we define uncertainty?

- Jaynes only considers D

hSh axiomatic basis of Shannon

(1948)

• However, a system: - need not be multinomial ! - need not be asymptotic !

Kullback-Leibler or Shannon functions will not give the MaxProb distribution

If know P P

mult or N , must include this information !

Application to Multinomial Systems (Asymptotic + Non-Asymptotic)

Why the Multinomial? (Niven, AIP Conf. Proc. 954 (2007) 133; Blower, pers. comm.)

Pmult = N ! qi

ni !i=1

with pi =

1. Frequentist approach

mult, {qi } = measurable frequencies

2. Bayesian approach

mult, {qi } = Bayesian probabilities

If ignorant about choice of model P , then all models equiprobable

must choose multinomial (“central model theorem”)

Asymptotic Analysis Jaynes’ (1957) Algorithm

Minimise

DKL = pi lnpi

subject to

s= 1 and

sfri = fr , r = 1,...,R

Form Lagrangian, differentiate w.r.t. pi

pi*= qi e 0

r frir=1

Zqi e r frir=1

Z = e 0

= qi e r frir=1

Boltzmann

distribution

with 0

Jaynes’ (1957, 1963, 2003) analysis many more (generic) relations

Isolated Thermodynamic System:

Isolated from rest of universe

e.g. microcanonical ensemble

Natural:

Mean energy:

si = U

*= qi e 0

1 i =1

Zqi e 1 i

compare S* = k lnZ +U

or F = kT lnZ = TS * + U

Hence 0

'= lnZ = and

Non-Asymptotic Analysis (Niven, Phys. Lett. A, 342(4) (2005) 286; Physica A, 365(1) (2006) 142)

Use raw multinomial:

Minimise

NlnN !+ ni lnqi

lnni !

subject to

s= N and

sfri= F

r, r = 1,...,R

Form Lagrangian, differentiate w.r.t. n

1 lnN !

N+ lnqi 0

(N)r(N)fri

where ( )= digamma function

Non - asymptotic

distribution

Example 1:

Multinomial

n1,n2,n3

subject to

Example 2: Multinomial

n1,n2,n3

subject to

i i= E

= 1,2,4[ ]

Example 2 (cont’d):

For constant

obtain

Thermodynamic Double System:

System 1: N

1 particles; energy levels

System 2: N

2 particles; energy levels

Probs. of realizations {ni }, {nj } are:

Maximise P1P

2 subject to

most probable distrib.:

+ lnqi 0a

(N1)1 i 1

1 lnN2 !

+ lnqj 0b

(N2)1 j 1

“Zeroth law” upheld

( 1 in common)

P1 = N1! qi

ni !i=1

, P2 = N2 ! qj

nj !j=1

ET = ni ii=1

s+ nj jj=1

Summary

- combinatorial approach straightforward analysis using system (not

ensemble) parameters

steps towards non-asymptotic thermodynamics

(without thermodynamic limit !)

Application to Information Theory

(Niven, Phys. Lett. A, 342(4) (2005) 286; Physica A, 365(1) (2006) 142)

Adopt Boltzmann principle as definition of information:

log2 W

log2 P

N (in bits)

non-asymptotic coding ?

non-asymptotic network theory ?

Application: (In)distinguishability

Statistics Consider role of distinguishability:

Disting. balls Indisting. balls

Disting.

Maxwell-Boltzmann

(Lynden-Bell)*

Bose-Einstein

(Fermi-Dirac)*

Indisting. boxes

* maximum of 1 ball per box

Statistics Consider role of distinguishability:

Disting.

Maxwell-Boltzmann

(Lynden-Bell)*

Bose-Einstein

(Fermi-Dirac)*

Indisting. boxes

D:I statistic I:I statistic

* maximum of 1 ball per box

(In)distinguishability Allocate students to PhD supervisors:

1. Disting. students disting. supervisors - consider personal interactions

2. Indisting. students disting. supervisors

- e.g. Dean

3. Disting. students indisting. supervisors

- e.g. student club

4. Indisting. students indisting. supervisors

- e.g. Government department Choice of statistic - and hence entropy - depends on purpose

Tseng & Caticha (2002): “Entropy is not a property of a system … [it] is a property of our description of a system.”

(a) Maxwell-Boltzmann

WMB = N ! gi

ni !i=1

(b) Bose-Einstein

WBE =(gi + ni 1)!

ni !(gi 1)!i=1

(c) Fermi-Dirac

WFD =gi !

ni !(gi ni )!i=1

Entropy functions: Name Asymptotic Entropy Non-asymptotic Entropy

(Niven, 2005, 2006) MB

hMB= pi ln

hMB(N)

Nln[(piN)!]

Npi ln[N!]+ pi lngi

hBE = ( i + pi )ln( i + pi )

i ln i pi lnpi

hBE(N)

Nln ( iN + piN 1)!{

ln ( iN 1)! ln (piN)! }

hFD = ( i pi )ln( i pi )

+ i ln i pi lnpi

hFD(N)

ln ( iN piN)!{

+ ln ( iN)! ln (piN)! }

where i = gi / N = relative degeneracy

MaxProb: maximise h subject to

pi frii=1

= fr , r = 1,...,R

Name Asymptotic distribution Non-asymptotic distribution (Niven, 2005, 2006)

pMB,i*

= gi e 0' r frir=1

pMB,i#

1 ln[N!]

N+ lngi 0 ' r fri

pBE,i*

e 0+ r frir=1

pBE,i#

1( iN + pBE,i

# N) 0 r frir=1

pFD,i*

e 0+ r frir=1

pFD,i#

1( iN pFD,i

# N +1) 0 r frir=1

where = digamma f’n; -1 =inverse digamma f’n

(d) D:I Statistic

(Niven, CTNEXT07)

- disting. entities

- indisting. categories, each with g indisting. subcategories

Can show

WD:I =N

n1,n2,...,nk ,0,...,0(g)

min(g,ni )

where rj = no. of occurrences of j in

g = Stirling no. of 2nd kind

Curious behaviour!

(e) I:I Statistic

(Niven, NEXT 07)

- indisting. entities

- indisting. categories, each with g indisting. subcategories

Can show

WI:I(g) =N

n1,n2,...,nk ,0,...,0(g)

= P ( j)

min(g, j)rj

where rj = no. of occurrences of j in

P ( j) = partition number

a + b + ...( )m

= Wronski aleph = “combinatorial polynomial”

Normal Polynomials Wronski (1811) alephs

(a + b)2 = a2+ 2ab + b

(a + b)3 = a3+ 3a

2b + 3ab

(a + b)m =m

(a + b)2 = a2+ ab + b

(a + b)3 = a3+ a

2b + ab

(a + b)m = atb

2,...,t

t2 ...a

t with

Example: Non-Degenerate MB and BE statistics

Example: Non-Degenerate D:I statistic

Example: Non-Degenerate I:I statistic

Summary (non-degenerate, no moment constraints)

Disting. boxes

MB statistic

MaxProb; MeanProb

Highly symmetric

Strongly asymptotic

uniform distrib.

BE statistic

MeanProb only

Highly symmetric

Strongly asymptotic uniform

distrib.

Indisting. boxes

D:I statistic

MaxProb; MeanProb

Highly asymmetric

Slowly asymptotic, s N

Non-asymptotic, s N ?

I:I statistic

MeanProb only

Highly asymmetric

Non-asymptotic, s N

Monotonic asymptote for s N

Application:

Pólya Distribution

Pólya Distribution (Grendar & Niven, cond-mat/0612697)

- urn: M disting. balls, with mi of each

category,

- sample: N balls, with ni of each category

- scheme: draw of ball of category i, return to urn + add c balls of same category to urn

“neither independent nor identically distributed” (ninid) sampling

Prob. of each realization {ni} is:

PPólya =M c + N 1

1 mi c + ni 1

Multivariate

distribution

Pólya cross-entropy: put qi = mi/M, and =N/M: Name Asymptotic cross-entropy Non-asymptotic cross-entropy

(without Stirling approx.) Pólya (c>0)

DPólyax

Npi ln

(N +1) ( Nc)

( Nc+ N)i=1

+ ln(qiN

c+ piN)

(piN +1) (qiN

Pólya (c<0)

DPólyaSt pi

c( c +1)ln( c +1)

(qi + pi c)ln(qi + pi c)

qi lnqi pi lnpi

DPólyax

Npi ln

(N +1) ( Nc

+1)i=1

(piN +1) (qiN

cpiN +1)

Pólya MaxProb

Name Asymptotic distribution Non-asymptotic distribution Pólya (c>0)

pPólya,i#

1F(N, c) + (

c+ pPólya,i

# N) 0 r frir=1

Pólya (c<0)

pPólya,i*

e 0+ r frir=1R

pPólya,i#

1K(N, c) (

cpPólya,i

# N +1) 0 r frir=1

Compare Acharya-Swamy (1994) ansatz for “anyons”

, with [ 1,1]

Future Applications: Graphs and Networks

Graph Entropy (Körner & Longo, 1973; Körner & Orlitsky, 1998)

- vertices = categories (alphabet)

- lines (edges) connect disting. categories

H(G,P) = limsupN

Nlog2 (G

N ) +1( )

where P = prob. distrib on vertex set

N ) = chromatic no. of graph G

N , for N-sequence

consider “heterogeneous” distinguishability of categories

BUT is asymptotic (does not consider entities)

Strong connection to networks + coding theory

Conclusions

Conclusions • MaxProb principle: choose realization of highest probability

principle of probabilistic inference

explanation for MaxEnt, MinXEnt

generalised definitions of D and h

• Non-asymptotic theory

- finite N thermodynamics (microcanonical) - other applications!

• Other statistics:

- MB, BE, FD - indisting. categories - Polya sampling (“ninid”)

• Strong connections to graphs, networks + coding

Acknowledgments:

Thanks to:

• The University of New South Wales, Australia

• The European Commission, for Marie Curie Incoming

International Fellowship at University of Copenhagen

• Dr Bjarne Andresen + Dr Flemming Topsøe

• COSNET, ANU and (Prof. R. Dewar)2 for opportunity to present

References Acharya, R., Narayana Swamy, P. (1994) J. Phys. A: Math. Gen., 27: 7247-7263. Boltzmann, L. (1872) Sitzungsberichte Akad. Wiss., Vienna, II, 66: 275-370; English transl.: Brush,

S.G. (1966) Kinetic Theory: Vol. 2 Irreversible Processes, Permagon Press, Oxford, 88-175. Boltzmann, L. (1877), Wien. Ber., 76: 373-435, English transl., Le Roux, J. (2002), 1-63,

http://www.essi.fr/~leroux/. Clausius, R. (1865) Poggendorfs Annalen 125: 335; English transl.: R.B. Lindsay, in J. Kestin (ed.)

(1976) The Second Law of Thermodynamics, Dowden, Hutchinson & Ross, PA, (1976) 162. Clausius, R. (1876) Die Mechanische Wärmetheorie (The Mechanical Theory of Heat), F. Vieweg,

Braunschwieg; English transl.: W.R. Browne (1879), Macmillan & Co., London. Edwards, A.W.F. (2002) Pascal’s Arithmetical Triangle: The Story of a Mathematical Idea, 2nd ed.,

John Hopkins U.P., Baltimore. Grendar, M., Grendar, M. (2001) What is the question that MaxEnt answers? A probabilistic

interpretation, in A. Mohammad-Djafari (ed.) Bayesian Inference and Maximum Entropy Methods in Science and Engineering, AIP (Melville), 83.

Grendar, M., Niven, R.K. (in submission), http://arxiv.org/abs/cond-mat/0612697. Jaynes, E.T. (1957), Physical Review, 106: 620-630. Jaynes, E.T. (Bretthorst, G.L., ed.) (2003) Probability Theory: The Logic of Science, Cambridge

U.P., Cambridge. Körner, J., Longo, G. (1973) IEEE Trans. Information Theory IT-19(6): 778. Körner, J., Orlitsky, A., (1998) IEEE Trans. Information Theory 44(6) 2207. Kullback, S., Leibler, R.A. (1951), Annals Math. Stat., 22: 79-86.

Lilly, S. (2002), A Practical Guide to Runes, Caxton Editions, London. Niven, R.K. (2005), Physics Letters A, 342(4): 286-293. Niven, R.K. (2006), Physica A, 365(1): 142-149. Niven, R.K. (in submission) CTNEXT07, 1-5 July 2007, Catania, Sicily, Italy, http://arxiv.org/

abs/0709.3124. Niven, R.K. (2005-07) Combinatorial information theory: I. Philosophical basis of cross-entropy

and entropy, cond-mat/0512017. Niven, R.K., Suyari, H. (in submission) Combinatorial basis and finite forms of the Tsallis entropy

function. Pascal, B. (1654), Traité du Triangle Arithmétique, Paris. Paxson, D.L. (2005) Taking Up the Runes, Red Wheel/Weiser, York Beach, ME, USA. Pennick, N. (2003) The Complete Illustrated Guide to Runes, HarperCollins, London. Planck, M. (1901) Annalen der Physik 4: 553. Sanov, I.N. (1957) Mat. Sb. 42, 11-44; English transl. Selected Transl. Math. Stat. Prob. 1 (1961),

213-224. Shannon, C.E. (1948), Bell System Technical Journal, 27: 379-423; 623-659. Suyari, H. (2006), Physica A 368(1): 63. Vincze, I, (1974) Progress in Statistics, 2: 869-895. Historical references prior to 1800AD are given in Edwards (2002).

Appendix 1: Runic alphabet: (Refs: Lilly, 2002; Pennick, 2003; Paxson, 2005; Wikipedia)

fuTarkgw hnijIpzs tbemlNod ...

f.u.th.a.r.k.g.w h.n.i.j.eo.p.z.s t.b.e.m.l.ng.o.d

- used across Germanic + central Europe, Britain + Scandinavia, 5th-10th cent.; in Sweden to 17th cent.

- derived from Etruscan alphabet (not Greek or Roman) - each rune has symbolic meaning Anglo-Saxon h (“Haegl”) = old German h (“Hagalaz”) = hail, hailstones - symbolic of destructive force of Nature, but melts and gives

new life - evokes need to accept what is inevitable; to “go with the flow”;

i.e. rune of transformation

The Combinatorial Basis of Entropy...

Documents

Transcript of The Combinatorial Basis of Entropy...

Entropy for complex systems - Universität Wien · Why talking of entropy of Complex Systems ? The central concept: understanding macroscopic system behavior on the basis of microscopic

Influence of Size and Shape Effects on the Solubility of Hydrocarbons the Role of the Combinatorial Entropy

Toric algebra of hypergraphs - link.springer.comKeywords Hypergraph · Edge ring ·Markov basis ·Toric ideal · Monomial walk · Sunﬂower ·Combinatorial discrepancy ·Parameter

Combinatorial Logic Design Multiplexers and ALUs · Combinatorial vs. Sequential Logic • The CPU schematic shows both combinatorial and sequential logic blocks • Combinatorial

Combinatorial Generation - 1stWorks1stworks.com/ref/RuskeyCombGen.pdf · Combinatorial Generation Combinatorial Generation Combinatorial Generatio ombinatorial Generation Combinatorial

Entropy-Based Algorithms for Best Basis

Combinatorial Algorithms - jsnu.edu.cnmathsqxx.jsnu.edu.cn/_upload/article/81/10/8db4ac934402a2003f300b... · Combinatorial Algorithms Lectures on Graph Theory, ... 2 Combinatorial

1 Combinatorial Dominance Analysis Keywords: Combinatorial Optimization (CO) Approximation Algorithms (AA) Approximation Ratio (a.r) Combinatorial Dominance.

Information Science and Statistics · 11/3/1990 · Rubinstein and Kroese: The Cross-Entropy Method: A Unified Approach to Combinatorial Optimization, Monte Carlo Simulation, and

40th Combinatorial Mathematics and Combinatorial Computing · 2017-01-17 · WELCOME Welcome to 40ACCMCC—the 40th Australasian Conference on Combinatorial Mathematics and Combinatorial

Author's personal copy - Ruđer Bošković Institute · Author's personal copy Maximum entropy algorithm with inexact upper entropy bound based on Fup basis functions with compact

RESIDUAL' . ENTROPY OF ICE, AND RELATED COMBINATORIAL PROBLEMS Bound... · in Ag2H310o must give rise to unequal a-priori probabilities ... all Npoints to be equal to (3J8)N ... RESIDUAL

COMBINATORIAL CHEMISTRY AND DYNAMIC COMBINATORIAL CHEMISTRY

Combinatorial Games Martin Müller. Contents Combinatorial game theory Thermographs Go and Amazons as combinatorial games.

The Cross-Entropy Method · The Cross-Entropy Method: A Uniﬁed Approach to Combinatorial Optimization, Monte Carlo Simulation and Machine Learn-ing, Springer-Verlag, New York, 2004.

Weighted Combinatorial Group Theory and Wild Metric Complexes · A basic construction in combinatorial group theory is the free group F(X)with basis X.HereXdenotes a set equipped

Combinatorial Optimization and Combinatorial Structure in Computational Biology

The Cross-Entropy Method - University of Queenslandpeople.smp.uq.edu.au/DirkKroese/ps/tutslides.pdf · 2004-10-31 · The Cross-Entropy Method: A Uniﬁed Approach to Combinatorial

COMBINATORIAL MODULUS, THE COMBINATORIAL LOEWNER PROPERTY ...bkleiner/coxeter.pdf · COMBINATORIAL MODULUS, THE COMBINATORIAL LOEWNER PROPERTY, ... and prove Cannon’s conjecture

Entropy and Isoperimetry for Linear and non-Linear Group Actions. · non-Linear Group Actions. Misha Gromov June 11, 2008 Contents 1 Concepts and Notation. 3 1.1 Combinatorial Isoperimetric