Classical inference in/for physics

Classical Inferencein Physics

Thiago Mosqueiro

Institute of Physics of Sao CarlosUniversity of Sao Paulo

July 31 2012

Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 1 / 38

Equilibrium criticality – Ising model

• Model of ferromagnetism in statisticalmechanics

• Contact with thermal reservoir

• Lattice of N binary elements – spins

• Each site of this lattice: or

Energy of a configuragion S = (S1, S2, . . .),

E(S) = −J∑〈p,j〉

SjSp

Probability of S = (S1, S2, . . .),

P (S) =

exp

(− JkT

∑〈p,j〉

SjSp

)∑∀S

exp

(− JkT

∑〈p,j〉

SjSp

)Monte Carlo step: each time step means N itera-

tions of the algorithm.


The Market for Lemons: Quality Uncertainty and the Market Mechanism, G. Akerlof.The quarterly journal of economics, 1970


Objectivies

• What’s the difference between Probability and Statistics?

• Inference

• Statistics – models, hypotehsis and estimation

• Main focus: Maximum likelyhood estimators

• Glimpse of hypothesis testing

“What! you have solved it already?”“Well, that would be too much to say.

I have discovered a suggestive fact, that is all. It is, however, very suggestive.”

Sign of Four (II), Sir Arthur Conan Doyle


Summary

1 Probability and random variables

2 Estimators

3 Maximum likelihood

4 Non-trivial example

5 One last example

6 Hypothesis testing

7 Conclusions


Where are we?


2 Estimators



5 One last example


7 Conclusions


Random variable

• Let |ψ〉 be the state of a particle

• Suppose we know for every n the solutions H |n〉 = εn |n〉

What is the probability the particle is in state |n〉?

• It can happen that |ψ〉 = |1〉 . . . or |ψ〉 = |2〉 . . . or . . .

• In this sense, |ψ〉 is a random variable

• Suppose you have n dice Side j of each die have probability pj

• The result of tossing the j-th die is Xj

What is the probability of Xj > x?

• Xj is another random variable


Random variables

• Events:X = x, X ≤ x, X = x and Y < y

• Probability of an event:P (X = x), P (X ≤ x), P (X = x, Y < y)

• Moments:〈X〉 :=

∑x

P(X = x)x, 〈X2〉 :=∑x

P(X = x)x2, . . .

• Surprisal:

I (X = x) = − log [P (X = x)]

• Entropy:

〈I (X = x)〉 = −∑x

P (X = x) log [P (X = x)]


Example: gaussian distributed variable

X ∼ N(µ, σ)

P(x < X ≤ x+ dx) = ρ(x)dx = A exp

(− (x− µ)2

σ2

)dx

- 3 - 2 - 1φ μ,σ

2(

0.8

0.6

0.4

0.2

0.0

−5 −3 1 3 5

x

1.0

−1 0 2 4−2−4

x)

0,μ=0,μ=

0,μ=

−2,μ=

2 0.2,σ =2 1.0,σ =2 5.0,σ =2 0.5,σ =


Example: gaussian distributed variable

X ∼ N(µ, σ)

P(x < X ≤ x+ dx) = ρ(x)dx = A exp

(− (x− µ)2

σ2

)dx

0

1

2

3

4

5

6

0 100 200 300 400 500 600 700

Eve

nt X

j

Realization j


Now the big problem...

What if I don’t have a model...?

How can one obtain information from observations?

How can I know my model fits the reality?


Where are we?


2 Estimators



5 One last example


7 Conclusions


Statistically infering...

• Statistics validates and fits Probabilistic models

• build a statistical model that should describe the process• interpret the data as realizations of your model

• Inference gives you a statistical proposition

• Models may be parametric, non-parametric or semi-parametric

• Of course let’s focus on parametric models.

Let’s guess our model:Xj ∼ N(µ, σ)

What’s your best guessabout µ and σ?

Usual way of doing this estimate isby means of an estimator

0

1

2

3

4

5

6

0 100 200 300 400 500 600 700

Eve

nt X

j

Realization j


Estimator

• Data: set of i.i.d. X1, X2, X3, . . . Xm

• Given a data set and a statistical model, we have a probability withsome parameter θ

• An estimator is a function of the data set to some sample estimates

Examples of estimators

• X =1

m

m∑j=1

Xj

• σ =1

m

m∑j=1

(Xj −X

)2• xmin = min (X1, X2, X3, . . . Xm)


Mean Square Error

• EQM[θ]

:=

⟨(θ − θ

)2⟩

• EQM[θ]

=⟨θ2 − 2θθ + θ2

⟩=⟨θ2⟩− 2θ

⟨θ⟩

+ θ2

=⟨θ2⟩−⟨θ⟩2

+⟨θ⟩2

− 2θ⟨θ⟩

+ θ2

EQM[θ]

= Var[θ]−B2(θ)

• Bias: B(θ) =⟨θ⟩− θ

• Non-biased estimator: B(θ) = 0 ⇐⇒⟨θ⟩

= θ


Example to gaussian mean and variance

• Random sample: X1, X2, X3, . . . , Xm

• Starting with the mean value: µ = X :=1

m

m∑j=1

Xj

• 〈µ〉 =

⟨1

m

m∑j=1

Xj

⟩=

1

m

m∑j=1

〈Xj〉 =1

m

m∑j=1

µ = µ

Thus, X is a non-biased estimator for µ

• Now let’s take a look at the

• EQM [µ] = Var[X]

= Var

[1

m

m∑j=1

Xj

]=

1

m2

m∑j=1

Var [Xj ] =1

m2

m∑j=1

σ2 =σ2

m

Moreover, X → µ when m→∞


Example to gaussian mean and variance

• Let’s now do it with the variance: σ2b =

1

m

m∑j=1

(Xj −X

)2•⟨σ2b

⟩=

⟨1

m

m∑j=1

(Xj −X

)2⟩=

1

m

m∑j=1

⟨(Xj −X

)2⟩=m− 1

mσ2

Thus, σb is asymptotically non-biased estimator for σ

• Conversely, let’s define σ2 =1

m− 1

m∑j=1

(Xj −X

)2• EQM [µ] = σ2

σ is an unbiased estimator for σ


Back to Gaussian random variables

0

1

2

3

4

5

6

0 100 200 300 400 500 600 700

Eve

nt X

j

Realization j

• Estimation: σb = 0.49944735873

• Actual value used to generate the data: σ = 0.49980448952


Where are we?


2 Estimators



5 One last example


7 Conclusions


Maximum Likelihood

• Maximize the ”likelihood“: get an estimator for te parameter of agiven statistical model

• p(x1, x2, x3, . . . |θ) :: given the value θ for the parameter,this is the probability that X1 = x1, X2 = x2, . . .

• p(x1, x2, x3, . . . |θ) = p(x1|θ)p(x2|θ)p(x3|θ) . . . =m∏j=1

p(xj |θ)

• Let x = x1, x2, x3, . . .

L(θ, x) =m∏j=1

p(xj |θ)

• To maximize it, we can use ln (L(θ, x)) =m∑j=1

p(xj |θ)

• ∂

∂θln (L(θ, x)) = 0 – solve it for θ


Example for gaussian

• Suppose Xj ∼ N(µ, σ) is our statistical model

• L(µ, x) =

(1√

2πσ2

)mexp

(1

2σ2

m∑j=1

(xj − µ)2

)

• By calculating∂

∂µln (L(µ, x)) = 0, we get to

m∑j=1

(xj − µ) =m∑j=1

xj −mµ = 0

• Finally, we get µ =1

m

m∑j=1

xj

• On the other hand,∂

∂σln (L(σ, x)) = 0 gives ........

• This is what is usually done to derive such an estimator


Example for gaussian

• Suppose Xj ∼ N(µ, σ) is our statistical model

• L(µ, x) =

(1√

2πσ2

)mexp

(1

2σ2

m∑j=1

(xj − µ)2

)

• By calculating∂

∂µln (L(µ, x)) = 0, we get to

m∑j=1

(xj − µ) =m∑j=1

xj −mµ = 0

• Finally, we get µ =1

m

m∑j=1

xj

• On the other hand,∂

∂σln (L(σ, x)) = 0 gives σ =

1

m

m∑j=1

(Xj −X

)2,

which we have already discovered to be biased!

• This is what is usually done to derive such an estimator


Deutsch Tank Problem

• Suppose you have a box with (unkown) n tickets, labled from 1 to n.

• You take one ticket, it’s label is x. What’s your best guess for n?


Estimation of the maximum

• Statistical model: p(x|n) =1

n, x ≤ n – a uniform distribution

• Remember p(x|n) = 0, x > n. This is important to this maximization.

• L(n, x) =1

n, which is maximized when n is the largest!

• Therefore, our best guess at this moment would be n = x.

• If we had conversely made several observations X1, X2, X3, . . ., then

n = max (X1, X2, X3, . . .)

• ”However, this is awful“ – it is a poor estimation!

• Other methods are far more accurate and, indeed, were succesfully used in WWII.


Where are we?


2 Estimators



5 One last example


7 Conclusions


What is this?

Large values significant?

Is the mean informative? Probably not.

0

10

20

30

40

50

60

70

0 20 40 60 80 100 120 140 160

Xj

Realization j


Empirical PDF

• We don’t know p(x)∆x, but we have a lot data...

• In this experiment, I will use 107 points.

• We can then calculate the following estimator:

p(x) =1

m

m∑j=1

δ (Xj ∈ [x, x+ ∆x])

10-8

10-7

10-6

10-5

10-4

10-3

10-2

10-1

100

100 101 102 103 104

P(X

= x

)

Event X = x

In fact, we can show thatp(x) is an unbiasedestimator for p(x).

Let’s propose then a model:p(x) ∼ x−α

But... α =?


Empirical CDF

If you want to fit, use the CDF

10-7

10-6

10-5

10-4

10-3

10-2

10-1

100

100 101 102 103 104 105

P(X

>=

x)

Event X = x

However, how about a maximum likelihood estimator?


Exponent estimator

• General power-law distribution: p(x)dx =α− 1

xmin

(x

xmin

)−αdx

• We have to maximize the likelihood: L (α|X1, . . . Xm) =m∏j=1

α− 1

xmin

(Xjxmin

)−α

• ln [L (α|X1, . . . Xm)] =m∑j=1

{ln(α− 1)− ln(xmin)− α ln

(Xjxmin

)}• Maximizing it...

α = 1 +m

m∑j=1

Xj/xmin


PDF

In our case, α ∼ 2.65

10-8

10-7

10-6

10-5

10-4

10-3

10-2

10-1

100

100 101 102 103 104

P(X

= x

)

Event X = x

Testing dataFitting curve


Where are we?


2 Estimators



5 One last example


7 Conclusions


What about this...

-250

-200

-150

-100

-50

0

50

100

150

0 50 100 150 200 250 300

Eve

nt X

j

Realization j

Let’s start by the empirical pdf!


Gaussian!!! :)

Again, we can propose a model: A Gaussian!

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

-4 -2 0 2 4

P(X

= x

)

Event X = x

Estimating the variance: σ ≈ 200


Gaussian!!! :(

Again, we can propose a model: A Gaussian!

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

-4 -2 0 2 4

P(X

= x

)

Event X = x

Empirical PDFFitting

Estimating the variance: σ ≈ 200 – Something is not right!


Okay, not a gaussian

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

-10 -5 0 5 10

P(X

= x

)

Event X = x


Okay, not a gaussian

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

-20 -15 -10 -5 0 5 10 15 20

P(X

= x

)

Event X = x


Power-law decay – Cauchy distributions

1e-06

1e-05

0.0001

0.001

0.01

0.1

1

1 10 100 1000 10000 100000

P(X

= x

)

Event X = x

Empirical CDF

Very suggestive indeed.


Where are we?


2 Estimators



5 One last example


7 Conclusions


Hypothesis testing

• What we have been doing till now is ∼ exploratory analysis

• We now want to confirm a prediction or hypothesis – confirmatory analysis

• Is this last data set gaussian or cauchy distributed?

• The general recipe:

- An initial guess, possibly true

- State an relevant null and its alternative hypothesis

- Formulate an appropriate test – T and a significance level τ

- Estimate the distribution of T under your null hypothesis

- Compute the observed quantity t and verify your null hypothesis


Let’s try this shit!

• H0: the data is normally distributed – N(0, σ)

• We have the estimated pdf of the data sample – these are ourm observations Oj

• The test will be

T =m∑j=1

(Oj − Ej)2

Ej

• Ej = A exp

(− (x)2

σ2

)are the expected frequencies!

• It is easy to derive that T ∼ χ2m−1 (Bolfarine)

• In our case, the observed t ≈ 400 and P (T = t)→ 0.

• This rejects H0.


Suggestions

• Other hypothesis testing techniques:

* τ -Student test

* minimax

* Lagrange multiplier

* Union-intersection

* Fisher test

* ...

• Non-parametric testins, such as Kolmogorov-Smirnov


Where are we?


2 Estimators



5 One last example


7 Conclusions


Classical inference in/for physics

Education

Transcript of Classical inference in/for physics