• Statistica 1: Neyman vs Bayes.

frequenze in esp. di conteggio

• Statistica 2: likelihood ratio e test di ipotesi

segnale su fondo

• Track fitting: tracking in GEANT3 &GEANT4

filtri di Kalman e global fitting

1

Alberto Rotondi Università di Pavia - Scuola di Alghero 2010

2

3

2

4

5

PHYSTAT 05 - Oxford 12th - 15th September 2005

Statistical problems in Particle Physics, Astrophysics and Cosmology

6

7

8

9

Frequentist confidenceintervals

x

10

x1x2

x

x= x1 < < x= x2 < < 2

x

1< < 2 when x1 < x <x2

True value

Possibleinterval

CL

P 1< < 2) = P(x1 < x<x2) = CL

11

NEYMAN INTEGRALS

x

x

Elementary statisticsmay be

WRONG!!

12

Because P{Q} does not

contain the parameter!

13

Estimation of the sample mean

since

Due to the Central Limit theorem we have a pivot quantity when N>>1

Hence:

14

Hence, we have three methods to find confidenceintervals with the Neyman (frequentist) technique:

• Graphical method (Neyman band)

•Analytic with the Neyman Integrals (Clopper Pearson method)

• Inversion method (pivot variable)

x

x

Counting experiments

CLtx

xP

][

||

CL is the asymptotic probability the interval will contain the true value

COVERAGE is the probability that the specific experiment does contain the true value irrespective of what the true value is

On the infinite ensemle of experiments, for a continuous variable Coverageand CL tend to coincide

In counting experiments the variables are discrete and CL and Coverage do not coincide

What is requested is the minimum overcoverage

16

Wald

1

)1(

4

1

2||)1(

||2

2

2

2

2

n

tn

ff

n

tt

n

tn

tf

pt

n

pp

pf

n

fftfpn )1(1

t is the quantile of the normal distribution

t

t=1, area 84%Quantile =0.84P[|f-p|<t ]= 68%

CLt

n

pp

pFPt

p

pFP

)1(

||

][

||

Counting experiments: Binomial case

Wilson interval(1934)

Wald (1950)Standard in Physics

17

xtxx

Counting experiments: Poisson case

Wilson interval (1934)

Wald (1950)Standard in Physics

42

|| 22 txt

txCLt

xP

Why to complicate all this?

18

Why to complicate all this?

19

n

fftfn )1(1

n=20

20

p1 p2

p

21

The 90% CL gaussian upper limit

Observedvalue

90% area

10% area

Meaning II: a larger upper limit should give values less than the observed one in less than 10% of the experiments

1.28

Meaning III: the probability to be wrong is 10%

Meaning I: with this upper limit, values less than theobserved one are possible with a probability <10%

22

23

muon a selects trigger the that prob.

pion a selects trigger the that prob.

piona betoprob.

muona betoprob.

trigger a give to pion a for prob.

trigger a give to muon a for prob.

)|(

)|(

90.0)(

10.0)(

05.0)|(

95.0)|(

TP

TP

P

P

TP

TP

The probability to be a muon after the trigger P( |T):

678.090.005.010.095.0

10.095.0

)()|()()|(

)()|()|(

PTPPTP

PTPTP

The trigger problem

10.000 particles

9000 1.000

950450

8550 50

enrichment 950/(950+450) = 68%

Efficiency (950+450)/10.000 = 14%

triggertrigger

prior

26

27

28

29

30

31

Bayesian credible interval

32

2n

33

34

35

BayesiansvsFrequentists

+

36

Why ML does work?

hypothesis observation

37

38

)

39

)();(ln2)(

2

1ln 2

2

2)(

2

1 2

XLX

ex

Gaussian variables: ML corresponds to Minimum 2

40

The weighted average

22

21

22

2

21

1

2

22

22

21

212

110

)(,

)()()(

xx

xx

Consider the well-known weighted mean:

122

2

2

1

2

112

2

2

1

2

12

2

21

2

2

2

2

1

1

2

2

2

1

2

2

2

1

2

2

2

1

2

2

2

2

1

1

11xxx

xxxx

xx

A simple algebraic manipulation gives the recursive form(Kalman filter):

measured point weight matrix predictionKalman= the measurement is weightedwith a model prediction (track following)

Gaussian variables:

weighted average = Bayes (uniform) = Likelihood

41

42

20 events have been generated and 5 passed the cut What is the estimation of the efficiency with CL=90%?

Frequentist result:

Bayesian result:

=[0.104, 0.455]

=[0.122, 0.423]

x=5, n=20, CL=90%

90.0

)1(

)1(

1

0

2

1

d

d

xnx

p

p

xnx

1, 2

1, 2

What meaning??

05.0)1( 22

0

knk

x

k k

n

05.0)1( 11knk

n

xk k

n

Elementary example

x

43

n

fftfn )1(1

Efficiency calculation: an OPEN PROBLEM!!

1

)1(

4

1

22

2

2

2

2

n

tn

ff

n

tt

n

tn

tf

2/)1( 11knk

n

xk k

n

?

)1(

)1(

1

0

2

1 CL

xnx

p

p

xnx

d

d

2/)1( 22

0

knk

x

k k

n

Wilson interval (1934)

Bayes.This is not frequentistbut can be testedin a frequentist way

Wald (1950)Standard in Physics

Exact frequentistClopper Pearson (1934) (PDG)

Coverage simulation

44

x = gRandom → Binomial(p,N) → x

2/)1( 11knk

n

xk

ppk

n

2/)1( 22

0

knk

x

k

ppk

n

1-CL =

p

p1p2

Tmath:: BinomialI(p,N,x)

k++=k/n

p2p1

0ne expects ~ CL

45

Simulate many x with a true pand check when the intervals contain the true value p . Compare this frequency with the stated CL

CL=0.95, n=50

Simulate many x with a true p and check when the intervals contain the true value p . Compare this frequency with the stated CL

46

CL=0.90, n=20

47

In the estimation of the efficiency (probability) the coverage is “chaotic”

The new standard (not yet for physicists)is to use the exact frequentist or the formula

The standard formula

should be abandoned

111,/

)1(1],,0[,0

)1(],1,[,,

1

)1(

4

1

2

2/

/122

/111

2

2/

2

2

2/2/

2

2/

2

2/

is gaussian, t,α-CLtnxf

CLppx

CLppnx

n

t

n

ff

n

tt

n

t

n

tf

n

n

n

fftf

)1(BYE-BYE

48

A further improvement:The continuity correction is equivalent toThe Clopper-Pearson formula

1is1 1 gaussian,

,/)5.0(,/)5.0(

)1(1],,0[,0

)1(],1,[,,

1

)1(

4

1

2

2/

/122

/111

2

2/

2

2

2/2/

2

2/

2

2/

t,α-CLt

nxfnxf

CLppx

CLppnx

n

t

n

ff

n

tt

n

t

n

tf

n

n

This should become the standard formula also for physicists

Correzione di continuità

6.5 7 7.5

Quando una distribuzione discreta (come la binomiale) è approssimata con una continua(come la gaussiana), l’area del rettangolo centrato su un valore discreto x= della variabile, che rappresenta la probabilità P( ), può essere stimata con l’area sottesa dalla curva continua nell’intervallo [ 0.5≤x≤ +0.5]

Ciò equivale a considerare il valore discreto della variabile come il punto medio dell’intervallo [ 0.5 , +0.5]

La probabilità binomiale per x= =7è stimata mediante l’area sottesa dalla curva normale approssimantenell’intervallo [6.5 ≤x≤ 7.5]

Binomiale (istogramma a rettangoli)Gaussiana (curva continua)

Se la probabilità si riferisce a una sequenza di valori discreti - cioè è richiesta la

probabilità P( 1)+P( 2)+…+ P( k) - questa si può approssimare con la probabilità

gaussiana nell’intervallo [ 1 0.5 ≤x≤ k+0.5]

Più in dettaglio, ecco come operare la correzione di continuità per diversi valori cercati:

Valore della probabilità Intervallo su cui stimare

binomiale la probabilità gaussianaP(x= ) [ 0.5≤x≤ +0.5]

P(x≤ ) [x≤ +0.5]

P(x< ) [x≤ 0.5]

P(x≥ ) [x≥ 0.5]

P(x> ) [x≥ 0.5]

P( 1≤x≤ k) [ 1 0.5 ≤x≤ k+0.5]

P( 1<x< k) [ 1 0.5 ≤x≤ k 0.5]

L’approssimazione gaussiana alla binomiale si rivela particolarmente utile per n»1(fattoriali grandi nei coefficienti binomiali), e se si sommano le probabilità per una lunga sequenza di valori. La correzione di continuità consente di migliorare l’approssimazione

Esempio

a) Trovare la probabilità che esca Testa 23 volte in 36 lanci di una moneta

La distribuzione richiesta è una binomiale con n=36, p=q=1/2. La variabile è =23

Approssimazione di Gauss con =np=36 1/2=18 e =√npq=√36 1/2 1/2=3

Tenendo conto della correzione di continuità, i due valori della variabile standardizzatacorrispondenti agli estremi dell’intervallo di integrazione [22.5, 23.5] sono:

La probabilità binomiale è: P ( ) n!

!(n !)p q

n 36!

23!13!

1

2

231

2

13

3.36%

z122.5 18

31.50 z2

23.5 18

31.83

P 1( ≤z≤1.50 )=43.32%

P 2( ≤z≤1.83 )=46.64%

Pertanto la probabilità binomiale che esca Testa 23 volte, con l’approssimazionedi Gauss, può essere stimata in questo modo:

P ( ≤z≤1.83 )= P 2 P 1 =3.32%

Il valore è abbastanza vicino a quello calcolato esattamente con la binomiale

Nota: per un dato valore della variabile , la probabilità con l’approssimazione di Gauss si può anche stimare dal valore assunto dalla densità normale per x=

Il valore di probabilità di una distribuzione continua per uno specifico valore della variabile è zero, come abbiamo visto. In questo caso però stiamo semplicemente approssimando il valore di una distribuzione discreta (la binomiale) con il valoreassunto per quel valore dalla curva normale.

b) Trovare la probabilità che esca Testa almeno 23 volte in 36 lanci

Con l’approssimazione di Gauss, dobbiamo trovare la probabilità normale nell’intervallo [22.5, ∞], che stima la probabilità binomiale che Testa esca 23 o più volte

P (z≥1.5)=50% P ( ≤z <1.5)=50% 43.32%=6.68%

Pbin( ) fG(x )

1

2e(x )

2/ 2

2

Pertanto possiamo porre:

Pbin( )

1

3 2e(23 18)

2/ 2 3

2

0.1330 0.2494 0.0332 3.32%

Nel caso dell’esempio:

che coincide con il valore trovato valutando l’area nell’intervallo [22.5, 23.5]

53

N=50 CL=0.90

54

N=50 CL=0.95

The likelihood ratio method

55

),(

),(ln2),(ln2

),(

),(),(

xpL

xpLxp

xpL

xpLxp

best

best

Maximize

Minimize

56

Binomial Coverage simulationmax likelihood constraint

nk k

Feldman & Cousins, Phys. Rev. D 57(1998)3873

UNIFIED method

p

fxN

p

fxNp

1

1ln)(ln2),,(ln2

),,(ln2),,(ln2and,0|

,)1()!(!

!

xNpkNpkkA

CLppkNk

NkNk

Ak

57

N=50 CL=0.90

58

N=50 CL=0.95

59

N=20 CL=090

The problem persists alsowith large samples!

60

0.90

0.95

0.86

61

N=20 CL=0.90

62

From coin tossing to physics:the efficiency measurement

Valid also for

k=0 and k=n

ArXiv:physics/0701199v1

63

N=20 CL=0.90 Interval amplitude

likelihood

frequentist

Wilson cc

Wilson

x

64

N=20 CL=0.90 Interval limits

x

65

(2001) n

fftfn )1(1

2/)1( 22

0

knk

x

k k

n

2/)1( 22

0

knk

x

k k

n1

)1(

4

1

22

2

2

2

2

n

tn

ff

n

tt

n

tn

tf

66

xtxx

Counting experiments: Poisson case

?

!

!

0

2

1 CL

x

e

x

e

x

x

d

d

Wilson interval (1934)

Bayes.This is not frequentistbut can be testedin a frequentist way

Wald (1950)Standard in Physics

Exact frequentistClopper Pearson (1934) (PDG)

42

)( 22 txt

txt

x

2/!

2

0

2 ek

x

k

k

2/!

11 e

kxk

k

67

Poissonian Coverage simulation

CL=68%

68

Poissonian Coverage simulation

CL=90%

The Neyman-FC integrals

70

xq2q1

Neyman integrals

),(

),(ln2),(ln2

xL

xLx best minimize

),(ln2),(ln2and,0|

,);(

xkkkA

CLxp

Ak

71

Poissonian Coverage simulationmax likelihood constraint

CLekAk

k

!

nk k

Feldman & Cousins, Phys. Rev. D 57(1998)3873

nnn

nen

nen

nn

n

ln)(!/

!/ln2),(ln2

72

Poissonian Coverage simulation

en

ek

kACLek

nk

Ak

k

!!:

!

CL=68%

73

Poissonian Coverage simulation

CL=90%

74

Counting experiments: new formula for the Poisson case

Wilson interval with Continuity correctiongives the same results as …

Exact frequentistClopper Pearson (1934) (PDG)

5.042

)( 22

xxt

xtt

xtx

2/!

2

0

2 ex

x

k

x

2/!

11 exxk

x

75

76

77

The Unitarity Triangle

d s b

u

c

t

V V V

V V V

V V V

ud us ub

cd cs cb

td ts tb

0***tbtdcbcdubud VVVVVV

*

*

td tb

cd cb

V V

V V

*

*

ud ub

cd cb

V V

V V

1

B=(1,0)

C=(0,0)

A=( , )

• Quark mixing is described

by the CKM matrix

• Unitarity relations on matrix

elements lead to a triangle

in the complex plane

IVVt

CP Violation

Luca Lista Statistical Methods for Data

Analysis

78

79

A Bayesian application: UTFit

• UTFit: Bayesian determination of the CKM unitarity triangle– Many experimental and theoretical inputs

combined as product of PDF– Resulting likelihood interpreted as Bayesian

PDF in the UT plane

• Inputs:

– Standard Model experimental measurements and parameters

– Experimental constraints

Luca Lista Statistical Methods for Data

Analysis

80

Combine the constraints

• Given {xi} parameters and {ci} constraints

• Define the combined PDF– ƒ( ρ, η, x1, x2 , ..., xN | c1, c2 , ..., cM ) ∝

∏j=1,M ƒj(cj | ρ, η, x1, x2 , ..., xN) ∏i=1,N ƒi(xi)⋅ ƒo (ρ, η)

– PDF taken from experiments, wherever it is possible

• Determine the PDF of (ρ, η) integrating over the remaining parameters– ƒ(ρ, η) ∝

∫ ∏j=1,M ƒj(cj | ρ, η, x1, x2 , ..., xN) ∏i=1,N ƒi(xi)⋅ ƒo (ρ, η)

A priori PDF

=

=

Statistical Methods for Data

Analysis

81

Unitarity Triangle fit

68%, 95%

contours

Luca Lista Statistical Methods for Data

Analysis

82

PDFs for and

Luca Lista Statistical Methods for Data

Analysis

83

Projections on other observables

84

A Frequentist application: RFit

• RFit: to choose a point in the plane, and ask for the best set of the parameters for this points. The 2

values give the requested confidence region.

• No a priori distribution of parameters is requested

85

)();(ln2)(

2

1ln 2

2

2)(

2

1 2

xLx

ex

86

87

88

89

Conclusions• The usual formulae used by physicists in counting

experimets should be abandoned

• By adopting a practical attitude, also bayesianformulae can be tested in a frequentist way

• frequentism is the best way to give the resultsof an experiment in the form

x but other forms are also possible

• physicists should use Bayes formulae to parametrizethe previous (th or exp)

knowledge, not the ignorance

90

Quantum Mechanics: frequentist or bayesian?Born or Bohr?

dx2||

The standard interpretation is frequentist

END

91

92

x

Neyman integrals

Bootstrap );(1);( xFxF

Search for pivotal variables

This method avoids the graphic procedure and the resolution of the Neyman integrals

93