131219 NuPhys Stat

Sta$s$cal issues in long baseline neutrino physics

Alessandra Tonazzo APC / Université Paris-‐Diderot

with the LAGUNA-‐LBNO Collabora$on

in par$cular L.Agos$no (APC/CNRS), D.Franco (APC/CNRS) and especially V.Galymov (CEA/IRFU)

Recent debate on MH determina$on in future experiments

Notable references (many others are available): •  [Qian] X.Qian et al., Phys.Rev. D86 (2012) 113011, arXiv:1210.3651 •  [CEZ] E.Ciuffoli, J.Evslin and X.Zhang, arXiv:1305.5150 •  [BCHS] M.Blennow, P.Coloma, P.Huber and T.Schwetz, arXiv:1311.1822 •  [Blen] M.Blennow, arXiv:1311.3183

I will discuss the approach chosen by the LBNO experiment

nσ ?

Median experiment ?

NuPhys Dec.19th,2013 A.Tonazzo 1

The LBNO experiment

LBNO will face two problems of hypothesis tes$ng: •  Mass Hierarchy: NH or IH ? •  CP viola$on in the leptonic sector: δCP=0,π or δCP≠0,π ?

CPV is dependent on knowledge of MH: LBNO strategy is to ensure MH is "done" at 5σ in the first years of running


EoI CERN-‐SPSC-‐2012-‐021 (SPSC-‐EOI-‐007) New beam from CERN WBB with <E>~ 4 GeV Baseline=2300km Pyhasalmi mine, Finland : @4000mwe depth GLACIER LAr TPC (+MIND) Phase I : SPS 750 kW (~1.2x1020pot/y)

20 kt LAr Phase II : HPPS 2MW (~3x1021pot/y)

70 kt LAr

Aim: test a “Null Hypothesis” H0 against an “alterna$ve hypothesis” H1.

Method (“frequen$st”): 1) define a “test sta$s$c” T, func$on of the data 2) construct the PDF of T under under each hypothesis 3) define a “cri$cal region” ΩC such that T in ΩC suggests H0 is true 4) Evaluate the probability to give the wrong answer

Test of hypothesis: the basics


PDF(T|H1) PDF(T|H0) Tcα

“cri$cal region” T>Tcα

Test of hypothesis: the basics 4) Evaluate the probability to give the wrong answer,

which you can do in two ways: –  reject H0 when it’s true : Error of type I or “loss”: α –  accept H0 when H1 is true: Error of type II or “contamina$on”: β

Defini$ons: Confidence Level CL = 1-‐α

Power p= 1-‐β


PDF(T|H1) PDF(T|H0)

Tcα

“cri$cal region” T>Tcα

β

Note: “3σ” means

if 1-‐sided integrals are

used

= PDF(T |H0 )dTTC

+∞

∫= PDF(T |H1)dT

−∞

TC

∫ PDF(T |H0 )dTTC

+∞

∫ = 99.73%

α β

reject H0 at CL (1-‐α): T<TCα

The test sta$s$c for MH If the two hypotheses are “simple”, the uniformly most powerful test is provided by the Likelihood Ra$o (Neyman-‐Pearson lemma) •  In general, the LR is a good choice •  For the MH problem, it is equivalent to a difference of χ2’s

(in the gaussian limit)

•  Defini$on of the test sta$s$c for MH : T = χ2IH-‐χ2NH (“Δχ2”)


− lnLNH = − ln L ni |µiNH( )

i∈bins∏

$

%&

'

() ≈ − ln exp − ni−µi

NH( )2

2µiNH

+

,-

.

/0

i∈bins∏

$

%&&

'

())=

ni−µiNH( )2

2µiNH

+

,-

.

/0

i∈bins∑ = χNH

2

ln LNHLIH

= −χNH2 + χ IH

2

idem for IH

PDFs of the test sta$s$c for MH If the hypotheses are “nested” (eg: H0 is {θ in ωCΩ} and H1 is {θ in Ω}), Wilk’s theorem states that

PDF(T|H0) = X2(1dof) => Nσ CL is ensured at Tc=N2

But, for the MH case, the hypotheses are NOT nested ! => PDF(T|H0) is not X2(1dof) and the Tc values must be computed based on

the correct distribu$on [CEZ,Qian]


Figure 1 of [Qian]

PDFs of the test sta$s$c for MH •  Ideally: get PDFs from toy-‐MC simula$ons •  A good approxima$on for the MH case [CEZ,Qian]: gaussians with σ=2√μ

PDF(T|NH)=N(T0NH,2√T0NH) PDF(T|IH)=N(-‐T0IH,2√T0IH)


T-150 -100 -50 0 50 100 150

A.U

.

0

0.02

0.04

0.06

True NH

True IH° = 90CPδ POT, 2010×LBNO: 2.25

0T20 40 60 80 100 120 140

0T/ T

σ1.6

1.8

2

2.2

2.4

0T = 2Tσ

/2π = CPδToys NH, /2π = 3CPδToys IH,

Τ0’s increase (almost linearly) with exposure @LBNO

The “factor 2”


1)  consider posi$ve values of T as sugges$ng NH (and nega$ve IH): require T to be “Nσ” from TCNH=TCIH=0 : T0> N 2√T0 #σ = √T0/2 and power=CL

“Crossing sensi$vity” in [BCHS]

Let’s assume, for simplicity, T0IH=T0NH

Possible approaches commonly used :

2)  consider NH suggested when values of T are more than

“Nσ” from TCNH=T0IH : T0-‐(-‐T0)> N 2 √T0 #σ = √T0 and power = 0.5

“Median experiment”

and don’t forget we should use one-‐sided integrals of PDFs (small difference, cfr [BCHS])

3σ 3σ

The “factor 2”


cfr also Fig.3 of [BCHS]

Median experiment power=0.5, TC=-‐T0 (or Tc=0 with N(T0,√T0))

“crossing” power=CL, TC=0

2χ Δ0 20 40 60 80 100 120 140

σN

0

2

4

6

8

10

2χΔQian et al.

/2π = CPδToys NH, /2π = 3CPδToys IH,

σ5

σ3

T

√T

The bo�omline: one must always quote the power along with the CL !

Defining the MH test(s) As proposed in [BCHS] 1.  Choose the CL you want 2.  NH is null hypothesis:

Define TCNH such that 3.  Compute power 4.  Quote power for a given T0NH or for all values of T0NH

5.  Repeat 2-‐4 for IH as null hypothesis (with a change of sign…) “cri$cal region” is T<TCIH :


Typically T0 increases with $me, so power can also be plo�ed vs $me/exposure

Fig.2 of [BCHS]

PDF(T |H0 )dTTCNH

+∞

∫ =CL

p = PDF(T |H1)dT−∞

TCNH

∫

Simple analy$cal formulae if PDF’s are gaussian, otherwise use toy-‐MC

PDF(T |H0 )dT−∞

TCIH

∫ =CL p = PDF(T |H1)dTTCIH

+∞

∫

Dependence on oscilla$on parameters •  As pointed out in [BCHS], we are actually dealing with “composite

hypotheses”: T0 depends on parameters (δCP,θ23,…) whose values will not be known at the $me of the experiments

•  Need to define Tc’s such that the desired CL is ensured for all values of the parameter(s)

•  The power will depend on the value of the parameter(s)


Fig.12 of [BCHS]

LBNO poten$al for MH •  T0NH and T0IH depend mainly on δCP •  From the figures below, one chooses:

for NH TCα(T0NH(δCP=90ο)) for IH TCα(T0IH(δCP=270ο)) •  Evaluate power vs exposure for all values of δCP


CPδTrue 0 1 2 3 4 5 6

2 χ Δ

0

50

100

150

200

250

300medium value

bandσ1 band σ3

σ3σ5

MH determination (NH assumed)50% nu+50% anu3.0e20 pots

myf

itter

H0=NH H0=IH

T

~2.5 years ~2.5 years

T

LBNO poten$al for MH •  Power vs exposure for all values of δCP (shaded bands)


Exposure (/1e20 POT)0 1 2 3 4 5

βp

= 1-

0

0.2

0.4

0.6

0.8

1

σ3

σ5Test power for NH

Exposure (/1e20 POT)0 1 2 3 4 5

βp

= 1-

0

0.2

0.4

0.6

0.8

1

σ3

σ5Test power for IH

p = 0.5 => “Median experiment” 50% chance to give the wrong answer if the alterna$ve hypothesis is true

WOULD YOU SPEND 20 YEARS ON SUCH AN EXPERIMENT?

p ~1 => “Full power experiment” ~0 chance to give the wrong answer if the alterna$ve hypothesis is true

THE LBNO CHOICE TO QUOTE

SENSITIVITY

~5 years

The Bayesian approach for MH In the “frequen$st approach” discussed so far, two sets of informa$on must be provided (for NH and for IH). With a Bayesian approach [Blen], a single set of values can contain all the informa$on on the test : •  You need a “prior” P on each hypothesis, providing a rela$ve normaliza$on

between the two PDFs •  It can be proven [D.F.] that P(NH)=P(IH)=0.5 is the most conserva$ve choice

1)  Define a threshold t 2)  Compute TcNH such that

and TCIH such that

.


PDF(T | NH )PDF(T | NH )+PDF(T | IH )

> t

for T>TCNH

PDF(T | IH )PDF(T | NH )+PDF(T | IH )

> t

for T<TCIH

NOTE !!! we are looking the “Odds” or “ra$o of posterior probabili$es” : IF YOUR EXPERIMENT GIVES A RESULT T, THE PROBABILITY THAT NATURE IS ACTUALLY NH(IH) is >CL . A ques$on that cannot be answered in the frequen$st approach !

The Bayesian approach for MH 3)  Define the “p-‐value” as For example, in D.Franco et al, JHEP 1304 (2013) 008 ArXiv 1301.4332 (ORCA/PINGU)


•  TC’ are set by the ra$o of the heights of the two gaussians at a given value of T

•  p-‐value is 0.5* (blue area + red area)

PDF(T | IH )dT +−∞

TCIH

∫ PDF(T | NH )dTTCNH

+∞

∫

3σ CL means t=99.73%

=T

What about δCP ?


/ ndf 2χ 40.63 / 24 0A 49.8± 4950

k 0.0116± 0.9635

2χΔ0 5 10 15 20

1

10

210

310

/ ndf 2χ 40.63 / 24 0A 49.8± 4950

k 0.0116± 0.9635

) from toy MC0

|H2χΔf(

distribution fitk2χ

δCP=0,π

/ ndf 2χ 27.53 / 27 0A 62.3± 1405

µ 0.580± 9.152 σ 0.242± 5.708 α 0.1774± 0.6272

2χΔ0 10 20 30 40

A.U

.

0

200

400

600

800

/ ndf 2χ 27.53 / 27 0A 62.3± 1405

µ 0.580± 9.152 σ 0.242± 5.708 α 0.1774± 0.6272

) from toy MC1

|H2χΔf(

skew normal fit

δCP=π/2

LBNO, 15x1020pot

•  We are in a case of “nested hypotheses”: H0 = {δCP=0 or π} , H1 = {0<δCP<2π}

•  Test sta$s$c Δχ2 = min(Δχ2δCP=0 , Δχ2δCP=π) ⇒ Wilk’s theorem hold ! PDF(Δχ2|H0)=Χ2(1dof) => TC=(# of σ)2

independent of exposure (unlike for MH) •  PDF(Δχ2|H1) not easy to predict, can be es$mated with toy-‐MC for each

value of δCP

What about δCP ? <Δx2> is strongly dependent on δCP (the impact of other oscilla$on parameters is much smaller)

Power vs exposure can be computed by integra$ng toyMC distribu$ons, it will be maximal for δCP=π/2,3π/2 and as low as zero for other values


CPδTrue 0 1 2 3 4 5 6

2 χ Δ

024

6

8101214

16

1820

90%C.L.

σ3

exclusion with NHπ = 0,CPδ

ν+25% ν75% C2P: 15e20 pots

myf

itter

Exposure (/1e20 POT)0 2 4 6 8 10 12 14 16

βp

= 1-

0

0.2

0.4

0.6

0.8

1 /2π = CPδ

/2π = 3CPδ

σ390%

LBNO Preliminary

<Δχ2>

Summary / Discussion •  Animated discussion, recent papers helped clarify the situa$on •  The star$ng point: get the PDF’s correct ! •  The main point: decide which ques$on you want to answer

•  A test of hypotheses is characterised by two values (CL and power), and both must be specified. For MH, two sets of values are necessary, unless a Bayesian approach is chosen.

•  For comparison between experiments at a given CL, the chosen value of power is not relevant – as long as it’s the same for all

•  For choice of a future project, the LBNO Collabora$on considers ~100% power as an important requisite for inves$ng money and $me in an experiment –  LBNO can determine MH at 5σ CL with ~100% power in ~5 years

•  A treatment of δCP discovery poten$al with analogous sta$s$cal approach has also been proposed


131219 NuPhys Stat

Documents

Transcript of 131219 NuPhys Stat