Post on 09-Jun-2022
Sta$s$cal issues in long baseline neutrino physics
Alessandra Tonazzo APC / Université Paris-‐Diderot
with the LAGUNA-‐LBNO Collabora$on
in par$cular L.Agos$no (APC/CNRS), D.Franco (APC/CNRS) and especially V.Galymov (CEA/IRFU)
Recent debate on MH determina$on in future experiments
Notable references (many others are available): • [Qian] X.Qian et al., Phys.Rev. D86 (2012) 113011, arXiv:1210.3651 • [CEZ] E.Ciuffoli, J.Evslin and X.Zhang, arXiv:1305.5150 • [BCHS] M.Blennow, P.Coloma, P.Huber and T.Schwetz, arXiv:1311.1822 • [Blen] M.Blennow, arXiv:1311.3183
I will discuss the approach chosen by the LBNO experiment
nσ ?
Median experiment ?
NuPhys Dec.19th,2013 A.Tonazzo 1
The LBNO experiment
LBNO will face two problems of hypothesis tes$ng: • Mass Hierarchy: NH or IH ? • CP viola$on in the leptonic sector: δCP=0,π or δCP≠0,π ?
CPV is dependent on knowledge of MH: LBNO strategy is to ensure MH is "done" at 5σ in the first years of running
NuPhys Dec.19th,2013 A.Tonazzo 2
EoI CERN-‐SPSC-‐2012-‐021 (SPSC-‐EOI-‐007) New beam from CERN WBB with <E>~ 4 GeV Baseline=2300km Pyhasalmi mine, Finland : @4000mwe depth GLACIER LAr TPC (+MIND) Phase I : SPS 750 kW (~1.2x1020pot/y)
20 kt LAr Phase II : HPPS 2MW (~3x1021pot/y)
70 kt LAr
Aim: test a “Null Hypothesis” H0 against an “alterna$ve hypothesis” H1.
Method (“frequen$st”): 1) define a “test sta$s$c” T, func$on of the data 2) construct the PDF of T under under each hypothesis 3) define a “cri$cal region” ΩC such that T in ΩC suggests H0 is true 4) Evaluate the probability to give the wrong answer
Test of hypothesis: the basics
NuPhys Dec.19th,2013 A.Tonazzo 3
PDF(T|H1) PDF(T|H0) Tcα
“cri$cal region” T>Tcα
Test of hypothesis: the basics 4) Evaluate the probability to give the wrong answer,
which you can do in two ways: – reject H0 when it’s true : Error of type I or “loss”: α – accept H0 when H1 is true: Error of type II or “contamina$on”: β
Defini$ons: Confidence Level CL = 1-‐α
Power p= 1-‐β
NuPhys Dec.19th,2013 A.Tonazzo 4
PDF(T|H1) PDF(T|H0)
Tcα
“cri$cal region” T>Tcα
β
Note: “3σ” means
if 1-‐sided integrals are
used
= PDF(T |H0 )dTTC
+∞
∫= PDF(T |H1)dT
−∞
TC
∫ PDF(T |H0 )dTTC
+∞
∫ = 99.73%
α β
reject H0 at CL (1-‐α): T<TCα
The test sta$s$c for MH If the two hypotheses are “simple”, the uniformly most powerful test is provided by the Likelihood Ra$o (Neyman-‐Pearson lemma) • In general, the LR is a good choice • For the MH problem, it is equivalent to a difference of χ2’s
(in the gaussian limit)
• Defini$on of the test sta$s$c for MH : T = χ2IH-‐χ2NH (“Δχ2”)
NuPhys Dec.19th,2013 A.Tonazzo 5
− lnLNH = − ln L ni |µiNH( )
i∈bins∏
$
%&
'
() ≈ − ln exp − ni−µi
NH( )2
2µiNH
+
,-
.
/0
i∈bins∏
$
%&&
'
())=
ni−µiNH( )2
2µiNH
+
,-
.
/0
i∈bins∑ = χNH
2
ln LNHLIH
= −χNH2 + χ IH
2
idem for IH
PDFs of the test sta$s$c for MH If the hypotheses are “nested” (eg: H0 is {θ in ωCΩ} and H1 is {θ in Ω}), Wilk’s theorem states that
PDF(T|H0) = X2(1dof) => Nσ CL is ensured at Tc=N2
But, for the MH case, the hypotheses are NOT nested ! => PDF(T|H0) is not X2(1dof) and the Tc values must be computed based on
the correct distribu$on [CEZ,Qian]
NuPhys Dec.19th,2013 A.Tonazzo 6
Figure 1 of [Qian]
PDFs of the test sta$s$c for MH • Ideally: get PDFs from toy-‐MC simula$ons • A good approxima$on for the MH case [CEZ,Qian]: gaussians with σ=2√μ
PDF(T|NH)=N(T0NH,2√T0NH) PDF(T|IH)=N(-‐T0IH,2√T0IH)
NuPhys Dec.19th,2013 A.Tonazzo 7
T-150 -100 -50 0 50 100 150
A.U
.
0
0.02
0.04
0.06
True NH
True IH° = 90CPδ POT, 2010×LBNO: 2.25
0T20 40 60 80 100 120 140
0T/ T
σ1.6
1.8
2
2.2
2.4
0T = 2Tσ
/2π = CPδToys NH, /2π = 3CPδToys IH,
Τ0’s increase (almost linearly) with exposure @LBNO
The “factor 2”
NuPhys Dec.19th,2013 A.Tonazzo 8
1) consider posi$ve values of T as sugges$ng NH (and nega$ve IH): require T to be “Nσ” from TCNH=TCIH=0 : T0> N 2√T0 #σ = √T0/2 and power=CL
“Crossing sensi$vity” in [BCHS]
Let’s assume, for simplicity, T0IH=T0NH
Possible approaches commonly used :
2) consider NH suggested when values of T are more than
“Nσ” from TCNH=T0IH : T0-‐(-‐T0)> N 2 √T0 #σ = √T0 and power = 0.5
“Median experiment”
and don’t forget we should use one-‐sided integrals of PDFs (small difference, cfr [BCHS])
3σ 3σ
The “factor 2”
NuPhys Dec.19th,2013 A.Tonazzo 9
cfr also Fig.3 of [BCHS]
Median experiment power=0.5, TC=-‐T0 (or Tc=0 with N(T0,√T0))
“crossing” power=CL, TC=0
2χ Δ0 20 40 60 80 100 120 140
σN
0
2
4
6
8
10
2χΔQian et al.
/2π = CPδToys NH, /2π = 3CPδToys IH,
σ5
σ3
T
√T
The bo�omline: one must always quote the power along with the CL !
Defining the MH test(s) As proposed in [BCHS] 1. Choose the CL you want 2. NH is null hypothesis:
Define TCNH such that 3. Compute power 4. Quote power for a given T0NH or for all values of T0NH
5. Repeat 2-‐4 for IH as null hypothesis (with a change of sign…) “cri$cal region” is T<TCIH :
NuPhys Dec.19th,2013 A.Tonazzo 10
Typically T0 increases with $me, so power can also be plo�ed vs $me/exposure
Fig.2 of [BCHS]
PDF(T |H0 )dTTCNH
+∞
∫ =CL
p = PDF(T |H1)dT−∞
TCNH
∫
Simple analy$cal formulae if PDF’s are gaussian, otherwise use toy-‐MC
PDF(T |H0 )dT−∞
TCIH
∫ =CL p = PDF(T |H1)dTTCIH
+∞
∫
Dependence on oscilla$on parameters • As pointed out in [BCHS], we are actually dealing with “composite
hypotheses”: T0 depends on parameters (δCP,θ23,…) whose values will not be known at the $me of the experiments
• Need to define Tc’s such that the desired CL is ensured for all values of the parameter(s)
• The power will depend on the value of the parameter(s)
NuPhys Dec.19th,2013 A.Tonazzo 11
Fig.12 of [BCHS]
LBNO poten$al for MH • T0NH and T0IH depend mainly on δCP • From the figures below, one chooses:
for NH TCα(T0NH(δCP=90ο)) for IH TCα(T0IH(δCP=270ο)) • Evaluate power vs exposure for all values of δCP
NuPhys Dec.19th,2013 A.Tonazzo 12
CPδTrue 0 1 2 3 4 5 6
2 χ Δ
0
50
100
150
200
250
300medium value
bandσ1 band σ3
σ3σ5
MH determination (NH assumed)50% nu+50% anu3.0e20 pots
myf
itter
H0=NH H0=IH
T
~2.5 years ~2.5 years
T
LBNO poten$al for MH • Power vs exposure for all values of δCP (shaded bands)
NuPhys Dec.19th,2013 A.Tonazzo 13
Exposure (/1e20 POT)0 1 2 3 4 5
βp
= 1-
0
0.2
0.4
0.6
0.8
1
σ3
σ5Test power for NH
Exposure (/1e20 POT)0 1 2 3 4 5
βp
= 1-
0
0.2
0.4
0.6
0.8
1
σ3
σ5Test power for IH
p = 0.5 => “Median experiment” 50% chance to give the wrong answer if the alterna$ve hypothesis is true
WOULD YOU SPEND 20 YEARS ON SUCH AN EXPERIMENT?
p ~1 => “Full power experiment” ~0 chance to give the wrong answer if the alterna$ve hypothesis is true
THE LBNO CHOICE TO QUOTE
SENSITIVITY
~5 years
The Bayesian approach for MH In the “frequen$st approach” discussed so far, two sets of informa$on must be provided (for NH and for IH). With a Bayesian approach [Blen], a single set of values can contain all the informa$on on the test : • You need a “prior” P on each hypothesis, providing a rela$ve normaliza$on
between the two PDFs • It can be proven [D.F.] that P(NH)=P(IH)=0.5 is the most conserva$ve choice
1) Define a threshold t 2) Compute TcNH such that
and TCIH such that
.
NuPhys Dec.19th,2013 A.Tonazzo 14
PDF(T | NH )PDF(T | NH )+PDF(T | IH )
> t
for T>TCNH
PDF(T | IH )PDF(T | NH )+PDF(T | IH )
> t
for T<TCIH
NOTE !!! we are looking the “Odds” or “ra$o of posterior probabili$es” : IF YOUR EXPERIMENT GIVES A RESULT T, THE PROBABILITY THAT NATURE IS ACTUALLY NH(IH) is >CL . A ques$on that cannot be answered in the frequen$st approach !
The Bayesian approach for MH 3) Define the “p-‐value” as For example, in D.Franco et al, JHEP 1304 (2013) 008 ArXiv 1301.4332 (ORCA/PINGU)
NuPhys Dec.19th,2013 A.Tonazzo 15
• TC’ are set by the ra$o of the heights of the two gaussians at a given value of T
• p-‐value is 0.5* (blue area + red area)
PDF(T | IH )dT +−∞
TCIH
∫ PDF(T | NH )dTTCNH
+∞
∫
3σ CL means t=99.73%
=T
What about δCP ?
NuPhys Dec.19th,2013 A.Tonazzo 16
/ ndf 2χ 40.63 / 24 0A 49.8± 4950
k 0.0116± 0.9635
2χΔ0 5 10 15 20
1
10
210
310
/ ndf 2χ 40.63 / 24 0A 49.8± 4950
k 0.0116± 0.9635
) from toy MC0
|H2χΔf(
distribution fitk2χ
δCP=0,π
/ ndf 2χ 27.53 / 27 0A 62.3± 1405
µ 0.580± 9.152 σ 0.242± 5.708 α 0.1774± 0.6272
2χΔ0 10 20 30 40
A.U
.
0
200
400
600
800
/ ndf 2χ 27.53 / 27 0A 62.3± 1405
µ 0.580± 9.152 σ 0.242± 5.708 α 0.1774± 0.6272
) from toy MC1
|H2χΔf(
skew normal fit
δCP=π/2
LBNO, 15x1020pot
• We are in a case of “nested hypotheses”: H0 = {δCP=0 or π} , H1 = {0<δCP<2π}
• Test sta$s$c Δχ2 = min(Δχ2δCP=0 , Δχ2δCP=π) ⇒ Wilk’s theorem hold ! PDF(Δχ2|H0)=Χ2(1dof) => TC=(# of σ)2
independent of exposure (unlike for MH) • PDF(Δχ2|H1) not easy to predict, can be es$mated with toy-‐MC for each
value of δCP
What about δCP ? <Δx2> is strongly dependent on δCP (the impact of other oscilla$on parameters is much smaller)
Power vs exposure can be computed by integra$ng toyMC distribu$ons, it will be maximal for δCP=π/2,3π/2 and as low as zero for other values
NuPhys Dec.19th,2013 A.Tonazzo 17
CPδTrue 0 1 2 3 4 5 6
2 χ Δ
024
6
8101214
16
1820
90%C.L.
σ3
exclusion with NHπ = 0,CPδ
ν+25% ν75% C2P: 15e20 pots
myf
itter
Exposure (/1e20 POT)0 2 4 6 8 10 12 14 16
βp
= 1-
0
0.2
0.4
0.6
0.8
1 /2π = CPδ
/2π = 3CPδ
σ390%
LBNO Preliminary
<Δχ2>
Summary / Discussion • Animated discussion, recent papers helped clarify the situa$on • The star$ng point: get the PDF’s correct ! • The main point: decide which ques$on you want to answer
• A test of hypotheses is characterised by two values (CL and power), and both must be specified. For MH, two sets of values are necessary, unless a Bayesian approach is chosen.
• For comparison between experiments at a given CL, the chosen value of power is not relevant – as long as it’s the same for all
• For choice of a future project, the LBNO Collabora$on considers ~100% power as an important requisite for inves$ng money and $me in an experiment – LBNO can determine MH at 5σ CL with ~100% power in ~5 years
• A treatment of δCP discovery poten$al with analogous sta$s$cal approach has also been proposed
NuPhys Dec.19th,2013 A.Tonazzo 18