Look elsewhere effect
Ofer VitellsStatistics miniworkshop at CERN , February 2013
LEE Topics• Introduction
– Definition of gaussian & gaussian-related fields
• Z-dependence of trial factor– Variance of m-hat – Bayesian comparison
• Different possibilities for critical region– Constatnt LR (“Tevatron” test statistic) curves– Leadbetter formula
• Location (“energy-scale”) uncertainties– Single channel– Combination
• Approximation/estimation problems– Sliding window effect on upcrossings counting– Uncertainty on observed number of upcrossings (poisson?)– When asymptotic formulae break down in practice
• Gaussian & Gaussian related fields
- The joint distribution of any collection {f(t1),f(t2),…,f(tn)} is multivariate Gaussian- Gaussian related fields are functions of Gaussian fields, e.g.
(chi-squared field)
2 2
1
( ) ( )k
k ii
t f t
t
f(t)Wilks :
2( 0)( ) 2 log
ˆ( , )q
LL
• Z-dependance
0p-value=P(max[ ( )] )q u
22 /2 /2
1 1 1
1( )
2u Z
localP u e p e N N
1ZNTrial-factor
( ( ) )i i in Poiss s m b
21
2
logˆ[ ] ( [ ])
LVar m E
m
222
2 2
( )log 1[ ]
( ( ) )i
i i i
s mLE
m s m b m
ˆ2
1 1 1ˆ[ ] , mVar m
Z
ˆ ( )m
rangeTF
Z
In the large sample limit
Example :
• Variance of : m
( )i ib s
Bayesian estimate
ˆ
1m
m
2
2( 0)2log
ˆ( )
ˆ( )
( 0)Z
q Z
e
LL
LL
( , )mL
There is less posterior probability in the peak as it narrows (~1/Z)
( 0) 1 ( )( ) ( )m
m m
mP Z
m Z Z
m
With a uniform prior:
“Trial factor”
Bayesian estimate
ˆ
1m
m
2
2( 0)2log
ˆ( )
ˆ( )
( 0)Z
q Z
e
LL
LL
( , )mL
There is less posterior probability in the peak as it narrows (~1/Z)
m
Jeffreys Prior:2
ˆ
2 2ˆ
0( , ) det
0 (1)m
m
Cancels the Z dependence
Example normalized likelihoods
0 20 40 60 80 100 120 140 160 180 200-5
-4
-3
-2
-1
0
1
2
3
4
5
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 20 40 60 80 100 120 140 160 180 200-5
-4
-3
-2
-1
0
1
2
3
4
5
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Background
Background + signal
m
Bayesian estimate
m
( , )mL
Integrate m ?
m
what exactly is meant by 'more impressive than observed in data' ?q(PL) vs. qTEV
2 22 ˆˆ ˆ(0)2 log 2
( )SM SM SM
TEVSM
Lq
L
2ˆ(0)
2 logˆ( )
Lq
L
SMSMZ
SM Expected significance (sensitivity):
observed significance : ˆobsZ q
22 obsTEV SM SMq Z Z Z
At a given mass point the two tests are equivalent (1-to-1 functions of ),
But give different answers to what is the “best fit” mass - or
* note qTEV = 0 if ZSM=0. max[qTEV] is generally not at the point of largest local significance.
max[ ( )]mq m max[ ( )]TEV
mq m
9
22 obsTEV SM SMq Z Z Z
SMZ
obsZ
2
2obs TEV SM
SM
q ZZ
Z
10
Curves of constant qTEV
2
2obs TEV SM
SM
q ZZ
Z
p-value = Prob( max[qTEV] > c )
A similar signal at 160 GeV would give much smaller global significance (because less consistent with the SM) - same as local 1σ @ 600 GeV
11
Curves of constant qTEV
2
2obs TEV SM
SM
q ZZ
Z
p-value = Prob( max[qTEV] > c )
Can be estimated with Leadbetter’s formula (upcrossings above a curve)
A similar signal at 160 GeV would give much smaller global significance (because less consistent with the SM) - same as local 1σ @ 600 GeV
12
Where to put the critical region
13
Energy-scale uncertainties
14
Likelihood at a fixed mass M0
Energy-scale nuisance parameter
“local” LEE (Leadbetter)
ATLAS combined Higgs workspace toy sampling at 126.5 GeV with ES uncertainty
Leadbetter formula with a parabolic curve (gaussian constraint)
Similar to a LEE in the range defined by
ES
combination
m1
m2
1 2
2 2
1 0 2 00 1 2
,1 2
( ) max ( , )m m
m m m mq m q m m
21 /2d Zp Z e (trial factor )
dZ
2D field
2
• Sliding windows (mass dependant cuts)==>discontinuity in q(m) due to events getting in/out
0 20 40 60 80 100 1200
10
20
30
40
50
Eve
nts
/ uni
t mas
s
0 20 40 60 80 100 1200
5
m
q(m
)
0 ( )q m
u
0q
q(m)
– Uncertainty on observed number of upcrossings
• Usually assumed Poisson • Effect on significance is logarithmic
– When & how asymptotic formulae break down in practice
• ?
2 /21 1( ) Z
global localp p e N N
Extra slides
Example of combination of channels with different mas resolutions
• Toy combination of two channels:(both gaussian signal + flat bkg)- channel 1: σm=1 GeV- channel 2: σm=10 GeV
Combination example
q0(mH) µ(mH)^
0 10 20 30 40 50 60 70 80 90 100-3
-2
-1
0
1
2
3
combined
channel 1channel 2
0 10 20 30 40 50 60 70 80 90 1000
1
2
3
4
5
6
7
8
9
combined
channel 1channel 2
Note the effect of the wide bump on the number of upcrossings at 1
mH mH
Average number of upcrossings
0 2 4 6 8 10 12 14 16 18 20 2210
-5
10-4
10-3
10-2
10-1
100
101
level
aver
age
num
ber
of u
pcro
ssin
gs
channel 1
channel 2Combined
/20( ) cN c N e
25,000Toy simulations
Note that the average number of upcrossins in the combination is alwayssmaller than in channel 1 alone
23
2-D exapmle #2: resonance search with unknown width
• Gaussian signal on exponential background• Toy model : 0<m<100 , 2<σ<6• Unbinned likelihood:
( ) ( )( | )s s i b s i
s bi s b
N f x N f xPoiss N N N
N N
L
( ) cxbf x ce
0 10 20 30 40 50 60 70 80 90 10010
0
101
102
10 20 30 40 50 60 70 80 902
2.5
3
3.5
4
4.5
5
5.5
60q
σ
m
2
2
( )
2
2
1( ; , )
2
x m
sf x m e
24
2-D exapmle #2: resonance search with unknown width
10 20 30 40 50 60 70 80 902
2.5
3
3.5
4
4.5
5
5.5
6
10 20 30 40 50 60 70 80 902
2.5
3
3.5
4
4.5
5
5.5
6
u=1 u=0
5 10 15 20 25 3010
-6
10-5
10-4
10-3
10-2
10-1
100
P-value0q
2 /21 2
1[ ( )] P( ) ( )
2u
uE A u u e N N
1
2
4 0.2
0.7 0.3
N
N
0 4.5 0.2 1 3 0.16
Excellent approximation above the ~2σ level
(2nd term is dominant for ) 1 2/ 5.7Z N N
20 40 60 80 100 120 140 160 180 200
20
40
60
80
100
120
140
160
180
200
m1
m2
1 2
2 2
1 0 2 00 1 2
,1 2
( ) max ( , )m m
m m m mq m q m m
Asymptotic formulae
To have the distribution well defined for , take , since
e.g. If take , such that constant ( )
In this limit is independent of up to
e.g.
N ' / N 1/ N
b s b /s b / 1/ 0s b b
(1/ )O N
/s s
O s bb s b
Top Related