G. Cowan RHUL Physics page 1 Status of search procedures for ATLAS ATLAS-CMS Joint Statistics...
-
Upload
gabriel-woods -
Category
Documents
-
view
214 -
download
0
Transcript of G. Cowan RHUL Physics page 1 Status of search procedures for ATLAS ATLAS-CMS Joint Statistics...
![Page 1: G. Cowan RHUL Physics page 1 Status of search procedures for ATLAS ATLAS-CMS Joint Statistics Meeting CERN, 15 October, 2009 Glen Cowan Physics Department.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697bfb51a28abf838c9d613/html5/thumbnails/1.jpg)
G. CowanRHUL Physics page 1
Status of search procedures for ATLAS
ATLAS-CMS Joint Statistics Meeting
CERN, 15 October, 2009
Glen CowanPhysics DepartmentRoyal Holloway, University of [email protected]/~cowan
![Page 2: G. Cowan RHUL Physics page 1 Status of search procedures for ATLAS ATLAS-CMS Joint Statistics Meeting CERN, 15 October, 2009 Glen Cowan Physics Department.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697bfb51a28abf838c9d613/html5/thumbnails/2.jpg)
G. Cowan / RHUL page 2
IntroductionThis talk describes (part of) the view from ATLAS with emphasis on searches using profile likelihood-based techniques;using as an example the combination of Higgs channels described in
Expected Performance of the ATLAS Experiment: Detector, Trigger and Physics, arXiv:0901.0512, CERN-OPEN-2008-20.
Other methods are also being explored and the view presentedhere should not be taken as a final word on methodology.
Some issues are:Basic formalism, definition of significanceMethod used for setting limitsLook-elsewhere effect
![Page 3: G. Cowan RHUL Physics page 1 Status of search procedures for ATLAS ATLAS-CMS Joint Statistics Meeting CERN, 15 October, 2009 Glen Cowan Physics Department.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697bfb51a28abf838c9d613/html5/thumbnails/3.jpg)
G. Cowan / RHUL page 3
Search formalism
Define a test variable whose distribution is sensitive to whetherhypothesis is background-only or signal + background.
E.g. count n events in signal region:
events found
expected signal expected background
strength parameter= s/ s,nominal
![Page 4: G. Cowan RHUL Physics page 1 Status of search procedures for ATLAS ATLAS-CMS Joint Statistics Meeting CERN, 15 October, 2009 Glen Cowan Physics Department.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697bfb51a28abf838c9d613/html5/thumbnails/4.jpg)
G. Cowan / RHUL page 4
Search formalism with multiple bins (channels)
Bin i of a given channel has ni events, expectation value is
Expected signal and background are:
is global strength parameter, common to all channels.= 0 means background only, = 1 is nominal signal hypothesis.
btot, s, b arenuisance parameters
![Page 5: G. Cowan RHUL Physics page 1 Status of search procedures for ATLAS ATLAS-CMS Joint Statistics Meeting CERN, 15 October, 2009 Glen Cowan Physics Department.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697bfb51a28abf838c9d613/html5/thumbnails/5.jpg)
G. Cowan / RHUL page 5
Subsidiary measurements for background
One may have a subsidiary measurement to constrain the background based on a control region where one expects no signal.
In bin i of control histogram find mi events; expectation value is
where the ui can be found from MC and includes parametersrelated to the background (mainly rate, sometimes also shape).
In some measurements there may be no explicit subsidiarymeasurement but the sidebands around a signal peak effectivelyplay the same role in constraining the background.
![Page 6: G. Cowan RHUL Physics page 1 Status of search procedures for ATLAS ATLAS-CMS Joint Statistics Meeting CERN, 15 October, 2009 Glen Cowan Physics Department.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697bfb51a28abf838c9d613/html5/thumbnails/6.jpg)
G. Cowan / RHUL page 6
Likelihood functionFor an individual search channel, ni ~ Poisson(si+bi), mi ~ Poisson(ui). The likelihood is:
Parameter of interest
Here represents allnuisance parameters
For multiple independent channels there is a likelihood Li(,i) for each. The full likelihood function is
![Page 7: G. Cowan RHUL Physics page 1 Status of search procedures for ATLAS ATLAS-CMS Joint Statistics Meeting CERN, 15 October, 2009 Glen Cowan Physics Department.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697bfb51a28abf838c9d613/html5/thumbnails/7.jpg)
G. Cowan / RHUL page 7
Systematics "built in" as long as some point in -space = "truth". Presence of nuisance parameters leads to broadening of theprofile likelihood, reflecting the loss of information, and givesappropriately reduced discovery significance, weaker limits.
![Page 8: G. Cowan RHUL Physics page 1 Status of search procedures for ATLAS ATLAS-CMS Joint Statistics Meeting CERN, 15 October, 2009 Glen Cowan Physics Department.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697bfb51a28abf838c9d613/html5/thumbnails/8.jpg)
G. Cowan / RHUL page 8
Modified test statistic for exclusion limits
For e.g. data generated with , -2 ln () can come out large for
If , then data more compatible with a higher value of . sodo not include this in critical region.
For upper limit, test hypothesis that strength parameter is ≥ .
Upper limit is smallest value of where this hypothesis can be rejected at significance level less than 1CL.
Critical region of test is region with less compatibility withthe hypothesis than the observed
![Page 9: G. Cowan RHUL Physics page 1 Status of search procedures for ATLAS ATLAS-CMS Joint Statistics Meeting CERN, 15 October, 2009 Glen Cowan Physics Department.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697bfb51a28abf838c9d613/html5/thumbnails/9.jpg)
G. Cowan / RHUL page 9
Test statistic for exclusion limitsTherefore for exclusion limits, define the test statistic to be
critical region
Thus distribution of modified q corresponds to lower branchonly of U-shaped plot above.
For low , this distribution falls off more quickly than theasymptotic chi-square form and thus gives conservative limit.
![Page 10: G. Cowan RHUL Physics page 1 Status of search procedures for ATLAS ATLAS-CMS Joint Statistics Meeting CERN, 15 October, 2009 Glen Cowan Physics Department.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697bfb51a28abf838c9d613/html5/thumbnails/10.jpg)
G. Cowan / RHUL page 10
p-value / significance of hypothesized
Test hypothesized by givingp-value, probability to see data with ≤ compatibility with compared to data observed:
Equivalently use significance,Z, defined as equivalent numberof sigmas for a Gaussian fluctuation in one direction:
![Page 11: G. Cowan RHUL Physics page 1 Status of search procedures for ATLAS ATLAS-CMS Joint Statistics Meeting CERN, 15 October, 2009 Glen Cowan Physics Department.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697bfb51a28abf838c9d613/html5/thumbnails/11.jpg)
G. Cowan / RHUL page 11
Distribution of qSo to find the p-value we need f(q|) .
Method 1: generate toy MC experiments with hypothesis , obtain at distribution of q.
OK for e.g. ~103 or 104 experiments, 95% CL limits.
But for discovery usually want 5, p-value = , so needto generate ~108 toy experiments (for every point in param. space).
Method 2: Wilk's theorem says that for large enough sample,
f(q|) ~ chi-square(1 dof)
This is the approach used in the ATLAS Higgs Combination exercise; not yet validated to 5 level.
If/when we are fortunate enough to see a signal, then focus MC resources on that point in parameter space.
![Page 12: G. Cowan RHUL Physics page 1 Status of search procedures for ATLAS ATLAS-CMS Joint Statistics Meeting CERN, 15 October, 2009 Glen Cowan Physics Department.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697bfb51a28abf838c9d613/html5/thumbnails/12.jpg)
G. Cowan / RHUL page 12
Example from validation exercise: ZZ(*) → 4l
5 level
(One minus)cumulativedistributions.Band gives 68%CL limits.
½2
½2
½2
½2
Distributions of q0 for 2, 10 fb from MC compared to ½2
![Page 13: G. Cowan RHUL Physics page 1 Status of search procedures for ATLAS ATLAS-CMS Joint Statistics Meeting CERN, 15 October, 2009 Glen Cowan Physics Department.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697bfb51a28abf838c9d613/html5/thumbnails/13.jpg)
G. Cowan / RHUL page 13
Significance from q
If we take f(q|) ~ 2 for 1dof, then the significance is simply:
For n ~ Poisson (s+b) with b known, testing = 0 gives
To quantify sensitivity give e.g. expected Z under s+b hypothesis
![Page 14: G. Cowan RHUL Physics page 1 Status of search procedures for ATLAS ATLAS-CMS Joint Statistics Meeting CERN, 15 October, 2009 Glen Cowan Physics Department.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697bfb51a28abf838c9d613/html5/thumbnails/14.jpg)
G. Cowan / RHUL page 14
SensitivityDiscovery:
Generate data under s+b (= 1) hypothesis;Test hypothesis = 0 → p-value → Z.
Exclusion:Generate data under background-only (= 0) hypothesis;Test hypothesis .If = 1 has p-value < 0.05 exclude mH at 95% CL.
Estimate median significance (sensitivity) either from MC or by using a single data set with observed numbers set equal tothe expectation values ("Asimov" data set).
![Page 15: G. Cowan RHUL Physics page 1 Status of search procedures for ATLAS ATLAS-CMS Joint Statistics Meeting CERN, 15 October, 2009 Glen Cowan Physics Department.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697bfb51a28abf838c9d613/html5/thumbnails/15.jpg)
G. Cowan / RHUL page 15
Example of ATLAS Higgs searchCombination of Higgs search channels (ATLAS) Expected Performance of the ATLAS Experiment: Detector, Trigger and Physics, arXiv:0901.0512, CERN-OPEN-2008-20.
Standard Model Higgs channels considered: H → H → WW (*)→ eH → ZZ(*) → 4l (l = e, ) H → → ll, lh
Not all channels included for now; final sensitivity will improve.
Used profile likelihood method for systematic uncertainties:nuisance parameers for: background rates, signal &
background shapes.
![Page 16: G. Cowan RHUL Physics page 1 Status of search procedures for ATLAS ATLAS-CMS Joint Statistics Meeting CERN, 15 October, 2009 Glen Cowan Physics Department.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697bfb51a28abf838c9d613/html5/thumbnails/16.jpg)
G. Cowan / RHUL page 16
Combined discovery significance
Discovery signficance (in colour) vs. L, mH:
Approximations used here not always accurate for L < 2 fb but in most cases conservative.
![Page 17: G. Cowan RHUL Physics page 1 Status of search procedures for ATLAS ATLAS-CMS Joint Statistics Meeting CERN, 15 October, 2009 Glen Cowan Physics Department.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697bfb51a28abf838c9d613/html5/thumbnails/17.jpg)
G. Cowan / RHUL page 17
Combined 95% CL exclusion limits
p-value of mH (in colour) vs. L, mH:
![Page 18: G. Cowan RHUL Physics page 1 Status of search procedures for ATLAS ATLAS-CMS Joint Statistics Meeting CERN, 15 October, 2009 Glen Cowan Physics Department.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697bfb51a28abf838c9d613/html5/thumbnails/18.jpg)
G. Cowan / RHUL page 18
Comment on combination softwareCurrent ATLAS Higgs combination shows median significances
Obtained using median significances from each channel
What we will need is the significance one would have from asingle (e.g. real) data sample.
Requires full likelihood function, global fit → software.
Since summer 2008 ATLAS/CMS decision to focus joint statisticssoftware effort in RooStats (based on RooFit, ROOT).
Provides facility to construct global likelihood forcombination of channels/experiments
Emphasis on retaining modularity for validation byswapping in/out different components.
![Page 19: G. Cowan RHUL Physics page 1 Status of search procedures for ATLAS ATLAS-CMS Joint Statistics Meeting CERN, 15 October, 2009 Glen Cowan Physics Department.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697bfb51a28abf838c9d613/html5/thumbnails/19.jpg)
G. Cowan / RHUL page 19
The "look-elsewhere effect"Look for Higgs at many mH values -- probability of seeing a large fluctuation for some mH increased.
Combined significance shown here relates to fixed mH.
False discovery prob enhanced by ~ mass region explored / m
For H→ and H→WW, studied by allowing mH to float in fit:
→
![Page 20: G. Cowan RHUL Physics page 1 Status of search procedures for ATLAS ATLAS-CMS Joint Statistics Meeting CERN, 15 October, 2009 Glen Cowan Physics Department.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697bfb51a28abf838c9d613/html5/thumbnails/20.jpg)
G. Cowan / RHUL page 20
Summary / conclusions
Current philosophy (ATLAS/CMS) is to encourage a variety of methods, e.g., for limits: classical (PL ratio), CLs, Bayesian,...
If the results agree, it's an important check of robustness.If the results disagree, we learn something (~ Cousins)
Discussion on look-elsewhere effect still ongoing
Derive trials factor from e.g. MC or other considerationsFloating mass search
D0, CDF, CMS, ATLAS need to compare like with like.
Need discussions on e.g. formalism for discovery,limits, combination, treatment of common systematics,…
![Page 21: G. Cowan RHUL Physics page 1 Status of search procedures for ATLAS ATLAS-CMS Joint Statistics Meeting CERN, 15 October, 2009 Glen Cowan Physics Department.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697bfb51a28abf838c9d613/html5/thumbnails/21.jpg)
G. Cowan / RHUL page 21
Extra slides
![Page 22: G. Cowan RHUL Physics page 1 Status of search procedures for ATLAS ATLAS-CMS Joint Statistics Meeting CERN, 15 October, 2009 Glen Cowan Physics Department.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697bfb51a28abf838c9d613/html5/thumbnails/22.jpg)
G. Cowan / RHUL page 22
Fast Fourier Transform method to find distribution; derivesn-event distribution from that of single event with FFT.
Hu and Nielson, physics/9906010
Solves "5-sigma problem".
Used at LEP -- systematics treated by averaging the likelihoodsby sampling new values of nuisance parameters for each simulated experiment (integrated rather than profile likelihood).
An alternative (in simple cases equivalent) test variable is
Comment on "LEP"-style methods
![Page 23: G. Cowan RHUL Physics page 1 Status of search procedures for ATLAS ATLAS-CMS Joint Statistics Meeting CERN, 15 October, 2009 Glen Cowan Physics Department.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697bfb51a28abf838c9d613/html5/thumbnails/23.jpg)
G. Cowan / RHUL page 23
Setting limits: CLs
Alternative method (from Alex Read at LEP); exclude if
where
This cures the problematic case where the one excludes parameterpoint where one has no sensitivity (e.g. large mass scale)because of a downwards fluctuation of the background.
But there are perhaps other ways to get around this problem,e.g., only exclude if both observed and expected p-value .
![Page 24: G. Cowan RHUL Physics page 1 Status of search procedures for ATLAS ATLAS-CMS Joint Statistics Meeting CERN, 15 October, 2009 Glen Cowan Physics Department.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697bfb51a28abf838c9d613/html5/thumbnails/24.jpg)
G. Cowan / RHUL page 24
![Page 25: G. Cowan RHUL Physics page 1 Status of search procedures for ATLAS ATLAS-CMS Joint Statistics Meeting CERN, 15 October, 2009 Glen Cowan Physics Department.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697bfb51a28abf838c9d613/html5/thumbnails/25.jpg)
G. Cowan / RHUL page 25
Some issuesThe profile likelihood method "includes" systematics to theextent that for some point in the model's parameter space, thedifference from the "truth" is negligible.
Q: What if the model is not good enough?
A: Improve the model, i.e., include additional flexibility (nuisance parameters).
Increased flexibility → decrease in sensitivity.How to achieve optimal balance in a general way is not obvious.
Corresponding exercise in Bayesian approach:
Include nuisance parameters in model with prior probabilities -- also not obvious in many important cases,
e.g., uncertainties in correlations.