Statistical aspects of Higgs analyses

23
Statistical aspects of Higgs analyses W. Verkerke (NIKHEF)

description

Statistical aspects of Higgs analyses. W. Verkerke (NIKHEF). Introduction. Enormous effort to search for Higgs signature in many decay channels Results  many plots with signal, background expectations, each with (systematic) uncertainties, and data - PowerPoint PPT Presentation

Transcript of Statistical aspects of Higgs analyses

Page 1: Statistical aspects of Higgs analyses

Statistical aspects of Higgs analyses

W Verkerke(NIKHEF)

Introductionbull Enormous effort to search for Higgs signature in many

decay channelsbull Results many plots with signal

background expectations each with (systematic) uncertainties and data

bull Q How do you conclude from this that yoursquove seen the Higgs (or not)ndash Want answer of type

lsquoWe can exclude that the Higgs exist at 95 CLrdquo or ldquoProbabilitythat background only caused observedexcess is 310-7

bull Here a short guide through how this is (typically) done

Quantifying discovery and exclusion ndash Frequentist approach

bull Consider the simplest case ndash a counting experimentndash Observable N (the event count)ndash Model F(N|s) Poisson(N|s+b) with b=5 known exactly

bull Predicted distributions of N for various values of s

s=0

s=5

s=10s=15

bull Now make a measurement N=Nobs (example Nobs=7)bull Can now define p-value(s) eg for bkg hypothesis

ndash Fraction of future measurements with N=Nobs (or larger) if s=0

Frequentist p-values ndash excess over expected bkg

)230()0(

obsNb dNbNPoissonp

s=0

s=5s=10

s=15

bull p-values of background hypothesis is used to quantify lsquodiscoveryrsquo = excess of events over background expectation

bull Another example Nobs=15 for same model what is pb

ndash Result customarily re-expressed as odds of a Gaussian fluctuation with equal p-value (35 sigma for above case)

ndash NB Nobs=22 gives pb lt 2810-7 (lsquo5 sigmarsquo)

Frequentist p-values - excess over expected bkg

)000220()0(

obsNb dNbNPoissonp

s=0

s=5s=10

s=15

Upper limits (one-sided confidence intervals)bull Can also defined p-values for hypothesis with signal ps+b

ndash Note convention integration range in ps+b is flipped

bull Convention express result as value of s for which p(s+b)=5 ldquosgt68 is excluded at 95 CLrdquo

Wouter Verkerke NIKHEF

obsN

bs dNsbNPoissonp )(

p(s=15) = 000025p(s=10) = 0007p(s=5) = 013

p(s=68) = 005

Modified frequentist upper limitsbull Need to be careful about interpretation p(s+b) in terms

of inference on signal onlyndash Since p(s+b) quantifies consistency of signal plus backgroundndash Problem most apparent when observed data

has downward stat fluctations wrt background expectationbull Example Nobs =2

bull Modified approach to protectagainst such inference on sndash Instead of requiring p(s+b)=5

require

ps+b(s=0) = 004

sge0 excluded at gt95 CL s=0

s=5

s=10s=15

51

b

bsS p

pCL

for N=2 exclude sgt34 at 95 CLs for large N effect on limit is small as pb0

p-values and limits on non-trivial analysis

bull Typical Higgs search result is not a simple number counting experiment but looks like this

bull Any type of result can be converted into a single number by constructing a lsquotest statisticrsquo ndash A test statistic compresses all signal-to-background

discrimination power in a single numberndash Most powerful discriminators

are Likelihood Ratios(Neyman Pearson)

)ˆ|()|(ln2

dataLdataLq

- Result is a distribution not a single number

- Models for signal and background have intrinsic uncertainties

The likelihood ratio test statistic

bull Definition μ = signal strength signal strength(SM)ndash Choose eg likelihood with nominal signal strength in numerator (μ=1)

bull Illustration on model with no shape uncertainties

)ˆ|()1|(ln21

dataLdataLq

μ is best fit value of μ^

lsquolikelihood of best fitrsquo

lsquolikelihood assuming nominal signal strengthrsquo

770~1 q 3452~

1 q

On signal-like data q1 is small On background-like data q1 is large

Distributions of test statisticsbull Value of q1 on data is now the lsquomeasurementrsquobull Distribution of q1 not calculable

But can obtain distribution from pseudo-experimentsbull Generate a large number of pseudo-experiments with a given value of mu

calculate q for each plot distribution

bull From qobs and these distributions can then set limitssimilar to what was shown for Poisson counting example

)ˆ|()1|(ln2~

1

dataLdataLq

q1 for experiments with signal

q1 for experiments with background only

Note analogyto Poisson

counting example

Incorporating systematic uncertaintiesbull What happens if models have uncertainties

ndash Introduction of additional model parameters θ that describe effect of uncertainties

)|()|(

dataLdataL

)~())()(|()|( pbsNPoissondataL iii

xx θ2θ1

Jet Energy ScaleQCD scaleluminosity

Incorporating systematic uncertainties

)~())()(|()|( pbsNPoissondataL iii

xx θ2θ1

Likelihood includes auxiliary measurement terms that constrains the nuisance parameters θ(shape is flat log-normal gamma or Gaussian)

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are

incorporated in test statistic using a profile likelihood ratio

bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments

ˆ0(with a constraint )

lsquolikelihood of best fitrsquo

lsquolikelihood of best fit for a given fixed value of μrsquo

)ˆ|()|(ln2

dataLdataLq

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are

incorporated using a profile likelihood ratio test statistic

bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments

ˆ0(with a constraint )lsquolikelihood of best fitrsquo

lsquolikelihood of best fit for a given fixed value of μrsquo

μ=035 μ=10 μ=18 μ=26

Putting it all together for one Higgs channelbull Result from data D(x)bull PDF F(x|mHμθ) that models

the data for a given true Higgs mass hypothesis

bull Construct test statistic

bull Obtain expected distributions of qμ for various μndash Determine lsquodiscoveryrsquo p-value

and signal exclusion limitbull Repeat for each assumed mH

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ|(

ln2)(~

H

HH mdataL

mdataLmq

Example ndash 95 Exclusion limit vs mH for HWWExample point asymp3 x SM HWW cross-section excluded at mH=125 GeV

Example point asymp05 x SM HWW cross-section excluded at mH=165 GeV

Higgs with 10x SM cross-section excluded at 95 CL for mH in range [150~187]

Expected exclusion limitfor background-only hypothesis

Combining Higgs channels (and experiments)bull Procedure define joint likelihood

bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration

bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

)()()( CMSCMSATLASATLASLHC LLL

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)

have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and

delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

lsquollll110rootrsquo lsquogg110rootrsquo

Example ndash joint ATLASCMS Higgs exclusion limit

Wouter Verkerke NIKHEF

Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ0|(ln2~ 0

0

dataLdataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming background onlyrsquo

)ˆˆ|()ˆ|(

ln2~

dataL

dataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming μ signal strengthrsquo

770~1 q

lsquodiscoveryrsquolsquoexclusionrsquo

7734~0 q

0~0 q3452~

1 q

simulateddata with signal+bkg

simulateddata with bkg only

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)

Note that lsquopeakrsquo around 160 GeV reflects increased

experimental sensitivity not SM prediction of Higgs mass

Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)

But need to be careful with local p-values

Search is executed for a wide mH mass range

Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1

Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction

Conclusionsbull We are early awaiting more data

Wouter Verkerke NIKHEF

  • Statistical aspects of Higgs analyses
  • Introduction
  • Quantifying discovery and exclusion ndash Frequentist approach
  • Frequentist p-values ndash excess over expected bkg
  • Frequentist p-values - excess over expected bkg
  • Upper limits (one-sided confidence intervals)
  • Modified frequentist upper limits
  • p-values and limits on non-trivial analysis
  • The likelihood ratio test statistic
  • Distributions of test statistics
  • Incorporating systematic uncertainties
  • Incorporating systematic uncertainties (2)
  • Dealing with nuisance parameters in the test statistic
  • Dealing with nuisance parameters in the test statistic (2)
  • Putting it all together for one Higgs channel
  • Example ndash 95 Exclusion limit vs mH for HWW
  • Combining Higgs channels (and experiments)
  • A word on the machinery
  • Example ndash joint ATLASCMS Higgs exclusion limit
  • Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
  • Conclusions
Page 2: Statistical aspects of Higgs analyses

Introductionbull Enormous effort to search for Higgs signature in many

decay channelsbull Results many plots with signal

background expectations each with (systematic) uncertainties and data

bull Q How do you conclude from this that yoursquove seen the Higgs (or not)ndash Want answer of type

lsquoWe can exclude that the Higgs exist at 95 CLrdquo or ldquoProbabilitythat background only caused observedexcess is 310-7

bull Here a short guide through how this is (typically) done

Quantifying discovery and exclusion ndash Frequentist approach

bull Consider the simplest case ndash a counting experimentndash Observable N (the event count)ndash Model F(N|s) Poisson(N|s+b) with b=5 known exactly

bull Predicted distributions of N for various values of s

s=0

s=5

s=10s=15

bull Now make a measurement N=Nobs (example Nobs=7)bull Can now define p-value(s) eg for bkg hypothesis

ndash Fraction of future measurements with N=Nobs (or larger) if s=0

Frequentist p-values ndash excess over expected bkg

)230()0(

obsNb dNbNPoissonp

s=0

s=5s=10

s=15

bull p-values of background hypothesis is used to quantify lsquodiscoveryrsquo = excess of events over background expectation

bull Another example Nobs=15 for same model what is pb

ndash Result customarily re-expressed as odds of a Gaussian fluctuation with equal p-value (35 sigma for above case)

ndash NB Nobs=22 gives pb lt 2810-7 (lsquo5 sigmarsquo)

Frequentist p-values - excess over expected bkg

)000220()0(

obsNb dNbNPoissonp

s=0

s=5s=10

s=15

Upper limits (one-sided confidence intervals)bull Can also defined p-values for hypothesis with signal ps+b

ndash Note convention integration range in ps+b is flipped

bull Convention express result as value of s for which p(s+b)=5 ldquosgt68 is excluded at 95 CLrdquo

Wouter Verkerke NIKHEF

obsN

bs dNsbNPoissonp )(

p(s=15) = 000025p(s=10) = 0007p(s=5) = 013

p(s=68) = 005

Modified frequentist upper limitsbull Need to be careful about interpretation p(s+b) in terms

of inference on signal onlyndash Since p(s+b) quantifies consistency of signal plus backgroundndash Problem most apparent when observed data

has downward stat fluctations wrt background expectationbull Example Nobs =2

bull Modified approach to protectagainst such inference on sndash Instead of requiring p(s+b)=5

require

ps+b(s=0) = 004

sge0 excluded at gt95 CL s=0

s=5

s=10s=15

51

b

bsS p

pCL

for N=2 exclude sgt34 at 95 CLs for large N effect on limit is small as pb0

p-values and limits on non-trivial analysis

bull Typical Higgs search result is not a simple number counting experiment but looks like this

bull Any type of result can be converted into a single number by constructing a lsquotest statisticrsquo ndash A test statistic compresses all signal-to-background

discrimination power in a single numberndash Most powerful discriminators

are Likelihood Ratios(Neyman Pearson)

)ˆ|()|(ln2

dataLdataLq

- Result is a distribution not a single number

- Models for signal and background have intrinsic uncertainties

The likelihood ratio test statistic

bull Definition μ = signal strength signal strength(SM)ndash Choose eg likelihood with nominal signal strength in numerator (μ=1)

bull Illustration on model with no shape uncertainties

)ˆ|()1|(ln21

dataLdataLq

μ is best fit value of μ^

lsquolikelihood of best fitrsquo

lsquolikelihood assuming nominal signal strengthrsquo

770~1 q 3452~

1 q

On signal-like data q1 is small On background-like data q1 is large

Distributions of test statisticsbull Value of q1 on data is now the lsquomeasurementrsquobull Distribution of q1 not calculable

But can obtain distribution from pseudo-experimentsbull Generate a large number of pseudo-experiments with a given value of mu

calculate q for each plot distribution

bull From qobs and these distributions can then set limitssimilar to what was shown for Poisson counting example

)ˆ|()1|(ln2~

1

dataLdataLq

q1 for experiments with signal

q1 for experiments with background only

Note analogyto Poisson

counting example

Incorporating systematic uncertaintiesbull What happens if models have uncertainties

ndash Introduction of additional model parameters θ that describe effect of uncertainties

)|()|(

dataLdataL

)~())()(|()|( pbsNPoissondataL iii

xx θ2θ1

Jet Energy ScaleQCD scaleluminosity

Incorporating systematic uncertainties

)~())()(|()|( pbsNPoissondataL iii

xx θ2θ1

Likelihood includes auxiliary measurement terms that constrains the nuisance parameters θ(shape is flat log-normal gamma or Gaussian)

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are

incorporated in test statistic using a profile likelihood ratio

bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments

ˆ0(with a constraint )

lsquolikelihood of best fitrsquo

lsquolikelihood of best fit for a given fixed value of μrsquo

)ˆ|()|(ln2

dataLdataLq

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are

incorporated using a profile likelihood ratio test statistic

bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments

ˆ0(with a constraint )lsquolikelihood of best fitrsquo

lsquolikelihood of best fit for a given fixed value of μrsquo

μ=035 μ=10 μ=18 μ=26

Putting it all together for one Higgs channelbull Result from data D(x)bull PDF F(x|mHμθ) that models

the data for a given true Higgs mass hypothesis

bull Construct test statistic

bull Obtain expected distributions of qμ for various μndash Determine lsquodiscoveryrsquo p-value

and signal exclusion limitbull Repeat for each assumed mH

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ|(

ln2)(~

H

HH mdataL

mdataLmq

Example ndash 95 Exclusion limit vs mH for HWWExample point asymp3 x SM HWW cross-section excluded at mH=125 GeV

Example point asymp05 x SM HWW cross-section excluded at mH=165 GeV

Higgs with 10x SM cross-section excluded at 95 CL for mH in range [150~187]

Expected exclusion limitfor background-only hypothesis

Combining Higgs channels (and experiments)bull Procedure define joint likelihood

bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration

bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

)()()( CMSCMSATLASATLASLHC LLL

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)

have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and

delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

lsquollll110rootrsquo lsquogg110rootrsquo

Example ndash joint ATLASCMS Higgs exclusion limit

Wouter Verkerke NIKHEF

Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ0|(ln2~ 0

0

dataLdataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming background onlyrsquo

)ˆˆ|()ˆ|(

ln2~

dataL

dataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming μ signal strengthrsquo

770~1 q

lsquodiscoveryrsquolsquoexclusionrsquo

7734~0 q

0~0 q3452~

1 q

simulateddata with signal+bkg

simulateddata with bkg only

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)

Note that lsquopeakrsquo around 160 GeV reflects increased

experimental sensitivity not SM prediction of Higgs mass

Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)

But need to be careful with local p-values

Search is executed for a wide mH mass range

Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1

Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction

Conclusionsbull We are early awaiting more data

Wouter Verkerke NIKHEF

  • Statistical aspects of Higgs analyses
  • Introduction
  • Quantifying discovery and exclusion ndash Frequentist approach
  • Frequentist p-values ndash excess over expected bkg
  • Frequentist p-values - excess over expected bkg
  • Upper limits (one-sided confidence intervals)
  • Modified frequentist upper limits
  • p-values and limits on non-trivial analysis
  • The likelihood ratio test statistic
  • Distributions of test statistics
  • Incorporating systematic uncertainties
  • Incorporating systematic uncertainties (2)
  • Dealing with nuisance parameters in the test statistic
  • Dealing with nuisance parameters in the test statistic (2)
  • Putting it all together for one Higgs channel
  • Example ndash 95 Exclusion limit vs mH for HWW
  • Combining Higgs channels (and experiments)
  • A word on the machinery
  • Example ndash joint ATLASCMS Higgs exclusion limit
  • Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
  • Conclusions
Page 3: Statistical aspects of Higgs analyses

Quantifying discovery and exclusion ndash Frequentist approach

bull Consider the simplest case ndash a counting experimentndash Observable N (the event count)ndash Model F(N|s) Poisson(N|s+b) with b=5 known exactly

bull Predicted distributions of N for various values of s

s=0

s=5

s=10s=15

bull Now make a measurement N=Nobs (example Nobs=7)bull Can now define p-value(s) eg for bkg hypothesis

ndash Fraction of future measurements with N=Nobs (or larger) if s=0

Frequentist p-values ndash excess over expected bkg

)230()0(

obsNb dNbNPoissonp

s=0

s=5s=10

s=15

bull p-values of background hypothesis is used to quantify lsquodiscoveryrsquo = excess of events over background expectation

bull Another example Nobs=15 for same model what is pb

ndash Result customarily re-expressed as odds of a Gaussian fluctuation with equal p-value (35 sigma for above case)

ndash NB Nobs=22 gives pb lt 2810-7 (lsquo5 sigmarsquo)

Frequentist p-values - excess over expected bkg

)000220()0(

obsNb dNbNPoissonp

s=0

s=5s=10

s=15

Upper limits (one-sided confidence intervals)bull Can also defined p-values for hypothesis with signal ps+b

ndash Note convention integration range in ps+b is flipped

bull Convention express result as value of s for which p(s+b)=5 ldquosgt68 is excluded at 95 CLrdquo

Wouter Verkerke NIKHEF

obsN

bs dNsbNPoissonp )(

p(s=15) = 000025p(s=10) = 0007p(s=5) = 013

p(s=68) = 005

Modified frequentist upper limitsbull Need to be careful about interpretation p(s+b) in terms

of inference on signal onlyndash Since p(s+b) quantifies consistency of signal plus backgroundndash Problem most apparent when observed data

has downward stat fluctations wrt background expectationbull Example Nobs =2

bull Modified approach to protectagainst such inference on sndash Instead of requiring p(s+b)=5

require

ps+b(s=0) = 004

sge0 excluded at gt95 CL s=0

s=5

s=10s=15

51

b

bsS p

pCL

for N=2 exclude sgt34 at 95 CLs for large N effect on limit is small as pb0

p-values and limits on non-trivial analysis

bull Typical Higgs search result is not a simple number counting experiment but looks like this

bull Any type of result can be converted into a single number by constructing a lsquotest statisticrsquo ndash A test statistic compresses all signal-to-background

discrimination power in a single numberndash Most powerful discriminators

are Likelihood Ratios(Neyman Pearson)

)ˆ|()|(ln2

dataLdataLq

- Result is a distribution not a single number

- Models for signal and background have intrinsic uncertainties

The likelihood ratio test statistic

bull Definition μ = signal strength signal strength(SM)ndash Choose eg likelihood with nominal signal strength in numerator (μ=1)

bull Illustration on model with no shape uncertainties

)ˆ|()1|(ln21

dataLdataLq

μ is best fit value of μ^

lsquolikelihood of best fitrsquo

lsquolikelihood assuming nominal signal strengthrsquo

770~1 q 3452~

1 q

On signal-like data q1 is small On background-like data q1 is large

Distributions of test statisticsbull Value of q1 on data is now the lsquomeasurementrsquobull Distribution of q1 not calculable

But can obtain distribution from pseudo-experimentsbull Generate a large number of pseudo-experiments with a given value of mu

calculate q for each plot distribution

bull From qobs and these distributions can then set limitssimilar to what was shown for Poisson counting example

)ˆ|()1|(ln2~

1

dataLdataLq

q1 for experiments with signal

q1 for experiments with background only

Note analogyto Poisson

counting example

Incorporating systematic uncertaintiesbull What happens if models have uncertainties

ndash Introduction of additional model parameters θ that describe effect of uncertainties

)|()|(

dataLdataL

)~())()(|()|( pbsNPoissondataL iii

xx θ2θ1

Jet Energy ScaleQCD scaleluminosity

Incorporating systematic uncertainties

)~())()(|()|( pbsNPoissondataL iii

xx θ2θ1

Likelihood includes auxiliary measurement terms that constrains the nuisance parameters θ(shape is flat log-normal gamma or Gaussian)

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are

incorporated in test statistic using a profile likelihood ratio

bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments

ˆ0(with a constraint )

lsquolikelihood of best fitrsquo

lsquolikelihood of best fit for a given fixed value of μrsquo

)ˆ|()|(ln2

dataLdataLq

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are

incorporated using a profile likelihood ratio test statistic

bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments

ˆ0(with a constraint )lsquolikelihood of best fitrsquo

lsquolikelihood of best fit for a given fixed value of μrsquo

μ=035 μ=10 μ=18 μ=26

Putting it all together for one Higgs channelbull Result from data D(x)bull PDF F(x|mHμθ) that models

the data for a given true Higgs mass hypothesis

bull Construct test statistic

bull Obtain expected distributions of qμ for various μndash Determine lsquodiscoveryrsquo p-value

and signal exclusion limitbull Repeat for each assumed mH

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ|(

ln2)(~

H

HH mdataL

mdataLmq

Example ndash 95 Exclusion limit vs mH for HWWExample point asymp3 x SM HWW cross-section excluded at mH=125 GeV

Example point asymp05 x SM HWW cross-section excluded at mH=165 GeV

Higgs with 10x SM cross-section excluded at 95 CL for mH in range [150~187]

Expected exclusion limitfor background-only hypothesis

Combining Higgs channels (and experiments)bull Procedure define joint likelihood

bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration

bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

)()()( CMSCMSATLASATLASLHC LLL

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)

have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and

delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

lsquollll110rootrsquo lsquogg110rootrsquo

Example ndash joint ATLASCMS Higgs exclusion limit

Wouter Verkerke NIKHEF

Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ0|(ln2~ 0

0

dataLdataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming background onlyrsquo

)ˆˆ|()ˆ|(

ln2~

dataL

dataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming μ signal strengthrsquo

770~1 q

lsquodiscoveryrsquolsquoexclusionrsquo

7734~0 q

0~0 q3452~

1 q

simulateddata with signal+bkg

simulateddata with bkg only

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)

Note that lsquopeakrsquo around 160 GeV reflects increased

experimental sensitivity not SM prediction of Higgs mass

Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)

But need to be careful with local p-values

Search is executed for a wide mH mass range

Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1

Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction

Conclusionsbull We are early awaiting more data

Wouter Verkerke NIKHEF

  • Statistical aspects of Higgs analyses
  • Introduction
  • Quantifying discovery and exclusion ndash Frequentist approach
  • Frequentist p-values ndash excess over expected bkg
  • Frequentist p-values - excess over expected bkg
  • Upper limits (one-sided confidence intervals)
  • Modified frequentist upper limits
  • p-values and limits on non-trivial analysis
  • The likelihood ratio test statistic
  • Distributions of test statistics
  • Incorporating systematic uncertainties
  • Incorporating systematic uncertainties (2)
  • Dealing with nuisance parameters in the test statistic
  • Dealing with nuisance parameters in the test statistic (2)
  • Putting it all together for one Higgs channel
  • Example ndash 95 Exclusion limit vs mH for HWW
  • Combining Higgs channels (and experiments)
  • A word on the machinery
  • Example ndash joint ATLASCMS Higgs exclusion limit
  • Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
  • Conclusions
Page 4: Statistical aspects of Higgs analyses

bull Now make a measurement N=Nobs (example Nobs=7)bull Can now define p-value(s) eg for bkg hypothesis

ndash Fraction of future measurements with N=Nobs (or larger) if s=0

Frequentist p-values ndash excess over expected bkg

)230()0(

obsNb dNbNPoissonp

s=0

s=5s=10

s=15

bull p-values of background hypothesis is used to quantify lsquodiscoveryrsquo = excess of events over background expectation

bull Another example Nobs=15 for same model what is pb

ndash Result customarily re-expressed as odds of a Gaussian fluctuation with equal p-value (35 sigma for above case)

ndash NB Nobs=22 gives pb lt 2810-7 (lsquo5 sigmarsquo)

Frequentist p-values - excess over expected bkg

)000220()0(

obsNb dNbNPoissonp

s=0

s=5s=10

s=15

Upper limits (one-sided confidence intervals)bull Can also defined p-values for hypothesis with signal ps+b

ndash Note convention integration range in ps+b is flipped

bull Convention express result as value of s for which p(s+b)=5 ldquosgt68 is excluded at 95 CLrdquo

Wouter Verkerke NIKHEF

obsN

bs dNsbNPoissonp )(

p(s=15) = 000025p(s=10) = 0007p(s=5) = 013

p(s=68) = 005

Modified frequentist upper limitsbull Need to be careful about interpretation p(s+b) in terms

of inference on signal onlyndash Since p(s+b) quantifies consistency of signal plus backgroundndash Problem most apparent when observed data

has downward stat fluctations wrt background expectationbull Example Nobs =2

bull Modified approach to protectagainst such inference on sndash Instead of requiring p(s+b)=5

require

ps+b(s=0) = 004

sge0 excluded at gt95 CL s=0

s=5

s=10s=15

51

b

bsS p

pCL

for N=2 exclude sgt34 at 95 CLs for large N effect on limit is small as pb0

p-values and limits on non-trivial analysis

bull Typical Higgs search result is not a simple number counting experiment but looks like this

bull Any type of result can be converted into a single number by constructing a lsquotest statisticrsquo ndash A test statistic compresses all signal-to-background

discrimination power in a single numberndash Most powerful discriminators

are Likelihood Ratios(Neyman Pearson)

)ˆ|()|(ln2

dataLdataLq

- Result is a distribution not a single number

- Models for signal and background have intrinsic uncertainties

The likelihood ratio test statistic

bull Definition μ = signal strength signal strength(SM)ndash Choose eg likelihood with nominal signal strength in numerator (μ=1)

bull Illustration on model with no shape uncertainties

)ˆ|()1|(ln21

dataLdataLq

μ is best fit value of μ^

lsquolikelihood of best fitrsquo

lsquolikelihood assuming nominal signal strengthrsquo

770~1 q 3452~

1 q

On signal-like data q1 is small On background-like data q1 is large

Distributions of test statisticsbull Value of q1 on data is now the lsquomeasurementrsquobull Distribution of q1 not calculable

But can obtain distribution from pseudo-experimentsbull Generate a large number of pseudo-experiments with a given value of mu

calculate q for each plot distribution

bull From qobs and these distributions can then set limitssimilar to what was shown for Poisson counting example

)ˆ|()1|(ln2~

1

dataLdataLq

q1 for experiments with signal

q1 for experiments with background only

Note analogyto Poisson

counting example

Incorporating systematic uncertaintiesbull What happens if models have uncertainties

ndash Introduction of additional model parameters θ that describe effect of uncertainties

)|()|(

dataLdataL

)~())()(|()|( pbsNPoissondataL iii

xx θ2θ1

Jet Energy ScaleQCD scaleluminosity

Incorporating systematic uncertainties

)~())()(|()|( pbsNPoissondataL iii

xx θ2θ1

Likelihood includes auxiliary measurement terms that constrains the nuisance parameters θ(shape is flat log-normal gamma or Gaussian)

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are

incorporated in test statistic using a profile likelihood ratio

bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments

ˆ0(with a constraint )

lsquolikelihood of best fitrsquo

lsquolikelihood of best fit for a given fixed value of μrsquo

)ˆ|()|(ln2

dataLdataLq

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are

incorporated using a profile likelihood ratio test statistic

bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments

ˆ0(with a constraint )lsquolikelihood of best fitrsquo

lsquolikelihood of best fit for a given fixed value of μrsquo

μ=035 μ=10 μ=18 μ=26

Putting it all together for one Higgs channelbull Result from data D(x)bull PDF F(x|mHμθ) that models

the data for a given true Higgs mass hypothesis

bull Construct test statistic

bull Obtain expected distributions of qμ for various μndash Determine lsquodiscoveryrsquo p-value

and signal exclusion limitbull Repeat for each assumed mH

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ|(

ln2)(~

H

HH mdataL

mdataLmq

Example ndash 95 Exclusion limit vs mH for HWWExample point asymp3 x SM HWW cross-section excluded at mH=125 GeV

Example point asymp05 x SM HWW cross-section excluded at mH=165 GeV

Higgs with 10x SM cross-section excluded at 95 CL for mH in range [150~187]

Expected exclusion limitfor background-only hypothesis

Combining Higgs channels (and experiments)bull Procedure define joint likelihood

bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration

bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

)()()( CMSCMSATLASATLASLHC LLL

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)

have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and

delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

lsquollll110rootrsquo lsquogg110rootrsquo

Example ndash joint ATLASCMS Higgs exclusion limit

Wouter Verkerke NIKHEF

Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ0|(ln2~ 0

0

dataLdataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming background onlyrsquo

)ˆˆ|()ˆ|(

ln2~

dataL

dataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming μ signal strengthrsquo

770~1 q

lsquodiscoveryrsquolsquoexclusionrsquo

7734~0 q

0~0 q3452~

1 q

simulateddata with signal+bkg

simulateddata with bkg only

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)

Note that lsquopeakrsquo around 160 GeV reflects increased

experimental sensitivity not SM prediction of Higgs mass

Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)

But need to be careful with local p-values

Search is executed for a wide mH mass range

Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1

Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction

Conclusionsbull We are early awaiting more data

Wouter Verkerke NIKHEF

  • Statistical aspects of Higgs analyses
  • Introduction
  • Quantifying discovery and exclusion ndash Frequentist approach
  • Frequentist p-values ndash excess over expected bkg
  • Frequentist p-values - excess over expected bkg
  • Upper limits (one-sided confidence intervals)
  • Modified frequentist upper limits
  • p-values and limits on non-trivial analysis
  • The likelihood ratio test statistic
  • Distributions of test statistics
  • Incorporating systematic uncertainties
  • Incorporating systematic uncertainties (2)
  • Dealing with nuisance parameters in the test statistic
  • Dealing with nuisance parameters in the test statistic (2)
  • Putting it all together for one Higgs channel
  • Example ndash 95 Exclusion limit vs mH for HWW
  • Combining Higgs channels (and experiments)
  • A word on the machinery
  • Example ndash joint ATLASCMS Higgs exclusion limit
  • Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
  • Conclusions
Page 5: Statistical aspects of Higgs analyses

bull p-values of background hypothesis is used to quantify lsquodiscoveryrsquo = excess of events over background expectation

bull Another example Nobs=15 for same model what is pb

ndash Result customarily re-expressed as odds of a Gaussian fluctuation with equal p-value (35 sigma for above case)

ndash NB Nobs=22 gives pb lt 2810-7 (lsquo5 sigmarsquo)

Frequentist p-values - excess over expected bkg

)000220()0(

obsNb dNbNPoissonp

s=0

s=5s=10

s=15

Upper limits (one-sided confidence intervals)bull Can also defined p-values for hypothesis with signal ps+b

ndash Note convention integration range in ps+b is flipped

bull Convention express result as value of s for which p(s+b)=5 ldquosgt68 is excluded at 95 CLrdquo

Wouter Verkerke NIKHEF

obsN

bs dNsbNPoissonp )(

p(s=15) = 000025p(s=10) = 0007p(s=5) = 013

p(s=68) = 005

Modified frequentist upper limitsbull Need to be careful about interpretation p(s+b) in terms

of inference on signal onlyndash Since p(s+b) quantifies consistency of signal plus backgroundndash Problem most apparent when observed data

has downward stat fluctations wrt background expectationbull Example Nobs =2

bull Modified approach to protectagainst such inference on sndash Instead of requiring p(s+b)=5

require

ps+b(s=0) = 004

sge0 excluded at gt95 CL s=0

s=5

s=10s=15

51

b

bsS p

pCL

for N=2 exclude sgt34 at 95 CLs for large N effect on limit is small as pb0

p-values and limits on non-trivial analysis

bull Typical Higgs search result is not a simple number counting experiment but looks like this

bull Any type of result can be converted into a single number by constructing a lsquotest statisticrsquo ndash A test statistic compresses all signal-to-background

discrimination power in a single numberndash Most powerful discriminators

are Likelihood Ratios(Neyman Pearson)

)ˆ|()|(ln2

dataLdataLq

- Result is a distribution not a single number

- Models for signal and background have intrinsic uncertainties

The likelihood ratio test statistic

bull Definition μ = signal strength signal strength(SM)ndash Choose eg likelihood with nominal signal strength in numerator (μ=1)

bull Illustration on model with no shape uncertainties

)ˆ|()1|(ln21

dataLdataLq

μ is best fit value of μ^

lsquolikelihood of best fitrsquo

lsquolikelihood assuming nominal signal strengthrsquo

770~1 q 3452~

1 q

On signal-like data q1 is small On background-like data q1 is large

Distributions of test statisticsbull Value of q1 on data is now the lsquomeasurementrsquobull Distribution of q1 not calculable

But can obtain distribution from pseudo-experimentsbull Generate a large number of pseudo-experiments with a given value of mu

calculate q for each plot distribution

bull From qobs and these distributions can then set limitssimilar to what was shown for Poisson counting example

)ˆ|()1|(ln2~

1

dataLdataLq

q1 for experiments with signal

q1 for experiments with background only

Note analogyto Poisson

counting example

Incorporating systematic uncertaintiesbull What happens if models have uncertainties

ndash Introduction of additional model parameters θ that describe effect of uncertainties

)|()|(

dataLdataL

)~())()(|()|( pbsNPoissondataL iii

xx θ2θ1

Jet Energy ScaleQCD scaleluminosity

Incorporating systematic uncertainties

)~())()(|()|( pbsNPoissondataL iii

xx θ2θ1

Likelihood includes auxiliary measurement terms that constrains the nuisance parameters θ(shape is flat log-normal gamma or Gaussian)

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are

incorporated in test statistic using a profile likelihood ratio

bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments

ˆ0(with a constraint )

lsquolikelihood of best fitrsquo

lsquolikelihood of best fit for a given fixed value of μrsquo

)ˆ|()|(ln2

dataLdataLq

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are

incorporated using a profile likelihood ratio test statistic

bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments

ˆ0(with a constraint )lsquolikelihood of best fitrsquo

lsquolikelihood of best fit for a given fixed value of μrsquo

μ=035 μ=10 μ=18 μ=26

Putting it all together for one Higgs channelbull Result from data D(x)bull PDF F(x|mHμθ) that models

the data for a given true Higgs mass hypothesis

bull Construct test statistic

bull Obtain expected distributions of qμ for various μndash Determine lsquodiscoveryrsquo p-value

and signal exclusion limitbull Repeat for each assumed mH

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ|(

ln2)(~

H

HH mdataL

mdataLmq

Example ndash 95 Exclusion limit vs mH for HWWExample point asymp3 x SM HWW cross-section excluded at mH=125 GeV

Example point asymp05 x SM HWW cross-section excluded at mH=165 GeV

Higgs with 10x SM cross-section excluded at 95 CL for mH in range [150~187]

Expected exclusion limitfor background-only hypothesis

Combining Higgs channels (and experiments)bull Procedure define joint likelihood

bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration

bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

)()()( CMSCMSATLASATLASLHC LLL

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)

have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and

delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

lsquollll110rootrsquo lsquogg110rootrsquo

Example ndash joint ATLASCMS Higgs exclusion limit

Wouter Verkerke NIKHEF

Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ0|(ln2~ 0

0

dataLdataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming background onlyrsquo

)ˆˆ|()ˆ|(

ln2~

dataL

dataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming μ signal strengthrsquo

770~1 q

lsquodiscoveryrsquolsquoexclusionrsquo

7734~0 q

0~0 q3452~

1 q

simulateddata with signal+bkg

simulateddata with bkg only

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)

Note that lsquopeakrsquo around 160 GeV reflects increased

experimental sensitivity not SM prediction of Higgs mass

Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)

But need to be careful with local p-values

Search is executed for a wide mH mass range

Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1

Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction

Conclusionsbull We are early awaiting more data

Wouter Verkerke NIKHEF

  • Statistical aspects of Higgs analyses
  • Introduction
  • Quantifying discovery and exclusion ndash Frequentist approach
  • Frequentist p-values ndash excess over expected bkg
  • Frequentist p-values - excess over expected bkg
  • Upper limits (one-sided confidence intervals)
  • Modified frequentist upper limits
  • p-values and limits on non-trivial analysis
  • The likelihood ratio test statistic
  • Distributions of test statistics
  • Incorporating systematic uncertainties
  • Incorporating systematic uncertainties (2)
  • Dealing with nuisance parameters in the test statistic
  • Dealing with nuisance parameters in the test statistic (2)
  • Putting it all together for one Higgs channel
  • Example ndash 95 Exclusion limit vs mH for HWW
  • Combining Higgs channels (and experiments)
  • A word on the machinery
  • Example ndash joint ATLASCMS Higgs exclusion limit
  • Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
  • Conclusions
Page 6: Statistical aspects of Higgs analyses

Upper limits (one-sided confidence intervals)bull Can also defined p-values for hypothesis with signal ps+b

ndash Note convention integration range in ps+b is flipped

bull Convention express result as value of s for which p(s+b)=5 ldquosgt68 is excluded at 95 CLrdquo

Wouter Verkerke NIKHEF

obsN

bs dNsbNPoissonp )(

p(s=15) = 000025p(s=10) = 0007p(s=5) = 013

p(s=68) = 005

Modified frequentist upper limitsbull Need to be careful about interpretation p(s+b) in terms

of inference on signal onlyndash Since p(s+b) quantifies consistency of signal plus backgroundndash Problem most apparent when observed data

has downward stat fluctations wrt background expectationbull Example Nobs =2

bull Modified approach to protectagainst such inference on sndash Instead of requiring p(s+b)=5

require

ps+b(s=0) = 004

sge0 excluded at gt95 CL s=0

s=5

s=10s=15

51

b

bsS p

pCL

for N=2 exclude sgt34 at 95 CLs for large N effect on limit is small as pb0

p-values and limits on non-trivial analysis

bull Typical Higgs search result is not a simple number counting experiment but looks like this

bull Any type of result can be converted into a single number by constructing a lsquotest statisticrsquo ndash A test statistic compresses all signal-to-background

discrimination power in a single numberndash Most powerful discriminators

are Likelihood Ratios(Neyman Pearson)

)ˆ|()|(ln2

dataLdataLq

- Result is a distribution not a single number

- Models for signal and background have intrinsic uncertainties

The likelihood ratio test statistic

bull Definition μ = signal strength signal strength(SM)ndash Choose eg likelihood with nominal signal strength in numerator (μ=1)

bull Illustration on model with no shape uncertainties

)ˆ|()1|(ln21

dataLdataLq

μ is best fit value of μ^

lsquolikelihood of best fitrsquo

lsquolikelihood assuming nominal signal strengthrsquo

770~1 q 3452~

1 q

On signal-like data q1 is small On background-like data q1 is large

Distributions of test statisticsbull Value of q1 on data is now the lsquomeasurementrsquobull Distribution of q1 not calculable

But can obtain distribution from pseudo-experimentsbull Generate a large number of pseudo-experiments with a given value of mu

calculate q for each plot distribution

bull From qobs and these distributions can then set limitssimilar to what was shown for Poisson counting example

)ˆ|()1|(ln2~

1

dataLdataLq

q1 for experiments with signal

q1 for experiments with background only

Note analogyto Poisson

counting example

Incorporating systematic uncertaintiesbull What happens if models have uncertainties

ndash Introduction of additional model parameters θ that describe effect of uncertainties

)|()|(

dataLdataL

)~())()(|()|( pbsNPoissondataL iii

xx θ2θ1

Jet Energy ScaleQCD scaleluminosity

Incorporating systematic uncertainties

)~())()(|()|( pbsNPoissondataL iii

xx θ2θ1

Likelihood includes auxiliary measurement terms that constrains the nuisance parameters θ(shape is flat log-normal gamma or Gaussian)

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are

incorporated in test statistic using a profile likelihood ratio

bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments

ˆ0(with a constraint )

lsquolikelihood of best fitrsquo

lsquolikelihood of best fit for a given fixed value of μrsquo

)ˆ|()|(ln2

dataLdataLq

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are

incorporated using a profile likelihood ratio test statistic

bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments

ˆ0(with a constraint )lsquolikelihood of best fitrsquo

lsquolikelihood of best fit for a given fixed value of μrsquo

μ=035 μ=10 μ=18 μ=26

Putting it all together for one Higgs channelbull Result from data D(x)bull PDF F(x|mHμθ) that models

the data for a given true Higgs mass hypothesis

bull Construct test statistic

bull Obtain expected distributions of qμ for various μndash Determine lsquodiscoveryrsquo p-value

and signal exclusion limitbull Repeat for each assumed mH

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ|(

ln2)(~

H

HH mdataL

mdataLmq

Example ndash 95 Exclusion limit vs mH for HWWExample point asymp3 x SM HWW cross-section excluded at mH=125 GeV

Example point asymp05 x SM HWW cross-section excluded at mH=165 GeV

Higgs with 10x SM cross-section excluded at 95 CL for mH in range [150~187]

Expected exclusion limitfor background-only hypothesis

Combining Higgs channels (and experiments)bull Procedure define joint likelihood

bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration

bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

)()()( CMSCMSATLASATLASLHC LLL

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)

have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and

delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

lsquollll110rootrsquo lsquogg110rootrsquo

Example ndash joint ATLASCMS Higgs exclusion limit

Wouter Verkerke NIKHEF

Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ0|(ln2~ 0

0

dataLdataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming background onlyrsquo

)ˆˆ|()ˆ|(

ln2~

dataL

dataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming μ signal strengthrsquo

770~1 q

lsquodiscoveryrsquolsquoexclusionrsquo

7734~0 q

0~0 q3452~

1 q

simulateddata with signal+bkg

simulateddata with bkg only

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)

Note that lsquopeakrsquo around 160 GeV reflects increased

experimental sensitivity not SM prediction of Higgs mass

Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)

But need to be careful with local p-values

Search is executed for a wide mH mass range

Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1

Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction

Conclusionsbull We are early awaiting more data

Wouter Verkerke NIKHEF

  • Statistical aspects of Higgs analyses
  • Introduction
  • Quantifying discovery and exclusion ndash Frequentist approach
  • Frequentist p-values ndash excess over expected bkg
  • Frequentist p-values - excess over expected bkg
  • Upper limits (one-sided confidence intervals)
  • Modified frequentist upper limits
  • p-values and limits on non-trivial analysis
  • The likelihood ratio test statistic
  • Distributions of test statistics
  • Incorporating systematic uncertainties
  • Incorporating systematic uncertainties (2)
  • Dealing with nuisance parameters in the test statistic
  • Dealing with nuisance parameters in the test statistic (2)
  • Putting it all together for one Higgs channel
  • Example ndash 95 Exclusion limit vs mH for HWW
  • Combining Higgs channels (and experiments)
  • A word on the machinery
  • Example ndash joint ATLASCMS Higgs exclusion limit
  • Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
  • Conclusions
Page 7: Statistical aspects of Higgs analyses

Modified frequentist upper limitsbull Need to be careful about interpretation p(s+b) in terms

of inference on signal onlyndash Since p(s+b) quantifies consistency of signal plus backgroundndash Problem most apparent when observed data

has downward stat fluctations wrt background expectationbull Example Nobs =2

bull Modified approach to protectagainst such inference on sndash Instead of requiring p(s+b)=5

require

ps+b(s=0) = 004

sge0 excluded at gt95 CL s=0

s=5

s=10s=15

51

b

bsS p

pCL

for N=2 exclude sgt34 at 95 CLs for large N effect on limit is small as pb0

p-values and limits on non-trivial analysis

bull Typical Higgs search result is not a simple number counting experiment but looks like this

bull Any type of result can be converted into a single number by constructing a lsquotest statisticrsquo ndash A test statistic compresses all signal-to-background

discrimination power in a single numberndash Most powerful discriminators

are Likelihood Ratios(Neyman Pearson)

)ˆ|()|(ln2

dataLdataLq

- Result is a distribution not a single number

- Models for signal and background have intrinsic uncertainties

The likelihood ratio test statistic

bull Definition μ = signal strength signal strength(SM)ndash Choose eg likelihood with nominal signal strength in numerator (μ=1)

bull Illustration on model with no shape uncertainties

)ˆ|()1|(ln21

dataLdataLq

μ is best fit value of μ^

lsquolikelihood of best fitrsquo

lsquolikelihood assuming nominal signal strengthrsquo

770~1 q 3452~

1 q

On signal-like data q1 is small On background-like data q1 is large

Distributions of test statisticsbull Value of q1 on data is now the lsquomeasurementrsquobull Distribution of q1 not calculable

But can obtain distribution from pseudo-experimentsbull Generate a large number of pseudo-experiments with a given value of mu

calculate q for each plot distribution

bull From qobs and these distributions can then set limitssimilar to what was shown for Poisson counting example

)ˆ|()1|(ln2~

1

dataLdataLq

q1 for experiments with signal

q1 for experiments with background only

Note analogyto Poisson

counting example

Incorporating systematic uncertaintiesbull What happens if models have uncertainties

ndash Introduction of additional model parameters θ that describe effect of uncertainties

)|()|(

dataLdataL

)~())()(|()|( pbsNPoissondataL iii

xx θ2θ1

Jet Energy ScaleQCD scaleluminosity

Incorporating systematic uncertainties

)~())()(|()|( pbsNPoissondataL iii

xx θ2θ1

Likelihood includes auxiliary measurement terms that constrains the nuisance parameters θ(shape is flat log-normal gamma or Gaussian)

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are

incorporated in test statistic using a profile likelihood ratio

bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments

ˆ0(with a constraint )

lsquolikelihood of best fitrsquo

lsquolikelihood of best fit for a given fixed value of μrsquo

)ˆ|()|(ln2

dataLdataLq

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are

incorporated using a profile likelihood ratio test statistic

bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments

ˆ0(with a constraint )lsquolikelihood of best fitrsquo

lsquolikelihood of best fit for a given fixed value of μrsquo

μ=035 μ=10 μ=18 μ=26

Putting it all together for one Higgs channelbull Result from data D(x)bull PDF F(x|mHμθ) that models

the data for a given true Higgs mass hypothesis

bull Construct test statistic

bull Obtain expected distributions of qμ for various μndash Determine lsquodiscoveryrsquo p-value

and signal exclusion limitbull Repeat for each assumed mH

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ|(

ln2)(~

H

HH mdataL

mdataLmq

Example ndash 95 Exclusion limit vs mH for HWWExample point asymp3 x SM HWW cross-section excluded at mH=125 GeV

Example point asymp05 x SM HWW cross-section excluded at mH=165 GeV

Higgs with 10x SM cross-section excluded at 95 CL for mH in range [150~187]

Expected exclusion limitfor background-only hypothesis

Combining Higgs channels (and experiments)bull Procedure define joint likelihood

bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration

bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

)()()( CMSCMSATLASATLASLHC LLL

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)

have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and

delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

lsquollll110rootrsquo lsquogg110rootrsquo

Example ndash joint ATLASCMS Higgs exclusion limit

Wouter Verkerke NIKHEF

Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ0|(ln2~ 0

0

dataLdataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming background onlyrsquo

)ˆˆ|()ˆ|(

ln2~

dataL

dataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming μ signal strengthrsquo

770~1 q

lsquodiscoveryrsquolsquoexclusionrsquo

7734~0 q

0~0 q3452~

1 q

simulateddata with signal+bkg

simulateddata with bkg only

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)

Note that lsquopeakrsquo around 160 GeV reflects increased

experimental sensitivity not SM prediction of Higgs mass

Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)

But need to be careful with local p-values

Search is executed for a wide mH mass range

Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1

Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction

Conclusionsbull We are early awaiting more data

Wouter Verkerke NIKHEF

  • Statistical aspects of Higgs analyses
  • Introduction
  • Quantifying discovery and exclusion ndash Frequentist approach
  • Frequentist p-values ndash excess over expected bkg
  • Frequentist p-values - excess over expected bkg
  • Upper limits (one-sided confidence intervals)
  • Modified frequentist upper limits
  • p-values and limits on non-trivial analysis
  • The likelihood ratio test statistic
  • Distributions of test statistics
  • Incorporating systematic uncertainties
  • Incorporating systematic uncertainties (2)
  • Dealing with nuisance parameters in the test statistic
  • Dealing with nuisance parameters in the test statistic (2)
  • Putting it all together for one Higgs channel
  • Example ndash 95 Exclusion limit vs mH for HWW
  • Combining Higgs channels (and experiments)
  • A word on the machinery
  • Example ndash joint ATLASCMS Higgs exclusion limit
  • Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
  • Conclusions
Page 8: Statistical aspects of Higgs analyses

p-values and limits on non-trivial analysis

bull Typical Higgs search result is not a simple number counting experiment but looks like this

bull Any type of result can be converted into a single number by constructing a lsquotest statisticrsquo ndash A test statistic compresses all signal-to-background

discrimination power in a single numberndash Most powerful discriminators

are Likelihood Ratios(Neyman Pearson)

)ˆ|()|(ln2

dataLdataLq

- Result is a distribution not a single number

- Models for signal and background have intrinsic uncertainties

The likelihood ratio test statistic

bull Definition μ = signal strength signal strength(SM)ndash Choose eg likelihood with nominal signal strength in numerator (μ=1)

bull Illustration on model with no shape uncertainties

)ˆ|()1|(ln21

dataLdataLq

μ is best fit value of μ^

lsquolikelihood of best fitrsquo

lsquolikelihood assuming nominal signal strengthrsquo

770~1 q 3452~

1 q

On signal-like data q1 is small On background-like data q1 is large

Distributions of test statisticsbull Value of q1 on data is now the lsquomeasurementrsquobull Distribution of q1 not calculable

But can obtain distribution from pseudo-experimentsbull Generate a large number of pseudo-experiments with a given value of mu

calculate q for each plot distribution

bull From qobs and these distributions can then set limitssimilar to what was shown for Poisson counting example

)ˆ|()1|(ln2~

1

dataLdataLq

q1 for experiments with signal

q1 for experiments with background only

Note analogyto Poisson

counting example

Incorporating systematic uncertaintiesbull What happens if models have uncertainties

ndash Introduction of additional model parameters θ that describe effect of uncertainties

)|()|(

dataLdataL

)~())()(|()|( pbsNPoissondataL iii

xx θ2θ1

Jet Energy ScaleQCD scaleluminosity

Incorporating systematic uncertainties

)~())()(|()|( pbsNPoissondataL iii

xx θ2θ1

Likelihood includes auxiliary measurement terms that constrains the nuisance parameters θ(shape is flat log-normal gamma or Gaussian)

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are

incorporated in test statistic using a profile likelihood ratio

bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments

ˆ0(with a constraint )

lsquolikelihood of best fitrsquo

lsquolikelihood of best fit for a given fixed value of μrsquo

)ˆ|()|(ln2

dataLdataLq

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are

incorporated using a profile likelihood ratio test statistic

bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments

ˆ0(with a constraint )lsquolikelihood of best fitrsquo

lsquolikelihood of best fit for a given fixed value of μrsquo

μ=035 μ=10 μ=18 μ=26

Putting it all together for one Higgs channelbull Result from data D(x)bull PDF F(x|mHμθ) that models

the data for a given true Higgs mass hypothesis

bull Construct test statistic

bull Obtain expected distributions of qμ for various μndash Determine lsquodiscoveryrsquo p-value

and signal exclusion limitbull Repeat for each assumed mH

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ|(

ln2)(~

H

HH mdataL

mdataLmq

Example ndash 95 Exclusion limit vs mH for HWWExample point asymp3 x SM HWW cross-section excluded at mH=125 GeV

Example point asymp05 x SM HWW cross-section excluded at mH=165 GeV

Higgs with 10x SM cross-section excluded at 95 CL for mH in range [150~187]

Expected exclusion limitfor background-only hypothesis

Combining Higgs channels (and experiments)bull Procedure define joint likelihood

bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration

bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

)()()( CMSCMSATLASATLASLHC LLL

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)

have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and

delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

lsquollll110rootrsquo lsquogg110rootrsquo

Example ndash joint ATLASCMS Higgs exclusion limit

Wouter Verkerke NIKHEF

Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ0|(ln2~ 0

0

dataLdataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming background onlyrsquo

)ˆˆ|()ˆ|(

ln2~

dataL

dataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming μ signal strengthrsquo

770~1 q

lsquodiscoveryrsquolsquoexclusionrsquo

7734~0 q

0~0 q3452~

1 q

simulateddata with signal+bkg

simulateddata with bkg only

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)

Note that lsquopeakrsquo around 160 GeV reflects increased

experimental sensitivity not SM prediction of Higgs mass

Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)

But need to be careful with local p-values

Search is executed for a wide mH mass range

Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1

Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction

Conclusionsbull We are early awaiting more data

Wouter Verkerke NIKHEF

  • Statistical aspects of Higgs analyses
  • Introduction
  • Quantifying discovery and exclusion ndash Frequentist approach
  • Frequentist p-values ndash excess over expected bkg
  • Frequentist p-values - excess over expected bkg
  • Upper limits (one-sided confidence intervals)
  • Modified frequentist upper limits
  • p-values and limits on non-trivial analysis
  • The likelihood ratio test statistic
  • Distributions of test statistics
  • Incorporating systematic uncertainties
  • Incorporating systematic uncertainties (2)
  • Dealing with nuisance parameters in the test statistic
  • Dealing with nuisance parameters in the test statistic (2)
  • Putting it all together for one Higgs channel
  • Example ndash 95 Exclusion limit vs mH for HWW
  • Combining Higgs channels (and experiments)
  • A word on the machinery
  • Example ndash joint ATLASCMS Higgs exclusion limit
  • Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
  • Conclusions
Page 9: Statistical aspects of Higgs analyses

The likelihood ratio test statistic

bull Definition μ = signal strength signal strength(SM)ndash Choose eg likelihood with nominal signal strength in numerator (μ=1)

bull Illustration on model with no shape uncertainties

)ˆ|()1|(ln21

dataLdataLq

μ is best fit value of μ^

lsquolikelihood of best fitrsquo

lsquolikelihood assuming nominal signal strengthrsquo

770~1 q 3452~

1 q

On signal-like data q1 is small On background-like data q1 is large

Distributions of test statisticsbull Value of q1 on data is now the lsquomeasurementrsquobull Distribution of q1 not calculable

But can obtain distribution from pseudo-experimentsbull Generate a large number of pseudo-experiments with a given value of mu

calculate q for each plot distribution

bull From qobs and these distributions can then set limitssimilar to what was shown for Poisson counting example

)ˆ|()1|(ln2~

1

dataLdataLq

q1 for experiments with signal

q1 for experiments with background only

Note analogyto Poisson

counting example

Incorporating systematic uncertaintiesbull What happens if models have uncertainties

ndash Introduction of additional model parameters θ that describe effect of uncertainties

)|()|(

dataLdataL

)~())()(|()|( pbsNPoissondataL iii

xx θ2θ1

Jet Energy ScaleQCD scaleluminosity

Incorporating systematic uncertainties

)~())()(|()|( pbsNPoissondataL iii

xx θ2θ1

Likelihood includes auxiliary measurement terms that constrains the nuisance parameters θ(shape is flat log-normal gamma or Gaussian)

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are

incorporated in test statistic using a profile likelihood ratio

bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments

ˆ0(with a constraint )

lsquolikelihood of best fitrsquo

lsquolikelihood of best fit for a given fixed value of μrsquo

)ˆ|()|(ln2

dataLdataLq

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are

incorporated using a profile likelihood ratio test statistic

bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments

ˆ0(with a constraint )lsquolikelihood of best fitrsquo

lsquolikelihood of best fit for a given fixed value of μrsquo

μ=035 μ=10 μ=18 μ=26

Putting it all together for one Higgs channelbull Result from data D(x)bull PDF F(x|mHμθ) that models

the data for a given true Higgs mass hypothesis

bull Construct test statistic

bull Obtain expected distributions of qμ for various μndash Determine lsquodiscoveryrsquo p-value

and signal exclusion limitbull Repeat for each assumed mH

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ|(

ln2)(~

H

HH mdataL

mdataLmq

Example ndash 95 Exclusion limit vs mH for HWWExample point asymp3 x SM HWW cross-section excluded at mH=125 GeV

Example point asymp05 x SM HWW cross-section excluded at mH=165 GeV

Higgs with 10x SM cross-section excluded at 95 CL for mH in range [150~187]

Expected exclusion limitfor background-only hypothesis

Combining Higgs channels (and experiments)bull Procedure define joint likelihood

bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration

bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

)()()( CMSCMSATLASATLASLHC LLL

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)

have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and

delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

lsquollll110rootrsquo lsquogg110rootrsquo

Example ndash joint ATLASCMS Higgs exclusion limit

Wouter Verkerke NIKHEF

Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ0|(ln2~ 0

0

dataLdataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming background onlyrsquo

)ˆˆ|()ˆ|(

ln2~

dataL

dataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming μ signal strengthrsquo

770~1 q

lsquodiscoveryrsquolsquoexclusionrsquo

7734~0 q

0~0 q3452~

1 q

simulateddata with signal+bkg

simulateddata with bkg only

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)

Note that lsquopeakrsquo around 160 GeV reflects increased

experimental sensitivity not SM prediction of Higgs mass

Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)

But need to be careful with local p-values

Search is executed for a wide mH mass range

Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1

Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction

Conclusionsbull We are early awaiting more data

Wouter Verkerke NIKHEF

  • Statistical aspects of Higgs analyses
  • Introduction
  • Quantifying discovery and exclusion ndash Frequentist approach
  • Frequentist p-values ndash excess over expected bkg
  • Frequentist p-values - excess over expected bkg
  • Upper limits (one-sided confidence intervals)
  • Modified frequentist upper limits
  • p-values and limits on non-trivial analysis
  • The likelihood ratio test statistic
  • Distributions of test statistics
  • Incorporating systematic uncertainties
  • Incorporating systematic uncertainties (2)
  • Dealing with nuisance parameters in the test statistic
  • Dealing with nuisance parameters in the test statistic (2)
  • Putting it all together for one Higgs channel
  • Example ndash 95 Exclusion limit vs mH for HWW
  • Combining Higgs channels (and experiments)
  • A word on the machinery
  • Example ndash joint ATLASCMS Higgs exclusion limit
  • Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
  • Conclusions
Page 10: Statistical aspects of Higgs analyses

Distributions of test statisticsbull Value of q1 on data is now the lsquomeasurementrsquobull Distribution of q1 not calculable

But can obtain distribution from pseudo-experimentsbull Generate a large number of pseudo-experiments with a given value of mu

calculate q for each plot distribution

bull From qobs and these distributions can then set limitssimilar to what was shown for Poisson counting example

)ˆ|()1|(ln2~

1

dataLdataLq

q1 for experiments with signal

q1 for experiments with background only

Note analogyto Poisson

counting example

Incorporating systematic uncertaintiesbull What happens if models have uncertainties

ndash Introduction of additional model parameters θ that describe effect of uncertainties

)|()|(

dataLdataL

)~())()(|()|( pbsNPoissondataL iii

xx θ2θ1

Jet Energy ScaleQCD scaleluminosity

Incorporating systematic uncertainties

)~())()(|()|( pbsNPoissondataL iii

xx θ2θ1

Likelihood includes auxiliary measurement terms that constrains the nuisance parameters θ(shape is flat log-normal gamma or Gaussian)

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are

incorporated in test statistic using a profile likelihood ratio

bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments

ˆ0(with a constraint )

lsquolikelihood of best fitrsquo

lsquolikelihood of best fit for a given fixed value of μrsquo

)ˆ|()|(ln2

dataLdataLq

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are

incorporated using a profile likelihood ratio test statistic

bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments

ˆ0(with a constraint )lsquolikelihood of best fitrsquo

lsquolikelihood of best fit for a given fixed value of μrsquo

μ=035 μ=10 μ=18 μ=26

Putting it all together for one Higgs channelbull Result from data D(x)bull PDF F(x|mHμθ) that models

the data for a given true Higgs mass hypothesis

bull Construct test statistic

bull Obtain expected distributions of qμ for various μndash Determine lsquodiscoveryrsquo p-value

and signal exclusion limitbull Repeat for each assumed mH

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ|(

ln2)(~

H

HH mdataL

mdataLmq

Example ndash 95 Exclusion limit vs mH for HWWExample point asymp3 x SM HWW cross-section excluded at mH=125 GeV

Example point asymp05 x SM HWW cross-section excluded at mH=165 GeV

Higgs with 10x SM cross-section excluded at 95 CL for mH in range [150~187]

Expected exclusion limitfor background-only hypothesis

Combining Higgs channels (and experiments)bull Procedure define joint likelihood

bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration

bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

)()()( CMSCMSATLASATLASLHC LLL

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)

have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and

delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

lsquollll110rootrsquo lsquogg110rootrsquo

Example ndash joint ATLASCMS Higgs exclusion limit

Wouter Verkerke NIKHEF

Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ0|(ln2~ 0

0

dataLdataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming background onlyrsquo

)ˆˆ|()ˆ|(

ln2~

dataL

dataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming μ signal strengthrsquo

770~1 q

lsquodiscoveryrsquolsquoexclusionrsquo

7734~0 q

0~0 q3452~

1 q

simulateddata with signal+bkg

simulateddata with bkg only

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)

Note that lsquopeakrsquo around 160 GeV reflects increased

experimental sensitivity not SM prediction of Higgs mass

Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)

But need to be careful with local p-values

Search is executed for a wide mH mass range

Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1

Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction

Conclusionsbull We are early awaiting more data

Wouter Verkerke NIKHEF

  • Statistical aspects of Higgs analyses
  • Introduction
  • Quantifying discovery and exclusion ndash Frequentist approach
  • Frequentist p-values ndash excess over expected bkg
  • Frequentist p-values - excess over expected bkg
  • Upper limits (one-sided confidence intervals)
  • Modified frequentist upper limits
  • p-values and limits on non-trivial analysis
  • The likelihood ratio test statistic
  • Distributions of test statistics
  • Incorporating systematic uncertainties
  • Incorporating systematic uncertainties (2)
  • Dealing with nuisance parameters in the test statistic
  • Dealing with nuisance parameters in the test statistic (2)
  • Putting it all together for one Higgs channel
  • Example ndash 95 Exclusion limit vs mH for HWW
  • Combining Higgs channels (and experiments)
  • A word on the machinery
  • Example ndash joint ATLASCMS Higgs exclusion limit
  • Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
  • Conclusions
Page 11: Statistical aspects of Higgs analyses

Incorporating systematic uncertaintiesbull What happens if models have uncertainties

ndash Introduction of additional model parameters θ that describe effect of uncertainties

)|()|(

dataLdataL

)~())()(|()|( pbsNPoissondataL iii

xx θ2θ1

Jet Energy ScaleQCD scaleluminosity

Incorporating systematic uncertainties

)~())()(|()|( pbsNPoissondataL iii

xx θ2θ1

Likelihood includes auxiliary measurement terms that constrains the nuisance parameters θ(shape is flat log-normal gamma or Gaussian)

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are

incorporated in test statistic using a profile likelihood ratio

bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments

ˆ0(with a constraint )

lsquolikelihood of best fitrsquo

lsquolikelihood of best fit for a given fixed value of μrsquo

)ˆ|()|(ln2

dataLdataLq

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are

incorporated using a profile likelihood ratio test statistic

bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments

ˆ0(with a constraint )lsquolikelihood of best fitrsquo

lsquolikelihood of best fit for a given fixed value of μrsquo

μ=035 μ=10 μ=18 μ=26

Putting it all together for one Higgs channelbull Result from data D(x)bull PDF F(x|mHμθ) that models

the data for a given true Higgs mass hypothesis

bull Construct test statistic

bull Obtain expected distributions of qμ for various μndash Determine lsquodiscoveryrsquo p-value

and signal exclusion limitbull Repeat for each assumed mH

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ|(

ln2)(~

H

HH mdataL

mdataLmq

Example ndash 95 Exclusion limit vs mH for HWWExample point asymp3 x SM HWW cross-section excluded at mH=125 GeV

Example point asymp05 x SM HWW cross-section excluded at mH=165 GeV

Higgs with 10x SM cross-section excluded at 95 CL for mH in range [150~187]

Expected exclusion limitfor background-only hypothesis

Combining Higgs channels (and experiments)bull Procedure define joint likelihood

bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration

bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

)()()( CMSCMSATLASATLASLHC LLL

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)

have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and

delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

lsquollll110rootrsquo lsquogg110rootrsquo

Example ndash joint ATLASCMS Higgs exclusion limit

Wouter Verkerke NIKHEF

Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ0|(ln2~ 0

0

dataLdataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming background onlyrsquo

)ˆˆ|()ˆ|(

ln2~

dataL

dataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming μ signal strengthrsquo

770~1 q

lsquodiscoveryrsquolsquoexclusionrsquo

7734~0 q

0~0 q3452~

1 q

simulateddata with signal+bkg

simulateddata with bkg only

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)

Note that lsquopeakrsquo around 160 GeV reflects increased

experimental sensitivity not SM prediction of Higgs mass

Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)

But need to be careful with local p-values

Search is executed for a wide mH mass range

Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1

Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction

Conclusionsbull We are early awaiting more data

Wouter Verkerke NIKHEF

  • Statistical aspects of Higgs analyses
  • Introduction
  • Quantifying discovery and exclusion ndash Frequentist approach
  • Frequentist p-values ndash excess over expected bkg
  • Frequentist p-values - excess over expected bkg
  • Upper limits (one-sided confidence intervals)
  • Modified frequentist upper limits
  • p-values and limits on non-trivial analysis
  • The likelihood ratio test statistic
  • Distributions of test statistics
  • Incorporating systematic uncertainties
  • Incorporating systematic uncertainties (2)
  • Dealing with nuisance parameters in the test statistic
  • Dealing with nuisance parameters in the test statistic (2)
  • Putting it all together for one Higgs channel
  • Example ndash 95 Exclusion limit vs mH for HWW
  • Combining Higgs channels (and experiments)
  • A word on the machinery
  • Example ndash joint ATLASCMS Higgs exclusion limit
  • Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
  • Conclusions
Page 12: Statistical aspects of Higgs analyses

Incorporating systematic uncertainties

)~())()(|()|( pbsNPoissondataL iii

xx θ2θ1

Likelihood includes auxiliary measurement terms that constrains the nuisance parameters θ(shape is flat log-normal gamma or Gaussian)

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are

incorporated in test statistic using a profile likelihood ratio

bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments

ˆ0(with a constraint )

lsquolikelihood of best fitrsquo

lsquolikelihood of best fit for a given fixed value of μrsquo

)ˆ|()|(ln2

dataLdataLq

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are

incorporated using a profile likelihood ratio test statistic

bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments

ˆ0(with a constraint )lsquolikelihood of best fitrsquo

lsquolikelihood of best fit for a given fixed value of μrsquo

μ=035 μ=10 μ=18 μ=26

Putting it all together for one Higgs channelbull Result from data D(x)bull PDF F(x|mHμθ) that models

the data for a given true Higgs mass hypothesis

bull Construct test statistic

bull Obtain expected distributions of qμ for various μndash Determine lsquodiscoveryrsquo p-value

and signal exclusion limitbull Repeat for each assumed mH

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ|(

ln2)(~

H

HH mdataL

mdataLmq

Example ndash 95 Exclusion limit vs mH for HWWExample point asymp3 x SM HWW cross-section excluded at mH=125 GeV

Example point asymp05 x SM HWW cross-section excluded at mH=165 GeV

Higgs with 10x SM cross-section excluded at 95 CL for mH in range [150~187]

Expected exclusion limitfor background-only hypothesis

Combining Higgs channels (and experiments)bull Procedure define joint likelihood

bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration

bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

)()()( CMSCMSATLASATLASLHC LLL

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)

have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and

delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

lsquollll110rootrsquo lsquogg110rootrsquo

Example ndash joint ATLASCMS Higgs exclusion limit

Wouter Verkerke NIKHEF

Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ0|(ln2~ 0

0

dataLdataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming background onlyrsquo

)ˆˆ|()ˆ|(

ln2~

dataL

dataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming μ signal strengthrsquo

770~1 q

lsquodiscoveryrsquolsquoexclusionrsquo

7734~0 q

0~0 q3452~

1 q

simulateddata with signal+bkg

simulateddata with bkg only

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)

Note that lsquopeakrsquo around 160 GeV reflects increased

experimental sensitivity not SM prediction of Higgs mass

Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)

But need to be careful with local p-values

Search is executed for a wide mH mass range

Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1

Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction

Conclusionsbull We are early awaiting more data

Wouter Verkerke NIKHEF

  • Statistical aspects of Higgs analyses
  • Introduction
  • Quantifying discovery and exclusion ndash Frequentist approach
  • Frequentist p-values ndash excess over expected bkg
  • Frequentist p-values - excess over expected bkg
  • Upper limits (one-sided confidence intervals)
  • Modified frequentist upper limits
  • p-values and limits on non-trivial analysis
  • The likelihood ratio test statistic
  • Distributions of test statistics
  • Incorporating systematic uncertainties
  • Incorporating systematic uncertainties (2)
  • Dealing with nuisance parameters in the test statistic
  • Dealing with nuisance parameters in the test statistic (2)
  • Putting it all together for one Higgs channel
  • Example ndash 95 Exclusion limit vs mH for HWW
  • Combining Higgs channels (and experiments)
  • A word on the machinery
  • Example ndash joint ATLASCMS Higgs exclusion limit
  • Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
  • Conclusions
Page 13: Statistical aspects of Higgs analyses

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are

incorporated in test statistic using a profile likelihood ratio

bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments

ˆ0(with a constraint )

lsquolikelihood of best fitrsquo

lsquolikelihood of best fit for a given fixed value of μrsquo

)ˆ|()|(ln2

dataLdataLq

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are

incorporated using a profile likelihood ratio test statistic

bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments

ˆ0(with a constraint )lsquolikelihood of best fitrsquo

lsquolikelihood of best fit for a given fixed value of μrsquo

μ=035 μ=10 μ=18 μ=26

Putting it all together for one Higgs channelbull Result from data D(x)bull PDF F(x|mHμθ) that models

the data for a given true Higgs mass hypothesis

bull Construct test statistic

bull Obtain expected distributions of qμ for various μndash Determine lsquodiscoveryrsquo p-value

and signal exclusion limitbull Repeat for each assumed mH

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ|(

ln2)(~

H

HH mdataL

mdataLmq

Example ndash 95 Exclusion limit vs mH for HWWExample point asymp3 x SM HWW cross-section excluded at mH=125 GeV

Example point asymp05 x SM HWW cross-section excluded at mH=165 GeV

Higgs with 10x SM cross-section excluded at 95 CL for mH in range [150~187]

Expected exclusion limitfor background-only hypothesis

Combining Higgs channels (and experiments)bull Procedure define joint likelihood

bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration

bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

)()()( CMSCMSATLASATLASLHC LLL

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)

have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and

delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

lsquollll110rootrsquo lsquogg110rootrsquo

Example ndash joint ATLASCMS Higgs exclusion limit

Wouter Verkerke NIKHEF

Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ0|(ln2~ 0

0

dataLdataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming background onlyrsquo

)ˆˆ|()ˆ|(

ln2~

dataL

dataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming μ signal strengthrsquo

770~1 q

lsquodiscoveryrsquolsquoexclusionrsquo

7734~0 q

0~0 q3452~

1 q

simulateddata with signal+bkg

simulateddata with bkg only

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)

Note that lsquopeakrsquo around 160 GeV reflects increased

experimental sensitivity not SM prediction of Higgs mass

Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)

But need to be careful with local p-values

Search is executed for a wide mH mass range

Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1

Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction

Conclusionsbull We are early awaiting more data

Wouter Verkerke NIKHEF

  • Statistical aspects of Higgs analyses
  • Introduction
  • Quantifying discovery and exclusion ndash Frequentist approach
  • Frequentist p-values ndash excess over expected bkg
  • Frequentist p-values - excess over expected bkg
  • Upper limits (one-sided confidence intervals)
  • Modified frequentist upper limits
  • p-values and limits on non-trivial analysis
  • The likelihood ratio test statistic
  • Distributions of test statistics
  • Incorporating systematic uncertainties
  • Incorporating systematic uncertainties (2)
  • Dealing with nuisance parameters in the test statistic
  • Dealing with nuisance parameters in the test statistic (2)
  • Putting it all together for one Higgs channel
  • Example ndash 95 Exclusion limit vs mH for HWW
  • Combining Higgs channels (and experiments)
  • A word on the machinery
  • Example ndash joint ATLASCMS Higgs exclusion limit
  • Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
  • Conclusions
Page 14: Statistical aspects of Higgs analyses

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are

incorporated using a profile likelihood ratio test statistic

bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments

ˆ0(with a constraint )lsquolikelihood of best fitrsquo

lsquolikelihood of best fit for a given fixed value of μrsquo

μ=035 μ=10 μ=18 μ=26

Putting it all together for one Higgs channelbull Result from data D(x)bull PDF F(x|mHμθ) that models

the data for a given true Higgs mass hypothesis

bull Construct test statistic

bull Obtain expected distributions of qμ for various μndash Determine lsquodiscoveryrsquo p-value

and signal exclusion limitbull Repeat for each assumed mH

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ|(

ln2)(~

H

HH mdataL

mdataLmq

Example ndash 95 Exclusion limit vs mH for HWWExample point asymp3 x SM HWW cross-section excluded at mH=125 GeV

Example point asymp05 x SM HWW cross-section excluded at mH=165 GeV

Higgs with 10x SM cross-section excluded at 95 CL for mH in range [150~187]

Expected exclusion limitfor background-only hypothesis

Combining Higgs channels (and experiments)bull Procedure define joint likelihood

bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration

bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

)()()( CMSCMSATLASATLASLHC LLL

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)

have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and

delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

lsquollll110rootrsquo lsquogg110rootrsquo

Example ndash joint ATLASCMS Higgs exclusion limit

Wouter Verkerke NIKHEF

Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ0|(ln2~ 0

0

dataLdataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming background onlyrsquo

)ˆˆ|()ˆ|(

ln2~

dataL

dataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming μ signal strengthrsquo

770~1 q

lsquodiscoveryrsquolsquoexclusionrsquo

7734~0 q

0~0 q3452~

1 q

simulateddata with signal+bkg

simulateddata with bkg only

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)

Note that lsquopeakrsquo around 160 GeV reflects increased

experimental sensitivity not SM prediction of Higgs mass

Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)

But need to be careful with local p-values

Search is executed for a wide mH mass range

Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1

Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction

Conclusionsbull We are early awaiting more data

Wouter Verkerke NIKHEF

  • Statistical aspects of Higgs analyses
  • Introduction
  • Quantifying discovery and exclusion ndash Frequentist approach
  • Frequentist p-values ndash excess over expected bkg
  • Frequentist p-values - excess over expected bkg
  • Upper limits (one-sided confidence intervals)
  • Modified frequentist upper limits
  • p-values and limits on non-trivial analysis
  • The likelihood ratio test statistic
  • Distributions of test statistics
  • Incorporating systematic uncertainties
  • Incorporating systematic uncertainties (2)
  • Dealing with nuisance parameters in the test statistic
  • Dealing with nuisance parameters in the test statistic (2)
  • Putting it all together for one Higgs channel
  • Example ndash 95 Exclusion limit vs mH for HWW
  • Combining Higgs channels (and experiments)
  • A word on the machinery
  • Example ndash joint ATLASCMS Higgs exclusion limit
  • Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
  • Conclusions
Page 15: Statistical aspects of Higgs analyses

Putting it all together for one Higgs channelbull Result from data D(x)bull PDF F(x|mHμθ) that models

the data for a given true Higgs mass hypothesis

bull Construct test statistic

bull Obtain expected distributions of qμ for various μndash Determine lsquodiscoveryrsquo p-value

and signal exclusion limitbull Repeat for each assumed mH

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ|(

ln2)(~

H

HH mdataL

mdataLmq

Example ndash 95 Exclusion limit vs mH for HWWExample point asymp3 x SM HWW cross-section excluded at mH=125 GeV

Example point asymp05 x SM HWW cross-section excluded at mH=165 GeV

Higgs with 10x SM cross-section excluded at 95 CL for mH in range [150~187]

Expected exclusion limitfor background-only hypothesis

Combining Higgs channels (and experiments)bull Procedure define joint likelihood

bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration

bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

)()()( CMSCMSATLASATLASLHC LLL

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)

have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and

delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

lsquollll110rootrsquo lsquogg110rootrsquo

Example ndash joint ATLASCMS Higgs exclusion limit

Wouter Verkerke NIKHEF

Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ0|(ln2~ 0

0

dataLdataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming background onlyrsquo

)ˆˆ|()ˆ|(

ln2~

dataL

dataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming μ signal strengthrsquo

770~1 q

lsquodiscoveryrsquolsquoexclusionrsquo

7734~0 q

0~0 q3452~

1 q

simulateddata with signal+bkg

simulateddata with bkg only

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)

Note that lsquopeakrsquo around 160 GeV reflects increased

experimental sensitivity not SM prediction of Higgs mass

Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)

But need to be careful with local p-values

Search is executed for a wide mH mass range

Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1

Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction

Conclusionsbull We are early awaiting more data

Wouter Verkerke NIKHEF

  • Statistical aspects of Higgs analyses
  • Introduction
  • Quantifying discovery and exclusion ndash Frequentist approach
  • Frequentist p-values ndash excess over expected bkg
  • Frequentist p-values - excess over expected bkg
  • Upper limits (one-sided confidence intervals)
  • Modified frequentist upper limits
  • p-values and limits on non-trivial analysis
  • The likelihood ratio test statistic
  • Distributions of test statistics
  • Incorporating systematic uncertainties
  • Incorporating systematic uncertainties (2)
  • Dealing with nuisance parameters in the test statistic
  • Dealing with nuisance parameters in the test statistic (2)
  • Putting it all together for one Higgs channel
  • Example ndash 95 Exclusion limit vs mH for HWW
  • Combining Higgs channels (and experiments)
  • A word on the machinery
  • Example ndash joint ATLASCMS Higgs exclusion limit
  • Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
  • Conclusions
Page 16: Statistical aspects of Higgs analyses

Example ndash 95 Exclusion limit vs mH for HWWExample point asymp3 x SM HWW cross-section excluded at mH=125 GeV

Example point asymp05 x SM HWW cross-section excluded at mH=165 GeV

Higgs with 10x SM cross-section excluded at 95 CL for mH in range [150~187]

Expected exclusion limitfor background-only hypothesis

Combining Higgs channels (and experiments)bull Procedure define joint likelihood

bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration

bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

)()()( CMSCMSATLASATLASLHC LLL

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)

have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and

delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

lsquollll110rootrsquo lsquogg110rootrsquo

Example ndash joint ATLASCMS Higgs exclusion limit

Wouter Verkerke NIKHEF

Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ0|(ln2~ 0

0

dataLdataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming background onlyrsquo

)ˆˆ|()ˆ|(

ln2~

dataL

dataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming μ signal strengthrsquo

770~1 q

lsquodiscoveryrsquolsquoexclusionrsquo

7734~0 q

0~0 q3452~

1 q

simulateddata with signal+bkg

simulateddata with bkg only

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)

Note that lsquopeakrsquo around 160 GeV reflects increased

experimental sensitivity not SM prediction of Higgs mass

Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)

But need to be careful with local p-values

Search is executed for a wide mH mass range

Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1

Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction

Conclusionsbull We are early awaiting more data

Wouter Verkerke NIKHEF

  • Statistical aspects of Higgs analyses
  • Introduction
  • Quantifying discovery and exclusion ndash Frequentist approach
  • Frequentist p-values ndash excess over expected bkg
  • Frequentist p-values - excess over expected bkg
  • Upper limits (one-sided confidence intervals)
  • Modified frequentist upper limits
  • p-values and limits on non-trivial analysis
  • The likelihood ratio test statistic
  • Distributions of test statistics
  • Incorporating systematic uncertainties
  • Incorporating systematic uncertainties (2)
  • Dealing with nuisance parameters in the test statistic
  • Dealing with nuisance parameters in the test statistic (2)
  • Putting it all together for one Higgs channel
  • Example ndash 95 Exclusion limit vs mH for HWW
  • Combining Higgs channels (and experiments)
  • A word on the machinery
  • Example ndash joint ATLASCMS Higgs exclusion limit
  • Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
  • Conclusions
Page 17: Statistical aspects of Higgs analyses

Combining Higgs channels (and experiments)bull Procedure define joint likelihood

bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration

bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

)()()( CMSCMSATLASATLASLHC LLL

)ˆˆ|()ˆ|(

ln2~

dataLdataL

q

A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)

have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and

delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

lsquollll110rootrsquo lsquogg110rootrsquo

Example ndash joint ATLASCMS Higgs exclusion limit

Wouter Verkerke NIKHEF

Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ0|(ln2~ 0

0

dataLdataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming background onlyrsquo

)ˆˆ|()ˆ|(

ln2~

dataL

dataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming μ signal strengthrsquo

770~1 q

lsquodiscoveryrsquolsquoexclusionrsquo

7734~0 q

0~0 q3452~

1 q

simulateddata with signal+bkg

simulateddata with bkg only

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)

Note that lsquopeakrsquo around 160 GeV reflects increased

experimental sensitivity not SM prediction of Higgs mass

Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)

But need to be careful with local p-values

Search is executed for a wide mH mass range

Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1

Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction

Conclusionsbull We are early awaiting more data

Wouter Verkerke NIKHEF

  • Statistical aspects of Higgs analyses
  • Introduction
  • Quantifying discovery and exclusion ndash Frequentist approach
  • Frequentist p-values ndash excess over expected bkg
  • Frequentist p-values - excess over expected bkg
  • Upper limits (one-sided confidence intervals)
  • Modified frequentist upper limits
  • p-values and limits on non-trivial analysis
  • The likelihood ratio test statistic
  • Distributions of test statistics
  • Incorporating systematic uncertainties
  • Incorporating systematic uncertainties (2)
  • Dealing with nuisance parameters in the test statistic
  • Dealing with nuisance parameters in the test statistic (2)
  • Putting it all together for one Higgs channel
  • Example ndash 95 Exclusion limit vs mH for HWW
  • Combining Higgs channels (and experiments)
  • A word on the machinery
  • Example ndash joint ATLASCMS Higgs exclusion limit
  • Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
  • Conclusions
Page 18: Statistical aspects of Higgs analyses

A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)

have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and

delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward

Wouter Verkerke NIKHEF

)()()()( HZZZZHWWWWHcomb LLLL

lsquollll110rootrsquo lsquogg110rootrsquo

Example ndash joint ATLASCMS Higgs exclusion limit

Wouter Verkerke NIKHEF

Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ0|(ln2~ 0

0

dataLdataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming background onlyrsquo

)ˆˆ|()ˆ|(

ln2~

dataL

dataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming μ signal strengthrsquo

770~1 q

lsquodiscoveryrsquolsquoexclusionrsquo

7734~0 q

0~0 q3452~

1 q

simulateddata with signal+bkg

simulateddata with bkg only

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)

Note that lsquopeakrsquo around 160 GeV reflects increased

experimental sensitivity not SM prediction of Higgs mass

Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)

But need to be careful with local p-values

Search is executed for a wide mH mass range

Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1

Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction

Conclusionsbull We are early awaiting more data

Wouter Verkerke NIKHEF

  • Statistical aspects of Higgs analyses
  • Introduction
  • Quantifying discovery and exclusion ndash Frequentist approach
  • Frequentist p-values ndash excess over expected bkg
  • Frequentist p-values - excess over expected bkg
  • Upper limits (one-sided confidence intervals)
  • Modified frequentist upper limits
  • p-values and limits on non-trivial analysis
  • The likelihood ratio test statistic
  • Distributions of test statistics
  • Incorporating systematic uncertainties
  • Incorporating systematic uncertainties (2)
  • Dealing with nuisance parameters in the test statistic
  • Dealing with nuisance parameters in the test statistic (2)
  • Putting it all together for one Higgs channel
  • Example ndash 95 Exclusion limit vs mH for HWW
  • Combining Higgs channels (and experiments)
  • A word on the machinery
  • Example ndash joint ATLASCMS Higgs exclusion limit
  • Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
  • Conclusions
Page 19: Statistical aspects of Higgs analyses

Example ndash joint ATLASCMS Higgs exclusion limit

Wouter Verkerke NIKHEF

Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ0|(ln2~ 0

0

dataLdataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming background onlyrsquo

)ˆˆ|()ˆ|(

ln2~

dataL

dataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming μ signal strengthrsquo

770~1 q

lsquodiscoveryrsquolsquoexclusionrsquo

7734~0 q

0~0 q3452~

1 q

simulateddata with signal+bkg

simulateddata with bkg only

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)

Note that lsquopeakrsquo around 160 GeV reflects increased

experimental sensitivity not SM prediction of Higgs mass

Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)

But need to be careful with local p-values

Search is executed for a wide mH mass range

Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1

Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction

Conclusionsbull We are early awaiting more data

Wouter Verkerke NIKHEF

  • Statistical aspects of Higgs analyses
  • Introduction
  • Quantifying discovery and exclusion ndash Frequentist approach
  • Frequentist p-values ndash excess over expected bkg
  • Frequentist p-values - excess over expected bkg
  • Upper limits (one-sided confidence intervals)
  • Modified frequentist upper limits
  • p-values and limits on non-trivial analysis
  • The likelihood ratio test statistic
  • Distributions of test statistics
  • Incorporating systematic uncertainties
  • Incorporating systematic uncertainties (2)
  • Dealing with nuisance parameters in the test statistic
  • Dealing with nuisance parameters in the test statistic (2)
  • Putting it all together for one Higgs channel
  • Example ndash 95 Exclusion limit vs mH for HWW
  • Combining Higgs channels (and experiments)
  • A word on the machinery
  • Example ndash joint ATLASCMS Higgs exclusion limit
  • Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
  • Conclusions
Page 20: Statistical aspects of Higgs analyses

Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation

Wouter Verkerke NIKHEF

)ˆˆ|()ˆ0|(ln2~ 0

0

dataLdataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming background onlyrsquo

)ˆˆ|()ˆ|(

ln2~

dataL

dataLq

lsquolikelihood of best fitrsquo

lsquolikelihood assuming μ signal strengthrsquo

770~1 q

lsquodiscoveryrsquolsquoexclusionrsquo

7734~0 q

0~0 q3452~

1 q

simulateddata with signal+bkg

simulateddata with bkg only

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)

Note that lsquopeakrsquo around 160 GeV reflects increased

experimental sensitivity not SM prediction of Higgs mass

Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)

But need to be careful with local p-values

Search is executed for a wide mH mass range

Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1

Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction

Conclusionsbull We are early awaiting more data

Wouter Verkerke NIKHEF

  • Statistical aspects of Higgs analyses
  • Introduction
  • Quantifying discovery and exclusion ndash Frequentist approach
  • Frequentist p-values ndash excess over expected bkg
  • Frequentist p-values - excess over expected bkg
  • Upper limits (one-sided confidence intervals)
  • Modified frequentist upper limits
  • p-values and limits on non-trivial analysis
  • The likelihood ratio test statistic
  • Distributions of test statistics
  • Incorporating systematic uncertainties
  • Incorporating systematic uncertainties (2)
  • Dealing with nuisance parameters in the test statistic
  • Dealing with nuisance parameters in the test statistic (2)
  • Putting it all together for one Higgs channel
  • Example ndash 95 Exclusion limit vs mH for HWW
  • Combining Higgs channels (and experiments)
  • A word on the machinery
  • Example ndash joint ATLASCMS Higgs exclusion limit
  • Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
  • Conclusions
Page 21: Statistical aspects of Higgs analyses

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)

Note that lsquopeakrsquo around 160 GeV reflects increased

experimental sensitivity not SM prediction of Higgs mass

Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)

But need to be careful with local p-values

Search is executed for a wide mH mass range

Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1

Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction

Conclusionsbull We are early awaiting more data

Wouter Verkerke NIKHEF

  • Statistical aspects of Higgs analyses
  • Introduction
  • Quantifying discovery and exclusion ndash Frequentist approach
  • Frequentist p-values ndash excess over expected bkg
  • Frequentist p-values - excess over expected bkg
  • Upper limits (one-sided confidence intervals)
  • Modified frequentist upper limits
  • p-values and limits on non-trivial analysis
  • The likelihood ratio test statistic
  • Distributions of test statistics
  • Incorporating systematic uncertainties
  • Incorporating systematic uncertainties (2)
  • Dealing with nuisance parameters in the test statistic
  • Dealing with nuisance parameters in the test statistic (2)
  • Putting it all together for one Higgs channel
  • Example ndash 95 Exclusion limit vs mH for HWW
  • Combining Higgs channels (and experiments)
  • A word on the machinery
  • Example ndash joint ATLASCMS Higgs exclusion limit
  • Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
  • Conclusions
Page 22: Statistical aspects of Higgs analyses

Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)

But need to be careful with local p-values

Search is executed for a wide mH mass range

Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1

Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction

Conclusionsbull We are early awaiting more data

Wouter Verkerke NIKHEF

  • Statistical aspects of Higgs analyses
  • Introduction
  • Quantifying discovery and exclusion ndash Frequentist approach
  • Frequentist p-values ndash excess over expected bkg
  • Frequentist p-values - excess over expected bkg
  • Upper limits (one-sided confidence intervals)
  • Modified frequentist upper limits
  • p-values and limits on non-trivial analysis
  • The likelihood ratio test statistic
  • Distributions of test statistics
  • Incorporating systematic uncertainties
  • Incorporating systematic uncertainties (2)
  • Dealing with nuisance parameters in the test statistic
  • Dealing with nuisance parameters in the test statistic (2)
  • Putting it all together for one Higgs channel
  • Example ndash 95 Exclusion limit vs mH for HWW
  • Combining Higgs channels (and experiments)
  • A word on the machinery
  • Example ndash joint ATLASCMS Higgs exclusion limit
  • Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
  • Conclusions
Page 23: Statistical aspects of Higgs analyses

Conclusionsbull We are early awaiting more data

Wouter Verkerke NIKHEF

  • Statistical aspects of Higgs analyses
  • Introduction
  • Quantifying discovery and exclusion ndash Frequentist approach
  • Frequentist p-values ndash excess over expected bkg
  • Frequentist p-values - excess over expected bkg
  • Upper limits (one-sided confidence intervals)
  • Modified frequentist upper limits
  • p-values and limits on non-trivial analysis
  • The likelihood ratio test statistic
  • Distributions of test statistics
  • Incorporating systematic uncertainties
  • Incorporating systematic uncertainties (2)
  • Dealing with nuisance parameters in the test statistic
  • Dealing with nuisance parameters in the test statistic (2)
  • Putting it all together for one Higgs channel
  • Example ndash 95 Exclusion limit vs mH for HWW
  • Combining Higgs channels (and experiments)
  • A word on the machinery
  • Example ndash joint ATLASCMS Higgs exclusion limit
  • Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
  • Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
  • Conclusions