Overview of MIC-TRK Physics Performance · 21 February 2018 K. McDermott –MIC-TRK F2F 2018 11 •...

Overview of MIC-TRK Physics Performance

Kevin McDermott

MIC-TRK F2F: February 21, 2018

Outline

21 February 2018 K. McDermott – MIC-TRK F2F 2018 2

• Broad overview of classes of measurements and tools

• Overview of measurements• Overview of association methods• Overview of samples• Current performance• Discussion of GH issues

Measurements and Tools


• Text file based: use cout statements after building to make plots of kinematic distributions and residuals using truth info– Relevant code

Ø quality_process() in mkFit/MkBuilder.ccØ plotting/textDumpFiles.shØ plotting/makePlotsFromDump.[py,C] +

plotting/PlotsFromDump.[hh,cpp]– Enabled by: --dump-for-plots --quality-val

• ROOT based: meat of this presentation and tools– Rate measurements, kinematic residuals vs. truth info– Relevant code in Backup: “Additional Resources”– Enabled by: --cmssw-val or --sim-val– Includes a host of configuration options

• Physics validation performance included in benchmarking– Standard benchmarking: ./xeon_scripts/runBenchmarks.sh– Validation: ./val_scripts/validation-snb-cmssw-benchmarks.sh

Reference tracks


• Validation is performed with respect to a set of reference tracks – Simtracks: either from cylindrical cow or CMSSW

tracking particles– CMSSW tracks: iter0 reconstructed tracks

• For validation purposes, we also specify “findable tracks” to make fair comparisons– If a track is labeled “unfindable”: excluded from

denominator (and numerator) of rate measurements– Criteria for findability

ØGeneric: require reference tracks to have at least seven total unique layers

ØWhen using simtracks as reference with CMSSW seeds: require simtracks to share at least four hits with a single CMSSW seed

Overview of measurements


Definitions of measurements


• Efficiency: how often we match one of our candidate tracks to a reference track – Plotted vs. reference track observable

• Duplicate Rate: how often we match to a reference track more than once– Plotted vs. reference track observable

• Fake Rate: how often we do not match one of our candidate tracks to a reference track – Plotted vs. candidate track observable

• Each measurement plotted vs. track pT, φ, η• Can include cut on pT of track being plotted against

– e.g. pT cut on candidate track for FR; reference track for DR– Current pT cuts: pT > 0.0, pT > 0.9, pT > 2.0 GeV/c

Efficiency


• Denominator: Set of all findable reference tracks• Numerator: Set of all findable reference tracks that

have at least one reco track match

TTbar No PUSimtrack EfficiencyMC track pT > 0.0

Duplicate Rate


• Denominator: Set of all findable reference tracks that have at least one reco track match

• Numerator: Set of all findable reference tracks that have more than one reco track match

TTbar No PUSimtrack Duplicate RateMC track pT > 0.0

Fake Rate


• Denominator: Set of all “good” reco tracks• Numerator: Set of all ”good” reco tracks that

are NOT matched to a reference track

TTbar No PUSimtrack Fake RateMC track pT > 0.0

“good”: only include reco tracks with at least seven total hits, can include with --inc-shortsDefault: exclude short tracks

Overview of association methods


Hit based: Simtracks


• “Found seeds” or CMSSW seeds– Require at least 75% of hits on reco track are

shared with a single simtrack– Denominator is nFoundHits of reco track– Example: 8 out of 10 reco hits match a single

simtrack = match

• Sim seeds (i.e. sim tracks form seeds)– Require at least 50% of hits on reco track

after the seed match the simtrack the seed is from

– Denominator is nFoundHits of reco track– Example: 4 out of 6 reco hits match hits

from original simtrackthat the seed is from = match

Reco track

Seed

Simtrack

Hit based: CMSSW tracks


• “Pure” CMSSW seeds (i.e. CMSSW seeds that formed CMSSW tracks)– Require at least 50% of hits on reco track after the

seed match the CMSSW track seed is from– Denominator is nFoundHits of reco track– Example: 5 out of 8 reco hits match hits from

original CMSSW track that the seed is from = match

– Enabled with: --cmssw-val-label

• All other CMSSW seed config– Require at least 75% of hits on reco track are shared

with a single CMSSW track – Denominator is nUniqueLayers of potential

CMSSW track– Example: 9 out of 10 reco hits match a single

CMSSW track = match– Enabled with: --cmssw-val-hit

Reco track

Seed

CMSSW track

Track parameter based


pT Range !" cut ∆$window%& < 1.0 +, < 100 ∆- < 0.2

1.0 ≤ %& < 2.0 +, < 50 ∆- < 0.12.0 ≤ %& < 5.0 +, < 50 ∆- < 0.055.0 ≤ %& < 10.0 +, < 30 ∆- < 0.01%& ≥ 10.0 +, < 20 ∆- < 0.005

• Perform 2D χ2 with 1/pT and track η• Compute track φ residual– For forward candidate tracks: propagate φ of

potential CMSSW track to radius of reco track using: φ0 – 2*arctan[ΔR/176/(pT*q)]

– Problematic for pT < 0.5 --> NaN• Cuts for each computation binned in reco track pT

• Enabled for forward candidate tracks with: --cmssw-val-hit

• Default matching for bkFit tracks to PCA

Overview of Samples


Current Samples, I


Sample nEvents nEventsfor val

nSeedTracks nSeedTracks(N2-cleaned)

10muon HS 100000 20000 17.60 9.13TTbar NoPU 1000 1000 141.61 86.01TTbar Pu35 100 100 1084.95 626.45TTbar PU 70 100 100 2013.09 1157.53TTbar PU 70 HS 4550 500 2101.54 1208.61

Sample nSimTracks nSimTracks: findable, pT > 0.0

nSimTracks: findable, pT > 0.9

nSimTracks: findable, pT > 2.0

10muon HS 9.76 5.88 5.65 4.98TTbar NoPU 328.01 47.95 31.13 16.48TTbar Pu35 5099.53 324.66 131.64 33.74TTbar PU 70 9873.33 577.43 222.48 46.70TTbar PU 70 HS 10081.8 602.42 232.23 49.86

“nTracks” is average per event, “findable” sim tracks is defined w.r.t. to CMSSW seeds

Current Samples, II


Sample nCMSSWTracks nCMSSWTracks: findable, pT > 0.0

nCMSSWTracks: findable, pT > 0.9

nCMSSWTracks: findable, pT > 2.0

10muon HS 7.33 6.93 6.68 5.88TTbar NoPU 66.43 55.51 37.66 20.19TTbar Pu35 457.72 357.88 157.5 40.67TTbar PU 70 826.39 650.89 273.35 57.04TTbar PU 70 HS 859.8 679.27 285.39 59.98

“nTracks” is average per event, “findable” sim tracks is defined w.r.t. to CMSSW seeds

Sample Discussion


• Will need to regenerate binary files– Issue 121:

https://github.com/cerati/mictest/issues/121– To add:

– Are there other things we wish to add besides what is already listed?

• Do we want to use physics data?– JetHT from 2017: probably want to pick an era when

pixels were “stable”: 2017 D,E

https://github.com/cerati/mictest/issues/121

Current Performance


Cylindrical Cow with Lids


• Currently not in use for validation–Navigation foos not implemented

• Validation is with respect to simtracks, using hit-based matching– Simtracks directly as seeds: match is declared

if reco track has 50% of its hits after the seed match the hits on the sim track

– Find seeds with our algo: match is declared if reco track has 75% of its hits match a single sim track


Current Performance: CMSSW• Three groups of performance

1. Performance vs. simtracks: 75% hit matching for mkFit candidate tracks2. Performance vs. CMSSW tracks: 75% hit matching for mkFit candidate tracks3. Performance vs. CMSSW tracks: track-parameter matching for mkFit backward

fit tracks to PCA

• For each group, show of a subset of results– Efficiency, Duplicate Rate, and Fake Rate– vs. pT– vs. η, for pT > 0.9– Samples: TTbar NoPU, TTbar PU35, TTbar PU70, 10mu HS, TTbar PU70 HS

• Additional configuration info – CMSSW N2-cleaned seeds– nTH = 24, nVU = 8int– Platform = SNB

• Full validation plots: https://kmcdermo.web.cern.ch/kmcdermo/face2face_2018/

https://kmcdermo.web.cern.ch/kmcdermo/face2face_2018/

Simtrack validation: hit based matching


Efficiency vs pT


TTbar NoPU TTbar PU35 TTbar PU70

TTbar PU70 HS10mu HS

Efficiency vs η, pT > 0.9



TTbar PU70 HS10mu HS10mu HS

Duplicate Rate vs pT




Duplicate Rate vs η, pT > 0.9




Fake Rate vs pT




Fake Rate vs η, pT > 0.9




CMSSW track validation: hit based matching


Efficiency vs pT




Fake Rate vs pT




CMSSW track validation: track parameter matching


Efficiency vs pT




Fake Rate vs pT




Open Issues for Physics


Screenshot on GitHub


Loss of efficiency with param B-field


• For some reason, when turning on parameterized magnetic field, efficiency degrades in transition region (0.8 < |η| < 1.2)

• Above: turned on param B-field for both building+fitting

• Issue 124: https://github.com/cerati/mictest/issues/124

10mu HSStatic B-field

10mu HSParam B-field

Track parameter-based matching with fit to PCA


Loss of FV efficiency with hit-based matching


• When using hit-based matching for FV, there are noticeable dips in efficiency 1.2 < |η| < 2.0 compared to CE

• Not present in track parameter-based matching


TTbar PU70 HSpT > 0.0

Hit-based matching to CMSSW tracks



η Residual in FV Track Parameter Matching


• In track parameter-based matching of backward fit tracks, residuals of matched tracks in FV are much wider than CE

• Not present in hit-based matching



Track parameter-based matching to CMSSW tracks



Lots o’ NaNs


Issue 131: https://github.com/cerati/mictest/issues/131


PCA calculation: require some consensus


Issue 125: https://github.com/cerati/mictest/issues/125

https://github.com/cerati/mictest/files/1653557/pca.pdf

Related Issue 121: https://github.com/cerati/mictest/issues/121

https://github.com/cerati/mictest/issues/125https://github.com/cerati/mictest/files/1653557/pca.pdfhttps://github.com/cerati/mictest/issues/121

Conclusions


• Outlined validation methods and underlying association techniques

• With backward fit in place, have two ways to measure performance vs. CMSSW

• Overall, performance is in good shape and have a suite of tools to check performance regularly

• Some gaps in performance look to be bugs, although trouble with NaNs and magnetic slightly worrying

Backup


Additional Resources


• Validation manifesto: https://github.com/cerati/mictest/blob/devel/validation-desc.txt– ~90% up to date, needs info for backward fit tracks + hit based matching for CMSSW tracks– Dense, but thorough – Some text explanations of measurements and association methods– Definitions of various IDs and masks– Pseudo-code for function calls

• Validation block diagram: https://indico.cern.ch/event/656884/contributions/2676532/attachments/1513662/2363067/validation_flow_diagram-v4.pdf– ~90% up to date, same state as manifesto– High level flow chart to demonstrate flow, as well as low-level pseudo-code flow

• The code itself, with most relevant files:– Event.[h,cc]: some containers for validation, high-level validation routine calls– Track.[h,cc]: TrackExtra class, association routines, ID modification routines– TTreeValiation.[h,cc]: heavy lifting for validation – mapping routines, mask assignments, filling branches– mkFit/mkFit.cc: main executable with command line options listed– mkFit/buildtestMPlex.[h,cc]: outline of high-level validation calls– mkFit/MkBuilder.[h,cc]: interface for validation and tracks from building/fitting – prepping tracks and extras,

mapping hits– plotting/runValidation.C + plotting/PlotValidation.[hh,cpp]: more heavy lifting – use root files to produce

measurements (efficiency, fake rate, duplicate rate)– plotting/makeValidation.C + plotting/StackValidation.[hh,cpp]: overlay histograms– val_scripts/validation-snb-cmssw-benchmarks.sh: standard benchmarking validation script

https://github.com/cerati/mictest/blob/devel/validation-desc.txthttps://indico.cern.ch/event/656884/contributions/2676532/attachments/1513662/2363067/validation_flow_diagram-v4.pdf

Phase0 Geometry


Phase1 Pixel Geometry


Phase1 geom is pictured y > 0Phase0 geom is pictured y < 0

http://iopscience.iop.org/article/10.1088/1748-0221/12/02/C02033/pdf
http://iopscience.iop.org/article/10.1088/1748-0221/12/02/C02033/pdf

Overview of MIC-TRK Physics Performance · 21 February 2018 K. McDermott –MIC-TRK F2F 2018 11 •...

Documents

Transcript of Overview of MIC-TRK Physics Performance · 21 February 2018 K. McDermott –MIC-TRK F2F 2018 11 •...