Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 1

Trigger UpgradeWesley H. Smith

University of Wisconsin Darin Acosta

University of FloridaSergo Jindariani

Fermilab

US CMS PMG, January 16, 2013

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 2

Performance and Schedule of the LHC

Need to handle PU~50, and L2E34 @ 13-14 TeV

Effectively a factor of ~6 (or larger) increase in current L1 rates

M.Lamont, CMS Week, also shown to Council, Dec.2012

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 3

Motivation: Trigger Rate Projections

Trigger rate studies for the special 8 TeV high-PU runs

!

Linear with PU and lumi, but rates growing large

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 4

Motivation: Projected L1 Rates @ 2E34

Single e/γ trigger Black - 14 TeV MC (50 PU) Red - 8 TeV data (66 PU)

Lepton triggers scale by ~2 for increased center of mass energy. Muons have poor control of rates at high thresholds

HT (ET sum of jets) Black - 8 TeV data (66 PU) Red - 8 TeV data (45 PU)

Rates shown for Linst=2×1034cm-2s-1

Single muon trigger Black - 14 TeV MC (50 PU) Red - 8 TeV data (66 PU)

Jet trigger rates are strongly dependent on PU

Rate in kHz !

100

100

100

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 5

US

Upg

.

US Upg.

Existing CMS Trigger & DAQOverall Trigger & DAQ Architecture: 2 Levels:Level-1 Trigger:

• 25 ns input• 3.2 s latency

Design: Interaction rate: 1 GHzBunch Crossing rate: 40 MHz

40 MHz x 25 PU = 1 GHz

Level 1 Output: 100 kHz Output to Storage: 300-400 HzAverage Event Size: 0.5 MBData production 1 TB/day

UXC

USC

Calorimeters:

Muon Systems:

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 6

Goals for CSC Trig. Upgrade- U. Florida, Rice, Texas A&M

Remove limit of 3 segments per Muon Port Card• Each Muon Port Card covers one sector• Particularly problematic for collimated multi-muons w/rising occupancy

Increase bandwidth in trigger links• Occupancy of segments from chambers will exceed optical link bandwidth to CSC

Track-Finder Sector Processors• Dropped segments will degrade performance: lose momentum precision (higher

rate) and/or tracks (inefficiency). Improve momentum resolution

• Make full use of all track information to approach best precision achievable for standalone muon reconstruction

• Steeper rate vs. pT threshold curve increases safety margin to high luminosity and high pile-up

Deliver higher precision on output track quantities to Global Trigger upgrade, & more μ candidates• η ϕ = 0.05 2.5° → 0.0125 0.015° ✕ ✕ ✕• Accommodate new algorithms like those in Higher Level Trigger

• invariant mass cuts, jet-lepton matching, … • Possible seeding of future inner tracking trigger• Requires new high bandwidth (optical) links to Global Trigger

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 7

EMU Trigger UpgradeCavern Counting Room

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 8

Muon Port Card UpgradeUse the existing MPC main boardBackplane interface to TMB remains unchanged3 original optical links are still availableNew mezzanine card with new fpga and new

links

x 60 +20 spares

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 9

EMU Track-Finder Upgrade

“BackplaneConnector”

“Sector Processor” x 12

Connection toVT892 standardbackplane

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 10

EMU CSCTF Chassis

Not US*

*Processing the resulting tracks with RPC or CSC hits is a US responsibility

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 11

Muon Trigger M&STOTAL: $701K

• Muon Port Cards: 80 x $1480 = $126,400

• Optical Fibers & installation: $51,600• New estimate 4x as much under investigation

• CSC Track-Finder: $523,246• Module Preproduction (3): $63,630• Module Production (18): $357,480• uTCA chassis, optical parts, COTS: $102,136

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 12

Upgrade Cal. Trig. Algos.- U. Wisconsin

Particle Cluster Finder• Applies tower thresholds to Calorimeter• Creates overlapped 2x2 clusters

Cluster Overlap Filter• Removes overlap between clusters• Identifies local maxima• Prunes low energy clusters

Cluster Isolation and Particle ID• Applied to local maxima• Calculates isolation deposits around 2x2,2x3 clusters• Identifies particles

Jet reconstruction• Applied on filtered clusters• Groups clusters to jets

Particle Sorter• Sorts particles & outputs the most energetic ones

MET,HT,MHT Calculation• Calculates Et Sums, Missing Et from clusters

All coded in Firmware & Tested (latency/resources)

ECA

LHCAL Δη x Δφ=0.087x0.087

e/γ

ECA

L

HCAL

τ

ECA

L

HCAL

jet

ηφ

η

φ

η

φ

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 13

Upgrade Cal. Trig. PerformanceSimulation work on stage-1 calorimeter trigger by FNAL, UI-Chicago, UC Davis, UC San Diego, U Wisconsin, MIT, Ohio State U. (subset of stage 2)

Also exploring muon isolation

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 14

Calo Trigger Upgrade in Parallel: Split inputs from ECAL & HCAL

HCALenergy

ECALenergy

Regional Calo Trigger

Global Calo Trigger

EMcandidates

Regionenergies

HFenergy

HCALenergy

Layer 1Calo Trigger

Layer 2 Calo TriggerC

urre

nt L

1 Tr

igge

r Sys

tem

Upg

rade

L1

Trig

ger S

yste

m

oSLB

oRM

• Install optical SLB and optical RM mezzanines during LS1• Install HCAL passive optical splitters during LS1 or YETS• Install HCAL backend μHTR cards for input to new trigger• Install HCAL frontend electronics after LS2 (finer segmentation)

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 15

Calorimeter Trigger Evolution3 new calorimeter trigger μTCA crates

601 + spares Optical Receiver Modules

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 16

Calorimeter Trigger ProcessorVirtex-6 Prototype Board (CTP-6)

Back End FPGAXC6VHX250T/XC6VHX380T

Front End FPGAXC6VHX250T/XC6VHX380T

Avago AFBR-810B Tx Module

4X Avago AFBR-820B Rx Module

MMC Circuitry

JTAG/USB ConsoleInterface Mezzanine

Power Modules

Dual SDRAM for dedicated DAQ and TCP/IP buffering

12x Multi Gig Backplane Connections

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 17

Calorimeter Trigger Processor Virtex-7 (CTP-7)

• Replace 2 Virtex-6s with a Virtex 7 for processing+ZYNQ for embedded TCP/IP endpoint

• 30A, 1V power module for FPGA logic core

• 3x CXP Pluggable modules for 36 Tx + 36 Rx 10G optical links

• 2x AFBR-820 modules for 24 Rx 10G optical links

• Simpler design to execute than the CTP-6

• 36 Total + spares

Virtex-7 VX690T FPGA

ZYNQXC7Z030

EPP(optional)

1.5V

Sup

ply

2.5V

Sup

ply 3.

3V S

uppl

y

1V 3

0A

Supp

ly

CXP Module

12Tx + 12 Rx

CXP Module12Tx + 12Rx

CXP Module

12Tx + 12 Rx12XRx

12XRx

(CTP-6 CAD View)

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 18

CIO-X: crate interconnections(2/crate x 3 crates = 6 + spares)

Controller (MMC and link mgmt)

4X Avago AFBR-79EQDZ QSFP+ ModulePositions

4x4 Lane Bidirectional Multi Gig Backplane Connections

Backplane Rx/Tx

Redriver ICs (top and

bottom sides)

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 19

VT894 Crate Test Setup(Final system: 3 crates w/ 12 CTP7 ea.)

BU

AM

C13

Vada

tech

M

CH U

W C

TP-

6

UW

CT

P-6

TTC Downlink

UW

Aux

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 20

Final Calo Trigger Upgrade(“Stage 2”)

Two modes of connectivity required

• Keep new trigger flexible in order to adapt to the needs of the evolving CMS physics program

• Both architectures have two processing layers• Layer 1 optimized for backplane connectivity, Layer 2 for optical

• TMT architecture chosen as baseline

Fully Pipelined Calorimeter Trigger

Time Multiplexed Calorimeter Trigger

Layer 1

Layer 2

Demux

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 21

Stage 2 Cal. Trig. Upgrade(TMT Architecture)

US:

UK:

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 22

Calorimeter Trigger M&STOTAL: 1,149 K$

• 42 CTP7s @ 17.5K$ each (36 needed plus 6 spares): 735 K$• FPGA: 12 K$• Optical: 3 K$• Board, Fabrication, Assembly, other 2.5 K$

• 6 CTP7 prototypes @ 17.5 K$ each: 105 K$• 9 CIOx @ 2K$ each (6 needed plus 3 spares): 18 K$

• Optical: 1.2 K$• Board, Fab Assembly, other 0.8 K$

• 3 CIOx prototypes @ 2K$ each: 6 K$• 4 Vadatech μTCA crates : 40 K$

• incl. MCH & PS (3 needed plus spare) @ 10 K$• 4 AMC13 Modules (3 needed plus spare) @ 5 K$: 15 K$• Optical Cables & Patch Panel between Layers 1 & 2: 20 K$• oRM’s: 700 (601 plus spares) @ 300 $: 210 K$.

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 23

Trigger Labor & Travel(non-physicist)

Total Labor: $2.07M• Muon Trigger Labor over 4 years: $ 840K

• MPC Electronic & Firmware Engineering: $160K• CSCTF Electronic & Firmware Engineering: $480K• Software Engineering: $200K

• Calorimeter Trigger Labor over 4 years: $ 1,230K• Electronic Engineering: $ 380K• Firmware Engineering: $ 530K• Technician: $160K• Software Engineering: $160K

• Travel over 4 years: $130K• Muon: $70K• Calorimeter: $60K

NB: Resource Loading is not complete

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 24

Trigger WBS High LevelNotes:

• Starts Nov. 1, 2013• Prototyping complete

• Not included• Dictionary structure only• Next: Schedule integration & Resources

• Needs alignment with CMS Structure• Underway• All WBS US only

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 25

Trigger WBS Detail

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 26

Cal. Trig. Layer 1 Schedule(official CMS version)

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 27

EMU Trigger Schedule(official CMS version)

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 28

Cal. Trig. oSLB-oRM Schedule

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 29

Schedule & CompletionTrigger threshold KPP: Demonstration of 99.9% agreement between

upgrade trigger electronics and software emulation of this electronics through test pattern injection based on data taken after LS1 followed by demonstration of reduction of calorimeter and endcap muon trigger rates for electrons, photons, muons and taus with respect to the present system by a factor of two for a reduction of less than 15% in efficiency using the trigger emulator run on data taken after LS1. Incorporation of unganged ME1/1a data into the endcap muon trigger logic.

Trigger objective KPP: Demonstration of 99.99% agreement between upgrade trigger electronics and software emulation of this electronics through test pattern injection based on data taken after LS1 followed by demonstration of reduction of calorimeter and endcap muon trigger rates for electrons, photons, muons and taus with respect to the present system by a factor of two for a reduction of less than 10% in efficiency using the trigger emulator run on data taken after LS1. Calorimeter Trigger electron and photon position resolution improved from DeltaEta=DeltaPhi=0.35 to DeltaEta=DeltaPhi=0.05. Incorporation of unganged ME1/1a data into the endcap muon trigger logic.

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 30

RisksMPC installation incomplete by end LS1

• Consequence: unable to provide full inputs to upgrade CTCTF• Mitigation: installation of remaining MPC mezzanines during

2015-2016 YETS.oSLB-oRM installation not complete by end LS1

• Consequence: full parallel operation of final calorimeter trigger not possible until 2018

• Mitigation: use parallel operation involving stage-1 calorimeter trigger hardware (oRSC) or use slice for validation

Full HCAL μHTR system not commissioned by end 2016• Consequence: operation of final calorimeter trigger not

possible during 2016• Mitigation: Continue to use stage-1 upgrade calorimeter

trigger hardware (oRSC).

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 31

BACKUP

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 32

Cal. Trig. Stage 1

UK

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 33

Muon Trigger WBS401.04.01 Trigger Milestones & Interfaces 401.04.03.05 Endcap Muon Track-Finder401.04.02 Trigger Management 401.04.03.05.01 EMUTF Production Manage401.04.03 Muon Trigger 401.04.03.05.02 EMUTF Software401.04.03.01 Muon Trigger Milestones & Interfaces 401.04.03.05.03 EMUTF Firrmware401.04.03.02 Muon Trigger Management 401.04.03.05.04 EMUTF Components401.04.03.03 Muon Port Card Mezzanine 401.04.03.05.04.01 EMUTF Optics401.04.03.03.01 MPCM Production Manage 401.04.03.05.04.02 EMUTF FPGAs401.04.03.03.02 MPCM Software 401.04.03.05.04.03 EMUTF Misc. Components401.04.03.03.03 MPCM Firmware 401.04.03.05.05 EMUTF PCB Fabrication401.04.03.03.04 MPCM Components 401.04.03.05.05.01 EMUTF Processing Card Fab.401.04.03.03.04.01 MPCM Optics 401.04.03.05.05.02 EMUTF Optics Card Fab.401.04.03.03.04.02 MPCM FPGAs 401.04.03.05.05.03 EMUTF Memory Card Fab.401.04.03.03.04.03 MPCM Misc. Components 401.04.03.05.05.04 EMUTF Backplane Connector Fab.401.04.03.03.05 MPCM PCB Fabrication 401.04.03.05.06 EMUTF Assembly401.04.03.03.06 MPCM Assembly 401.04.03.05.06.01 EMUTF Processing Card Assy.401.04.03.03.07 MPCM Testing 401.04.03.05.06.02 EMUTF Optics Card Assy.401.04.03.03.08 MPCM Ship 401.04.03.05.06.03 EMUTF Memory Card Assy.401.04.03.03.09 MPCM on-site Testing 401.04.03.05.06.04 EMUTF Backplane Connector Assy.401.04.03.03.10 MPCM Installation 401.04.03.05.07 EMUTF Testing401.04.03.03.11 MPCM Comissioning 401.04.03.05.08 EMUTF Ship401.04.03.04 MPC-EMUTF Optical Fibers 401.04.03.05.09 EMUTF on-site Testing401.04.03.04.01 MPC-EMUTF OF Production Manage 401.04.03.05.10 EMUTF Installation401.04.03.04.02 MPC-EMUTF OF Components 401.04.03.05.11 EMUTF Comissioning401.04.03.04.03 MPC-EMUTF OF Fabrication 401.04.03.06 EMU TF Infrastructure401.04.03.04.04 MPC-EMUTF OF Testing 401.04.03.06.01 EMUTF Infrastructure Manage401.04.03.04.05 MPC-EMUTF OF Ship 401.04.03.06.02 EMUTF Vadatech Crates401.04.03.04.06 MPC-EMUTF OF Installation 401.04.03.06.03 EMUTF AMC13401.04.03.04.07 MPC-EMUTF OF Comissioning 401.04.03.06.04 EMUTF Optical Splitters & Patch Panel

401.04.03.06.05 EMUTF Infrastucture Testing401.04.03.06.06 EMUTF Infrastructure Ship401.04.03.06.07 EMUTF Infrastructure on-site Testing401.04.03.06.08 EMUTF Infrastructure Installation401.04.03.06.09 EMUTF Infrastructure Comissioning

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 34

Calorimeter Trigger WBS401.04.04 Calorimeter Trigger 401.04.04.05 CTP Infrastructure401.04.04.01 Calorimeter Trigger Milestones & Interfaces 401.04.04.05.01 CTP Infrastructure Manage401.04.04.02 Calorimeter Trigger Management 401.04.04.05.02 CTP Vadatech Crates401.04.04.03 Calorimeter Trigger Processor 401.04.04.05.03 CTP AMC13401.04.04.03.01 CTP7 Production Manage 401.04.04.05.04 CTP Optical Cables & Patch Panel401.04.04.03.02 CTP7 Software 401.04.04.05.05 CTP Infrastucture Testing401.04.04.03.03 CTP7 Firmware 401.04.04.05.06 CTP Infrastructure Ship401.04.04.03.04 CTP7 Components 401.04.04.05.07 CTP Infrastructure on-site Testing401.04.04.03.04.01 CTP7 Optics 401.04.04.05.08 CTP Infrastructure Installation401.04.04.03.04.02 CTP7 FPGAs 401.04.04.05.09 CTP Infrastucture Comissioning401.04.04.03.04.03 CTP7 Misc. Components 401.04.04.06 Optical Receiver Mezzanines401.04.04.03.05 CTP7 PCB Fabrication 401.04.04.06.01 oRM Production Manage401.04.04.03.06 CTP7 Assembly 401.04.04.06.02 oRM Software401.04.04.03.07 CTP7 Testing 401.04.04.06.03 oRM Firmware401.04.04.03.08 CTP7 Ship 401.04.04.06.04 oRM Components401.04.04.03.09 CTP7 on-site Testing 401.04.04.06.04.01 oRM Optics401.04.04.03.10 CTP7 Installation 401.04.04.06.04.02 oRM FPGAs401.04.04.03.11 CTP7 Comissioning 401.04.04.06.04.03 oRM Misc. Components401.04.04.04 CIOx Card 401.04.04.06.05 oRM PCB Fabrication401.04.04.04.01 CIOx Production Manage 401.04.04.06.06 oRM Assembly401.04.04.04.02 CIOx Software 401.04.04.06.07 oRM Testing401.04.04.04.03 CIOx Firmware 401.04.04.06.08 oRM Ship401.04.04.04.04 CIOx Components 401.04.04.06.09 oRM on-site Testing401.04.04.04.04.01 CIOx Optics 401.04.04.06.10 oRM Installation401.04.04.04.04.02 CIOx FPGAs 401.04.04.06.11 oRM Comissioning401.04.04.04.04.03 CIOx Misc. Components 401.04.04.07 oSLB-oRM Optical Fibers401.04.04.04.05 CIOx PCB Fabrication 401.04.04.07.01 oSLB-oRM Production Manage401.04.04.04.06 CIOx Assembly 401.04.04.07.02 oSLB-oRM OF Components401.04.04.04.07 CIOx Testing 401.04.04.07.03 oSLB-oRM OF Fabrication401.04.04.04.08 CIOx Ship 401.04.04.07.04 oSLB-oRM OF Testing401.04.04.04.09 CIOx on-site Testing 401.04.04.07.05 oSLB-oRM OF Ship401.04.04.04.10 CIOx Installation 401.04.04.07.06 oSLB-oRM OF Installation401.04.04.04.11 CIOx Comissioning 401.04.04.07.07 oSLB-oRM OF Comissioning

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 35

Calorimeter Trigger PrimitivesPresent Connections

RegionalCalorimeter

Trigger To DAQ

Via GCT

HCAL HTR Cards

To DAQ

ViaHCALDCC2

Existing Copper Cables

ECAL TCCsTo DAQ

Via ECALDCC

Existing Copper Cables

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 36

Calorimeter Trigger PrimitivesConnection Evolution

RegionalCalorimeter

Trigger

HCAL uHTR Cards

ECAL TCCsSLHC

Cal TriggerProcessor

CardsOptical Ribbons

OSLBs

ORMs

TriggerPrimitiveOptical

Patch Panel

ECAL Opti. RibbonsECAL

Indiv. Fibers

(LC)

Optical Ribbons

To DAQ

Via BU“AMC13”

To DAQ

Via GCT

To DAQ

Via BU“AMC13”

To DAQ

Via ECALDCC

HCAL HTR Cards

To DAQ

ViaHCALDCC2

Existing Copper Cables

HCAL Opti. Ribbons

Optical Splitter Upstream of

HCAL HTR/uHRT

Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 37

Calorimeter Trigger PrimitivesFinal Situation

HCAL uHTR Cards

ECAL TCCsSLHC

Cal TriggerProcessor

CardsOptical Ribbons

OSLBs

ECAL Opti. Ribbons

Optical Ribbons

To DAQ

Via BU“AMC13”

To DAQ

Via BU“AMC13”

To DAQ

Via ECALDCC

HCAL Opti. Ribbons

TriggerPrimitiveOptical

Patch Panel