The Next Steps - Upgrading L1Calo: Why, What, When and Who An essentially random set of borrowed and...
-
Upload
roxanne-shelton -
Category
Documents
-
view
214 -
download
1
Transcript of The Next Steps - Upgrading L1Calo: Why, What, When and Who An essentially random set of borrowed and...
The Next Steps -Upgrading L1Calo: Why, What,
When and Who
An essentially random set of borrowed and recycled slides to provide a flavour of
the planned work
Where I’ve borrowed, many thanks
Where I’ve used my own and far better exist elsewhere, many apologies
L1Calo Upgrade: Why
We worked hard for 15+ years to get here, where did we go wrong?
The obvious answer: LHCand the physics available
• LHC is now the main flagship for particle physics – For many years to come– Pressure to squeeze as many physics results out as is
humanly possible• Mass of Higgs is in a very interesting, but tricky
region– Ambition of analyses exceeds pre-LHC expectations
• Hope of any other ‘easy’ discoveries is fading– Need to maintain performance at lower energies as
Higgs/EW physics. It may be all we’ll get
LHC Schedule
• I got this from a Phil Allport talk in July, so it must be right
Typical quotedluminosities*:
Run 2: 1-2 x design
Run 3: 2-3 x design
Run 4+: 5 x design
design = 1034 cm-2s-1
* could be on low side
Focussing on TDAQ
• The (original) TDR aims were well known and well motivated
• But even in 2012-2013 we were pushing beyond the original rate goals
• Except at Level-1, as HARD hardware limited– As distinct from the
rest of TDAQ
Run 1 Realities• Moore’s Law helps HLT +
Offline• Doesn’t do much for Level-1
Level-1Max 75 kHz,
Typical 60 kHz
Level-2Max 5.5 kHz,Typical 4 kHz
Event FilterMax 600 Hz,
Typical 400 Hz
Also watch out for dead-time
• Even when we did run at 75 kHz, dead-time started to creep up
• Aim for 2015 is to try to go faster
• But I have my doubts
The consequence
• The ATLAS trigger performance depends increasingly on Level-1
Level-1 1/500
Level-2 1/20
Evt Filter 1/20
DESIGN
Level-1 1/500
Level-2 1/10
Evt Filter 1/10
RUN 1
• And that’s before the planned HLT upgrades…
HLT beyond 2015
• More hardware with more nodes (of course)• Distinction Level-2 and EF is blurred/removed
– A more incremental approach to algorithms– Along with an increased event building capacity
• Baseline recording rate increased to 1 kHz– With potential for further data ‘parking’ and an offline Level-4 trigger
• Effectively picture could look something like below– And who knows how much more capacity in 2020+
Level-1 1/500
‘Level-2’ 1/7
‘Evt Filter’ 1/7
RUN 1
L1Calo response
• Increasing pressure to improve selection• Several different strands possible
– Address pile-up dependency– More flexibility– Introduce topology– Use more detailed calorimeter information
• When all else fails– Increase Level-1 rate (and call it Level-0!)– With all the detector work that implies
L1Calo Upgrade: What
You will be upgraded. You will become like us.*
*A. Cyberman
L1Calo Rates: the Good, the Badand Missing Energy
Well behaved triggers scale with luminosity– Useful to study trigger ‘cross-section’ as
function of pile-up factor <μ>
Triggers affected by pile-up very dependent on LHC bunch structure– Typically missing energy, forward
jets, multi-jets
Missing Energy: the Pile-Up effect
• L1Calo experiences pedestal shifts due to unbalanced overlaying of signals at the start of the train– This was something we always expected– But perhaps we underestimated the extent– And the horrified reaction from others!
• On-the-fly pedestal correction reduces the problem enormously– Also different types of filters can help too
• This sounded the death knell for the final major ASIC in the system– But gave a lovely new project for Heidelberg
The New Multi-Chip Module
USA 15 system alreadyupgraded with nMCMsrunning in legacy mode
FPGAs: Our Flexible Friends
• Elsewhere FPGAs dominated the old system– Which has definitely proved useful– Already in Run 1– But also for current upgrades
• CPM and JEM will send more detailed information on backplane– Mostly used for topological information– But also allows, for example, isolation fraction cuts
• Required the replacement of the CMM– Which can produce more, and better, triggers
CMX: what does it stand for?
• The CMX breaks the mould of L1Calo modules by being green!
• More modules appearing all the time in USA 15
Towards Topology
• The CMX adds extra functionality and flexibility to the original counting triggers
• Crucially it also provides a new data path to a ‘Topo Box’
• The Topo module adds topological functionality that runs on the current RoIs– It is also built with further upgrade inputs in mind
• In the absence of tracking information at Level-1, this provides another useful lever
The Topo Box, or Boxes
• First use of modern optical links in L1Calo (6.4 Gb/s)• Pioneering ATCA usage (replacement for VME)
– Installed recently in USA15 (first ATCA in ATLAS?)
If you thought 6.4 Gb/s was fast…
• Longer term upgrade:– Use more detailed calorimeter information– Typically each tower becomes 10 ‘super-cells’– Digitisation on detector, optical data received at
speeds of up to ~12 Gb/s
Two become Three:eFEX, jFEX and gFEX
The (not so) far future
• For Run4+, many ATLAS detectors need to be replaced• Potential Options for changing underlying trigger
architecture– Include track trigger– Introduce a higher rate Level-0
• Which means we can ‘relax’, a la HLT and let the upgraded Level-1 (now Level-0) run at a higher rate
• Need to define what the new Level-1 is like– Details are being hammered out, but this is still a very open area– Eg Protocols, Rates– May be possible to implement more software like algorithms in
GPUs, also use of CAMs
L1Calo Upgrade: When
No Gantt charts allowed!More a history of upgrade evolution
L1Calo, the next ten years
• Long Shutdown 1 (NOW)– MCM becomes nMCM– Introduce CMX to RX/TX RoIs rather than hits– Add Topological Trigger
• Long Shutdown 2 (2018-2019)– New digital high granularity trigger towers (super-cells)– Digital eFEX, jFEX ang gFEX run initially in parallel with legacy-L1Calo
• Only LARg for Run 3
– Topological trigger augmented for new signals
• Long Shutdown 3 (2024-2025)– Level-1 split into two stage trigger (L0 and L1)– FEX systems fully populated including Tile and used as L0Calo– New L1Calo fed from detector readout information
L1Calo Upgrade:the early days
• (Probably) the first serious attempt to think ahead happened right here in January 2008:– https://indico.cern.ch/event/24112/
• Here’s an interesting slide, what do we learn?
a) I haven’t changed my PPT styleb) We were still relying on Eric…c) … and Wesleyd) Most of the points are accuratee) Weren’t yet thinking Topology
Rest of 2008
• Not surprisingly commissioning was the major focus
• Nevertheless, thoughts in several institutes were looking at directions we have now taken:– High speed optical link research– (µ)ATCA developments– Clock jitter cleaning
• Also new institutes starting to take an interest
RAL and Mainz 2009:The Upgrade comes of age
• (Maybe because LHC was broken?)• Argonne, Cambridge and MSU officially accepted as
new upgrade collaborators• First mention of nMCM• Backplane tester• Discussion of CMM speeds…• …and alternatives• By the end of Mainz meeting (June), seeds of
current (LS1) upgrade were sewn• And Eric then retired
L1Calo: Generations
What about calorimeters?
• Francesco attended the meeting in January 2008
• Ideas became more focussed after a Tile Upgrade meeting in February 2008
• Idea was then to be on LS3 time-scale
• Discussions on this option continued regularly with both Tile and LARg
Then 2011 came along
• The idea of the trigger dedicated DPS appeared– Largely motivated by electron selectivity, though
with lots of other potential too• I clearly remember saying to many people that
I expected no calorimeter upgrade on the LS2 time-scale– I was utterly wrong!
• Though there were rumblings before, it mostly erupted at Stockholm meeting in June
The ubiquitous diagram
The project blossomed!
• Ambitious, large scale, high speed, digital electronics– Exactly what the doctor ordered
• There was/is/will be one big problem– Timescale– For such an ambitious project, it seemed a little late to be starting– And our record on sticking to deadlines is not perfect
• Still for many reasons, the show had to go on– So draw up the Gantt charts, and cross the fingers
• For many years (1995-2008) I wondered how it was we always produced impossible looking schedules– Which we then invariably failed to meet
• Now I know how it happens
The collaboration also blossomed!
Moving on rapidly
• Thoughts on sATLAS and sLHC continued• Nikos and others have been pushing Track Trigger for
many years• But it was never clear if it could contribute at the current
Level-1 latency• Another approach was needed: Level-0• The idea had been around for some time, but first
presented in SLAC, November 2010– There were no great objections– But then maybe no-one understood it– In fact I’m still not sure I do
As presented back then
L0A: 500 kHz 2.5 µsL1A: 100 kHz 6.4 µs
• Largely speaking this architecture has been accepted– Or something like it (rates, and particularly latencies
changed/changing)• Many of us are still finding our feet with the implications
Developments
Anotherslide from
2010
L1Calo
The Lament of the Level-1 ‘Expert’
The lament of the Level-1 expert (1)
• Level-1 has to do more rejection than HLT– In 2012, Level-1 took ~15 MHz input down to 75 kHz
• 75 kHz is the advertised figure, actually more like 65 kHz, constrained by detector readout
– Whereas HLT is allowed to output up to 1 kHz• Data can be buffered for processing during quiet periods (eg LS1)
• And yet is also expected to achieve near 100% efficiency– HLT is allowed to at least 5% inefficient for important signals– HLT is better understood so efficiency can be compared more easily to
Monte-Carlo. (So people keep telling me.)
The lament of the Level-1 expert (2)
• Everyone compares to offline reconstruction• There’s no MC ‘truth’ in real data
– Normally I’m all in favour of data driven analyses• But in the absence of ‘truth’ people tend to assume offline energy is the truth
– Sadly this is not always the case– Sometimes the trigger path works better than the full detector data path– And both routes can make mistakes in the presence of large pile-up
• HLT uses offline-like quantities from detector readout, so is less affected by these expectations (ie HLT and offline make the same mistakes)
Detector Path OK OK Broken Broken
Trigger Path OK Broken OK Broken
Common Perception
Level-1 might be
right
Level-1 is wrong
Level-1 is wrong
Level-1 is wrong
We say Phew It’s a fair cop Hang on a moment
Not our fault!
The lament of the Level-1 expert (3)
• Common complaint: Level-1 is the limiting factor for many triggers– There’s much truth in this
• Hence we’re often under a lot of pressure
– HLT gains hugely by having tracking information
• But it’s not true for everything– For example photon and two-photon channels
• (I’m reliably informed that the two-photon channel has been quite interesting)
– High-pT jets are not a problem
• And it’s partly due to a lack of adventure!– Level-1 is expected to be ‘stable’, can’t change too often
• Tuning for high-pileup was delayed and could probably be improved
– Reluctance to use features built in exactly to keep rates down• Eg electron isolation
L1Calo Upgrade: Who
You know those rock-group line-up charts?
Well this isn’t one of those
The original line-up(1996ish-2009)
• Birmingham on the CPM• Heidelberg underlying everything with the PPM• Mainz accompanying with the larger scale JEM• QMUL: management and programming• RAL providing rhythm and support with the
TCM, CMM and ROD• Stockholm providing the flourish and firmware
The first upgrade wave
• Joining in March 2009– Argonne (and occasionally leaving)– Cambridge– MSU
• Joining in March 2011– Berlin Humboldt
The Upgrade-TDR wave
• Completing the set in 2013– Brookhaven – Stony Brook– Krakow (Jagiellonian)– Oregan– (UBC/Wojtek)
L1Calo Upgrade: The Collaboration
L1Calo Collaboration
Some Last Words
• Digitisation: I still believe, that an 8 bit FADC is sufficient for threshold-settings in Processors.
• [ It is winter: there are no tomatoes to throw. ]
From Paul Hanke
From Steve
It’s been wonderful working with you.
Thanks for re-introducing me to ‘The Now Show’.