Calibration streams in the Event Filter. Status report Mainz, Thursday 13 October 2005 Sander Klous...
-
Upload
kerry-moore -
Category
Documents
-
view
215 -
download
1
Transcript of Calibration streams in the Event Filter. Status report Mainz, Thursday 13 October 2005 Sander Klous...
![Page 1: Calibration streams in the Event Filter. Status report Mainz, Thursday 13 October 2005 Sander Klous – NIKHEF On behalf of the EF calibration team: Martine.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2359/html5/thumbnails/1.jpg)
Calibration streams in the Event Filter.
Status report
Mainz, Thursday 13 October 2005
Sander Klous – NIKHEFOn behalf of the EF calibration team:
Martine Bosman, Andrea Negri, Serge Sushkov and Sarah Wheeler.
![Page 2: Calibration streams in the Event Filter. Status report Mainz, Thursday 13 October 2005 Sander Klous – NIKHEF On behalf of the EF calibration team: Martine.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2359/html5/thumbnails/2.jpg)
13 October 2005 TDAQ workshop - Mainz 2
Physics streams40 MHz x 1.5 MByte
Level 1
Level 2
75 kHz x 1.5 MByte
Definition and scope.
Calibration issues in the EF.– At the moment only EF output.
• Identify calibration types.– Size, rate, contents.
• Requirements for the EF.– Data flow.– Processing issues.
• Implementation scenarios.• Design and modifications.
– Processing.– Memory management.– Networking/Timing.
• Plan of work.
Processing time:1 Sec/Evt
Number of nodes:1500
Output: 200 Hz320 MB/s
Calibrationstreams
Level 1
Level 2
Other people
Event Filter
?2 kHz x 1.5 MByte
![Page 3: Calibration streams in the Event Filter. Status report Mainz, Thursday 13 October 2005 Sander Klous – NIKHEF On behalf of the EF calibration team: Martine.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2359/html5/thumbnails/3.jpg)
13 October 2005 TDAQ workshop - Mainz 3
Use cases(identification of calibration streams).
Based on the Hawkings/Gianotti document. – All listed calibration types are identified at HLT level (after PESA).
• Known calibration types in the HLT.– Various duplicates of physics streams: e.g. inclusive high pT electrons and
muons (tracking), Z to di-lepton (energy), minimum bias (background).• Total: 35 MB/s (10% of physics data).
– Liquid Argon Calorimeter.Pulse shape analysis, timing calibration and tuning of filter coefficients.
• High pT electron sample• Electro Magnetic data only. ROI only.• Raw data, 5 consecutive samples (i.e. special event type).
– Calorimeters and TRT.Hadronic response studies, comparison to test beam data.TRT: e/ separation.
• High pT isolated hadrons.• All subdetectors. ROI only.• RAW data.
![Page 4: Calibration streams in the Event Filter. Status report Mainz, Thursday 13 October 2005 Sander Klous – NIKHEF On behalf of the EF calibration team: Martine.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2359/html5/thumbnails/4.jpg)
13 October 2005 TDAQ workshop - Mainz 4
Use cases – continued…(identification of calibration streams).
– MDT small chambers.Hourly realignment.
• Small muon sample.• MDT information only. Overlap regions only.• Reprocessing of raw data.
– Inner Detector subdetectors (Pixel, SCT and TRT).ROD monitoring (TRT only) and alignment.
• Generic high pT events.• All subdetectors. ROI only.• Post-processing of track fit information on HLT level.
• Other foreseen calibration types.– Liquid Argon Calorimeter might need Z ee calibration at HLT level.– High statistics (1 kHz) ROI muon sample, containing MDT, CSC and
RPC/TGC information.– Your favorite missing calibration stream…
![Page 5: Calibration streams in the Event Filter. Status report Mainz, Thursday 13 October 2005 Sander Klous – NIKHEF On behalf of the EF calibration team: Martine.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2359/html5/thumbnails/5.jpg)
Special events
EF Node n
EF Node n Transporttime: 19 ms
EF Node n
Data flow characterization.
SFO
320 MB/s
Physics
Full calibration events
SFO
32 MB/s
High pT
SFO
1.6 MB/s
Z di-lepton
SFO
1.6 MB/s
Min. bias
EF processing: 1 second/event
SFOSFO
5 kB/s
OverlapMu
2.5 MB/s
LAr
SFO
2 MB/s
IsoHad
SFO
4 MB/s
GenPT
Stripping/CollectingPartial calibration events
Transport times/rates:LArOverlapMuIsoHadGenPT
0.5 ms0.05 ms
5 ms0.5 ms
50 Hz5 Hz5 Hz
100 Hz
Output
1 MB/s
LVL2Cal
SubdetectorFragment 1
ROI info
Transport time:LVL2Cal 0.01 ms
Partial event
SFI?
Transport times: 19 msSubdetectorFragment 1
SubdetectorFragment N
Lvl 1/2 info
Lvl 1/2 info
SFI
AdditionalProcessing
1 kHz!
Sorting
EF processing:1 second/event
![Page 6: Calibration streams in the Event Filter. Status report Mainz, Thursday 13 October 2005 Sander Klous – NIKHEF On behalf of the EF calibration team: Martine.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2359/html5/thumbnails/6.jpg)
13 October 2005 TDAQ workshop - Mainz 6
EF processing issues.• Definitions:– Sorting for calibration: CalID.– Stripping/Collecting for calibration: CalCollect.– Detector calibration: CalDetect or Calibration.
• Full event streams.– Very similar to physics streams.– Output to multiple SFOs/streams.– Sorting of events for calibration (CalID).
• Partial event streams.– Processing similar to physics stream.– Stripping and collecting (CalCollect).– Handling of different output event size.– Sometimes requires additional processing
(CalDetect/Calibration).
• Special event streams.– Processing times completely different from
physics stream.– Handling of different input and output event
sizes.
• Central issue: Robustness of the EFD
Main output stream
SFO
Diagnostic
SFO
Node n
EFD
PT#1
PTIO
SFI
Input
ExtPTs
OutputOutput
Trash
ExtPTsPT
cal
PTIO
Calibration Stream
Output
SFO
PT#2
PTIO
EventResult
SharedHeap
![Page 7: Calibration streams in the Event Filter. Status report Mainz, Thursday 13 October 2005 Sander Klous – NIKHEF On behalf of the EF calibration team: Martine.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2359/html5/thumbnails/7.jpg)
13 October 2005 TDAQ workshop - Mainz 7
Main output stream
SFO
EF processing issues.
Node n
EFD
SFI
Input
ExtPTs
Output
Event
SharedHeap
Dataflow application
• Definitions:– Sorting for calibration: CalID.– Stripping/Collecting for calibration: CalCollect.– Detector calibration: CalDetect or Calibration.
• Full event streams.– Very similar to physics streams.– Output to multiple SFOs/streams.– Sorting of events for calibration (CalID).
• Partial event streams.– Processing similar to physics stream.– Stripping and collecting (CalCollect).– Handling of different output event size.– Sometimes requires additional processing
(CalDetect/Calibration).
• Special event streams.– Processing times completely different from
physics stream.– Handling of different input and output event
sizes.
• Central issue: Robustness of the EFD
![Page 8: Calibration streams in the Event Filter. Status report Mainz, Thursday 13 October 2005 Sander Klous – NIKHEF On behalf of the EF calibration team: Martine.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2359/html5/thumbnails/8.jpg)
13 October 2005 TDAQ workshop - Mainz 8
Calibration stream scenarios (1).
Additional functionality:• CalID algorithm.
• Parallel output streams.
Node n
EFD
SFI
Input
ExtPTs
Trash
Main output stream
SFO
Output
EventResult
Physics only events Full calibration events. e.g. Z di-lepton
PT#1
PTIO
PESA
Main output stream
SFO
Node n
EFD
SFI
Input
ExtPTs
OutputTrash
EventResult
PT#1
PESA
PTIO
Calibration Stream
SFO
Output
CalID
![Page 9: Calibration streams in the Event Filter. Status report Mainz, Thursday 13 October 2005 Sander Klous – NIKHEF On behalf of the EF calibration team: Martine.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2359/html5/thumbnails/9.jpg)
Calibration Stream 1
Main output stream
SFO SFO
Node n
EFD
SFI
Input
ExtPTs
OutputOutput
EventResult
PT#1
PESACalID
PTIO
Calibration stream scenarios (2).
Additional functionality:• PT for calibration.
– Information handling– Stripping/collecting.
• Memory management.
Partial calibration events. e.g. GenPT Node n
EFD
SFI?
Input
ExtPTs
Event
Calibration Stream
EF output
Output
PT#1
CalID
PTIO
Calibration Stream 2
Output
SFO
ExtPTs
PT cal P
TIO StrippingCollecting
CalResult
Special streams. e.g. LVL2Cal
Calibration
• Networking/Timing issues.
Sorting
![Page 10: Calibration streams in the Event Filter. Status report Mainz, Thursday 13 October 2005 Sander Klous – NIKHEF On behalf of the EF calibration team: Martine.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2359/html5/thumbnails/10.jpg)
13 October 2005 TDAQ workshop - Mainz 10
Design and modifications (1).• CalID algorithm.
– Lightweight algorithm.– Runs after PESA in physics PT – Stability issues.
Athena configuration: multiple top algorithms.
– Workload:Low – implementation only, thorough testing required.
– Impact:High – required for (almost) all calibration streams.
– Coordination:Sorting – New PT answers should be discussed.
• Parallel output streams.– Slight modification of existing algorithm.– Runs in EFD, probably required for PESA as well.– Workload:
Low – Modification of standard EFD task.
– Impact:High – Required for most (calibration) streams.
![Page 11: Calibration streams in the Event Filter. Status report Mainz, Thursday 13 October 2005 Sander Klous – NIKHEF On behalf of the EF calibration team: Martine.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2359/html5/thumbnails/11.jpg)
13 October 2005 TDAQ workshop - Mainz 11
Design and modifications (2).• PT for calibration.
1. Create stripping/collection algorithm.• Requires new eformat (see next slide).• Requires modifications in output task.
2. Allow multiple PTs to run consecutively (works already).3. Transfer information between these PTs.
– Should be possible with new eformat for EFResult / CalResult.
4. It might be interesting to transfer “intermediate results”.– Would avoid to
• Run calibration algorithms in the same PT as PESA.• Reanalyze complete event in second PT.
– Since EF is a dataflow application, this should be accomplished by writing an extra “Intermediate Result” object in the Shared Heap.• This requires ByteStream conversion for complex classes.
– Workload:Medium – With exception of item 4 (no use cases yet).
– Impact:High – Maybe with exception of item 4.
![Page 12: Calibration streams in the Event Filter. Status report Mainz, Thursday 13 October 2005 Sander Klous – NIKHEF On behalf of the EF calibration team: Martine.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2359/html5/thumbnails/12.jpg)
13 October 2005 TDAQ workshop - Mainz 12
Memory management and information handling.
New eformat.
Node n EFD
SFI
Input
ExtPTs
ExtPTs
SharedHeap
EventEFResult
PT#1
PESACalID
PTIO
CalResult
SFO
Output
PT cal
PTIOStripping
Collecting
1 0 1 0 0 1
- x - 1 - 0 x
1 1 1 0 0 1
• Integrate with:– Virtual event.
– SharedHeap.
– Event handling in the EF.
– Event modification in EF, i.e. stripping/collecting.
Event fragments
1 0 1 0 0 1 1
1 0 1 0 0 1 1
EFResult
- - - - - - - 1
Stripping
CalResult
Virtual event
![Page 13: Calibration streams in the Event Filter. Status report Mainz, Thursday 13 October 2005 Sander Klous – NIKHEF On behalf of the EF calibration team: Martine.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2359/html5/thumbnails/13.jpg)
13 October 2005 TDAQ workshop - Mainz 13
Open issuesMany new developments on a very tight schedule.• Memory and performance.
– Management: move from open/close backpressure mechanism (barrier) to analog (Nano sleeps).
– Timing: revise SFI – EFD – SFO protocol.
• Coordination…– Investigate common/similar design issues with monitoring– 128 bit header word. First discussion yesterday.
• 32 bits to register appropriate output streams.• Investigate usage of this header. Streaming only, since EFResult fragment
contains much more detail and is “just around the corner”.
– PT answer to EFD. Composite structure.– EFD – SFO sorting, i.e. how is an output stream defined?
• Load balancing between calibration and physics.• Distribution of calibration constants to EF software.
– Communication between Athena algorithms and configuration and calibration databases.
– Could this be done via e.g. the information service?
![Page 14: Calibration streams in the Event Filter. Status report Mainz, Thursday 13 October 2005 Sander Klous – NIKHEF On behalf of the EF calibration team: Martine.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2359/html5/thumbnails/14.jpg)
13 October 2005 TDAQ workshop - Mainz 14
Plan of work.• Short time scale.
– Run calibration algorithm in PT.– Implement parallel output streams.
• Medium time scale.– Implement CalID algorithm and sorting.– Eliminate dead time in input and output tasks.
• Medium – Long timescale.– Change memory management.– Implement new eformat.– Implement stripping/collecting.
• Long timescale.– Transfer of intermediate results between first and second PT
![Page 15: Calibration streams in the Event Filter. Status report Mainz, Thursday 13 October 2005 Sander Klous – NIKHEF On behalf of the EF calibration team: Martine.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2359/html5/thumbnails/15.jpg)
13 October 2005 TDAQ workshop - Mainz 15
Conclusions.
• Our understanding of calibration streams in the Event Filter improved a lot.
• We think we have a realistic overview of the workload involved in modifications of the Event Filter.
• Implementation has started and this will lead to even better understanding of the topic (and of the work involved).
• There are still some (many) open issues. Coordination is important, especially because of the tight schedule.
• More information:https://uimon.cern.ch/twiki/bin/view/Atlas/EventFilterCalibration
![Page 16: Calibration streams in the Event Filter. Status report Mainz, Thursday 13 October 2005 Sander Klous – NIKHEF On behalf of the EF calibration team: Martine.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2359/html5/thumbnails/16.jpg)
13 October 2005 TDAQ workshop - Mainz 16
Conclusions.
• Our understanding of calibration streams in the Event Filter improved a lot.
• We think we have a realistic overview of the workload involved in modifications of the Event Filter.
• Implementation has started and this will lead to even better understanding of the topic (and of the work involved).
• There are still some (many) open issues. Coordination is important, especially because of the tight schedule.
• More information:https://uimon.cern.ch/twiki/bin/view/Atlas/EventFilterCalibration
![Page 17: Calibration streams in the Event Filter. Status report Mainz, Thursday 13 October 2005 Sander Klous – NIKHEF On behalf of the EF calibration team: Martine.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2359/html5/thumbnails/17.jpg)
13 October 2005 TDAQ workshop - Mainz 17
Appendix
![Page 18: Calibration streams in the Event Filter. Status report Mainz, Thursday 13 October 2005 Sander Klous – NIKHEF On behalf of the EF calibration team: Martine.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2359/html5/thumbnails/18.jpg)
13 October 2005 TDAQ workshop - Mainz 18
A distributed trigger for calibration?Appears to fit with solutions shown in LVL2Mu presentations.• Ultralight project (Manuela Cirilli).• LVL2Mu calibration stream (Speranza Falciano).• Etc. (Enrico Pasqualucci, Alessandro de Salvo).
Additional functionality:
• HLT output
Node n
EFD
SFI?
Input
ExtPTs
EventPT#1 Event
Distributor
PTIO
Output
Moore’s law for networkingGary Stix, Scientific American, January 2001
The Event Filter is CPU dominated. You would like it to be bandwidth dominated…
Calibration Stream
HLToutput
![Page 19: Calibration streams in the Event Filter. Status report Mainz, Thursday 13 October 2005 Sander Klous – NIKHEF On behalf of the EF calibration team: Martine.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2359/html5/thumbnails/19.jpg)
13 October 2005 TDAQ workshop - Mainz 19
ByteStream conversion
• Write converters?– No, a lot of work.
– Robustness issues.
• Generic ByteStream conversion?– No support for complicated classes
(e.g. multiple inheritance, polymorphism).
• Something else?• Not on the priority list.• Under discussion…
![Page 20: Calibration streams in the Event Filter. Status report Mainz, Thursday 13 October 2005 Sander Klous – NIKHEF On behalf of the EF calibration team: Martine.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2359/html5/thumbnails/20.jpg)
13 October 2005 TDAQ workshop - Mainz 20
Memory management and networking/timing
Node n EFD
SFI
Input
ExtPTs
ExtPTsPT
cal PTIO
SFO
SharedHeapEvent
EFResultEvent
EFResultEvent
EFResultCalResult
IntermediateResults
PT#1
PESACalID
PTIO
Virtual event
OutputStripping
Collecting
19 milliseconds transport time
19 milliseconds transport time
0.25 ms dead time
0.25 ms dead time
1+ second processing time
Barrier
![Page 21: Calibration streams in the Event Filter. Status report Mainz, Thursday 13 October 2005 Sander Klous – NIKHEF On behalf of the EF calibration team: Martine.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2359/html5/thumbnails/21.jpg)
13 October 2005 TDAQ workshop - Mainz 21
Memory management and networking/timing
Node n EFD
SFI
Input
ExtPTs
ExtPTsPT
cal PTIO
SFO
SharedHeapEvent
EFResultCalResult
EventEFResultCalResult
EventEFResultCalResult
IntermediateResults
IntermediateResults
IntermediateResults
PT#1
PESACalID
PTIO
Virtual event
Output
Barrier
0.01 ms transport time
0.01 ms transport time
25 ms dead time
25 ms dead time
0 sec. processing time
• Eliminate dead time.– Redesign of SFI – EFD – SFO
communication protocol.– Coordination with networking
group.
• Barrier is insufficient.– Oscillations.– Other memory requests.
Stripping
Collecting
![Page 22: Calibration streams in the Event Filter. Status report Mainz, Thursday 13 October 2005 Sander Klous – NIKHEF On behalf of the EF calibration team: Martine.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2359/html5/thumbnails/22.jpg)
13 October 2005 TDAQ workshop - Mainz 22
Memory management and networking/timing
Node n EFD
SFI
Input
ExtPTs
ExtPTsPT
cal PTIO
SFO
SharedHeapEvent
EFResultCalResult
EventEFResultCalResult
EventEFResultCalResult
IntermediateResults
IntermediateResults
IntermediateResults
PT#1
PESACalID
PTIO
Virtual event
Output
Barrier
0.01 ms transport time
0.01 ms transport time
25 ms dead time
25 ms dead time
0 sec. processing time
• Eliminate dead time.
• Barrier is insufficient.– Oscillations.
– Other memory requests.
Stripping
Collecting
![Page 23: Calibration streams in the Event Filter. Status report Mainz, Thursday 13 October 2005 Sander Klous – NIKHEF On behalf of the EF calibration team: Martine.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649eba5503460f94bc2359/html5/thumbnails/23.jpg)
13 October 2005 TDAQ workshop - Mainz 23
Memory management and networking/timing
Node n EFD
SFI
Input
ExtPTs
ExtPTsPT
cal PTIO
SFO
SharedHeapEvent
EFResultCalResult
EventEFResultCalResult
EventEFResultCalResult
IntermediateResults
IntermediateResults
IntermediateResults
PT#1
PESACalID
PTIO
Virtual event
Output
• Eliminate dead time.
• Barrier is insufficient.– Oscillations.
– Other memory requests.Nano sleeps
0.01 ms transport time
0.01 ms transport time
25 ms dead time
25 ms dead time
Nano sleeps
Stripping
Collecting
0 sec. processing time
• Solution: Nano sleeps,
However…
– multiple control loops.
– Risk of oscillations.
– Additional complexity.
• Workload:
High – Especially testing.
• Impact:
???