Framework
1
Evaluation of the Optical Link Card Evaluation of the Optical Link Card for the Phase II Upgrade of TileCal for the Phase II Upgrade of TileCal Detector Detector F. Carrió F. Carrió 1 , V. Castillo , V. Castillo 2 , A. Ferrer , A. Ferrer 2 , V. González , V. González 1 , E. Higón , E. Higón 2 , C. Marín , C. Marín 1 , , P. Moreno P. Moreno 2 , E. Sanchis , E. Sanchis 1 , C. Solans , C. Solans 2 , A. Valero , A. Valero 2 , J. Valls , J. Valls 2 1 Dept. of Electronic Engineering, Universitat de València, Spain Dept. of Electronic Engineering, Universitat de València, Spain 2 Dept. of Atomic, Molecular and Nuclear Physics, IFIC, Valencia, Spain Dept. of Atomic, Molecular and Nuclear Physics, IFIC, Valencia, Spain Framework Upgrade in ATLAS OLC Tests Conclusion and future work Specifications Specifications ROD double PU Board format (4.72 in x 6.85 in) ROD double PU Board format (4.72 in x 6.85 in) 12 input channels up to 75 Gbps 12 input channels up to 75 Gbps 12 output channels up to 75 Gbps 12 output channels up to 75 Gbps Optical Link Card Description Technical details Technical details 1 Altera Stratix II GX EP2SGX60E 1 Altera Stratix II GX EP2SGX60E FPGA FPGA 1152 pins 1152 pins 12 high speed transceivers @ 12 high speed transceivers @ 6.375Gbps 6.375Gbps 534 I/Os 534 I/Os 2 SNAP12 connectors (12 fibers @ 2 SNAP12 connectors (12 fibers @ 6.25Gbps) 6.25Gbps) 1 SFP connector @640Mbps 1 SFP connector @640Mbps 12 copper layers 12 copper layers Different thickness of Different thickness of dielectric: 2.5, 4 and 6 mils dielectric: 2.5, 4 and 6 mils Need high capacitance between Need high capacitance between planes but sufficient thickness planes but sufficient thickness for striplines for striplines Over 2000 routes 5 mils width Over 2000 routes 5 mils width Over 2400 vias 10 mils Over 2400 vias 10 mils Design details Design details Power Integrity techniques Power Integrity techniques Custom Power Distribution Network Custom Power Distribution Network with FDTIM method with FDTIM method High plane capacitance High plane capacitance Proper location for capacitor Proper location for capacitor Signal Integrity techniques Signal Integrity techniques Copper plane removal below Copper plane removal below coupling capacitor pads coupling capacitor pads Via design optimization Via design optimization Low inductance connections Low inductance connections Dielectric material selection Dielectric material selection Shielding lines Shielding lines Ground return vias Ground return vias GBT Implementation TileCal detector measures the energy of TileCal detector measures the energy of hadrons in ATLAS experiment hadrons in ATLAS experiment TileCal detector is divided in 64 modules in TileCal detector is divided in 64 modules in and 3 sections in z for a total of 256 modules and 3 sections in z for a total of 256 modules The readout is carried out using 10.000 The readout is carried out using 10.000 electronic channels which, after digitization, electronic channels which, after digitization, come out of the detector using optical fibers come out of the detector using optical fibers towards the Read Out Driver (ROD) module towards the Read Out Driver (ROD) module ATLAS requires radiation tolerant electronics capable of stable ATLAS requires radiation tolerant electronics capable of stable operation for, at least, 10 years operation for, at least, 10 years The TileCal phase II upgrade is focused on replacement of most of The TileCal phase II upgrade is focused on replacement of most of the readout electronics, including the links between on- and off- the readout electronics, including the links between on- and off- detector electronics. This development is driven by the detector electronics. This development is driven by the requirements for increased radiation tolerance, and the need to requirements for increased radiation tolerance, and the need to provide the level-1 trigger with more detailed information provide the level-1 trigger with more detailed information . . The off-detector electronics consists of the preprocessor and the The off-detector electronics consists of the preprocessor and the Read-Out-Drivers (RODs). The pre processor is responsible for Read-Out-Drivers (RODs). The pre processor is responsible for receiving the data from the on-detector electronics and preparing receiving the data from the on-detector electronics and preparing it for the level-1 trigger and for the RODs it for the level-1 trigger and for the RODs . . The upgrade Phase II of the ATLAS Tile Calorimeter The upgrade Phase II of the ATLAS Tile Calorimeter implies a complete redesign of the read-out electronics. implies a complete redesign of the read-out electronics. In order to increase radiation tolerance pipelines, In order to increase radiation tolerance pipelines, derandomizers and L1 trigger preprocessor will be placed derandomizers and L1 trigger preprocessor will be placed within the off-detector electronics. within the off-detector electronics. With this new architecture the data transmission for With this new architecture the data transmission for the read-out is one of the greatest challenges due to the read-out is one of the greatest challenges due to the large amount of data transferred out of the the large amount of data transferred out of the detector. Considering a sampling rate of 40Msps, bi-gain detector. Considering a sampling rate of 40Msps, bi-gain transmission, 12-bit samples and a total amount of 9856 transmission, 12-bit samples and a total amount of 9856 channels with redundancy read-out, the complete detector channels with redundancy read-out, the complete detector requires a total data bandwidth of around 20Tbps. requires a total data bandwidth of around 20Tbps. The Read-Out Driver functionality would reconstruct the The Read-Out Driver functionality would reconstruct the Energy, Phase and Quality Factor at the Level1 rate. Energy, Phase and Quality Factor at the Level1 rate. The Optimal Filtering algorithm can be implemented with The Optimal Filtering algorithm can be implemented with DSP-like slices available in present FPGAs . DSP-like slices available in present FPGAs . The ROD system includes Energy calibration to different The ROD system includes Energy calibration to different units, histogramming and monitoring for different units, histogramming and monitoring for different magnitudes, Level 2 trigger algorithms (low Pt muon magnitudes, Level 2 trigger algorithms (low Pt muon tagging and total transverse energy computation for a tagging and total transverse energy computation for a whole slice), data quality checks and raw data whole slice), data quality checks and raw data compression for offline data re-processing compression for offline data re-processing GBT protocol implementation resources occupation GBT protocol implementation resources occupation Instantiation of 1, 2, 4, 8 and 12 GBT links without Instantiation of 1, 2, 4, 8 and 12 GBT links without optimization optimization Architecture of the Upgrade TileCal read-out electronics Read-Out Driver block architecture Test 1: Measurement of latency and maximum Test 1: Measurement of latency and maximum data bandwidth data bandwidth Raw data: 32 bit counter Raw data: 32 bit counter No protocol No protocol 6.25 Gbps per fiber 6.25 Gbps per fiber Latency of 108ns - 17 clock cycles Latency of 108ns - 17 clock cycles Data correlation with Signal Tap Data correlation with Signal Tap Total measured bandwidth of 75 Gbps Total measured bandwidth of 75 Gbps Power consumption 11.71 W Power consumption 11.71 W Test 2: 1 link GBT Test 2: 1 link GBT 1 link GBT protocol 1 link GBT protocol Power supply from external supply Power supply from external supply Receiver and transmitter located in Receiver and transmitter located in different GX transceiver block different GX transceiver block 48 hours without errors 48 hours without errors BER of 3.61·10 BER of 3.61·10 -15 -15 with a confidence of 95% with a confidence of 95% Total bandwidth of 4.8 Gbps Total bandwidth of 4.8 Gbps Power consumption 7.128 W Power consumption 7.128 W Test 3: 12 link GBT Test 3: 12 link GBT 12 link GBT protocol 12 link GBT protocol Power supply from OMB Power supply from OMB Receiver and transmitter located in Receiver and transmitter located in different GX transceiver block different GX transceiver block Internal FIFO compensation needed Internal FIFO compensation needed BER of 6.05·10 BER of 6.05·10 -16 -16 with a confidence of 95% with a confidence of 95% Total bandwidth of 57.6 Gbps Total bandwidth of 57.6 Gbps Power consumption 14.025 W Power consumption 14.025 W Block description Block description Configuration unit Configuration unit Power supply unit Power supply unit FPGA FPGA Test bench Test bench IO modules IO modules Thermal management Thermal management unit unit Clock unit Clock unit Timing problems Timing problems Setup and hold required times not met Setup and hold required times not met Errors due to non-synchronization Errors due to non-synchronization between logic and GX transceivers between logic and GX transceivers Timing problems increase with the Timing problems increase with the number of GBT links implemented number of GBT links implemented Solved with FIFO compensation Solved with FIFO compensation FIFO compensation between FPGA FIFO compensation between FPGA logic clock and GX transmitter logic clock and GX transmitter fixed this problem in OLC fixed this problem in OLC Timing requirements not met using Timing requirements not met using optimization 2,3 and 4 optimization 2,3 and 4 First PreROD prototype fully functional First PreROD prototype fully functional and working and working Optical Link Card tested succesfully Optical Link Card tested succesfully Implement up to 12 links GBT protocol Implement up to 12 links GBT protocol in Optical Link Card @ 4.8Gbps each in Optical Link Card @ 4.8Gbps each one one Ongoing test: OLC working with ROD Ongoing test: OLC working with ROD Test assembly Test assembly OLC connected to OMB as PU OLC connected to OMB as PU OLC with SNAP12 connectors in OLC with SNAP12 connectors in loopback loopback OMB connected to ROD with G-link OMB connected to ROD with G-link OLC purpose OLC purpose Simulate Front End data and sends Simulate Front End data and sends its to SNAP12 loopback its to SNAP12 loopback Send received Front End data to OMB Send received Front End data to OMB through Mezzanine connectors through Mezzanine connectors OMB purpose OMB purpose Send Front End data to ROD through Send Front End data to ROD through G-link G-link Confidence Level for case 2 and Confidence Level for case 2 and 3 Test assembly for 12 links in Test assembly for 12 links in bld. 175 bld. 175 GX transceiver latency GX transceiver latency Test assembly for 1 link Test assembly for 1 link Resource occupation for OLC Resource occupation for OLC FIFO compensation FIFO compensation OLC working with ROD scheme OLC working with ROD scheme Violated setup timing diagram Violated setup timing diagram Block diagram Block diagram Optical Link Card Optical Link Card ACES 2011 - Common ATLAS CMS Electronics Workshop for sLHC 9-11 March, 2011
description
Framework. Upgrade in ATLAS. The upgrade Phase II of the ATLAS Tile Calorimeter implies a complete redesign of the read-out electronics. In order to increase radiation tolerance pipelines, derandomizers and L1 trigger preprocessor will be placed within the off-detector electronics. - PowerPoint PPT Presentation
Transcript of Framework
Evaluation of the Optical Link Card for the Evaluation of the Optical Link Card for the Phase II Upgrade of TileCal DetectorPhase II Upgrade of TileCal Detector
F. Carrió F. Carrió11, V. Castillo, V. Castillo22, A. Ferrer, A. Ferrer22, V. González, V. González11, E. Higón, E. Higón22, C. Marín, C. Marín11,,P. MorenoP. Moreno22, E. Sanchis, E. Sanchis11, C. Solans, C. Solans22, A. Valero, A. Valero22, J. Valls, J. Valls22 11Dept. of Electronic Engineering, Universitat de València, SpainDept. of Electronic Engineering, Universitat de València, Spain
22Dept. of Atomic, Molecular and Nuclear Physics, IFIC, Valencia, SpainDept. of Atomic, Molecular and Nuclear Physics, IFIC, Valencia, Spain
Framework Upgrade in ATLAS
OLC Tests
Conclusion and future work
SpecificationsSpecifications ROD double PU Board format (4.72 in x 6.85 in)ROD double PU Board format (4.72 in x 6.85 in) 12 input channels up to 75 Gbps12 input channels up to 75 Gbps 12 output channels up to 75 Gbps12 output channels up to 75 Gbps
Optical Link Card Description
Technical detailsTechnical details 1 Altera Stratix II GX EP2SGX60E FPGA1 Altera Stratix II GX EP2SGX60E FPGA
1152 pins1152 pins 12 high speed transceivers @ 12 high speed transceivers @
6.375Gbps6.375Gbps 534 I/Os534 I/Os
2 SNAP12 connectors (12 fibers @ 2 SNAP12 connectors (12 fibers @ 6.25Gbps)6.25Gbps)
1 SFP connector @640Mbps1 SFP connector @640Mbps 12 copper layers12 copper layers Different thickness of dielectric: 2.5, 4 Different thickness of dielectric: 2.5, 4
and 6 milsand 6 mils Need high capacitance between planes Need high capacitance between planes
but sufficient thickness for striplinesbut sufficient thickness for striplines Over 2000 routes 5 mils widthOver 2000 routes 5 mils width Over 2400 vias 10 mils Over 2400 vias 10 mils
Design detailsDesign details Power Integrity techniquesPower Integrity techniques
Custom Power Distribution Network with Custom Power Distribution Network with FDTIM methodFDTIM method
High plane capacitanceHigh plane capacitance Proper location for capacitorProper location for capacitor
Signal Integrity techniquesSignal Integrity techniques Copper plane removal below coupling Copper plane removal below coupling
capacitor padscapacitor pads Via design optimizationVia design optimization Low inductance connectionsLow inductance connections Dielectric material selectionDielectric material selection Shielding lines Shielding lines Ground return viasGround return vias
GBT Implementation
TileCal detector measures the energy of hadrons in TileCal detector measures the energy of hadrons in ATLAS experimentATLAS experiment
TileCal detector is divided in 64 modules in TileCal detector is divided in 64 modules in and 3 and 3 sections in z for a total of 256 modulessections in z for a total of 256 modules
The readout is carried out using 10.000 electronic The readout is carried out using 10.000 electronic channels which, after digitization, come out of the channels which, after digitization, come out of the detector using optical fibers towards the Read Out detector using optical fibers towards the Read Out Driver (ROD) moduleDriver (ROD) module ATLAS requires radiation tolerant electronics capable of stable operation for, at ATLAS requires radiation tolerant electronics capable of stable operation for, at least, 10 yearsleast, 10 years
The TileCal phase II upgrade is focused on replacement of most of the readout The TileCal phase II upgrade is focused on replacement of most of the readout electronics, including the links between on- and off-detector electronics. This electronics, including the links between on- and off-detector electronics. This development is driven by the requirements for increased radiation tolerance, development is driven by the requirements for increased radiation tolerance, and the need to provide the level-1 trigger with more detailed informationand the need to provide the level-1 trigger with more detailed information..
The off-detector electronics consists of the preprocessor and the Read-Out-The off-detector electronics consists of the preprocessor and the Read-Out-Drivers (RODs). The pre processor is responsible for receiving the data from the Drivers (RODs). The pre processor is responsible for receiving the data from the on-detector electronics and preparing it for the level-1 trigger and for the on-detector electronics and preparing it for the level-1 trigger and for the RODsRODs..
The upgrade Phase II of the ATLAS Tile Calorimeter implies a The upgrade Phase II of the ATLAS Tile Calorimeter implies a complete redesign of the read-out electronics.complete redesign of the read-out electronics.
In order to increase radiation tolerance pipelines, derandomizers In order to increase radiation tolerance pipelines, derandomizers and L1 trigger preprocessor will be placed within the off-detector and L1 trigger preprocessor will be placed within the off-detector
electronics.electronics. With this new architecture the data transmission for the read-out With this new architecture the data transmission for the read-out
is one of the greatest challenges due to the large amount of data is one of the greatest challenges due to the large amount of data transferred out of the detector. Considering a sampling rate of transferred out of the detector. Considering a sampling rate of 40Msps, bi-gain transmission, 12-bit samples and a total amount 40Msps, bi-gain transmission, 12-bit samples and a total amount of 9856 channels with redundancy read-out, the complete of 9856 channels with redundancy read-out, the complete detector requires a total data bandwidth of around 20Tbps. detector requires a total data bandwidth of around 20Tbps.
The Read-Out Driver functionality would reconstruct the Energy, The Read-Out Driver functionality would reconstruct the Energy, Phase and Quality Factor at the Level1 rate. The Optimal Filtering Phase and Quality Factor at the Level1 rate. The Optimal Filtering algorithm can be implemented with DSP-like slices available in algorithm can be implemented with DSP-like slices available in present FPGAs . present FPGAs .
The ROD system includes Energy calibration to different units, The ROD system includes Energy calibration to different units, histogramming and monitoring for different magnitudes, Level 2 histogramming and monitoring for different magnitudes, Level 2 trigger algorithms (low Pt muon tagging and total transverse trigger algorithms (low Pt muon tagging and total transverse energy computation for a whole slice), data quality checks and energy computation for a whole slice), data quality checks and raw data compression for offline data re-processingraw data compression for offline data re-processing
GBT protocol implementation resources occupationGBT protocol implementation resources occupation Instantiation of 1, 2, 4, 8 and 12 GBT links without optimizationInstantiation of 1, 2, 4, 8 and 12 GBT links without optimization
Architecture of the Upgrade TileCal read-out electronics
Read-Out Driver block architecture
Test 1: Measurement of latency and maximum Test 1: Measurement of latency and maximum data bandwidthdata bandwidth
Raw data: 32 bit counterRaw data: 32 bit counter No protocolNo protocol 6.25 Gbps per fiber6.25 Gbps per fiber Latency of 108ns - 17 clock cyclesLatency of 108ns - 17 clock cycles Data correlation with Signal TapData correlation with Signal Tap Total measured bandwidth of 75 GbpsTotal measured bandwidth of 75 Gbps Power consumption 11.71 WPower consumption 11.71 W
Test 2: 1 link GBTTest 2: 1 link GBT 1 link GBT protocol1 link GBT protocol Power supply from external supplyPower supply from external supply Receiver and transmitter located in different GX Receiver and transmitter located in different GX
transceiver blocktransceiver block 48 hours without errors48 hours without errors BER of 3.61·10BER of 3.61·10-15-15 with a confidence of 95% with a confidence of 95% Total bandwidth of 4.8 GbpsTotal bandwidth of 4.8 Gbps Power consumption 7.128 WPower consumption 7.128 W
Test 3: 12 link GBTTest 3: 12 link GBT 12 link GBT protocol12 link GBT protocol Power supply from OMBPower supply from OMB Receiver and transmitter located in different GX Receiver and transmitter located in different GX
transceiver blocktransceiver block Internal FIFO compensation neededInternal FIFO compensation needed BER of 6.05·10BER of 6.05·10-16-16 with a confidence of 95% with a confidence of 95% Total bandwidth of 57.6 GbpsTotal bandwidth of 57.6 Gbps Power consumption 14.025 WPower consumption 14.025 W
Block descriptionBlock description Configuration unitConfiguration unit Power supply unitPower supply unit FPGAFPGA Test benchTest bench IO modulesIO modules Thermal management Thermal management
unitunit Clock unitClock unit
Timing problemsTiming problems Setup and hold required times not metSetup and hold required times not met
Errors due to non-synchronization Errors due to non-synchronization between logic and GX transceiversbetween logic and GX transceivers
Timing problems increase with the Timing problems increase with the number of GBT links implementednumber of GBT links implemented
Solved with FIFO compensationSolved with FIFO compensation FIFO compensation between FPGA logic FIFO compensation between FPGA logic
clock and GX transmitter fixed this clock and GX transmitter fixed this problem in OLCproblem in OLC
Timing requirements not met using Timing requirements not met using optimization 2,3 and 4optimization 2,3 and 4
First PreROD prototype fully functional and First PreROD prototype fully functional and workingworking
Optical Link Card tested succesfullyOptical Link Card tested succesfully Implement up to 12 links GBT protocol in Implement up to 12 links GBT protocol in
Optical Link Card @ 4.8Gbps each oneOptical Link Card @ 4.8Gbps each one Ongoing test: OLC working with RODOngoing test: OLC working with ROD
Test assemblyTest assembly OLC connected to OMB as PUOLC connected to OMB as PU OLC with SNAP12 connectors in loopbackOLC with SNAP12 connectors in loopback OMB connected to ROD with G-linkOMB connected to ROD with G-link
OLC purposeOLC purpose Simulate Front End data and sends its to Simulate Front End data and sends its to
SNAP12 loopbackSNAP12 loopback Send received Front End data to OMB Send received Front End data to OMB
through Mezzanine connectorsthrough Mezzanine connectors OMB purposeOMB purpose
Send Front End data to ROD through G-linkSend Front End data to ROD through G-link
Confidence Level for case 2 and 3Confidence Level for case 2 and 3 Test assembly for 12 links in bld. 175Test assembly for 12 links in bld. 175
GX transceiver latencyGX transceiver latency
Test assembly for 1 linkTest assembly for 1 link
Resource occupation for OLCResource occupation for OLC
FIFO compensationFIFO compensation
OLC working with ROD schemeOLC working with ROD scheme
Violated setup timing diagramViolated setup timing diagram
Block diagramBlock diagram
Optical Link CardOptical Link Card
ACES 2011 - Common ATLAS CMS Electronics Workshop for sLHC9-11 March, 2011
