High Speed Links
26
2 October 2013 [email protected] 1 High Speed Links 1. High speed links in LHC and commercial applications 2. On-going common projects for LHC Phase I upgrades 3. Towards HL-LHC 4. Conclusions Francois Vasey, CERN PH-ESE
description
High Speed Links. Francois Vasey, CERN PH-ESE. High speed links in LHC and commercial applications On-going common projects for LHC Phase I upgrades Towards HL-LHC Conclusions. 1. High Speed Links in LHC. Trigger. Timing/triggers/ sync. Control/monitoring. Switching network. CPU. - PowerPoint PPT Presentation
Transcript of High Speed Links
[email protected] 12 October 2013
High Speed Links
1. High speed links in LHC and commercial applications2. On-going common projects for LHC Phase I upgrades3. Towards HL-LHC4. Conclusions
Francois Vasey, CERN PH-ESE
1. High Speed Links in LHC
22 October 2013
CPU
CPU
CPU
CPU
CPU
CPU
Switching network
Front-end DAQ interfaceFront-end DAQ
interfaceFront-end DAQ interfaceFront-end DAQ
interface 1
Front-end DAQ interface N
Front-end DAQ interface 2
Timing/triggers/sync Control/monitoringTrigger
1.1 For instance: Link diversity in ATLAS
32 October 2013
Detector Purpose Media Dir. Rate Mbits/s
Quantity Comment
Pixel TTC/DCS/DAQ
Optical Down/up
40 (80) 250 Custom
SCT TTC/DCS/DAQ
Optical Down/up
40 8200 Custom
TRT TTC/DCS LVDS-Cu
Down 40 400 Custom
DAQ/DCS LVDS – Optical
Up 40 –1600 400 GOL
Ecal TTC Optical Down 80 TTC link
DCS LVDS-Cu
Down/Up
SPAC
DAQ Optical Up 1600 1600 Glink
Trigger Copper
Up Sub-det. Analog
Hcal TTC Optical Down 80 TTC link
DCS Copper
Down/up
CAN
DAQ Optical Up 1600 512 Glink
Trigger Copper
Up Analog
CSC, RPC, TGC DAQ Optical Up 1600 1200 Glink
MDT DAQ Optical Up 1600 1200 GOL
CSC TTC Optical Down 1600 200 Glink
CSC, RPC, TGC, MDT
DCS Copper
Down/up
CAN
RPC Trigger Optical Up 1600 Glink
~15’000
1.2 For instance: Link diversity in CMS
42 October 2013
Detector Purpose Media Dir. Rate Mbits/s
Quantity Comment
Pixel - strip TTC/DCS Optical Down/up
80 3.250 CCU, Custom
DAQ Optical Up 40Msamp~320
40.000 Custom analog
Ecal TTC/DCS Optical Down/up
80 3.000 CCU, Custom
DAQ Optical Up 800 3.000 GOL
Trigger Optical Up 800 5.300 GOL
Hcal TTC Optical Down 80 160
DCS Electrical
Down/up
n/a 160
DAQ & Trig Optical Up 1600 2.000 GOL
Muons TTC Optical Down 80 240 DT
DCS Optical Down/up
80 150 RPC
DAQ Electrical
Up parallel 800 DT
Trigger Optical Up 1.600 600 RPC
Trigger Electrical
Up parallel 800 DT,CSC
… etc
~60’000
[email protected] 52 October 2013
1.3 High Speed Optical Links in LHC
Large quantity o(100k), large diversity Majority running @ o(1Gbps) From full custom to qualified COTS
Tuned to specific application and environment Developed by independent teams
Successful adoption of technology in HEP
[email protected] 62 October 2013
1.4 High Speed Optical Links in LHC: Lessons Learned
Increase Link Bandwidth amortize system cost better
Share R&D effort use limited resources optimally
Strengthen quality assurance Programs Identify problems early Test at system level
Joint ATLAS/CMS NOTE ATL-COM-ELEC-2007-001
CMS-IN-2007/066 https://edms.cern.ch/document/882775/3.8
[email protected] 72 October 2013
1.5 High Speed Optical Links outside LHC
Rapid progress driven by: Successful standardization effort
100 GbE standard ratified in 2010 by IEEE Availability of hardware cores embedded in FPGAs
50+ x 10G transceivers in modern FPGAs Commercial availability of MultiSourceAgreement-based hardware
Commodity 10G and 40G devices Emerging 100G and 400G parallel optics engines
Current LAN rates @ o(10Gbps), ramping up to 40Gbps
Widening performance gap compared to HEP But consider:
Specific environmental constraints and long qualification time Long detector development time: vintage 2000 hardware in LHC
(short distance)
R&D necess
ary to ke
ep up,
use and deve
lop tech
nology
[email protected] 82 October 2013
2. On-going Development Projects for LHC Phase I Upgrades
Initiatives initially aiming at a single target: SLHC Launched in 2008, timely for phase I upgrades
Working Groups Microelectronics User Group (MUG) Optoelectronics Working Group (Opto WG)
Topical Workshop on Electronics for Particle Physics
Common Projects Rad Hard Optical Link
GigaBit Transceiver (GBT) project (chip-set) & GBT-FPGA project
Versatile Link (VL) project (opto) & Gigabit Link Interface Board (GLIB)
Many others …
[email protected] 92 October 2013
2.1 Rad Hard Optical Link Common Project
Requirements: General
Bi-directional High data rate: 4.8Gbits/s Low and constant latency (for TTC and trigger data paths) Error detection and correction
Environment Radiation hard ASICs (130nm) and radiation qualified opto-electronics at
Front-End Magnetic Field tolerant devices at Front-End Flexible chip interface (e-links) to adapt to various architectures Compatibility with legacy fibre plants
Back-end COTS High-end FPGAs with embedded transceivers
GBT-FPGA firmware Parallel optics
Commitment to deliver to LHC experiments in 2014-2015
2.2 Impact on System Architecture Custom development for
difficult Front-End Firmware only for FPGA-
based Back-End Evaluation platform for
system-tests
CPU
CPU
CPU
CPU
CPU
CPU
Switching network
Front-end interface
Front-end interface
Front-end interface
Timing/triggers/sync
Control/monitoring
Trigger
DCS network
Rad-hard optical links
2 October 2013
2.3 Optical Link Project Status: GBT
Project started in 2008 GBT (Serializer/Deserializer)
GBT-Serdes prototype in 2009 GBTx in 2012 Packaging in 2013 2nd iteration and prod in 2014
GBLD (Laser Driver) Final iteration (V4.1/V5) in 2013
GBTIA (Pin Diode Receiver) Final iteration (V3) in 2012
GBTSCA (Slow Control ASIC) Final version expected in 2014
GBT-FPGA firmware Tracking evolution of major FPGA families Available
Project delivers Chipset for Front-End GBT-FPGA Back-End
firmware
2 October 2013
~30 Man-Years
[email protected] 122 October 2013
2.4 Optical Link Project Status: VL Kick-off: April08 Proof of concept: Sep09 Feasibility demo: Sep11 Project delivers
Custom built Rad Hard VTRx Production readiness: Apr14 Early delivery of rad-soft VTTX to
CMS-Cal-Trig: Dec13 Recommendations for
Fibre and connectors Backend optics
Evaluation Interface boards (GLIB) Experiments
Design their own system Select passive and backend components
based on VL recommendations and on their own constraints
~40 Man-Years
2.5 Packaging and Interconnects Status
GBT 20x20 BGA with on-package
crystal and decoupling capacitors
CERN<>Distributor<>Company<>Company
5 iterations to freeze design 1-4 weeks per iteration 6 months to first prototype Mask error, re-spin, +2months
VL High speed PCB simulation
and design Injection-moulded ULTEM
2xLC connector latch and pcb support
Prototyping started 2009, moulded parts delivered 2013
2 October 2013
[email protected] 142 October 2013
2.6 Rad Hard Optical Link Project Effort
6 years of development Launched in 2008 Delivery in 2014-15
6 institutes involved CERN, FNAL, IN2P3, INFN, Oxford, SMU Estimated 80 Man-Years + 2-3 MCHF material
One of the largest common efforts in the community
[email protected] 152 October 2013
3. Towards HL-LHC
Higher Data-rate Lower Power Smaller Footprint Enhanced Radiation Resistance
Not to be forgotten: Fast electrical links Radiation-soft links
Not all features in same link
[email protected] 162 October 2013
3.1 Higher Data-Rate and Lower Power
ASICs: migrate to a more advanced technology node: ≤65nm Qualify technology for environment Establish stable support framework and design tools for full
duration of development Design new ASICs taking advantage of technology advantages
Either high speed (multiply by two) Or low power (divide by four)
Opto: qualify new components and emerging technologies VL opto are already 10Gbps capable
Electrical interconnects and packaging become performance limiters
Build up expertise Train with relevant simulation and design Tools Establish relationship with selected suppliers
[email protected] 172 October 2013
3.2 Smaller Footprint
GBT package size can be shrunk by limiting the number of IO pads and going to fine pitch BGA
Will affect host board design
VTRx concept has been pushed to its size limit: SF-VTRx
Not sufficient for some tracker layouts
Tracker frontends will need custom packaging
Industry to be approached
VTRx
SF-VTRx
[email protected] 182 October 2013
3.3 Enhanced Radiation Resistance
ASICs likely to be OK
Active opto devices OK except for pixels
Tight margins Are there
alternatives for fluences beyond 1016 cm-2 ?
Reconsider Passives?
modulators
Tx
Rx
HL-LHC TK
HL-LHC TK
[email protected] 192 October 2013
3.4 Si-Photonics, a paradigm changing technology?
Si is an excellent optical material with high refractive index (but indirect
bandgap) Is widely available in high quality grade Can be processed with extreme precision using deep submicron CMOS
processing techniques So, why not build a photonic circuit in a CMOS Si-wafer?
[email protected] 202 October 2013
3.5 Si-Photonics, status in the community
Commercial devices tested Excellent functional performance Moderate radiation resistance
limited by controller ASIC failure On-going collaborations with
academic and industrial partners
Simulation tools in hands Selected building blocks
under test No usable conclusion so far,
much more work needed Packaging is challenging
Assess radiation hardness first !
Luxtera QSFP+ Si-Photonics chip
[email protected] 212 October 2013
3.6 Not to be forgotten
High speed electrical data links are not obsolete !!!
Short distance, on-board serial links Aggregation to high speed opto-hubs Low mass, highly radiation resistant
(HL-LHC pixels) Develop expertise and tools
Detectors with modest radiation levels may not need custom front-ends Qualify COTS and/or Radiation-soft
components Shortlist recommended parts Continuously track market evolution
22
4. Conclusions (1/2) High speed links are the umbilical cords of the experiments Meeting the HL-LHC challenge will require:
Qualifying new, emerging technologies and components Designing electronics, interconnects, packages and perhaps even
optoelectronics Maintaining expertise, tools and facilities Investing heavily with a few selected industrial partners
The community is healthy, but small and fragmented Existing working groups and common projects are effective and
should be continued for phase II upgrades Additional projects and working groups could be created
WG on fast electrical links & signal integrity WG on radiation-soft links & qualification Exploratory Project on Si-photonics for HEP applications
Manpower is the real bottleneck Close to or below critical mass in several institutes
Development
Service
Liaison with Industry
Design
Common projects
Working groups
People
23
4. Conclusions (2/2)
Development time remains very long in comparison to industry
HL-LHC environment is unique and requires specific R&D and qualification procedures
Common building blocks are desirable, but… … Take time to be specified … Must be made available early to detector development teams
Limited manpower results in longer development time Master schedule and requirements are evolving 6 years, 6 institutes, 80 MY were required to reach production
readiness for phase I 2014+6=2020
Common optical link project for HL-LHC must be started now !
Evolving from phase I “Rad-Hard Optical Link” technological solution
Reusing and possibly expanding existing collaboration framework
Strengthening teams and avoiding parallel efforts wherever possible
Leaving door open to selected exploratory R&D, as long as schedule is still fluid
Time
1.1 Many different Link types Readout - DAQ:
Unidirectional Event frames. High rate Point to point
Trigger data: Unidirectional High constant data rate Short and constant latency Point to point
Detector Control System Bidirectional Low/moderate rate (“slow
control”) Bus/network or point to
point Timing: Clock, triggers,
resets Precise timing (low jitter
and constant latency) Low latency Fan-out network
(with partitioning)
25
Different link types remain physically separate, each with their own specific implementation
2 October 2013
