Data Acquisition Systems
description
Transcript of Data Acquisition Systems
![Page 1: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/1.jpg)
Data Acquisition Systems
Ryan RiveraFermilab Detector R&D Retreat
May 5, 2011
![Page 2: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/2.jpg)
Wikipedia
• Data acquisition– is the process of sampling signals that measure
real world physical conditions and converting the resulting samples into digital numeric values that can be manipulated by a computer.
Data Acquisition Systems - Ryan Rivera 2Detector R&D Retreat
![Page 3: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/3.jpg)
This Talk
1. Current DAQ program2. Future projections– Direction of Current Program– Direction of New R&D
Data Acquisition Systems - Ryan Rivera 3Detector R&D Retreat
![Page 4: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/4.jpg)
Current Program
• Optical Links • CAPTAN• xTCA• Mu2e
Data Acquisition Systems - Ryan Rivera 4Detector R&D Retreat
![Page 5: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/5.jpg)
Current Program
• Optical Links • CAPTAN• xTCA• Mu2e
Data Acquisition Systems - Ryan Rivera 5Detector R&D Retreat
![Page 6: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/6.jpg)
Optical Links
• Versatile Link Common Project• Free Space Optical Transmission
Data Acquisition Systems - Ryan Rivera 6Detector R&D Retreat
![Page 7: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/7.jpg)
Optical Links
• Versatile Link Common Project• Free Space Optical Transmission
Data Acquisition Systems - Ryan Rivera 7Detector R&D Retreat
![Page 8: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/8.jpg)
Versatile Link Common Project
Data Acquisition Systems - Ryan Rivera 8
• A CERN-organized common project for ATLAS and CMS.• Charge: Develop general purpose optical link @ 5 Gbps data rates.• Fermilab has back-end component responsibilities.• Leading parallel optics investigations.
Detector R&D Retreat
![Page 9: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/9.jpg)
Optical Transceiver Test Measurements
Data Acquisition Systems - Ryan Rivera 9
Eye Diagram Measurements:• Optical Modulation Amplitude• Extinction Ratio• Rise/Fall Times
Jitter Analysis:• Deterministic Jitter Decomposition• Random Jitter (Gaussian, Unbounded)• Eye Opening @ 10-12 BER
Industry Standard Measurements and Apparatus
Detector R&D Retreat
![Page 10: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/10.jpg)
CMS Pixel Optohybrid (POH)
• Status: Current laser no longer available– Upgrades will require new devices
• Requirements:– Rad hard– 640 Mbps
• Next: FNAL responsible for testing and characterizing devices
Detector R&D Retreat Data Acquisition Systems - Ryan Rivera 10
![Page 11: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/11.jpg)
Parallel Optics – Technology Evolution
Data Acquisition Systems - Ryan Rivera 11Detector R&D Retreat
![Page 12: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/12.jpg)
Parallel Optics – Device Evaluation
Detector R&D Retreat Data Acquisition Systems - Ryan Rivera 12
Parallel Optical Engine Transceiver (4 channels, 6.25 Gbps/channel)
SNAP12 Transmitter (12 channels,
2.7 Gbps/channel)
SFP+ Single Channel Transceiver (10 Gbps)
![Page 13: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/13.jpg)
Optical Links
• Versatile Link Common Project
• Free Space Optical Transmission
Data Acquisition Systems - Ryan Rivera 13Detector R&D Retreat
![Page 14: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/14.jpg)
10 Gbps Optical Transmitters at
different wavelengths
Silicon Detectors
10Gb/s Optical Receivers
~10- 50cm
~50-100 cm
~100-150 cm
Beam Line Center
Free Space Optical Transmission• Motivation
– Reduce material budget– Work within rigid space constraints
Detector R&D Retreat Data Acquisition Systems - Ryan Rivera 14
1440 1460 1480 1500 1520 1540 1560 1580 16000.000
0.100
0.200
0.300
0.400
0.500
0.600
Measured Transmission Spectrum of Silicon(IC grade doped)
Wavelength (nm)
Tran
smis
sion
![Page 15: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/15.jpg)
Lens Lens
CWDM Mux
Tx LVDS Electrical Bit Streams
~2 mm Si ~2 mm Si
FPGA-Based BERT
8cm21 cm
l1
l2
l3
l4
CWDM DeMux
l4
l3
Tx
Tx
Tx
Tx
Rx
Rx
Rx
Rx
l2
l1
Free SpaceProof of Concept l1 = 1470 nm
l2 = 1490 nml3 = 1510 nml4 = 1530 nm
15
Rx LVDS Electrical Bit Streams
CWDM: Coarse Wavelength Division Multiplexing
Next Step: Evolve the Optical Design for Detector ApplicationsDetector R&D Retreat Data Acquisition Systems - Ryan Rivera
![Page 16: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/16.jpg)
16
Operated error free for over 48 hours at 1 Gbps on all 4 four channels
16Detector R&D Retreat Data Acquisition Systems - Ryan Rivera
Free Space Proof of ConceptLab Test
![Page 17: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/17.jpg)
Optical Links Future
• Parallel optics– Initiated US R&D request to recent DOE detector
solicitation. Need KA15 money until DOE approval.– Working with vendors on emerging devices, and
SMU/OSU on custom devices.• Versatile Link and CMS Phase 1 Optohybrid– Activities not funded by KA15, funded through CMS.
• Free space optics– Exploring low mass/power options (modulators).
Detector R&D Retreat Data Acquisition Systems - Ryan Rivera 17
![Page 18: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/18.jpg)
Current Program
• Optical Links
• CAPTAN• xTCA• Mu2e
Data Acquisition Systems - Ryan Rivera 18Detector R&D Retreat
![Page 19: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/19.jpg)
CAPTAN
Data Acquisition Systems - Ryan Rivera 19
What is the CAPTAN system?
Detector R&D Retreat
![Page 20: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/20.jpg)
Compact And Programmable daTa Acquisition Node
Data Acquisition Systems - Ryan Rivera 20Detector R&D Retreat
![Page 21: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/21.jpg)
Attributes
Data Acquisition Systems - Ryan Rivera 21
• It is simple.– It is 6” x 6” and for many systems, the only
external connections are a 3.3V power supply and a standard Ethernet cable.
Detector R&D Retreat
![Page 22: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/22.jpg)
Attributes
Data Acquisition Systems - Ryan Rivera 22
• It is flexible.– The user can stack the foundation boards in
different combinations to give unique functionality.
Detector R&D Retreat
![Page 23: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/23.jpg)
Attributes
Data Acquisition Systems - Ryan Rivera 23
• It is scalable.– In addition to the vertical stacking, the stacks can
be repeated arbitrarily and connected with one or many PCs in an Ethernet network.
Detector R&D Retreat
![Page 24: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/24.jpg)
OPTICAL BUS
COOLING CHANNELVERTICAL BUS
LATERAL BUS MOUNTING HOLE
ELECTRONICS
User Template
Data Acquisition Systems - Ryan Rivera 24
• The CAPTAN architecture consists of a few core boards but is intended to be augmented by custom boards designed and built by users.
Detector R&D Retreat
![Page 25: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/25.jpg)
Core Boards
Data Acquisition Systems - Ryan Rivera 25
“Green Board”NPCB – Node Processing and Control Board
“Blue Board”DCB – Data Conversion Board
“Red Board”PDB – Power Distribution Board.
Detector R&D Retreat
![Page 26: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/26.jpg)
CAPTAN User Community• Fermilab• Brown• Purdue• Colorado• Milano• Lecce
Detector R&D Retreat Data Acquisition Systems - Ryan Rivera 26
![Page 27: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/27.jpg)
Applications
• The CAPTAN system was designed to handle common data acquisition, control, and processing challenges within high energy physics.
• Examples of such applications are tracker readout systems, R&D test stands, and parallel data processing.
• As the CAPTAN system is a modular system it can be used for a wide range of applications, from very small to very large.
Data Acquisition Systems - Ryan Rivera 27Detector R&D Retreat
![Page 28: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/28.jpg)
FTBF Pixel Telescope
Data Acquisition Systems - Ryan Rivera 28Detector R&D Retreat
![Page 29: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/29.jpg)
PPD VIP Test Stand
Data Acquisition Systems - Ryan Rivera 29Detector R&D Retreat
![Page 30: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/30.jpg)
QIE Irradiation BER Testing
Detector R&D Retreat Data Acquisition Systems - Ryan Rivera 30
![Page 31: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/31.jpg)
IHEP Telescope - Beijing
Data Acquisition Systems - Ryan Rivera 31Detector R&D Retreat
![Page 32: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/32.jpg)
T980 Crystal Collimation Telescope
Detector R&D Retreat Data Acquisition Systems - Ryan Rivera 32
![Page 33: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/33.jpg)
CAPTAN Future
• Port CAPTAN Software to Linux
• Explore parallel processing power
• Integrate with xTCA
Data Acquisition Systems - Ryan Rivera 33Detector R&D Retreat
![Page 34: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/34.jpg)
Current Program
• Optical Links • CAPTAN
• xTCA• Mu2e
Data Acquisition Systems - Ryan Rivera 34Detector R&D Retreat
![Page 35: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/35.jpg)
xTCA
• ATCA (Advanced Telecommunications Computing Architecture)– Spec put forth by PICMG (PCI Industrial Computer
Manufacturers Group: a consortium of over 250 companies)
• AMC plugs into ATCA card– Or MicroTCA crate
• ATCA + MicroTCA = xTCA• Successor to VME?
Detector R&D Retreat Data Acquisition Systems - Ryan Rivera 35
12U 14-slot ATCA shelf
![Page 36: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/36.jpg)
Advanced Mezzanine Card (AMC)
Data Acquisition Systems - Ryan Rivera 36Detector R&D Retreat
![Page 37: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/37.jpg)
xTCA and CAPTANIntegration
Data Acquisition Systems - Ryan Rivera 37Detector R&D Retreat
![Page 38: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/38.jpg)
VME vs. xTCA
+ Number of new developments decreasing, sales still constant
+ Many I/O modules available− Developed in 1981− Bus technology has speed limitations− Wide busses create a lot of noise in analog channels− No standard management at crate level− No standard management at module level− One damaged bus line stops entire crate
Data Acquisition Systems - Ryan Rivera 38Detector R&D Retreat
![Page 39: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/39.jpg)
VME vs. xTCA
+ Scalable modern architecture From 1 slot MicroTCA to full mesh ATCA
+ 1 Gbps serial communication links+ Standard PCIe and Ethernet communication+
Well defined management+ Hot-swappable+ Safe against hardware/software failures+ 99.999% availability is possible
Data Acquisition Systems - Ryan Rivera 39Detector R&D Retreat
![Page 40: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/40.jpg)
VME vs. xTCA
− Many crates and modules from different manufacturers Is there a 'standard' ?
− Learning curve to software/hardware specifications
− Need functional MMC/IPMI code for power delivery
− Imposed system management
Data Acquisition Systems - Ryan Rivera 40Detector R&D Retreat
![Page 41: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/41.jpg)
xTCA Future
• Large experiments are considering xTCA over VME.– CMS– ATLAS– LHCb– Experiments at DESY
• Dream is to develop a readout system for strips/pixels– Evaluate for future FNAL Experiments (CMS upgrade, Mu2e)
• SBIR submitted with industry partners.• Vince Pavlicek is on committee to develop the
PhysMTCA or MTCA.4 (MicroTCA for Physics) spec.
Data Acquisition Systems - Ryan Rivera 41Detector R&D Retreat
![Page 42: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/42.jpg)
Current Program
• Optical Links • CAPTAN• xTCA
• Mu2e
Data Acquisition Systems - Ryan Rivera 42Detector R&D Retreat
![Page 43: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/43.jpg)
Mu2eTracker
21,600 Straws216 Readout Controllers~65 GBps*
Calorimeter2112 Crystals 44 Readout Controllers~15 GBps*
* zero-suppressed data
- Additional data (< 1 GBps) from Cosmic Ray Veto system and extinction/target monitors
Data Acquisition Systems - Ryan Rivera 43Detector R&D Retreat
![Page 44: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/44.jpg)
Architecture Alternatives
• Streaming (this is the selected architecture)+ less front-end hardware (no concentrators or L1 Trigger)+ simpler design+ flexible trigger− higher cost
• Triggered+ less back-end hardware (networking and processing)+ lower cost+ allows higher digitization rate (e.g., waveform digitizers)− fixed L1 trigger
Data Acquisition Systems - Ryan Rivera 44Detector R&D Retreat
![Page 45: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/45.jpg)
Implementation Alternatives
Data Acquisition Systems - Ryan Rivera 45Detector R&D Retreat
Standalone(rack-mount)
DTC
Servers
EVB Switch
EVB Switch
DTC
PCIe DTCDTC
EVB SwitchDTC NIC PCIe DTC
(event building in server)
Servers
Servers
NIC
![Page 46: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/46.jpg)
Comparing Technology
• To meet the demands of a 100 GBps streaming DAQ, Mu2e will leverage…– 10 Gbps Networking– High Density FPGAs– High Performance Processors– Fiber-optic Communications (synergy with Optical Links)
CDF DAQ (1987) Mu2e DAQ
Data Rate 20 100,000 MBps
Buffering 400 2,000,000 MB
Processing (FP) 10 30,000,000 MFLOPS
Processing (INT) 200 60,000,000 MIPS
Historical Perspective
Data Acquisition Systems - Ryan Rivera 46Detector R&D Retreat
![Page 47: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/47.jpg)
Design Philosophy• Commercial Slow Control hardware and software is used for
development– needed early, can’t wait for custom HW/SW– hopefully, utility extends into production and running– LabVIEW or EPICS
• Embedded Processors– use commercial modules if possible to allow early software development– embedded Ethernet and HTTP with eye towards debugging ease
Data Acquisition Systems - Ryan Rivera 47Detector R&D Retreat
![Page 48: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/48.jpg)
Mu2e Future
• One year from starting prototype phase.• ~6 years to installation complete.• Mu2e funded by Mu2e, but components of
DAQ architecture may apply to future experiments (Project X and others).
Data Acquisition Systems - Ryan Rivera 48Detector R&D Retreat
![Page 49: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/49.jpg)
DAQ Discussion Points
• Program Resources• National Picture Comparison• Best Future R&D Avenues
Detector R&D Retreat Data Acquisition Systems - Ryan Rivera 49
![Page 50: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/50.jpg)
DAQ Program Resource Issues
• Limited manpower– Has hindered xTCA effort
• xTCA development accessibility– Community MMC design effort?
• FPGA power/expense/expertise
Data Acquisition Systems - Ryan Rivera 50Detector R&D Retreat
![Page 51: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/51.jpg)
Summary of National Picture vs. Current Program
• Breaking new ground with optics, free space– Submitted SBIR and DOE R&D proposal
• Optics: NTT (Nippon Telegraph and Telephone) achieved 69.1 Tbps transmission using WDM of 432 wavelengths with a capacity of 171 Gbps over a single 240 km optical fiber on March 25, 2010. Highest optical transmission speed ever recorded at time.
• xTCA falling behind?• GPUs falling behind?• DAQ Software falling behind?
– COMEDI, EPICS, LabVIEW, MATLAB, Visual C++, ladder logic?
Data Acquisition Systems - Ryan Rivera 51Detector R&D Retreat
![Page 52: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/52.jpg)
Best Future DAQ R&D Avenues
• Ethernet/InfiniBand: InfiniBand prevalent as the interconnect between nodes of the highest performing super computers
• Off-the-shelf components• xTCA• GPUs: Fastest computer in the world, Tianhe-1A in China uses 7,168
NVidia Fermi GPUs and 14,336 Intel Xeon CPUs. Would require 50,000 CPUs for same performance with CPUs alone.
• Optics: Commonly 100 Gbps with distances over 100 m. Becoming interconnect of choice in high speed data centers and HPC clusters. There are 40 Gbps and 100 Gbps optical Ethernet standards.
• FPGAs• EPICS: The tool is designed to help develop systems which often feature
large numbers of networked computers providing control and feedback.
Data Acquisition Systems - Ryan Rivera 52Detector R&D Retreat
![Page 53: Data Acquisition Systems](https://reader034.fdocuments.in/reader034/viewer/2022051317/56816380550346895dd46196/html5/thumbnails/53.jpg)
Thanks!
Data Acquisition Systems - Ryan Rivera 53Detector R&D Retreat