Post on 23-Dec-2015
The TrainBuilder ATCA Data Acquisition Board for the European XFEL
The TrainBuilder ATCA Data Acquisition Boardfor the European-XFEL
John Coughlan, Chris Day, James Edwards, Ed Freeman, Senerath Galagedera and Rob Halsall
Science & Technology Facilities CouncilRutherford Appleton Laboratory Oxfordshire, United Kingdom
E-mail: john.coughlan@stfc.ac.uk
Presented by John Coughlan. TWEPP Oxford. September 2012.
The TrainBuilder ATCA Data Acquisition Board for the European XFEL
2
European-XFEL DAQ TrainBuilder demonstrator board FPGA Firmware System Applications
Presented by John Coughlan. TWEPP Oxford. September 2012.
Topics
The TrainBuilder ATCA Data Acquisition Board for the European XFEL
XFEL DAQ Data Flow
• 3 Large 2D Pixel Detectors• 1M Pixel, 2B/Pixel, 512 Images @10Hz (X-Ray pulse Train)• 10 Gbyte per second per Megapixel rate• 10G Ethernet using UDP protocol• Modularity : 16 x 10G Links off detector per Megapixel• Scalable• Full Data Rate to PC Layer
3
Train BuilderDetector 2D Camera 10G Switch 10G serversPC Layer
10G 10G 10G
Common Systems XFEL DAQ
Presented by John Coughlan. TWEPP Oxford. September 2012.
The TrainBuilder ATCA Data Acquisition Board for the European XFEL
Image Train Building
4Presented by John Coughlan. TWEPP Oxford. September 2012.
FEMs
Train Builder
FEMs
PCSWITCH PC
PCSWITCH PC
Train Builder
N x 512N x 512
10Gbps 10Gbps
Train BuilderDetector 2D Camera
10G serversPC Layer
Partial Images 128 KB Full Images 2 MB
Detector
SpecificCommon
XFEL DAQ
N= 16
Deep Buffers
Switch
10G Links
The TrainBuilder ATCA Data Acquisition Board for the European XFEL
TrainBuilder Demonstrator
5Presented by John Coughlan. TWEPP Oxford. September 2012.
• Train Builder Demonstrator• ATCA form factor• 5 x Xilinx V5FXT100 FPGAs• Analogue X-point 80x80
• 4 x FMCs dual SFP+ 10Gb• 8 x 2GB DDR2 SODIMM VLP
Modules• 4 x QDRII SRAMs
• Links Configured as Input or Output by software
• Connects to eitherDetectors or PC Farm
The TrainBuilder ATCA Data Acquisition Board for the European XFEL
TrainBuilder FE x 4
6Presented by John Coughlan. TWEPP Oxford. September 2012.
Virtex5 FX100T, Dual PPC
DESY FMC2 x 10Gbps SFP+
2 x DDR2 VLP SODIMMs2 GByte
QDRII SRAM 8 MByte
SRAM PPC code
The TrainBuilder ATCA Data Acquisition Board for the European XFEL
TrainBuilder FE x 4
7Presented by John Coughlan. TWEPP Oxford. September 2012.
Virtex5 FX100T, Dual PPC
DESY FMC2 x 10Gbps SFP+
QDRII SRAM 8 MByte
QDRII SRAMPixel reordering
DDR2Partial Images train buffer
To Xpoint10G Inputs fromDetector
Configured as INPUT : Data from Detectors
2 x DDR2 VLP SODIMMs2 GByte
The TrainBuilder ATCA Data Acquisition Board for the European XFEL
TrainBuilder FE x 4
8Presented by John Coughlan. TWEPP Oxford. September 2012.
Virtex5 FX100T, Dual PPC
DESY FMC2 x 10Gbps SFP+
QDRII SRAM 8 MByte
10G Outputs to Farm
QDRII SRAM Image Tiling
DDR2Full Images train buffer
From Xpoint
Configured as OUTPUT : Data to PC Farm
2 x DDR2 VLP SODIMMs2 GByte
The TrainBuilder ATCA Data Acquisition Board for the European XFEL
TrainBuilder 10G link FMC
9Presented by John Coughlan. TWEPP Oxford. September 2012.
FPGA Mezzanine Card FMCANSI/VITA 57
Developed by DESY Electronics Group
Dual 10 Gbps SFP+
Vitesse 10G PHYs
XAUI 4 x 3.125 Gbps
The TrainBuilder ATCA Data Acquisition Board for the European XFEL
TrainBuilder Demonstrator
10Presented by John Coughlan. TWEPP Oxford. September 2012.
Master FPGA
Boot MMCSpartan3AN FPGA
Xpoint • Crosspoint Switch• Mindspeed 80x80 6.5 Gbps• 64 Tx & Rx @ 2.5 Gbps• Switch changes @ ~ 10 Hz
• Master FPGA Virtex5 FXT100• Manages data switching• PPC GbE TCP/IP server• Python scripts on client PC
FPGA Configuration CF card
• Boot Spartan3 with FLASH
GbE Ctrl
GbE Ctrl RJ45
The TrainBuilder ATCA Data Acquisition Board for the European XFEL
TrainBuilder ATCA Interfaces
11Presented by John Coughlan. TWEPP Oxford. September 2012.
ATCA Zone 3Cable/Rear Transition Module
ATCAZone 2GbE CtrlPCIe
ATCAZone 1JTAG
JTAG
• Zone 3 Custom RTM• 32 Tx & Rx data lanes
to Crosspoint+ clock&controls
• Zone 2 • Std interfaces
• Zone 1• Power 48V• MMC 3V3
ATCA 48V DC-DC
The TrainBuilder ATCA Data Acquisition Board for the European XFEL
TrainBuilder Data Rates
12Presented by John Coughlan. TWEPP Oxford. September 2012.
• 10G
XAUI 4 x 3.125 Gbps
> 1 GB/sec
• Crosspoint• Aurora 4 lanes 64B/66B @ 2.5 Gbps
> 1 GB/sec
• DDR2 (1 per link) 2 GB• Dual PPC Hard IP Embedded
~ 1 GB/sec Read & Write (Long bursts)
• QDRII SRAM (1 per 2 links) 8 MB• 32 bit R & W @ 250 MHz
~ 1 GB/sec per link (2 word burst)
• Detector (512 bunches/train)
~ 640 MB/sec per 10G LINK
• Total B/W per TB board 8 Links
~ 8 GB/sec
The TrainBuilder ATCA Data Acquisition Board for the European XFEL
13
Firmware
Presented by John Coughlan. TWEPP Oxford. September 2012.
The TrainBuilder ATCA Data Acquisition Board for the European XFEL
FE FPGA Firmware
14Presented by John Coughlan. TWEPP Oxford. September 2012.
AuroraCore
0
XPOINT
4 TX PAIRS
4 RX PAIRS
4 TX PAIRS
4 RX PAIRS
AuroraCore
1
XAUICore
0
XAUICore
1
Power PCMemory
ControllerSubsystem
4 TX PAIRS
4 RX PAIRS
4 TX PAIRS
4 RX PAIRS
XTALPLL
XTALPLL
DUAL10G PHY
FMC
PH
Y
SFP+
PH
Y
SFP+
TB FPGA
10GUDPCore
1
Power PCMemory
ControllerSubsystem
10GUDPCore
0
‘XCTRL’
DDR2 SDRAMImage BuffersStorage
DDR2 SDRAM
QDRII SRAMImageManipulation
QDRII SRAM
Aurora 64B/66BXAUI
QDRIIInterface
QDRIIInterface
LOCAL LINK Protocol
The TrainBuilder ATCA Data Acquisition Board for the European XFEL
FE FPGA Firmware
15Presented by John Coughlan. TWEPP Oxford. September 2012.
AuroraCore
0
XPOINT
4 TX PAIRS
4 RX PAIRS
4 TX PAIRS
4 RX PAIRS
AuroraCore
1
XAUICore
0
XAUICore
1
Power PCMemory
ControllerSubsystem
4 TX PAIRS
4 RX PAIRS
4 TX PAIRS
4 RX PAIRS
XTALPLL
XTALPLL
DUAL10G PHY
FMC
PH
Y
SFP+
PH
Y
SFP+
TB FPGA
10GUDPCore
1
Power PCMemory
ControllerSubsystem
10GUDPCore
0
‘XCTRL’
DDR2 SDRAMImage BuffersStorage
DDR2 SDRAM
QDRII SRAMImageManipulation
QDRII SRAM
Aurora 64/66BXAUI
QDRIIInterface
QDRIIInterface
LOCAL LINK Protocol
QDRII SRAMPixel reordering
DDR2Partial Images train buffer
To Xpoint10G Inputs fromDetector
The TrainBuilder ATCA Data Acquisition Board for the European XFEL
FE FPGA Firmware
16Presented by John Coughlan. TWEPP Oxford. September 2012.
AuroraCore
0
XPOINT
4 TX PAIRS
4 RX PAIRS
4 TX PAIRS
4 RX PAIRS
AuroraCore
1
XAUICore
0
XAUICore
1
Power PCMemory
ControllerSubsystem
4 TX PAIRS
4 RX PAIRS
4 TX PAIRS
4 RX PAIRS
XTALPLL
XTALPLL
DUAL10G PHY
FMC
PH
Y
SFP+
PH
Y
SFP+
TB FPGA
10GUDPCore
1
Power PCMemory
ControllerSubsystem
10GUDPCore
0
‘XCTRL’
DDR2 SDRAMImage BuffersStorage
DDR2 SDRAM
QDRII SRAMImageManipulation
QDRII SRAM
Aurora 64/66BXAUI
QDRIIInterface
QDRIIInterface
LOCAL LINK Protocol
10G Outputs to Farm
QDRII SRAM Image Tiling
DDR2Full Images train buffer
From Xpoint
The TrainBuilder ATCA Data Acquisition Board for the European XFEL
TrainBuilder Firmware System
17
PHY XAUISFP UDP PPC AUR
PHY XAUISFP UDP PPC AUR
FPGA 1
XP
RAM
RAM
PHY XAUISFP UDP PPC AUR
PHY XAUISFP UDP PPC AUR
FPGA 2RAM
RAM
PHY XAUISFP UDP PPC AUR
PHY XAUISFP UDP PPC AUR
FPGA 3RAM
FPGA
GIGE
RAM
PHY XAUISFP UDP PPC AUR
PHY XAUISFP UDP PPC AUR
FPGA 4RAM
RAM
PHYXAUI SFPUDPPPCAUR
PHYXAUI SFPUDPPPCAUR
FPGA 5RAM
RAM
PHYXAUI SFPUDPPPCAUR
PHYXAUI SFPUDPPPCAUR
FPGA 6RAM
RAM
PHYXAUI SFPUDPPPCAUR
PHYXAUI SFPPEPPCAUR
FPGA 7RAM
RAM
PHYXAUIPEPPCAUR
PHYXAUI SFPPEPPCAUR
FPGA 8RAM
RAM
SFPPHYXAUIUDPPPC
RAM
ASICRX
SFPPHYXAUIUDPPPC
RAM
TB
ASICRX
FEM
SFP
FPGA
FPGA
Presented by John Coughlan. TWEPP Oxford. September 2012.
TrainBuilderFEMs
The TrainBuilder ATCA Data Acquisition Board for the European XFEL
18
Applications
Presented by John Coughlan. TWEPP Oxford. September 2012.
The TrainBuilder ATCA Data Acquisition Board for the European XFEL
Status
•First ATCA boards manufactured in Q1/2012
• 4 Boards all passed Boundary SCAN first time.
• Basic Tests working•10G links Tx and Rx•DDR2•QDRII•Crosspoint•GbE controls
• Performance testing at RAL. Data Rates 1 GB/s/link? Soak Tests.
• Board in Hamburg for XFEL DAQ PC Farm tests since July.
19Presented by John Coughlan. TWEPP Oxford. September 2012.
The TrainBuilder ATCA Data Acquisition Board for the European XFEL
First Detector Application
20
1 x TB configured as4 Inputs and 4 Outputs
Presented by John Coughlan. TWEPP Oxford. September 2012.
x4
Large Pixel Detector Prototype Quadrant. 4 FEMs 10G links
The TrainBuilder ATCA Data Acquisition Board for the European XFEL
2 x TBs : ½ Mpixel
21Presented by John Coughlan. TWEPP Oxford. September 2012.
2 x TB System : 8 Links½ Mpixel
PassiveCablesTx & Rx
x8
INPUT OUTPUT
The TrainBuilder ATCA Data Acquisition Board for the European XFEL
FUTURE 4 x TBs : 1 Mpixel 2013
22Presented by John Coughlan. TWEPP Oxford. September 2012.
1 Mpixel with 4 x ATCA TB Demonstrators. 16 LINKs
x16
Parallel Opto LinksSNAP 12 Tx / Rx
Scaleable to larger systems
NEW ATCA Switch Board
NEW Optical RTMs
@ 2.5 Gbps
The TrainBuilder ATCA Data Acquisition Board for the European XFEL
Summary & Plans
• Developed an ATCA board for common XFEL DAQ• Firmware infrastructure already operating incl. 10G UDP• 2 x Demonstrator ATCA can instrument ½ Mpixel today• Baseline is 1 Mpixel. Scheme exists to scale to larger detectors.
•Plans•Complete Performance Tests on prototypes•Develop Optical RTM and Switch card 2013•Integrate with 1 Mpixel detectors 2014•Install XFEL in 2015•Data taking at XEFL in 2016
Acknowledgements:M. Zimmer, I. Sheviakov (DESY Electronics) C. Youngman (XFEL DAQ)
23Presented by John Coughlan. TWEPP Oxford. September 2012.
The TrainBuilder ATCA Data Acquisition Board for the European XFEL
24
Questions
Presented by John Coughlan. TWEPP Oxford. September 2012.
The TrainBuilder ATCA Data Acquisition Board for the European XFEL
25
Spare Slides
Presented by John Coughlan. TWEPP Oxford. September 2012.
The TrainBuilder ATCA Data Acquisition Board for the European XFEL
European-XFEL Hamburg
26Presented by John Coughlan. TWEPP Oxford. September 2012.
2D Camera captures 512 Images per TrainTrain repetition rate = 10 Hz
~5,000 fps
Xray Free Electron Laser
The TrainBuilder ATCA Data Acquisition Board for the European XFEL
Crosspoint Switch Operation
27Presented by John Coughlan. TWEPP Oxford. September 2012.
Switch Protocol : Xilinx Aurora 64/66B @ 2.5 GbpsXpoint Switch changes only @ 10 Hz
Partial Images 128 KBTime Ordered
Full Image Trains512 x 2 MB
A->Y
B->Z
C->W
D->X
A->X
B->Y
C->Z
D->W
A->Z
B->W
C->X
D->Y
A->W
B->X
C->Y
D->Z
A->Y
B->Z
C->W
D->X
A->X
B->Y
C->Z
D->W
A->Z
B->W
C->X
D->Y
A->W
B->X
C->Y
D->Z
1234A
EXT RAM
FPGA FIFO
X1234B
FIFO
1234C
RAM
FIFO
1234
1234
1234
1234
1234D
RAM
FIFO
A1D1C1B1 W
RAM
FIFO A1B1C1D1
B2A2D2C2
FIFO A2B2C2D2
C3B3A3D3 Y
RAM
FIFO A3B3C3D3
D4C4B4A4 Z
RAM
FIFO A4B4C4D4
CrosspointSwitch64 x 64Bi-dir links
8 Lanes@ ~2.5 Gbps
1 2 3 4
IIIIIIIV IIIIIIIV
2 x 10 Gbps2 x 10 GbLinks 1234A
EXT RAM
FPGA FIFO
X1234B
FIFO
1234C
RAM
FIFO
1234
1234
1234
1234
1234D
RAM
FIFO
A1D1C1B1 W
RAM
FIFO A1B1C1D1
B2A2D2C2
FIFO A2B2C2D2
C3B3A3D3 Y
RAM
FIFO A3B3C3D3
D4C4B4A4 Z
RAM
FIFO A4B4C4D4
CrosspointSwitch64 x 64Bi-dir links
8 Lanes@ ~2.5 Gbps
1 2 3 4
IIIIIIIV IIIIIIIV
2 x 10 Gbps2 x 10 GbLinks
A
C
B
D
A
C
B
D
REARINPUTS
FRONTOUTPUTS
FEEs PC Farm
DDR2 DDR2
The TrainBuilder ATCA Data Acquisition Board for the European XFEL
10G UDP
28
LL TX PKT
SPLIT
LL RXPKT
COMB
UDP TXBlock
UDP RXBlock
XAUICORE
Local Link Tx
Local Link Rx
64D8C
64D8C
RDMA RDMAREG
4 TX4RX312.5MHz 156.25 MHz
50 MHz
MDIO
XAUIPHY
GA0,GA1,SCLSDA
Ctrl & Mon
TX/RX ALARMRESET
ROM
• LL FRAME (64 bit) <-> UDP PKT STREAM• FPGA <-> FPGA - FPGA <-> PC
• Tx : MAC, IP ADDR & PORT LUT (LL Frame Header) *• Rx : IP, PORTS, MAC, Filter for PLAYBACK mode• Software Programmable : Packet Length (Jumbo), IFG ...etc
*Plan to add Resends on Tx for “Reliable UDP” (TB to PC Farm)
10G PHYFPGA
Presented by John Coughlan. TWEPP Oxford. September 2012.
Local Link
The TrainBuilder ATCA Data Acquisition Board for the European XFEL
PPC DDR2 Memory Controller
29Presented by John Coughlan. TWEPP Oxford. September 2012.
DDR2SODIMM
64 bit
2 GB
V5 FX100T 2 x PPC440 Hard IP blocksEach with 4 DMA engines Tx&Rx
DDR2 controller500 MT/s
Local Link Interface 32 bit @ 250 MHz
~ 1 GB/sec Concurrent Read & Write
LL Write
LL Read
Embedded C code manages DMA engines/buffers