Switching for BTeV Level 1 Trigger

Jinyuan Wu(For the BTeV Collaboration)

Jinyuan WuBTeV trigger

Level 1 vertex trigger architecture

FPGA segment finders

Merge

Trigger decision to Global Level 1

Switch: sort by crossing number

track/vertex farm(~2500 processors)

30 station pixel detector

Jinyuan WuBTeV trigger

DAQ From M. Bowden’s Talk

Jinyuan WuBTeV trigger

GlobalLevel-1

ITCH

Information Transfer Control Hardware

GL1

Level-1 Buffers

12 x 24-port Fast Ethernet Switches

Level 2/3Processor Farm

Pixel Processors

FPGA Segment Finder

Track/Vertex Farm

Gigabit Ethernet Switch

Data Combiners +Optical Transmitters

OpticalReceivers

BTeV Detector

Front End Boards

8 Data Highways

Data Logger

Cross Connect Switch

BTeV Trigger (From M. Wang’s Talk)

Jinyuan WuBTeV trigger

Data From Pixel, Organized in Highways

This is DCB.

The PDCB and DCB look similar.

There are 10 sub-racks for pixel system.

There are 8 cables per sub-rack.

Each cable has 12 fibers.

• 80 cables, 960 fibers• 10 cables/hwy

Each card sees ¼ station, 12

cards/sub-rack.

Each cable goes to one

highway.

Jinyuan WuBTeV trigger

Pre-Processor

Pre-Processor

Pre-Processor

Pre-Processor

1

2

F

0

1

2

F

0

1

2

F

0

1

2

F

0

1

2

F

0

1

2

F

0

1

2

F

0

1

2

F

0

1

2

F

0

1

2

F

0

Optical Receivers to Segment Finder

Optical Receivers, 10 cards/hwy Pre-processors, 32 cards/hwy

SegmentFinder

Segment Finders, 32 cards/hwy

TO Farmlet

Pre-Processor

SegmentFinder

TO L1 Buffer:

From PDCB

Jinyuan WuBTeV trigger

Optical Receiver

0,10,20…

1,11,21…

2,12,22…

F,1F,2F…

IN: 12 fibers 1 cable/card

OUT: 16 cables/cardLVDS 4 pairs/cable<470 Mb/s per pair

1

2

F

0

Rate/cable:2.5 Gb/s x 12/16= 2.5 Gb/s x 3/4LVDS 4 pairs/cable<470 Mb/s per pair

Rate/fiber:2.5 Gb/s

Optical Receivers, 10 cards/hwy, 80 cards total

Jinyuan WuBTeV trigger

Pre-processor

Rate/cable:2.5 Gb/s x 12/16 x 5/4= 2.5 Gb/s x 15/16LVDS 4 pairs/cable<<585 Mb/s per pair

Pre-processors, 32 cards/hwy, 256 cards total

Pre-Processor

IN from Optical Receivers5 cables/card

OUT to Segment Finders: 4 cables/card

OUT to L1 Buffers: 4 cables/card

Rate/cable:2.5 Gb/s x 12/16= 2.5 Gb/s x 3/4LVDS 4 pairs/cable<470 Mb/s per pair

Rate/cable:2.5 Gb/s x 12/16 x 5/4= 2.5 Gb/s x 15/16LVDS 4 pairs/cable<<585 Mb/s per pair

Jinyuan WuBTeV trigger

Segment Finder

Rate/cable:2.5 Gb/s x 12/16 x 5/4= 2.5 Gb/s x 15/16LVDS 4 pairs/cable<<585 Mb/s per pair

Segment Finders, 32 cards/hwy, 256 cards total

IN from Segment Finders: 4 cables/card

OUT to DSP Farmlets: 4 cables/card

Rate/cable:2.5 Gb/s x 12/16 x 5/4= 2.5 Gb/s x 15/16LVDS 4 pairs/cable<<585 Mb/s per pair

SegmentFinder

Jinyuan WuBTeV trigger

Pre-Processor

Pre-Processor

Pre-Processor

Pre-Processor

1

2

F

0

1

2

F

0

1

2

F

0

1

2

F

0

1

2

F

0

1

2

F

0

1

2

F

0

1

2

F

0

1

2

F

0

1

2

F

0

Optical Receivers to Segment Finder

IN: 120 fibers 10 cables/hwy

OR_PP: 160 cables

Optical Receivers, 10 cards/hwy Pre-processors, 32 cards/hwy

Ch: (1/120) BCO: (1/8)

Ch: (1/10) BCO: (1/8 x 1/16)

SegmentFinder

Segment Finders, 32 cards/hwy

Ch: (1/2) BCO: (1/8 x 1/16 x 5/4)

Ch: (all) BCO: (1/8 x 1/16 x 5/4 x 2/4)

TO Farmlet 128 cables

PP_SF:128 cables

Pre-Processor

SegmentFinder

TO L1 Buffer: 128 cables

Jinyuan WuBTeV trigger

Short Compares

TDR New

L1 Switching Function

To Farmlet: Yes

To L1 Buffer: No

To Farmlet: Yes

To L1 Buffer: Yes

L1 Trigger Switch Hardware

Yes ($800K) No ($0)

Bandwidth from Preprocessor to Segment Finder

2.8 x

Input bandwidth

1 x

Input bandwidth

Number of cards/hwy

OR: 10

PP: 120 (60, 30?)

SF: 56 (28?)

OR: 10

PP: 32

SF: 32 (64?)

Optical Receiver Simple, FPGA not even needed

Need FPGA

Latency Minimum Longer, Need Study

Jinyuan WuBTeV trigger

To Do and Questions

• Optical receiver.• Latency budget.• Last stations.• Reduce switching in DAQ stages?• Further absorb segment finder into preprocessor?

Jinyuan WuBTeV trigger

End

End

Jinyuan WuBTeV trigger

Pixel data readout

Counting RoomCollision Hall

Pixelprocessor

Pixelprocessor

Pixelprocessor

FPGAsegment finder

to neighboring FPGAsegment finder

to neighboring FPGAsegment finder

Pixel stations

Optical links

Pixel processor

time-stamp expansion

time ordering

clustering algorithm

xy table lookup

FPIX2 Read-out chip

DCB

DCB

DCB

Data combiners

Row (7bits) Column (5bits) BCO (8bits) ADC (3bits)

sync (1bit)

Chip ID (13bits)

Jinyuan WuBTeV trigger

Optical Receiver

Optical Receivers, 80 cards full system

12 fibers in 1 cable/card

8 highways/card16 cables/card

0,8,16,24,32,40,48,56,64,72,80,88,96,104,112,120

1,9,17,25,33,41,49,57,65,73,81,89,97,105,113,121

2,10,18,26,34,42,50,58,…

3,11,19,27,35,43,51,59,…

4,12,20,28,36,44,52,60,…

5,13,21,29,37,45,53,61,…

6,14,22,30,38,46,54,62,…

7,15,23,31,39,47,55,63,…

0,64,…8,72,…16,80,…24,88,…

32,96,…40,104,…48,112,…56,120,…

LVDS 4 pairs/cable LVDS 8 pairs/card/hwy

<470 Mb/s per pair

Jinyuan WuBTeV trigger

The 4x4 Cable Bundle

Jinyuan WuBTeV trigger

Switching in the Processing Stages

Optical Receivers, 80 cards full system

12 fibers in 1 cable/card

8 highways/card16 cables/card

LVDS 4 pairs/cable<470 Mb/s per pair

Pre-processors, 32 cards/hwy

160 cables/hwy640 pairs/hwy

80 cables960 fibers

5 cables20 pairs/card

To L1Buffer

To SegmentFinder

Cable bundle4 x 4

Ch: (1/80) BCO: (all)

Ch: (1/80) BCO: (1/8 x 1/2)

Ch: (1/4) BCO: (1/8 x ½ x ¼ )

Segment Finders,40 cards/hwy 4 cables/card

Ch: (all) BCO: (1/8 x ½ x ¼ x 1/5)

To DSPFarmlet

Jinyuan WuBTeV trigger

L2/L3 PC

To L1 Buffer, Etc.

Pre-processors, 32 cards/hwy

5 cables20 pairs/card

To L1Buffer

To SegmentFinder

Cable bundle4 x 4

Ch: (1/4) BCO: (1/8 x ½ x ¼ )

Segment Finders,40 cards/hwy 4 cables/card

Ch: (all) BCO: (1/8 x ½ x ¼ x 1/5)

To DSPFarmlet

4 cables/card

Ch: (all) BCO: (1/8 x ½ x ¼ x 1/5)

L1 Buffer

SF SF SF SF

Jinyuan WuBTeV trigger

In One Highway:

Optical Receivers, 10 cards/hwy

12 fibers in 1 cable/card

16 cables/cardLVDS 4 pairs/cable<470 Mb/s per pair

Pre-processors, 32 cards/hwy

160 cables/hwy

10 cables/hwy120 fibers/hwy

5 cablesTo L1Buffer

To SegmentFinder

Ch: (1/10) BCO: (1/8)

Ch: (1/10) BCO: (1/8 x 1/16)

Ch: (1/2) BCO: (1/8 x 1/16 )

Segment Finders,40 cards/hwy 4 cables/card

Ch: (all) BCO: (1/8 x 1/16 x 2/5)

To DSPnodes

Jinyuan WuBTeV trigger

In One Highway:

Optical Receivers, 10 cards/hwy

IN: 12 fibers 1 cable/card

OUT: 16 cables/cardLVDS 4 pairs/cable<470 Mb/s per pair

Pre-processors, 32 cards/hwy

160 cables/hwy

10 cables/hwy120 fibers/hwy

IN: 5 cablesOUT: 8 cables

To L1Buffer

To SegmentFinder

Ch: (1/10) BCO: (1/8)

Ch: (1/10) BCO: (1/8 x 1/16)

Ch: (1/2) BCO: (1/8 x 1/16 )

Segment Finders,64 cards/hwy IN: 4 cables

Ch: (all) BCO: (1/8 x 1/16 x 4/8)

To DSPnodes

Jinyuan WuBTeV trigger

1

2

F

0

1

2

F

0

1

2

F

0

1

2

F

0

1

2

F

0

1

2

F

0

1

2

F

0

1

2

F

0

1

2

F

0

1

2

F

0

Optical Receivers to Segment Finder

IN: 120 fibers 10 cables/hwy

OR_PP: 160 cables

Optical Receivers, 10 cards/hwy

Pre-Processor

Pre-processors, 32 cards/hwy

Ch: (1/120) BCO: (1/8)

Ch: (1/10) BCO: (1/8 x 1/16)

Pre-Processor

Pre-Processor

Pre-Processor

Pre-Processor

SegmentFinderSegment

FinderSegment

Finder

Segment Finders, 64 cards/hwy

Ch: (1/2) BCO: (1/8 x 1/16 x 1/8)

Ch: (all) BCO: (1/8 x 1/16 x 1/8 x 2/4)

TO Farmlet 256 cables

PP_SF:256 cables

Jinyuan WuBTeV trigger

BTeV trigger overview

BTeV detector

L1 muon

L1 vertex

GlobalLevel-1

Level-1

Level 2/3 Crossing Switch

Data Logging

Front-end electronics

Level-1 Buffers

Level-2/3 Buffers

Information Transfer Control Hardware

ITCH

Level-2/3 Processor Farm#1

#2#m-1

#m

RDY

Crossing #N

Req. data for crossing #N

Level-3 accept

GL1 accept

PIX

> 2 x 10 channels7

800 GB/s7.6 MHz

L1 rate reduction: ~100x

L2/3 rate reduction: ~20x

4 KHz