Legnaro Event Builder Prototypes

G. Maron CPT Week CERN, 23 April 2001 1

LNL

CMS

Legnaro Event Builder PrototypesLegnaro Event Builder Prototypes

Luciano Berti, Gaetano MaronLuciano Berti, Gaetano Maron

INFN – Laboratori Nazionali di LegnaroINFN – Laboratori Nazionali di Legnaro


LNL

CMS

GE Event BuilderGE Event Builder

Test conditions:Test conditions:• No command or event aggregation (each packet No command or event aggregation (each packet transports a command or data frame relative to a single transports a command or data frame relative to a single event)event)• full data transfer from/to PC memoryfull data transfer from/to PC memory• recovery from packet lossrecovery from packet loss• fixed fragment sizes are varied 400-4000 bytesfixed fragment sizes are varied 400-4000 bytes

Components:Components:Hardware: Hardware: • switch: FoundryNet FastIron switch: FoundryNet FastIron • NIC: SysKonnect SK9821NIC: SysKonnect SK9821• PC: Supermicro PIII (i840) PC: Supermicro PIII (i840) Software: vxWorks basedSoftware: vxWorks based 15 x 15


LNL

CMS

Event builder layoutEvent builder layout

1 1515141413131212111110109988776655443322

1 1515141413131212111110109988776655443322

EVMEVM

RUsRUs

BUs

Slot 1Slot 1 Slot 2Slot 2 Slot 3Slot 3 Slot 4Slot 4

RUs and BUs distributed in all switch slots:RUs and BUs distributed in all switch slots:

– Part of the traffic localized within the slotPart of the traffic localized within the slot

– Reduces switch backplane utilizationReduces switch backplane utilization

RU performanceRU performance problem found withproblem found with this configuration this configuration


LNL

CMS

Modified Event Builder layoutModified Event Builder layout

11 1515141413131212111110109988776655443322

11 1515141413131212111110109988776655443322

EVMEVM

RUsRUs

BUsBUs

Slot 1Slot 1 Slot 2Slot 2 Slot 3Slot 3 Slot 4Slot 4

Request data Request data commandscommands

Fast EthernetFast EthernetSlotSlot

Request data Request data commandscommands

- RU fast control message over FE (PCI 32/33)RU fast control message over FE (PCI 32/33)- RU data transfer on GE (PCI 64/66) RU data transfer on GE (PCI 64/66)


LNL

CMS

The GE Event BuilderThe GE Event Builder


LNL

CMS

EB protocolEB protocol

BUBU

allocateallocate

EVMEVM

confirmconfirm

11 22 33 nn

RUsRUs

sendsend

cachecache


LNL

CMS

Concurrent building threads in the same BUConcurrent building threads in the same BU

BUBU EVMEVM 11 22 33 nn

RUsRUs

BU thread 1BU thread 1BU thread 2BU thread 2BU thread 3BU thread 3


LNL

CMS

BU

allocate

EVM

confirm

Sequential readingSequential reading

1 2 3 n

RUs

BU

allocate

EVM

confirm

Random reading

1 2 3 n

RUs

4 5

send

cache

send

cache

Sequential vs Random readingSequential vs Random reading


LNL

CMS

BU

allocate

EVM

confirm

1 2 3 n

RUs

4 5

send

cache

““Sliding Window”Sliding Window”

• multiple send to Rus multiple send to Rus

• reduce the total reduce the total rebuilding timerebuilding time

• less events in the less events in the BusBus

• not yet testednot yet tested


LNL

CMS

• No difference on performanceNo difference on performance

• But more allocated event But more allocated event needed on BUs, needed on BUs,

• All the measurements with All the measurements with random readingrandom reading

Sequential readingSequential reading Random readingRandom reading

Sequential - random reading comparisonSequential - random reading comparison


LNL

CMS

Recovery from Packets lossRecovery from Packets loss

BUBU

Req. DataReq. Data

RURU timertimer

startstart

EvtDataEvtData

Req. DataReq. Data( retry ) ( retry )

startstart

cancelcancel

timeouttimeout

EvtDataEvtData

BUBU

Req. EvtIdReq. EvtId

EVMEVM timertimer

startstart

EvtIdEvtId

Req. EvtIdReq. EvtId(retry)(retry)startstart

cancelcancel

timeouttimeout

EvtIdEvtId

BU – EVM communicationBU – EVM communication BU – RU communicationBU – RU communication

TimeoutsTimeouts80 - 160 ms80 - 160 ms


LNL

CMS

EVB 15x15 performance - ThroughputEVB 15x15 performance - Throughput

0

20

40

60

80

100

120

140

0 500 1000 1500 2000 2500 3000 3500 4000

Fragment Size (Byte)

Thro

ughp

ut p

er n

ode

(MB/

s)

• Throughput up to 116 MB/s, ie 93% link speedThroughput up to 116 MB/s, ie 93% link speed• no packet loss observed (as expected)no packet loss observed (as expected)

15 x 15


LNL

CMS

EVB ScalingEVB Scaling

0

20

40

60

80

100

120

140

0 2 4 6 8 10 12 14 16

N

Th

rou

gh

pu

t p

er

no

de

(M

B/s

)

4000 bytes

2000 bytes

400 bytes


LNL

CMS

EVB Performance – Event RateEVB Performance – Event Rate

0

20

40

60

80

100

120

0 500 1000 1500 2000 2500 3000 3500 4000 4500

fragment size (bytes)

Fra

gm

en

t ra

te p

er

no

de

(k

Hz)

Nominal fragmentNominal fragmentsize 2kbytes:size 2kbytes:event rate = 52 kHzevent rate = 52 kHz

15 x 15


LNL

CMS

Conic Event BuilderConic Event Builder

EventEvent ManagerManager

Builder NetworkBuilder Network

RURU

BUBU

FUFU

FUFU

FUFU

FUFU

EventEvent ManagerManager

RURU

FUFU FUFU FUFU FUFU FUFU FUFU FUFU FUFU

Builder NetworkBuilder Network

FUFU FUFU FUFU FUFU

symmetric EVBsymmetric EVB conic EVBconic EVB

• faster ports at Rusfaster ports at Rus• slower ports at BUsslower ports at BUs


LNL

CMS

Conic Event Builder: LayoutConic Event Builder: Layout

11 443322

11 292927272525232321211919171715151313111199775533

EVMEVM

RUsRUs

FUsFUs

GE GE Slot 1Slot 1

Request Data Request Data CommandCommand

39393737353533333131

22 30302828262624242222202018181616141412121010886644 40403838363634343232

FE FE Slot 1Slot 1

FE FE Slot 2Slot 2


LNL

CMS

EVB throughput – Conic vs SymmetricEVB throughput – Conic vs Symmetric

0

20

40

60

80

100

120

140

0 500 1000 1500 2000 2500 3000 3500 4000

Fragment Size (Byte)

Th

rou

gh

pu

t p

er

no

de

(M

B/s

)

symmetric 15x15

conic 4x40

conic EVB: no performance degradation vs symmetric conic EVB: no performance degradation vs symmetric

4 x 40


LNL

CMS

EVB Conic – ScalingEVB Conic – Scaling

1 x 10

4 x 40

2 x 20

0

20

40

60

80

100

120

140

0 1 2 3 4 5

N

Th

rou

gh

pu

t p

er

no

de

(M

B/s

)

4000 bytes2000 bytes400 bytes


LNL

CMS

Conic: RU/FU Conic: RU/FU Throughput ratioThroughput ratio

1 x n

2 x n

4 x n


LNL

CMS

To be done and test To be done and test

• variable size eventsvariable size events

• EB performances with the new implemented “ window” EB performances with the new implemented “ window” mechanismmechanism

• latency times measurementslatency times measurements

• Fault generation with the new implemented Random Error Fault generation with the new implemented Random Error Generator to check the error recovery procedureGenerator to check the error recovery procedure


LNL

CMS

Multistage Event BuilderMultistage Event Builder

• All our results have been obtained with a single All our results have been obtained with a single switch event builder configurationswitch event builder configuration

• We propose to extend our tests to a multistage We propose to extend our tests to a multistage ethernet switches topology and to study the behavior ethernet switches topology and to study the behavior of this configuration.of this configuration.


LNL

CMS

Plain TopologyPlain Topology

• In the Event In the Event Builder application Builder application data flows in only data flows in only one directionone direction

• The inter-switch The inter-switch Gigabit Ethernet Gigabit Ethernet links are full-links are full-duplexduplex

• Result : half of the Result : half of the inter-switch inter-switch bandwidth bandwidth available is wastedavailable is wasted

RUsRUs

BUsBUs


LNL

CMS

Full Mesh TopologyFull Mesh Topology

• RU and BU RU and BU distributed in distributed in all the switchesall the switches

• Inter-switch Inter-switch links are used links are used in both in both directiondirection

• Same number Same number of ports of the of ports of the plain topologyplain topology

• Twice of the Twice of the bandwidth of bandwidth of the plain the plain topology in the topology in the inter-switch inter-switch linkslinks

RUsRUs

BUsBUs


LNL

CMS

Plain and mesh topology limitsPlain and mesh topology limits

• Each couple of switches is connected by a single linkEach couple of switches is connected by a single link• This is a bottleneck if the traffic is not uniformly randomThis is a bottleneck if the traffic is not uniformly random• The network is blocking for certain traffic patternsThe network is blocking for certain traffic patterns


LNL

CMS

Traffic with patternsTraffic with patterns

• If traffic has patterns (for example this could happens in the case If traffic has patterns (for example this could happens in the case the event builder is performed in steps) it could make sense to the event builder is performed in steps) it could make sense to introduce an artificial mechanism that randomise the traffic. introduce an artificial mechanism that randomise the traffic.

• This mechanism exist and it is called Universal Routing This mechanism exist and it is called Universal Routing


LNL

CMS

Universal Routing ReferenceUniversal Routing Reference

• Discovered by L.G. Valiant in 1980Discovered by L.G. Valiant in 1980

• See: M.D May, P.W. Thompson, P.H. Welch See: M.D May, P.W. Thompson, P.H. Welch NETWORKS,ROUTERS & TRASPUTER available NETWORKS,ROUTERS & TRASPUTER available on : http://www.pact.srf.ac.uk/macrame/papers/bluebook.htmlon : http://www.pact.srf.ac.uk/macrame/papers/bluebook.html

• Those papers describe the Universal Routing applied to Those papers describe the Universal Routing applied to Transputers Networks, a wormhole routing based networkTransputers Networks, a wormhole routing based network

• We adapted the same concept to a packet switched network We adapted the same concept to a packet switched network like Gigabit Ethernetlike Gigabit Ethernet


LNL

CMS

Universal Routing with GigaEthernet Universal Routing with GigaEthernet

• Based on Clos topologyBased on Clos topology• Multiple path available Multiple path available

between each couple of between each couple of switchesswitches

• Every packets is sent to Every packets is sent to a randomly chosen a randomly chosen intermediate switchintermediate switch

• The intermediate The intermediate switch send the packet switch send the packet to the final destinationto the final destination

• Full bandwidth Full bandwidth between each between each couple of switches couple of switches and uniform buffer and uniform buffer utilizationutilization


LNL

CMS

Universal RoutingUniversal Routing

• Transformation of the CLOS topology to a folded CLOS Transformation of the CLOS topology to a folded CLOS • The resulting number of ports is the same of the plain topologyThe resulting number of ports is the same of the plain topology

Half duplex links Full duplex links

RUs

BUs

RUs

BUs


LNL

CMS

Large (500x500) multistage GE network (1)Large (500x500) multistage GE network (1)

1111

22

2525

33

22

20 BUs20 BUs

2020

44

20 RUs20 RUs

20 Ports20 Ports 25 Ports25 Ports

40 Ports40 Ports

• 25 switches with 25 switches with 60 x 1Gb ports 60 x 1Gb ports

• 20 switches with 20 switches with 25 x 1 Gb ports 25 x 1 Gb ports


LNL

CMS

Large (500x500) multistage GE network (2)Large (500x500) multistage GE network (2)

• 25 switches with 25 switches with 40 x 1Gb ports + 40 x 1Gb ports + 2 x 10 Gb uplinks 2 x 10 Gb uplinks

• 2 switches with 2 switches with 25 x 10 Gb ports 25 x 10 Gb ports

11

11

22

2525

22

20 BUs

20 RUs

2 Ports 10G2 Ports 10G

25 Ports 10G25 Ports 10G40 Ports


LNL

CMS

Proposal for a multistage event builder demonstratorProposal for a multistage event builder demonstrator

• Multistage event builders can be emulated using the much Multistage event builders can be emulated using the much cheaper fast ethernet connections and switches. The GE speed cheaper fast ethernet connections and switches. The GE speed is not needed in these topological investigationsis not needed in these topological investigations

• The proposal is to have prototypes for:The proposal is to have prototypes for:– Full Mesh TopologyFull Mesh Topology

– Folded CLOS topology with (and without) Universal Routing Folded CLOS topology with (and without) Universal Routing mechanism mechanism


LNL

CMS

Full Mesh 64x64 Event Builder PrototypeFull Mesh 64x64 Event Builder Prototype

11

55

77

33

22

66

88

44

8 RUs8 RUs 8 BUs8 BUs- 1 host node 4 Rus / 4 BUs or - 1 host node 4 Rus / 4 BUs or a mix of thema mix of them- 32 hosts- 32 hosts- 128 FE NICS (56 + - 128 FE NICS (56 + 7272))- - 8 24 FE ports switch8 24 FE ports switch

Missing componentsMissing components


LNL

CMS

Folded CLOS 64x64 Event Builder PrototypeFolded CLOS 64x64 Event Builder Prototype

1111

22

44

33

22

16 BUs16 BUs

1616

44

16 RUs16 RUs

16 Ports16 Ports 4 Ports4 Ports

32 Ports32 Ports

33

FastIron with 3 24 FE ports modsFastIron with 3 24 FE ports mods

- 1 host node 4 Rus / 4 BUs or - 1 host node 4 Rus / 4 BUs or a mix of thema mix of them- 32 hosts- 32 hosts- 128 FE NICS (56 + - 128 FE NICS (56 + 7272))- - 4 48 FE ports switch4 48 FE ports switch- 2 24 FE ports FastIron module- 2 24 FE ports FastIron module+ 1 24 FE ports FastIron module+ 1 24 FE ports FastIron module

Missing componentsMissing components

- 1 host node 3 Rus / 3 BUs or - 1 host node 3 Rus / 3 BUs or a mix of thema mix of them- 32 hosts- 32 hosts- 96 FE NICS (56 + - 96 FE NICS (56 + 4040))- - 4 36 FE ports switch4 36 FE ports switch- 2 24 FE ports FastIron module- 2 24 FE ports FastIron module

64x6464x64

48x4848x48


LNL

CMS

Folded CLOS 80x80 Event Builder PrototypeFolded CLOS 80x80 Event Builder Prototype

11

11

22

44

20 BUs20 BUs

20 RUs20 RUs

2 GE Ports2 GE Ports 4 Ports4 Ports

40 FE40 FEPortsPorts

33

FastIron with 8 GE ports moduleFastIron with 8 GE ports module(1000 BaseT or (1000 BaseT or 1000 BaseSX1000 BaseSX))

- 1 host node 4 Rus / 4 BUs or - 1 host node 4 Rus / 4 BUs or a mix of thema mix of them- 40 hosts (32+- 40 hosts (32+88))- 160 FE NICS (56 +- 160 FE NICS (56 +104104))- - 4 48 FE ports + 2 GE links switch4 48 FE ports + 2 GE links switch- 1 8 GE (- 1 8 GE (Base SXBase SX) ports FastIron ) ports FastIron modulemodule

Missing componentsMissing components80x8080x80

22


LNL

CMS

8x80 Conic Event Builder Prototype8x80 Conic Event Builder Prototype

11

22

44

20 FUs20 FUs

2 GE Ports2 GE Ports

20 FE20 FEPortsPorts

33

RU1RU1

RU2RU2

RU3RU3

RU4RU4

RU5RU5

RU6RU6

RU7RU7

RU8RU8

FastIron with 2 8 GE ports modulesFastIron with 2 8 GE ports modules(1000 BaseT or (1000 BaseT or 1000 BaseSX1000 BaseSX))

- 1 host node 4 FUs - 1 host node 4 FUs - 20 hosts - 20 hosts - 80 FE NICS (56 + - 80 FE NICS (56 + 2424))- - 4 24 (48) FE ports + 2 GE up 4 24 (48) FE ports + 2 GE up links switchlinks switch- 2 8 GE (- 2 8 GE (Base SXBase SX) ports FastIron ) ports FastIron modulemodule

Missing componentsMissing components8x808x80


LNL

CMS

Material for the event builder multistage prototypesMaterial for the event builder multistage prototypes

Mesh 64x64Mesh 64x64- 72 FE NICs- 72 FE NICs- - 8 24 FE ports8 24 FE ports

Folded CLOS 80x80Folded CLOS 80x80- 8 PCs- 8 PCs- 104 FE NICS- 104 FE NICS- - 4 48 FE ports with 2 GE uplinks 1000 baseT4 48 FE ports with 2 GE uplinks 1000 baseT

if the 1000 baseT uplinks are not available:if the 1000 baseT uplinks are not available:

1) Folded CLOS 64x641) Folded CLOS 64x64::72 FE NICs72 FE NICs4 48 FE ports switch4 48 FE ports switch1 24 FE ports FastIron module1 24 FE ports FastIron module

2) Folded CLOS 48x482) Folded CLOS 48x48::40 FE NICs40 FE NICs4 >36 FE ports switch4 >36 FE ports switch

Legnaro Event Builder Prototypes

Documents

Transcript of Legnaro Event Builder Prototypes