First P2P Measurements on Infiniband Luciano Berti INFN Laboratori Nazionali di Legnaro
7
CMS week, June 2002, CERN 1 First P2P Measurements on Infiniband Luciano Berti INFN Laboratori Nazionali di Legnaro
description
First P2P Measurements on Infiniband Luciano Berti INFN Laboratori Nazionali di Legnaro. Infiniband in brief. Storage Target. Channel based message passing. TCA. Internet Intranet. Mem Cntlr. CPU. 1. HCA. Link. Link. Link. Link. Switch. Mem Cntlr. CPU. HCA. 2. Link. Link. - PowerPoint PPT Presentation
Transcript of First P2P Measurements on Infiniband Luciano Berti INFN Laboratori Nazionali di Legnaro
CMS week, June 2002, CERN 1
First P2P Measurements on Infiniband
Luciano BertiINFN Laboratori Nazionali di Legnaro
CMS week, June 2002, CERN 2
Infiniband in brief
Same networkto transport lowlatency ipc, storage I/O and network I/O
Internet Intranet
Link speed1x 2.5 Gbps4x 10.0 Gbps.12x 30 Gbps
Router
xCA
Lin
k
Link
CPUMemCntlr HCA
CPUMemCntlr HCA
CPUMemCntlr HCA
Link
CPUMemCntlr HCA
Switch
Lin
k
Link
Link
Lin
k
LinkTCA
N/WTarget
TCA
StorageTarget
Link
Link
Lin
k
1
2
3
n
Channel basedmessage passing
1000’s nodeper subnet
1 x Link 4 x Link 12 x Link
CMS week, June 2002, CERN 3
Infiniband Transport Protocols
IBA has been developed with Virtual Interface in mind. VIPL 2.0 includes IBA extensions and RDMA operations.
SCSI RDMA Protocol (SRP). It is a T10 standard.– SRP defines mapping to IBA architecture– it is the transport protocol over IBA– SRP is based on VI
Direct Access Files System (DAFS) Direct Access Socket (DAS)
– TCP/IP functionality over VI/IB
IBA Host Channel Adapter
Virtual Interface over IB
DAS DAFS SRP
fast, low latencyTCP/IP sockets File Access Block Access
fast, low latency network storage
CMS week, June 2002, CERN 4
LNL Infiniband Test Bed
Leaf Switch32 1x (2.5 Gbps) ports
in 1 U chassis
PCI-X (max ~ 380 Mbyte/s)
Supermicro P4DL6
IBA Host Channel Adapter
CMS week, June 2002, CERN 5
Status of Infiniband test bed All the hardware has been provided by Infiniswitch (1 switch + 4 HCA)
All the hardware is up and running
First p2p measurements have been performed
Software– Virtual Interface Library (VIPL) as provided by Infiniswitch
Send/Receive over reliable connections RDMA over reliable connections
– Sourceforge has a infiniband project over Linux VIPL source is available. Compiled and works! Performance as the infiniswitch VIPL (probably they are the same code)
Results– Round trip time small buffers ~ 40 sec (latency 20 sec)– P2p ~ 80% link saturation
CMS week, June 2002, CERN 6
First IB p2p tests220 Link Saturation
Mb
yte
/s
Buffer Size (Bytes)
~ 40 Mbyte/s
CMS week, June 2002, CERN 7
Work Plan
2 months IB hardware evaluation (started end of May)
Mainly focus on functionality tests both of the hardware and software available:– SRP – SCSI remote protocol – already got, to be tested– DAS and DAFS if early version will be available
Contacts started with major companies (IBM, Dell, Fujitsu/Siemens) to understand their IB products road map
