First P2P Measurements on Infiniband

Luciano BertiINFN Laboratori Nazionali di Legnaro

Infiniband in briefSame networkto transport lowlatency ipc, storage I/O and network I/OInternet IntranetLink speed1x 2.5 Gbps4x 10.0 Gbps.12x 30 Gbps

RouterxCALinkLinkCPUMem CntlrHCACPUMem CntlrHCACPUMem CntlrHCALinkCPUMem CntlrHCASwitchLinkLinkLinkLinkLinkTCAN/WTargetTCAStorageTargetLinkLinkLink123nChannel basedmessage passing1000s nodeper subnet1 x Link4 x Link12 x Link

Infiniband Transport ProtocolsIBA has been developed with Virtual Interface in mind. VIPL 2.0 includes IBA extensions and RDMA operations. SCSI RDMA Protocol (SRP). It is a T10 standard.SRP defines mapping to IBA architectureit is the transport protocol over IBASRP is based on VIDirect Access Files System (DAFS)Direct Access Socket (DAS)TCP/IP functionality over VI/IBIBA Host Channel AdapterVirtual Interface over IBDASDAFSSRPfast, low latencyTCP/IP socketsFile AccessBlock Accessfast, low latency network storage

LNL Infiniband Test BedLeaf Switch32 1x (2.5 Gbps) ports in 1 U chassisPCI-X (max ~ 380 Mbyte/s)Supermicro P4DL6IBA Host Channel Adapter

Status of Infiniband test bedAll the hardware has been provided by Infiniswitch (1 switch + 4 HCA)

All the hardware is up and running

First p2p measurements have been performed

SoftwareVirtual Interface Library (VIPL) as provided by InfiniswitchSend/Receive over reliable connectionsRDMA over reliable connections

Sourceforge has a infiniband project over LinuxVIPL source is available. Compiled and works!Performance as the infiniswitch VIPL (probably they are the same code)ResultsRound trip time small buffers ~ 40 msec (latency 20 msec)P2p ~ 80% link saturation

First IB p2p tests220Link SaturationMbyte/sBuffer Size (Bytes)~ 40 Mbyte/s

Work Plan2 months IB hardware evaluation (started end of May)

Mainly focus on functionality tests both of the hardware and software available:SRP SCSI remote protocol already got, to be testedDAS and DAFS if early version will be available

Contacts started with major companies (IBM, Dell, Fujitsu/Siemens) to understand their IB products road map