CMS week, June 2002, CERN 1

First P2P Measurements on Infiniband

Luciano BertiINFN Laboratori Nazionali di Legnaro

CMS week, June 2002, CERN 2

Infiniband in brief

Same networkto transport lowlatency ipc, storage I/O and network I/O

Internet Intranet

Link speed1x 2.5 Gbps4x 10.0 Gbps.12x 30 Gbps

Router

xCA

Lin

k

Link

CPUMemCntlr HCA

CPUMemCntlr HCA

CPUMemCntlr HCA

Link

CPUMemCntlr HCA

Switch

Lin

k

Link

Link

Lin

k

LinkTCA

N/WTarget

TCA

StorageTarget

Link

Link

Lin

k

1

2

3

n

Channel basedmessage passing

1000’s nodeper subnet

1 x Link 4 x Link 12 x Link

CMS week, June 2002, CERN 3

Infiniband Transport Protocols

IBA has been developed with Virtual Interface in mind. VIPL 2.0 includes IBA extensions and RDMA operations.

SCSI RDMA Protocol (SRP). It is a T10 standard.– SRP defines mapping to IBA architecture– it is the transport protocol over IBA– SRP is based on VI

Direct Access Files System (DAFS) Direct Access Socket (DAS)

– TCP/IP functionality over VI/IB

IBA Host Channel Adapter

Virtual Interface over IB

DAS DAFS SRP

fast, low latencyTCP/IP sockets File Access Block Access

fast, low latency network storage

CMS week, June 2002, CERN 4

LNL Infiniband Test Bed

Leaf Switch32 1x (2.5 Gbps) ports

in 1 U chassis

PCI-X (max ~ 380 Mbyte/s)

Supermicro P4DL6

IBA Host Channel Adapter

CMS week, June 2002, CERN 5

Status of Infiniband test bed All the hardware has been provided by Infiniswitch (1 switch + 4 HCA)

All the hardware is up and running

First p2p measurements have been performed

Software– Virtual Interface Library (VIPL) as provided by Infiniswitch

Send/Receive over reliable connections RDMA over reliable connections

– Sourceforge has a infiniband project over Linux VIPL source is available. Compiled and works! Performance as the infiniswitch VIPL (probably they are the same code)

Results– Round trip time small buffers ~ 40 sec (latency 20 sec)– P2p ~ 80% link saturation

CMS week, June 2002, CERN 6

First IB p2p tests220 Link Saturation

Mb

yte

/s

Buffer Size (Bytes)

~ 40 Mbyte/s

CMS week, June 2002, CERN 7

Work Plan

2 months IB hardware evaluation (started end of May)

Mainly focus on functionality tests both of the hardware and software available:– SRP – SCSI remote protocol – already got, to be tested– DAS and DAFS if early version will be available

Contacts started with major companies (IBM, Dell, Fujitsu/Siemens) to understand their IB products road map