OpenFabrics 2.0or libibverbs 1.0

Sean Hefty

Intel Corporation

Pre-OpenIB

• Multiple IB vendors each with a proprietary software stack– Multiple versions of the ‘verbs’ interface– Binary incompatible

• Formation of OpenIB– Focused on creating a single, open source software

stack for InfiniBand– Apps have 1 API to target, distros have 1 solution to

support– Greatly increased adoption

www.openfabrics.org 2

OpenIB

• Resulting software stack was hardware facing– Verbs is a hardware semantic– Was never intended to be an API– Application developers have always disliked it

• OpenIB expanded to include iWarp– Changed name to OpenFabrics– Folded iWarp support under InfiniBand API

www.openfabrics.org 3

OpenFabrics Today

• The fundamental stack is largely unchanged• The result:

– There are multiple, binary incompatible versions of the ‘verbs’ interface

– There are vendor-specific APIs• FCA, MXM, PSM, UCCS

• Sounds a lot like the pre-OpenIB days

www.openfabrics.org 4

Why Not Just Extend Verbs?

• Author’s prediction: will not succeed– OIB created a single verbs solution, which later split

• The split was a direct result of needing extensions

– Additional APIs have been already been developed that are NOT part of verbs or extended verbs

• Existence proof against

– There are fundamental issues with the API that will prohibit scaling to very large systems

www.openfabrics.org 5

Proposed Alternative

• Develop application facing APIs and let each vendor determine the best way to support those APIs– There is no intent to disadvantage any hardware

solution

• Define mechanisms for migrating providers and apps to the new framework

www.openfabrics.org 6

(Scalable) Fabric Interfaces

Q: What is implied by incorporating interface sets under a single framework?

Objects exist that are usable between the interfacesIsolated interfaces turn the framework into a complex dlopen

Interfaces are composableMay be used together

www.openfabrics.org 7

Fabric InterfacesMessage Queue

ControlInterface RDMA Atomics

Active Messaging

Tag Matching

Collective OperationsCM Services

Cohesive interfaces,not merely a union

Migrating Providers from Verbs to FI

libfabric

CM Services

libibverbs

FI

RDM

A CM

Verbs Provider

Message Queue RDMA

Verbs

Verbs Provider

ibverbs abstraction layer

Providers can integrate into framework with minimal changes

Providers leverage abstraction layer and RDMA CM capabilities

Add optimize data transfers and completion processing calls to avoid translation overhead

www.openfabrics.org 8

Migrating Apps from Verbs to FI

• Expose ‘verbs’ interfaces directly from FI

• Use macros to convert ‘libibverbs’ exported calls to FI calls

• Or layer libibverbs over libfabric

• Applications must recompile• Minimal benefit to app

FID

Domain

Shared resources

Socket

Reliable datagram message

Message queue

Memory descriptor

Event collector

Completion queue

CM events

Counter

Address vector

Maps

Tables

Interface

uverbs

ucma

www.openfabrics.org 9

Migrating Apps from Verbs to FI

• Define ‘verbs’ compatibility mode– Allows mapping objects

between interfaces– E.g. QP fabric socket

CQ EC– Restricts implementation– Mapping must be documented

• Allow software to adopt new interfaces selectively– E.g. send/recv/ec_read

libfabric

Dual-Provider Library

verbs

libibverbs

SFI

SFI ProviderVerbs Provider

www.openfabrics.org 10