© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr

1

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 1

Open Standards for Interoperability

OpenFabrics

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialCisco booth 2

The OpenFabrics Alliance

Alliance of InfiniBand andiWarp vendors

Produce a common driverstackInteroperability between allvendors

Open source driversDrivers in Linux kernel treeDistributed in Red Hat andSuSE

© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr

2

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialCisco booth 3

Open source development

All InfiniBand vendors participate in developmentSource code in OpenFabrics Subversion and Git repositoriespublicly available

Cisco drives the verbs developmentKernel and user layer APIsMellanox hardware drivers

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialCisco booth 4

OpenFabrics Software Stack

RDMA NICR-NIC

Host Channel AdapterHCA

User Direct AccessProgramming Lib

UDAPL

Reliable Datagram ServiceRDS

iSCSI RDMA Protocol(Initiator)

iSER

SCSI RDMA Protocol(Initiator)

SRP

Sockets Direct ProtocolSDP

IP over InfiniBandIPoIB

Performance Manager AgentPMA

Subnet Manager AgentSMA

Management DatagramMAD

Subnet AdministratorSA

Common

InfiniBand

iWARP

Key

InfiniBand HCAInfiniBand HCA iWARP R-NICiWARP R-NIC

HardwareSpecific Driver

Hardware SpecificDriver

ConnectionManagerMAD

InfiniBand Verbs / API

SA Client

ConnectionManager

Connection ManagerAbstraction (CMA)

User Level Verbs / API

SDPIPoIB SRP iSER RDS

UDAPL

SDP Library

User Level MAD API

Open SM

DiagTools

Hardware

Provider

Mid-Layer

UpperLayerProtocol

UserAPIs

Kernel SpaceUser Space

NFS-RDMARPC

ClusterFile Sys

ApplicationLevel

SMA

R-NIC Driver API

ClusteredDB Access

(Oracle10g RAC)

SocketsBasedAccess

(IBM DB2)

VariousMPIs

Access to File

Systems

BlockStorageAccess

IP BasedApp

Access

Apps & Access

Methodsfor usingOF Stack

© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr

3

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialCisco booth 5

OpenFabrics Software Stack

RDMA NICR-NIC

Host Channel AdapterHCA

User Direct AccessProgramming Lib

UDAPL

Reliable Datagram ServiceRDS

iSCSI RDMA Protocol(Initiator)

iSER

SCSI RDMA Protocol(Initiator)

SRP

Sockets Direct ProtocolSDP

IP over InfiniBandIPoIB

Performance Manager AgentPMA

Subnet Manager AgentSMA

Management DatagramMAD

Subnet AdministratorSA

Common

InfiniBand

iWARP

Key

InfiniBand HCAInfiniBand HCA iWARP R-NICiWARP R-NIC

HardwareSpecific Driver

Hardware SpecificDriver

ConnectionManagerMAD

InfiniBand Verbs / API

SA Client

ConnectionManager

Connection ManagerAbstraction (CMA)

User Level Verbs / API

SDPIPoIB SRP iSER RDS

UDAPL

SDP Library

User Level MAD API

Open SM

DiagTools

Hardware

Provider

Mid-Layer

UpperLayerProtocol

UserAPIs

Kernel SpaceUser Space

NFS-RDMARPC

ClusterFile Sys

ApplicationLevel

SMA

R-NIC Driver API

ClusteredDB Access

(Oracle10g RAC)

SocketsBasedAccess

(IBM DB2)

VariousMPIs

Access to File

Systems

BlockStorageAccess

IP BasedApp

Access

Apps & Access

Methodsfor usingOF Stack

Developed by Cisco

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialCisco booth 6

OpenFabrics Enterprise Distribution

Release vehicle for OpenFabrics softwareSingle stack supported by all InfiniBand vendors

Enterprise-class supportFully supported by Cisco Technical Assistance Center

© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr

4

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialCisco booth 7

Software Availability

Community source availableOFED releases available on www.openfabrics.com

Cisco-packaged RPMs available on www.cisco.comThoroughly qualified and tested with Cisco hardware

Full documentation available

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 8

Open MPI

Open standards for interoperability

© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr

5

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialCisco booth 9

MPI From Scratch!

Developers of FT-MPI, LA-MPI, LAM/MPIKept meeting at conferences in 2003Culminated at SC 2003: Let’s start overOpen MPI was born

Started serious design and coding work January 2004All of MPI except one-sided operationsFirst release 1Q 2005

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialCisco booth 10

MPI From Scratch: Why?

Each prior project had different strong pointsCould not easily combine into one code base

New concepts could not easily be accommodated in oldcode bases

Easier to start overStart with a blank sheet of paperMany years of collective implementation experience

© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr

6

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialCisco booth 11

MPI From Scratch: Why?

• Started as merger of ideas fromFT-MPI (U. of Tennessee)LA-MPI (Los Alamos, Sandia)LAM/MPI (Indiana U.)PACX-MPI (HLRS, U. Stuttgart)

• Grew into much more than that

PACX-MPILAM/MPI

LA-MPIFT-MPI

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialCisco booth 12

Current Members

Academia / Research

HLRS

Indiana University

Sandia National Laboratory

Los Alamos NationalLaboratory

University of Dresden

University of Houston

University of Tennessee

Industry

Cisco

IBM

Mellanox

Myricom

QLogic

Sun

Voltaire

© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr

7

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialCisco booth 13

Other contributors

Technical U. Chemnitz

U. Jenna

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialCisco booth 14

Open MPI Project Goals

All of MPI (i.e., MPI-1 and MPI-2) Open source

Vendor-friendly license (BSD)

Prevent “forking” problemCommunity / 3rd party involvementProduction-quality research platform (targeted)Rapid deployment for new platforms

Shared development effort

© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr

8

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialCisco booth 15

Design Goals

Extend / enhance previous ideas

Message fragmentation / reassembly

Design for heterogeneous environmentsMultiple networksNode architecture (data type representation)

Automatic error detection / retransmission

Process fault tolerance

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialCisco booth 16

Design Goals

Design for a changing environmentHardware failureResource changesApplication demand (dynamic processes)

Portable efficiency on any parallel resourceSmall cluster“Big iron” hardwareGrid…

© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr

9

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialCisco booth 17

Implementation Goals

All of MPI

Low latencyE.g., minimize memory management traffic

High bandwidthE.g., stripe messages across multiple networks

Production quality

Thread safety and concurrency(MPI_THREAD_MULTIPLE)

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialCisco booth 18

Implementation Goals

Based on a componentarchitecture

Flexible run-time tuning

“Plug-ins” for differentcapabilities (e.g., differentnetworks)

Natively support commoditynetworks

Myrinet GM / MX

Infiniband OpenFabrics / VAPI

InfiniPath

Portals

Shared memory

TCP

uDAPL

© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr

10

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialCisco booth 19

Current Status

Open MPI v1.1.2 current stable releaseIncluded in OFED distributions

Open MPI v1.2b1 available for previewhttp://www.open-mpi.org/

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialCisco booth 20

The Power ofOpen Standards

© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr

11

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialCisco booth 21

Sandia Thunderbird cluster

#6 on the Top 500 list

Powered by OpenFabrics andOpen MPI

53 teraflops, 84.66% networkefficiency

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialCisco booth 22

Sandia Thunderbird cluster

#6 on the Top 500 list

Powered by OpenFabrics andOpen MPI

53 teraflops, 84.66% networkefficiency

© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr

12

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialCisco booth 23

Sandia Thunderbird cluster

#6 on the Top 500 list

Powered by OpenFabrics andOpen MPI

53 teraflops, 84.66% networkefficiency

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialCisco booth 24

Come join us!

Become part of the Open MPI teamhttp://www.open-mpi.org/community/contribute/