Interconnect Your Future With Mellanox

High-Performance Computing

March 2014

Interconnect Your Future With Mellanox

© 2014 Mellanox Technologies 2

Mellanox Performance Advantage (Source TopCrunch)

1000

1100

1200

1300

1400

1500

CRAY XC30/Aries2000 Cores

CRAY XC30/Aries4000 Cores

FDR InfiniBand (SGI)2000 Cores

LS-DYNA Applications, Car2Car Benchmark (Seconds)

All platforms use same CPUs of Intel® Xeon® E5-2690 v2 @3.00GHz, Cray platform is connected with Cray Aries interconnect, SGI platform is connected with Mellanox FDR InfiniBand

LS-DYNA is an advanced multiphysics

simulation software (CAE), developed by LSTC

Used in automotive, aerospace, military,

manufacturing, and bioengineering industries

Higher Performance

with Half of the System Size!

2014 Results

InfiniBand Delivers Highest System Performance, Efficiency and Scalability


Mellanox Performance Advantage (Source HPC Advisory Council)

More than

2X Performance!

InfiniBand Delivers Highest System Performance, Efficiency and Scalability

HOOMD-blue is an highly optimized object-oriented

many-particle dynamics applications that performs

general purpose particle dynamics simulations

Developed by the University of Michigan


InfiniBand Leadership in TOP500 Petascale-Capable Systems

Mellanox InfiniBand is the interconnect of choice for Petascale computing

• Accelerates 48% of the sustained Petaflop systems (19 systems out of 40)


Mellanox InfiniBand Connected Petascale Systems

Connecting Half of the World’s Petascale Systems Mellanox Connected Petascale System Examples


InfiniBand’s Unsurpassed System Efficiency

TOP500 systems listed according to their efficiency

InfiniBand is the key element responsible for the highest system efficiency

Mellanox delivers efficiencies of more than 97% with InfiniBand

Average Efficiency

• InfiniBand: 86%

• Cray: 80%

• 10GbE: 65%

• GigE: 44%


Mellanox FDR InfiniBand is the fastest interconnect solution on the TOP500

• More than 12GB/s throughput, less than 0.7usec latency

• Being used in 80 systems on the TOP500 list – 1.8X increase from the Nov’12 list

• Connects the fastest InfiniBand-based supercomputers – TACC (#7), LRZ (#10)

• Enables the two most efficient systems in the TOP200

Mellanox InfiniBand is the fastest interconnect technology on the list

• Enables the highest system utilization on the TOP500 – more than 97% system efficiency

• Enables the top seven highest utilized systems on the TOP500 list

Mellanox InfiniBand is the only Petascale-proven, standard interconnect solution

• Connects 19 out of the 40 Petaflop capable systems on the list

• Connects 4X the number of Cray based systems in the Top100, 6.5X in TOP500

Mellanox’s end-to-end scalable solutions accelerate GPU-based systems

• GPUDirect RDMA technology enables faster communications and higher performance

Mellanox in the TOP500 Supercomputing List (Nov’13)


20K InfiniBand nodes

Mellanox end-to-end FDR and QDR InfiniBand

Supports variety of scientific and engineering projects • Coupled atmosphere-ocean models

• Future space vehicle design

• Large-scale dark matter halos and galaxy evolution

System Example: NASA Ames Research Center Pleiades

Asian Monsoon Water Cycle

High-Resolution Climate Simulations


Leading Supplier of End-to-End Interconnect Solutions

MXM Mellanox Messaging

Acceleration

FCA Fabric Collectives

Acceleration

Management

UFM Unified Fabric Management

Storage and Data

VSA Storage Accelerator

(iSCSI)

UDA Unstructured Data

Accelerator

Comprehensive End-to-End Software Accelerators and Managment

Host/Fabric Software ICs Switches/Gateways Adapter Cards Cables/Modules

Comprehensive End-to-End InfiniBand and Ethernet Portfolio

Metro / WAN


Converged Interconnect Solutions to Deliver Highest ROI for all Applications

Accelerating Half of the World’s Petascale Systems Mellanox Connected Petascale System Examples

Businesses Success Depends on Mellanox

InfiniBand Enables Lowest Application Cost in the Cloud

(Examples) Dominant in Storage Interconnects


Mellanox Solutions


Virtual Protocol Interconnect (VPI) Technology

64 ports 10GbE

36 ports 40/56GbE

48 10GbE + 12 40/56GbE

36 ports IB up to 56Gb/s

8 VPI subnets

Switch OS Layer

Mezzanine Card

VPI Adapter VPI Switch

Ethernet: 10/40/56 Gb/s

InfiniBand:10/20/40/56 Gb/s

Unified Fabric Manager

Networking Storage Clustering Management

Applications

Acceleration Engines

LOM Adapter Card

3.0

From data center to

campus and metro

connectivity

Standard Protocols of InfiniBand and Ethernet on the Same Wire!

http://www.redhat.com/


OpenSHMEM / PGAS

Mellanox ScalableHPC Communication Library to Accelerate Applications

MXM • Reliable Messaging

• Hybrid Transport Mechanism

• Efficient Memory Registration

• Receive Side Tag Matching

FCA • Topology Aware Collective Optimization

• Hardware Multicast

• Separate Virtual Fabric for Collectives

• CORE-Direct Hardware Offload

MPI Berkeley UPC

0.0

20.0

40.0

60.0

80.0

100.0

0 500 1000 1500 2000 2500

La

ten

cy (

us

)

Processes (PPN=8)

Barrier Collective Latency

Without FCA With FCA

0

500

1000

1500

2000

2500

3000

0 500 1000 1500 2000 2500

La

ten

cy (

us

)

Processes (PPN=8)

Reduce Collective Latency

Without FCA With FCA


Mellanox Connect-IB The World’s Fastest Adapter

The 7th generation of Mellanox interconnect adapters

World’s first 100Gb/s interconnect adapter (dual-port FDR 56Gb/s InfiniBand)

Delivers 137 million messages per second – 4X higher than competition

World leading scalable transport – no dependency on system size


Smart Offloads for MPI/SHMEM/PGAS/UPC Collective Operations

CORE-Direct Technology

US Department of Energy (DOE) funded project – ORNL and Mellanox

Adapter-based hardware offloading for collectives operations

Includes floating-point capability on the adapter for data reductions

CORE-Direct API is exposed through the Mellanox drivers

Ideal System noise CORE-Direct (Offload) CORE-Direct - Asynchronous


GPUDirect RDMA for Highest GPU Performance

GPUDirect RDMA

CPU

GPU Chip

set

GPU Memory

InfiniBand

System

Memory

1

CPU

GPU Chip

set

GPU Memory

InfiniBand

System

Memory

1

67% Lower Latency 5X Increase in Throughput

Source: Prof. DK Panda

http://www.google.com/url?sa=i&source=images&cd=&cad=rja&docid=l4bgqVY3Z-5H9M&tbnid=VeKX0Kar856WBM:&ved=0CAgQjRwwADgF&url=https://twitter.com/OSUCATS&ei=5a65UcmnGaqG0AWJ1YGwBg&psig=AFQjCNFmgs1A9YUXxMlqqJPS30QSMEHV0Q&ust=1371209829447892


Remote GPU Access through rCUDA

GPU servers GPU as a Service

rCUDA daemon

Network Interface CUDA

Driver + runtime Network Interface

rCUDA library

Application

Client Side Server Side

Application

CUDA

Driver + runtime

CUDA Application

rCUDA provides remote access from

every node to any GPU in the system

CPU VGPU

CPU VGPU

CPU VGPU

GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU


Example:

Example:

4 MetroX TX6100 systems over 6 km

Example:

“A common problem is the time cost of moving data

between datacenters, which can slow computations

and delay results. Mellanox's MetroX lets us unify

systems across campus, and maintain the high-speed

access our researchers need, regardless of the

physical location of their work.”

Mike Shuey, Purdue University

2 MetroX TX6100 systems over 8 km

Campus and Metro RDMA Long Reach Solutions

4 MetroX TX6100 systems

• Connect IB over 2-4km

• Replace Obsidian SDR

http://www.google.com/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&docid=uDUa3XhmJt39UM&tbnid=a3RPp2yvuwHyfM:&ved=0CAUQjRw&url=http://www.myinternationalscholarships.com/gsia-offers-four-hedley-bull-scholarships/&ei=iz7ZUdz7F4PB0gGI4YCoBw&bvm=bv.48705608,d.dmg&psig=AFQjCNEzMFZn9sGeEV0JYBMD0-Aigkyo6g&ust=1373278210555298


Variety of Clustering Topologies

Typically enables best performance, lowest latency

Non-Blocking Network

Alleviates bandwidth bottleneck closer to the root.

Most common topology in many supercomputers

Blocking network, good for applications with locality

Support for dedicate sub-networks

Simple expansion for future growth

Not limited to storage connection only at cube edges

Concept of connecting “groups” together in a full-graph

Flexible definition of intra-group interconnection

Mesh / 3D Torus CLOS (Fat Tree)

Hypercube DragonFly+

Supported by SGI


Connect-IB delivers superior performance: 100Gb/s, 0.7usec latency, 137 million messages/sec

ScalableHPC software library provides leading performance for MPI, OpenSHMEM/PGAS and UPC

Superiors applications offloads: RDMA, Collectives, scalable transport (Dynamically Connected)

Flexible topologies: Fat Tree, mesh, 3D Torus, Dragonfly+

Standard based solution, Open source support, large eco-system, one solution for all applications

Converged I/O – compute, storage, management on single fabric

Long term roadmap

The Mellanox Advatage


Technology Roadmap – One-Generation Lead over the Competition

2000 2020 2010 2005

20Gbs 40Gbs 56Gbs 100Gbs

“Roadrunner” Mellanox Connected

1st 3rd

TOP500 2003 Virginia Tech (Apple)

2015

200Gbs

Mega Supercomputers

Terascale Petascale Exascale

Mellanox

http://www.google.com/url?sa=i&source=images&cd=&cad=rja&docid=WZQwW7JWurxgJM&tbnid=OvKa9JUp-zTPGM:&ved=0CAgQjRwwADixBg&url=http://www.gamesthirst.com/2013/06/24/microsoft-invests-678m-in-iowa-data-centers-to-support-growing-services-business-xbox-live/&ei=GyZRUpydE6PF0QWK94DgCw&psig=AFQjCNHRYWwRn1vFGaB9Mweu30OcDb1w2A&ust=1381136283359407


The Only Provider of End-to-End 40/56Gb/s Solutions

From Data Center to Metro and WAN

X86, ARM and Power based Compute and Storage Platforms

The Interconnect Provider For 10Gb/s and Beyond

Host/Fabric Software ICs Switches/Gateways Adapter Cards Cables/Modules

Comprehensive End-to-End InfiniBand and Ethernet Portfolio

Metro / WAN

For more information: [email protected]

Interconnect Your Future With Mellanox

Technology

Transcript of Interconnect Your Future With Mellanox