IBM Blue GeneIBM Blue Gene

Multiple Processor SystemsMultiple Processor SystemsSpencer MacDonaldSpencer MacDonald

Ryan WilliamsRyan Williams

AgendaAgenda

IntroductionIntroductionHardwareHardware

NodesNodesNetworkNetwork

Operating EnvironmentOperating EnvironmentCurrent and Future SystemsCurrent and Future Systems

Blue Gene/L - World’s fastest computer.

IBM Blue Gene Mission StatementIBM Blue Gene Mission Statement

“The IBM® System Blue Gene® solution is the result of an IBM “The IBM® System Blue Gene® solution is the result of an IBM supercomputing project begun over five years ago, dedicated to supercomputing project begun over five years ago, dedicated to building a new family of supercomputers optimized for building a new family of supercomputers optimized for bandwidth, scalability and the ability to handle large amounts obandwidth, scalability and the ability to handle large amounts of f data while consuming a fraction of the power and floor space data while consuming a fraction of the power and floor space required by today’s fastest systems.” required by today’s fastest systems.” ----IBMIBM

Blue gene systems currently hold the number 1 and 2 spot on the Blue gene systems currently hold the number 1 and 2 spot on the top 500 list of super computers.top 500 list of super computers.

Blue Gene Project GoalsBlue Gene Project Goals

Scientific ResearchScientific ResearchTop SupercomputerTop SupercomputerPetaflopPetaflop BarrierBarrierSelfSelf--Imposed ConstraintsImposed Constraints

Low Power ConsumptionLow Power ConsumptionLow Floor SpaceLow Floor SpaceHigh performance to area ratioHigh performance to area ratioHighly ScalableHighly Scalable

Different Size Blue Gene SolutionsDifferent Size Blue Gene Solutions

Scientific ApplicationsScientific ApplicationsG ProteinG Protein--Coupled ReceptorsCoupled Receptors

Half of pharmaceutical drugsHalf of pharmaceutical drugsTens of billions of dollars annuallyTens of billions of dollars annually

Lipid analysisLipid analysisLipids enable cell signaling and divisionLipids enable cell signaling and divisionCritical to understanding diseasesCritical to understanding diseases

OmegaOmega--3 Fatty acids and cholesterol3 Fatty acids and cholesterolStudy of how membrane proteins affect Study of how membrane proteins affect membrane environmentmembrane environment

Hardware OverviewHardware Overview

Blue Gene ASICBlue Gene ASICTwo power PC 440 CPU with Two power PC 440 CPU with Double Hummer FPUDouble Hummer FPUOperating at 700 MHzOperating at 700 MHz11.1 mm X 11.1 mm die11.1 mm X 11.1 mm dieNonNon--coherent 64K L1 Cachecoherent 64K L1 CacheCoherent 2 KB Fully associative Coherent 2 KB Fully associative cachecacheShared L3 CacheShared L3 CacheIntegrated Communication AssistIntegrated Communication Assist

IBM Blue Gene NodeIBM Blue Gene Node

IBM Blue Gene Node BoardIBM Blue Gene Node Board

16 Compute Cards per board16 Compute Cards per board00--2 I/O Cards per board2 I/O Cards per board9090--180 Gigaflops180 Gigaflops

The Communication NetworksThe Communication Networks

5 Networks5 Networks3D torus3D torusCollective NetworkCollective NetworkGlobal InterruptGlobal InterruptGigabit EthernetGigabit EthernetJTAGJTAG

Clock DistributionClock Distribution

3D torus3D torus

Each node is connected to Each node is connected to 6 nearest neighbors6 nearest neighborsHigh bandwidthHigh bandwidthNo edgesNo edgesDynamic and Deterministic Dynamic and Deterministic RoutingRoutingVirtual BufferingVirtual BufferingCutCut--through Routingthrough Routing1.4 1.4 Gb/sGb/s per linkper link1.05 GB/s per Node1.05 GB/s per Node

64 Rack Blue Gene System

Collective NetworkCollective Network

Max of 30 hops for 65,536 Node systemMax of 30 hops for 65,536 Node systemUsed for global minimum, maximum and sum.Used for global minimum, maximum and sum.For floating point sum operations, two passes For floating point sum operations, two passes are used on the networkare used on the network

One to find the maximum exponentOne to find the maximum exponentThe second to sum the mantissasThe second to sum the mantissas

Operating EnvironmentOperating EnvironmentCompute nodes run Compute Node Kernel (CNK)Compute nodes run Compute Node Kernel (CNK)I/O nodes run PowerPC LinuxI/O nodes run PowerPC Linux

Power on configuration is done via the service nodes using the cPower on configuration is done via the service nodes using the control ontrol networknetworkOne core on each I/O nodes is locked in an infinite loop.One core on each I/O nodes is locked in an infinite loop.

Compute Node EnvironmentCompute Node Environment

Compute nodes run Compute Node Kernel Compute nodes run Compute Node Kernel (CNK)(CNK)

A subset of POSIX A subset of POSIX -- One thread per CPUOne thread per CPUFixed flat address space. No PagingFixed flat address space. No PagingKernel and application share the same address Kernel and application share the same address spacespaceTorusTorus network mapped to user spacenetwork mapped to user space

CommunicationCommunication

Two supported Processor ModesTwo supported Processor ModesCoprocessor ModeCoprocessor Mode

One MPI task per node One MPI task per node –– split between processorssplit between processors

co_start, co_joinco_start, co_join

Virtual Node ModeVirtual Node ModeTwo MPITwo MPI--tasks per nodetasks per nodeProcessors communicate through message passingProcessors communicate through message passing

Communication can be done on 3 layers:Communication can be done on 3 layers:Packet layerPacket layerMessage layerMessage layerMessage Passing Interface (MPI) based on MPICH2.Message Passing Interface (MPI) based on MPICH2.

The Blue Gene SystemsThe Blue Gene Systems

Blue Gene/L Blue Gene/L –– Lawrence Livermore National LaboratoriesLawrence Livermore National Laboratories#1 Supercomputer #1 Supercomputer -- 360 Teraflops 360 Teraflops -- 64 racks, 65,536 nodes64 racks, 65,536 nodes

Blue Gene/W Blue Gene/W –– Thomas J. Watson Research CenterThomas J. Watson Research Center#2 Supercomputer #2 Supercomputer -- 114 Teraflops 114 Teraflops -- 20 racks, 20,480 nodes20 racks, 20,480 nodes

JuelichJuelich Blue Gene/LBlue Gene/L “JUBL” “JUBL” –– John von Neumann Institute for ComputingJohn von Neumann Institute for ComputingFastest in Europe Fastest in Europe -- 45.8 teraflops 45.8 teraflops -- 8 racks, 8192 nodes8 racks, 8192 nodes

Future MachinesFuture MachinesBlue Gene/P Blue Gene/P –– 1 1 petafloppetaflop targettargetBlue Gene/Q Blue Gene/Q –– 3 3 petafloppetaflop targettargetBlue Gene/C “Cyclops” Blue Gene/C “Cyclops” –– Cell architectureCell architecture

ConclusionConclusion

Top super computerTop super computerLow Power ConsumptionLow Power Consumption

3.6 percent of the power consumption of the Earth Simulator3.6 percent of the power consumption of the Earth Simulator**

Low Floor SpaceLow Floor Space1 percent of the size of the Earth Simulator1 percent of the size of the Earth Simulator**

High performance to area ratioHigh performance to area ratio

*When Blue Gene/L surpassed Earth Simulator as world’s fastest computer.