IBM Systems & Technology Group and IBM Research IBM …Blue Gene/P looks like a traditional...

© 2002 IBM Corporation

IBM Systems & Technology Group and IBM Research

Copyright 2008– IBM Corporation

IBM® System Blue Gene®/P Solution: Facilitating Extreme Scalability

IBM® Blue Gene®/P Solution Updates

Dino E. QuinteroWorldwide Technical Support Marketing Blue Gene & HPC SolutionsSenior Certified IT [email protected]

IBM® System Blue Gene®/P Solution © 2008 IBM Corporation2

IBM® System Blue Gene®/P Solution: Expanding the Limits of Breakthrough Science

Legal Notice and Disclaimer

This presentation is provided to recipient for informational purposes

only. All statements regarding IBM's future direction and intent are

subject to change or withdrawal without notice, and represent goals and

objectives only. Transactions between IBM and recipient for the sale and

purchase of hardware, software and services described herein are subject

to the terms and conditions of a purchase agreement.

IBM, the IBM logo, Blue Gene, PowerPC and eServer are trademarks of International Business Machines Corporation in the United States, other countries, or both.

Linux is a registered trademark of Linux Torvalds in the United States, other countries, or both.Other company, product and service names may be trademarks or service marks of others.

3


IBM® System Blue Gene®/P Solution © 2008 IBM Corporation

Did you know that …

� 2 racks of Blue Gene /P with 8000 processors will allow you to process

approximately 1500 square kilometers of seismic data per day, with 41 Million traces per line, which is 3-5 times faster than what's possible on typical clusters in use today. (With the near linear scalability of seismic processing, if you double the BG/P racks, you double the amount of work done.)

� The aggregate memory available in 2 racks of Blue Gene/P is enough to load and run the largest petroleum reservoir models in the world.

� Using 10 racks of BG/P, computation time for a detailed simulation of a human heart beating can be reduced from months to minutes compared to a traditional large cluster.

� Using 2 racks of BG/P one can conduct an in-silico clinical trial on 27 million simulated patients in 5 hours. A typical clinical trial may have several thousand patients and take years.

� A transonic flow simulation of a full-body aircraft structure with 300 million cells can be done in less than 1 hour on a single rack of BG/P.

� One BG/P rack can run 4000 independent pricing calculations in parallel while only consuming 35kW and occupying 1 square meter of data center power and space.

� One BG/P rack, with 2TB of memory and 4000 compute engines, can efficiently perform real-time analysis of multi-Petabyte data streams, which can be used for tracking news streams, updating trade data and monitoring events that impact financial markets.



Blue Gene/P system – expandable, flexible

Blue Gene/P Rack(s)•1,024 Quad-Core Compute Nodes per Rack > 13.7 TF/sec peak•2 GB main memory per Node > 2 TB main memory per Rack•Up to 64 I/O Nodes per Rack > 80 GB/sec peak•Scalable to 256 racks > 3.5 PF/sec

Host System•Service and Front End Nodes > System P•Software Stack > SLES10, DB2, XLF/C Compilers, HPC Cluster SW•Ethernet Switches > Force10, Myrinet•Storage System > IBM, DDN

5



Physics – Materials ScienceMolecular Dynamics

Environment and Climate Modeling Life Sciences: Sequencing

BiologicalModeling – Brain Science

Computational Fluid Dynamics

Life Sciences: In-Silico Trials, Drug Discovery

Financial ModelingStreaming Data Analysis

Geophysical Data ProcessingUpstream Petroleum

Why and for What is Blue Gene used?� Improve understanding – significantly larger scale, more complex and higher resolution

models; new science applications

� Multiscale and multiphysics – From atoms to mega-structures; coupled applications

� Shorter time to solution – Answers from months to minutes

6

IBM Blue Gene/P Solution: Expanding the Limits of Breakthrough Science


Who is Looking for a Solution like Blue Gene?

� For many engineering and scientific studies it has become far cheaper and faster to use computer simulation instead of more costly physical experiments. Competitive pressures are pushing many R&D groups to complete their research in a much shorter time frame.

� Among the fast-growing vertical segments are biosciences, geosciences (oil and gas), computer-aided engineering(CAE: CFD and EDA), defense, and university research.

“HPC Server Market Maintains Strong Growth in Third Quarter”, IDC Press Release, December 4, 2007

7



Blue Gene technology roadmap:

2004 20102007

Blue Gene/PPPC 450 @ 850MHzScalable to 3+ PF

Blue Gene/QPower Multi-CoreScalable to 10+ PF

Blue Gene/LPPC 440 @ 700MHzScalable to 360+ TF

Note: All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.

8



Both High Performance and High Throughput Computing Modes are possible on the Blue Gene

� High Performance Computing (HPC) Mode – best for Capability Computing

• Parallel, tightly coupled applications o Single Instruction, Multiple Data (SIMD) architecture

• Programming model: typically MPI

• Apps need tremendous amount of computational power over short time period

� High Throughput Computing (HTC) Mode – best for Capacity Computing

• Large number of independent taskso Multiple Instruction, Multiple Data (MIMD) architecture

• Programming model: non-MPI

• Apps need large amount of computational power over long time period

• Traditionally run on large clusters

� HTC and HPC modes co-exist on Blue Gene

• Determined when resource pool (partition) is allocated

9



Blue Gene/P Release 2 (May 2008)

V1R2 Updates in all the major systems:

� Control System� Toolchain� Kernels� Messaging libraries

� Compatibility of other IBM products with V1R2M0:

� SLES 10 � DB2 9.1� GPFS 3.2.1.2� LoadLeveler 3.4.3.1� ESSL 4.3.1.1� CSM 1.7.0.0 or later� XLC/C++ 9.0� XLF 11.1

� See “Memo To Users” for more details (KB 487217891)

10



What is High-Throughput Computing?

� High-Throughput Computing (HTC) is a new model for running applications on Blue Gene/P (BG/P)

� The HTC model of computing enables portions of the machine to be allocated for many single-node jobs

•Unlike parallel High Performance Computing (HPC) jobs, these HTC

jobs or "tasks" are all independent

•They start and end independently, and have no Message Passing

Interface (MPI) communication between the tasks.

� Blue Gene provides the flexibility to mix HPC and HTC workloads and maximize the use of the machine

� The HTC software feature is introduced in V1R2 of BG/P

11



What is High-Throughput Computing?

HTC provides a complete, integrated solution that gives users a simple, flexible mechanism for submitting single-node jobs.

Benefits of High-Throughput Computing (HTC) mode on Blue Gene/P:

� Blue Gene/P looks like a traditional "cluster" from an application’s point of view

� Enables a new class of workloads that use many single-node jobs

� Makes Blue Gene/P a viable solution for a wider spectrum of customers

� Blue Gene/P supports hybrid application environment, traditional HPC (MPI) and now HTC apps

� Differentiators (energy efficiency, processor density, reliability, capacity, administration):

� Leadership performance in a space-saving, processor dense, power-efficient package

� High reliability: designed for less then 1 failure per rack per year (7 days MTBF for 72 racks)

� Ultrascale capacity machine (“cluster buster”): run up to 4,096 HTC jobs on a single rack

� Easy administration using the powerful web based Blue Gene Navigator

Blue GeneEnvironment

Serial and Pleasantly Parallel Apps

Highly Scalable Msg Passing Apps

Paths Paths Toward aToward aGeneral Purpose General Purpose MachineMachine

*** NEW ***V1R2

HTC HPC (MPI)

12



Dual Mode� 2 cores run 1 process each

� Each process may spawn 1 thread on core not used by other process

� Each process gets ½ node memory

� MPI/OpenMP/HTC programming models

� 2 simultaneous HTC jobs per node (2048 per rack)

SMP Mode� 1 core runs 1 process

� Process may spawn threads on each of the other cores

� Process gets full node memory

� MPI/OpenMP/HTC programming models

� 1 HTC job per node (1024 per rack)

Quad Mode� Also called Virtual Node

Mode

� All 4 cores run 1 process each

� No threading

� Each process gets ¼ node memory

� MPI/HTC programming models

� 4 simultaneous HTC jobs per node (4096 per rack)

M

P

M

P

M

P

Memory address space

M

Co

re 0

P

Application

Co

re 1

Co

re 2

Co

re 3

Application

M

P

T

M

P

T

Co

re 0

Co

re 1

Co

re 2

Co

re 3


CPU2 CPU3

Application

M

P

T T T

Co

re 0

Co

re 1

Co

re 2

Co

re 3


BG/P Job Modes allow Flexible use of Compute Node Resources

� Similar to HPC jobs, users choose their job mode for HTC based on memory requirements and/or thread requirements in the application

13



Partitioning

� Two Blue Gene/P racks mixing HTC and HPC

� Each midplane (top and bottom) has 512 compute nodes

� Partitions can be dedicated to HPC or HTC based on workload mix

HPC – 1,024 nodes

HTC Virtual Node Mode – 512 nodes

HTC Dual Mode - 256 nodes

HTC SMP Mode - 256 nodes

�HTC partitions can be as small as 16 nodes, or as large as the entire machine

�Users can allocate some portion of the machine for HTC and the other portion for HPC

�Users can combine multiple HTC partitions together to form “pools” of nodes dedicated to

running HTC jobs

Partitioning Example

14



Summary of High-Throughput Computing

� High-Throughput Computing (HTC) is an important new software feature of

BG/P available in V1R2� HTC enables BG/P to continue on the path toward general purpose usage� HPC and HTC workloads can co-exist, by partitioning the machine� Usage of HTC is available out-of-the-box

• HTC jobs can also be launched via a scheduler

� HTC features are integrated throughout the BG/P control system

• Navigator

• Console commands

• Submit command

• Scheduler APIs

� HTC scales very well for short-duration tasks� For more information, consult the Application Development Redbook, SG24-

7287, System Administration Redbook, SG24-7417 or the IBM Blue Gene

Knowledgebase.

15



Release 2 Control System (cont.)

� Navigator quick lookup� 3D Block visualizer in navigator Block Details view� Many new MMCS commands and features

• job_trace, job_status for debugging control problems

• tailjob command shows stdout for a job launched with submitjob

• genBPBlock now supports passthrough and split cables

• A command line interface to the navigator block builder

• locate neighbors verbose

• Duplicate mac address detection

� Two new boot timeout parameters:• boot-timeout-node-multiplier: seconds to wait per IO node for a boot to complete (default

120 sec per ION)

• boot-timeout-and-free: max time to wait for all IONs in a boot to complete (default 20 min)

16



Release 2 Control System (cont.)

� RAS events may now have control actions� The control action may mark hardware in error� Node card error events:

• Failure to configure global interrupts

• Unhandled interrupts or machine checks

• Kernel load and init CRC errors

� Cable failure events:• Link chip byte alignment failed

• Link chip parity alignment failed

� Service actions are required to re-enable the hardware

� See Memo To User for complete RAS list

17



Release 2 Toolchain

� Upgrade gcc 4.1.1 to 4.1.2 (same as SLES10 SP1)

� Using HTC mode to run compiler test suites

� PTF2 for XL C/C++ V9.0

� PTF2 for XL Fortran V11.1

18



Release 2 Kernels

� HTC Mode support• 1, 2 or 4 jobs per node

• Unique stdout/stderr

• Per job user and per job signaling

� Checkpoint/restart library memory image compression

� Debug watchpoint support

� Totalview debugger enablement

� Stack guard protection allowed while debugging

19



Release 2 Kernels (cont.)

� More shared memory segments allowed (was 2, now 8)

� Process memory windows• New syscalls to map memory windows between processes

• Allows access to what is normally non-shared memory

• Can be used by comm library to transfer data directly from sender’s buffers

� Ethernet hardware flow control enabled

� BG_COREDUMPONERROR=1 allows forced core files from nodes that try to exit with non-zero exit status

� Source for CNK and CIOD available under CPL license

20



Release 2 Messaging Libraries

� Change MPI_Aint to 64-bits• Corrects problems in HDF5 and MPI-IO implementations for large files

• Note that “address sized integers” are otherwise too small for file extents on a 32-bit system such as Blue Gene

� Numerous performance enhancements!

� Support alternative DMA fifo modes• provides more fifos to apps using direct SPI access to hardware

� Add the DCMF_Put() API and DMA protocol• for ARMCI and MPI 1-sided operations

� DMA Multisend performance optimizations and enhancements• new multisends (remote get flavor for NAMD) and allreduce as a reduce-scatter followed

by allgather



Blue Gene/P Ecosystem

� Hardware

– DataDirect Networks (DDN), IBM System DCS9550; storage

– Myricom, Force10; switches

� 3rd party ISVs

– Visual Numerics (VNI) developers of IMSL

– TotalView Technologies Inc. developers of TotalView Debugger

– Allinea developers of allinea ddt and allinea opt

– Tsunami Development LLC developers of the Tsunami Imaging Suite

– Condor developers of software tools (open source scheduler)

22



Where in the world is Blue Gene?

NARSS

DIAS

23



Q2 Events� Malaysia Petroleum Forum – May 5-6

� STG Sales Conference, LAX – May 5-8

� STG Technical Conference, LAX - May 12-15

� SP-XXL – SciCOMP, Hudson Valley – May 19-23

� STG North Region Customer Conference, Venezuela – May 20-23

� Blue Gene, DCV and HPC Enablement, Brazil – May 29

� Korea HPC Day – June 5

� HPCS, Quebec City, Canada – June 8-10

� European Association of Geoscientists and Engineers (EAGE), June 9-12

� STG South Region Customer Conference, Argentina – June 10-13

� ANL INCITE/BG Consortium BG/P porting and tuning workshop - May 7-8

� International Supercomputing Conference (ISC'08) – June 17-20

� UK Blue Gene Lead Generation Workshops June 24-28

� Blue Gene Enablement, Madrid/Barcelona – May 24-26

� Computational Mechanics Conference, Italy – June 30 to July 4

24



Blue Gene/P Collateral in Progress

� HTC validation with DOCK6 program Redpaper - Completed

� System Level Acceleration Redpaper - Completed

� Blue Gene/P Redbooks updates – April through September

� Julich/Force10/IBM case study and video – June

� Petroleum analyst case study update – June

� TCO analyst case study, green, and competitive update – June



More Information� Signed up for the Blue Gene Newsletter

– http://www.ibm.com/servers/deepcomputing/bluegene.html

� Top500 Supercomputer Sites– http://www.top500.org

� IBM Redbooks for Blue Gene (ibm.com/redbooks)– Application Development Guide

– System Administration Guide

– Performance Tools

� Sign up for the Blue Gene Consortium membership– http://www-fp.mcs.anl.gov/bgconsortium/

� Open Source Communities– http://www.alphaworks.ibm.com

� Argonne National Labs Blue Gene wiki– http://wiki.bgl.mcs.anl.gov/wiki/ANLBGLWiki_Home



Worldwide Blue Gene/P Solution Contacts

Masahiko ShinjohMasahiko Shinjoh/Japan/IBM@IBMJP

Japan

Philippe Bricard Philippe Bricard/France/IBM@IBMFR

SW

Pat CareyPatrick Carey/Minneapolis/IBM

WW

Mladen KarcicMladen Karcic/Bethesda/IBM@IBMUS

U.S. Federal

Sinisa NikolicSinisa Nikolic/Houston/IBM@IBMUS

AP

Chris Cooper Christopher Cooper1/UK/IBM@IBMGB

NE

Doug McGuire Douglas McGuire/Lexington/IBM@IBMUS

AG

Sales LeaderGeo

Masahiko ShinjohMasahiko Shinjoh/Japan/IBM@IBMJP

Japan

Philippe Bricard Philippe Bricard/France/IBM@IBMFR

SW

Pat CareyPatrick Carey/Minneapolis/IBM

WW

Mladen KarcicMladen Karcic/Bethesda/IBM@IBMUS

U.S. Federal

Sinisa NikolicSinisa Nikolic/Houston/IBM@IBMUS

AP

Chris Cooper Christopher Cooper1/UK/IBM@IBMGB

NE

Doug McGuire Douglas McGuire/Lexington/IBM@IBMUS

AG

Sales LeaderGeo

© 2002 IBM Corporation

IBM Systems & Technology Group and IBM Research

Copyright 2008– IBM Corporation

IBM® System Blue Gene®/P Solution: Facilitating Extreme Scalability

Questions and Answers

The New IBM® Blue Gene®/P SolutionDino E. Quintero

Worldwide Technical Support Marketing Blue Gene & HPC Solutions

Senior Certified IT Specialist

[email protected]

© 2008 IBM Corporation28

TrademarksThe following are trademarks of the International Business Machines Corporation in the United States, other countries, or both.

The following are trademarks or registered trademarks of other companies.

* All other products may be trademarks or registered trademarks of their respective companies.

Notes:

Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here.

IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.

All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions.

This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult your local IBM business contact for information on the product or services available in your area.

All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.

Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.

Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography.

Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.

Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license therefrom. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.

UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office.IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency, which is now part of the Office of Government Commerce.

For a complete list of IBM Trademarks, see www.ibm.com/legal/copytrade.shtml:

*, AS/400®, e business(logo)®, DBE, ESCO, eServer, FICON, IBM®, IBM (logo)®, iSeries®, MVS, OS/390®, pSeries®, RS/6000®, S/30, VM/ESA®, VSE/ESA, WebSphere®, xSeries®, z/OS®, zSeries®, z/VM®, System i, System i5, System p, System p5, System x, System z, System z9®, BladeCenter®

Not all common law marks used by IBM are listed on this page. Failure of a mark to appear does not mean that IBM does not use the mark nor does it mean that the product is not

actively marketed or is not significant within its relevant market.

Those trademarks followed by ® are registered trademarks of IBM in the United States; all others are trademarks or common law marks of IBM in the United States.

IBM Systems & Technology Group and IBM Research IBM …Blue Gene/P looks like a traditional...

Documents

Transcript of IBM Systems & Technology Group and IBM Research IBM …Blue Gene/P looks like a traditional...