HPCMP Benchmarking and

Performance Analysis

Mark Cowan

USACE ERDC ITL in support of DoD HPCMP

Tuesday, April 17, 2012

What is the HPCMP?

• Initiated in 1992• Congressional mandate to modernize DoD’s HPC capabilities• Assembled from collection of HPC departments across Army, Air Force, and Navy labs and test centers

What is the HPCMP?FOCUS• Solve military and security problems using HPC hardware and software• Assess technical and management risks

• Performance• Time• Available resources• Cost• Schedule

• Supports DoD objectives through research, development, test and evaluation

Where we benchmark

Migrate to a 2-year acquisition cycle

• Why the radical change?• Entice more vendors into the competition

• Vendor feedback remove or alleviate disincentives

• Review the entirety of the TI acquisition process• Line-by-line justification of benchmarking rules document

• Address both HPC community and vendor concerns

• Comprehensive reevaluation of how we benchmark• Analyze the codes• Justify the test cases

Migrate to a 2-year acquisition cycle

• Dangers?• Time the milestones poorly on the calendar and miss out on

release of cutting-edge technology• Difficult problem

• How to schedule activities to maximize likelihood of hitting publicly-available products months in advance, while being blind to intricacies of chip fabrication schedules and unforeseen recalls?

Codes considered for TI-11/12ABAQUS COBALT ICEPICABINIT CP2K LAMMPSACES CPMD LS-DYNA

ADCIRC CTH MATLABADH ETA OOCORE

ALE3D FDTD OVERFLOWALEGRA FLAPW SHAMRC

AMR FLUENT SIERRAAVUS GAMESS STAR CCM+CFD++ GASP VASP

CFDSHIP-IOWA GAUSSIAN WRFCOAMPS HYCOM XPATCH

TI-11/12 benchmarking applications• ADCIRC – Coastal Circulation

and Storm Surge model – 100% Fortran, MPI– Uses METIS library (C)– 205K LOC

• ALEGRA – Hydrodynamic and solid dynamics plus magnetic field and thermal transport– 96% C, 4% Fortran, MPI– 978K LOC

• AVUS (Cobalt-60) – Turbulent flow CFD code– Fortran, MPI, 29K LOC

• CTH – Shock physics code– ~58% Fortran/~42% C, MPI,

900K LOC

GAMESS – Quantum chemistry code– Fortran, MPI, 330K LOC

HYCOM – Ocean circulation modeling code– Fortran, MPI, 31K LOC

ICEPIC – Particle-in-cell magnetohydrodynamics code – C, MPI, 350K LOC

LAMMPS – Molecular dynamics code– C++, MPI, 45K LOC

█ Predicted█ Benchmarked

Components of testing packages• Applications tested on representative input sets

CODE CASE

Distinguished

Core Count

Time (sec) on

DIAMOND Core Counts

ADCIRC baroclinic 1024 8959 512, 768, 1024, 1280, 1536, 1792, 2048

ADCIRC hurricane 1280 2082 512, 768, 1024, 1280, 1536, 1792, 2048

ALEGRA obliqueImp 1536 1640 1024, 1280, 1536, 1792, 2048

ALEGRA explWire 256 944 256, 384, 512, 768, 1024

AVUS waverider 1024 941 384, 512, 768, 1024, 1536

AVUS turret-td 1280 1332 768, 1024, 1280, 1536, 2048

CTH fixed-grid 1280 3399 768, 1024, 1280, 1536, 2048

CTH amr 1280 2535 768, 1024, 1280, 1536, 2048

GAMESS DFT-grad 256 4701 128, 192, 256, 384, 512

GAMESS MP2-grad 512 2536 128, 256, 512, 768, 1024

GAMESS CC-energy 1024 3658 512, 768, 1024, 1536, 2048

HYCOM lrg 1353 3020 1001, 1353, 1516, 1770, 2045

ICEPIC magnetron 384 2559 256, 384, 512, 768, 1024

ICEPIC gyrotron 2048 3639 1536, 1792, 2048, 2304, 2560

LAMMPS Au 1024 3182 128, 256, 384, 512, 1024, 1280, 1536, 2048

Some components of HPC procurement cycle

Some components of HPC procurement cycle• Acquire new versions of codes• Port codes to various machines• Acquire test cases• Develop or acquire accuracy checks• Test codes, get times to compare• Assemble package for vendors

Some components of HPC procurement cycle

• Run codes with test cases on installed DSRC machines• Optimize! How fast can we go?

Some components of HPC procurement cycle• We review vendor submittal

• Anything suspicious?• How do vendor times compare to ours? How did vendors optimize? • How risky is vendor’s proposal?• Present our results

Components of testing packages continued

• Timers measure the elapsed running times• Accuracy checks ensure validity of output files• Often requires determination of acceptable error bounds

How the test packages are used• Run all test cases on 5 different DSRC machines to

acquire times• Debug test packages• Quantify variation across/within machines• Compare times to proposed systems

Machine attributes

        Architectures Used in Study      

DSRC Name Make Model Chip SetProcessor

Speed (GHz)

Interconnect Number of Cores

Cores per Node

Operating System

ERDC Diamond SGI Altix ICEIntel

Xeon QC 2.8DDR4

InfiniBand 15360 8 SUSE Linux

MHPCC Mana DellPowerEdge

M610Intel

Xeon QC 2.8DDR

InfiniBand 9216 8 Linux

NAVY DaVinci IBM Power6IBM P6

DC 4.7DDR

Infiniband 4800 32 AIX

NAVY Einstein Cray XT5

Cray Opteron

QC 2.3 SeaStar2+ 12736 8 CNL

ERDC Garnet Cray XE6

AMD Opteron

64-bit 2.4 Cray Gemini 20224 16 CLE

RESULTS! Graphs of runtimes

Risk Assessment: Major Areas Assessed• Compliance assessment

– Ability to follow benchmark rules– Number of test case results provided– Results within accuracy criteria

• Assessment of risk in meeting proposed times in acceptance tests– Differences between benchmarked and proposed system

• Processor , interconnect, and I/O system differences– Quality of estimation procedure

• Quality of explanation and soundness of estimation procedure– Aggressiveness of final estimate

• Comparison with measured benchmark system times• Comparison with predicted times

• Assessment of likelihood of users and/or developers using proposed code modifications– Acceptability of proposed code modifications

Benchmarking website

URL:http://www.benchmarking.hpc.mil/

Benchmarking website continued

Narrative of website purpose, codes tested

Heatmap of systems best suited for applications

Benchmarking website continued

Brief description of application

Brief description of test cases

Benchmarking website continued

An example of how we madethe heatmap for allocationchoices

Benchmarking website continued

Got a question? Want to suggest an improvement?

Contact us.

Performance Team Members

• Mark Cowan – ERDC – Chair• Larry Davis – HPCMPO• Lloyd Slonaker – AFRL • Tim Sell – AFRL • Laura Brown – ERDC• Mahbubur Rashid – ERDC• Christine Cuicchi – NAVO• Matt Grismer – AFRL• Jerry Boatz – AFRL

Performance Team Advisors

• William Ward – HPCMPO• Steve Finn – DTRA • Carrie Leach – ERDC• Paul Bennett – ERDC • Tom Oppe – ERDC• Henry Newman – Instrumental• Michael Laurenzano – SDSC

• Bronis de Supinski – LLNL• Joseph Swartz – LM • Allan Snavely – SDSC• Laura Carrington – SDSC• Robert Pennington – NSF• Nick Wright – NERSC• James Ianni – ARL

Questions?

Contact me…

Mark Cowan

USACE ERDC ITL Computational Analysis Branch3909 Halls Ferry RoadBuilding 8000, Room 1255Vicksburg, MS 39180(601) 634-2665

Mark.A.Cowan@usace.army.mil

ADDENDA

AVUS: Code description

• CFD code, formerly COBALT_60• Simulates 3-D turbulent viscous flow over

irregular geometries• Grid-based, reads a large grid file• AVUS: 29K lines of Fortran 90 code• Uses ParMETIS: 12K lines of C code• Parallelism via MPI, no OpenMP• Runs on Cray XT, IBM Power, SGI Altix, Linux

clusters

CTH: Code description• CTA: CSM (Computational Structural Mechanics)• Shock Physics • Two-step, 2nd order accurate Eulerian algorithm is used

to solve the mass, momentum, and energy conservation equations • An explicit approach that does not require solving a linear

system • Has both static and adaptive mesh capabilities • Parallelism via MPI • 900K LOC, 58% FORTRAN and 42% C • Uses NetCDF, supplied with distribution

GAMESS: Code description• CTA: CCM (Computational Chemistry, Biology, and

Materials Science) • Ab Initio Quantum chemistry • Computes many energy integrals with molecular data

in form of atom positions and electron orbitals • Communication depends on platform• LAPI, Sockets, SHMEM, MPI

• Code composition: 99% FORTRAN, 1% C

HYCOM: Code description

• CTA: Climate/Weather/Ocean Modeling and Simulation (CWO)

• A primitive equation ocean general circulation model

• Communication is MPI (MPI-2 is available)• 100% FORTRAN• Version 2.2.27

HYCOM: MPI-2 details

• HYCOM may be run with MPI or MPI-2• MPI-2 is MPI with additional features such as

parallel I/O, dynamic process management and remote memory operations

• HYCOM utilizes parallel I/O feature• Parallel I/O times required starting with TI-10

ICEPIC: Code description

• CTA: Computational Electromagnetics and Acoustics (CEA)• Particle-in-cell plasma physics code• Ions and electrons move under influence of electromagnetic

fields• Particles are updated in a grid-free manner; grouped in cells

which are periodically adjusted to preserve load balance• Fields calculated on a structured, static grid and dual grid

according to Maxwell's Equations• Can simulate plasmas contained in complex geometries• Used in electromagnetic device design• ~350K lines of code, 100% C++, C

LAMMPS: Code description• CTA: CCM (Comp Chemistry, Biology, & Material

Science) • Classical molecular dynamics code that models

particles in a liquid, solid, or gaseous state • Calculates atomic velocities, positions, system energy,

and temperature • After equilibration: surface tension, radial pressure, and phase change• Post-processing: pair-correlation function and diffusion coefficients

• All actions occur within box (usually orthogonal) • Distributed-memory message-passing parallelism (MPI) • Highly-portable C++ • Libraries needed: MPI and single-processor FFT

ADCIRC: Code descriptionADCIRC Coastal Circulation and Storm Surge Model Solves time dependent, free surface circulation and transport problems in 2 and 3 dimensions. Use the finite element method in space, which permits highly flexible, unstructured grids.

Typical ADCIRC applications have included:• Modeling tides and wind driven circulation,• Analysis of hurricane storm surge and flooding,• Dredging feasibility and material disposal studies,• Larval transport studies, and• Near shore marine operations

“BASE” ALEGRA: Code descriptionALE code -- Arbitrary Lagrangian-Eulerian -- provides flexibility, accuracy and reduced numerical dissipation over pure Eulerian code; modern remeshing technology allows for robust mesh smoothing and control.

Hydrodynamic and solid dynamics

Models large distortions and strong shock propagation in multiple-materials

Finite element code; descendent of PRONTO and uses some CTH Eulerian technology

Energy deposition and explosive burn models

Geometry -- 2D/3D Cartesian, 2D cylindrical Material Models in ALEGRA: Equations of StateElastic-Plastic ModelsFracture Models Pressure and temperature during

formation of jet from shaped charge

“BASE” ALEGRA: Code description

ALEGRA_MHD: Code descriptionAll hydrodynamics/solid dynamic modules of "base" ALEGRA PLUS magnetic field and thermal transport effects

Lorentz forces, Joule heating, thermal transport and simple models for radiating excess energy

2D and 3D versions2D modeling with the magnetic flux density vector components in or out of the

plane with the corresponding current density out of or in the plane, respectively.3D uses a magnetic diffusion solution based on edge and face elements which

maintains the discrete flux divergence-free property during magnetic solve and constrained transport remap stage

Lumped element coupled circuit equations

Magnetic and thermal conduction

Advanced models for thermal and electrical conductivity

Emission model radiates excess energy when medium is optically thin while accounting for reabsorption