Interactively Visualizing Science at Scale

Kelly Gaither Director of Visualization/Senior Research

Scientist

Texas Advanced Computing Center

November 13, 2012

Issues and Concerns – Maximizing Scientific Impact – Managing Data at Scale – Providing Resources at Scale – Ensuring Broad Accessibility/Developing

Ubiquitous Tools

Maximizing Scientific Impact

Image: Greg P. Johnson, Romy Schneider, TACC Image: Adam Kubach, Karla Vega, Clint Dawson

Image: Karla Vega, Shaolie Hossain, Thomas J.R., Hughes Greg Abram, Carsten Burstedde, Georg Stadler, Lucas C. Wilcox, James R. Martin, Tobin Isaac, Tan Bui-Thanh,and Omar Ghattas

Not Just Simulation Any More… Vastly more powerful

instruments and computers have led to an explosion of new data.

Modern science and engineering therefore is about managing and analyzing this data as well as modeling and simulation.

Visualization of Large Scale Turbulent Flow Kelly Gaither, Hank Childs, Greg Johnson, Karl Schulz, Cyrus Harrison,

Diego Donzis, Texas A&M; P.K. Yeung, Georgia Tech

• Remote interactive visualization of 17 time-steps (34 TB) of the largest turbulent flow simulation computed to date (40963).

• First time this had been visualized interactively at this scale.

• Equal parts data mining and remote interactive visualization – goal was to characterize flow behavior over time.

Gaither, K., Childs, H., Schulz, K., Harrison, C., Barth, W., Donzis, D., and Yeung, P.K., “Using Visualization and Data Analysis to Understand Critical Structures in Massive Time Varying Turbulent Flow Simulations,” IEEE Computer Graphics and Applications, 32(4), Jul/Aug 2012.

Stellar Magnetism Greg Foss, TACC; Ben Brown, University of Wisconsin, Madison

• A Sun-like star undergoes magnetic cyclic reversal shown by field lines.

• Shifts in positive and negative polarity demonstrate large-scale polarity changes in the star.

• Wreath-like areas in the magnetic field may be the source of Sun spots.

• Terabytes of data to mine through and visualize.

TACC Visualization Pipelines • Post-processing

– User generates a data set either through simulation or through measurement and saves the data set for later post-processing analysis and visualization.

– Reasonable model when interactive query is of primary interest and the data set size is manageable.

• In Situ – Data sets are growing at a staggering rate and the

traditional model of post-processing visualization becomes too costly to manage, particularly in those instances in which time to insight is of value.

– Requires instrumentation of the simulation code and hooks in the visualization software.

– We have facilitated this using VisIt and ParaView.

Managing Data at Scale

Large-Scale Visualization

Resource

HPC System

Data Archive

Display

Remote Site Wide-Area Network Local Site

Pixels

Mouse

Longhorn First NSF XD Visualization Resource

• 256 Dell Dual Socket, Quad Core Intel Nehalem Nodes

– 240 with 48 GB shared memory/node (6 GB/core) – 16 with 144 GB shared memory/node (18 GB/core) – 73 GB Local Disk – 2 Nvidia GPUs/Node (FX 5800 – 4GB RAM)

• ~13.5 TB aggregate memory • QDR InfiniBand Interconnect • Jobs launched through SGE

• ~6GB/s to scratch filesystem • ~6GB/s to Ranger filesystem

256 Nodes, 2048 Cores, 512 GPUs, 14.5 TB Memory

Kelly Gaither (PI), Valerio Pascucci, Chuck Hansen, David Ebert, John Clyne (Co-PI), Hank Childs

Supporting Visualization on Stampede

• Leverage 128 Kepler GPUs for interactive remote visualization using VNC and VirtualGL.

• Working with Intel graphics group to facilitate remote interactive visualization: – Porting OpenGL to MIC – Real time raytracing

Visualization Usage Modalities: • Remote/Interactive Visualization

– Highest priority jobs – Remote/Interactive capabilities facilitated through VNC – Run on 3 hour queue limit boundary

• GPGPU jobs – Run on a lower priority than the remote/interactive jobs – Run on a 12 hour queue limit boundary

• CPU jobs with higher memory requirements – Run on lowest priority when neither remote/interactive nor GPGPU

jobs are waiting in the queue – Run on a 12 hour queue limit boundary

Queue Structure

Example: qsub -q normal -P vis

Visualization Portal portal.longhorn.tacc.utexas.edu

• Developed to provide easy access to remote visualization systems and abstract away complexities involved with command line access

• Leverages XSEDE user portal codebase and employs a fraction of XUP developers to ensure continuity

• Used for all in-person remote visualization training

Visualization Portal portal.longhorn.tacc.utexas.edu

>5000 jobs submitted through the portal

Visualization Portal portal.longhorn.tacc.utexas.edu

• Specify type of session

• Specify resolution of vnc session

• Specify number of nodes needed and the wayness of the nodes

• Provides graphic of machine load

Visualization Software on All TACC Systems • Programming APIs: OpenGL, vtk (Not natively parallel)

– OpenGL – low level primitives, useful for programming at a relatively low level with respect to graphics

– VTK (Visualization Toolkit) – open source software system for 3D computer graphics, image processing, and visualization

– IDL • Visualization Turnkey Systems

– VisIt – free open source parallel visualization and graphical analysis tool

– ParaView – free open source general purpose parallel visualization system

– VAPOR – free flow visualization package developed out of NCAR – EnSight – commercial turnkey parallel visualization package

targeted at CFD visualization – Amira – commercial turnkey visualization package targeted at

visualizing scanned medical data (CAT scan, MRI, etc..)

Thoughts Towards Exascale:

• Data will get larger and more unwieldy – we will stop moving it around

• High performance computing environments will become high performance science environments that provide computing and analytics

• Rendering will continue to get less and less expensive.

• We will see a real blend in high performance environments of physical modeling and computer graphics.

Thank You

Kelly Gaither kelly@tacc.utexas.edu

Thoughts Towards Exascale:

• Data will get larger and more unwieldy – we will stop moving it around

• High performance computing environments will become high performance science environments that provide computing and analytics

• Rendering will continue to get less and less expensive.

• We will see a real blend in high performance environments of physical modeling and computer graphics.