John Clyne & Alan Norton Scientific Computing Division National Center for Atmospheric Research

Ultra-Scale Visualization WorkshopNovember 13, [email protected]

Supercomputing • Communications • Data

NCAR Scientific Computing Division

VAPORVisualization and Analysis Platform for Ocean, atmosphere, and solar Research

SC06 Ultra-Scale Visualization Workshop

John Clyne & Alan Norton

Scientific Computing Division

National Center for Atmospheric Research

Boulder, CO USA

This work is funded in part through a U.S. National Science Foundation, Information Technology Research program grant




[Numerical] models that can currently be run on typical supercomputing platforms produce data in amounts that make storage expensive, movement cumbersome, visualization difficult, and detailed analysis impossible. The result is a significantly reduced scientific return from the nation's largest computational efforts.

Mark RastUniversity of Colorado, LASP

Problem

• Numerical simulations in the earth sciences have reached such extraordinary sizes that researchers can no longer effectively extract insight from their simulation outputs.

• Result: loss of scientific productivity!!!




Dichotomy of simulation and analysis needs and resources in today’s HPC environments

Simulation Analysis

Large systems (O(1000) processors)

Small systems (O(10) processors)

Batch computing model Interactive computing model

Modest on-line storage requirements

Large on-line storage requirements

CPU/interconnect bound IO bound

Highly tuned, custom, parallel codes

COTS serial software applications

Historical focus of centers Emerging focus of centers




A sampling of various technology performance curves

Performance gains from 1980 to present

1

10

100

1000

10000

100000

1980 1984 1988 1992 1996 2000 2004

Improvement

Disk Drive Internal DataRate

Disk Drive InterfaceData Rate

Ethernet NetworkBandwidth

Intel MicroprocessorClock Speed

Drive Capacity

• Not all technologies advance at same rate• Impact of parallelization not shown




Communication limits for volume rendering assuming theoretical peak performance

2563 5123 10243 20483

GFX Memory

76 GB/sec(GeForce 7900 GX2)

4551 568 71 9

CPU Memory

10 GB/sec(AMD Opteron 1000 series)

599 75 9 1

PCI Express 16x

4 GB/sec

240 30 4 0.5

SATA 3.0

0.3 GB/sec

18 2 0.3 0.04

• Table shows limits expressed as frames per second imposed by communication alone

• Assumes only 8-bit data quantities




Visualization and Analysis Platform for Ocean, atmosphere, and solar Research (VAPOR)

Key components1. Domain specific application focus: simulated earth sciences fluid flow2. Coupled Visualization and quantitative data interrogation and

manipulation capabilities 3. Multiresolution enabled terascale data exploration on the desktop

Combination of visualization with multiresolution data representation that provide sufficient data reduction to enable interactive work on terascale data from a desktop

Visual data browsing

Datamanipulation

Quantitativeanalysis

Refine

Coarsen




Fluid flow in the geosciences

• E.g. Numerically simulated turbulence– Cartesian grids (usually)

• 5123 to 10243

• Up to 40963 “hero” calculations

– 5 to 8 variables• Temperature & Pressure

• Velocity field components

• Magnetic field components (MHD calculations)

– Hundreds of time steps saved• Terabytes of data per experiment

– Numerical “experiments”• Substantial analysis requirements

Yannick Ponty, CNRS 2006




Key Component (1) : Domain specific supportOnly limited support for:

– Grid & data types• Cartesian grids, stretched and uniform sampling• AMR grids• Scalar and vector quantities

– Visualization algorithms• Volume rendering, flow visualization, cutting

planes/probe

– Misc.• Publication quality graphics• Filters• File formats (one!)

• Extensive support for:– Time varying data

• Uniform as well as non-uniform sampling• Missing time steps

– Quantitative investigation• Mathematical operators and data manipulators

– Science driven specialized features

Keep it simple!Keep it focused!

Make it scientist friendly!




Interactive exploration of time varying data

• Reduce bandwidth requirements– Regions of interest– Multiresolution– Caching

2563 5123 10243 20483

GFX Memory

76 GB/sec

(GeForce 7900 GX2)

4551 568 71 9

CPU Memory

10 GB/sec

(AMD Opteron 1000 series)

599 75 9 1

PCI Express 16x

4 GB/sec

240 30 4 0.5

SATA 3.0

0.3 GB/sec

18 2 0.3 0.04




Future???VAPORInteractive visual browsing

IDLData manipulation &

analysis

VAPOR Data Collection

Disk Array

Multi-resolution access and rapid sub-region extraction

Key Component (2) : Coupled visualization, quantitative analysis and manipulation capabilities

• IDL - array based 4GL for scientific data processing– Thousands of mathematical functions

– Basic 2D plotting

– Array manipulation




Key component (3) : Multiresolution data access

• Wavelet transformed data– Two parameter linear function

decomposition

– Hierarchical data representation

– Invertible and lossless

– Numerically efficient (O(n))

• forward and inverse transform

– No additional storage cost

• Enable speed/quality tradeoffs

504x504x2048

Full

252x252x1024

1/8

126x126x512

1/64

63x63x256

1/512

€

f t( ) = a j ,k

j

∑k

∑ ψ j ,k t( )




Visual comparison of a 5123 compressible convection simulation

1283

coarsened

M. Rast, 20025123

native




Performance of forward and inverse Haar wavelet transform

Inverse Data Transform

0

10

20

30

40

50

60

70

80

90

100

128 3̂ 256 3̂ 512 3̂ 1024 3̂

Resolution

Time (seconds)

Read

Transform

Forward Data Transformation

0

10

20

30

40

50

60

70

80

90

100

128 3̂ 256 3̂ 512 3̂ 1024 3̂

Resolution

Time (seconds)

Write

Transform

System

• Linux RHEL 3.0

• 2 x Intel 3.4 GHz Xeon EMT64

• 8 GBs RAM

• 1Gb/sec Fibre Channel storage

Data

• Scalar

• Single precision

Gains in microprocessor technology enable transforms at very low cost




VAPOR Demo




Summary

• VAPOR is a domain-specific platform for analysis, not a general purpose visualization tool

• Target users: fluid flow researchers in earth sciences– Limited value for medical, oil & gas, aerospace, etc.

• Desktop data exploration of terabyte data possible– Visualization enables rapid ROI identification

– Multiresolution enables speed/quality tradeoffs




Acknowledgements

• Steering Committee– Nic Brummell - CU

– Yuhong Fan - NCAR, HAO

– Aimé Fournier – NCAR, IMAGe

– Pablo Mininni, NCAR, IMAGe

– Aake Nordlund, University of Copenhagen

– Helene Politano - Observatoire de la Cote d'Azur

– Yannick Ponty - Observatoire de la Cote d'Azur

– Annick Pouquet - NCAR, ESSL

– Mark Rast - CU

– Duane Rosenberg - NCAR, IMAGe

– Matthias Rempel - NCAR, HAO

– Geoff Vasil, CU

• Developers– Alan Norton – NCAR, SCD

– John Clyne – NCAR, SCD

– Kenny Gruchalla - CU

• Research Collaborators– Kwan-Liu Ma, U.C. Davis

– Hiroshi Akiba, U.C. Davis

– Han-Wei Shen, Ohio State

– Liya Li, Ohio State

• Systems Support– Joey Mendoza, NCAR, SCD




Questions???

www.vapor.ucar.edu




Inverse Haar transform with 1/8th volume subregion extraction

System

• Linux RHEL 3.0

• 2 x Intel 3.4 GHz Xeon EM64

• 8 GBs RAM

• 1Gb/sec Fibre Channel storage

Data

• Scalar

• Single precision

Inverse Data Transform with Subregion Extraction

0

2

4

6

8

10

12

14

128 3̂ 256 3̂ 512 3̂ 1024 3̂

Resolution

Time (seconds)

Read

Transform

Data blocking permits rapid subregion extraction




The Lifting Method of wavelet construction in the spatial domain[Sweldens, 95]

{ } Z∈<≤≡ kjk jkjj ,for 20|,λλ

kkjkj ∀= +− 12,,1 λγ

( ) kkjkjkj P ∀−= −+− ,112,,1 λλγ

kjkj 2,,1 λλ =−

( ) kkjkjkj U ∀+= −−− ,1,1,1 γλλ

kj ,λ SplitPredict

Update

kj ,1−γ

kj ,1−λkj ,2−γ

kj ,2−λk,0γ

k,0λk,1λTransform

1

Transform 2

Transform j

1) Split:

2) Predict:

3) Update:

Split signal into even (λ) and odd (γ) coefficients. λ will contain low frequency information, γ will contain high frequency information.

Local correlation permits prediction of odd samples by even using a prediction operator, P. Capture difference between prediction and actual coefficient value.

Update λ coefficients to preserve a property (e.g. mean) of original signal.

A signal λj consisting of 2j samples




Example: Lifting Method with the Haar Wavelet

1 7 3 1 6 0 9 5 ( )kkk P ,212,3,2 λλγ −= +

4 2 3 7

3 5

λ3,k

λ2,k

λ1,k

λ0,k

6 -2 -6 -4γ2,k

γ0,k

γ1,k

Haaroperators xxU

xxP

2

1)(

)(

=

=

( )kkk U ,2,2,2 γλλ +=

-2 4

4 2




NCAR Historical Estimated Sustained GFLOPS (Batch Production Systems)

0

100

200

300

400

500

600

700

800

900

1000

Jan-97 Jan-98 Jan-99 Jan-00 Jan-01 Jan-02 Jan-03 Jan-04 Jan-05 Jan-06

IBM p5-575/HPS(bluevista)

IBM Opteron/Linux(lightning)

IBM POWER4/Federation(thunder)

IBM POWER4/Colony(bluesky)

IBM POWER4 (bluedawn)

SGI Origin3800/128

IBM POWER3(blackforest)

IBM POWER3 (babyblue)

Compaq ES40/32(prospect)

SGI Origin2000/128 (ute)

HP SPP-2000/64 (sioux)

CRI Cray C90/16 (antero)

CRI Cray J90 series

ARCS Phase 4

Cray C90/16

HP SPP2000

SGI Origin2000

blackforest (WH-1)

SGI Origin3800

lightning

bluesky

blackforest

ARCS Phase 3

ARCS Phase 2

ARCS Phase 1

Linux

blackforest (WH-2/NH-2)

bluevista




NCAR Historical Estimated Sustained GFLOPS (Interactive Production Systems)

• Current NCAR visualization and analysis resources– ~32 processors

• 8 nodes (6 with gfx)

– ~100 TB on-line storage

– ~800 MBs/sec aggregate storage bandwidth

– ~100 users (99 of which will not leave office)

John Clyne & Alan Norton Scientific Computing Division National Center for Atmospheric Research

Documents

Transcript of John Clyne & Alan Norton Scientific Computing Division National Center for Atmospheric Research