Support for Adaptive Computations Applied to

Simulation of Fluids in Biological Systems

Kathy YelickU.C. Berkeley

Project Summary

• Provide easy-to-use, high performance tool for simulation of fluid flow in biological systems.– Using the Immersed Boundary Method

• Enable simulations on large-scale parallel machines. – Distributed memory machine including SMP

clusters

• Using Titanium, ADR, and KeLP with AMR• Specific demonstration problem: Simulation

of the heart model on Blue Horizon.

Outline

• Generic Immersed Boundary Method in Titanium (IBT)– Titanium status – Immersed Boundary method status

• Applications of the method– Heart model– Cochlea model

• Future Plans– Data analysis– Solvers

Titanium on Blue Horizon• Recent improvements:

– Support for PAPI (performance analysis)– Cache optimizations– Portable runtime layer (maintainable)– Faster LAPI-based implementation– Adapted to OS upgrade, ongoing issues

• Gcc does not on recent AIX’s• IBM C++ does not fully support templates

• Plans– Communication optimizations– Common runtime with UPC (possibly

CAF)

MPI vs. LAPI on the IBM SP

0

50

100

150

200

250

300

350

400

1 2 4 8

16

32

64

12

8

25

6

51

2

10

24

20

48

40

96

81

92

16

38

4

32

76

8

65

53

6

13

10

72

MB

/s IBM/MPI

IBM/LAPI

• LAPI bandwidth higher than MPI• Also better small-message overhead

– 9usec vs. 11usec

• Latest Titanium release leverages this

Immersed Boundary Method Structure

Fiber activation & force calculation

InterpolateVelocity

Navier-Stokes Solver

SpreadForce

4 steps in each timestep

Fiber Points

Interaction

Fluid Lattice

Immersed Boundary Method

• Recent Performance Improvements– Use of FFTW in Spectral solver

• 10x performance improvement on t3e• Use on BH still pending

– Use of scatter/gather communication• Copying bounding boxes is still faster• Depends on application and machine

– Load balancing• Alignment of fluid grid (in slabs) and fiber• Multigrid solver might offer more

possibilities

Load Balancing

Egg slicer Pizza cutter

0

50

100

150

200

250

300

350

400

450

0 5 10 15 20No. of processors.

MF

LO

PS

SpreadForce(Pizzacutter)

InterpolateVelocity(Pizzacutter)SpreadForce(Eggslicer)

InterpolateVelocity(Eggslicer)

Fluid grid is divided in slabs for 3D FFT

Application: Heart Simulation

05000

1000015000200002500030000350004000045000

Fiber Spreadforce

NS Solver Interpolatevelocity

Before

After

• Performance improvements over the last year

-64 node t3e ~= 2 node C90 ~= 1-node (8p) BH (probably)

Heart simulation on a Cray T3E

Heart Simulation

• Recent improvements– Support for heart input– Generate data for NYU visualization– Visualization is now OpenGL (ongoing)– Runs on

• Blue Horizon, NERSC SP, T3E, SGI, Millennium

• Short term plans– Finish checkpoint/restart– Larger runs– Complete performance model– Validation

Cochlea Model

• Model of the inner ear– Developed by Julian Bunn and Ed Givelberg

• Contains new features, e.g. membranes

• Implemented one of these last fall• Plan to have Givelberg here next year

Alpha Project Plans

• Support for new applications– Berkeley and Michigan

• Scaling– Berkeley

• Solvers– San Diego, Berkeley, LBNL

• Data analysis and model construction– Ohio, Maryland

Scallop: Multigrid Poisson Solver

• A latency tolerant elliptical solver library– Will be used to build Navier-Stokes Solver– Implemented in KeLP, with a simple interface

• Work by Scott Baden and Greg Balls– Based on Balls/Colella algorithm– 2D implementation in both KeLP and Titanium

• 3D Solver – Algorithm is complete– Implementation running, but performance

tuning is ongoing– Interface between Titanium and KeLP

developed

Elliptical solvers

• A finite-difference based solvers– Good for regular, block-structured domains

• Method of Local Corrections– Local solutions corrected by a coarse solution– Good accuracy, well-conditioned solutions

• Limited communication– Once to generate coarse grid values– Once to correct local solutions– Trades off extra computation for fewer

messages

KeLP implementation

• Advantages – abstractions available in C++– built in domain calculus– communication management– numerical kernels written in Fortran

• Simple interface– callable from other languages– no KeLP required in user code

Improved Heart Structure Model

• Current model is– Based on dog heart, textbook anatomy– Approximation by composing cones

• Building a more accurate model– Use modern imaging on human heart for model– Need to see individual fibers– Collaboration between

• Joel Saltz’s group and • Dr. Robert DePhilip in Anatomy Division of Biomedical

Informatics Dept. at Ohio State University

• Long term goal– Specialize model to patient using MRI data

Analysis of Cardiac Simulations

• Methods and tools to analyze 3D datasets from cardiac blood flow. – Outputs are velocity and pressure values on a

3D grid (1283) over many time steps.

• Characterize behavior under different pathological and physiological conditions. – Natural and artificial heart valves– Vary the initial values of

• Kinematic viscosity • Fluid density• Fiber stiffness and resting lengths

• Use parameter study to find conditions that may lead to aneurysm.

Analysis of Cardiac Simulations

• These queries require support for – spatial subsetting of one or more datasets – processing of the data of interest to

visualize and compare results from one or more datasets

– execution on high-performance machines and in a distributed environment.

• Implemented using DataCutter developed by Joel Saltz’s group.

Summary of Alpha Project Impact

• Several categories– Application development

• Heart and cochlea-component simulation

– Application-level package• Generic immersed boundary method• Parallel for shared and distributed memory• Enables new larger-scale simulations; finer grid

– Solver libraries• Method of Local Corrections• Improved scalability and load balance expected

– Data analysis• Building input data and analysis of results

applicationdata

softwaresystems