Cactus Tools for the Grid
Gabrielle Allen
Max Planck Institute for Gravitational Physics,
(Albert Einstein Institute)
CactusCactus
THE GRID: Dependable, consistent, pervasive access
to high-end resources
CACTUS is a freely available, modular,
portable and manageable environment
for collaboratively developing parallel, high-
performance multi-dimensional simulations
www.CactusCode.org
What is CactusWhat is Cactus Flesh (ANSI C) provides code infrastructure (parameter, variable, scheduling
databases, error handling, APIs, make, parameter parsing, ) Thorns (F77/F90/C/C++/Java/Perl/Python) are plug-in and swappable modules
or collections of subroutines providing both the computational instructructure and the physical application. Well-defined interface through 3 config files
Just about anything can be implemented as a thorn: Driver layer (MPI, PVM, SHMEM, …), Black Hole evolvers, elliptic solvers, reduction operators, interpolators, web servers, grid tools, IO, …
User driven: easy parallelism, no new paradigms, flexible Collaborative: thorns borrow concepts from OOP, thorns can be shared, lots
of collaborative tools Computational Toolkit: existing thorns for (Parallel) IO, elliptic, MPI unigrid
driver, Integrate other common packages and tools: HDF5, Globus, PETSc, PAPI,
Panda, FlexIO, GrACE, Autopilot, LCAVision, OpenDX, Amira, ... Trivially Grid enabled!
Current Version Cactus 4.0Current Version Cactus 4.0 Cactus 4.0 beta 1 released September
1999 Community code: Distributed under GNU
GPL Currently: Cactus 4.0 beta 8 Supported Architectures:
SGI Origin SGI 32/64 Cray T3E Dec Alpha Intel Linux IA32/IA64 Windows NT HP Exemplar IBM SP2 Sun Solaris Hitachi SR8000-F NEC SX-5 Mac Linux ...
0
20
40
60
80
100
120
0
20
40
60
80
10
0
12
0
Processors
Sc
alin
g
Origin
NT SC
Cactus Scaling on T3E-600
192
760
5980
47900
100
1000
10000
100000
1 10 100 1000
Number of Processors
Cactus on T3E 600 Total Mflops/sec
Cactus Computational Toolkit: Parallel utilities (thorns) for computational scientistCactus Computational Toolkit: Parallel utilities (thorns) for computational scientist
CactusBase Boundary, IOUtil, IOBasic, CartGrid3D,
IOASCII, Time
CactusBench BenchADM
CactusConnect HTTPD, HTTPDExtra
CactusExample WaveToy1DF77, WaveToy2DF77
CactusElliptic EllBase, EllPETSc, EllSOR, EllTest
CactusPUGH Interp, PUGH, PUGHSlab,
PUGHReduce
CactusPUGHIO IOFlexIO, IOHDF5, IsoSurfacer
CactusIO IOJpeg
CactusTest TestArrays, TestCoordinates,
TestInclude1, TestInclude2, TestComplex, TestInterp, TestReduce
CactusWave IDScalarWave, IDScalarWaveC,
IDScalarWaveCXX, WaveBinarySource, WaveToyC, WaveToyCXX, WaveToyF77, WaveToyF90, WaveToyFreeF90
CactusExternal FlexIO, jpeg6b
BetaThorns (In Development) IOStreamedHDF5, Renderer,
IOHDF5Util,…, many more
Cactus CommunityCactus Community DLR
Geophysics(Bosl)
Numerical Relativity CommunityCornell
Crack prop.
NASA NS GC
Livermore
SDSS(Szalay)
Intel
Microsoft
Clemson
“Egrid”NCSA, ANL, SDSC
AEI Cactus Group(Allen)
NSF KDI(Suen)
EU Network(Seidel)
Astrophysics(Zeus)
US Grid ForumDFN Gigabit
(Seidel)
“GRADS”(Kennedy, Foster,
Dongarra, et al)
ChemEng(Bishop)
San Diego, GMD, Cornell
Berkeley
Grid ComputingGrid Computing
AEI Numerical Relativity Group has access to high-end resources in over ten centers in Europe/USA
They want: Bigger simulations, more simulations and faster throughput Intuitive IO at local workstation No new systems/techniques to master!!
How to make best use of these resources? Provide easier access … noone can remember ten usernames, passwords,
batch systems, file systems, … great start!!! Combine resources for larger productions runs (more resolution badly
needed!) Dynamic scenarios … automatically use what is available Remote/collaborative visualization, steering, monitoring
Many other motivations for Grid computing ...
Grand PictureGrand PictureRemote steering and monitoring
from airport
Origin: NCSA
Remote Viz in St Louis
T3E: Garching
Simulations launched from Cactus PortalGrid enabled
Cactus runs on distributed machines
Remote Viz and steering from Berlin
Viz of data from previous simulations in
SF café
DataGrid/DPSSDownsampling
Globus
http
HDF5
IsoSurfaces
Remote VisualizationRemote Visualization
IsoSurfaces and Geodesics
Contour plots(download)
Grid FunctionsStreaming
HDF5
Amira
Amira
LCA Vision
OpenDXOpenDX
Remote Offline VisualizationRemote Offline VisualizationViz Client (Amira)
HDF5 VFD
DataGrid (Globus)
DPSS FTP HTTP
VisualizationClient
DPSS Server
FTP Server
Web Server Remote
Data Server
Downsampling, hyperslabs
Viz in Berlin
4TB at NCSA
Only what is needed
Remote Monitoring: Thorn HTTPDRemote Monitoring: Thorn HTTPD Thorn which allows
simulation any to act as its own web server
Connect to simulation from any browser anywhere … collaborate
Monitor run: parameters, basic visualization, ...
Change steerable parameters
See running example at www.CactusCode.org
Wireless remote viz, monitoring and steering
Remote Viz: OpenDXRemote Viz: OpenDX
Open source, (free), multiplatform, large active development community, easy to program
Reads HDF5 (Cactus) data from file or remotely streamed from Cactus
Simple GUI, select different hyperslabs from 3D data
Also support for streamed ASCII data from Cactus
Remote Viz: IsoViewRemote Viz: IsoView
Lightweight, free, isosurface viewer
Distributed from Cactus website
Displays isosurfaces streamed from Cactus (thorn IsoSurfacer calculates isosurfaces inline with the parallel simulation)
Connect to running simulation on any remote machine
Steer isosurface value from client
Cactus PortalCactus Portal KDI ASC Project Technology: Globus, GSI, Java Beans,
DHTML, Java CoG, MyProxy, GPDK, TomCat, Stronghold
Locates resources, builds/finds executables, central management of parameter files/job output, submit jobs to local batch queues, tracks active jobs
Submission of distributed runs Adding more collaborative features
(eg group accounting, …) Accesses the ASC Grid Testbed
(SDSC, NCSA, Argonne, ZIB, LRZ, AEI)
New Grid ApplicationsNew Grid Applications Dynamic Staging: move to faster/cheaper/bigger machine
“Cactus Worm”
Multiple Universe create clone to investigate steered parameter (“Cactus Virus”)
Automatic Convergence Testing from intitial data or initiated during simulation
Look Ahead spawn off and run coarser resolution to predict likely future
Spawn Independent/Asynchronous Tasks send to cheaper machine, main simulation carries on
Thorn Profiling best machine/queue choose resolution parameters based on queue ….
New Grid Applications (2)New Grid Applications (2) Dynamic Load Balancing
inhomogeneous loads multiple grids
Portal resource choosing simulation launching management
Intelligent Parameter Surveys farm out to different machines
Make use of Running with management tools such as Condor, Entropia, etc. Scripting thorns (management, launching new jobs, etc) Dynamic use of eg MDS for finding available resources
Go!
Dynamic Grid ComputingDynamic Grid Computing
Clone job with steered parameter
Queue time over, find new machine
Add more resources
Found a horizon,try out excision
Look forhorizon
Calculate/OutputGrav. Waves
Calculate/OutputInvariants
Find bestresources
Free CPUs!!
NCSA
SDSC RZG
SDSC
LRZ Archive data
Cactus Grid DevelopmentCactus Grid Development
Projects: GrADs, KDI Astrophysics Collaboratory, TIKSL/GriKSL, EGrid/GGF, Globus team, AEI
Grid Application Development Toolkit: Application developer should be able to build simulations with tools that easily enable dynamic
grid capabilities e.g. Query information server, Network Monitoring, Simulation instrumenation/Contract specification, Spawning and Migrating, Authentification Server, Data handling, Data transfer, Visualization, Steering, …
Remote visualization, steering, data management User Portal Dynamics Applications
More Information ...More Information ... Cactus:
Web Site: www.CactusCode.org (Documentation/Tutorials etc) Cactus Worm: www.CactusCode.org/Development/Egrid.html
Global Grid Forum (Egrid) www.egrid.org www.gridforum.org
ASC Portal www.ascportal.org
TIKSL Gigabit Computing www.zib.de/Visual/projects/TIKSL/
Black Holes and Neutron Star: Pictures and Movies jean-luc.aei.mpg.de
Any questions: [email protected]
Top Related