Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu [email protected].
-
Upload
norah-gilbert -
Category
Documents
-
view
219 -
download
0
Transcript of Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu [email protected].
![Page 2: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/2.jpg)
http://acts.nersc.gov/
What is the ACTS Toolkit?
• Advanced Computational Testing and Simulation• Part of the DOE 2000 Project
• Will likely shift to base DOE funding
• Tools (roughly 20) for developing parallel applications• Developed (primarily) at DOE Labs
• Separate projects originally
• ACTS is an “umbrella” project• Collecting tools
• Leveraging numerous independently funded projects
![Page 3: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/3.jpg)
http://acts.nersc.gov/
ACTS Project Goals
• Bringing tools together into a “Toolkit”
• Making the tools interoperable
• Providing consistent application interfaces
• Promoting general (not application specific) solutions to parallel programming needs
• Encouraging code re-use
• Impacting DOE science• ASCI, IT2(SSP)
• Enabling large scale applications• example follows: Oil Reservoir Simulation
![Page 4: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/4.jpg)
http://acts.nersc.gov/
Prometheus
• Multigrid solver
• Unstructured meshes in solid mechanics
• Modified matrix graphs
• 26 million DOF, 640 PEs
• Use PETSc
http://www.cs.berkeley.edu/~madams/prometheus
![Page 5: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/5.jpg)
http://acts.nersc.gov/
NERSC Activities
• Make ACTS tools available on NERSC platforms
• Provide technical support
• Find new ER users who can benefit from the tools
• Work with users to integrate tools into applications
• Evaluate the tools
• Create ACTS information center (online)
![Page 6: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/6.jpg)
ACTS Information Online
• One-stop Shopping • But coordinated and integrated with developer resources.
• User-oriented Information• Which tool should I use to solve my problem?
• Which tools are ready for prime time?
• Reviews: strengths, weaknesses, caveats, etc.
http://http://acts.nersc.gov/acts.nersc.gov/
![Page 7: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/7.jpg)
ACTS Support at NERSC
• Support for application development
• Technical support
• Tool installation (by request)
• Leverage with developers
• Support is also available through traditional NERSC channels (e.g. [email protected]).
![Page 8: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/8.jpg)
http://acts.nersc.gov/
Risk
• ACTS Tools are still research projects• Can’t guarantee indefinite support
• Can’t guarantee fixed interface
• But commercial software doesn’t either
• A NERSC goal is to minimize risk to users• Provide unbiased advice
• Promote tools that work
• Support tools with the best chance of surviving
• Be an advocate for users
![Page 9: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/9.jpg)
http://acts.nersc.gov/
Tool Catagorization
• Numerical• Libraries/data structures that implement numerical algorithms
• I.e. your “classic” numerical packages
• PETSc, Aztec, Hypre, ScaLAPACK, SuperLU, PVODE, Opt++
• Structural (Frameworks)• Libraries/data structures that manage data, communicate
• Global Arrays, Overture, POOMA , Cumulvs, PAWS, InDEPS
• Infrastructural• TAU (performance analysis)
• Globus (meta-computing)
• PADRE, Nexus, Tulip (middleware -- mostly used by other tools)
![Page 10: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/10.jpg)
http://acts.nersc.gov/
Tool Status at NERSC
• Installed on the T3E and fully supported by NERSC:• Aztec
• PETSc
• ScaLAPACK
• TAU
• Known to have been successfully used on NERSC’s T3E:• CUMULVS
• Global Arrays
• Globus
• POOMA
• PVODE
![Page 11: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/11.jpg)
Tools available on T3E
![Page 12: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/12.jpg)
http://acts.nersc.gov/
ScaLAPACK
• Installed on the T3E
• Parallel version of LAPACK
• Well known, trusted
• Basic dense linear algebra (BLAS)
• Direct solution of linear systems / Factorization• General, tri-diagonal, banded
• Dense matrix eigensolvers• Sustained 605 GFLOPS on ASCI Red in materials simulation
• Get started: “man scalapack” or “module help scalapack”
![Page 13: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/13.jpg)
http://acts.nersc.gov/
PETSc
• Installed on the T3E
• Functionality for solving PDEs in parallel
• The most widely used and well supported member of the ACTS toolkit
• Runs on serial and parallel machines (even NT)
• C and Fortran Programming Interfaces
• Modular, objected design methodology• Methodology extensible to User code
• Get started: “module help petsc” or “man petsc”
![Page 14: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/14.jpg)
http://acts.nersc.gov/
Aztec
• Installed on T3E
• Solving sparse linear systems on distributed machines
• Highly efficient and scalable• Some Apps have successfully scaled to 1000s of processors
• Krylov iterative methods:• CG, CGS, Bi-CG-Stab, GMRES, TFQMR
• Large preconditioning suite including• Jacobi, Gauss-Seidel, overlapping domain decomps (ILU et al)
• Get started: “module help aztec” or “man aztec”
![Page 15: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/15.jpg)
http://acts.nersc.gov/
Aztec
• User input matrix in DMSR or DVBR format
• Aztec sets up its own data structure for solving linear systems
• Used by groups of dedicated users• Salinas
• MPSalsa
• COYOTE II
• TOUGH
• ...
http://endo.sandia.gov/BB/comp_str_dyn/mp-fema.html
![Page 16: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/16.jpg)
http://acts.nersc.gov/
Using Aztec: basic steps
• Prepare your linear system• distribute the matrix
• call AZ_transform
• set up right-hand side and initial guess
• call AZ_reorder_vec on initial guess and right-hand side
• selective an iterative solver and a preconditioner
• call AZ_solve
• call AZ_invorder_vec on solution
![Page 17: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/17.jpg)
http://acts.nersc.gov/
Using Aztec: DMSR
PE 0
PE 1
PE 2
•local index to global index map
•ja: ja[0:nloc] - starting positionsja[nloc+1:] - global column indices
•valval[0:nloc-1] diagonal elementsval[nloc] not usedval[nloc+1:] off-diagonal nonzeros
![Page 18: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/18.jpg)
http://acts.nersc.gov/
Using Aztec
• Two re-ordering:• global re-ordering to distribute the matrix (done by user)
• local re-ordering to allow maximum overlapping of communication and computation (done by AZ_transform)
• Part of the right-hand side (elements beyond nloc) is used as scratch space internally
• more information at http://acts.nersc.gov/aztec/evaluation.html
• examples to look at• /usr/local/pkg/Aztec/Aztec-2.0/app
• /u1/kewu/azmm.tar
![Page 19: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/19.jpg)
http://acts.nersc.gov/
TAU
• Currently installed on the T3E
• Profiling of C++/C and Fortran programs
• Detailed information - much more than prof/gprof
• For C++: per-class and per-instance profiling
• Graphical display of profiling results• Built-in viewers, interface to VAMPIR
• Automatic instrumentation in the future
• Get started: “module help tau” or “man tau”
![Page 20: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/20.jpg)
http://acts.nersc.gov/
TAU Screen Shot
![Page 21: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/21.jpg)
http://acts.nersc.gov/
Using TAU: instrumentation
Template <class T>
class Templated {
private:
T Data;
public:
T SetData (T d) {
Data = d;
return Data;
}
}
TAU_TYPE_STRING(str, CT(Data)+“(“+CT(d)+“)”);TAU_PROFILE(“Templated::SetData()”, str, TAU_DEFAULT);
![Page 22: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/22.jpg)
http://acts.nersc.gov/
Using TAU: compilation
• Load module tau to define TAUROOTDIR
• include one of the Makefiles from ${TAUROOTDIR}/t3e/lib to define TAU_DEFS and TAULIBS
• Add ${TAULIBS} to link command
• Examples at ${TAUROOTDIR}/examples
• More help at http://acts.nersc.gov/tau/at-nersc.html
![Page 23: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/23.jpg)
http://acts.nersc.gov/
Using TAU: view traces
• Must use VAMPIR on T3E
• To turn TAU trace files to VAMPIR trace file• merge all TAU trace files into one (tau_merge)
• convert merged file with tau_convert -vampir
• see Makefile of examples in ${TAUROOTDIR}/examples
• To use VAMPIR• module load vampir
• vampir tracefile.pv
![Page 24: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/24.jpg)
Tools not on T3E or under development
![Page 25: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/25.jpg)
http://acts.nersc.gov/
SuperLU
• Direct solution of large sparse linear systems
• Single processor and multithread versions are available http://www.nersc.gov/~xiaoye
• Distributed memory (MPI) version under development
• Attained 10 GFLOPS (speedup 100) on 512 node T3E• 2-D matrix distribution
• static pivoting (replace small pivot with larger value)
• iterative refinement to ensure accuracy
![Page 26: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/26.jpg)
http://acts.nersc.gov/
SuperLU
http://www.nersc.gov/~xiaoye/SuperLU
![Page 27: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/27.jpg)
http://acts.nersc.gov/
Hypre
• Family of problem specific preconditioners
• Physics and grid based• Algebraic and structured multigrids
• Sparse approximate inverse
• Will be becoming available for use soon
• Should be useful stand-alone
• …and in the context of other ACTS tools
• http://www.llnl.gov/CASC/linear_solvers/
![Page 28: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/28.jpg)
http://acts.nersc.gov/
PVODE
• Solves ODE & DAE• Adams multi-step for non-stiff
problems
• BDF multi-step for stiff problems
• Can use linear solvers from ACTS• Integration with PETSc planned
• Now includes non-linear solvers (KINSOL)• Newton based solvers
• Globalization strategies (line search, trust region)
Http://www.llnl.gov/CASC/PVODE
![Page 29: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/29.jpg)
http://acts.nersc.gov/
ATLAS and PHiPAC
• Automatic generation of machine optimized BLAS
• Can beat hand-tuned performance
![Page 30: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/30.jpg)
http://acts.nersc.gov/
Global Arrays
• “Shared memory” programming interface on distributed memory computers• But doesn’t hide remote access characteristics
• Global array object abstraction
• One-sided communication
• Used as infrastructure for several computational chemistry packages (e.g. NWChem on the NERSC T3E)
![Page 31: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/31.jpg)
http://acts.nersc.gov/
Global Arrays
http://www.emsl.pnl.gov/pub/proj/tms/hpcc_actinides
![Page 32: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/32.jpg)
http://acts.nersc.gov/
Overture
• Complex geometry
• C++ Class Library
• Overlapping composite grids
![Page 33: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/33.jpg)
http://acts.nersc.gov/
POOMA
• C++ framework (class library) for high performance parallel computing
• Abstractions for arrays, grids, particles
• Allows application to be written with high level “data-parallel” style
• T3E specific difficulties have apparently been overcome• Extensive expression templates overwhelmed compilers
![Page 34: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/34.jpg)
http://acts.nersc.gov/
CUMULVS
• Simple mechanisms for• Computational steering
• Interactive visualization
• Fault tolerance
• Handles, for multiple attached viewers,• Data collection, distribution
• Computational parameter control (“steering”)
• Can be used in heterogeneous environment (layered on PVM)
![Page 35: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/35.jpg)
http://acts.nersc.gov/
Globus
• Provides many services to enable development of meta-computing applications:• Resource discovery, monitoring, allocation
• Process (and IO) staging, management
• Communication - varied protocols - “Nexus”
• User and resource authentication, security (RSA)
• Hides implementation details from application• Provides common “middleware” layer of interfaces
• Transparent remote access to resources• User authentication, File staging, Batch queuing, etc.,
through common interfaces
• Perhaps of greatest immediate interest to application programmer
![Page 36: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/36.jpg)
http://acts.nersc.gov/
PAWS
• Distributing data between separate parallel applications
• Coupling parallel simulation to parallel visualization tools that• require different parallel data distributions
• run on same machine
• ...or run across heterogeneous machines
• Minimal application coding requirements
• PAWS “Controller” handles interactions, data mapping
![Page 37: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/37.jpg)
http://acts.nersc.gov/
PAWS Screen Shot
Controller
Sim. Code
Viz. Code
![Page 38: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/38.jpg)
http://acts.nersc.gov/
Future Directions
• Comparisons of parallel linear system solvers• AZTEC
• BLOCKSOLVE
• ISIS++
• PETSc
• PSPARSLIB
• Test problems, anyone ?
![Page 39: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/39.jpg)
http://acts.nersc.gov/
Future Directions
• CCA (Common Component Architecture)• Developing standardized ways of managing numerical
“components” to allow mixing-and-matching
• Frameworks for gluing components together
• Similarities to CORBA, DCOM, Java Beans
• But:
• Scientific Interface Description Language (allowing Fortran)
• Designed for zero-copy interconnections
• ESI (Equation Solver Interface)• Developing standardized interfaces for scalable linear algebra
• Specific test case for CCA component design
![Page 40: Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e885503460f94b8c925/html5/thumbnails/40.jpg)
http://acts.nersc.gov/
Conclusion
• DOE has funded a nice set of tools
• We are here to maintain them for you
Any questions?
http://http://acts.nersc.gov/acts.nersc.gov/[email protected]