Post on 04-Jan-2016
Terascale Terascale Simulation Simulation Tools and Tools and TechnologiTechnologi
eses
The TSTT Interface Definition EffortLori Freitag Diachin
Lawrence Livermore National Lab
2
The TSTT Center
• Goal: To enable high-fidelity calculations based on multiple coupled physical processes and multiple physical scales – Adaptive methods– Composite or hybrid solution strategies– High-order discretization strategies
• Barrier: The lack of easy-to-use interoperable meshing, discretization, and adaptive tools requires too much software expertise by application scientists
The TSTT center recognizes this gap and is addressing the technical and human barriers preventing use of
adaptive, composite, hybrid methods
3
TSTT brings a broad set of meshing-discretization communities together
• Diverse Partners– Overture
– AOMD/Trellis
– CUBIT – NWGrid/Phys
– – Frontier
– Mesquite
– Nek5000
• Meshing Technologies– Structured/Overset– Mixed element hybrid– Unstructured hex– Tetrahedral– Variational approaches– Mesh quality improvement
• Adaptive Technologies– Mesh refinement– Front tracking
• Software Frameworks and PDE Solvers
• Significant Applications Experience
NWGrid mesh
MEGA Boundary Layer Mesh (RPI)
Overture Mesh (LLNL)
CUBIT Mesh (Sandia)
4
The Challenge
• These tools all meet particular needs, but– They do not interoperate to form hybrid, composite meshes– They cannot be easily interchanged in an application
• In general the technology requires too much software expertise from application scientists– Difficult to improve existing codes– Difficult to design and implement new codes
• Our goal is to develop interchangeable and interoperable software components for meshing and discretization
5
TSTT interoperability goals require services build on a common infrastructure
• All tools must build on a common abstract framework and data model– Provides a single reference for
PDE simulation data hierarchy
• Defining common interfaces is critical– Required for interchangeability of
similar tools– Allows vertical integration of tools
• Applications write to the TSTT specification once and can use all TSTT services
GeometryInformation(Level A)
Full GeometryMeshes(Level B)
MeshComponents(Level C)
6
TSTT data model abstracts PDE-simulation data hierarchy
• Core Data Types– Geometric Data: provides a high level description of the boundaries
of the computational domain; e.g., CAD, image, or mesh data
– Mesh Data: provides the geometric and topological information associated with the discrete representation of the domain
– Field Data: provides access to the time dependent physics variables associated with application solution. These can be scalars, vectors, tensors, and associated with any mesh entity.
• Data Relation Managers– Provides control of the relationships among the core data types
– Resolves cross references between entities in different groups
– Provides functionality that depends on multiple core data types
Each core data typehas a TSTT interface• Mesh: TSTTM• Geometry: TSTTG• Fields: TSTTF
Each core data typehas a TSTT interface• Mesh: TSTTM• Geometry: TSTTG• Fields: TSTTF
7
An overarching philosophy guidesour interface definition efforts
• Maintain data structure neutrality
• Create a small set of interfaces that existing packages can support– Small set of ‘core’ functions that must be
implemented
– Larger set of functions supported by reference implementations
• Balance performance and flexibility
• Work with a large tool provider and application community to ensure applicability
• Target both specific uses of TSTT services and broad dissemination as CCA components
Lowers the burden for adoption of the interface
Performance is critical for kernel computations involving mesh access; flexibility is critical for covering a broad usage spectrum
CCA provides infrastructure and guidance for domain-specific interface definition efforts
8
The TSTT mesh data model consists of entities, entity sets and meshes• Entity Definition
– Unique type and topology– Canonical ordering defines adjacency
relationships
• Entity Set Definition– Arbitrary grouping of TSTT mesh entities– There is a single “Root Set”– Relationships among entity sets
• Contained-in• Hierarchical
• Mesh Definition– Simple meshes are a set of connected, non-
overlapping entities related through adjacency relationships
– Meshes can contain collections of simple meshes
9
Core functions must be implemented to be TSTT compliant
• The smallest set of functions required to be considered TSTT compliant
• Provides basic access to vertex coordinates and adjacency information– Mesh loading and saving– Accessing global information such as the root set, geometric
dimension, number of entities of a given type or topology– Primitive array access for entity sets– Global entity arrays for entity sets
• Reference implementations can be used to provide advanced capabilities needed for TSTT services– Requires a copy of mesh data– Allows incremental implementation of advanced functions
10
Advanced traversal functions provide additional flexibility and performance
• Single Entity Traversal– Iterator functionality (init, next, reset, end)
– Query (topology, type, adjacency)
• Workset Traversal– Block iterators allow access of entities in a user-defined array size
– Array based query functions for topology, type, adjacency
– Increases performance by reducing the number of calls through the interface
– Interface is not as intuitive
11
Advanced functionality is supported through optional interfaces
• Entity Sets– Creation/Destruction– Boolean Operations– Relations
• Tags– Allows the incorporate user-defined
data in the TSTT data model– Associated with entities, meshes,
and entity sets• Mesh Modification
– Simple mesh geometry and topology modification– Requires simple classification mechanisms against the
geometric model– Intended to support higher-level functionality
12
The geometry interface provides access to the computational domain
• Must support – automatic mesh generation– mesh adaptation– tracking domain changes– relating information between alternative
discretizations
• Builds on boundary representations of geometry
• Used to support various underlying representations – Commercial modelers (e.g., Parasolid, ACIS)– Modelers that operate from standard files (e.g. IGES, STEP)– Models constructed from an input mesh
13
Both basic and advanced functionalities are supported in the geometry interface
• Model loading and initiation• Topological queries of entities
and adjacencies• Pointwise geometric shape
interrogation• Parametric coordinate
systems• Model topology modification• Tags
14
Relating mesh and geometry data is critical for advanced TSTT services
• Required for e.g., adaptive loops, mesh quality improvement
• Mesh/Geometry Classification Interface– Manages the relationship between the high level
geometric description and the mesh– Called by an application that knows about both
• Capabilities– For a given mesh entity, get the geometric entity
against which it is classified– Establish a classification relationship between a
mesh entity and a geometric entity
15
Preliminary interface implementations are well underway
• Working documents describing the mesh and geometry data models as well as the overall vision
• TSTT Mesh Interface (version 0.6) complete• Preliminary interfaces for geometry and mesh/geometry
classification defined• Implementations
– AOMD, FronTier, GRUMMP, MOAB, NWGrid, Overture for mesh– CGM for geometry
• C, C++, and Fortran language interoperability through SIDL/Babel (CCA)– Analyzing performance ramifications of SIDL/Babel language
interoperability tools (joint with the CCA)
16
Performance studies will provide ‘best practices’ guidelines
• Working with SIDL/Babel team to characterize the interface performance
• Focus on entity-based access– Single entity iterators– Workset iterators
• Compare– Native data structures– Native language interfaces– SIDL/Babel interfaces
Workset Iterators
Initialize Workset(entity_type, num_entities, &wrk_iter)
While (!done){
GetNextWorkset(&entity_handles[ ], &num, wrk_iter);
GetInfo(entity_handles);
for i=0:num-1 {
do_something(entity_handle[i]);
}
}
Destroy Workset(wrk_iter);
Workset Iterators
Initialize Workset(entity_type, num_entities, &wrk_iter)
While (!done){
GetNextWorkset(&entity_handles[ ], &num, wrk_iter);
GetInfo(entity_handles);
for i=0:num-1 {
do_something(entity_handle[i]);
}
}
Destroy Workset(wrk_iter);
17
Preliminary performance focus on TSTT mesh access patterns
Relative Wall Clock Time for Each Variant
0
100
200
300
400
500
1 100Work Set Size
Pe
rce
nta
ge
of
Na
tiv
e L
ink
ed
Lis
t
Native Array
Native Linked List
Native Interface
SIDL Direct
SIDL Memcpy
SIDL For-loop
• Workset Iterators– Increased workset size amortizes overhead – 15-40% compared to native data structures– SIDL a few percent more than native language– Single entity access via workset iterators 2X-
4X for native and SIDL interfaces
• Single entity iterators have similar performance to a workset size of 5-10
• Array based access for NWGrid has .5% overhead independent of mesh size
• Ongoing research to further improve performance when interoperability is not critical (w/ PERC)
Relative Wall Clock Time for Interfaces
75
100
125
150
175
200
225
0 25 50 75 100Work Set Size
Per
cen
tag
e o
f N
ativ
e In
terf
ace
Native Interface
SIDL Direct
SIDL Memcpy
SIDL For-loop
18
The TSTT mesh interface working group uniquely engages a large number of people
Internal– ANL: Fischer,
Buschelman– LLNL: Freitag, Chand,
Henshaw, Dahlgren– PNNL: Trease– RPI: Shephard, Seole– SNL: Tautges, Kim– SUNY SB: Li, Fix
Internal– ANL: Fischer,
Buschelman– LLNL: Freitag, Chand,
Henshaw, Dahlgren– PNNL: Trease– RPI: Shephard, Seole– SNL: Tautges, Kim– SUNY SB: Li, Fix
External (active)– UofC: Loy– UBC: Ollivier-Gooch
External (active)– UofC: Loy– UBC: Ollivier-Gooch
External (monitoring)– UIUC: Guoy– Boeing: Michal (UGC)– LLNL: Keasler– SNL: Pebay, Haynes,
Edwards, Jung– ElemTech: Meyers– UU: Parker– ORNL: Bernholdt
External (monitoring)– UIUC: Guoy– Boeing: Michal (UGC)– LLNL: Keasler– SNL: Pebay, Haynes,
Edwards, Jung– ElemTech: Meyers– UU: Parker– ORNL: Bernholdt
Managing the interactions across institutions• Face to face meetings (all-hands and opportunistic)• Telecons (approximately every three weeks)• Email discussion group (39 participants from 13 institutions)• Working documents and SIDL files• CVS repository
19
Defining the TSTT interfaces has been an interesting challenge
• Interface definition is harder than we anticipated• Cannot achieve the 100 percent solution, so...
– What level of functionality should be supported?– How to best support existing (different) packages?
• The devil is in the details– Initial interface completed a year ago– Lots of interesting issues
• Invariant handles
• Consistent error handling across packages
• Memory management issues
• Core interface functions
• Performance, performance, performance
20
TSTT services support SciDAC applications developers
• Domains from multiple geometry source• Use of alternative mesh generators• Use alternative equation discretization methods and
tools• Application of adaptive control of the
– Geometry - shape optimization
– Mesh - discretization error control
– Both - evolving geometry simulations
• Combining multiple analysis codes to solve multiphysics problems
21
TSTT services
• “Current” TSTT Services – Mesh services libraries (AOMD, MOAB, NWGrid)– Mesh quality improvement (Mesquite)– Discretization libraries (Overture, Trellis)– Parallel mesh generation (MOAB) – Front tracking (FronTier)– Adaptive control loops (combination of tools)
• Services Under Development– Geometry interface functions (not doing geometry)– Fields library– Mesh control for shape optimization
22
Mesquite Mesh Quality Improvement
• Stand alone library of mesh quality improvement algorithms– 2D/3D unstructured and hybrid meshes– Triangles, Quads, Tets, Hex elements– State-of-the-art optimization-based improvement schemes– Joint work with TOPS created fastest mesh quality optimizer in Mesquite
tool suite – improves 1M elements in 15 seconds• TSTT interface to access mesh and geometry information
– Vertex coordinates, element connectivity, tags– Successfully integrated with AOMD, MOAB, Overture, Cubit
• Application uses– Surface mesh smoothing on complex geometries for accelerator modeling– Mesh quality analysis for SLAC Tau3P code– Mesh deformation in design optimization problems– Anisotropic smoothing for error minimization
23
TSTT developed a new interoperable AMR/Front tracking scheme
• Combine front tracking in Frontier with adaptive techniques in Overture– Increase problem resolution using
AMR– Maintain exact interface
representation capabilities
• Applications– Richtmeyer-Meshkov instability– Diesel jet spray break up
• Using TSTT interface to combine front-tracking schemes with NWgrid capabilities
24
TSTT/SLAC collaboration provides adaptive mesh capabilities for Omega3P
• Omega3P is SLAC’s quadratic FEM eigensolver for computing normal modes in RF cavities
• Extremely accurate solutions (0.01% error) are required for requirements of accelerator design
• Using new TSTT technology developed at RPI, adaptive mesh capability has been provided for Omega3P
• Simulations achieve same accuracy with 1/3 the number of unknowns
3 levels of refinement
First Refinement Final Adaptive MeshInitial Mesh
25
Using geometry operatorsmeans alternate solid modelers can be inserted
Using geometry operatorsmeans alternate solid modelers can be inserted
Using TSTT mesh operatorsmeans alternate mesh generatorsand mesh adaptation procedurescan be inserted
Using TSTT mesh operatorsmeans alternate mesh generatorsand mesh adaptation procedurescan be inserted
Using TSTT field operators allows easy construction of alternative error estimators
Components used in construction of adaptive loop for SLAC
Projection-based error estimatorused to construct new mesh sizefield given to mesh modification
Mesh adaptation based on local modification linkeddirectly to CAD
Mesh adaptation based on local modification linkeddirectly to CAD
UnalteredSLAC code
UnalteredSLAC code
Error estimatorsfrom RPI and SLAC
Error estimatorsfrom RPI and SLAC
26
Contact Information
• TSTT Web Site: www.tstt-scidac.org
• David Brown: dlb@llnl.gov• Lori Freitag Diachin: diachin2@llnl.gov• Jim Glimm: glimm@ams.sunysb.edu