1 FEM Framework Tutorial Sayantan Chakravorty 10/19/2004.
-
Upload
neil-sherman -
Category
Documents
-
view
236 -
download
0
Transcript of 1 FEM Framework Tutorial Sayantan Chakravorty 10/19/2004.
1
FEM Framework Tutorial
Sayantan Chakravorty10/19/2004
2
Roadmap
Why use FEM? FEM Concepts FEM Program Structure FEM Basic Calls FEM Advanced Calls Extra Features
3
Why use FEM?
4
Why use the FEM Framework?
Makes parallelizing a serial code faster and easier Handles mesh partitioning Handles communication Handles load balancing (via Charm)
Allows extra features NetFEM Visualizer Collision Detection Library Parallel Refinement Library IFEM Matrix Library
5
Why not use the FEM Framework?
Does not help you write serial code But it does help with parallel
Another thing to learn But it has a manual and examples
Another thing to break But it runs on native MPI as well as
AMPI
6
Fem Framework Users CSAR
Rocflu: Fluids solver CPSD
SpaceTime meshing Frac3D
Fracture Mechanics Dendritic Growth
Metal Solidification process
7
FEM Concepts
8
FEM Basics
FEM programs manipulate elements and nodes
Element is a portion of problem domain, surrounded by nodes
Node is one point in the domain
9
Serial FEM Mesh
N5N4N2E3
N4N2N1E2
N4N3N1E1
Surrounding Nodes
Element
10
Partitioned Mesh
N3N2N1E2
N4N3N1E1
Surrounding Nodes
Element
N3N2N1E1
Surrounding Nodes
Element
Shared Nodes
N3N4
N1N2
BA
11
FEM Parallel Model: Shared Nodes
“Shared Node” model Element computations based on
values of surrounding nodes Node values are sum of
surrounding elements
Example: Mechanical Simulation Element stresses are computed
from locations of element’s nodes Sum up forces at nodes
12
FEM Mesh: Node Communication
Summing forces from other processors only takes one call:
FEM_Update_field
Adds values from shared nodes
13
FEM Parallel Model: Ghosts
“Ghost” model Element computations based only
on values of surrounding nodes and elements
Example: Fluid Dynamics Element pressures and velocities
come from neighboring element pressures and velocities, and node locations
14
FEM Mesh: Ghosts
1 2 3 4
1 2Ghostof3
Ghost of2
3 4
Serial Mesh
Left Chunk Right Chunk
Partitioning
Communication
15
FEM Program Structure
16
Serial FEM Program Structure
read mesh, connectivity, boundary conditionstime loop element loop- Element deformation applies forces to surrounding nodes node loop- Forces and boundary conditions change node positions end time loopwrite out mesh data for postprocessing
17
Parallelization
Partition the FEM Mesh into multiple chunks
Distribute elements, replicate shared nodes and/or add ghosts Keep track of communication
Partition so that communication is minimized
18
Parallel FEM Program
read/get chunk mesh data, connectivity, shared nodeschunk time loop element loop- Element deformation applies forces to surrounding nodes <update forces on shared nodes> node loop- Forces and boundary conditions change node positions end time loopwrite chunk mesh data for postprocessing
19
FEM Framework Program
Consists of at least two user-written subroutines init driver
init is called on chunk 0 driver is called on every chunk
20
init
subroutine init read the serial mesh and configuration data inform the framework about the mesh end subroutine
21
driver
subroutine driver get local mesh chunk time loop FEM computations communication more FEM computations end time loop end subroutine
22
Structure of an FEM Application
init()
Update Update Update
driver driver driver
23
FEM Mesh Access Calls
24
void FEM_Mesh_data(
int mesh,int entity,int attr, void *data, int first,
int length,int datatype,
int width);
void FEM_Mesh_data(
int mesh,int entity,int attr, void *data, int first,
int length,int datatype,
int width);
Get/Set, and multi-mesh Get/Set, and multi-mesh supportsupportNODE, ELEM, SPARSE NODE, ELEM, SPARSE
(+GHOST)(+GHOST)DATA, CONN, SYM, DATA, CONN, SYM, GLOBALNO,…GLOBALNO,…
Apply to first…first+length-1Apply to first…first+length-1
User data: width x length User data: width x length arrayarray
User data User data formattingformatting
25
FEM_Mesh_data(
mesh,FEM_NODE,FEM_DATA+23, coord, 0,nNodes, FEM_DOUBLE,3
);
Mesh Access: Example
e.g., e.g., FEM_Mesh_default_read()FEM_Mesh_default_read()We’re changing node We’re changing node valuesvaluesUser data (tag User data (tag 23)23)
Change all the Change all the nodesnodes
An array of 3 x nNodes An array of 3 x nNodes doublesdoubles
3 3 doubles/noddoubles/nodee
26
FEM Communication Calls
27
Node Fields Framework handles combining data
for shared nodes and keeps them in sync
Framework does not understand meaning of node fields, only their location and types
Framework needs to be informed of locations and types of fields
Create_field once, Update_field every timestep
28
Create a Field
integer function FEM_Create_simple_field(datatype, len)
integer, intent(in) :: datatype, len
29
Update Field: Shared Nodes
subroutine FEM_Update_Field(fid,nodes) integer, intent(in) :: fid varies, intent(inout) :: nodes
30
FEM Ghost Layers
31
Ghost Elements: Overview Most FEM programs
communicate via shared nodes Some computations require
read-only copies of remote elements—“ghosts” Stencil-type finite volume
computation Many kinds of mesh modification
32
Ghosts: 2D Example
1 2 3 4
1 2Ghostof3
Ghost of2
3 4
Serial Mesh
Left Chunk Right Chunk
33
Building Ghosts: Add ghost elements layer-by-layer from init A chunk will include ghosts of all the
elements it is connected to by “tuples”—sets of nodes
For 2D, a tuple might be a 2-node edge For 3D, a tuple might be a 4-node face You specify a ghost layer with
FEM_Add_ghost_layer(tupleSize,ghostNodes) ghostNodes indicates whether to add ghost
nodes as well as ghost elements.
34
Building Ghosts: FEM_Add_ghost_elem(e,t,elem2tuple) e is the element type t is the number of tuples per element elem2tuple maps an element to its tuples:
A tupleSize by t array of integers Contains element-local node numbers
Repeat this call for each ghost element type
35
Ghosts: Node adjacency/* Node-adjacency: triangles have 3 nodes */
FEM_Add_ghost_layer(1,0); /* 1 node per tuple */
const static int tri2node[]={0,1,2};
FEM_Add_ghost_elem(0,3,tri2node);
0
1 2
36
Ghosts: Edge adjacency
/* Edge-adjacency: triangles have 3 edges */
FEM_Add_ghost_layer(2,0); /* 2 nodes per tuple */
const static int tri2edge[]={0,1, 1,2, 2,0};
FEM_Add_ghost_elem(0,3,tri2edge);
0
1 2
37
Extracting and Using Ghosts Ghosts are always given larger
numbers than non-ghosts—that is, ghosts are at the end
FEM_Get_node_ghost() and FEM_Get_elem_ghost(e) Return the index of the first ghost node
or element
FEM_Update_ghost_field(fid,e,data) Obtain other processor’s data (formatted
like fid) for each ghost element of type e
0 eg
38
Update Field: Ghosts
subroutine FEM_Update_ghost_field(fid,elType,elts) integer, intent(in) :: fid,elType varies, intent(inout) :: elts
39
Ghost Example: Mesh
40
Ghost Example: Ghost Elements
41
FEM Installation
42
Where to Get It ?FEM Framework is included in Charm++ distribution, available under CVSCSH:setenv CVSROOT ":pserver:[email protected]:/cvsroot"Or BASH:export CVSROOT=":pserver:[email protected]:/cvsroot"
You should now be able to do a> cvs login(no password needed, just type [Enter] at prompt)
and then> cvs co -P charmto get the entire Charm++ source: FEM is in charm/src/libs/ck-libs
43
How to Build It ?
> cd charm
and do
> ./build FEM net-linux -O
This will make a net-linux directory, with bin, include, lib etc subdirectories.
Platforms: net-sol, mpi-origin, mpi-linux etc.
44
How to Compile & Link ? Use “charmc”: available under bin
a multi-lingual compiler driver, understands f90
Knows where modules and libraries are Portable across machines and compilers
Linking use “-language femf” : for F90 Use “–language fem” : for C/C++
See example Makefiles charm/examples/fem/…
45
How to Run ?
Charmrun A portable parallel job execution
script Specify number of processors: +pN Specify number of chunks: +vpN Special “nodelist” file for net-*
versions
46
Example
Nodelist File: $(HOME)/.nodelist
group main host tur0001.cs.uiuc.edu host tur0002.cs.uiuc.edu host tur0003.cs.uiuc.eduetc…
./charmrun pgm +p4
47
Advanced FEM Calls
48
Advanced: FEM Migration
49
Advanced API: Migration Chunks may not be computationally
equal: Results in load imbalance Multiple chunks per processor Chunks cannot have writable global
data Automatic load balancing
Migrate chunks to balance load How to migrate allocated data for
chunks ? Embed it in a user-defined type
50
Chunk Data Example
MODULE my_block_mod TYPE my_block INTEGER :: n1,n2x,n2y REAL*8, POINTER, DIMENSION(:,:) :: arr END TYPE END MODULE
51
Pack/Unpack (PUP) RoutineSUBROUTINE pup_my_block(p,m) USE my_block_mod USE pupmod INTEGER :: p TYPE(my_block) :: m call fpup_int(p,m%n1) call fpup_int(p,m%n2x) call fpup_int(p,m%n2y) IF (fpup_isUnpacking(p)) THEN ALLOCATE(m%arr(m%n2x,m%n2y)) END IF call fpup_doubles(p,m%arr,m%n2x*m%n2y) IF (fpup_isDeleting(p)) THEN DEALLOCATE(m%arr) END IF END SUBROUTINE
52
Registering Chunk Data
!- Fortran driver subroutine use my_block_mod interface subroutine pup_my_block(p,m) use my_block_mod INTEGER :: p TYPE(my_block) :: m end subroutine end interface
TYPE(my_block) :: m
CALL FEM_Register(m,pup_my_block)
53
Migration Every chunk driver calls
FEM_Migrate Framework calls PUP
for getting the size of packed data For packing data
Chunk migrates to new processor
Framework calls PUP for unpacking
Driver returns from FEM_Migrate
54
Advanced: Complicated Fields
55
Node Fields
56
FEM_Create_Fieldfunction integer :: FEM_Create_Field( base_type, vec_len, offset, dist) integer, intent(in) :: base_type, vec_len, offset, dist
Base_type
•FEM_BYTE- INTEGER*1, or CHARACTER*1 •FEM_INT- INTEGER*4 •FEM_REAL- REAL*4 •FEM_DOUBLE- DOUBLE PRECISION, or REAL*8
57
Create_field Example
! 3D Force for each node! stored as 3*n real*8 array
REAL*8 ALLOCATABLE, DIMENTION(:) :: nodeForce INTEGER :: fid
... allocate nodeForce as 3*n_nodes...
fid = FEM_Create_Field(FEM_DOUBLE,3, foffsetof(nodeForce(1),nodeForce(1)), foffsetof(nodeForce(1),nodeForce(4))
58
Create_field Example
! 3D force is contained as fXYZ variable! in a user-defined type node_type
TYPE(node_type), ALLOCATABLE, DIMENTION(:) :: nodes INTEGER :: fid
...allocate nodes array as n_nodes...
fid = FEM_Create_Field(FEM_DOUBLE,3, foffsetof(nodes(1), nodes(1)%fXYZ), foffsetof(nodes(1), nodes(2)) )
59
Advanced: FEM on MPI
60
FEM on MPI
FEM routines perform their communication using MPI calls
This means you can call FEM routines from an MPI program; or call MPI routines from an FEM program
61
Chunk Data Example
PROGRAM main include ‘femf.h’ include ‘mpif.h’ CALL MPI_Init() CALL FEM_Init(MPI_COMM_WORLD) ... MPI or FEM routines ...END PROGRAM
62
Extra FEM Features
63
Collision Detection
64
Charm++ Collision Detection Detect collisions (intersections) between objects
scattered across processors
Built on Charm++ Arrays Overlay regular 3D sparse grid of voxels (boxes) Send objects to all voxels they touch Collect collisions from each voxel
Collision response is left to caller
65
Mesh Adaptation
66
Parallel Mesh Refinement 2D refinement 2D coarsening 3D refinement Integration with FEM
67
Triangular Mesh Refinement To refine, split the longest edge:
But if split neighbor has a longer edge, split his edge first
Refinement propagates across mesh, but preserves mesh quality
Initial 2D parallel implementation built on Charm++ Integrated with FEM Framework
FEM_Refine2D_Newmesh FEM_Refine2D_Split
• Input: Nodes, coordinates, elements and desired areas for elements
• Optional: Boundary values for edges• Effect: Updates FEM mesh, interpolates user
data and sets boundary values for new nodes
68
TMR : Results
Initial Mesh: coarse
Refined Mesh: later in the simulation
69
2D coarsening Collapse the smallest edge of an element Makes the adjacent elements larger Being integrated into FEM
70
Parallel 3D refinement Longest Edge based refinement Longest Face based refinement Internal refinement
71
NetFEM
72
NetFEM Client: pretty pictures
Wave dispersion off a crack (simplified frac3d)
73
NetFEM: Easy Visualization Can interact with a running FEM
application Uses Charm++ CCS (Client-server) library
Easy to “publish” attributes
NetFEM n=NetFEM_Begin(2,t,NetFEM_POINTAT);
NetFEM_Nodes(n,nnodes,(double *)g.coord,"Position (m)"); NetFEM_Vector(n,(double *)g.d,"Displacement (m)"); NetFEM_Vector(n,(double *)g.v,"Velocity (m/s)");
NetFEM_Elements(n,nelems,3,(int *)g.conn,"Triangles"); NetFEM_Scalar(n,g.S11,1,"X Stress (pure)"); NetFEM_Scalar(n,g.S22,1,"Y Stress (pure)"); NetFEM_Scalar(n,g.S12,1,"Shear Stress (pure)");
NetFEM_End(n);
Author:
Orion Lawlor
74
NetFEM: Easy visualization
75
NetFEM: Zoom in
76
NetFEM: Outline Elements
77
NetFEM: Point Nodes
78
NetFEM Server Side: Overview To allow the NetFEM client to connect,
you add NetFEM registration calls to your server Register nodes and element types Register data items: scalars or spatial
vectors associated with each node or element
You provide the display name and units for each data item
Link your program with “-module netfem”
Run with “++server”, and connect!
79
NetFEM Server Side: Setup n=NetFEM_Begin(FEM_My_partition(),ti
mestep, dim,NetFEM_POINTAT) Call this each time through your
timeloop; or skip timestep identifies this data update dim is the spatial dimension—must be
2 or 3 Returns a NetFEM handle n used by
everything else NetFEM_End(n)
Finishes update n
80
NetFEM Server Side: Nodes NetFEM_Nodes(n,nnodes,coord,”Posi
tion (m)”) Registers node locations with NetFEM—future
vectors and scalars will be associated with nodes n is the handle returned by NetFEM_Begin nnodes is the number of nodes coord is a dim by nnodes array of doubles The string describes the coordinate system and
meaning of nodes
Currently, there can only be one call to nodes
81
NetFEM: Node Displacement
82
NetFEM Server Side: Elements NetFEM_Elements(n,nelem,nodeper,
conn,”Triangles”) Registers elements with NetFEM—future vectors
and scalars will be associated with these elements
n is the handle returned by NetFEM_Begin nelem is the number of elements nodeper is the number of nodes per element conn is a nodeper by nelem array of node indices The string describes the kind of element
Repeat to register several kinds of element Perhaps: Triangles, squares, pentagons, …
83
NetFEM: Element Stress
84
NetFEM Server Side: Vectors NetFEM_Vector(n,val,”Displacement (m)”)
Registers a spatial vector with each node or element
• Whichever kind was registered last n is the handle returned by NetFEM_Begin val is a dim by nitems array of doubles
• There’s also a more general NetFEM_Vector_field in the manual
The string describes the meaning and units of the vectors
Repeat to register multiple sets of vectors Perhaps: Displacement, velocity,
acceleration, rotation, …
85
NetFEM: Element Velocity
86
NetFEM Server Side: Scalars NetFEM_Scalar(n,val,s,”Displacement (m)”)
Registers s scalars with each node or element• Whichever kind was registered last
n is the handle returned by NetFEM_Begin val is an s by nitems array of doubles
• There’s also a more general NetFEM_Scalar_field in the manual
s is the number of doubles for each node or element
The string describes the meaning and units of the scalars
Repeat to register multiple sets of scalars Perhaps: Stress, plasticity, node type,
damage, …
87
NetFEM Server Side: 2D Exampleinteger :: t,n, numnp, numel
real*8, dimension(2,numnp) :: coor,d,v,a
integer, dimension(3,numel) :: conn
n=NetFEM_Begin(FEM_My_partition(),t,2,NetFEM_POINTAT)
CALL NetFEM_Nodes(n,numnp,coor,'Position (m)')
CALL NetFEM_Vector(n,d,'Displacement (m)')
CALL NetFEM_Vector(n,v,'Velocity (m/s)')
CALL NetFEM_Vector(n,a,'Acceleration (m/s^2)')
CALL NetFEM_Elements(n,numel,3,conn,'Triangles')
CALL NetFEM_Scalar(n,stress,1,'Stress (pure)')
CALL NetFEM_End(n)
88
NetFEM: Conclusion Easy, general way to get output from an
FEM computation Client configures itself based on server Client can be run anywhere (from
home!) Server performance impact minimal
(1s!) Future work:
Support multiple chunks per processor Movie mode
89
Conclusions: Charm++ FEM
Easy to parallelize existing codes
Flexible: shared nodes or ghosts High performance Extra features
Visualization with NetFEM Collision Matrix methods with IFEM
98
Additional features IFEM: Iterative FEM Linear Solver
Interface FEM supports Symmetries Provides support for data
transfer between meshes