Ghost ElementsGhost Elements

Ghost Elements: OverviewGhost Elements: Overview

Most FEM programs communicates via Most FEM programs communicates via shared nodes, using FEM_Update_fieldshared nodes, using FEM_Update_field

Some computations require read-only Some computations require read-only copies of remote copies of remote elementselements—“ghosts”—“ghosts” Stencil-type finite volume computationsStencil-type finite volume computations The push form of matrix-vector productThe push form of matrix-vector product Many kinds of mesh modificationMany kinds of mesh modification

Ghosts are a recent addition to the FEM Ghosts are a recent addition to the FEM frameworkframework

Ghosts: 2D ExampleGhosts: 2D Example

1 2 3 4

1 2Ghostof3

Ghost of2

3 4

Serial Mesh

Left Chunk Right Chunk

Building Ghosts:Building Ghosts:

Add ghost elements layer-by-layer from initAdd ghost elements layer-by-layer from init A chunk will include ghosts of all the A chunk will include ghosts of all the

elements it is connected to by “tuples”—sets elements it is connected to by “tuples”—sets of nodesof nodes

For 2D, a tuple might be a 2-node edgeFor 2D, a tuple might be a 2-node edge For 3D, a tuple might be a 4-node faceFor 3D, a tuple might be a 4-node face You specify a ghost layer with You specify a ghost layer with

FEM_Add_ghost_layer(FEM_Add_ghost_layer(tupleSize,ghostNodestupleSize,ghostNodes)) ghostNodesghostNodes indicates whether to add ghost indicates whether to add ghost

nodes as well as ghost elements.nodes as well as ghost elements.

Building Ghosts:Building Ghosts:

FEM_Add_ghost_elem(FEM_Add_ghost_elem(e,t,elem2tuplee,t,elem2tuple)) ee is the element type is the element type tt is the number of tuples per element is the number of tuples per element elem2tupleelem2tuple maps an element to its maps an element to its

tuples:tuples: A A tupleSizetupleSize by by tt array of integers array of integers Contains element-local node numbersContains element-local node numbers

Repeat this call for each ghost element Repeat this call for each ghost element typetype

Ghosts: Node adjacencyGhosts: Node adjacency/* Node-adjacency: triangles have 3 nodes */

FEM_Add_ghost_layer(1,0); /* 1 node per tuple */

const static int tri2node[]={0,1,2};

FEM_Add_ghost_elem(0,3,tri2node);

0

1 2

Ghosts: Edge adjacencyGhosts: Edge adjacency/* Edge-adjacency: triangles have 3 edges */

FEM_Add_ghost_layer(2,0); /* 2 nodes per tuple */

const static int tri2edge[]={0,1, 1,2, 2,0};

FEM_Add_ghost_elem(0,3,tri2edge);

0

1 2

Extracting and Using GhostsExtracting and Using Ghosts Ghosts are always given larger numbers Ghosts are always given larger numbers

than non-ghosts—that is, ghosts are at the than non-ghosts—that is, ghosts are at the endend

FEM_Get_node_ghost() and FEM_Get_node_ghost() and FEM_Get_elem_ghost(FEM_Get_elem_ghost(ee)) Return the index of the first ghost node or Return the index of the first ghost node or

elementelement

FEM_Update_ghost_field(FEM_Update_ghost_field(fid,e,datafid,e,data)) Obtain other processor’s Obtain other processor’s datadata (formatted like (formatted like

fidfid) for each ghost element of type ) for each ghost element of type ee

0 eg

Ghost Elements: Sub-MeshGhost Elements: Sub-Mesh

Ghost ElementsGhost Elements

Ghosts and SymmetriesGhosts and Symmetries

Ghosts and SymmetriesGhosts and Symmetries

In addition to cross-processor ghosts, can In addition to cross-processor ghosts, can build ghosts to model build ghosts to model problemproblem symmetriessymmetries Translational and rotational periodicitiesTranslational and rotational periodicities Mirror symmetryMirror symmetry

FEM_Add_linear_periodicity(FEM_Add_linear_periodicity(nFaces,nPer, nFaces,nPer, facesA,facesB, nNodes,nodeLocsfacesA,facesB, nNodes,nodeLocs)) Identify these two lists of faces under linear Identify these two lists of faces under linear

periodicity, and build ghosts to matchperiodicity, and build ghosts to match

Symmetry Ghosts: 2D Symmetry Ghosts: 2D ExampleExample

1 2 3 4

1 2Ghostof3

Ghost of2

3 4

Serial Mesh

Left Chunk Right Chunk

Sym. Ghost4

Sym.Ghost1

Horizontal Periodicity

Symmetry Ghost ElementsSymmetry Ghost Elements

Symmetry-Ghost ElementsSymmetry-Ghost Elements

Ghosts and Symmetries: Ghosts and Symmetries: UpdateUpdate

NetFEM ClientNetFEM Client

NetFEM Client: Pretty pictures NetFEM Client: Pretty pictures ofofwave dispersion around a wave dispersion around a crackcrack

NetFEM: Zoom inNetFEM: Zoom in

NetFEM: Outline ElementsNetFEM: Outline Elements

NetFEM: Point NodesNetFEM: Point Nodes

NetFEM ServerNetFEM Server

NetFEM Server Side: OverviewNetFEM Server Side: Overview

To allow the NetFEM client to connect, you To allow the NetFEM client to connect, you add NetFEM registration calls to your add NetFEM registration calls to your serverserver Register nodes and element typesRegister nodes and element types Register data items: scalars or spatial vectors Register data items: scalars or spatial vectors

associated with each node or elementassociated with each node or element You provide the display name and units for You provide the display name and units for

each data itemeach data item Link your program with “-module netfem”Link your program with “-module netfem” Run with “++server”, and connect!Run with “++server”, and connect!

NetFEM Server Side: SetupNetFEM Server Side: Setup

nn=NetFEM_Begin(FEM_My_partition(),=NetFEM_Begin(FEM_My_partition(),timestimesteptep, , dimdim,NetFEM_POINTAT),NetFEM_POINTAT) Call this each time through your timeloop; or Call this each time through your timeloop; or

skipskip timesteptimestep identifies this data update identifies this data update dimdim is the spatial dimension—must be 2 or 3 is the spatial dimension—must be 2 or 3 Returns a NetFEM handle Returns a NetFEM handle nn used by everything used by everything

elseelse NetFEM_End(NetFEM_End(nn))

Finishes update Finishes update nn

NetFEM Server Side: NodesNetFEM Server Side: Nodes

NetFEM_Nodes(NetFEM_Nodes(n,nnodes,coordn,nnodes,coord,”Position ,”Position (m)”)(m)”) Registers node locations with NetFEM—future Registers node locations with NetFEM—future

vectors and scalars will be associated with nodesvectors and scalars will be associated with nodes nn is the handle returned by NetFEM_Begin is the handle returned by NetFEM_Begin nnodesnnodes is the number of nodes is the number of nodes coordcoord is a is a dimdim by by nnodesnnodes array of doubles array of doubles The string describes the coordinate system and The string describes the coordinate system and

meaning of nodesmeaning of nodes Currently, there can only be one call to nodesCurrently, there can only be one call to nodes

NetFEM: Node NetFEM: Node DisplacementDisplacement

NetFEM Server Side: ElementsNetFEM Server Side: Elements

NetFEM_Elements(NetFEM_Elements(n,nelem,nodeper, n,nelem,nodeper, connconn,”Triangles”),”Triangles”) Registers elements with NetFEM—future vectors Registers elements with NetFEM—future vectors

and scalars will be associated with these elementsand scalars will be associated with these elements nn is the handle returned by NetFEM_Begin is the handle returned by NetFEM_Begin nelemnelem is the number of elements is the number of elements nodepernodeper is the number of nodes per element is the number of nodes per element connconn is a is a nodepernodeper by by nelemnelem array of node indices array of node indices The string describes the kind of elementThe string describes the kind of element

Repeat to register several kinds of element Repeat to register several kinds of element Perhaps: Triangles, squares, pentagons, …Perhaps: Triangles, squares, pentagons, …

NetFEM: Element StressNetFEM: Element Stress

NetFEM Server Side: VectorsNetFEM Server Side: Vectors

NetFEM_Vector(NetFEM_Vector(n,valn,val,”Displacement (m)”),”Displacement (m)”) Registers a spatial vector with each node or elementRegisters a spatial vector with each node or element

Whichever kind was registered lastWhichever kind was registered last nn is the handle returned by NetFEM_Begin is the handle returned by NetFEM_Begin valval is a is a dimdim by by nitemsnitems array of doubles array of doubles

There’s also a more general NetFEM_Vector_field in the manualThere’s also a more general NetFEM_Vector_field in the manual The string describes the meaning and units of the vectorsThe string describes the meaning and units of the vectors

Repeat to register multiple sets of vectorsRepeat to register multiple sets of vectors Perhaps: Displacement, velocity, acceleration, rotation, …Perhaps: Displacement, velocity, acceleration, rotation, …

NetFEM: Element VelocityNetFEM: Element Velocity

NetFEM Server Side: ScalarsNetFEM Server Side: Scalars

NetFEM_Scalar(NetFEM_Scalar(n,val,s,n,val,s,”Displacement (m)”)”Displacement (m)”) Registers Registers ss scalars with each node or element scalars with each node or element

Whichever kind was registered lastWhichever kind was registered last nn is the handle returned by NetFEM_Begin is the handle returned by NetFEM_Begin valval is an is an ss by by nitemsnitems array of doubles array of doubles

There’s also a more general NetFEM_Scalar_field in the There’s also a more general NetFEM_Scalar_field in the manualmanual

ss is the number of doubles for each node or element is the number of doubles for each node or element The string describes the meaning and units of the The string describes the meaning and units of the

scalarsscalars Repeat to register multiple sets of scalars Repeat to register multiple sets of scalars

Perhaps: Stress, plasticity, node type, damage, …Perhaps: Stress, plasticity, node type, damage, …

NetFEM Server Side: 2D NetFEM Server Side: 2D ExampleExampleinteger :: t,n, numnp, numel

real*8, dimension(2,numnp) :: coor,d,v,a

integer, dimension(3,numel) :: conn

n=NetFEM_Begin(FEM_My_partition(),t,2,NetFEM_POINTAT)

CALL NetFEM_Nodes(n,numnp,coor,'Position (m)')

CALL NetFEM_Vector(n,d,'Displacement (m)')

CALL NetFEM_Vector(n,v,'Velocity (m/s)')

CALL NetFEM_Vector(n,a,'Acceleration (m/s^2)')

CALL NetFEM_Elements(n,numel,3,conn,'Triangles')

CALL NetFEM_Scalar(n,stress,1,'Stress (pure)')

CALL NetFEM_End(n)

NetFEM: ConclusionNetFEM: Conclusion

Easy, general way to get output from an Easy, general way to get output from an FEM computationFEM computation

Client configures itself based on serverClient configures itself based on server Client can be run anywhere (from home!)Client can be run anywhere (from home!) Server performance impact minimal Server performance impact minimal

(1(1s!)s!) Future work:Future work:

Support multiple chunks per processorSupport multiple chunks per processor Non-network, file-based versionNon-network, file-based version Movie modeMovie mode

Multiple ModulesMultiple Modules

Multiple ModulesMultiple Modules

Use of 2 or more CHARM++ frameworks in the Use of 2 or more CHARM++ frameworks in the same programsame program FEM—multiple unstructured mesh chunksFEM—multiple unstructured mesh chunks MBLOCK—multiple structured mesh blocksMBLOCK—multiple structured mesh blocks AMPI—Adaptive MPI-on-Charm++AMPI—Adaptive MPI-on-Charm++ All based on the Threaded CHARM++ framework All based on the Threaded CHARM++ framework

(TCHARM)(TCHARM)

For example, we may want to use AMPI in our For example, we may want to use AMPI in our FEM program for exchanging information FEM program for exchanging information between FEM chunksbetween FEM chunks

DetailsDetails

Can compose FEM programs with other modules by Can compose FEM programs with other modules by just calling that module’s attach routine from just calling that module’s attach routine from init()init()

For example:For example:void init(void)void init(void)

{{

//Start AMPI, to allow drivers to use MPI calls://Start AMPI, to allow drivers to use MPI calls:

MPI_Attach(“myAMPIFEM”);MPI_Attach(“myAMPIFEM”);

// .. Use FEM_Set() calls as usual ..// .. Use FEM_Set() calls as usual ..

}}

ExampleExample

#include ‘fem.h’#include ‘fem.h’#include ‘mpi.h’#include ‘mpi.h’

void driver(void) {void driver(void) { //..use FEM_Get calls as usual..//..use FEM_Get calls as usual.. //Broadcast “data” from chunk 0://Broadcast “data” from chunk 0: MPI_Bcast(&data,1,MPI_DOUBLE,0,MPI_COMM_WORLD);MPI_Bcast(&data,1,MPI_DOUBLE,0,MPI_COMM_WORLD); //...timeloop: Use FEM_Update_field calls as usual...//...timeloop: Use FEM_Update_field calls as usual... if (dataToSend)if (dataToSend) MPI_Send(&data,4,MPI_INT,dest,tag,MPI_COMM_WORLD);MPI_Send(&data,4,MPI_INT,dest,tag,MPI_COMM_WORLD); elseelse MPI_Recv(&data,4,MPI_INT,src,tag,MPI_COMM_WORLD,&status);MPI_Recv(&data,4,MPI_INT,src,tag,MPI_COMM_WORLD,&status);}}

Multiple Modules: ConclusionMultiple Modules: Conclusion

Easy to use other modules from FEM frameworkEasy to use other modules from FEM framework Just call Just call MPI_AttachMPI_Attach from init, and link with “-module ampi” from init, and link with “-module ampi”

We could also have specified how to combine We could also have specified how to combine frameworks by writing a special startup routine named frameworks by writing a special startup routine named TCHARM_User_setup()TCHARM_User_setup() Not FEM-centric: overrides the normal call to Not FEM-centric: overrides the normal call to init()init() Allows you to call main computation routine something other than Allows you to call main computation routine something other than driver()driver()

See TCHARM manual for detailsSee TCHARM manual for details