PETSc and Neuronal Networks Toby Isaac VIGRE Seminar, Wednesday, November 15, 2006.

PETSc and Neuronal Networks

Toby IsaacVIGRE Seminar, Wednesday,

November 15, 2006

Tips for general ODEs


Recall: have an input program to convert to PETSc binary format



e.g.: Vec for initial values, Mat for linear ODE, adjacency/connectivity Mat



e.g.: Vec for initial values, Mat for linear ODE, adjacency/connectivity Mat

PetscBinaryView for arrays of scalars (see exampleinput.c)


To keep scalar parameters organized (end time, dt, # cells, etc.) use a PetscBag:



Allows you to save a struct in binary and read in to all processors



Allows you to save a struct in binary and read in to all processors

No need to keep track of order in which scalars are written/read


Recall: using “Load” functions, parallel layout is specified at read in…



Except for arrays: only go to first processor




Use MPI_Bcast to send those arrays to all processors




Use MPI_Bcast to send those arrays to all processors

e.g.: piecewise constant inj. current


A TS object keeps track of the settings for time-stepping



Same old song: TSCreate and TSDestroy




TSSetType: forward Euler, backward Euler, “ode45”, (pseudo-timestepping)




TSSetType: forward Euler, backward Euler, “ode45”, (pseudo-timestepping)

TSSetProblemType: linear, nonlinear


TSSetSolution: set initial conditions


TSSetSolution: set initial conditions TSSetRHSFunction/

TSSetRHSMatrix:


TSSetSolution: set initial conditions TSSetRHSFunction/

TSSetRHSMatrix: Specified functions has format

rhsfunc(ts, t, u, du, void *additional arguments)


TSSetSolution: set initial conditions TSSetRHSFunction/TSSetRHSMatrix: Specified functions has format

rhsfunc(ts, t, u, du, void *additional arguments)

Create a struct for passing additional arguments


TSSetRHSJacobian, if method calls for it



TSSetInitialTimeStep (that is, initial time and initial time step)




TSSetDuration




TSSetDuration TSRKSetTolerance:




TSSetDuration TSRKSetTolerance: Control absolute error over whole

time of integration: a bit sketchy


If only interested in final state, run TSStep to execute



If interested in progress along the way, you need a monitor function:



If interested in progress along the way, you need a monitor function:

Runs after every time step, can output, plot, change parameters, change time-step etc.


Multiple monitor functions can run: e.g. one for parameter changes, one for output



Attention IAF modelers: you can change the state vector too!



Attention IAF modelers: you can change the state vector too!

Syntax: TSSetMonitor, monitor(ts, iter#, t, u, void *args)

Tips for Homogeneous Nets


Most dependency occurs within cell: bad to have one cell divided across processors


Most dependency occurs within cell: bad to have one cell divided across processors

No guarantee that PETSC_DECIDE won’t split your vector this way


Have a vector y of length = # cells


Have a vector y of length = # cells PETSc evenly distributes this

vector



vector nlocal = VecGetLocalSize(y)



vector nlocal = VecGetLocalSize(y) VecCreateMPI(…,

neqns*nlocalcells, PETSC_DETERMINE,&x);


VecSetBlockSize: set this to the number of equations per cell



VecStrideGather: send value from same index for each block to another vector



VecStrideGather: send value from same index for each block to another vector

VecStrideScatter: send values from a vector to the same index for each block


Paradigm for ease/simplicity: gather like indices, make changes, scatter back


Paradigm for ease/simplicity: gather like indices, make changes, scatter back

VecStrideGatherAll/VecStrideScatterAll: take the state vector, break it up into an array of vectors, one for each equivalent index


In RHSFunction: Vec U and Vec DU are inputs



Declare arrays Vec u[neqns], du[neqns]




VecStrideGatherAll at the start




VecStrideGatherAll at the start Set du[i] in terms of u[] for each i




VecStrideGatherAll at the start Set du[i] in terms of u[] for each I VecStrideScatterAll at the end


For very large networks, large number of processors: message passing will take its toll



Order cells so that connections occur between close numbers



Order cells so that connections occur between close numbers

MatGetOrdering, MATORDERING_RCM, MatPermute

Tips for Inhomogeneous Nets


VecSetBlockSize no longer an option



Can be reproduced with more generic VecGather/VecScatter



Can be reproduced with more generic VecGather/VecScatter

Requires the creation of arrays of VecScatter objects: one for each state


VecSetBlockSize no longer an option Can be reproduced with more generic

VecGather/VecScatter Requires the creation of arrays of

VecScatter objects: one for each state VecScatter created from two IS index

objects: TO Vec and FROM Vec


Different types: gathered on separate processors or mixed?



Gathered is easiest to implement: specify which processor, treat like the homogeneous case



Gathered is easiest to implement: specify which processor, treat like the homogeneous case

Mixed is faster: balance processor load, potentially less message passing


If there are ODEs for each connection, then the need for mixed distribution is greater



Ideally (if disparity in # eqns isn’t great):



Ideally (if disparity in # eqns isn’t great):

Lump all cells together, RCM permute, equal # cells per processor

Tips for Adaptive Time Step


PETSc RK45 < ode45


PETSc RK45 < ode45 Will not integrate across

discontinuities



discontinuities For discontinuities you know of

(e.g. time dependent forcing function):



discontinuities For discontinuities you know of (e.g.

time dependent forcing function): Loop: TSSetDuration to

discontinuity, TSStep

PETSc and Neuronal Networks Toby Isaac VIGRE Seminar, Wednesday, November 15, 2006.

Documents

Transcript of PETSc and Neuronal Networks Toby Isaac VIGRE Seminar, Wednesday, November 15, 2006.