HAL Id: hal-01379164https://hal.inria.fr/hal-01379164

Submitted on 11 Oct 2016

HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, estdestinée au dépôt et à la diffusion de documentsscientifiques de niveau recherche, publiés ou non,émanant des établissements d’enseignement et derecherche français ou étrangers, des laboratoirespublics ou privés.

Real-time Simulation of Distributed Control Systems:The example of Functional Electrical Stimulation

Daniel Simon, David Andreu

To cite this version:Daniel Simon, David Andreu. Real-time Simulation of Distributed Control Systems: The exampleof Functional Electrical Stimulation. ICINCO: International Conference on Informatics in Control,Automation and Robotics, Jul 2016, Lisboa, Portugal. pp.455-462, �10.5220/0005967804550462�. �hal-01379164�

Real-time Simulation of Distributed Control Systems : the example ofFunctional Electrical Stimulation

Daniel Simon1, David Andreu21 CAMIN team, INRIA, LIRMM Bat 5 CC05 017, 860 Rue Saint Priest, Montpellier Cedex 5, France

2 CAMIN team, Univ. Montpellier, LIRMM Bat 5 CC05 017, 860 Rue Saint Priest, Montpellier Cedex 5, Francedaniel.simon@inria.fr, andreu@lirmm.fr

Keywords: Hybrid simulation, Real-time control, Control architecture validation

Abstract: The paper presents a real-time open software simulation framework, dedicated to the analysis of control sys-tems deployed over distributed execution resources and wireless links. It is able to consistently simulate inparallel the numerical devices (real-time tasks and communication links) and the evolution of the controlledcontinuous time plant. It is applied to foresee future enhancements of a Functional Electrical Stimulation(FES) system used in therapy for rehabilitation or substitution for disabled people. It is a distributed controlsystem using electrodes to interface a digital control system with livings. Hence the whole system gatherscontinuous-time (muscles and nerves) and discrete-time (controllers and wireless links) components. Duringthe design process, realistic simulation remains a precious tool ahead of real experiments to check without dan-ger that the implementation matches the functional and safety requirements. The simulation tool is especiallydevoted to the joint design and analysis of control loops and real-time features.

1 FROM CONTROL DESIGN TOREAL-TIME EXPERIMENTS

The design and validation of complex systems, likecyber-physical systems which include both physicaland computational devices, is costly, time consum-ing and needs knowledge and cooperation of differentdisciplines (Ben Khaled-El Feki, 2014). For example,medical robotics belong to such category and requirethe coordinated design of both bio-mechanical, elec-trical, and chemical models (from a physical point ofview) and many-sided controllers (from a computa-tional point of view).

Simulations, handling both the components andtheir interaction, are needed from the early stageto speed-up the design, development and validationphases, especially when real experiments are costlyor dangerous for human beings.

1.1 Simulation needs

Understanding and modeling the influence of an im-plementation (support system) on the QoC (Qual-ity of control) is a challenging objective in con-trol/computing co-design process. Real-time prop-erties (task response times) and the network Qual-ity of Service (QoS) influence the controlled system

properties and QoC. However real-life size problems,involving non-linear systems and uncertain compo-nents, still escape from a purely theoretic framework.In this design process, simulation is an indisputablestep between concept design and prototype validation.

Models are needed to describe the mechatroniccontinuous system, to compute the discrete time con-trol laws and diagnosers, and to handle the networkbehavior. As usual, choosing the right model resultsof a trade-off between complexity and fidelity. Indeedseveral models with different granularity are used dur-ing the design and validation process, from fast proto-typing until real-time simulation and implementation.

1.2 Numerical simulation of controlprocess

The main purpose of the numerical simulation is toapproximate as faithfully as possible the behavior ofthe complex dynamic system. In other words, bound-ing the simulation errors is an important goal of nu-merical simulations so that the designers can be con-fident with the prediction.

A distinctive feature of real-time control systemsis that they are hybrid. Indeed, they gather compo-nents with different time nature. Digital components,

such as CPUs and networks, run over discrete timescale while handling real-time tasks and communica-tion packets. Digital components can be easily exe-cuted on the CPUs, using the multitasking capabili-ties of the simulation cores to simulate the concurrentnature of the distributed control system.

On the other hand, the physical components ofthe control system – such as actuators, sensors andmechatronic parts – are continuous time componentswhich can be described by some kind of non-lineardifferential equations. As real system are too complexto lead to explicit solutions of their dynamic behavior,they must be integrated on computers using a numeri-cal solver. Numerical integration is still an active fieldin numerical analysis, and provides many effectivenumerical solvers and associated software suites, ableto solve the continuous models either using time dis-cretization or state quantization (Ben Khaled-El Feki,2014).

Finally, all the components of the simulated sys-tem must be carefully interconnected, and their timescales must be meshed to provide consistent simula-tion results over the different time representations andscales. Both modeling and numerical integration dealwith approximations, hence it is first needed to finda satisfactory trade-off between the simulation speedand precision.

Real-time simulation needs to consider two timescales :

• Real-time : it is the time reference of the real physicalsystem that is modeled and simulated, e.g. as mea-sured by an absolute wall clock;

• Simulated time : it is the time elapsed during the ex-ecution of the simulation that can be measured by thenumerical integrator clock (i.e., by the accumulationof integration steps).

To be consistent, the simulated time and real-timemust be carefully meshed to meet at some precisepoints (Ben Khaled-El Feki, 2014).

Several simulation steps of growing accuracy andfidelity to the real implementation can be processed,from basic functional investigation in continuous timeuntil hardware-in-the-loop setups including parts ofthe final components, just prior to real tests.

Model-in-the-loop A first step is the choice of a con-trol structure, based on a model of the plant and onthe control objective. The control toolbox now con-tains many tools to state and solve a large varietyof control problems applied to various plants. Be-sides control theory, preliminary design and tests of-ten rely on simulation tools such as Scilab/Scicos orMatlab/Simulink.

Software-in-the-loop As for MIL, the SIL phaseconsiders only simulated (i.e. software) elements,but it takes into account the actual production code,including implementation related constraints such asfixed-point computations and memory size limits.

Hardware-in-the-loop After validating the softwarein a real-time SIL simulation, the production hard-ware can be checked in a real-time Hardware-In-the-Loop (HIL) simulation. A HIL simulation is made ofmixed simulated and real components, which meansthat some real components –usually the physical con-trol target– are replaced by their software avatar.

The next sections are devoted to the developmentof a simulation framework dedicated to the case of adistributed electrical functional electrical system.

2 THE FES FRAMEWORK

Functional electrical stimulation (FES) is one of ex-isting rehabilitation techniques to restore lost motorfunctions for motor-impaired people. For example, incase of spinal cord injuries, the natural pathways be-tween the central nervous system (CNS) and periph-eral nerves are open under the lesion level. Thereforesuch lesions forbid both the activation of movementsthrough motor nerves and the collection of sensoryinformation by the CNS.

In FES systems, electrodes are connected tonerves or muscles and generate electrical pulses to in-duce contractions of muscles. Similar electrodes canbe used to collect activity signals from muscles (Elec-troMyoGrams EMG) or from nerves (ElectroNeuro-Grams ENG). FES is primarily used for rehabilitationof functions for people with disabilities. Until nowFES is mainly used in open-loop mode, where thetherapist selects predefined currents patterns (e.g., fre-quency, intensity) according to the desired effect onthe patient. It is argued, e.g. (Zhang et al., 2013), thatclosed-loop FES is necessary to safely control and co-ordinate the numerous electrodes and muscles neededto generate complex movements such as walking.

Sensors 1st ctrl Sensors 2nd ctrl Actuators Ctrl 1 Actuators Ctrl 2

Ctrl law 1

PODEMG

PODSTIM

Set-point

Controller

Ctrl law 2Set-point

PODEMG

PODEMG

PODSTIM

Sharedsensors

Mutual exclusion between Controllers

Figure 1: Feedback FES

In that case (Figure 1), the stimulation controlleris fed back by various sensors, such as limbs joints

angles, IMUs providing accelerations, EMG sig-nals. . . These signals are used by feedback controllersto accurately control the artificially actuated limbs.

Indeed it is well known that feedback control pro-vides adaptability w.r.t. varying operation conditionsand robustness w.r.t. the uncertainties spoiling thecontrol loops components.

Compared with mechanical devices, the large un-certainties and variability of livings make them espe-cially difficult to be accurately modeled and safelycontrolled. Conversely, considering humans in theloop, the safety of the technology must be validated,e.g. formally assessed, to allow for its certificationand usage in everyday life on a large scale.

2.1 The Phenix Liberty FES system

The Phenix Liberty FES system has been developedalong a partnership between the Demar research teamand the Vivaltis company 1. It is devoted to externalstimulation, i.e. the electrodes are glued on the skin,facing the target muscle or nerve. Compared withits competitors, a distinctive feature of the system isits wireless implementation. Considering the multi-ple sensors and actuators needed to control complexmotion, distributed systems over wireless links com-ply with mobility constraints, leading to acceptancefrom the human users. The marketed system executesa preset stimulation sequence in open-loop. Althoughit is fluently used to correct, e.g., foot drop, finite-state controllers typically are not adequate to adaptstimulation patterns for frequent changes of walkingspeed. FES also fastly involves muscular fatique,thus needed an on-line detection and adaptation of thestimulation profiles. Current research aims to designand implement closed-loop FES using this system.

PC

Control device

Stimulation unit

Electrode

Micro sensor

Acquisition unit

Figure 2: The Phenix FES system

The main components are :

• A main PC is used as a end-users (i.e. a therapist)interface, to select the stimulation pattern out of a li-brary of pre-defined profiles, and to provide the gate-way with the controller using an USB link;

• The control device manages the communications withthe distributed stimulation units through the wireless

1http://www.vivaltis.com/

network running the protocol stack. For future feed-back controlled stimulation it will also run the real-time control loops. The controller board is equippedwith a Freescale MC1322X micro-controller and anintegrated IEEE802.15.4 2.4 GHz Radio Frequency(RF) emitting/receiving chip.

There are several kind of distributed units (DSUs, alsonamed PODs) :

• The ”Stim/Bio” DSUs allows for both the stimula-tion of muscles for movement generation and for theacquisition of EMG signals for biofeedback purpose.They are made of two electronic boards :

– A generic board embeds the housekeeping hardwareand software to provide the electrical supply (bat-tery), switches and LEDs, and a physical networktransceiver. The protocol stack and the requests (stim-ulation and acquisition) processing are handled by aFreescale MC1322X micro-controller.

– A daughter board executes the stimulation patterns orthe acquisition requests through two input/output ana-logue channels.

• The ”universal” DSUs use the same generic board, butembeds in its daughter board specific sensors. For ex-ample, IMUs or goniometers are planned to be em-bedded in a near future.

The communication architecture uses a specifi-cally designed 3-layer protocol stack, compliant withthe reduced OSI model. These layers are the Applica-tion layer, the MAC layer and the Physical layer. Thephysical layer manages the wireless medium. TheMAC layer ensures a deterministic medium sharing,so that only one unit can speak at a time (no colli-sion), to provide known and bounded response times.It allows for either a single (unicast) or a group of tar-get DSUs (multicast) by using an individual or groupidentifier (Godary-Dejean et al., 2011).

These various tasks run on the µcontrollers undercontrol of a CMX real-time multi-tasking executive.

3 SIMULATION SETUP

The long term objective is the safe and efficient im-plementation of motion control using FES feedbackloops. As the analysis of such complex hybrid systemis out of scope of pure theoretic analysis, we need asoftware tool to help for the design, sizing, analysisand tuning of the components and of their interaction,w.r.t. specified performance and safety levels.

More precisely, we need that the system uses anopen source framework and run in real-time on stock

PCs. It must easily evolve from functional simula-tion to HIL to cover the different phases of softwaredevelopment. Hence, according to the design cur-rent phase, some simulated components might usequite simple models while others should be more de-tailed. For example, a preliminary analysis of the con-trol loops sampling rate do not need a very detailedphysical model of the communication medium (whichcould be added later on for refined tuning).

3.1 Simulation architecture

Processes and threads Based on previous expe-rience the simulation software is designed as a setof Unix processes and Posix threads running underLinux on stock multi-core PCs. The first version ofthe simulator is a pure software which does not inte-grate any piece of real hardware, and calculations areperformed with the usual floating point capabilities ofthe PC.

However, to mimic the real distributed architec-ture, it is split in 3 processes so that it can be run ona multi-core PC (see Figure 3), for example one coresimulates the controller and each DSU is simulated onanother core :

• the ”Controller” process executes the FES controllersrunning on the control chip under supervision ofa multitasks O.S.; it also simulates the networktransceiver through a simplified model of the commu-nication stack;

• the ”Medium” process is used to simulate the physicalfeatures of the wireless link;

• the ”DSU” process handles models of the Stim andAcquisition DSU fonctions in dedicated threads. Italso handles the computation thread running the nu-merical integrator.

Using the (non portable)pthread attr setaffinity np CPU allocation state-ment, each of these process can be run on a differentcore. In the future these processes might be easilyallocated on different chips connected by the realtransceivers to enhance the simulation frameworktowards an HIL simulation.

Communication and synchronization A simpli-fied version of the communication protocol stack isimplemented in dedicated communication threads.The data exchanged between the controller and theDSUs are encapsulated in packets whose structure issimilar to those handled by the real network. In thesimulation, the packets are handled in two steps byUDP Unix sockets (Figure 4) :

• the packet is first transmitted using a point-to-pointUDP socket from the communication manager of thecontroller (or of the Pod) to the medium process re-ceiving thread;

• in the current version, the model of the medium justadd a constant delay to the packet transmission, anddata loss are handled by a simple random process.

• after being delayed, the packet is sent to all re-ceivers using a diffusion UDP socket (using the”so broadcast” parameter of UDP sockets). This fea-ture allows for sending data with a unique time stampto groups of Pods.

broadcast socket

point−to−point

DSU

Stimulation

DSU

Acquisition

Medium Control Device

sockets

Figure 4: Communication sockets

Timing values All threads in the simulation can beunlocked either by signals from clocks (e.g. for pe-riodic control tasks), by blocking waiting messageson an incoming socket or by semaphores signaled bycommunication requests sent by another thread. Thethreads are scheduled by the Linux kernel, set with theRT FIFO real-time mode, according to their prioritiesand locks. However stock Linux kernels sometimesprovides inconsistent behaviors, although the simula-tion works coherently with kernel compiled with real-time options, i.e. with high resolution timers and lowlatency preemption model under superuser privilege.

The transmission delays on the wireless links aresimulated by pure delays in the medium component.Probes using functions of the Posix real-time librarycan be inserted anywhere in the threads to measure,e.g., communication delays between two tags or torecord missed deadlines.

Solver triggering The numerical integration threadis a special case. Indeed this thread is not the ghost ofa real time task existing in the real implementation.

In real life the state of the physical process nat-urally evolves in parallel with the numerical execu-tion of the control and communication systems. Thetime scales of the continuous and numerical entitiesare meshed at some meeting instants at the occurenceof different kind of events :

Medium Process

Controler Process

PO

D P

roce

ss

Acquisition DSUthreads

Stimulation DSUthreads

Spy

C−>P

P−>CPatientmodelthread

UDP sockets

Unix process

Posix thread

Sensorprocessing

Comm.manager

functionsFES

protocol stack

MAC layer

Motionsensing

Biomechanical models

Muscles Jointskinematics

Comm.manager

Stimulationprocessing

FESfunctions

Probes

MAC layerprotocol stack

controlFES

laws

Messagesscheduler

Taskscheduler

Schedulingcontroler

Controlsupervision

Disturbances

medium

medium

model

model

protocol stackMAC layer

Communication

Communication

Numerical integrator

StimulationCurrents

Figure 3: Simulation architecture

• when a control system requests an updated sensorvalue;

• when a control system sends a new control value;

• when the simulation system needs a snapshot of thecontrolled process and of its environment;

• when the continuous process raises an event (e.g., viaan asynchronous sensor or crossing a model disconti-nuity).

The integration thread is triggered by any of theseevents. At each triggering event the integrator is firstrun from the last interruption until the current timeto update the state of the continuous process, then itdelivers the new state or accept new control inputs.

In practice, for the current set-up the simulationruns very comfortably faster than real-time on a mod-ern laptop (e.g., 100 times faster on a i7 CPU). Com-putation times and overruns can be easily checked bysoftware probes and reported to provide informationand warnings about the timing behavior and qualityof the simulation.

Note that, if the numerical integration cost be-comes prohibitive to run the simulation on a singleCPU w.r.t. real-time, the model can be split overseveral CPUs. Slackened synchronization betweendecoupled sub-models allows for significant speed-ups while keeping the integration errors under control(Ben Khaled-El Feki, 2014).

3.2 Continuous plant and numericalintegration

Continuous model The model of the continuousplant is usually given as a set of ordinary differ-ential equations (ODEs), differential algebraic equa-tions (DAEs) or partial derivative equations (PDEs),where the continuous (physical) time is an indepen-dent variable.

Building high fidelity system-level models ofcomplex systems, like mechatronic systems and evenmore systems involving livings, is a challenging duty.One problem is the diversity of modeling and simula-tion environments used by the various involved multi-disciplinary teams. Particular environments are pre-ferred for a specific use due to distinctive strengths(modeling language, libraries, solvers, cost, etc.), forexample the automatic derivation of the kinematic anddynamic equations of a multi-body system can be adistinctive feature.

Perfect modeling must be forgot. A model is al-ways an approximation of reality, both in its structureand in its parameters values. A problem for simula-tion, due to both model complexity and fast systemdynamics, is the prohibitive CPU times which can beobserved when high-fidelity models are run, thereforepreventing any real-time simulation. It is needed tofind a satisfactory trade-off between the simulationspeed and precision. A good model captures the es-sential behavior of the physical plant. Therefore, ac-cording to its user and objectives (e.g. an control sci-entist dealing with gains tuning, or a therapist willingto choose the stimulation frequency), different modelsof a given physical plant can be chosen.

Building a high-fidelity model needs preciseknowledge about the plant itself, here coming frombio-mechanics and neuroscience. The simulationsetup has been tested using the model of a human kneecontrolled by electrostimulation, a topic for whichthe Demar team developed a strong experience overyears. The main structure and parameters of themodel were mainly taken from (Benoussaad, 2009)for artificially excited muscles and from (McFaull andLamontagne, 1998) for the joint non-linear parame-ters.

Figure 5: Simplified knee kinematics

It is given as a small set of non-linear ODEs. Acontractile element (CE) dynamics is given by (Be-noussaad, 2009) :

Kc = −Kc|u|+ αKcmFlc(εc)|u|+ −Kc|εc| (1)

Fc = −Fc|u|+ αFcmFlc(εc)|u|+ − Fc|εc|+ Lc0Kc εc

where Kc and Fc are the stiffness and force devel-oped by the CE,Kcm and Fcm their maximum values,u and |u|+ chemical control variables coming from anactivation model, Flc(εc) the force/length relation ofthe CE, α the recruitment rate, and εc the contractionspeed of the CE.

The numerical integration of the later variable,given by

εc =ksL0ε+ Fc|u| − αFcmFlc(εc)|u|+

ksLc0 +KcLc0 − svFc(2)

needs a special attention as it involves the sign func-tion :

sv = sign(ksL0ε+ Fc|u| − αFcmFlc(εc)|u|+) (3)

Correctly handling this non-linearity needs an inte-grator enabled with a root-finding capability to pro-vide a fast and stable integration (see next section).

The two muscles involved in the knee flex-ion/extension, i.e. the quadriceps and hamstring, havesimilar models with different parameters and are ac-tuated by their own electrode. For our purpose, theknee joint is considered to be a perfect cylindricald.o.f. leading the usual equation of motion (Figure5

Θ =1

J(F2r2−F1r1−MgLog sin(Θ−Θ0)−BΘ+Te)

(4)

where Fi is the force induced by muscle i, B a non-linear damping and Te a passive elasticity torque.

Finally the model is encoded for integration as aset of 8 first order non-linear ODEs, three for eachmuscle and two for the joint dynamics.

Numerical integration The non-linear model ofa robot (or of most mechatronic systems involv-ing mechanical and electrical components) is usu-ally described by a set of ordinary differential equa-tions (ODEs) (or sometimes by a set of differential-algebraic equations (DAEs)) (Arnold et al., 2011).There already exist a broad family of algorithmsand corresponding software used to numerically solvesuch a set of ODEs.

With most existing integration packages, the end-user’s choices can be :

• set the value of a fixed-step integrator, leading to apredictable number of steps and computing time, butthe precision cannot be controlled and may lead tofalse results;

• specify the precision of the integration and delegatethe adaptive step management to the integrator, but inthat case the computing time cannot be predicted.

With an adaptive step integration method the con-trolled variable is the precision (or more exactly theestimate of the integration error). The open-sourceLSODAR package we have elected (Hindmarsh andPetzold, 1995) for this experiment uses a variablestep, variable number of steps integration algorithm.It is suitable for both stiff and non-stiff systems,and automatically selects between non-stiff and stiffmethods.

It is associated with a root-finding stopping cri-teria : when it detects a sign change in a particularfunction of the states (root detection), it runs a root-finding algorithm to find the exact time point at whichthe sign change occurred. The later feature is essentialto correctly integrate continuous plants with disconti-nuities, i.e. where the values of some states or pa-rameters, or even the model structure itself, suddenlychange where a given threshold is crossed (zero-crossing). It is for example the case for models ofimpacts, dry friction, and here for muscles contrac-tion (due to eq. (3)).

3.3 Control design

To generate movement for a joint, two muscles (an ag-onist and antagonist pair) must be stimulated. For theknee these two muscles are the quadriceps on the an-terior face (extension actuator) and the hamstring onthe posterior face (flexion actuator). In a first simple

control approach, the two muscles will be actuated inmutual exclusion according to the flexion and exten-sion phases of the movements, see Figure 6.

sat

sat

+−

angle

desired angleAntagonist muscle

Agonist muscle

Velocity

Position

Sensors

−1

PID

Figure 6: Control scheme

As usual, the first control algorithm to be consid-ered is a PID, a simple controller able to control manysingle input/single output (SISO) systems through avery simple modeling and tuning effort. Here it isdiscretized using a backward difference method (withTs the sampling period), yielding

uk = uk−1+K[ek−ek−1]+K · Ts

Ti

ek+K · Td

Ts[ek−2ek−1+ek−2]

(5)

4 PRELIMINARY EXPERIMENTS

In this preliminary version, the simulation system is a”Model-in-the-loop” simulation, where all the objectsare modeled, the physical plant state is evaluated bya numerical integrator and the control functions areencoded in plain C and executed using the floatingpoint features of a modern laptop. Anyway, it goesbeyond usual simulations with Scicos or Simulink asit handles quite precisely some implementation andscheduling related features such as tasks and eventsinterleaving, or computation and communication in-duces delays. Hence it can be already used to explorethe possible performance of FES feedback loops ac-cording to various scheduling parameters.

4.1 PID tuning and sampling rate

A simple PID controller has been setup for prelimi-nary tests of the simulation platform. Indeed a PIDwith fixed gains would perform uniformly for a linearsystem, which is not the case for the knee joint, whereeven a simplified model shows several sources of non-linearities. Hence the PID has been empirically tunedto provide acceptable responses for the whole reach-able motion amplitude.

Initially simulations were made with a 40 msecsampling rate, i.e. the one used by the marketedversion of the Phenix system. In fact, the end-to-end communication time (including control tasks ex-ecution and messages transmission and acknowledg-ment) between the controller and the DSUs is lowerthan 5 msecs (Figure 7b). Lowering the samplingrate to 5 msecs shows an improved performance of

Kn

ee a

ng

le (

rad

)

Time (sec)

sampling 40 msec

sampling 5 msec

sampling 2 msec

Time (sec)

En

d−

to−

en

d c

om

mu

nic

ati

on

tim

e (

mse

c)

Figure 7: a) Knee position – b) End-to-end delay

the joint controller (Figure 7a). Note that, as the net-work model is included in the simulation, samplingrates smaller than 5 msec induce data loss and poorcontrol performance. Hence the real-time simulatorallows for a fine tuning of control gains w.r.t the exe-cution resources at hand, far beyond the usually rec-ommended sampling rates between 3 and 10 times theShannon sampling rate.

4.2 Network protocol enhancements

The current communication threads do not implementthe production code of the full protocol stacks, butonly its main features. The simulated transceiver en-capsulates data in frames according to the STIMAPprotocol (Godary-Dejean et al., 2011) used to con-nect the controller and DSUs. The frames are furthersent/received over UDP sockets. Several versions ofthe communication frames were tested, and their im-pact on the feedback control performance and on thecommunication load was evaluated (see Figure 8).Single receiver (Production version) (in red) Adata frame issued by the controller contains theaddress of a single receiving/answering DSU, and thepayload only contains the stimulation parameters forthis particular DSU (red plot, 6 ms average delay);

Single receiver with optimization (in green) Theoptimization uses the mutual exclusion between theagonist/antagonist muscle and send only frames to theactive muscle in the current motion phase. Howeversome safety messages are sent (at a slower rate com-pared with control messages) to keep alive the DSUof the passive muscle, leading to irregular timing pat-terns (green plot, 75% of samples near 3 ms and 25%of samples near 6 ms);

Multiple receivers (in blue) Messages are addressedto a group of DSUs, with a payload made of the con-catenation of the individual stimulation parameters:the frame is longer than a single receiver frame butshorter than the concatenation of the single receiverframes, thus saving communication bandwidth (blueplot, 4.5 ms average delay);The observation of the simulations traces show thatusing optimized versions of the protocol may signif-icantly save bandwidth and decrease the end-to-end

communication delay between sensing and actuation,leading to improvements in the performance and ro-bustness of the FES control law. Such results are use-ful to motivate (even costly) modifications of the pro-duction hardware and software.

Time (sec)

En

d−

to−

end

co

mm

un

ica

tio

n t

ime

{mse

c)

Multiple receivers

Single receiver

Single receiver, optimized

Figure 8: End-to-end communication time

5 SUMMARY AND FUTUREDEVELOPMENTS

Realistic simulations are effective tools to design andtune complex systems whose analysis cannot be pro-vided only by theory. Several simulation steps can beexplored, from simple functional analysis until HIL,to design, test, tune and validate both the single com-ponents of the system and their interactions in a dis-tributed architecture. Simulations are precious, asthey allow for non-destructive trials, anticipation forfuture technological development at lowered cost andexploration of a system behavior until its limits withno danger.

For the particular setup presented here, it is ex-pected that such simulator will provide inputs in twomain directions. Firstly it allows for preliminary test-ing new FES protocols prior to experiments with pa-tients, and may help for writing the ethical protocolsneeded for any experiments involving livings. Sec-ondly it can be used for preliminary evaluation of newtechnologies or implementations, without costly re-working of existing electronic chips or certified com-ponents.

The simulation software is open, so that enhance-ments w.r.t. to the original release can be added uponrequest of various designers and to fulfill various de-sign objectives. For example, some future enhance-ments may be :

• Development and test of feedback schedulers for au-tomatic adaptation of the CPU and communicationloads in complex FES applications such as walking;

• Refinement of the physical communication mediummodel, so that the wireless link fading due to ofobstacles can be evaluated, leading to the designof a feedback control of the emission power of thetransceivers;

• Integration of others continuous plants. The dynamicmodel of a human hand with 23 d.o.f. is currentlydeveloped to provide a complex case for numericalintegration and control coordination;

• Progressive integration of the production code ofsome components, e.g., the full protocol stack, and ofsome real components, e.g., wireless transceivers be-tween simulated nodes, to incrementally provide HILsimulation setups.

As for any simulation, a crucial problem is the cal-ibration of the models and the validation of the over-all simulation system. This process needs, at somepoints, comparisons with observations of real experi-ments, which is costly but necessary. Anyway this isneeded to gain the confidence of the various users ofthe system.

REFERENCES

Arnold, M., Burgermeister, B., Fuhrer, C., Hippmann, G.,and Rill, G. (2011). Numerical methods in vehiclesystem dynamics: State of the art and current devel-opments. Vehicle System Dynamics.

Ben Khaled-El Feki, A. (2014). Distributed real-time sim-ulation of numerical models: application to power-train. PhD thesis, EEATS, Automatique Productique,Universite de Grenoble. http://www.theses.fr/2014GRENT033.

Benoussaad, M. (2009). Protocole d’identification sousFES et synthese des sequences de stimulation chez leblesse medullaire. PhD thesis, Univ. Montpellier II.

Godary-Dejean, K., Andreu, D., and Romain, R. (2011).Temporal bounds verification of the STIMAP proto-col. In RTNS 2011 : 19th International Conferenceon Real-Time and Network Systems, page 10, Nantes,France.

Hindmarsh, A. C. and Petzold, L. R. (1995). Algo-rithms and software for Ordinary Differential Equa-tions and Differential-Algebraic Equations, part II:Higher-order methods and software packages. Com-put. Phys., 9:148–155.

McFaull, S. R. and Lamontagne, M. (1998). In vivomeasurement of the passive viscoelastic properties ofthe human knee joint. Human Movement Science,17(2):139 – 165.

Zhang, Q., Hayashibe, M., and Azevedo Coste, C.(2013). Evoked Electromyography-Based Closed-Loop Torque Control in Functional Electrical Stimula-tion. IEEE Transactions on Biomedical Engineering,60(8):2299–2307.