http://nrg.mbi.ufl.edu1

Towards Real-Time Distributed Signal Modeling for Brain-Machine Interfaces

Justin C. Sanchez, Ph.D.

Neuroprosthetics Research Group (NRG)http://nrg.mbi.ufl.edu

jcs77@ufl.edu

On behalf of the DDDBMI PI team:José Fortes, Renato Figueiredo, Linda Hermer-

Vasquez, and José Principe

http://nrg.mbi.ufl.edu2

Enabling Neurotechnologies for Overcoming Neurological Disorders

Develop direct neural interfaces to bypass injury. Communicate and control (closed-loop, real-time) via the interface.Spinal Cord InjuryMovement DisabilitiesStroke

Leuthardt

http://nrg.mbi.ufl.edu

Goals of this Project

ImmediateA test bed for real-time, closed-loop BMI modeling and experimentation Advance behavioral brain modeling, BMI experimentation, algorithms and Grid-computing.

Long-TermCyberworkstation for real-time neurophysiologicalexperiments.A general purpose platform for modeling and experimentation in neurophysiology. Impact the speed and complexity of new ideas that can be tested in the neurophysiology laboratory

http://nrg.mbi.ufl.edu

Integrating the Multidisciplinary Team

Here we present the critical architecture and infrastructure to support theory of brain information processing and motor control function.

http://nrg.mbi.ufl.edu

BMI Portal – Provides the Bridge

Reservation of resourcesFor online experiments

Access to data setsFor replay and analysis of

experiments

Specification of modelsFor use in either offline or

online experiments

Access to computational tools

For analysis, simulation, visualization ….

http://nrg.mbi.ufl.edu

Middleware Architecture

http://nrg.mbi.ufl.edu

BMI Workstation Components

Data ExchangesRat & Pentusa

PentusaData.exe GlobalBMI LocalBMI

Bin numbers, masks, thresholds

InputDatInputDat

OutputDat

Robot Command

Robotic Arm

NRG ACIS

http://nrg.mbi.ufl.edu

Data Flow and Scheduling

http://nrg.mbi.ufl.edu

Multiple paired models and the responsibility predictor for the DDDAS based BMI

http://nrg.mbi.ufl.edu

General RequirementsNumber of pairs of “internal” models

10s – 100s for simple tasks (e.g. press lever)1000s (?) for complex tasks

Types of “internal” modelsLinear (filters): Wiener, NLMS, PVA, …Nonlinear (neural nets): TDNN, RMLP, RNN, NMCLMState-based: Kalman filters, Bayesian classifiers, HMMs, RLBMI

Complexity of modelsO(n), O(n2), O(mn2), O(n3), …for n neurons, m models

http://nrg.mbi.ufl.edu

One Realization of the Computational Structure

Online – real-time (100ms - hard deadline)Offline – recreation of experiments from data in storage

module module module…

data

…

training/gating

…

+10 – 100 msHard (neural

sampling)

10 – 500 msSoft (model update)

Reinforcement Learning

http://nrg.mbi.ufl.edu

Experimental Paradigm

-3 -2 -1 0 1 2 3

01

23

0

1

2

3

IncorrectTarget

CorrectTarget

StartingPosition

Match LEDs

Grid-space

Match LEDs

Rat’s Perspective

Water Reward

Map workspace to grid

Rat

Robot Arm

Left Lever Right Lever

http://nrg.mbi.ufl.edu

Feasibility of Architecture –Offline Performance Evaluation

7.3%34.3%43.8%MLP**3

11%23.1%43.8%MLP*2

Xx31.2%SLP1

10.8%68.1%81.3%MLP**3

10%61.9%81.3%MLP*2

xx93.7%SLP1

VarianceMeanMaxStructureDimension

*H = 2 **H = 3SLP – Single Layer Perceptron

MLP – Multilayer Perceptron

Surrogate

http://nrg.mbi.ufl.edu

Real-Time Experimentation across Campus

ACISlab

NRGlab

http://nrg.mbi.ufl.edu

Closed-loop BMI Timing Result

No deadline missedTotal closed loop time allows more modules to be addedVirtualization can improve resource utilization

17.5

62

3.59

4471 6.50

2927

4.27

0123

0.04

8975

1

24.1

83

2.40

0925

0

5

10

15

20

25

30

Computation (µs) Training (ms) Acquisition (ms) Transfer (ms) Robot (ms)

Physica ResourceVirtual Resource

http://nrg.mbi.ufl.edu

Real-Time, Closed-Loop Robot under Brain Control using DDDBMI Architecture

Non-functional levers

Functional levers

Robot workspace in rat visual field of view.

BLUE – Robot

GREEN - Lever

Top-view of the rat behavioral cage.

http://nrg.mbi.ufl.edu

Milestones MetNeural recordings time synchronized with behavior from multiple arrays. Assessment of (open loop and closed loop) cortical (M1) contribution to the lever pressing task. Statistical comparison of training and testing MSE for the proposed algorithms (80-90% accuracy). Benchmark of virtual application performance. All met 100ms harddeadline. Implemented and tested gating function training (Reinforcement Learning) Evaluated parameters for symbiotic training using VM cluster in real-time closed loop operation Implemented spike train learning for the forward modelInitiated study of adaptive virtual application rescheduling, VMreservation middleware with animal dataSpecified bottlenecks in computing architecture for real-time experiments.

http://nrg.mbi.ufl.edu

Student Support through the DDDASBMI

18

Prapaporn Rattanatamrong -ECE

Jack DiGiovanna -BME

BabakMahmoudi -BME

ShalomDarmanjian -ECE

Ming Zhao -ECE

This work is supported by NSF project No. CNS-0540304