Decoding task states from distributed local field potential recordings
By
Nicole Renee Provenza
Sc.B., Brown University, 2015
Thesis
Submitted in partial fulfillment of the requirements for the Degree of Master of Science in the
Center of Biomedical Engineering at Brown University
PROVIDENCE, RHODE ISLAND
MAY 2017
ii
This thesis by Nicole Renee Provenza is accepted in its present form by the Center of
Biomedical Engineering as satisfying the thesis requirements for the degree of Master of
Science.
Date: __________ Signature: ______________________________
Dr. David Borton, Advisor
Date: __________ Signature: ______________________________
Dr. Leigh Hochberg, Reader
Date: __________ Signature: ______________________________
Dr. Wilson Truccolo, Reader
Date: __________ Signature: ______________________________
Dr. Alik Widge, Reader
Approved by the Graduate Council
Date: __________ Signature: ______________________________
Dr. Andrew G. Campbell, Dean of the Graduate School
iii
VITA Nicole Renee Provenza was born on August 21st, 1993 to Sherri and Clint Provenza in Columbia,
MD. She attended Brown University for her undergraduate studies where she was a four-year
athlete and captain of the varsity swimming and diving team. She graduated in 2015 with a
Bachelor of Science in biomedical engineering with honors. Her honors undergraduate thesis
work is titled, ‘Mapping Parkinson’s disease progression as observed by the motor cortex’, and
was written under the advisement of Dr. David Borton. Nicole received the Draper Fellowship
for her master’s research in the Borton Laboratory at Brown University. During her master’s
program, she conducted research as part of the Defense Advanced Research Projects Agency
(DARPA) program, Systems-Based Neurotechnology for Emerging Therapies (SUBNETS), in
collaboration with Massachusetts General Hospital (MGH), Draper, and Brown University. After
graduation, she will enter the doctoral program in biomedical engineering at Brown University to
continue her research on distinguishing brain states relevant to the closed-loop treatment of
neuropsychiatric illness.
iv
PREFACE This work was conducted at Brown University in collaboration with Massachusetts General
Hospital (MGH - Boston, MA) and Draper (Cambridge, MA), as part of the Defense Advanced
Research Projects Agency (DARPA) program, Systems-Based Neurotechnology for Emerging
Therapies (SUBNETS) led by Principal Investigators Dr. Alik Widge, Dr. Emad Eskandar, Dr.
Sydney Cash, and Dr. Darin Dougherty. The MGH team conducted all clinical recordings and
performed data preprocessing steps including filtering, referencing, channel selection, and
electrode localization. The corresponding methods sections were generously provided by the
MGH team. Individual contributions are outlined below.
Human physiology was conducted by Dr. Sydney Cash, Britni Crocker, Kara Farnes, Dr. Todd
Herrington, Erica Johnson, Dr. Shaun Patel, and Nicole Rivilis. Animal lab work was conducted
by Dr. Angelique Paulk, Helen Deng,Madeleine Robertson. Neuro-Imaging was conducted by
Afsana Afzal, Dr. Thilo Deckersbach, Dr. Kristen Ellard, Anna Gilmour, Dr. Matti Hamalainen,
Dr. Roan LaPlante, Dr. Noam Peled, Dr. Steve Stufflebeam, and Sam Zorowitz. Analysis and
modeling was conducted by Dr. Ishita Basu, Dr. Emery Brown, Dr. Uri Eden, Dr. Mark Kramer,
Dr. Angelique Paulk, Nicole Provenza, Dr. Patrick Stokes, and Dr. Ali Yousefi. The program
was managed by Dr. Karl Helmer, Mark Helfant, Matthew Thombs, and Graham Warner.
Hardware was developed by Dr. Chris Salthouse, Jesse Wheeler, and Caroline Bjune.
Patents related to this work are currently under review at Massachusetts General Hospital and
Brown University.
This research was funded by DARPA under Cooperative Agreement Number W911NF-14-2-
0045 issued by the Army Research Office contracting office in support of DARPA’s SUBNETS
program. The views, opinions, and/or findings expressed are those of the authorand should not
be interpreted as representing the official views or policies of the Department of Defense or the
U.S. Government.
This work was sponsored by DARPA Biological Technologies Office under the auspices of Dr.
Doug Weber through the DARPA Contracts Management Office Grant/Contract No.
D15AP00112.
v
ACKNOWLEDGEMENTS Foremost, I wish to thank my advisor, Dr. David Borton, for his outstanding mentorship,
patience, enthusiasm, and encouragement. It has been a privilege and a joy to be a member of his
lab over the past three years. Next, I wish to thank my advisor-away-from-Brown, Dr. Alik
Widge, who generously welcomed me onto the MGH team, and whose expertise and enthusiasm
continues to guide my research. I would like to express thanks to Dr. Angelique Paulk for single-
handedly organizing all the data that made my thesis possible, and answering my endless
questions on all things data related and beyond. I owe a special thanks to Draper for their
encouragement and support of my continued academic pursuits, and believing in my research
enough to fund me twice. I’d like to thank my Draper supervisor, Dr. Marc McConley, who
continues to make sure I have everything I need to be successful. I would be remiss if I did not
mention my lab-mates at Brown University, who are a perpetual source of inspiration, humor,
and fun. Lastly, I am incredibly thankful for my parents, Sherri and Clint, sister, Christina, and
boyfriend, Robert, for celebrating my successes and lamenting my frustrations, and for their
invaluable love and support.
vi
TABLE OF CONTENTS
VITA .......................................................................................................................................................................... III
PREFACE ................................................................................................................................................................. IV
ACKNOWLEDGEMENTS ....................................................................................................................................... V
TABLE OF CONTENTS ......................................................................................................................................... VI
LIST OF FIGURES ................................................................................................................................................. VII
LIST OF TABLES ................................................................................................................................................. VIII
ABSTRACT .............................................................................................................................................................. IX
INTRODUCTION ....................................................................................................................................................... 1
Closed-loop deep brain stimulation for neuropsychiatric illness .............................................................................. 2 Observing functional connectivity via fMRI ............................................................................................................. 5 Observing functional connectivity via local field potential recordings ..................................................................... 9 Considerations for closed-loop algorithm development for implantable systems ................................................... 12 Specific Aims of the Study ...................................................................................................................................... 14
METHODS ................................................................................................................................................................. 16
Behavioral tasks ...................................................................................................................................................... 16 Patients .................................................................................................................................................................... 17 Electrode localization .............................................................................................................................................. 18 Invasive electroencephalography recordings .......................................................................................................... 20 Data preprocessing .................................................................................................................................................. 20 Feature extraction .................................................................................................................................................... 23 Classification ........................................................................................................................................................... 24 Feature selection ...................................................................................................................................................... 27 Evaluation of decoder stability ................................................................................................................................ 28
RESULTS ................................................................................................................................................................... 29
Functional inference distinguishes task engagement ............................................................................................... 29 Classification accuracy depends on a subset of optimal features per patient .......................................................... 32 Optimal feature sets shift over time ........................................................................................................................ 35 Training on temporally separated data shows potential for decoding stability ....................................................... 37
DISCUSSION ............................................................................................................................................................. 39
Classification performance ...................................................................................................................................... 39 Feature selection ...................................................................................................................................................... 42 Decoder stability ..................................................................................................................................................... 43 Applicability ............................................................................................................................................................ 45
CONCLUSIONS ........................................................................................................................................................ 47
REFERENCES .......................................................................................................................................................... 49
APPENDIX I: SUPPLEMENTAL METHODS ...................................................................................................... 54
APPENDIX II: INVENTION DISCLOSURE ........................................................................................................ 56
Essence of the invention .......................................................................................................................................... 56 Novelty and major advantages ................................................................................................................................ 56
vii
LIST OF FIGURES Figure 1: Disability-adjusted life years (DALYs) for mental illness .............................................. 1
Figure 2: Illustration of closed-loop DBS for PTSD. ..................................................................... 3
Figure 3: Illustrations of a transdiagnostic framework ................................................................... 4
Figure 4: Single subject fMRI response during MSIT.................................................................... 7
Figure 5: Functional connectivity illustration ................................................................................. 9
Figure 6: Canonical correlation is a robust metric for functional connectivity ............................ 11
Figure 7: Bipolar referencing. ....................................................................................................... 12
Figure 8: Schematics of the MSIT and ECR tasks ....................................................................... 17
Figure 9: Example electrode localization ..................................................................................... 20
Figure 10: High-pass filter removes frequency components less than 2 Hz................................. 22
Figure 11: Task and non-task dataset sizes. .................................................................................. 25
Figure 12: Classification pipeline for the identification of task-like brain states ......................... 26
Figure 13: Feature selection using SVM-Based criteria ............................................................... 27
Figure 14: Classification performance for MSIT vs. non-task classification ............................... 30
Figure 15: Classification performance for ECR tsak vs. nono-task classification ....................... 31
Figure 16: Window size effects on classification of CCA features .............................................. 32
Figure 17: Feature pruning reveals significant features for MSIT ............................................... 34
Figure 18: Feature pruning reveals significant features for ECR task .......................................... 34
Figure 19: Example feature sets required to reach 80% accuracy for MSIT ................................ 35
Figure 20: Example feature sets required to reach 80% accuracy for ECR task .......................... 35
Figure 21: Feature pruning reveals significant features across multiple recording sessions ........ 36
Figure 22: Overlap in significant feature sets over multiple recording sessions .......................... 37
Figure 23: Classification performance across multiple recording sessions .................................. 38
viii
LIST OF TABLES Table 1: iEEG recording summary ............................................................................................... 21
Table 2: Bipolar electrode localization summary for the MSIT. .................................................. 54
Table 3: Bipolar electrode localization summary for the ECR task ............................................. 55
ix
ABSTRACT
The brain is a distributed network that operates at small and large spatial and temporal scales to
meet the needs of the task at hand. Certain tasks generate conflict by requiring attention to
relevant stimuli among distractions. Engagement in conflict-based tasks elicits network activity
that differentiates these tasks from other behaviors. Brain states associated with conflict in
experimental behavioral contexts may be an important marker of dysfunction related to
neuropsychiatric illness, yet there are no documented decoders that predict engagement in
conflict-based tasks. In addition, there is evidence that precisely timed deep brain stimulation
(DBS) could restore normal task behavior in neuropsychiatric patients; however, these findings
are constrained to features elicited during experimental behavioral tasks, and do not yet
generalize to real world situations. In the present study, the investigators have developed a
decoding strategy to accurately predict task engagement by harnessing canonical correlation
analysis (CCA), a measure of functional connectivity between regions, in tandem with a support
vector machine (SVM) classifier. Cortical and sub-cortical invasive local field potential
recordings were collected from patients engaged in one of two Stroop-like tasks: the Multi-
Source Interference Task (MSIT) or the Emotional Conflict Resolution task (ECR). Canonical
correlation coefficients were extracted by performing singular value decompositions on windows
of data across all channels within each region and transforming the resulting singular vectors in a
way that maximally correlates activity between each region pair. These CCA features were used
as inputs to an SVM classifier to differentiate functional connectivity between task engagement
and free behavior. A mean classification accuracy of 95.3% (MSIT med: 96.7 ± 2.3%, ECR med:
95.0 ± 3.2%) was achieved for both tasks across 14 patients and a subset of features per patient
required to maintain high accuracy were isolated (MSIT: med 3 ± 3.4 region pairs, ECR: med 4
± 6.6 region pairs). The reduced feature set can be used to reduce the computational complexities
of the algorithm moving forward, which points to potential tractability for driving a simple
algorithm to detect task engagement on existing systems. While decoder stability was not
achieved over two recordings, classification accuracy improved when trained on both recordings.
This increase in accuracy suggests that training the classifier on many temporally separated
datasets could improve its stability. Detection of task-associated brain states could ultimately be
a closed-loop strategy for delivering real-time therapy to patients with neuropsychiatric illness.
1
INTRODUCTION
Mental illness is the leading cause of disability worldwide (Figure 1).1 Despite its prevalence,
mental illness has historically not been a global health priority. Only in recent generations have
clinicians and researchers begun to approach mental illnesses as brain circuit disorders.2,3
Despite the efforts of researchers and clinicians over the past several decades, current treatment
options still do not meet patient needs. For example, antidepressant medications commonly
prescribed for depressive disorders are only 30%-40% efficacious at best.4,5 Psychotherapy is
more effective in some cases, however, well-trained therapists are expensive and often
inaccessible for most patients.6
Figure 1: Adapted from Whiteford et al. (2013). Disability-adjusted life years (DALYs) for mental and substance use disorders in
2010, shown by age.
Deep brain stimulation (DBS) has recently emerged as a candidate for improving treatment
outcomes for patients suffering with neuropsychiatric illness. We have known for decades that
targeted stereotactic lesions can be effective for refractory obsessive compulsive disorder (OCD),
and depression.7–11 DBS has long been known as a means for the rapid and reversible modulation
2
of neural function, and may reproduce the lesioning effects in a reversible manner. While DBS
showed early promise for treatment of refractory MDD and OCD12–14, results were not replicable
during randomized controlled clinical trials.15,16 DBS is available for OCD patients under a
humanitarian device exemption17, but is currently not clinically approved for any other
neuropsychiatric disorders. The disappointments of past clinical trials could perhaps be explained
by the open loop nature of traditional DBS that fails to address the dynamic nature of
neuropsychiatric illness.6 In the following section, we will discuss potential avenues for
addressing this limitations.
Closed-loop deep brain stimulation for neuropsychiatric illness
Truly harnessing the full potential of DBS will require the development of systems that are
capable of adjusting stimulation parameters based on detected neural signatures or mental states
related to patient symptoms.18 An adaptive treatment strategy is ideal for neuropsychiatric
illness, as symptoms of neuropsychiatric illness are not static. Instead, symptoms flare and
resolve over a wide range of temporal scales depending on the illness, ranging from minutes to
days. Traditional open loop DBS delivers constant stimulation without regard to patient
symptoms, and adjustments in stimulation parameters are only possible during infrequent clinical
visits. A ‘closed-loop’ DBS system would deliver stimulation to modulate patient brain activity
in real-time only when needed, and allow for remote adjustments of parameters.
As an example, we will discuss how closed-loop DBS could be used to treat patients suffering
with post-traumatic stress disorder (PTSD). Our choice of PTSD is motivated by its prevalence
among soldiers and veterans after returning from combat in Iraq and Afghanistan (approximately
20% of soldiers exposed to combat meet the criteria for PTSD19,20). During closed-loop DBS for
PTSD, implanted electrodes would continuously monitor brain activity. Collected neural activity
3
would then be sent to an implanted device to decode brain states via feature estimation and
classification. When a brain state related to psychological distress is detected, specific
stimulation commands would be sent back to electrodes (Figure 2). Stimulation would ideally
nudge the network out of the distressed state to relieve symptoms.
Figure 2: Illustration of closed-loop DBS for PTSD.
One essential component to enable a closed-loop DBS system is the identification of the neural
signatures of neuropsychiatric illness. Despite progress in recent years, investigators have not yet
identified definitive, stable biomarkers for neuropsychiatric illness. A major challenge for
identifying disease-specific biomarkers is that phenotypes overlap between multiple diagnoses,
and phenotypes are not homogenous for single diagnoses. One proposition toward overcoming
this hurdle is identifying biomarkers that pertain to phenotype, rather than diagnosis. For
example, Figure 3A illustrates how three patients with different diagnoses (TBI: traumatic brain
injury, GAD: generalized anxiety disorder, and MDD: major depressive disorder) share an
4
overlapping phenotype, in that they all demonstrate cognitive rigidity and are unable to adapt in
changing environments. Conversely, Figure 3B illustrates how three patients with the same
diagnosis, MDD, do not share overlapping phenotypes. One patient exhibits emotional lability,
while another patient is emotionally flat.6
Figure 3: Adapted from Widge et al 2017. Illustrations of a transdiagnostic framework dependent on phenotypic symptoms
relating to cognitive flexibility, emotional lability, and approach and avoidance to stimuli.
Before the realization of closed-loop DBS as a clinically relevant therapy for patients, it will be
essential to identify biomarkers of specific phenotypes exhibited by patients suffering with
neuropsychiatric illness. Much of what we know about potential biomarkers of neuropsychiatric
illness has been derived from fMRI paradigms designed to expose specific differences in
behavior and regional connectivity between healthy participants and neuropsychiatric patients.
However, current closed-loop platforms rely on local field potential (LFP) activity rather than
fMRI activity. fMRI is not conducive to delivering closed-loop therapy for many reasons; most
obviously, it is not portable and has poor temporal resolution. In addition, robust LFP activity
can be recorded from electrodes long after they are implanted, and LFPs can be recorded from
electrodes during ongoing DBS.21 While identifying biomarkers rooted in LFP recordings is the
goal, the wealth of knowledge derived from fMRI study findings offer a starting point that
should guide future invasive analysis.
5
Observing functional connectivity via fMRI
The brain consists of a network of regions that perform distinct, separable functions. The flow of
information within the network is coordinated by modulating the strength, pattern, and frequency
of neural activity. Specific patterns of synchrony between individual neurons, populations of
neurons, or entire brain regions are associated with complex behaviors, such as attention and
conflict resolution. In other words, brain regions are functionally connected. Observing and
quantifying functional connectivity between regions of interest (ROIs) during different tasks is a
classic strategy for gaining insight on the link between neural activity and behavior.
Observing functional connectivity during task-based imaging studies has greatly contributed to
relating brain states to abnormal behavior observed during neuropsychiatric illness. In particular,
abnormalities in the anterior cingulate cortex (ACC) have been observed in the pathophysiology
of neuropsychiatric illness, including attention deficit disorder (ADD)22,23, anxiety disorders24,25,
OCD26, and PTSD8. Based on many single unit recording studies in nonhuman primate literature,
we know that the dorsal anterior cingulate cortex (dACC) is not a homogenous region and
instead is composed of distinct cognitive, emotional, and motor subdivisions.27–31 Hinging on
implications of ACC activity for neuropsychiatric illness, tasks have been developed to reliably
activate each distinct subdivision to better relate neural activity to behavior. To focus the scope
of the discussion, we will discuss two tasks related to conflict interference and resolution, and
associated findings. We will later present data from these tasks in our results.
The Multi-Source Interference Task
6
The Multi-Source Interference Task (MSIT) is designed to maximize cognitive interference and
dACC activation to study attention, response selection, and cognition. Elements of the Stroopa,
Eriksenb, and Simonc interference effects32–35, which are all known to activate the dACC during
group analysis, were combined in MSIT to maximally activate the dACC. During MSIT,
participants are given a keypad and instructed that the buttons represent the numbers one, two,
and three from left to right. Sets of three numbers are presented as stimuli, where one number is
different than the other two. Participants are asked to report the value of the unique number,
regardless of its position and the value of the distractors. During congruent trials, the target
number is placed in the position that corresponds to its value. During incongruent trials, the
target number is placed in a position that does not correspond to its value, and reaction time
increases.35,36
MSIT improves upon previous interference based tasks by reliably and robustly activating the
cingulo-frontal-parietal (CFP) cognitive/attention network during interference trials at the single
subject level (Figure 4).35,36 The CFP is composed of the dACC, dorsolateral prefrontal cortex
(dlPFC), and superior parietal cortex. In addition, consistent with the results of other complex
cognitive tasks, the perigenual anterior cingulate cortex (pACC) has been found to respond
reciprocally to dACC activation. During studies of complex cognitive tasks, an increase in dACC
activity is associated with a decrease in pACC activity, and vice versa. During tasks involving
a During the Color Stroop task, participants are asked to report the color ink a word is written in, ignoring the color
word, and have slower reactions when the color ink the word is written in different than the color word (e.g. blue ink
color, when the color word is ‘red’). b During the Eriksen Flanker tasks, participants are asked to report a centrally located target letter, ignoring letters
that flank the target letter. Reaction times are slower when flanker letters are different than the target letter (e.g.
‘DDTDD’, rather than ‘TTTTT’). c During the Simon task, participants are asked to press a button on the right when they see a red stimulus, and a
button on the left when the see a green stimulus. Reaction times are slower when the relative spatial location of the
stimulus is farther from the target button (e.g. red stimulus on the left of the screen).
7
emotion however, an increase in dACC activity increased pACC activity has been observed.24,37–
40
Figure 4: Adapted from Bush et al. 2006. Single subject fMRI response during MSIT. Light gray = gyri, dark gray = sulci. Note
the robust bilateral activation (P< 10-4) in the daMCC, dlPFC, and the superior parietal cortex. Often, significant activity is also
seen in the ventrolateral prefrontal cortex (vlPFC).36
Functional connectivity differences between neuropsychiatric patients and healthy controls have
been identified during MSIT, although no significant differences in task performance have been
found. For example, patients with Major Depressive Disorder (MDD) exhibit decreased
connectivity between the subgenual ACC and the vlPFC, and between the the subgenual ACC
and ventral striatum relative to healthy controls.41 ADHD patients treated with stimulant
medications have been found to exhibit increased dACC activation during the task relative to no
stimulant medication.42 OCD patients have been found to exhibit greater supplementary motor
area (SMA) activation and greater rostral anterior cingulate cortex (rACC) deactivation during
interference trials, along with hyper-activation of the dACC and insular cortices.43,44
The Emotional Conflict Resolution Task
The Emotional Conflict Resolution (ECR) task was designed to gain insight on the neural
mechanisms by which emotional conflict is monitored and resolved. The task combines elements
8
from the Stroop task1, and the emotional Stroop taskd.32,45,46 Unlike the classic Stroop task, there
is no element of interference in the emotional Stroop task, as the meaning of the word is
irrelevant to the task (i.e. distracting information does not lead to responses that compete with
the correct response option). During the ECR task, participants are presented with photos of
faces with fearful or happy expressions, with the words ‘happy’ or ‘fear’ overlaid across them.
Subjects are asked to identify the emotion of the facial expression while ignoring the text. During
congruent trials, the facial expression and word match. During incongruent trials, the facial
expression and word do not match, and reaction times increase. As compared to the emotional
Stroop task, there is an emotional interference effect elicited by the ECR task.47
The ECR task generates robust neural activity related to the amount and resolution of emotional
conflict on a trial-to-trial basis. Activity in the amygdala, dorsomedial prefrontal cortex
(dmPFC), and the dlPFC reflect the amount of behavioral conflict, while activity in the rostral
cingulate cortex is related to reduction in conflict. Activation of the rostral cingulate is
accompanied by a simultaneous decrease in amygdala activation. This inhibitory relationship
suggests that the rostral cingulate cortex may resolve conflict through top-down inhibition of the
amygdala, whereas the dmPFC and the dlPFC monitor conflict.47
Functional connectivity differences between neuropsychiatric patients and healthy controls
during the ECR task have implications for mood and anxiety disorders. Patients with generalized
anxiety disorder have been found to be completely unable to regulate emotional conflict during
the ECR task as compared to both healthy controls and patients with depression.48,49 Separate
from the ECR task, findings from many studies support the notion that rostral cingulate
d During the emotional Stroop task, participants are asked to report the color ink a word is written in, ignoring the
content of the word. Words are either emotionally salient (e.g. ‘death’) or emotionally neutral (e.g. ‘chair’). Reaction
times are either not affected at all, or habituate rapidly.
9
inhibition of the amygdala is important for resolving emotional conflict. PTSD patients show a
hypoactive rostral cingulate during the recollection of trauma50 and the severity of symptoms
correlates with the degree of rostral cingulate hypoactivation.8 The hypoactivity of the rostral
cingulate cortex is also related to refractory depression51 and predicts a poor response to
antidepressant medications.52
Observing functional connectivity via local field potential recordings
While functional connectivity in humans has traditionally been explored during task-based fMRI
studies, functional connectivity can be estimated from data types that vary widely in temporal
and spatial scales, including EEG and ECoG (Figure 5). Accordingly, it is necessary to develop
methods to assess functional connectivity during high-density multi-sensor Local Field Potential
(LFP) recordings. In particular, we are motivated to measure functional connectivity via LFP
recordings to identify the neural signatures of neuropsychiatric illness.
Figure 5: Brain regions communicate by changing the strength, pattern, and frequency of neural activity in space and time. The
signal can be broken down into frequency components. If two signals are oscillating at the same frequency with a phase lag, we
can say they are oscillating in synchrony and draw conclusions about functional connectivity.
Quantifying functional connectivity poses challenges for researchers, who are tasked with
choosing a meaningful metric and interpreting results. Many metrics have been developed, each
with its own advantages and limitations. To focus the scope of this discussion, we will discuss
the advantages and limitations of one method, canonical correlation analysis, in terms of closed-
loop feature identification for neuropsychiatric illness.
10
Canonical correlation analysis
Canonical correlation analysis (CCA) is a robust tool for detecting connectivity between pre-
defined ROIs. During CCA, individual sensors are grouped into pre-defined anatomical ROIs.
Variability within individual regions is assessed through singular value decompositions on all the
sensors grouped to each region. Then, the resulting singular vectors are transformed in a way that
maximally correlates activity between each region pair. The technique both reduces the
dimensionality of the network from individual sensors to ROIs and successfully estimates spatial
correlation between ROIs.53
CCA has been proven to accurately predict regional connectivity in the presence of high SNR
and background correlations (Figure 6). In addition, the region-level analysis intrinsic to CCA is
advantageous for identifying biomarkers of neuropsychiatric illness for several reasons. First,
because the fMRI results we outlined are in terms of ROI analysis, it may be possible to draw
parallels between CCA results and fMRI studies. Second, CCA provides means for a regional
analysis without resorting to signal averaging within ROIs. This is important for neural time-
series data because if two signals within a region are not phase aligned, signal averaging can
cause reduction in signal-to-noise ratio (SNR) by canceling out the signals of interest. Third,
canonical correlation improves detectability of weak correlations present between many
individual nodes. In other words, CCA has been proven to detect correlations between regions
that would not be detectable at the level of individual nodes.53
11
Figure 6: Adapted from Stephen et al., (2014). Network diagrams show that CCA detects correlation between regions in the
presence of varying signal to noise ratios (A), and background correlations (B).53
Mitigating spurious functional connectivity estimates
Measured signals contain both the signal of interest, signals of no interest, and noise, which
means that nonzero connectivity estimates do not always imply direct interaction between
regions, regardless of the chosen metric.54 For example, local field potential recordings are often
referenced with one distant scalp electrode. Fluctuations in electric potential that occur at the
reference location may be reflected in neighboring electrodes. Therefore, the reference signal
common across different channels gives rise to spurious connectivity estimates.
Bastos and Schoffelen demonstrate the common reference problem with simulated data, and
propose a bipolar referencing strategy to mitigate the spurious effects54. If the neural signals
from adjacent electrodes are subtracted from each other, the reference common to both signals
will cancel out (Figure 7). This method operates under the assumption that the reference is
12
equally present in each channel, and that each channel reflects a different mixture of underlying
neural signals.
Figure 7: Adapted from Bastos and Schoffelen (2015)54. (A) Data 1 and Data 2 reflect both the signal of interest and the common
reference signals. When the sources are not coupled through the simulation, a spurious coherence estimate appears. If the sources
are coupled, the coherence estimate reflects both coupling and the spurious estimate. (B) Signals from adjacent electrodes are
subtracted from each other during a bipolar referencing scheme. When the sources are not coupled via the simulation, there is no
spurious coherence estimate.
Considerations for closed-loop algorithm development for implantable systems
Evaluation of performance
Requirements for acceptable levels of sensitivity and specificity for brain machine interfaces
(BMIs) are not standard and should depend on clinical application. Performance standards for
assistive communication devices might be different than performance standards for seizure
detection. For example, in one recent study, a 70% accuracy was claimed to be good enough for
severely-disabled amyotrophic lateral sclerosis (ALS) patients using a BCI for
communication.55,56 However, if seizure detection was only 70% accurate, patients would still
13
not be able to risk participating in activities of daily life (e.g. driving).57 Many BMI studies
report classification performance only in terms of specific utility, and define the threshold of
success based on patient surveys. For example, for assistive communication BMIs, success rates
are often reported in characters typed per minute.58
Latency between state onset and delivery of therapy is another important consideration for
closed-loop algorithm performance, and is likewise dependent on application. For example, brain
states associated with cursor control are transient and require immediate action timescales less
than one second. However, brain states associated with neuropsychiatric illness might wax and
wane over minutes (e.g. anxiety, PTSD) to days (e.g. bipolar disorder, depression).6 Computation
of neural features must be computed in time to detect and treat neuropsychiatric illness as brain
states evolve and symptoms arise.
In addition to achieving high performance, closed-loop decoding algorithms must be reliable.
Because the neural features of neuropsychiatric illness are still unknown, we do not know if the
features will be constant over time. Motor BMI studies indicate that any closed-loop algorithm
based on neural signals will likely require both frequent recalibrations and online updating to
reduce non-stationarities.59–61
Successful demonstrations of closed loop DBS
Research for more than a decade has focused on developing ‘closed-loop’ or ‘adaptive’ DBS
systems to improve upon traditional open-loop DBS therapy. A closed-loop system could reduce
latencies for therapy adjustments, and allow for therapies to be tailored based on patient needs.
Recent demonstrations of closed loop DBS for Parkinson’s disease and epilepsy are promising
for the future of closed-loop DBS.62–64 However, these demonstrations were not fully
14
implantable. Instead, they relied on data streaming to an external computer that detected state
and streamed commands back to the stimulator accordingly.
Computational and power limitations
There has been progress on developing platforms for fully implantable closed-loop systems that
give insight on computational and power limitations of future implantable closed-loop devices.
For example, Afshar et al. (2013) developed an investigational platform for an implantable
device that delivers stimulation. They tested the system using a classifier based on a simple
thresholding algorithm, and estimated that their device would draw 10% less power than current
open-loop devices used for Parkinson’s disease.65
More recently, Khanna et al. (2015) deployed a fully implantable, externally programmable
closed-loop system by providing a firmware update to the investigational Medronic Activa PC +
S system. Because the firmware update is noninvasive, the system acts as a platform for testing
closed-loop strategies in clinical trial patients already implanted with PC+S. Through offline
analysis, they estimated that a closed loop paradigm based on a simple thresholding algorithm
would draw less current than open-loop stimulation. Assuming that stimulation would be on for
90% of waking hours (based on testing), they estimate that closed-loop stimulation would still
draw 26% less current than open-loop mode.66
Specific Aims of the Study
Engaging in effortful mental action, such as trying to decide between conflicting choices or
suppress an unwanted emotion, activates specific brain networks. This activity might
differentiate one behavior from another. Detecting these specific brain states might help us
understand how the brain becomes dysfunctional in neuropsychiatric illness, including but not
15
limited to anxiety, depression, bipolar disorder, substance abuse, attention deficit disorder, post-
traumatic stress disorder, and traumatic brain injury. It would be possible to identify when a
patient is trying to do a critical mental task, but failing. An intervention, like deep brain
stimulation (DBS), could be applied to augment the brain's function during that specific effort.
To our knowledge, there are no documented algorithms for classifying engagement in effortful
mental action based on electrical activity in the brain. In the present study, we aim to predict
human task engagement by exploiting patterns of functional connectivity in cortical and sub-
cortical local field potential recordings. Our specific aims are outlined below:
Aim 1: Design a classification algorithm that distinguishes task engagement from free behavior.
(1.1) We will select neural features that distinguish periods of task engagement from periods of
free behavior. (1.2) Using the neural features as input, we will design a classifier that will
accurately predict task vs. non-task engagement.
Aim 2: Identify a subset of neural features critical for high classification performance. (2.1) We
will identify a subset of the features critical to classification performance. (2.2) We will compare
the identified feature sets across patients.
Aim 3: Evaluate decoder stability over time. (3.1) We will evaluate classification performance
over multiple recording sessions during which the patient was engaged in the same task.
16
METHODS
Behavioral tasks
Participants engaged in either one or two conflict based tasks: the Multi-Source Interference
Task (MSIT) and the Emotional Conflict Resolution (ECR) task. During both tasks, stimuli were
presented with software (either Presentation® by Neurobehavioral Systems or Psychophysics
Toolbox 67–69).
MSIT in the epilepsy monitoring unit was composed of at least one and up to five 64-trial blocks.
During each trial, images were presented for two seconds with a fixation cross-presented for
between two and four seconds between images. Images were three numbers between zero and
three. One of the numbers, the ‘distractor,’ was different than the other two. Participants were
given a keypad and instructed that keypad numbers one, two, and three represented the numbers
that appeared on the screen from left to right. Participants were then instructed to identify the
distractor by pressing the button corresponding to its value, ignoring its position. The value of
the distractor either matched the button corresponding to its position (congruent) or did not
match (incongruent) for each trial (Figure 8A).35,36 Congruence changes from trial to trial were
evenly balanced in number and frequency within each block.
The ECR task in the epilepsy monitoring unit was composed of at least one and up to six 64-trial
blocks of images presented for one second with a fixation cross presented for between two and
four seconds between images. Images were emotive faces cropped to show either happy or
fearful expressions, drawn from a set of expressions with five identifiable male faces and six
identifiable female faces.47 The words “FEAR” and “HAPPY” were overlaid on the faces such
that the word and expression either matched (congruent) or did not match (incongruent) for each
17
trial (Figure 8B). The images were presented in a pseudorandom order such that the identity,
gender, and valence (fear or happy) were shown randomly, though the congruence changes were
balanced within each block.
Figure 8: Schematics of the MSIT (left) and ECR (right) conflict-based tasks. During MSIT, the participant is asked to report the
value of the distractor, ignoring its position. During the ECR task, the participant is asked to report the emotion of the facial
expression, ignoring the text.
Patients
Fourteen participants, each with a history of long-standing pharmaco-resistant complex partial
seizures, underwent clinically-indicated invasive monitoring as part of the process of confirming
hypothesized seizure focus. Our research on neural activity during the MSIT and ECR tasks was
completed while the patients were in the hospital awaiting seizures for clinical mapping of foci.
The decision to implant electrodes and the number, types, and location of implantations were all
determined on clinical grounds by a team of caregivers independent of this study. Participants
were informed that participation in the experiment would not alter their treatment in any way and
that they could withdraw at any time without jeopardizing their clinical care. Each patient gave
fully informed consent according to NIH guidelines, and all procedures were approved by the
local institutional review board at Partners Healthcare (Massachusetts General Hospital).
18
Depth electrodes were stereotactically implanted to monitor and identify seizure foci. The
implantation procedure involved placing the patient under general anesthesia, followed by
implantation of multi-lead depth electrodes in stereotactically identified regions (Ad-tech,
Wisconsin, USA). Depth electrodes (Ad-tech Medical, Racine WI, USA, or PMT, Chanhassen,
MN, USA) consisted of 8-16 platinum/iridium-contact leads 2.4 mm long. Angled electrode
trajectories were achieved through a ROSA® robotic stereotactic system (Med-Tech,
Montpellier, France). The patients received bilateral electrodes ranging from five to nine
electrodes in the right hemisphere, and five to eight electrodes in the left hemisphere.
Electrode localization
Electrodes were localized by using a volumetric image co-registration procedure (Figure 9).
Using Freesurfer scripts (http://surfer.nmr.mgh.harvard.edu), the preoperative T1-weighted MRI
(showing brain anatomy) was aligned with a postoperative CT (showing electrode locations).
Electrode coordinates were manually determined from the CT and placed into the native space.70
Mapping to brain regions was performed using the electrode labeling algorithm (ELA) below.
Signal source for local field potentials are typically related to dipole source localization. The
main purpose of the ELA is to estimate the probability that a particular brain region contributes
to the dipoles that constitute the source of the signal. The ELA uses a purely anatomical
approach. Electrodes implanted in the brain receive signals from multiple sources, including
white matter and gray matter. Current thought proposes that signals are generated from grey
matter (whether cortical or subcortical), while white matter simply provides fibers for the signal
to pass through. We developed a simple and approachable ELA to identify the signal source by
identifying the probability that a given electrode is in a labeled region of the brain (gray matter).
19
Using the Freesurfer software and DKT (Desikan-Killiany-Tourville) brain atlas, we mapped the
brain electrodes to regions.71,72
The ELA operates with the assumption that the probability for each label to be a source of a
given electrode is estimated based on the Euclidean distance between the electrode and the brain
label that can be defined as brain label voxels and electrode voxels. Brain label voxels are all
voxels where at least one of the label vertices is positioned inside the voxel volume as mapped in
the structural MRI. Electrode voxels where the distance from the center of the electrode is
smaller than a given threshold, Dc, circumscribe a 3D cloud around the electrode. We chose Dc
to be 3mm. Each electrode is modeled as a three-dimensional thin cylinder oriented in a specific
direction. To estimate the probability, the ELA counts the number of intersection voxels between
the electrode voxel cloud and the label voxels and divides that count by the number of electrode
voxels. The output of the algorithm is an E x L matrix, X, where E is the electrodes number and L
is the number of the label. Xij is the probability that electrode i is getting a signal from label j.
In the case where no label voxels intersect with the electrode voxels for a given electrode, Dc is
expanded by ΔDc and the length is increased by Δl. We chose ΔDc to be 0.5mm and Δl to be
1mm. This expansion continued until we found a detectable intersection between the label voxel
and the electrode voxels, then decreased the probability the electrode is in each label. Therefore,
an electrode in white matter will have a low probability of existing within a label, while an
electrode entirely encased in grey matter will have a high probability of being within a label. The
number of bipolar electrodes assigned to each label is included in Table 2 and Table 3 of
Appendix I.
20
Figure 9: Example electrode localization using the ELA algorithm. The right panel shows electrode locations determined by
manual alignment of the pre-operative T1 weighted MRI and post-operative CT scan. The right panel shows the output of the
ELA algorithm – each electrode is assigned to a particular region of interest, including the dorsolateral prefrontal cortex (dlPFC),
dorsomedial prefrontal cortex (dlPFC), orbitofrontal cortex (OFC), ventrolateral prefrontal cortex (vlPFC), cingulate cortex,
temporal lobe, insula, hippocampus, caudate, and amygdala (for this particular patient dataset).
Invasive electroencephalography recordings
Intracranial local field potential (LFP) recordings were acquired using one or two Neural Signal
Processor (NSP) recording systems (Blackrock Microsystems Inc., Salt Lake City, UT) at a
sampling rate of 2 kHz. Table 1 includes a summary of the invasive electroencephalography
(iEEG) LFP recordings collected per patient. At the time of acquisition, depth recordings were
referenced to one scalp EEG.
Data preprocessing
Data analysis was performed using custom analysis code in Matlab (MathWorks) and Fieldtrip,
an open source software implemented in Matlabe.73 All data were down-sampled to 1000 Hz and
demeaned relative to the entire recording. Line noise and its harmonics up to 200 Hz were
removed by subtracting the band passed filtered signal from the raw signal on each channel.
e http://www.ru.nl/neuroimaging/fieldtrip
21
Table 1: iEEG recording summary in each of the 14 patients, including number of task sessions completed, number of NSPs used
during signal acquisition, and number of electrodes implanted in each hemisphere.
# Patient Label MSIT sessions ECR sessions # of NSPs # of Electrodes
(8-16 contacts ea.)
1 2 1 2 Left Right
1 79 x 1 6 6
2 85 x 1 5 7
3 86 x x 1 9 5
4 88 x 1 6 6
5 89 x x 1 5 6
6 90 x x 1 7 7
7 95 x 2 7 7
8 96 x x 2 8 5
9 99 x x x x 2 8 6
10 101 x 2 9 4
11 102 x x 2 9 7
12 104 x x x 2 9 5
13 106 x x 2 9 7
14 107 x x x x 2 9 8
Channel selection
Channels that exhibited excessive line noise or no discernible signal were removed from the
analysis. Based on clinical reports and on visual inspection, electrodes surrounding the epileptic
focus and/or exhibiting abnormal activities were also excluded.
Bipolar referencing
Neighboring channels were bipolar referenced relative to one another to account for volume
conduction. The signal recorded from each channel represents the reference potential, R,
subtracted from the electric potentials gathered at the site of two electrodes, source1 and
source2. To eliminate the reference component of the signal that is common across nearby
electrodes, adjacent signals are subtracted from one another [Eqn 1]. The result is the bipolar
referenced signal [Eqn 2].
22
𝑏𝑖𝑝𝑜𝑙𝑎𝑟 𝑠𝑖𝑔𝑛𝑎𝑙 = (𝑠𝑜𝑢𝑟𝑐𝑒1 − 𝑅) − (𝑠𝑜𝑢𝑟𝑐𝑒2 − 𝑅) [Eqn 1]
𝑏𝑖𝑝𝑜𝑙𝑎𝑟 𝑠𝑖𝑔𝑛𝑎𝑙 = 𝑠𝑜𝑢𝑟𝑐𝑒1 − 𝑠𝑜𝑢𝑟𝑐𝑒2 [Eqn 2]
Evoked response potential mitigation
Low frequency components of the signal are thought to be related to evoked-response potentials
(ERPs) that occur during task image onset or reaction. A high-pass filter (HPF) was applied to
eliminate frequency components of the signal below a cutoff frequency of 2 Hz. Figure 10A and
B shows the neural signal across all trials was averaged relative to the image onset or to the time
the patient reacted to the picture. The resulting ERPs were averaged across all bipolar electrodes
in each region, the attenuation of the average ERP per region after filtering was visually
inspected to determine adequate ERP removal.
Figure 10: High-pass filter removes frequency components less than 2 Hz. Voltage traces on each channel in the left dorsal
anterior cingulate cortex and right vlPFC was averaged across all trials relative to the image onset, and an HPF was applied to
eliminate frequency components of the signal less than 2 Hz. (A) Bipolar signal average in the left daCC is shown in blue, and
the HPF bipolar signal average is shown in red. (B) Bipolar signal average in the right vlPFC is shown in green, and the HPF
bipolar signal average is shown in orange. ERP Attenuation value is the peak-to-peak amplitudes of the bipolar and filtered
signals, normalized by the amplitude of the bipolar signal.
23
Drift elimination across multiple recording systems
Each NSP samples data according to its own clock cycle, controlled by crystal oscillators. Each
clock oscillates at a slightly different frequency. At a sampling rate of 2000 Hz, the difference in
oscillation rates generates drift between recordings between approximately one and two samples
every 20 minutes. In order to synchronize recordings, event time stamps saved along with neural
data from both NSPs were used as synchronization points. If the number of samples between
event time stamps did not match across both recordings, the shorter interval of data was adjusted
via linear interpolation so that its length matched the size of the longer interval of data.
Feature extraction
Canonical correlation analysis (CCA) was used to compute features that contain information
about the functional connectivity of the network. Here, we exploit canonical correlation by first
organizing channels into corresponding regions of interest using the electrode localization
algorithm. For example, regions X and Y contain two groups of nx and ny signals that vary with
time, t.
𝑋 = [
𝑋1,1 ⋯ 𝑋1,𝑡
⋮ ⋱ ⋮𝑋𝑛,1 ⋯ 𝑋𝑛,𝑡
] [Eqn 3]
𝑌 = [
𝑌1,1 ⋯ 𝑌1,𝑡
⋮ ⋱ ⋮𝑌𝑛,1 ⋯ 𝑌𝑛,𝑡
] [Eqn 4]
Two singular value decompositions are performed on the groups of signals, X and Y. The
resulting change in bases maximally correlates the signals within each set.53
𝑋 = 𝑈𝑥Ʃ𝑥𝑉𝑥† Y = 𝑈𝑦Ʃ𝑦𝑉𝑦
† [Eqn 5 & Eqn 6]
24
A matrix multiplication is performed to find the maximum correlation between each set of
signals. The first singular value of the product matrix, Qxy, is the canonical correlation
coefficient, which represents spatial correlation between the two sets of signals.
𝑄𝑥𝑦 = 𝑈𝑥𝑉𝑥†𝑉𝑦𝑈𝑦
† [Eqn 7]
One feature is calculated via canonical correlation between each region pair. Canonical
correlation values are in the range of zero to one.
Window size of analysis
Canonical correlation carries an innate bias. Coefficients increase as the window size of analysis
decreases. At small window sizes, coefficients approach their maximum value of one. Thus, task
and non-task distributions are less distinguishable at smaller window sizes. However, large
window sizes induce greater computational requirements, and are not ideal for eventual real-time
implementation. In addition, longer window sizes make it more uncertain when detectable brain
states begin and end. We sought to optimize the tradeoff between accuracy and efficiency by
computing canonical correlation features using window sizes ranging from 200 ms to 20 s with
an overlap of 200 ms. The corresponding classification accuracies were computed (procedure
described below). Based on the results, a window size of five seconds was selected and used for
all following analysis. Every five second window of data on each channel was normalized by
subtracting the mean and dividing by the standard deviation.
Classification
Class label assignment
Each five second window of data was labeled as ‘task’ or ‘non-task’ depending on whether the
patient was actively engaged in the MSIT and ECR behavioral tasks. Both image presentation
25
and fixation periods are included in the ‘task’ label assignment. A five-second buffer was placed
at the start and end of ‘task’ periods to allow for ramp-up and ramp-down periods. The length of
‘task’ periods are determined by the number of task blocks the patient agreed to participate in.
‘Non-task’ labels were assigned to portions of the recording collected immediately before and
after task engagement, as well as any breaks the patient opts to take while playing the task. This
decision was based on our assumption that rest data interleaved with periods of blocked task
engagement are well suited for functional connectivity estimations of the resting state.74 The
length of ‘non-task’ periods is arbitrarily determined by the number and length of breaks the
patient opted to take during the experiment, and the time between the start and end of the
recording and the beginning and end of the task, respectively. During ‘non-task’ periods, the
patient is behaving freely and is not engaged in the task.
Figure 11: Task and non-task datasets vary in lengths across patients and recording sessions.
For 21 of 23 recording sessions analyzed, task data far exceeded non-task data (Figure 11). To
balance unequal ‘task’ and ‘non-task’ class sizes, we used a data-level approach agnostic of the
classification algorithm. The class of smaller size was augmented by random oversampling with
26
replacement to make up the difference between class sizes.75 The supplemental feature set was
then concatenated to the original feature set.
Classification model and prediction
A Support Vector Machine (SVM) classifier with a linear kernel was designed using the
LIBSVM Matlab implementationf. A start to finish pipeline of preprocessing, feature extraction,
and classification is included in Figure 12. Canonical correlation features and corresponding
‘task’ vs. ‘non-task’ class labels were used as classifier input. A five-fold cross validation
strategy was used to determine training data and testing data for the model in a way that ensured
classifier performance was not overly influenced on any one portion of the dataset. The data was
separated into five folds. Four folds were used for training and the remaining fold was used for
testing. This process was repeated until every fold was used for testing. Accuracy, true positive
rate (sensitivity), true negative rate (specificity), false positive rate, and false negative rate were
calculated on each iteration of the five-fold cross validation strategy to gauge classifier
performance.
Figure 12: Classification pipeline for the identification of task-like brain states. Data is preprocessed via a low-pass filter to 1000
Hz, bipolar referenced, and normalized by subtracting the mean and dividing by the standard deviation. Canonical correlation
features are extracted from the preprocessed data, and fed to a Support Vector Machine that predicts ‘task’ vs. ‘non-task’ labels.
f (https://www.csie.ntu.edu.tw/~cjlin/libsvm/
27
Chance classification performance
Classification performance due to chance was calculated by shuffling class labels before class
sizes were balanced. Labels were randomly assigned in a way that preserved original class sizes.
Feature selection
Important features were selected through a ranking process based on SVM criteria (Figure 13).
For example, consider a feature set, Var, of size N. During the feature selection process,
classification accuracy is re-calculated for the feature set (of size N-1) after each feature is
removed and subsequently replaced. The feature removal that least impacts classification
accuracy is removed from the feature set, Var. This process is repeated until the feature set is
empty and all the features have been ranked. The last feature removed is ranked as most
important and conversely, the first feature removed is ranked as least important.
Figure 13: Feature selection using SVM-Based criteria, based on Rakotomamonjy et al. 2003.76.
28
Evaluation of decoder stability
Two patients, MG99 and MG104, completed two sessions of both the MSIT and ECR tasks, and
one patient, MG107, completed two sessions of the ECR task. These five datasets were the basis
for decoder stability analysis. Classification performance and feature selection were compared
within patients over both sessions of each task. To make fair comparisons, only bipolar
electrodes present in both recordings after preprocessing were included in the analysis. For
example, if an electrode was included in the first recording session and eliminated during the
second recording session (e.g. due to excessive line noise), the electrode would be excluded in
both test datasets.
29
RESULTS
Functional inference distinguishes task engagement
Spatial correlation via canonical correlation features successfully distinguished periods of task
engagement from free behavior for both the MSIT and ECR tasks. Mean sensitivity and
specificity for MSIT vs. non-task classification was 97.51 ± 2.20%, and 95.30 ± 2.94% (n = 11),
respectively (Figure 14). There was no significant difference between MSIT vs. non-task and
ECR vs. non-task classification performance. Mean sensitivity and specificity for ECR task vs.
non-task classification was 96.08 ± 3.05% and 94.33 ± 3.57% (n = 12), respectively (Figure 15).
We found that the canonical correlation algorithm was not dependent on the ERPs inherent to the
task-related data. Image presentation and movement associated with reaction are reliably
associated with a wave of slow frequency activity across most channels. While ERPs also appear
throughout the non-task data, free behavior does not require focused attention to an image that
appears on a screen approximately every five seconds. If classification performance was
dependent on low frequency activity, the real-world implications of the classifier would be
limited as the algorithm would be tied to contexts related to experimental behavioral tasks. To
test whether classification performance is dependent on task-related cyclic ERPs, low frequency
activity (<2 Hz) was removed from the signal by applying a high-pass filter. Canonical
correlation features were recalculated using signals from the filtered data. No significant
differences were found between classification performance before and after filtering.
While there was a positive bias in chance classification performance, true performance
significantly outperformed chance performance (MSIT vs. non-task: p <0.01; ECR task vs. non-
task: p < 0.01). The classification accuracy we expected after shuffling labels was approximately
30
50%. We found the mean classification accuracy to be 55.2 ± 2.62% and 54.3 ± 2.18% for the
MSIT and ECR tasks, respectively. A positive bias was likely introduced by the resampling
procedure used to balance class sizes. If one sample is overrepresented in the balanced dataset
due to resampling, the same sample is likely to be included in both testing and training datasets.
Testing the classifier on a portion of training data introduces a positive bias, evident from the
inflated chance classification performance.
Figure 14: (A) Boxplots show classification performance metrics for MSIT vs. non-task classification (accuracy, true positive
rate, and true negative rate). SVM classifier performance with canonical correlation features are highlighted in gray (mean acc:
96.4 ± 2.32%). Classifier performance after high-pass filtering for ERP removal (mean acc: 95.7 ± 2.37%), and due to chance
after shuffling labels (mean acc: 55.2 ± 2.62%) are included for comparison. (B) Receiver operating characteristic (ROC) curves
show sensitivity (true positive rate) vs. specificity (true negative rate). (C) A diagnostic table shows the mean true and false
positive rates, and the true and false negative rates across all patients.
31
Figure 15: (A) Boxplots show classification performance metrics for ECR task vs. non-task classification (accuracy, true positive
rate, and true negative rate). SVM classifier performance with canonical correlation features are highlighted in gray (mean acc:
95.2 ± 3.02%). Classifier performance after high-pass filtering for ERP removal (mean acc: 94.43 ± 3.48%), and due to chance
after shuffling labels (mean acc: 54.27 ± 2.19%) are included for comparison. (B) Receiver Operating Characteristic (ROC)
curves show sensitivity (true positive rate) vs. specificity (true negative rate). (C) A diagnostic table shows the mean true and
false positive rates, and the true and false negative rates across all patients.
A five-second window size was selected for canonical correlation feature extraction. Canonical
correlation carries an innate bias in that coefficients increase as sample lengths decrease. At
small window sizes, coefficients approach their maximum value of one. Thus, task and non-task
distributions are less distinguishable at smaller window sizes. This phenomenon is illustrated in
Figure 16 C, D & E. Over a 200 ms window (Figure 16 C & D, bottom panels), canonical
correlation coefficients are skewed towards the right for both non-task and task distributions
(Figure 16 E, bottom panel). Considering a five-second window (Figure 16 C & D, top panels),
canonical correlation coefficients are less skewed and distributions become more distinguishable
(Figure 16 E, top panel). To select a window size that balances tradeoffs between computational
limitations and classification performance, classification accuracies were calculated across
window sizes ranging from 200 ms to 20 s (Figures 16A and B). Most of the exponential
32
increase in classification accuracy occurs before five seconds. Therefore, we chose a five-second
window for all following analyses.
Figure 16: (A, B): Classification accuracy across patients obtained using windows of data ranging from 200 ms to 20 s. (C)
Illustration of canonical correlation features across multiple window sizes. The leftmost column shows voltage traces from all the
electrodes in the left dorsomedial prefrontal cortex (dmPFC) over a five-second example window (top) and 200 ms example
window (bottom). Likewise, the center column shows voltage traces from all the electrodes in the right dorsolateral prefrontal
cortex (dlPFC) over a five-second example window (top) and 200 ms example window (bottom). The rightmost column shows
canonical correlation values computed over an entire task recording session divided into five-second windows (top) and 200 ms
windows (bottom).
Classification accuracy depends on a subset of optimal features per patient
The most influential region pairs among the feature set were selected through a ranking process
based on SVM criteria, during which the feature dropped at each iteration contributed least to
33
upholding SVM accuracy. The threshold for clinical success was defined as 80% accuracy based
on current motor brain machine interface decoding standards.55,56,61 The ranking process is
displayed in Figure 17 A and Figure 18 A, which show SVM accuracy decreasing as region pairs
are iteratively dropped from the feature set. When accuracy reached the 80% threshold marked
by the dotted lines, we deemed the remaining features significant.
Significant features required to maintain 80% accuracy vary across patients in quantity, as shown
in the inset of Figure 17 A and Figure 18 A (MSIT: med 3 ± 3.4 region pairs, ECR: med 4 ± 6.6
region pairs). Quantifying the number of significant features allows for assessment of recording
requirements necessary to reach the threshold for clinical success. The number of regions (rather
than region pairs) required to meet 80% accuracy ranges from two to 19 for both the MSIT (med
6.0; std 5.0) and ECR (med 7.0; std 4.9) tasks (Figure 17 B and Figure 18 B). The total number
of channels (before bipolar referencing) implanted across the set of significant regions ranges
from 16 to 153 for the MSIT (med 50; std 37.6), and 17 to 148 for the ECR task (med 76.0; std
36.6).
In addition to varying in quantity, significant features are unique to each patient with limited
overlap between patients engaged in the same task. For example, the unique feature sets required
to reach the 80% accuracy for two patients, MG86 and MG96, during the MSIT and ECR tasks
are shown as network diagrams on the left panels of Figure 19 and Figure 20. The number of raw
channels implanted in each significant region is shown on the right panels of Figure 19 and
Figure 20.
34
Figure 17: (A) Feature pruning reveals features required to maintain 80% accuracy for MSIT vs. non-task classification across
patients. The significant feature sets consist of (B) 6±5 regions of interest, and (C) 50±37.6 non-bipolar channels across patients.
Figure 18: (A) Feature pruning reveals features required to maintain 80% accuracy for ECR task vs. non-task classification
across patients. The significant feature sets consist of (B) 7±4.9 regions of interest, and (C) 76±36.6 non-bipolar channels across
patients.
35
Figure 19: Network diagrams illustrate two examples (patients MG86 and MG96) of feature sets required to reach 80% accuracy
for MSIT vs. non-task classification. (A, B) These diagrams demonstrate significant network connectivity for MG86 and MG96,
along with the corresponding number of channels in each ROI.
Figure 20: Network diagrams illustrate two examples (patients MG86 and MG96) of feature sets required to reach 80% accuracy
for ECR task vs. non-task classification. (A, B) These diagrams demonstrate significant network connectivity for MG86 and
MG96, along with the corresponding number of channels in each ROI.
Optimal feature sets shift over time
Feature sets required to meet 80% classification accuracy varied across multiple sessions of the
same task within each patient. The most influential region pairs among feature sets across
multiple recording sessions were again selected through a ranking process based on SVM
criteria. The ranking process is displayed in Figure 21 A, where the callout displays the number
of features required to meet 80% accuracy (med 5.0; std 5.65). The number of regions required
to meet 80% accuracy ranges from two to 17 (med 7; std 4.79) (Figure 21B). The total number of
channels (before bipolar referencing) implanted across the sets of significant regions ranges from
14 to 143 (med 64.5; std 28.22). It is interesting to note that the second ECR session completed
by MG107 yielded the only classification performance above 80% accuracy based on only one
36
feature, the canonical correlation between the right dorsal anterior cingulate cortex and the left
caudate.
Figure 21: (A) Feature pruning reveals features required to maintain 80% accuracy for task vs. non-task classification across
patients that performed multiple sessions of the same task. Unbroken lines and filled circles correspond to the first recording
session, and dotted lines and unfilled circles correspond to the second recording session. The significant feature sets vary across
recording sessions and consist of (B) 7±4.79 regions of interest and (C) 64.5±32.9 non-bipolar channels across patients.
Significant features vary from session to session, with fluctuating degrees of overlap among
significant regions between sessions. The unique feature sets required to reach 80% accuracy for
one example pair of recording sessions, MG99 engaged in the ECR task, are shown as network
diagrams in Figure 22 A and B. No features deemed significant during the first recording session
were declared significant during the second recording session. However, there is some overlap in
individual regions of interest (rather than in terms of the features, which are region pairs). The
left hippocampus, left dorsolateral pre-frontal cortex (dlPFC), and right lateral orbitofrontal
cortex (lOFC) are shared as significant ROIs between sessions one and two. There was overlap
between required regions in three of four session pairs analyzed. The mean percentages of
significant regions from session one that exist in session two and vice versa are ~55% and ~62%,
37
respectively. The distribution of overlap varies widely across recording sessions, ranging from
0% for MG99 during MSIT to ~92% for MG104 during the second session of the ECR task.
Figure 22: (A, B) Network diagrams illustrate one example of the feature sets required to reach 80% accuracy for ECR task vs.
non-task classification. Significant network connectivity for MG99 during session one and session two is shown in the left and
right network diagrams, respectively. (C) The number of channels in each region represented in the significant feature set is
shown for session 1 (white bars), overlapping between session 1 and session 2 (gray bars), or session 2 (black bars). (D) The
degree to which the regions in significant ROI pairs overlap between session 1 and session 2 is shown by the boxplots.
Training on temporally separated data shows potential for decoding stability
When training and testing on the same session, the classifier yields accuracies greater than 90%
for each of the session pairs. To test classification performance across multiple sessions of data,
the classifier was trained on features from one session and tested on the other session in its pair.
Training and testing the classifier on different sessions caused accuracy to degrade significantly,
38
although performance was still above chance levels (Figure 23A). We observed a general trend
where accuracy degraded as the amount of time between sessions increased (Figure 23B).
When training and testing the classifier on data from both sessions, mean classifier performance
accuracy across patients was approximately 89%. The classification performance improvement
that we observe indicates that training on temporally separated data may improve classification
performance over longer periods of time.
Figure 23: (A) Boxplots show accuracy of classifiers (1) trained on session 1 and tested on session 1, (2) trained on session 2 and
tested on session 2, (3) trained on session 1 and tested on session 2, and (4) trained on session 2 and tested on session 1. Accuracy
reported is an average testing accuracy after five-fold cross validation. (B) Scatterplot shows results for classifiers trained and
tested on different sessions for the same patient. Classifier accuracy drops as time between sessions increases. (C) Boxplot shows
accuracy of classifier trained and tested on both sessions of data. Accuracy reported is an average testing accuracy after five-fold
cross validation.
39
DISCUSSION
Detecting brain states associated with effortful mental action is important for the translation of
closed-loop DBS for neuropsychiatric illness. Beyond accurately detecting relevant brain states,
a clinically viable decoder must consistently enable levels of relief desired by the patient and
ultimately operate within the constraints of device firmware. Working towards clinical
translation, we have demonstrated successful detection of conflict related brain states and have
begun to address these challenges in terms of our proof of concept algorithm.
Classification performance
We have taken a known technique, CCA, and applied it to neural data collected during
experimental behavioral tasks to extract neural features related to effortful mental activity. These
neural features were used as input to an SVM classifier to predict task-related brain states. Using
these methods, we developed an algorithm that successfully distinguishes task from non-task
related mental effort.
When selecting window size for CCA feature extraction, we considered requirements imposed
by the ultimate application and platform for the algorithm. To be useful as a therapeutic tool, the
algorithm must accurately and quickly identify brain states shifting in time and run on an
independent platform limited in memory and power. Therefore, we selected a five second
window size based on our aims to maximize accuracy while minimizing the computational
requirements that stem from holding substantial amounts of data in memory.
The threshold for clinical viability in this study was set to 80% accuracy based on performance
standards in clinically used BMIs based on LFP decoding.55,56,61 However, requirements for
acceptable sensitivity and specificity ranges are not standard and instead should depend on
40
clinical application and needs of the patient population.57,58 The ROC curves (Figures 14 & 15)
highlight a tradeoff between relieving symptoms and avoiding side effects. The chosen
sensitivity and specificity point on the curve will depend on side effects. For example, perfect
specificity may be desirable for a person that experiences undesirable side effects when
stimulation is triggered after a false detection. While we claim success on our classification rates
with mean sensitivity and specificity rates greater than 94%, there is little precedent for
acceptable rates of performance for decoding brain states associated with neuropsychiatric
illness. This gap in knowledge indicates a need for systematically gauging the needs of the
patient population (e.g. surveys).
One important performance measure for decoders independent of application is the ability to
predict above chance levels.57 Chance prediction in the present study was approximately 55%,
which is slightly greater than the chance level expected for a two-class classification problem
(1/2 or 50%). We suspect that chance prediction for our classifier was slightly inflated by the
method chosen for balancing class sizes. Random labels were assigned to data samples in a way
that preserved original class sizes before the smaller class size was augmented via resampling.
Choosing to assign random labels before resampling likely introduced a bias in classification
performance due to the overrepresentation of a subset of samples in the dataset. Additionally, the
inflated chance performance indicates that the overall classification performance is inflated as
well. For 15 of 23 recordings analyzed, the non-task class size was less than half the task class
size. During future experimental recordings, it will be important to better control class sizes by
prioritizing non-task data collection.
In addition to the traditional performance measures discussed, the timeframe between brain state
onset and application of therapy will affect both efficacy and patient satisfaction with treatment.
41
The algorithm we developed extracted canonical correlation features from five-second windows
of data. While the five-second window size satisfies our assumptions about performance and
computational needs, it automatically sets a baseline for how quickly closed-loop intervention
can occur. For example, if the algorithm is operating in real-time and the onset of a pathological
brain state begins toward the start of the five second window, at least five seconds will pass
before the brain state is detected while the algorithm gathers additional data to fill the rest of the
window. The five-second estimate does not include the additional time required for feature
extraction, classification, and control. This timeframe would not be acceptable in the realm of
motor brain machine interfaces (BMIs), where naturalistic control of effectors (e.g. cursors,
prosthetic limbs) requires immediate action, and ‘thought-to-action’ brain states are transient.
However, a delay on the order of approximately 10 seconds could be acceptable for treating
neuropsychiatric illness if brain states are slowly evolving in a way that mirrors behavioral
symptoms. Moving forward, it is important that the algorithm is tested on an existing real-time
system to gauge computation time, requirements, and online performance, as prediction power of
decoders trained offline often does not translate to prediction power during closed-loop
performance.60
The canonical correlation feature is limited in that it does not address correlations in the
frequency domain. Canonical correlation coefficients are likely dominated by large amplitude
low frequency components. A spectral decomposition approach could be combined with CCA in
the future. Following, examining higher frequency bands might allow for shorter time windows
of analysis, because optimal window length is likely to depend on frequency band. If analyzing
high frequency activity allows for a decrease in window size without sacrificing classification
42
performance, this would strengthen the decoder by decreasing the latency between state onset
and detection.
Feature selection
To gain insight on functional connectivity important for distinguishing task-related brain states,
we identified a subset of features for each patient that are likely important for maintaining 80%
classification performance based on SVM criteria. We found that significant features differed
across tasks and patients and offer several possible explanations for these discrepancies. First,
electrode placement was determined purely by clinical criteria, which means that the regions
implanted and the number of electrodes in each region were different for each patient. Feature
sets across patients could be more consistent if the electrode implantation scheme was the
uniform across patients. Second, the sheer number of electrodes implanted in each region may
impose a bias on which regions are deemed significant. It is interesting to note, however, that
classification performance was not dependent on any one implant scheme. Successful
classification performance across many configurations of electrodes suggests extensive regional
network involvement in generating task-related brain states.
Selecting a subset of important features could be useful for guiding implant locations and
reducing computational requirements as the work matures. There is potential for further
reduction in channel count beyond the region and electrode count we have reported for
maintaining 80% accuracy. All electrodes with a nonzero probability of falling within a region
label via the ELA were included in analysis. Electrodes localized to a particular region may
actually be recording from passing fibers in white matter, which may carry signals coming from
another region. Thresholding electrodes by setting a minimum gray matter probability or
imposing a cap on the number of electrodes in each region would increase probability that
43
signals are generated by the target regions and reduce channel count per region. For example,
over 20 channels are implanted in the left temporal lobe of MG96, while only two channels are
implanted in the right dorsal anterior cingulate cortex (dACC). Both regions were significant;
however, the left temporal lobe requires 10 times the channel count than the right dACC, which
increases computational requirements. We have not explored whether computational complexity
could be reduced by selecting important channels without sacrificing classification performance.
The median number of features required to reach 80% classification accuracy was 6 and 7 region
pairs for the MSIT and ECR tasks, respectively. Only one recording, the second session of ECR
for MG107, yielded accuracy above 80% for just one feature. We were surprised to find that
classification task engagement is dependent on the functional connectivity between such a vast
array of structures. This is something that should be studied further, but is beyond the scope of
current analysis.
Decoder stability
Decoder stability is critical for clinical translation of the classifier. The instability may be due to
non-stationarities in the data related to physical movement of electrodes, dropped signals,
changes in neural activity due to inflammation after surgery. While stability was not achieved
over pairs of recording sessions spaced between a range of four hours and one week, we are
hopeful to achieve stability in the future with access to more temporally separated data. Repeated
recording sessions collected over tens to hundreds of days may be necessary to achieve stability.
Evidence from the motor BMI field is encouraging. For example, Vansteensel et al. describes a
ECoG based BCI that allows for control of a typing program. After 67 two-hour sessions over
157 days, they achieved control signal stability.61 Considering these findings and others from
LFP-based motor BCI literature, we are neither surprised nor alarmed by the lack in stability and
44
the lack of overlap in significant feature sets across sessions. In the final use case, we expect that
closed loop decoder adaptation algorithms along with remote calibration will be necessary for
long-term stabilization.55,56,59,60
The lack of reliability across sessions is likely due to the canonical correlation coefficient being
very susceptible to ongoing spurious events, given that a new linear combination is found that
maximizes correlation for every new time window. It is necessary to mitigate this effect in the
future by fixing the canonical basis for task and non-task data. By fixing the bases, only the
projections and canonical correlation coefficients would be calculated for each new time
window. In addition to potentially stabilizing the feature set over time, this method is also more
computationally efficient because singular value decompositions are not recomputed for each
time window. Further, including a statistical test could further increase the robustness of
canonical correlation over time.
Expanding the breadth of non-task related brain states used to train the classifier could strengthen
classification performance over a wide range of activities and potentially improve stability.
Currently, our data labeled as the ‘non-task’ state is composed of free behavior immediately
surrounding task engagement. During this time, the patient could be engaged in any number of
activities. Additionally, non-task activities during the first recording session may be completely
different than that of the second recording session. The lack of a robust non-task dataset might
contribute to the lack in overlap of significant features between pairs of recording sessions. A
potential challenge in including a wide array of non-task data is that task-related networks may
spontaneously reactivate during rest. Capturing a wide variety of activities during non-task state
recordings will be important for training and stabilizing the classifier.
45
Applicability
As discussed, the most immediate use case for our decoder is related to closed-loop DBS for
psychiatric purposes. Detecting when a patient is in a ‘task-like state’ or ‘vulnerable network
state’ could trigger a stimulation control policy to relieve symptoms of patients when they are
engaged in effortful mental activities during which they tend to need assistance.
We propose that our decoder could apply to contexts beyond the two experimental behavioral
contexts analyzed, including detecting task-related brain states both in and outside of
experimental behavioral contexts. We suggest that our classification pipeline is not task
dependent based on the high performance achieved across two different tasks. In addition, we
considered that task and non-task feature sets might be distinguishable due to the ERPs generated
during the repetitive image presentations and reactions that occur in experimental behavior tasks.
To mitigate ERP effects, we applied a filter to eliminate signal below 2 Hz. No significant
difference was found between classification performance before and after filtering. Therefore,
we claim that classification performance is not dependent on low frequency activity related to
repetitive ERPs inherent to the structure of experimental behavioral tasks. Further, we extend
this claim by hypothesizing that it may one day be possible to predict task-related states outside
of experimental behavioral contexts. The jump to detecting effortful mental action in real world
situations is one of many essential steps for establishing closed-loop DBS as a clinically viable
option for people suffering with neuropsychiatric illness.
While developing a decoding strategy for closed-loop DBS to treat neuropsychiatric illness has
largely motived this work, we imagine that detecting cognitive brain states could have
implications far beyond treating refractory neuropsychiatric illness. For example, knowledge of
cognitive states could be insightful during monitoring of coma patients, rehabilitating cognitive
46
function after strokes, gauging awareness and emotional state of soldiers in combat, or
identifying vulnerable periods for addicts. While there is much to accomplish before our proof of
concept algorithm becomes clinically viable, we believe the outlook is positive.
47
CONCLUSIONS
We have demonstrated the ability to accurately identify task-related brain states via invasive,
large-scale local field potential recordings during human engagement in conflict-based tasks.
Canonical correlation features were extracted and used as inputs to a classifier that predicted task
vs. non-task engagement for the MSIT and ECR tasks. Classification performance was not
dependent on the ERPs inherent to experimental behavioral tasks and was significantly greater
than chance performance. A subset of features related to regional connectivity were identified
that are likely important for maintaining the threshold for clinical accuracy, which we defined as
80%. The important feature sets were unique across patients engaged in the same task, as well as
across multiple task recording sessions. While significant feature sets were not maintained across
multiple sessions, there were varying degrees in overlap of significant regions across multiple
sessions. Successful classification was not achieved after training and testing the classifier on
different sessions. However, classification performance improved after training and testing the
classifier on data from multiple recording sessions. This improvement in classification
performance indicates the importance of training on temporally separated data.
While we have developed a proof of concept for detecting task-related brain states, further
investigation is required to establish the classification algorithm as a clinically viable element of
treatment. The amount of data required by the algorithm must be reduced beyond the identified
significant feature sets without sacrificing classification performance. The algorithm should be
tested in real-time to gauge both computational requirements and the minimum amount of time
required between state detection and application of therapy. The non-task state should
incorporate as many different types of non-task activities as possible to limit false detection of
conflict-related brain states in real world situations. Establishing decoder stability is essential. If
48
these limitations are addressed, task state detection could have potential for guiding closed-loop
treatment for neuropsychiatric illness.
49
REFERENCES 1. Whiteford, H. A. et al. Global burden of disease attributable to mental and substance use
disorders: Findings from the Global Burden of Disease Study 2010. Lancet 382, 1575–
1586 (2013).
2. Insel, T. R. Faulty circuits. Sci. Am. 302, 44–51 (2010).
3. Insel, T. R. & Wang, P. S. Rethinking mental illness. Jama 303, 1970–1971 (2010).
4. Gaynes, B. N. et al. What did STAR*D teach us? Results from a large-scale, practical,
clinical trial for patients with depression. Psychiatr. Serv. 60, 1439–45 (2009).
5. Warden, D., Rush, A. J., Trivedi, M. H., Fava, M. & Wisniewski, S. R. The STAR*D
project results: A comprehensive review of findings. Curr. Psychiatry Rep. 9, 449–459
(2007).
6. Widge, A. S. et al. Treating refractory mental illness with closed-loop brain stimulation:
Progress towards a patient-specific transdiagnostic approach. Exp. Neurol. (2017).
doi:10.1016/j.expneurol.2017.07.021
7. Ballantine, H. T., Bouckoms, A. J., Thomas, E. K. & Giriunas, I. E. Treatment of
psychiatric illness by stereotactic cingulotomy. Biol. Psychiatry 22, 807–819 (1987).
8. Shin, L. M. et al. An fMRI study of anterior cingulate function in posttraumatic stress
disorder. Biol. Psychiatry 50, 932–42 (2001).
9. Dougherty, D. D. et al. Prospective long-term follow-up of 44 patients who received
cingulotomy for treatment-refractory obsessive-compulsive disorder. Am. J. Psychiatry
159, 269–275 (2002).
10. Greenberg, B. D., Rauch, S. L. & Haber, S. N. Invasive circuitry-based neurotherapeutics:
stereotactic ablation and deep brain stimulation for OCD. Neuropsychopharmacology 35,
317–336 (2010).
11. Yang, J. C., Ginat, D. T., Dougherty, D. D., Makris, N. & Eskandar, E. N. Lesion analysis
for cingulotomy and limbic leucotomy: comparison and correlation with clinical
outcomes. J. Neurosurg. 120, 152–63 (2014).
12. Mayberg, H. S. Targeted electode-based modulation of neural circuits for depression. J.
Clin. Invest. 119, 717–725 (2009).
13. Malone, D. A. et al. Deep Brain Stimulation of the Ventral Capsule/Ventral Striatum for
Treatment-Resistant Depression. Biol. Psychiatry 65, 267–275 (2009).
14. Greenberg, B. D. et al. Deep brain stimulation of the ventral internal capsule/ventral
striatum for obsessive-compulsive disorder: worldwide experience. Mol. Psychiatry 15,
64–79 (2010).
15. Dougherty, D. D. et al. A Randomized Sham-Controlled Trial of Deep Brain Stimulation
of the Ventral Capsule/Ventral Striatum for Chronic Treatment-Resistant Depression.
Biol. Psychiatry 78, 240–248 (2015).
16. Morishita, T., Fayad, S. M., Higuchi, M.-A., Nestor, K. a & Foote, K. D. Deep Brain
50
Stimulation for Treatment-resistant Depression: Systematic Review of Clinical Outcomes.
Neurotherapeutics 11, 475–84 (2014).
17. Garnaat, S. L. et al. Who qualifies for deep brain stimulation for OCD? Data from a
naturalistic clinical sample. J. Neuropsychiatry Clin. Neurosci. 26, 81–6 (2014).
18. Ward, M. P. & Irazoqui, P. P. Evolving refractory major depressive disorder diagnostic
and treatment paradigms: toward closed-loop therapeutics. Front. Neuroeng. 3, 7 (2010).
19. Thomas, J. L., Wilk, J. E., Riviere, L. A., McGurk, D., Castro, C. A., & Hoge, C. W.
Prevalence of mental health problems and functional impairment among active component
and National Guard Soldiers 3 and 12 months following combat in Iraq. Arch. Gen.
Psychiatry 67, 614–623 (2010).
20. Hoge, C. W., Riviere, L. A., Wilk, J. E., Herrell, R. K. & Weathers, F. W. The prevalence
of post-traumatic stress disorder (PTSD) in US combat soldiers: A head-to-head
comparison of DSM-5 versus DSM-IV-TR symptom criteria with the PTSD checklist. The
Lancet Psychiatry 1, 269–277 (2014).
21. Priori, A., Foffani, G., Rossi, L. & Marceglia, S. Adaptive deep brain stimulation (aDBS)
controlled by local field potential oscillations. Exp. Neurol. 245, 77–86 (2013).
22. Bush, G. et al. Anterior cingulate cortex dysfunction in attention-deficit/hyperactivity
disorder revealed by fMRI and the Counting Stroop. Biol. Psychiatry 45, 1542–1552
(1999).
23. Swanson, J., Castellanos, F. X., Murias, M., LaHoste, G. & Kennedy, J. Cognitive
neuroscience of attention deficit hyperactivity disorder and hyperkinetic disorder. Curr.
Opin. Neurobiol. 8, 263–271 (1998).
24. Simpson, J. R., Snyder, A. Z., Gusnard, D. a. & Raichle, M. E. Emotion-induced changes
in human medial prefrontal cortex: I. During cognitive task performance. Proc. Natl.
Acad. Sci. U. S. A. 98, 683–687 (2001).
25. Whalen, P. J. et al. The emotional counting stroop paradigm: A functional magnetic
resonance imaging probe of the anterior cingulate affective division. Biol. Psychiatry 44,
1219–1228 (1998).
26. Gehring, W. J., Himle, J. & Nisenson, L. G. Action-Monitoring Dysfunction in Obsessive-
Compulsive Disorder Author ( s ): William J . Gehring , Joseph Himle and Laura G .
Nisenson Published by : Sage Publications , Inc . on behalf of the Association for
Psychological Science Stable URL : http://www. 11, 1–6 (2016).
27. Koyama, T., Kato, K., Tanaka, Y. Z. & Mikami, A. Anterior cingulate activity during
pain-avoidance and reward tasks in monkeys. Neurosci. Res. 39, 421–430 (2001).
28. Niki, H. & Watanabe, M. Prefrontal and cingulate unit activity during timing behavior in
the monkey. Brain Res. 171, 213–224 (1979).
29. Nishijo, H. et al. Single neuron responses in the monkey anterior cingulate cortex during
visual discrimination. Neurosci. Lett. 227, 79–82 (1997).
51
30. Procyk, E., Tanaka, Y. L. & Joseph, J. P. Anterior cingulate activity during routine and
non-routine sequential behaviors in macaques. Nat. Neurosci. 3, 502–508 (2000).
31. Shima, K. Role for Cingulate Motor Area Cells in Voluntary Movement Selection Based
on Reward. Science (80-. ). 282, 1335–1338 (1998).
32. Stroop, J. R. APA CENTENNIAL FEATURE Studies of Interference in Serial Verbal
Reactions. 121, 15–23 (1992).
33. Eriksen, B. A. & Eriksen, C. W. Effects of noise letters upon the identification of a target
letter in a nonsearch task. Percept. Psychophys. 16, 143–149 (1974).
34. Simon, J. R. & Berbaum, K. Effect of conflicting cues on information processing: The
‘Stroop effect’ vs. the ‘Simon effect’. Acta Psychol. (Amst). 73, 159–170 (1990).
35. Bush, G., Shin, L. M., Holmes, J., Rosen, B. R. & Vogt, B. a. The Multi-Source
Interference Task: validation study with fMRI in individual subjects. Mol. Psychiatry 8,
60–70 (2003).
36. Bush, G. & Shin, L. M. The Multi-source interference task: An fMRI task that reliably
activates the cingulo-frontal-parietal cognitive/attention network. Nat. Protoc. 1, 308–313
(2006).
37. Mayberg, H. S. et al. Reciprocal limbic-cortical function and negative mood: Converging
PET findings in depression and normal sadness. Am. J. Psychiatry 156, 675–682 (1999).
38. Gusnard, D. A., Akbudak, E., Shulman, G. L. & Raichle, M. E. Medial prefrontal cortex
and self-referential mental activity: relation to a default mode of brain function. Proc.
Natl. Acad. Sci. U. S. A. 98, 4259–64 (2001).
39. Bush, G., Luu, P. & Posner, M. I. Cognitive and emotional influences in anterior cingulate
cortex. 4, 215–222 (2000).
40. Drevets, W. C. & Raichle, M. E. Reciprocal suppresion of regional cerebral blood flow
during emotional versus higher cognitive processes: Impliation for interaction between
emotion and cognition. Cogn. Emot. 12, 353–385 (1998).
41. Davey, C. G., Yücel, M., Allen, N. B. & Harrison, B. J. Task-related deactivation and
functional connectivity of the subgenual cingulate cortex in major depressive disorder.
Front. Psychiatry 3, 1–8 (2012).
42. Bush, G. et al. Functional magnetic resonance imaging of methylphenidate and placebo in
attention-deficit/hyperactivity disorder during the multi-source interference task. Arch.
Gen. Psychiatry 65, 102–114 (2008).
43. Cocchi, L. et al. Functional alterations of large-scale brain networks related to cognitive
control in obsessive-compulsive disorder. Hum. Brain Mapp. 33, 1089–1106 (2012).
44. Yücel, M. et al. Functional and Biochemical Alterations of the Medial Frontal Cortex in
Obsessive-Compulsive Disorder TL - 64. Arch. Gen. Psychiatry 64 VN-r, 946–955
(2007).
45. McKenna, F. P. Effects of unattended emotional stimuli on color-naming performance.
52
Curr. Psychol. 5, 3–9 (1986).
46. Mathews, A. & Macleod, C. Selective Cues Processing in Anxiety of Threat. Behav. Res.
Ther 23, 563–569 (1985).
47. Etkin, A., Egner, T., Peraza, D. M., Kandel, E. R. & Hirsch, J. Resolving Emotional
Conflict: A Role for the Rostral Anterior Cingulate Cortex in Modulating Activity in the
Amygdala. Neuron 51, 871–882 (2006).
48. Etkin, A., Prater, K. E., Hoeft, F., Menon, V. & Schatzberg, A. F. Failure of anterior
cingulate activation and connectivity with the amygdala during implicit regulation of
emotional processing in generalized anxiety disorder. Am. J. Psychiatry 167, 545–554
(2010).
49. Etkin, A. & Schatzberg, A. F. Common abnormalities and disorder-specific compensation
during implicit regulation of emotional processing in generalized anxiety and major
depressive disorders. Am. J. Psychiatry 168, 968–978 (2011).
50. Hull, A. M. Neuroimaging findings in post-traumatic stress disorder. Systematic review.
Br. J. Psychiatry 181:102-10, 102–110 (2002).
51. Kumari, V. et al. Neural abnormalities during cognitive generation of affect in treatment-
resistant depression. Biol. Psychiatry 54, 777–791 (2003).
52. Etkin, A. et al. Toward a neurobiology of psychotherapy: basic science and clinical
applications. J. Neuropsychiatry Clin. Neurosci. 17, 145–58 (2005).
53. Stephen, E. P. et al. Assessing dynamics, spatial scale, and uncertainty in task-related
brain network analyses. Front. Comput. Neurosci. 8, 31 (2014).
54. Bastos, A. M. & Schoffelen, J.-M. A Tutorial Review of Functional Connectivity Analysis
Methods and Their Interpretational Pitfalls. Front. Syst. Neurosci. 9, 175 (2016).
55. Sellers, E. W., Vaughan, T. M. & Wolpaw, J. R. A brain-computer interface for long-term
independent home use. Amyotroph. Lateral Scler. 11, 449–455 (2010).
56. Zeitlin, D., Wolpaw, J. R. & Vaughan, T. M. Amyotrophic Lateral Sclerosis. (2015).
doi:10.3109/21678421.2013.865750.Brain-computer
57. Gadhoumi, K., Lina, J. M., Mormann, F. & Gotman, J. Seizure prediction for therapeutic
devices: A review. J. Neurosci. Methods 260, 270–282 (2016).
58. Pandarinath, C. et al. High Performance communication by people with paralysis using an
intracortical brain-computer interface. 1–27 (2016). doi:10.7554/eLife.18554
59. Jarosiewicz, B. et al. Virtual typing by people with tetraplegia using a stabilized, self-
calibrating intracortical brain-computer interface. IEEE BRAIN Gd. Challenges Conf.
Washington, DC 7, 1–11 (2014).
60. Dangi, S., Orsborn, A. L., Moorman, H. G. & Carmena, J. M. Design and Analysis of
Closed-Loop Decoder Adaptation Algorithms for Brain-Machine Interfaces. Neural
Comput. 25, 1693–1731 (2013).
53
61. Vansteensel, M. J. et al. Fully Implanted Brain–Computer Interface in a Locked-In Patient
with ALS. N. Engl. J. Med. NEJMoa1608085 (2016). doi:10.1056/NEJMoa1608085
62. Little, S. et al. Adaptive deep brain stimulation in advanced Parkinson disease. Ann.
Neurol. 449–457 (2013). doi:10.1002/ana.23951
63. Rosin, B. et al. Closed-loop deep brain stimulation is superior in ameliorating
parkinsonism. Neuron 72, 370–384 (2011).
64. Sun, F. T., Morrell, M. J. & Wharen, R. E. Responsive cortical stimulation for the
treatment of epilepsy. Neurotherapeutics 5, 68–74 (2008).
65. Afshar, P. et al. A translational platform for prototyping closed-loop neuromodulation
systems. Front. Neural Circuits 6, 117 (2012).
66. Khanna, P. et al. with Downloadable Firmware Upgrades. (2015).
67. Kleiner, Mario; Brainard, David; Pelli, Denis; Ingling, Allen; Murray, Richard; Broussard,
C. What’s new in Psychtoolbox-3. Perception 36, 1 (2007).
68. Pelli, D. G. The VideoToolbox software for visual psychophysics: transforming numbers
into movies. Spatial Vision 10, 437–442 (1997).
69. Brainard, D. The Psychophysics Toolbox. Spat. Vis. 10, 433–436 (1997).
70. Dykstra, A. R. et al. Individualized localization and cortical surface-based registration of
intracranial electrodes. Neuroimage 59, 3563–3570 (2012).
71. Desikan, R. S. et al. An automated labeling system for subdividing the human cerebral
cortex on MRI scans into gyral based regions of interest. Neuroimage 31, 968–980 (2006).
72. Fischl, B. FreeSurfer. Neuroimage 62, 774–781 (2012).
73. Oostenveld, R., Fries, P., Maris, E. & Schoffelen, J. M. FieldTrip: Open source software
for advanced analysis of MEG, EEG, and invasive electrophysiological data. Comput.
Intell. Neurosci. 2011, (2011).
74. Fair, D. A. et al. A method for using blocked and event-related fMRI data to study ‘resting
state’ functional connectivity. Neuroimage 35, 396–405 (2007).
75. Galar, M., Fernandez, A., Barrenechea, E., Bustince, H. & Herrera, F. A review on
ensembles for the class imbalance problem: Bagging-, boosting-, and hybrid-based
approaches. IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. 42, 463–484 (2012).
76. Rakotomamonjy, A. Variable Selection Using SVM-based Criteria. J. ofMachine Learn.
Res. 3, 1357–1370 (2003).
54
APPENDIX I: SUPPLEMENTAL METHODS Table 2: Bipolar electrode localization summary for MSIT. Numbers indicate the number of bipolar electrodes included within
each region label.
MG Label 79 86 89 90 95 96 99 102 104 106 107
Left
Accumb. 2 1 1
Amyg. 2 4 4 4 5 2 5 5 2 4
Caudate 7 3 2 7 7
Hipp 1 5 2 5 7 4 10 6 11
dACC 4 1 1 5 3 5 4 2 6
dlPFC 12 8 10 9 15 15 31 16 43 33 35
dmPFC 7 1 3 3 2 19 1 4 6
Insula 1 2 3
lOFC 3 4 2 7 8 7 8 8 4 8
mOFC 2 2 2 1 2 1 3 1 1 3 1
paraHipp 3
post. CC 1
raCC 3 2 3 1 3
Temporal 4 6 6 7 10 19 22 17 13 19 22
vlPFC 1 1 6 9 12 4 12 1 4 4
Right
Accumb. 3 2
Amyg. 4 1 5 5 2 5 5
Caudate 1 8 6 6
Hipp 2 5 6 3 7 3 1 6 11 7
dACC 2 1 1 6 2 2 7
dlPFC 6 6 14 10 23 12 7 1 10 24 20
dmPFC 3 7 3 6 12 3 1 1 8
Insula 2
lOFC 3 5 3 4 8 8 2 13 3 10 7
mOFC 3 1 2 1 2 2
paraHipp 3
post. CC 1
raCC 3 2 1 1 1 3
Temporal 6 4 5 10 12 16 16 23 10 18 8
vlPFC 7 3 3 4 12 3 12 15 2 5 14
Total 61 69 64 83 150 141 139 147 120 162 189
55
Table 3: Bipolar electrode localization summary for the ECR task. Numbers indicate the number of bipolar electrodes included
within each region label.
MG 85 86 88 89 90 96 99 101 102 104 106 107
Left
Accumb.
1
Amyg. 3 4 4 4 4 2 5 3 5
2 4
Caudate
7 3 10 2 7
5
Hipp 2 5 3 3 5 4 10 4
6 11
dACC 3 2 1 1 5
5
4 3 6
dlPFC 9 12 11 10 9 15 31 17 16 43 34 35
dmPFC 2 6 3 1
3 2 10 19 1 4 6
Insula
2 3
lOFC 1 4 4
2 8 7 10 8 8 4 8
mOFC
2 1 1 1 1 3 4 1 1 3 1
paraHipp
3
post. CC
2 1
raCC 1
1
2 3
3
Temporal 5 7 5 6 8 17 22 17 13 13 21 22
vlPFC 2 1
6 12 4 6 11 1 4 4
Right
Accumb.
2
Amyg. 2 2 3 1
5
2 5 5
Caudate 1
1
6
6
Hipp 3 3 2 5 6 7 3
1 6 11 7
dACC 2
2
1 6 1
2 2 7
dlPFC 6 6 9 14 10 12 7 2 1 10 22 20
dmPFC 3 7 5 3 6 4 1 10
1 8
Insula
2
lOFC 2 4 6 3 4 8 2 15 13 3 10 7
mOFC 1
1
1
4 2 1 1 2
paraHipp
3
post. CC 2
3
raCC 2
2
1 1 1
3
Temporal 8 4 7 5 10 14 16
21 10 17 8
vlPFC 5 2
3 4 3 12 8 15 2 5 13
Total 65 71 71 64 84 133 139 122 139 119 162 186
56
APPENDIX II: INVENTION DISCLOSURE Title of the invention: A method for classifying a subject's mental state from electrical brain
activity
Inventors: Nicole Provenza1,2,3, Emad Eskandar3, Darin Dougherty4, Alik Widge4**, David
Borton1,5,6*
Affiliations: 1Brown University School of Engineering, 2Draper, 3Massachusetts General
Hospital Neurosurgery Research, 4Massachusetts General Hospital Psychiatry, 5Brown Institute
for Brain Science, 6Department of Veterans Affairs Center for Neurorestoration and Neurology
Date of Invention Disclosure submission: April, 14th, 2017
Essence of the invention
Engaging in effortful mental action, such as trying to decide between conflicting choices or
suppress an unwanted emotion, activates specific brain networks. This activity differentiates one
behavior from another. Detecting these specific brain states might help us understand how the
brain becomes dysfunctional in neuropsychiatric illness, including but not limited to anxiety,
depression, bipolar disorder, substance abuse, attention deficit disorder, post-traumatic stress
disorder, and traumatic brain injury. It would be possible to identify when a patient is trying to
do a critical mental task, but failing. An intervention, like deep brain stimulation (DBS) could
then be applied to augment the brain's function during that specific effort. The problem is that
there are no documented algorithms for monitoring the brain's electrical activity at a network
level and classifying what the patient is trying to do.
We here propose a novel approach to accurately predict task engagement by harnessing
functional connectivity between brain regions, in tandem with a classifier. Our results show that
we are able to accurately distinguish task engagement from free behavior in humans.
Attached please find a manuscript that discusses the proposed algorithm that has been used to
distinguish task engagement from free behavior using invasive local field potential recordings.
The specific instantiation is conflict tasks and invasive brain recordings, but we believe this
could be extended to almost any mental process and likely to non-invasive recording methods
such as EEG.
A disclosure form has also been submitted to Massachusetts General Hospital, where David
Glass will be leading the patent prosecution.
Novelty and major advantages
We are taking a known technique, canonical correlation analysis, applying it to neural data to
extract neural features, and using these neural features to detect task-related brain states. Based
on classification performance criteria, we identify the specific regional connectivity that
distinguishes task-related brain states. That is, we can now read the brain's network activity,
expressed as the degree to which each pair of regions activates/de-activates in a correlated way.
From that network map, we can tell not what someone is thinking, but what he/she is trying to
do. To our knowledge, this has not been done before.
57
The most immediate use case is closed-loop DBS for psychiatric purposes. We have algorithms
that can change behavior using DBS during standard cognitive tasks. This new invention adds a
missing piece -- telling when a patient is in a "task-like state" or "vulnerable network state".
When we detect that, we would then apply our existing brain stimulation algorithms. Basically,
this new algorithm would detect when a patient is trying to do something with which he/she
tends to need help, then would apply that help.
Each of these network decoders turns out to rely on a small subset of brain regions to guide
accurate classification. The subset of important features we have identified per patient could be
useful for guiding personalized implant locations. If we know that network activity in only these
regions is sufficient to understand someone's mental state, then we should target stimulating and
recording implants to those regions.
We imagine that detecting cognitive brain states could have implications beyond treating
refractory neuropsychiatric illness. For example, knowledge of cognitive state could be insightful
during monitoring of coma patients, rehabilitating cognitive function after strokes, gauging
awareness and emotional state of soldiers, or identifying vulnerable periods for addicts.
Top Related