Joint Brain Connectivity Estimation from Diffusion andFunctional MRI Data

Shu-Hsien Chu, Christophe Lenglet and Keshab K. Parhi

University of Minnesota, Minneapolis, MN, USA

ABSTRACT

Estimating brain wiring patterns is critical to better understand the brain organization and function. Anatomicalbrain connectivity models axonal pathways, while the functional brain connectivity characterizes the statisticaldependencies and correlation between the activities of various brain regions. The synchronization of brainactivity can be inferred through the variation of blood-oxygen-level dependent (BOLD) signal from functionalMRI (fMRI) and the neural connections can be estimated using tractography from diffusion MRI (dMRI).Functional connections between brain regions are supported by anatomical connections, and the synchronizationof brain activities arises through sharing of information in the form of electro-chemical signals on axon pathways.Jointly modeling fMRI and dMRI data may improve the accuracy in constructing anatomical connectivity aswell as functional connectivity. Such approach may lead to novel multimodal biomarkers potentially able tobetter capture functional and anatomical connectivity variations.

We present a novel brain network model which jointly models the dMRI and fMRI data to improve theanatomical connectivity estimation and extract the anatomical subnetworks associated to specific functionalmodes by constraining the anatomical connections as structural supports to the functional connections. Thekey idea is similar to a multi-commodity flow optimization problem that minimizes the cost or maximizes theefficiency for flow configuration and simultaneously fulfills the supply-demand constraint for each commodity. Inthe proposed network, the nodes represent the grey matter (GM) regions providing brain functionality, and thelinks represent for white matter (WM) fiber bundles connecting those regions and delivering information. Thecommodities can be thought of as the information corresponding to brain activity patterns as obtained for instanceby independent component analysis (ICA) of fMRI data. The concept of information flow is introduced andused to model the propagation of information between GM areas through WM fiber bundles. The link capacity,i.e., ability to transfer information, is characterized by the relative strength of fiber bundles, e.g., fiber countgathered from the tractography of dMRI data. The node information demand is considered to be proportionalto the correlation between neural activity at various cortical areas involved in a particular functional mode (e.g.visual, motor, etc.). These two properties lead to the link capacity and node demand constraints in the proposedmodel. Moreover, the information flow of a link cannot exceed the demand from either end node. This is capturedby the feasibility constraints. Two different cost functions are considered in the optimization formulation in thispaper. The first cost function, the reciprocal of fiber strength represents the unit cost for information passingthrough the link. In the second cost function, a min-max (minimizing the maximal link load) approach is used tobalance the usage of each link. Optimizing the first cost function selects the pathway with strongest fiber strengthfor information propagation. In the second case, the optimization procedure finds all the possible propagationpathways and allocates the flow proportionally to their strength. Additionally, a penalty term is incorporatedwith both the cost functions to capture the possible missing and weak anatomical connections. With this set

Work partly supported by NIH grants P41 EB015894, P30 NS076408, R01 EB008432.Data were provided in part by the Human Connectome Project, WU-Minn Consortium (Principal Investigators: DavidVan Essen and Kamil Ugurbil; 1U54MH091657) funded by the 16 NIH Institutes and Centers that support the NIHBlueprint for Neuroscience Research; and by the McDonnell Center for Systems Neuroscience at Washington University.

Further author information: (Send correspondence to S.C.)S.C: E-mail: chuxx214@umn.edu, Telephone: 1 612 501 3204C.L.: E-mail: clenglet@umn.edu, Telephone: 1 612 625 7576K.K.P.: E-mail: parhi@umn.edu, Telephone: 1 612 624 4116

of constraints and the proposed cost functions, solving the network optimization problem recovers missing andweak anatomical connections supported by the functional information and provides the functional-associatedanatomical subnetworks.

Feasibility is demonstrated using realistic diffusion and functional MRI phantom data. It is shown thatthe proposed model recovers the maximum number of true connections, with fewest number of false connectionswhen compared with the connectivity derived from a joint probabilistic model using the expectation-maximization(EM) algorithm presented in a prior work.1 We also apply the proposed method to data provided by the HumanConnectome Project (HCP).

Keywords: Brain connectivity, functional MRI, diffusion MRI, network, tractography, information flow

1. INTRODUCTION

Human brains are known be a large-scale neuron network which is complex, hierarchical and can be decomposedaccordingly at different scales. Brain connectivity refers to the pattern of anatomical links (anatomical connec-tivity), statistical dependencies (functional connectivity) and causal interactions (effective connectivity) whichcan be determined by observing structural links such as fiber pathways, by representing statistical relationshipsmeasured as cross-correlations and coherence, and causal relationship, respectively, between parcels of the brain.However, the relationship between connectivity pattern and brain functionality is not well understood. Thus,understanding the cognitive mechanisms and functioning of the brain from connectivity patterns is of significantinterest.2–5

In a brain neuron system, signal is processed in the grey matter (GM) area and transmitted between GMregions through axons located mainly in white matter region (WM). Functional MRI (fMRI) measures the bloodoxygen level dependent (BOLD) signal that describes the contrast of oxygen consumption over time and iscorrelated to neural activities.6 fMRI data is often used to analyze and correlate the neuronal activity betweenregions, and to construct the functional connectivity pattern. Several methods have been widely used to generateand analyze the functional connectivity:7 seed-based correlation,8 independent component analysis (ICA)9 andgraph-theoretic network analysis.10 Moreover, the WM microstructure can be captured non-invasively throughdiffusion MRI (dMRI) which relies on the water molecule anisotropic diffusion along axons. Diffusion MRI hasbeen used to estimate brain connectivity pattern in many clinical applications11 such as Alzheimers disease,12

neuroscience research, and the construction of the human connectome.13 Furthermore, many neurological diseasesresult from abnormal changes of brain anatomical connectivity.12,14 However, dMRI and associated analysismethods such as tractography15 suffer from certain limitations which restrict the accuracy with which fiberpathways can be estimated, thereby possibly leading to missed connections. In the complex area, a major partof connections is missed, or weakened.

Despite significant advances in multimodal imaging techniques and analysis approaches, unimodal studiesare still the most common way to investigate brain changes or group differences. It has been found that thefunctional connectivity is significantly shaped by the anatomical structure.16–18 More specifically, brain regionswith high anatomical connectivity usually have high functional connectivity as well,16–18 but the converse is notnecessarily true. Extracting information jointly from each both dMRI and fMRI may lead to more accurateconnectivity.18,19 Recently, a joint connectivity approach under a probabilistic framework using expectation-maximization algorithm has been proposed for a group study1 and a functional connectivity pattern estimationbased on sparse Gausian Graphical Model penalized by structural connectivity has been presented.20

In this paper, we propose a novel network flow model to improve the estimation of brain connectivity patternbased on joint analysis of dMRI and fMRI. In the model, the nodes represent brain functional regions, the linksrepresent connections between regions, and the flow models the information propagation between functionalregions through structural pathways. The capability for transmitting information on each link, i.e., link capacityconstraint, is defined as the fiber count obtained from dMRI data and tractography; the demand of informationon each regions, i.e., demand constraint for each region, for each functional mode is computed from the fMRIICA result. In the ICA framework, the fMRI data corresponding to a brain function is decomposed into manyspatio-temporal components, i.e., functional modes, where modes relate to specific brain functions and the spatialmap provides the spatial distribution for each mode associated with the temporal activities. Through the spatial

distribution, the amount of information needed to trigger the neural activities is estimated and considered as ademand. In the network, the active regions within a functional mode share information. However, the amount ofshared information cannot exceed the demand from either end node. It is modeled by feasibility constraints. Asolution to the network flow problem based on three sets of constraints leads to structural connectivity patternswhich are supporting the functional network. The proposed network flow model is validated on a diffusion andfunctional MRI phantom data with various structural complexities, i.e., fiber crossing, to show the improvement.

This paper is organized as follows. Section 2 describes the data and processing pipeline used in this paper.Section 3 presents the proposed network flow model and network analysis techniques and Section 4 presents theconnectivity results derived from the proposed model.Comparison of the brain connectivity with prior work isalso presented in Section 4.

2. DATA ACQUISITION AND PROCESSING

A realistic phantom, Fiber Cup,21,22 dMRI data set with known fiber layout and synthetic fMRI data that areperfectly aligned with the true connectivity pattern is first used to verify the model, demonstrate the improvementfrom combining fMRI data and compare the performance with existing methods. For brain connectivity patternstudy, two MRI data sets are used in this paper: one from the phantom and the other from the Human ConnectomProject (HCP).

2.1 Phantom Data Set

Figure 1. The phantom21,22 data set: (a) Ground truth fiber connections used in dMRI scanning; (b) b0 image and theparcellation of 16 regions, P1, P2, . . . , and P16, centered at the seed voxels given in the 2009 MICCAI Fiber Cup contest.The color of the regions represent the true connectivity pattern. In other words, the regions sharing the same color areanatomically connected to each other. (c) ground truth anatomical connections in a network formation with regions asnodes and connections as edges (Generated using BrainNet Viewer23).

The phantom21,22 used in the FiberCup tractography challenge in the 12th International Conference onMedical Image Computing and Computer Assisted Intervention (MICCAI) in London in 2009 and made pub-licly available on the webpage (http://www.lnao.fr/spip.php?rubrique79) of LNAO laboratory is used forvalidating the result and comparing performance. Ground truth for 16 fibers shown in Figure 1 and seed pointswhich are centered at each region in Figure 1 (b) are available with the data. The phantom comprises sevendistinct bundles, and contains 3 crossings, 1 kissing, and 3 bundles that split. The various crossing, splittingand kissing fiber configurations simulate a coronal section of the human brain and can be used to evaluate thequality of connectivity estimation. The network presentation of true anatomical connections in Figure 1 (c)is generated using BrainNet Viewer23 by taking the regions as nodes and connections as edges. The networkcontains 5 separated sub-networks which coincide with the true connectivity pattern.

2.1.1 Diffusion-weighted MRI acquisitions

Diffusion-weighted data of the phantom21,22 are 3mm isotropic with 64 directions uniformly distributed over thesphere. The data with b = 2000 were used in this paper.

2.1.2 Functional MRI simulation

The fMRI BOLD signal on each voxel were simulated using a modification of “Generate synthetic fMRI data”function published on http://www.mathworks.com/matlabcentral/fileexchange/. A design matrix shown inFigure 2a (upper right panel) consisting in 5 randomly generated and minimally correlated binary stimulatingsequences was first generated. The sequences were assigned to the regions based on the rule that regions in eachconnected pattern share the same stimulating sequence.

Then, the BOLD signal in Figure 2a (bottom panel) is obtained from the convolution of the design matrixwith hemodynamic response function illustrated in Figure 2a (upper left panel). The BOLD signals were spreadout spatially with exponential decade and additive white Gaussian noise was added to the whole fMRI data setwith average SNR equal to 5.36. The imaging parameters for the simulated data are TR = 720ms and 1200time points. In order to verify the largest improvement we may gain from the joint analysis, the fMRI datais designed to have a fixed functional connectivity which coincides with the true anatomical connectivity. Thevalidations are shown in Figure 2 (b) and (c).

Figure 2. The fMRI time courses were simulated according to the 5 true anatomical connectivity patterns. They wereassigned with a stimulating pattern encoded in the design matrix, (a) upper right panel. The BOLD signal, (a) bottompanel, for each region is then given by the convolution of hemodynamic response function, (a) upper left panel, andthe design matrix. Figure (b) shows simulation result at t = 9, 34, 44, 76. It demonstrates that the fMRI signals aresynchronized for the connected regions. It can also be verified by checking the correlation, shown in (c), of signalsbetween each region where thicker the edge, stronger the correlation. Nodes are strongly correlated only when theyare anatomically connected, i.e., in the same color, otherwise they are minimally correlated. The numbers on the linksrepresent the correlation coefficients.

2.2 Human Connectome Project Data Set

The HCP24,25 data is publicly available on its website (http://www.humanconnectome.org/data) with multipleimaging modalities including diffusion imaging (dMRI), resting-state fMRI (R-fMRI), task-evoked fMRI (T-fMRI), T1- and T2-weighted MRI which can be used to characterize brain connectivity and function and theirvariability in healthy adults. T1-weighted MRI, diffusion-weighted MRI, and resting-state functional MRI dataare used for constructing brain networks in our study.

Data was acquired on a 3T Siemens scanner using the following parameters for fMRI: TR = 720ms, TE =33.1ms, FA = 52◦, 2 × 2 × 2mm voxels, FOV = 208 × 180mm, and 72 oblique axial slices alternated betweenphase encoding in a right to left direction in one run and phase encoding in a left to right direction in theother run.26–29 Each functional run lasted 14.55 min (1200 time points). Diffusion data sets were acquiredwith 1.25 × 1.25 × 1.25mm voxels, FOV = 210mm, 111 slices, TE = 89.50ms, TR = 5520ms, b-value=1000, 2000, 3000s/mm2, 90 directions per b-values and 18 non-diffusion-weighted volumes.

2.3 Data Preprocessing

The anatomical connectivity estimated from dMRI data is referred to as the link capacity Dl and the regioninformation requirement, Rmi , is calculated from the spatial maps in ICA of the fMRI data and represents how

significant the activity is in that region corresponding to the mth mode. Both link capacity Dl and regioninformation requirement Rmi are the inputs of the proposed model. The details regarding the model will beprovided in the next section. Here, we first introduce the process for getting these input data.

2.3.1 Diffusion

Data were preprocessed using the default HCP preprocessing pipeline (v.3.4),30 which includes correction forsusceptibility, motion and eddy current distortions.

In the preprocessing step, the DWI data were corrected for eddy current distortions and local fiber orientationswere estimated using bedpostx.31 Probabilistic fiber tracking with 5000 samples in each seeded voxel and distancecorrection is performed to generate the brain tractography. The number of tracks connecting two regions, Dl,was normalized by the number of samples and seeded region volume and taken as the fiber strength acting asthe capacity of each link in the network model.

2.3.2 rfMRI preprocessing

We utilized the fMRI preprocessed data released by the HCP.30 Before processing, the first 21 time stamps offMRI data were discarded. Then, high pass filtering with a cutoff frequency 1/60 Hz, motion correction, 4mmspatial smoothing, and FILM pre-whitening were applied using FSL.31

Spatial Independent Component Analysis (spatial ICA)

The spatial independent component analysis (ICA), using FSL melodic,32 is performed to decompose the brainfMRI into spatial independent components which are composed from correlation maps and temporal series. Thedecomposition is illustrated in Figure 3. The observed fMRI data is arranged in a time by voxel matrix Y whichrepresents the combination of functional components. Each component is associated with a time series which isa column of T and a spatial map that is described by a row of M . Here, the maps are spatially independentand do not overlap with others. From a matrix multiplication perspective, the absolute component value in aspatial map can be considered as the weight factor corresponding to a time course for each voxel. Therefore,these weights in the ith region are used to generate Rmi as the amount of information associated to mode m. Themeaningful components were selected automatically33 based on the skewness of the histogram of the spatial mapand the spectrum analysis of the corresponding temporal signal from the noise components. Furthermore, thespatial map outputs were refined by applying the existing parcellation and classifying each voxel into active orinactive status using K-means. For each selected mode, the regional mean correlation level was calculated fromthe absolute value from the voxels in the region. The levels are further used for identifying the activation statusof the region by K-means.34 The functional demand Rmi in the model was set to the mean correlation level foractive regions, otherwise the Rmi was set to zero.

Figure 3. Functional MRI data organized in a time × space matrix can be decomposed into the multiplication of twomatrices. One contains the mean temporal signals and the other matrix contains the spatial independent maps whichillustrate the correlation between the BOLD signals in each voxel and the temporal signal.

3. BRAIN NETWORK MODEL

The brain can be viewed as a multi-commodity supporting network with replicable goods. The analogies aremade between: 1) GM region and commodity factories, 2) WM fiber and delivering pathways, and 3) commodityand information. The brain function is accomplished by the interconnected neurons and as a result of information

processing and exchange. The neuronal cell bodies are majorly responsible for information processing and mostlylocated in the grey matter (GM). The parcels in GM are the nodes in the brain network, similar to the factoriesin a commodity network. Information is the commodity and the axon fiber bundles, the links in a network, in thewhite matter (WM) are the exchanging pathways which are similar to the delivering pathway in a commoditynetwork. Combining functional and structural MRI improves the estimation of anatomical connectivity bymaintaining the supply-demand relationship. Assuming that 1) a neuron reacts to the received signal, 2) aneuron doesn’t produce signal with no excitation, and 3) the synchronization is positively correlated to theamount of received signal and the capability to generate new signal, the underestimated fibers including missingconnections can be recovered from retrieved fMRI data.

Table 1. Definition for variables, parameters and symbols in the information network model

Symbol Description

fml Information flow variables for mode m on link lPl Comprehensive anatomical network variable that captures the missing connections l in

Dl networkRmi The region information demand regarding to mth mode and ith regionDl The capacity of link l for delivering informationl = (i, j) The connection l with associated end regions i and jN(i) The collection of links associated with region iγ Unit conversion between information and capacityβ Information sharing rateρ Penalty parameter for Pl

In the proposed network flow model, the nodes represent brain functional regions and the links between thenodes represent potential fiber connections. The information flow is used to describe the information propagation.The flow of information through the links is influenced by the strength of the anatomical connectivity measuredby dMRI: the stronger the connection, the larger the flow can be. The demand associated with a node is modeledseparately for each functional mode of the fMRI. The notations used in this paper to describe the network flowformulation are summarized in Table 1. Let fml represent the information flow through link l associated withfunctional mode m, the capacity of a link be denoted as Dl, and the information demand associated with nodei corresponding to functional mode be denoted by Rmi . The modes corresponding to the independent temporal-spatial components of the fMRI are obtained by independent component analysis (ICA). The proposed networkmodel for brain connectivity is described by equations (1)-(5).

Input: Dl, Rmi , ρ, γ ; Output: fml , Pl

minfml, Pl

L∑l=1

[1Dl

(M∑m=1

fml

)+ ρ(1 +Dl)Pl

](1)

s.t. ∑l∈N(i)

fml ≥ Rmi ∀m = 1, 2 . . .M, i = 1, 2, . . . N (2)

M∑m=1

fml ≤ γ(Dl + Pl) ∀l = 1, 2, . . . L (3)

fml ≤ βmin{Rmi , Rmj } where l = (i, j), ∀l = 1, 2, . . . L (4)

0 ≤ fml , 0 ≤ Pl ∀m, l (5)

The objective function described in (1) minimizes the overall cost for link usage and the unit cost is de-fined by the reciprocal of capacity, i.e., the summation of link usage cost which is the summed informationflow multiplied by the unit cost. The second term of the objective function minimizes the penalty for using

strength-underestimated links. Incorporating the reciprocal of link capacity as part of the cost function favors aconnectivity network where the links with larger capacity (strong links) have higher probability to be selected.

The constraints imposed by fMRI and dMRI are captured by the three constraints (2)-(4). These constraintsinclude: information demand constraint, link capacity constraint, and feasibility constraint.

Information demand constraint: This constraint, described by (2), states that for each functional mode ofthe fMRI, the gathered information at a region from all neighboring links must exceed the information demandfor that mode. The information demand can be interpreted as the signal needed to have resulting neural activityobserved in fMRI. Moreover, the received information is replicable. It can be further shared with other regions.Thus, the region information demand represents a lower bound on the sum of information on all links passingthrough the region, for a specific functional mode of the fMRI.

Link flow capacity constraint: This constraint, described by (3), states that the total flow through a linkshould not exceed its capacity obtained from the dMRI. Thus, the link capacity represents an upper bound onthe link information flow. In (3), the parameter γ is necessary for unit conversion between information andcapacity, i.e., fiber count.

Feasibility constraint: This constraint, described by (4), states that the link flow for a specific mode m ona link l must be less than the minimum of the information demand corresponding to the same mode of thetwo regions associated with the link. From the sharing prospective, the shared information cannot exceed theexisting information at the sharing end; on the other node, the obtained information will result in the stimuluswhich the demand is based on, thus the shared information cannot exceed the demand at the receiving end aswell.

minf11 ,f

12 ,f

13

1D1f11 + 1

D2f12 + 1

D3f13 (6)

s.t.

f11 + f12 ≥ R11, f

11 + f13 ≥ R1

2, f12 + f13 ≥ R1

3 (7)

f11 ≤ γD1, f12 ≤ γD2, f

13 ≤ γD3 (8)

f11 ≤ min{R11, R

12}, f12 ≤ min{R1

1, R13}, f13 ≤ min{R1

2, R13} (9)

0 ≤ f11 , f12 , f13 (10)Figure 4. A simple example has one mode m = 1, three regions with R1

1 = R12 = R1

3 = 10, and three undirected links withcapacity D1 = 10, D2 = 100, D3 = 100.

Example: To illustrate the proposed network flow model, we consider a brain network containing three regionswith input parameters shown in Figure 4. The network flow formulation for this example is given by (6)-(10).

In this example, we assume P1 = P2 = P3 = 0. The solution to this network flow problem is given by f11 =0, f12 = 10, f13 = 10 and the corresponding value of objective function is 0.2. Note that f11 = 5, f12 = 5, f13 = 5leads to the cost function of 0.6. Intuitively the solution favors links with higher capacity.

These optimization problems can be solved easily by commercial off-the-shelf software once Rml and Dl areobtained from the ICA result of the fMRI and the analysis of the dMRI, respectively. The proposed networkmodel has the following properties:

• the missing and weak anatomical connections can be discovered from Pl;

• the more complete anatomical network can be formed from Dl + Pl;

• the anatomical subnetwork for each individual functional mode m can be extracted from fml called thefunctional-associated anatomical subnetwork;

• the links with higher anatomical connectivity have higher probability to be selected.

4. RESULTS AND DISCUSSION

We first present the phantom validation results for verifying the true connections and comparing the performanceto an existing work.1 Then, the brain example based on HCP data will be presented in the next section.

4.1 Phantom Validation

This section presents connectivity results based on three methods: 1) anatomical connectivity defined by fibercount from probabilistic fiber tracking using PROBTRACKX231 and the dMRI data only, 2) the proposednetwork flow model using dMRI and fMRI data jointly, and 3) a probabilistic framework with Estimation-Maximization (EM) algorithm.1 Both separate and joint frameworks1 are implemented and the results arecompared.

(a) (b)Figure 5. (a) Tractography constructed by TrackVis;35 (b) The anatomical network generated through probabilistictracking with 5000 samples per seed voxel. The numbers on edges represent the numbers of tracks connecting thecorresponding pairs of regions. Note the false-positive links from regions P1, P2, P3, P4, and P5 to region P16 and fromregions P7, P8, P10, and P11 to region P9; the missing link P5-P7; and weak connections P6-P9, P9-P12, and P11-P15.

In probabilistic tracking, Monte Carlo simulations with, typically, 5000 samples are performed and the numberof streamlines can be interpreted as the probability and the strength of the anatomical connectivity as shownin Figure 5 (b). This network contains false links from regions P1, P2, P3, P4, and P5 to region P16 and fromregions P7, P8, P10, and P11 to region P9, since these connections are not part of the ground truth shownin Figure 1 (b). These links are therefore artifacts due to the diffusion pre-processing step. Furthermore, theconnection P5-P7 is missing and connections P6-P9, P9-P12, and P11-P15 are weak. In summary, probabilistictracking leads to some inaccurately estimated connections; therefore, without other information or refinementwe can only trade off between false connections and the absence of true connections.

Figure 6 shows the connectivity derived from the proposed network flow model for all 5 functional modes.Based on Rmi and Dl found from fMRI data and dMRI data, the network flow model was solved and the linkflow fml are shown in Figure 6 where the values of the flow correspond to the thicknesses of the links.

The connectivity network and the part of the network that is missing from the ground truth are shown inFigures 7 (a) and (b), respectively. Figure 7 (a) is obtained by combining the connectivity networks of 5 modes.Figure 7 (b) contains the anatomical links that are absent from Figure 5 (b) but are part of the ground truth.Note that it is very hard to find the connection P5-P7; however this connection is captured by the proposedmodel. Furthermore, the links P6-P9, P9-P12, and P8-P10, that are weak in Figure 5 (b), are also strengthenedby the proposed model.

The connectivity result of using EM algorithm based on probabilistic framework1 is shown in Figure 8.Figures 8 (a) and (b) illustrate the anatomical connectivity obtained only from dMRI and functional connectivityonly from fMRI, respectively, and Figures 8 (c) and (d) represent the anatomical connectivity and functionalconnectivity jointly estimated from dMRI and fMRI. Note that joint estimation helps find extra anatomicalconnections and eliminates the false functional connections.

(a) (b) (c)

(d) (e)Figure 6. The information network with respect to functional modes (m = 1, 2, 3, 4, 5). The size of node represents thevalue Rm

i and the width of link stands for fml . The anatomical connections were selected to form 5 subnetworks that

support the 5 functional modes, respectively.

(a) (b)Figure 7. (a) connectivity network obtained from combining the networks for all 5 modes. (b) connections that are partof the proposed model but are missing in Figure 5 (b).

(a) (b) (c) (d)Figure 8. (a) and (b) represent the estimated anatomical network and functional network, respectively, by EM approach1

obtained from dMRI and fMRI data individually. (c) and (d) represent the results obtained from joint estimation. Byjoint estimation, 5 missing links were discovered in (c) but with 10 additional false connections. In functional network(d), the misleading negative correlation between blue mode and orange mode is eliminated.

Table 2. Performance Comparison on Synthetic Data. The number of false, missing, and correct anatomical connectionsare compared for various schemes.

False Missing CorrectScheme Connections Connections Connections Data Set

Ground Truth - - 12 -Probabilistic Trackingα 7 8 4 dMRISeparate EM1β 7 10 2 dMRIJoint EM.1 16 4 8 dMRI+fMRI(Proposed) Network Flow Framework 7 3 9 dMRI+fMRI

α The links are thresholded at 5% of the maximum.β A simplified version of joint EM1 that estimates connectivity separately.

The numbers of false connections, missing connections, and correct connections with respect to all threeschemes and the ground truth (Figure 1 (b)) are summarized in Table 2. Combining fMRI data helps findmissing anatomical connections. Moreover, in the proposed network flow model, the false connections are thefewest, the missing connections are the least, and the correct connections are the highest. The proposed modelleads to the best connectivity for the synthetic data considered in the paper.

4.2 Brain Network from the Human Connectome Project Data

The network model is solved to obtain the anatomical network, Dl + Pl , and functional-associated networks,fml . The anatomical networks are composed from the connections estimated by probabilistic tractography andadditional connections found by the joint modeling. The functional-associated networks are the subnetworks ofthe anatomical network dedicated to each ICA component. Because the order of ICA components may vary, wefocus on the aggregated functional-associated network, fl =

∑m f

ml , instead of each individual sub-network.

Both anatomical network and aggregated functional-associated network are undirected and weighted. To explorevarious levels of significance, we further threshold both networks into unweighted networks of sparsity level from1% to 30% with 1% steps according to the weights on each link.36

For a functional-associated network, the subnetwork is picked according to the objective function (1) in themodel. Moreover, since the objective function listed in the equation (1) constrains the model to select thestrongest pathways, we also propose to use another min-max objective function to find the maximal portion ofanatomical network related to the fMRI data.

minfml

maxl

∑Mm=1 f

ml

Dl + Pl(11)

(a) (b) (c)Figure 9. The adjacent matrices of anatomical network (Left panel, presented in logarithm scale) and resting-state-associated anatomical subnetworks (Middle panel is with objective function (1) and right panel is with objective function(11) ) obtained from an HCP subject using the proposed joint model on dMRI and rfMRI modalities. Color code indicatesthe regions which are corpus callosum, left hemisphere (frontal, parietal, occipital, temporal, and limbic and sub corticalareas), and right hemisphere use the same as previous list.

The min-max objective function balances the capacity usage for each link by spreading the flow out propor-tionally according to the capacity of links. Therefore, we are able to obtain the largest portion of the anatomicalnetwork that are relevant to the brain function of the fMRI data.

5. CONCLUSION AND FUTURE WORK

This paper has presented a new network flow formulation for computing brain connectivity from the jointanalysis of dMRI and fMRI. Using a synthetic data example, it is shown that the proposed model leads to thebest connectivity network with the fewest number of false or missing connections, and the highest number ofcorrect connections. Future work will be directed towards validating the proposed model on a larger databaseof fMRI and dMRI data such as the HCP. Future work will also be directed towards investigating the effect ofalternative cost functions on the resulting network connectivity.

REFERENCES

[1] Venkataraman, A., Rathi, Y., Kubicki, M., Westin, C.-F., and Golland, P., “Joint modeling of anatomicaland functional connectivity for population studies,” Medical Imaging, IEEE Transactions on 31(2), 164–182(2012).

[2] Rabinovich, M. I., Afraimovich, V. S., Bick, C., and Varona, P., “Information flow dynamics in the brain,”Physics of life reviews 9(1), 51–73 (2012).

[3] Salinas, E. and Sejnowski, T. J., “Correlated neuronal activity and the flow of neural information,” NatureReviews Neuroscience 2(8), 539–550 (2001).

[4] Kriegeskorte, N., Goebel, R., and Bandettini, P., “Information-based functional brain mapping,” Proceedingsof the National Academy of Sciences of the United States of America 103(10), 3863–3868 (2006).

[5] Korzeniewska, A., Manczak, M., Kaminski, M., Blinowska, K. J., and Kasicki, S., “Determination of infor-mation flow direction among brain structures by a modified directed transfer function (dDTF) method,”Journal of neuroscience methods 125(1), 195–207 (2003).

[6] Biswal, B., Zerrin Yetkin, F., Haughton, V. M., and Hyde, J. S., “Functional connectivity in the motorcortex of resting human brain using echo-planar MRI,” Magnetic resonance in medicine 34(4), 537–541(1995).

[7] Lee, M., Smyser, C., and Shimony, J., “Resting-state fMRI: a review of methods and clinical applications,”American Journal of Neuroradiology 34(10), 1866–1872 (2013).

[8] Uddin, L. Q., Clare Kelly, A., Biswal, B. B., Xavier Castellanos, F., and Milham, M. P., “Functionalconnectivity of default mode network components: correlation, anticorrelation, and causality,” Human brainmapping 30(2), 625–637 (2009).

[9] Fox, M. D., Greicius, M., Fox, M., Greicius, M., et al., “Clinical applications of resting state functionalconnectivity,” Frontiers in systems neuroscience 4, 19 (2010).

[10] Wang, J., Zuo, X., and He, Y., “Graph-based network analysis of resting-state functional MRI,” Frontiersin systems neuroscience 4 (2010).

[11] Assaf, Y. and Pasternak, O., “Diffusion tensor imaging (DTI)-based white matter mapping in brain research:a review,” Journal of Molecular Neuroscience 34(1), 51–61 (2008).

[12] Bozzali, M., Falini, A., Franceschi, M., Cercignani, M., Zuffi, M., Scotti, G., Comi, G., and Filippi, M.,“White matter damage in alzheimer’s disease assessed in vivo using diffusion tensor magnetic resonanceimaging,” Journal of Neurology, Neurosurgery & Psychiatry 72(6), 742–746 (2002).

[13] Sporns, O., “The human connectome: a complex network,” Annals of the New York Academy of Sci-ences 1224(1), 109–125 (2011).

[14] Griffa, A., Baumann, P. S., Thiran, J.-P., and Hagmann, P., “Structural connectomics in brain diseases,”Neuroimage 80, 515–526 (2013).

[15] Johansen-Berg, H. and Behrens, T. E., “Just pretty pictures? what diffusion tractography can add inclinical neuroscience,” Current opinion in neurology 19(4), 379 (2006).

[16] Honey, C., Sporns, O., Cammoun, L., Gigandet, X., Thiran, J.-P., Meuli, R., and Hagmann, P., “Predict-ing human resting-state functional connectivity from structural connectivity,” Proceedings of the NationalAcademy of Sciences 106(6), 2035–2040 (2009).

[17] Honey, C. J., Thivierge, J.-P., and Sporns, O., “Can structure predict function in the human brain?,”Neuroimage 52(3), 766–776 (2010).

[18] Damoiseaux, J. S. and Greicius, M. D., “Greater than the sum of its parts: a review of studies combiningstructural connectivity and resting-state functional connectivity,” Brain Structure and Function 213(6),525–533 (2009).

[19] Zhu, D., Zhang, T., Jiang, X., Hu, X., Chen, H., Yang, N., Lv, J., Han, J., Guo, L., and Liu, T., “FusingDTI and fMRI data: A survey of methods and applications,” NeuroImage 102, 184–191 (2013).

[20] Ng, B., Varoquaux, G., Poline, J.-B., and Thirion, B., “A novel sparse graphical approach for multimodalbrain connectivity inference,” in [Medical Image Computing and Computer-Assisted Intervention–MICCAI2012 ], 707–714, Springer (2012).

[21] Fillard, P., Descoteaux, M., Goh, A., Gouttard, S., Jeurissen, B., Malcolm, J., Ramirez-Manzanares, A.,Reisert, M., Sakaie, K., Tensaouti, F., et al., “Quantitative evaluation of 10 tractography algorithms on arealistic diffusion MR phantom,” Neuroimage 56(1), 220–234 (2011).

[22] Poupon, C., Rieul, B., Kezele, I., Perrin, M., Poupon, F., and Mangin, J.-F., “New diffusion phantoms ded-icated to the study and validation of high-angular-resolution diffusion imaging (HARDI) models,” MagneticResonance in Medicine 60(6), 1276–1283 (2008).

[23] Xia, M., Wang, J., and He, Y., “Brainnet viewer: a network visualization tool for human brain connec-tomics,” PLoS One 8(7), e68910 (2013).

[24] Van Essen, D. C., Ugurbil, K., Auerbach, E., Barch, D., Behrens, T., Bucholz, R., Chang, A., Chen, L.,Corbetta, M., Curtiss, S. W., et al., “The Human Connectome Project: a data acquisition perspective,”Neuroimage 62(4), 2222–2231 (2012).

[25] Van Essen, D. C., Smith, S. M., Barch, D. M., Behrens, T. E., Yacoub, E., and Ugurbil, K., “The WU-Minnhuman connectome project: an overview,” NeuroImage 80, 62–79 (2013).

[26] Feinberg, D. A., Moeller, S., Smith, S. M., Auerbach, E., Ramanna, S., Glasser, M. F., Miller, K. L.,Ugurbil, K., and Yacoub, E., “Multiplexed echo planar imaging for sub-second whole brain fMRI and fastdiffusion imaging,” PLoS One 5(12), e15710 (2010).

[27] Moeller, S., Yacoub, E., Olman, C. A., Auerbach, E., Strupp, J., Harel, N., and Ugurbil, K., “Multibandmultislice GE-EPI at 7 tesla, with 16-fold acceleration using partial parallel imaging with application tohigh spatial and temporal whole-brain fMRI,” Magnetic Resonance in Medicine 63(5), 1144–1153 (2010).

[28] Setsompop, K., Gagoski, B. A., Polimeni, J. R., Witzel, T., Wedeen, V. J., and Wald, L. L., “Blipped-controlled aliasing in parallel imaging for simultaneous multislice echo planar imaging with reduced g-factorpenalty,” Magnetic Resonance in Medicine 67(5), 1210–1224 (2012).

[29] Xu, J., Moeller, S., Strupp, J., Auerbach, E., Chen, L., Feinberg, D., Ugurbil, K., and Yacoub, E., “Highlyaccelerated whole brain imaging using aligned-blipped-controlled-aliasing multiband EPI,” in [Proceedingsof the 20th Annual Meeting of ISMRM ], 2306 (2012).

[30] Glasser, M. F., Sotiropoulos, S. N., Wilson, J. A., Coalson, T. S., Fischl, B., Andersson, J. L., Xu, J., Jbabdi,S., Webster, M., Polimeni, J. R., et al., “The minimal preprocessing pipelines for the Human ConnectomeProject,” NeuroImage 80, 105–124 (2013).

[31] Jenkinson, M., Beckmann, C. F., Behrens, T. E., Woolrich, M. W., and Smith, S. M., “FSL,” Neuroim-age 62(2), 782–790 (2012).

[32] Beckmann, C. F. and Smith, S. M., “Probabilistic independent component analysis for functional magneticresonance imaging,” Medical Imaging, IEEE Transactions on 23(2), 137–152 (2004).

[33] Storti, S. F., Formaggio, E., Nordio, R., Manganotti, P., Fiaschi, A., Bertoldo, A., and Toffolo, G. M.,“Automatic selection of resting-state networks with functional magnetic resonance imaging,” Frontiers inneuroscience 7 (2013).

[34] Spath, H., [Cluster dissection and analysis: theory, FORTRAN programs, examples ], Computers and theirapplications, Horwood (1985).

[35] Wang, R., Benner, T., Sorensen, A., and Wedeen, V., “Diffusion toolkit: a software package for diffusionimaging data processing and tractography,” in [Proc Intl Soc Mag Reson Med ], 15, 3720 (2007).

[36] Sporns, O., [Networks of the Brain ], MIT press (2011).