Nikon D7100 Manuadownload.nikonimglib.com/archive2...Nikon D7100 Manua
A big data approach to functional characterization of the...
Transcript of A big data approach to functional characterization of the...
![Page 1: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/1.jpg)
A big data approach tofunctional characterization
of the human brain
Jack GallantUniversity of California at Berkeley
CCC-NSF Meeting Dec. 2014
![Page 2: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/2.jpg)
How complicated is the brain?
How much data will we need to model it?
How can we model the system?
How do we deal with non-stationarity?
What about individual differences?
Can we do brain decoding?
Can we get better data?
![Page 3: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/3.jpg)
Macaque vision is a typical brain subsystem
Felleman and Van Essen, Cerebral Cortex, 1992
● Dozens of distinct anatomical and functional areas arranged in a hierarchical, parallel network with substantial recurrent feedback.
● Transformations between areas are nonlinear.
● Each area represents a different high-dimensional feature space projected onto the cortical surface.
● Each area is differentially affected by bottom-up and top-down information.
![Page 4: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/4.jpg)
Modha & Singh, PNAS, 2010
The brain is a big placeMacaque brain:2 billion cortical neurons1-10 thousand connections/neuron383 total areas and nuclei6602 bi-directional, inter-areal connections
Human brain:18 billion cortical neurons1-10 thousand connections/neuron5 million cortical columns ?500 areas and nuclei?12000 inter-areal connections?
![Page 5: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/5.jpg)
Information is organized at multiple scales
OrientationSpatialfrequency
Ocular dom.
Issa, Trepel & Stryker, J. Neurosci (2000)
RetinotopicMappingStimulus
2DGmap onflattenedMacaquecortex
Tootel et al., J. Neurosci (1988)
![Page 6: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/6.jpg)
Structural and functional measurementsare complimentary
Diffusion Imaging fMRI
![Page 7: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/7.jpg)
How complicated is the brain?
How much data will we need to model it?
How can we model the system?
How do we deal with non-stationarity?
What about individual differences?
Can we do brain decoding?
Can we get better data?
![Page 8: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/8.jpg)
Predictions are limited by data size
Approximatesize of MTdata sets forsingle neurons
Oliver & Gallant, in preparation
1.38 Hours of video
MikeOliver
![Page 9: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/9.jpg)
Methods are time and/or space limited
Less activity Average activity More activity
Nishimoto et al., Current Biology, 2011; Huth et al., Neuron, 2012;Data visualization by James Gao's pyCortex (http://pycortex.org)Online visualization available at http://gallantlab.org
JamesGao
![Page 10: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/10.jpg)
How complicated is the brain?
How much data will we need to model it?
How can we model the system?
How do we deal with non-stationarity?
What about individual differences?
Can we do brain decoding?
Can we get better data?
![Page 11: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/11.jpg)
s(x,y,t) r(t)H(x,y,t,a)
Neuroscience as system identification
The amount of data required to fit the modelIs proportional to the degree of nonlinearity.
State variables (e.g. Attention) multiplyThe dimensionality of the problem.
![Page 12: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/12.jpg)
The explicit feature space approach
Nonlineartransformation
Lineartransformation
Linear or Nonlineartransformation
Hypothesis Linearregression
Experimentalmeasurement
![Page 13: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/13.jpg)
Modeling multiple, explicit feature spaces
![Page 14: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/14.jpg)
Explicit modeling of MT neurons
Nishimoto & Gallant, Neuron, 2011
ShinjiNishimoto
![Page 15: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/15.jpg)
A motion-energy model for low-level vision
Nishimoto, Vu, Naselaris, Benjamini, Yu and Gallant, Current Biology, 2011
ShinjiNishimoto
![Page 16: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/16.jpg)
A semantic model for high-level vision
Huth, Nishimoto, Vu & Gallant, Neuron, 2012
AlexHuth
![Page 17: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/17.jpg)
The machine learning approach
NonlinearTransform
(Hypothesis)
Linear or NonlinearTransform
(Measurement)
Nonlineartransformation
Linear or Nonlineartransformation
Learningalgorithm orregressionalgorithm
Experimentalmeasurement
![Page 18: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/18.jpg)
A deep network for V2 neurons
Oliver & Gallant, in preparation
MikeOliver
Deep time-delay network trainedwith DropOut and Poisson loss
![Page 19: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/19.jpg)
A convolutional network for fMRI data
Note: The CNN used herecontained 5 convolutionallayers and 2 classifier layers.
Agrawal, Cheung, Stansbury, Malik & Gallant, in preparation
PulkitAgrawal
![Page 20: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/20.jpg)
How complicated is the brain?
How much data will we need to model it?
How can we model the system?
How do we deal with non-stationarity?
What about individual differences?
Can we do brain decoding?
Can we get better data?
![Page 21: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/21.jpg)
Attention alters tuning in single neurons
Connor, Gallant, Preddie & Van Essen, J. Neurophys., 1996David, Mazer, Hayden & Gallant, Neuron, 2008
Olshausen, Anderson & Van Essen,J. Neurosci., 1993
![Page 22: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/22.jpg)
Attention changes cortical semantic maps
Cukur, Nishimoto, Huth & Gallant, Nature Neuroscience, 2013
TolgaCukur
![Page 23: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/23.jpg)
How complicated is the brain?
How much data will we need to model it?
How can we model the system?
How do we deal with non-stationarity?
What about individual differences?
Can we do brain decoding?
Can we get better data?
![Page 24: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/24.jpg)
Narrative languagesemantic maps for5 subjects
Huth, de Heer, Griffiths, Theunissen & Gallant, in review
![Page 25: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/25.jpg)
PrAGMATiC: Probabilistic And GenerativeModel of Areas Tiling Cortex
Huth, de Heer, Griffiths, Theunissen & Gallant, in review
![Page 26: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/26.jpg)
Likelihood tests for PrAGMATiC
Huth, de Heer, Griffiths, Theunissen & Gallant, in review
77Areas
64Areas
FiducialMap
![Page 27: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/27.jpg)
How complicated is the brain?
How much data will we need to model it?
How can we model the system?
How do we deal with non-stationarity?
What about individual differences?
Can we do brain decoding?
Can we get better data?
![Page 28: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/28.jpg)
Using encoding models to decode
P ( f (S )∣R)∝P (R∣ f (S ))P ( f (S ))
![Page 29: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/29.jpg)
Decoding semantics from higher visual areas
MovieLikely
Objects and Actions
Huth, Lee, Nishimoto & Gallant, in review
![Page 30: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/30.jpg)
You can decode anything that is representedin measured brain activity
Gao & Gallant, unpublished
![Page 31: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/31.jpg)
The current and future state of brain decoding
Joseph Niepce, 1825
● Our ability to measure brain activity.● Our ability to model measured activity.● Computer power.
![Page 32: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/32.jpg)
How complicated is the brain?
How much data will we need to model it?
How can we model the system?
How do we deal with non-stationarity?
What about individual differences?
Can we do brain decoding?
Can we get better data?
![Page 33: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/33.jpg)
Improved methods for recording brain data
Functional MRI
Optical imaging
Electrocorticography
![Page 34: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/34.jpg)
Trying to measure orientation columns
Yacoub, Harel & Ugurbil, PNAS, 2008
![Page 35: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/35.jpg)
The brain is very, very complicated structure.
Current methods for measuring the brain have limited spatial, temporal and conditional sampling.
To make progress we need to push the boundaries of spatial, temporal and conditional sampling.
If we do this then we can use a variety of computational methods to build robust computational models that predict accurately and generalize well.
When the resultant models perform poorly (which is likely), then we can focus on improving computational theory rather than mere neuroscience data collection.
Summary and conclusions
![Page 36: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/36.jpg)
We are entering the age of “Big Data” in neuroscience, and there is a pressing need for data processing algorithms that can operate on these meso-scale data.
The required algorithms fall into several domains: matrix (tensor) libraries, image processing algorithms, data mining tools, database tools and visualization tools.
The brain is a nonlinear dynamical system with feedback, but these are the least well understood sorts of systems from a theoretical point of view.
Therefore, even if we had a complete functional (or structural) map of the brain, we wouldn't know what to do with the data or how to model it!
This would be a good problem to have, and would likely drive theoretical developments in computer science and mathematics.
Computer science versus neuroscience
![Page 37: A big data approach to functional characterization of the ...archive2.cra.org/ccc/files/docs/meetings/BRAIN/12-14.CCC...We are entering the age of “Big Data” in neuroscience, and](https://reader036.fdocuments.in/reader036/viewer/2022071117/6003484d6d1c6676fc40613c/html5/thumbnails/37.jpg)
Natalia BilenkoJames GaoAlex Huth
Fatma ImamogluMark Lescroart
Tolga Cukur (Bilkent U)Stephen David (OHSU))Kate Gustavsen (UTVM)Kathleen Hansen (NIH)
Ben Hayden (URochester)Kendrick Kay (WashU)
James Mazer (Yale)
Thomas Naselaris (MUSC)Shinji Nishimoto (NIST Japan)
Ryan Prenger (LANL))Ben Willmore (Cambridge)
Bill Vinje (Salk)An Vu (UCB)
Michael Wu (Lithium Inc)
This work was supported by NEI, NIMH and NSF
Lydia MajurAnwar NunezMichael OliverStorm Slivcoff
Dustin Stansbury
Collaborating labsFrederic Theunissen, Bin Yu,Tom Griffiths, Cheryl Olman
Ralph Adolphs, Bertrand Thirion, Essa Yacoub, Kamil Ugurbil
Peter Bandettini
Current Lab Members
Past Lab Members