Big Data and the Brain MICHA E L O L IVE R G A L L A NT L A B UC BER KELEY

The brain is gigantic

• The human brain has ~100 billion neurons connected by ~100 trillion synapses

• Multiple levels of organization

But our data is only “Big”

• Electrophysiology experiments can record from ~100 neurons simultaneously

• fMRI experiments we can record from ~90,000 voxels of about 20 mm3

• There are over 2,000,000 neurons per voxel

The visual brain

System identification and neuroscience

• Model each neuron based on relationship between stimulus and response

• Evaluate models based on their ability to predict responses to novel stimuli

Why system identification in visual cortex is hard

• Non-linear

• High dimensional

• Interpretability is important!

Car Eiffel Tower

Linearized regression

p1

p2

p3 Pixel Space

f1

f2

f3 Feature Space

a1

a2

a3 Brain Activity Space

Nonlinear Feature Mapping

Encoding Model

Decoding Model

Feature spaces

Feature Mapping

Responses Stimuli

W1

W2

Wn

No

nlin

eari

ty

+

Model neuron/voxel

Linear Weighting

Movie reconstruction from fMRI data

Nishimoto S, et al. Curr Biol. 2011

Oct 11;21(19):1641-6

Van Essen DC, Gallant JL. Neuron. 1994 Jul;13(1):1-10.

Tuning in V4

We need big data!

• V4 receptive fields are moderately large

• Potential stimulus space is very large

• Natural images span the relevant space

• Response is highly nonlinear

How to get big data

• Implantable electrodes allow us to record from the same cell over many days

• We used over 1 million frames of natural movies, the largest ever stimulus set in V4

General nonlinear modeling

• The Volterra Series:

• Can control model flexibility by order choice

• Parameter space grows quickly with order

x1 x2

h11 h12

h21 h22 h111 h121

h211 h221

h111 h121

h111 h121

The scale of the problem

• If we have 1000 pixels

• 2nd Order: 500,000+ coefficients

• 3rd Order: 160 million+ coefficients

• 4th Order: 40 billion+ coefficients

• But we actually have about 196,608 pixels…

• 2nd Order: 19 billion+ coefficients

• 3rd Order: 1.2 quadrillion+ coefficients

• 4th Order: 62 quintillion+ coefficients

Pixels 1.2x1015

Taming dimensionality

Pixels 1.2x1015

PCA 1.6x108

Kernel regression

For all i,j =1:n in training data

Calculate weights for kernel regression model

Use weights to make predictions for new x

Kernel function equivalent to dot product in feature space

Pixels 1.2x1015

PCA 1.6x108

Kernel 1x106

The Inhomogeneous Polynomial Kernel

2nd Order IHP:

Implicitly Maps to feature space containing all first and second order terms

Neural networks as kernel machines

Input Hidden Units

Output In a standard NN:

From NN perspective, kernel regression with a tanh kernel function is equivalent to a NN with hidden units = training samples

Pixels 1.2x1015

PCA 1.6x108

Kernel 1x106

IPKN 1.5x104

Stochastic gradient boosting

Volterra Space Error Surface Data

Prediction performance by model order

Color constancy in V4

V4 Tuning for Color

Kusunoki M et al. J Neurophysiol 2006;95:3047-3059

Color Tuning of V4 Cells

First Order Color Coefficients

V4 tuning to curvature

Pasupathy A , and Connor C E J

Neurophysiol 2001;86:2505-2519

V4 tuning to Non-Cartesian Gratings

Gallant JL, Connor CE, Rakshit S, Lewis JW, Van Essen DC.

J Neurophysiol. 1996 Oct;76(4):2718-39.

Eigenvectors of second order V4 receptive field model

Eigenvectors of second order V4 receptive field model

Shape tuning of a V4 cell’s Volterra model

Shape tuning of a V4 cell’s Volterra model

Embracing the Complexity

• Demonstrated a way to make this big problem tractable

• Shown many reported features of V4 tuning can exist in a single cell

• Interpretation of large models is still a major problem

• Need tensor libraries that exploit symmetry to decompose large models

Thank you!

V4 High Response Movie Frames

Stochastic Gradient Boosted IPKNs

• Use IPKNs as the weak learners

• Fit to sample of data using backprop w/ a stopping set

• Perform line search to determine step size that minimizes error on sample

• Multiply step size by learning rate and update function

• Ensemble is equivalent to a single Volterra model!

Some Important Unanswered Questions

• What information are our models missing in V2, V4 and beyond?

• Do we need nonlinear combinations of basis functions?

• Can we derive new basis functions?

Extracting Coefficients from Model

• Create a design matrix or the desired order of interactions from the support vectors/input weights

• Multiply by the output weights and weight by correction factor

• Extract coefficients from each iteration’s network, weight by step size and sum to get final set of coefficients

Feature Spaces

x =

Response to each

category was found

using regularized

linear regression

Movies were shown

to subjects

Wom

an

Talkin

g

Text

Car

Build

ing

Category labels

2.8 -0.4

-0.9

1.7 -0.8

-1.0

-2.1

0.7

0.1 -2.70.1

-3.7

-3.2-1.5

-1.6

BOLD responsesCategory modelweights

Natu

ral m

ovie

s

Movies were labeled

with 1705 nouns

and verbs

BOLD responses were

recorded from the whole

brain using fMRI

120 minutes 120 minutes

Constructing a Semantic Space

person

athlete

plantorganism

animal

arthropod

reptile

bird fish

mammal

carnivore talk

communicate

text

group

herd

attribute

dirt

measure

communication

event

wave

rodeo atmospheric

phenomenon

mist

bodypart

eye

leg

matter

food

underwatersky

material bamboo

location

city

grassland

geol. formation

hill

plant organ

artifactway

clothing

road

structure

room

shop

door

building

furniture

container

bottle

device

laptop

gas pump

weapon

tool

kettle

ball

vehicleboat

wheeled

vehiclecar

travel

breathe

fastenmove

(transitive)

touch

hit

jump

change

walk

gallop

rappel

drag

turn

spin

bloom

lean

rodent

ungulate

consume

rub

move

C

equipment

Coefficient on 1st PC– +

person

mammal

bodypart

vehicle structure

location

plant organ

athlete

2nd PC

+

–+

+3rd PC

4th PCB

(Red channel)

(Green channel)

(Blue channel)

city

RGB colormap

for the group

semantic

space

sky

atmos.

phenom.

A

Semantic Decoding

Flattening the Brain

Visualizing Semantic Space

CoS

ITS

MTS

STS

IPS

CiSmr

PoCeS

Sylvian

Fissure

CeS

IFS

SFS

CiS

STSMTS

IPS

CiSmr

PoCeS

CeS

SFS

IFS

CiS

ITS

CoS

Sylvian

Fissure

V1

V2

V3

V4

V3b

V3a

V7IPS

MT+

EBA

OFA

FFA

PPA

RSC

Speech

FO

FEF

M1M

S1M

M1H

S1H

M1F S1F

SMHA

SMFA

AC

SpeechVper

Vper

pSTS

A1

V1

V2V3 V4

PPA

FFA

EBA

MT+LO

V3b

V3a

V7

IPS

RSC

FO

FEF

SpeechSEF

SMFA

S1M

M1MS1H

M1H

M1F

S1F

AC

Vper

Vper pSTS

A1

Speech

PrCu

TPJTPJ

PrCu

LO

TOS

SEF

TOS

SMHA

!

!!

!

!

!

!

!

!!LHanterior

su

pe

rio

r

!

!!

!

!

!

!

!

!!

!

!!

!

!

!

!

!

!!

Correlation with 1st PC- +

RHsu

pe

rio

r

anterior

WordNet LSA

LabelsCoS

ITSMTS

STS

IPS

CiSmr

PoCeS

Sylvian

Fissure

CeS

IFS

SFS

CiS

STS

MTS

IPS

CiSmr

PoCeS

CeS

SFS

IFS

CiS

ITS

CoS

Sylvian

Fissure

V1

V2

V3V4

V3b

V3aV7IPS

MT+

EBA

OFA

FFA

PPA

RSC

IFSFP

Speech

FO

FEF

M1M

S1M

M1H

S1H

M1FS1F

SMHASMFA

AC

Speech

Vper

VperpSTS

A1

V1

V2

V3

V4

PPA

FFA

OFA

EBA

MT+

LO

V3b

V3a

V7

IPS

RSC

FBA

IFSFP

FO

FEF

Speech

Speech

SEF

SMFA

S1M

M1M

S1H

M1H

M1FS1F

AC

Vper

Vper

pSTS

A1 Speech

PrCu

TPJTPJ

PrCu

!

!

!

!

!

!

!

!

!

!

EBA

LO

LHanterior

su

pe

rio

r

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!

B

Correlation with 1st PC- +

A

RHsu

pe

rio

r

anterior

C D