Backup: Machine Learning - UniTrento Enver Sangineto... · 2020. 6. 9. · Backup: Machine Learning...
Transcript of Backup: Machine Learning - UniTrento Enver Sangineto... · 2020. 6. 9. · Backup: Machine Learning...
Backup: Machine Learning
Enver Sangineto DISI
University of Trento, Italy
• My (general) research interests
• Backup: Hybrid biological-artificial networks
• Backup: Neuromorphic computing
Overview
• My (general) research interests
• Backup: Hybrid biological-artificial networks
• Backup: Neuromorphic computing
Overview
My Research Area
• Deep Learning:
– Discriminative methods
– Generative methods
Current (non-Backup) Research Interests
• Discriminative training with minimal human supervision – Weakly-supervised Object Detection
– Anomaly Detection
– Few-Shot Learning
– Domain Adaptation
• GAN-based Image Generation – GANs conditioned on structured input
– Improving GAN stability
• My (general) research interests
• Backup: Hybrid biological-artificial networks
• Backup: Neuromorphic computing
Overview
Goals: 1. Use an ANN to predict what a biological net "thinks"
2. Perform hybrid, joint artificial-biological computations
Hybrid biological-artificial nets
Goals: 1. Use an ANN to predict what a biological net "thinks"
2. Perform hybrid, joint artificial-biological computations
Hybrid biological-artificial nets
Predicting the behaviour of a biological network
• Can we read and predict what a brain thinks?
• There are some in-vivo experiments using human beings
9
"Mind reading"
• Reconstructing the brain signal (e.g., fMRI) using a neural network
Image reconstruction examples
• Common problems of the brain-signal reconstruction approaches: – Voxels have a low spatial resolution
– Small datasets
• A possible alternative: in-vitro experiments
• Goal: – To access each biological neuron
– To collect large training datasets
Brain reading in the Backup project
Photonic circuits and optogenetics used to stimulate and read biological neuron activations
Accessing individual neurons
• Let B be the biological network with n neurons
• Let A be the ANN with m neurons
• A and B do NOT need to share the same structure
Biological net’s activation prediction: a possible schema
• Let st be the light-based stimulus of B at time t
• E.g., st Rn represents the individual-neuron stimulation
• In a sparse stimulus, for most i (1 <= i <= n), st,i = 0
Biological net’s activation prediction: a possible schema
• After a time delay (k), let rt+k, be the activation state of B, i.e., B’s "response" induced by st as measured at time t + k.
• rt+k Rn
• rt+k is the light-based “readout” of B
Biological net’s activation prediction: a possible schema
• We can collect a virtually unlimited dataset
D = {(st , rt+k)}
• D is used to train A
Biological net’s activation prediction: a possible schema
• A learns to predict rt+k from st
• Formally: A(st) = rt+k
• A is a "functional copy" (a backup…) of the memories of B
Biological net’s activation prediction: a possible schema
Goals: 1. Use an ANN to predict what a biological net "thinks"
2. Perform hybrid, joint artificial-biological computations
Hybrid biological-artificial nets
Hybrid computational systems
• Can we develop hybrid computational systems?
• Can we replace arbitrary parts of the biological network with an artificial network, still preserving the same functional behaviour?
Hybrid computational systems: a possible schema
• B is arbitrarily split in two sub-nets
• Analogously:
st = (st(1), st
(2))
rt+k = (rt+k(1), rt+k
(2))
• B1 is inhibited or removed
• B1 is replaced with an ANN A
• A is connected with B2
Hybrid computational systems: a possible schema
• A and B2 exchange information
• Goal: B2 ‘s response (rt+k
(2)) should be statistically similar to what is obtained without amputation
Hybrid computational systems: a possible schema
• My (general) research interests
• Backup: Hybrid biological-artificial networks
• Backup: Neuromorphic computing
Overview
Neuromorphic Computing
Goal: to implement ANNs using photonic circuits
This is motivated by the much higher speed and lower power consumption of a photonic circuit w.r.t. an electric circuit
Our current solutions
• MultiLayer Perceptron (MLP)
• Reservoir Computing
Our current solutions
• MultiLayer Perceptron (MLP)
• Reservoir Computing
MLP using silicon photonics
• Our MLP is based on a microring resonator whose (thermal) nonlinear response corresponds to the neuron activation function
MLP using silicon photonics
• Only light intensity is used (no phase information)
MLP using silicon photonics
• Constraints:
– All net’s weights and activations should be positive
– The sum of the weights associated with the connections exiting from a neuron should be limited by 1
MLP using silicon photonics
• We solved this constrained-optimization problem using a technique called Projected Gradient Descent
• In each SGD step, w is projected onto the admissible area defined by our constraints.
MLP using silicon photonics
Preliminary Results:
• We used a software-based simulation
• MNIST dataset (60,000 28X28 digit images)
• MLP structure: 784-200-10
• Our simulation: 92% accuracy
• Standard, non-constrained ANN (same structure): 97%
• MNIST “linear regime”: 88%
Our solutions
• MultiLayer Perceptron (MLP)
• Reservoir Computing
Reservoir Computing (RC): Introduction
• It is an RNN with:
– input-to-hidden layer weights randomly fixed
– hidden-to-hidden layer weights randomly fixed
– hidden-to-output layer weights (readout) learned
• Main advantage: it is easy to train
• Can be implemented using photonic circuits
Reservoir Computing (RC): Introduction
• Sparsity is obtained by setting most of the connection weights to 0:
Reservoir Computing (RC): Introduction
• Difference with a standard RNN:
– Testing: no difference
– Training: only Who needs to be trained
Reservoir Computing (RC): Introduction
RC: Our solution
• Our trainable last layer (Who) is a Perceptron
• Specifically, using both phase and intensity, we have complex-valued activations and weights (CV-Perceptron)
Our CV-Perceptron
• The output neuron activation function is the squared light intensity
• All computations, except the last activation function are performed using photonic circuits
• This is different from common implementations [1], in which Who ht is computed electronically
[1] Amin et al., Single-layered complex-valued neural network for real-valued classification problems. Neurocomputing (72) 2009
Our CV-Perceptron: Simulation Results
Real datasets (normalized test error):
Symmetry detection (@ 0 error):
[1] Amin et al., Single-layered complex-valued neural network for real-valued classification problems. Neurocomputing (72) 2009
Dataset Amin [1] (act-fun. 1)
Amin [1] (act-fun. 2)
RV Perceptron
Ours
Fisheriris 0.19 - 0.15 0.11
Diabetes 0.37 0.47 0.23 0.27
Cancer 0.026 0.025 0.014 0.08
Dataset Amin [1] (act-fun. 1)
Amin [1] (act-fun. 2)
RV Perceptron
Ours
Longest seq. 7 6 1 6
Thank you!
Questions?