Autonomy Incubator Seminar Series: Neuromorphic solutions for autonomous land and aerial vehicles
-
Upload
autonomyincubator -
Category
Technology
-
view
310 -
download
2
description
Transcript of Autonomy Incubator Seminar Series: Neuromorphic solutions for autonomous land and aerial vehicles
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 1
Neuromorphic solutions
for autonomous land
and aerial vehicles Massimiliano Versace
CEO, Neurala Inc.
Director, Boston University Neuromorphics Lab
www.neurala.com nl.bu.edu [email protected]
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 2
To produce virtual and robotic agents
that autonomously learn to perform
useful tasks
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 3
limited intelligence
1:1 ratio robot/
human operator
barriers to adoption
Why are not robots everywhere?
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 4
Three ingredients for success
Smart mind
Powerful brain Inexpensive body
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 5
The body
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 6
Low cost of sensors, new actuators
accelerometer
gyroscope
magnetometer
front and rear cameras
NFC
barometer
speaker
microphone
proximity
light sensor
Bluetooth
GPS
WiFi + cellular
humidity
temperature
navigation
vision
audition
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 7
Three ingredients for success
Smart mind
Powerful brain Inexpensive body
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 8
Many robots, many minds
R&D $ R&D $$$
$$ensors
R&D $$
$ensors
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 9
One mind for autonomous robots?
learning
parallel processing
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 10
Why do we LOVE this mess??!!
Neuromorphic = of the form of neurons, brains Even in apparently simple tasks, many brain
functions work in synergy to help us solve
otherwise VERY HARD PROBLEMS
The brain does not work in silos of competences.
E.g., as you move around in the world, vision helps navigation (e.g., landmark
recognition, obstacle avoidance), navigation helps vision (what to expect in one
environment vs. another).
Ever thought why you can’t recognize the teller of your grocery at the movie theater?
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 11
“Stovepiped” sensemakig Rock detector Sky detector & cloud detector Dust devil detector
Onboard Autonomous Science Investigation System for Opportunistic Rover Science, Castano et al., 2007
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 12
Extremely un-stovepiped sensemaking!
object recognition
optic flow stereo & depth
environment map
with objects
segmentation
bias where to look
(data acquisition)
coarse
view
scenes help
define objects
object help
define scenes
map helps to find objects
objects help
orient
(e.g. Citgo)
to: obstacle
avoidance
scene recognition
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 13
Feedforward and feedback processing
Brain is a massively recurrent architecture
10X E.g. in the visual system, for each output
fiber from thalamus to the visual cortex, the
thalamus receives ~10 feedback fibers
What is this feedback for?
One key aspect of neuromorphic computing is feedback processing
Feedback happens all over the brain
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 14
Leave your desk.. ..walk to the conference room.. ..sit in the conference room
Feedforward information flow:
- leave the desk for the talk
- walk down the corridor to the conference room
- find the conference room, find a free spot as close to the exit as humanely
possible
Feedback information flow:
At every moment, you are MATCHING memory-based expectations with sensory
(and other cortical) INPUT
INPUT INPUT INPUT INPUT INPUT
New post-it (Visual)
Desk 0.5 cm lower
(Proprioceptive)
Chanel #5 smell
(Olfactory)
Different door knob (Tactile)
New speakers (Auditory)
EXPECTATION EXPECTATION EXPECTATION EXPECTATION EXPECTATION
Mark coming to the conference room
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 15
Feedback focuses on anomalous information
The president of the United States of America
?
& ?
Match memory with current input
Automatically and effortlessly focus on crucial differences
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 16
Predictive feedback is ubiquitous in the brain
eel is on the ...
The sentence “eel is on the ...” is preceded by white noise
Percept may be determined by the sound that one expects to hear in auditory
context based on previous language experiences
Expectations amplify consistent components of the white noise, while
suppressing inconsistent components
orange peel is on the orange
wagon wheel is on the wagon
shoe heel is on the shoe
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 17
Adaptive Resonance Theory
F2
+ +
- F1
X
Y
+
RESET
ATTENTIONAL
SUBSYSTEM
ORIENTING
SUBSYSTEM
Σ -
+ +
MATCHING
CRITERION
ADAPTIVE RESONANCE THEORY
(ART)
MATCH MISMATCH
Σ Σ
Math behind neuromorphic systems?
x
y
w
𝑑𝑦𝑗
𝑑𝑡= −𝐴𝑦𝑗 + (𝐵 − 𝑦𝑗) 𝑤𝑖,𝑗𝑥𝑖 − (𝑦𝑗 + 𝐶) 𝑦𝑘
𝑛
𝑘≠𝑗
𝑛
𝑖=0
𝑑𝑤𝑖,𝑗
𝑑𝑡= −𝐷𝑦𝑗𝑤𝑖,𝑗 + 𝐸𝑥𝑖𝑦𝑗
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 18
NASA STTR Phase II
With Mark Motter, NASA Langley
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 19
One algorithm for all robots
Sensory
Motivation
Navigation
Classification
Prefrontal
Cortex
Binding What
& Where
Unclear
Self-
Localization
Multiple Areas
Position-
Invariant Cells
IT
Complex
Cells
V2
Oriented
Edges
V1
Saliency
Map
Posterior PC
Luminance
LGN
Pre-
Processing
Retina
Color
Opponency
LGN?
Curiosity
Drive
Unclear
Goal
Selection Map
Unclear
Reward
Location Map
Unclear
Current
Location
Enthorinal Ctx
Active Goal
Location
Prefrontal Ctx
Reward
Signal
Basal Ganglia
Forward
Spread Trigger
Dentate Gyrus
Reverse
Spread
Prefrontal Ctx
Forward
Spread
CA3
Desired
Next Location
CA1
Motor Turn
Command
Motor Cortex
Motor Run
Command
Motor Cortex
Lack of
Comfort Drive
Unclear
Path-Finding
Reset
Medial Septum
Desired
Heading
Cingulate Ctx
Virtual Environment or Robotic Platform
~32M neurons, ~13B synapses
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 20
Example application: MoNETA
Morris water maze
Familiarity
map
Goal
Selection
map
MoNETA’s visual input
Brain diagram
Reverse
spread
Obstacle
map 1st run 4th run 6th run 8th run
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 21
One algorithm for all robots
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 22
Simulated Mars Rover
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 23
fusion
Vestibular
gyro &
accel.
Visual
optic flow
Motor
wheel
velocity
(1) (2)
Feedback improves heading estimation
Source Average error (deg) Avg. error rate (deg/hr) Vestibular 62.8 1.63 Visual 35.9 0.916 Motor 29.9 0.983 Integrated ring 14.9 0.111
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 24
How should robots “see”?
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 25
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 26
Biological vision is:
Multimodal (focus vision on speaker)
• computational savings (up to 30x documented vs. uniformly
sampled images)
• conformal mapping
• some invariance to changes in scale
and rotation Traver and
Bernardino (2010)
Selective (non-uniform sampling)
Predictive (look where the ball is going)
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 27
Artificial visual system
exogenous
spatial attention
WHERE
Visual System
Preprocessing
surface
representations
learning
control
object/scene
input
image
cognitive
biases
label
where
eye
movements
WHAT
what
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 28
Learning to classify Martian rocks
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 29
Expanding the visual system
Temporal continuity:
“how do you know that
the thing you are
looking at now is the
same you look at 1
second ago?”
Upcoming development under the STTR Phase II program:
1) Depth processing, optic-flow aided segmentation, scene “gists”
2) Implementation on robotic platforms
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 30
Robotic applications
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 31
One brain for all robots: UAVs?
Video courtesy of Mark Motter
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 32
Optic flow as a collision clue
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 33
Sense and Avoid from Flow Expansion
SAFE model
with Mark Motter, NASA Langley, CIF 2012-13
past environment (s) - action (a) pair
learning rate
reward/punishment value current state/action
random action
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 34
Obstacle avoidance via optic flow learning
Cost of crashing ~= cost of deviating course
Cost of crashing >> cost of deviating course
Each point = MAvg over 200 trials
Collision rate with other UAVs in collision path
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 35
Obstacle avoidance via optic flow learning
Each point = MAvg over 200 trials
Rate at which the UAV turns off-course in the
absence of a colliding UAV
Cost of crashing ~= cost of deviating course
Cost of crashing >> cost of deviating course
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 36
Obstacle avoidance via optic flow learning
Each point = MAvg over 200 trials
Rate at which the UAV turns off-course
after a UAV has passed
Cost of crashing ~= cost of deviating course
Cost of crashing >> cost of deviating course
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 37
Identifying other UAVs: virtual environment
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 38
Identifying other UAVs: real video
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 39
How do pilots look at things?
1a) Develop navigation and space variant vision on land robot;
1b) Develop collision avoidance for UAV;
2) Merge 1 & 2 on UAV;
NASA STTR Phase II timeline
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 40
Three ingredients for success
Smart mind
Powerful brain Inexpensive body
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 41
The brain
100 billion neurons
250 trillion synapses
+ =
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 42
Brain power
+
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 43
Brains, GPUs, multicore CPUs
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 44
Videogames and smartphones
+ =
Graphic Processing Units
(GPUs), now in mobile
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 45
Parallel vs. serial
CPU
GPU
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 46
2012 2017
# n
eu
ron
s/c
m2, p
ow
er s
up
ply
2023
2012
2015
2017
8M 0.87V
2013
24M 0.65V
2018
12M 0.78V
digital
GF
LO
PS
pe
r W
att
(G
PU
) &
# m
ob
ile
CP
U c
ore
s
6
4
2014
12?
2016
25?
48?
Computing power roadmap
“fancy”
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 47
2012 2017
# n
eu
ron
s/c
m2, p
ow
er s
up
ply
2023
Mouse brain
2017
2015
2018
6 cm2
28 V
3 cm2
5 V
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 48
2012 2017
# n
eu
ron
s/c
m2, p
ow
er s
up
ply
2023
Human brain
2017
2015 35 m2
2 kV
2018
70 m2
5 kV
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 49
The Human Brain Project
$100M/yr.
2013-23
BRAIN Initiative,
projected $300M/yr.
10 yrs.
2012 2017 2023 2017
Faster than we think?
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 50
Not only academia!
The NPU will
"not only mimic human-like perception
but also have the ability
to learn how biological brains do”
* See also IBM, Intel
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 51
Towards autonomous machines
Smart mind
Powerful brain Inexpensive body
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 52
Outreach: Boston Museum of Science
1.5 million visitors: the most visited cultural attraction in Boston
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 53
NASA?...no, MASA!
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 54
~1800 visitors used the app Snow inches by day - 2014
0
0.2
0.4
0.6
0.8
1
1.2
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 55
Neurala Massimiliano Versace
CEO, Neurala Inc.
www.neurala.com
in partnership with
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 56
56
To What
Pathway
Hot Spot (log-polar)
Hot Spot (Linear)
Eye mov. (Linear)
Eye/head
mov.
(Linear)
Working
memory (head cent.)
Working
memory (retinal)
Input (log-polar)
Edges
Diffusion
Contours
Working
memory kernel
Shroud (log-polar)
Reset Fit Shroud (log-polar)
Early Vision Processing
Attention Shroud
Working Memory
Eye/Head Movement
Where pathway
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 57
57
log-polar input representation
hot-spot working memory new hot spot
(where the eye will look next)
reset signal (different object)
shroud forming (bottom)
and fully formed (top)
Where pathway
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 58
Boundary/ completed
Shroud from Where
pathway
View Category
Object Category
Name Category
Teacher Where reset
Associative learning
ART learning
Recurrent excitation
Inhibition Spatial shroud exists Top-down priming
From “where”
pathway
From external
source
Core of “what”
pathway
Generates a name for the foveated object
(unsupervised). Can also take top down
name assigned by external teacher
(supervised)
Groups multiple views of a single object
into an object category. Reset of object
category is gated by Where signal
Groups similar log-polar views
of same object into view categories
What pathway
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 59
59
Confidential
Neurala – AFRL site visit,
November 14, 2012
59
bottom-up
activated name
category
top-down
teaching signal
current input
view-
specific
neurons
view-specific neuron weights
learning multiple views
of the same digit
What pathway
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 60
Semantic influence on data collection
5
learning
invariance
new instance
(ambiguous view)
?
0
1
2
3
4
5
6
7
8
9
Few presentations
“heat maps”
… 0 1 2 5
initial
hypothesis
“heat maps”
0 5
Where to look next
to maximize
discrimination
Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 61
Obstacle avoidance via optic flow learning
Each point = MAvg over 200 trials
Collision rate with otherwise “missed” UAVs
Cost of crashing ~= cost of deviating course
Cost of crashing >> cost of deviating course