eMorph: towards neuromorphic robotic vision · A PCI based high-fanout AER mapper with 2 GiB RAM...
1
eMorph: towards neuromorphic robotic vision Information technology has not yet delivered artificial systems that can compare with biology in reliably, robustly and efficiently extracting information from the often noisy and ambigu- ous real world, and interact- ing with the world by generating appropriate behaviours. Tracker Motion Sensor - measures contrast, spatial and temporal derivative and velocity of moving edges - comprises circuits that implement a model of “selective attention” for a smart readout eMorph is an EU funded project - ICT-FET 231467 Visual System Visual Primitives Technology eMorph will deliver a methodology for event-driven artificial vision on a robotic demonstrator , giving the chance to the robotic community to have a ready-to-use platform to directly interact and experience the power of neuromorphic tools. Visual system integration We developed a series of printed circuit boards to integrate on the iCub event driven sensors and dedicated processors. iCub navigation We developed an autonomous vehicle with 6 holonomic, omnidirectional wheels. Asynchronous, event-driven, space variant visual sensors Goal: to design asynchronous, data-driven, biologically inspired, vision sensors with non-uniform morphology, using analog VLSI neuromorphic circuits High level event-driven visual algorithms Goal: to develop a supporting data-driven asynchronous computational paradigm for machine-vision that is radically different from conventional image processing System integration Goal: to validate the system on the humanoid robot iCub fitted with the neuromorphic sensors, and controlled by the asynchronous machine-vision algorithms in real-time behavioral tasks Publications Bartolozzi, C. and Indiveri, G., Selective Attention in Multi-Chip Address-Event Systems, Sensors, 2009 Orchard, G., Bartolozzi, C. and Indiveri, G., Applying neuromorphic vision sensors to planetary landing tasks, IEEE Biomedical Circuits and Systems Conference, 2009. Hofstätter, M., Schön, P. and Posch, C., A SPARC-compatible general purpose Address-Event processor with 20-bit 10ns-resolution asynchronous sensor data interface in 0.18μm CMOS, IEEE International Symposium on Circuits and Systems, 2010. Posch, C., Matolin, D., Wohlgenannt, R., Hofstaetter, M., Schoen, P., Litzenberger, M., Bauer, D., and Garn, H., Live Demonstration: Asynchronous Time-Based Image Sensor (ATIS) Camera with Full-Custom AE Processor, IEEE International Symposium on Circuits and Systems, 2010. Awarded as best live demonstration Fasnacht, D.B. and Indiveri, G., A PCI based high-fanout AER mapper with 2 GiB RAM look-up table, 0.8 us latency and 66MHz output event-rate, Conference on Information Sciences and Systems (CISS 2011), 2011. Bartolozzi, C., Metta, G., Hofstaetter, M. and Indiveri, G., Event-driven vision for the iCub, Bio-Mimetic and Hybrid Approach to Robotics, Workshop at IEEE International Conference on Robotics and Automation, 2011. Bartolozzi, C., Mandloi, N.K. and Indiveri, G., Attentive motion sensor for mobile robotic applications, IEEE International Symposium on Circuits and Systems, 2011. Bartolozzi, C., Fasnacht, D.B., Rea, F., Hofstaetter, M., Clercq, C., Metta, G. and Indiveri, G., Embedded neuromorphic vision for humanoid robots, Seventh IEEE Workshop on Embedded Computer Vision, 2011 Benosman, R., Ieng, S., Clercq, C., Bartolozzi, C., Asynchronous Frameless Event-based Optical Flow, IEEE Transactions on Neural Networks (submitted) Space variant event-driven sensor prototype Log Polar Mapping temporal contrast pixel illumination spatial derivative noise reduction fovea and periphery Temporal derivative, stability analisys of feedback loop (left) and measured traces corresponding to different biases (right) Adaptive photoreceptor and spatial derivative C. Bartolozzi1, F. Rea1, C. Clercq1,5, N. Mandloi1, G. Indiveri2, D.B. Fasnacht2, M. Hofstaetter3, G. Metta4, R. Benosman5 0.01 0.015 0.02 0.025 0.03 0.035 0.04 0.045 0 1 2 3 Pulse(V) Pxl 2, Slow Pxl 1, Fast 0.01 0.015 0.02 0.025 0.03 0.035 0.04 0.045 1 2 3 Time(s) Pulse(V) Pxl 1, Slow Pxl 2, Fast Simulation of velocity circuits Response to an edge traveling across two pixels at different speeds. The velocity is calculated as time-to-travel across neighbouring pixels Bias Generator Asynchronous Logic Asynchronous Logic Integrate & Fire Change Detector Light Intensity Meta pixel periphery fovea CD CD CD CD CD CD CD CD LI LI LI LI IF CD Dedicated embedded AER hardware Goal: to develop dedicated embedded infrastructure supporting efficient processing of asynchronous data GAEP SPARC-compatible general purpose Address-Event processor with 20-bit 10ns-resolution asynchronous sensor data interface in 0.18μm CMOS FPGA Orange: Interface logic, blue/striped: FIFOs, green: internal logic & control. Arrows: AER data-path (solid), bias control (dashed). iHead Miniaturized embedded board for AER processing, configuration and routing. Comprises the FPGA and the GAEP, CCSAER, USB and ETH connections DVS_L DVS_R FPGA DVS_L_AER DVS_nReset DVS_L_BG BG_PwrDown GAEP JTAG ETH USB BG commands AER data GAEP config & program FPGA code & debug iHEAD CCSAER USB-FX2 Flash Memory SRAM Clock Reset GAEP config DVS_R_AER DVS_R_BG AER data UART BG commands AER data Chip block diagram PC104 i7 CPU Router WiFi Optical flow Event-driven sensors transmit information about local changes in their field of view at the time they occur. The optical flow is updated for each event resulting in extremely sparse computation over time and space. Events from a bouncing ball Consecutive “frames” of accumulated events. Events at t i+1, events computed from optical flow at t i , lag between prediction and real position. Estimated optical flow The computation is frame-less and event-driven. Optical flow snapshot for a 5ms sequence Space-time representation of a rotating white disk with a black bar. Bar orientation estimated by the events-based (+) and by the frame-based (-) optical flow algorithms versus the true bar orientation over time. 1 Istituto Italiano di Tecnologia (IIT) - 2 Institute of Neuroinformatics (INI|UZH|ETH) 3 Austrian Institute of Technology (AIT) - 4 Università di Genova (UNIGE) 5 Institut de le Vision - Université Pierre et Marie Curie
Transcript of eMorph: towards neuromorphic robotic vision · A PCI based high-fanout AER mapper with 2 GiB RAM...
eMorph: towards neuromorphic robotic vision
Information technology has not yet delivered arti�cial
systems that can compare with biology in reliably, robustly and e�ciently
extracting information from the often noisy and ambigu-ous real world, and interact-
ing with the world by generating appropriate
behaviours.
Tracker Motion Sensor- measures contrast, spatial and temporal derivative and velocity of moving edges- comprises circuits that implement a model of “selective attention” for a smart readout
eMorph is an EU funded project - ICT-FET 231467
Visual System Visual Primitives Technology
eMorph will deliver a methodology for event-driven arti�cial vision on a robotic demonstrator, giving the chance to the robotic
community to have a ready-to-use platform to directly interact and experience the power of neuromorphic tools.
Visual system integrationWe developed a series of printed circuit boards to integrate on the iCub event driven sensors and dedicated processors.
iCub navigationWe developed an autonomous vehicle with 6 holonomic, omnidirectional wheels.
Asynchronous, event-driven, space variant visual sensorsGoal: to design asynchronous, data-driven, biologically inspired, vision sensors with non-uniform morphology, using analog VLSI neuromorphic circuits
High level event-driven visual algorithmsGoal: to develop a supporting data-driven asynchronous computational paradigm for machine-vision that is radically di�erent from conventional image processing
System integrationGoal: to validate the system on the humanoid robot iCub �tted with the neuromorphic sensors, and controlled by the asynchronous machine-vision algorithms in real-time behavioral tasks
PublicationsBartolozzi, C. and Indiveri, G., Selective Attention in Multi-Chip Address-Event Systems, Sensors, 2009Orchard, G., Bartolozzi, C. and Indiveri, G., Applying neuromorphic vision sensors to planetary landing tasks, IEEE Biomedical Circuits and Systems Conference, 2009. Hofstätter, M., Schön, P. and Posch, C., A SPARC-compatible general purpose Address-Event processor with 20-bit 10ns-resolution asynchronous sensor data interface in 0.18μm CMOS, IEEE International Symposium on Circuits and Systems, 2010.Posch, C., Matolin, D., Wohlgenannt, R., Hofstaetter, M., Schoen, P., Litzenberger, M., Bauer, D., and Garn, H., Live Demonstration: Asynchronous Time-Based Image Sensor (ATIS) Camera with Full-Custom AE Processor, IEEE International Symposium on Circuits and Systems, 2010. Awarded as best live demonstrationFasnacht, D.B. and Indiveri, G., A PCI based high-fanout AER mapper with 2 GiB RAM look-up table, 0.8 us latency and 66MHz output event-rate, Conference on Information Sciences and Systems (CISS 2011), 2011.Bartolozzi, C., Metta, G., Hofstaetter, M. and Indiveri, G., Event-driven vision for the iCub, Bio-Mimetic and Hybrid Approach to Robotics, Workshop at IEEE International Conference on Robotics and Automation, 2011.Bartolozzi, C., Mandloi, N.K. and Indiveri, G., Attentive motion sensor for mobile robotic applications, IEEE International Symposium on Circuits and Systems, 2011.Bartolozzi, C., Fasnacht, D.B., Rea, F., Hofstaetter, M., Clercq, C., Metta, G. and Indiveri, G., Embedded neuromorphic vision for humanoid robots, Seventh IEEE Workshop on Embedded Computer Vision, 2011Benosman, R., Ieng, S., Clercq, C., Bartolozzi, C., Asynchronous Frameless Event-based Optical Flow, IEEE Transactions on Neural Networks (submitted)
Space variant event-driven sensor prototypeLog Polar Mapping
temporal contrastpixel illumination spatial derivativenoise reductionfovea and periphery
Temporal derivative, stability analisys of feedback loop (left) and measured traces corresponding to di�erent biases (right)
Adaptive photoreceptorand spatial derivative
C. Bartolozzi1, F. Rea1, C. Clercq1,5, N. Mandloi1, G. Indiveri2, D.B. Fasnacht2, M. Hofstaetter3, G. Metta4, R. Benosman5
0.01 0.015 0.02 0.025 0.03 0.035 0.04 0.0450
1
2
3
Pul
se(V
)
Pxl 2, SlowPxl 1, Fast
0.01 0.015 0.02 0.025 0.03 0.035 0.04 0.045
1
2
3
Time(s)
Pul
se(V
)
Pxl 1, SlowPxl 2, Fast
Simulation of velocity circuits Response to an edge traveling
across two pixels at di�erent speeds. The velocity is
calculated as time-to-travel across neighbouring pixels
Bias Generator
Asynchronous Logic
Asy
nchr
onou
s Lo
gic
Integrate & Fire
Change Detector
Light Intensity
Meta pixel
periphery
fovea
CD CD
CD CDCD
CD CDCD
LILI
LI
LI
IF
CD
Dedicated embedded AER hardwareGoal: to develop dedicated embedded infrastructure supporting e�cient processing of asynchronous data
GAEP SPARC-compatible general purpose Address-Event processor
with 20-bit 10ns-resolution asynchronous sensor data interface in 0.18µm CMOS
FPGA Orange: Interface logic, blue/striped: FIFOs, green: internal logic & control. Arrows: AER data-path (solid), bias control (dashed).
iHeadMiniaturized
embedded board for AER
processing, con�guration
and routing. Comprises the FPGA and the GAEP, CCSAER, USB and ETH connections
DVS_L
DVS_RFPGA
DVS_L_AER
DVS_nReset
DVS_L_BG
BG_PwrDown
GAEP
JTAG
ETH
USB BG commandsAER data
GAEP con�g& program
FPGA code& debug
iHEAD
CCSAER
USB-FX2
Flash Memory SRAM
Clock
Reset
GAEP con�g
DVS_R_AERDVS_R_BG
AER
dat
a
UART
BG commandsAER data
Chip block diagram
PC104
i7 CPU
RouterWiFi
Optical �owEvent-driven sensors transmit information about local changes in their �eld of view at the time they occur. The optical �ow is updated for each event resulting in extremely sparse computation over time and space.
−20 0 20 40 60 80 100 120 140−20
0
20
40
60
80
100
120
140
Events from a bouncing ballConsecutive “frames” of accumulated events.
Events at ti+1, events computed from optical �ow at ti, lag between prediction and real position.
Estimated optical �owThe computation is frame-less and
event-driven.
Optical �ow snapshot for a 5ms sequence
Space-time representation of a rotating white disk with a black bar. Bar orientation estimated by the events-based (+) and by the frame-based (-) optical �ow algorithms versus the true bar orientation over time.
1 Istituto Italiano di Tecnologia (IIT) - 2 Institute of Neuroinformatics (INI|UZH|ETH)3 Austrian Institute of Technology (AIT) - 4 Università di Genova (UNIGE)
5 Institut de le Vision - Université Pierre et Marie Curie
