Industrial DVS Design; Key Features and...

Industrial DVS Design;Key Features and Applications

Hyunsurk Eric Ryu, PhD

System LSI, Samsung Electronics

Hyunsurk Eric Ryu Industrial DVS Design 1 of 27

Outline

Industrial DVS DesignHyunsurk Eric Ryu

Samsung S.LSI Dynamic Vision Sensor

Key Features

◼ Low Latency

◼ Minimized Motion Artifacts

◼ Anti-Flicker

◼ Low Power Operation

Applications

◼ Sparse Edge-based Object Recognition

◼ Pose Estimation with Visual Information: DVS-*SLAM

Summary and Discussion*SLAM: Simultaneous Localization and Mapping

2 of 27

DVS Gen1 (R&D Ver., 2014)


Brief Specifications

◼ 640 x 480 Pixel Array with 1/2.5-inch optics

◼ Pixel size : 9 um

◼ Dynamic range : 66 dB (5~10,000 Lux)

◼ Max. event processing rate : 6.5 Meps*

◼ Minimum detectable contrast (50% response)

: < 19%

◼ Interface : 20-bit Parallel

◼ Typical power consumption : 15 mW

Key Features

◼ Small pixel size (9 um)

9.7mm

8.0

mm

640 x 480 Pixel Array

Column Arbiter + Addr. Decoder

Ro

w A

rbite

r + A

dd

r. De

co

de

r

Co

ntro

lL

og

ic +

AM

UX

Bia

s

Ge

n.

Pixel

PD

9um

9u

m

*eps: event per second

Full custom logic design

Fabricated using the Samsung 90-nm Back Side Illumination (BSI) CIS process

3 of 27

DVS Gen2 (R&D Ver., 2016)


(ISSCC ’17)





◼ Max. event processing rate : 300 Meps

◼ Minimum detectable contrast (50% response)

: < 19%

◼ Interface : MIPI(1Gbps 4-lane), USB connectable

Parallel, I2C


Key Features

◼ Standard digital I/F IP supported

◼ Digitally synthesized G-AER* for high throughput*G-AER: Group Address Event Representation

High data throughput using digitally synthesized G-AER scheme

4 of 27

DVS Gen3 (Product Ver., 2018)


Motion image artifact minimized by using global hold and global reset

Timestamp error minimized by applying sequential column scan readout

8.0mm

5.8

mm

640 x 480 Pixel Array

Bias Generator

Pixel Layout

PD

Digital Scan Logic

Event Storage

9um

9u

m

Dig

ital L

og

ic &

Fu

nctio

n B

locks





◼ Effective frame rate : > 2,000 fps (MIPI)

◼ Minimum detectable contrast (99.9% response)

: < 27.5%

◼ Interface : MIPI(1Gbps 4-lane), USB connectable

Parallel, I2C


Key Features

◼ Global hold, Global reset, Column scan readout

5 of 27

DVS Gen4 (R&D Ver.)



◼ 1280 x 960 Pixel Array with 1/2-inch optics

◼ Pixel size : 4.95 um


◼ Frame rate : fixed 1,000 fps (MIPI)

◼ Minimum detectable contrast (99.9% response)

: < 27.4%


Key Features

◼ Pixel circuit split using two-stack wafer bonding

◼ 2nd CCI supported, CIS RGB Format, De-noise,

Anti-Flicker

8372.7um

76

40

.1u

m

1280 x 960 Pixel Array (TOP/BOT)

4.95um

Digital Scan / Core Logic (BOT)

AP

S-D

igit

al

Inte

rfa

ce

(T

OP

)

AP

S-D

igit

al

Inte

rfa

ce

(T

OP

)

Bias GenTIA

ADC

MIPIOTP ABBG TEMP

Event Storage

GH

LD

Comparator Output Logic + Reset Logic

ON

Output

Logic

REQY

ACKY

Reset

LogicRST

OFF

ON_HLD

OFF_HLD

SE

LX

M

Capacitive-feedback

Amplifier (CapAMP)

-A2

C1 C2

Source

Follower (SF)PD + Log I-V Amp (LogAMP)

MFBN

MSF

VLOG

IPD

ISF

-A1VPD

VSF VOUT

GR

ST

TOP (2.8V)

→ BOT (1.8V)

MFBPVBIAS

(simplified) Pixel Diagram

상판 pixel (4x4) C2C (4x4) 하판 pixel (4x4)

Event Storage

GH

LD

Comparator Output Logic + Reset Logic

ON

Output

Logic

REQY

ACKY

Reset

LogicRST

OFF

ON_HLD

OFF_HLD

SE

LX

M

Capacitive-feedback

Amplifier (CapAMP)

-A2

C1 C2

Source

Follower (SF)PD + Log I-V Amp (LogAMP)

MFBN

MSF

VLOG

IPD

ISF

-A1VPD

VSF VOUT

GR

ST

TOP (2.8V)

→ BOT (1.8V)

MFBPVBIAS

(simplified) Pixel Diagram

상판 pixel (4x4) C2C (4x4) 하판 pixel (4x4)

4.9

5u

m

2x2 Pixel

(Bottom)

2x2 Pixel

(Top)

─ Top─ Bot─ Top/Bot

Pixel circuit split by using two-stack wafer bonding for smaller pixel size

Event signal processing block (Anti-flicker, De-noise , etc.) added

6 of 27

DVS Gen2 Results, only valid for small events

Industrial DVS Design 7 of 27Hyunsurk Eric Ryu

DVS Scene Characteristics


0

200

400

600

800

1000

1200

1400

1600

1800

0 10 20 30 40 50 60 70 80

Time (sec)

Eve

nt Rate

(ke

ps)

Event Probability


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0 10 20 30 40 50 60 70

Even

t P

rob

abili

ty

Temporal Contrast (%)

100Lux

5 Lux

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0 5 10 15 20 25 30

Even

t P

rob

abili

ty

Temporal Contrast (%)

…Pixel Array

…

unfair arbitration

Column selection

Column Arbiter

Artifacts and delay by unfair arbitration


Event scanning with a capacitor event memory

…Pixel Array

Fast column sequential read



Original AER handles the individual pixel data with address, polarity, and event generation time.

Group addressing reduces the latency by the interface bandwidth limitation

Pixel Array(640 x 480)

DVS with Event-Group Handshaking Readout

Sam

pling C

ircuit

Dig

itally S

ynth

esiz

ed

Gro

up *

AER

E[0]

E[1]

E[479]

Column Arbiter

CReq

AReq

CAddr[9:0] CAddr_BUF[9:0]

GAddr[5:0]

GData[7:0]

COL_yyyCOL_xxx

CAddr [9:0]

CAddr_BUF[9:0]

COL_yyyCOL_xxx

CReq

AReq

GAddr[5:0]

CLK

GData[7:0]

AER induced latency

*AER: Address Event Representation

12 of 27

AER induced motion artifact


Image artifact could be induced by the mismatch between event generation

time and readout time under high event rate condition

320 320

160 160 160 160

640

…Pixel Array

480

…

Arbitration

Arbitrary column selection

Column Arbiter

640

…Pixel Array

480

…

Sequential Scanning

Sequential column selection

Column Scanner

1

10

100

1000

1 10 100 1000

Sequential Column Selection

Word-serial AER using Arbiter

Tim

esta

mp Err

or

(Late

ncy)

(μsec)

Average Event Rate (Mega event per second)

Input: Poisson random events

13 of 27

Global Hold, Global Reset


Global Hold Off Global Hold On

rotating fan@8,000rpm

rotating fan@8,000rpm

Global Reset Off Global Reset On

One global reset every 32 frames

Tail Events Suppressed Tail Events

Jello Effect

Global hold is implemented with an event storage in each pixel and its global

control signal

To minimize unwanted tail event, global pixel-reset function is applied

14 of 27

Imbedded Anti-Flicker


Event Pass Rate vs. Flicker Frequency

Event Pass Rate vs. Non-Flicker Recordings

Average Non-flicker Event Pass Percentage

97.2%

Shaking Hand before Flickering Monitor

Walking person(AF @non-flickering environment)

15 of 27

Low Power Operation


Activity Detection Spatial Histogram

32

32

(0,0) (0,1) (0,2) (0,3) (0,16) (0,17) (0,18) (0,19)

(1,0) (1,1) (1,2) (1,3) (1,16) (1,17) (1,18) (1,19)

(14,0) (14,1) (14,2) (14,3) (14,16) (14,17) (14,18) (14,19)

One patch comprises 32x32 pixel array

16 of 27

DVS Human Detection


Recognition with securing privacy

Scene DVS RecognitionDVS Output

(Edge Image)

17 of 27


(person, x:230, y:410) (dog, x:70, y:432) (cat, x:321, y:125)

(chair, x:322, y:268) (box, x:120, y:122) (electric fan, x:621, y:325)

Total 850K images (11 categories) are used for network training.

1M-labeled VGA data set, 2.5x faster training and easy converging

Smaller Database and Faster Training

18 of 27


Algorithm Flow

Input Event Stream

Feature Extractor

(Densenet,Darknet)

RegionProposalBranch

Refine-ROIBranch

FRCNN structure

Event Pre-process

HumanDetection

Spatial and Temporal

voting

CountingResult

C++ based Caffe v1 with CPU only mode (no CUDA lib)

◼ DVS event pre-process

Use historic event timestamp of DVS sensor to compensate the sparsity issue.

◼ Deep learning Based Human detection

Faster Region Convolutional Neural Network (FRCNN)

Calculate the position of objects and classify objects.

◼ Spatial and temporal voting

Fusion different detection results to improve accuracy.

19 of 27


Distance ≥ 10 lux 5 ~ 10 lux

Recall

0.5 ~ 1.0m 98% 96%

1.0 ~ 5.0m 99% 98%

5.0 ~ 7.0m 96% 92%

False Alarm Rate (FAR) 1%

Accuracy Test FAR Test

Human Detection in Edge Device (ARMv8 Board)

92ms @ Exynos 7570, 2 cores (60%)

20 of 27

Smaller Network with DVS Images


Sparsity and binary features of DVS images

◼ Few kernel numbers

◼ Few convolutional layers

◼ Large stride for layers in the front

Small and fast network without accuracy degradation.

DVS-Based human detection solution (smart home/IoT)

◼ Computational speed: 140ms/frame @Exynos 7570(1.4GHz, 8% CPU usages)

FRCNN+FPN# of

*Layers*FLOPs

ProcessingTime (ms)

CIS-based 91 5.8G 172

DVS-based 24 81M 15

➔~11.4x speed up

*backbone network**computed on Titan XCIS DVS

21 of 27


DVS SLAM

Durrant-Whyte, Bailey: “Simultaneous Localisation and Mapping ( SLAM ) : Part I The Essential Algorithms”

VO/VIO/VI-SLAM Algorithms

◼ Track location of device in unknown environment

◼ Use visual and inertial sensors

Challenges with standard image sensors

◼ Lack robustness to fast motion (blur)

◼ Lack robustness to HDR (loss of features)

◼ High latency (VR & AR, fast control loops)

DVS intrinsic properties deal with these

challenges naturally

D. Scaramuzza, “Visual Odometry and SLAM: past, present, and the robust-perception age”

22 of 27

DVS-SLAM Algorithm


Yes

DVS Events

IMU

DVS Framer IMU Integration

Frame

Pose EstimationKeyFrameSelection

1. FAST Feature Extraction2. 3D Map Points Projection to

Frame3. Feature Matching4. 3D-2D PnP (Perspective-n-Points)

Map Points Insertion

Using Triangulation

KeyFrame,Map Points

KeyFrames,Map Points

Bundle Adjustment

Optimizeddata

Failure Recovery

Map Points

to Pose Estimation

Location Manager

100 Hz 6-DoF Pose

Mid-termFrame based6DoF Pose

DVS-SLAM Video


DVS-SLAM provides robust tracking even at high-speed motions, thanks to high frame rate (>1000 FPS), sophisticated processing, low

computing load with DVS signal characteristics.


Measured Performances

Comparison with a CIS Platform (May 2019)

DVS-SLAM A CIS Solution

Average Median Average Median

Last Position Error (cm) 3.2 2.7 130 125

AAPE (cm) 2.7 2.3 45.4 39.8

Relative Drift Error (%) 0.28% 0.24% 13.61% 12.6%

Relative AAPE (%) 0.25% 0.16% 4.96% 4.02%

A CIS loses tracking

25 of 27


DVS+CIS SLAM

DVS + CIS: best of both worlds?

◼ DVS provides low latency visual sensing, robustness to fast

motion & HDR

◼ CIS provides precise long term feature tracking, loop closure

& re-localization ability

Sample performance of hybrid DVS+CIS SLAM on

“The Event-Camera Dataset”:

ScenarioVINS CIS Only

AAPE [m]VINS CIS + DVS

AAPE [m]

boxes_6dof 0.29 0.26

boxes_translation 0.09 0.07

hdr_boxes 0.18 0.16

poster_6dof 0.29 0.19

poster_translation 0.02 0.03

dynamic_translation 0.04 0.02

shapes_6dof 0.62 0.49

26 of 27

Summary and discussion


• There were several design changes for the latency, motion artifact, low

power operation, smaller pixel size operation, etc.

• DVS based visual recognition shows fewer and faster NN without loss of

classification accuracy.

• DVS SLAM shows robustness when the motion is fast, and/or for the high

dynamic range scene. DVS with CIS provides better performance than DVS

or CIS only cases.

• Remaining issues and research topics

• bandwidth minimization for higher event rate and spatial resolution

• improving the pixel sensitivity (response time) at dark illumination condition

• A way of high level visual information processing combined with SLAM

information

27 of 27

Industrial DVS Design; Key Features and...

Documents

Transcript of Industrial DVS Design; Key Features and...