Digital Foveation: an Energy-Aware Machine Vision Framework

Digital Foveation:an Energy-Aware Machine Vision Framework

Ekdeep Singh Lubana† and Robert P. Dick‡

[email protected] and [email protected]

† Department of Electronics and Communication EngineeringIndian Institute of Technology, Roorkee

‡ Department of Electrical Engineering and Computer ScienceUniversity of Michigan

Machine Vision: Achievements and LimitationsDigital Foveation: Energy Efficient Machine VisionEvaluation on Various Machine Vision Applications

Machine vision importanceEnergy efficiencyBiologically inspired machine vision

Outline

1. Machine Vision: Achievements and Limitations

2. Digital Foveation: Energy Efficient Machine Vision

3. Evaluation on Various Machine Vision Applications

2 Lubana & Dick Digital Foveation



Machine vision important and becoming ubiquitous

Machine vision: use imaging systems to capture/process spatial information.

Market value: approx. $15 billion within next 5 years.

Security: License platedetection.

Banking: Automatedreading of check

contents.Surveillance: Pedestrian

detection.




Machine vision is energy-intensive

High energy consumption limits embedded deployment scenarios.

Example: Continuous face detection drains Google Glass battery in 38 minutes.

“Draining our Glass: An Energy and Heat Characterization of Google Glass”(Tech. Report; Likamwa et al. 2014).




Machine vision is energy-intensive

Computer vision energy generally proportional to throughput.

Pedestrian detection using HOG detector on ARM Cortex A53.Time and energy scale with throughput (image size).




Reasons for high-energy consumption

Machine vision pipeline is unidirectional–using uniform and high resolutiondata across the system.

Masterclock

I/O controller

Memory

EncoderDigital processing

Image signal processorImage sensor

controllerI/O Readout

Decoder

Pixel array

PLL

algorithms

Analysis via

Host processor

detection

Conventional pipeline.




How does human vision minimize energy consumption?

Foveated sensing: variable, multi-resolution spatio-temporal sensing patterns.

Peripheral region

Fovea

Sparsely sampling peripheral region isused for detection; densely samplingfovea is used for characterization.

Comparison between the image humansperceive (left) vs. the image they

actually see (right). A very narrow regionis sampled at high resolution

(approximately 1% of the scene).



Outline






Digital Foveation: biological inspiration

Adaptive throughput optimization via multi-resolution processing.

Drop pixels asper subsampling

level

Subsampled imageDetected object

Segmented object

of interest

Detection

Fe

ed

ba

ck



Digital Foveation: integration with camera hardware

Row/Column skipping, frame preservation, and random access via readoutcircuitry.

Object detection

Host processor

Foveal coordinates(Object of interest)

Image signalprocessor

ing/encodingDigital process−

Subsamp−led image

Reuse captured frame

Skip

Read

Skip

Skip

Image signal processor

encodingDigital processing/Characterization

offoveal image

Image sensor Row/Columnskipping

Skip

Read

Skip

Read

Random accessvia row/column

decoder



Energy evaluationPerformance evaluation

Outline







Evaluation setup

Image Sensor: Sony IMX219.

Microcontroller: Raspberry Pi 3 (ARM Cortex A53 with Videocore IV GPU).




Energy characterization of imaging pipeline

Model power as function of component power management state–durations.Time

Idle

Idle

Standby

Active


Host processor

Peripherals

Object detectionReadoutExposuresampled, full−resolution imagesDigital processing of uniformly

Image sensor Active

Active

Active

Idle

Idle

Conventional machine vision pipeline.

Rea

dout

of fo

veal

regi

on

Exp

osur

e

Rea

dout

subs

ampl

ed

proc

essing

of

Dig

ital

imag

e Obj

ect

dete

ctio

n

proc

essing

of

Dig

ital

fove

al im

age

Time

Active

Idle


Host processor

Peripherals

Image sensor StandbyIdle Active

Idle

Active

Active

Active

Active

Standby

Idle

Idle

complete;Analysis

may move on toprocess nextimage or sleep

State-timing diagram for Digital Foveation pipeline.




Time characterization of imaging pipeline

Time consumptions for license plate detection application.




Evaluation results

Energy consumptions for license plate detection application.




The energy-accuracy tradeoff

Use of low resolution images for detection reduces the accuracy of thedetection algorithm.

For given accuracy constraints, the following optimization problem needs tobe solved:

minerror≤thresh

System Energy




Averting energy-performance tradeoff

Significance of energy-performance tradeoff depends on application setupand user-defined accuracy constraints.

Sufficient rounds to optimize performance, under accuracy constraints, indifferent single frame capture scenarios.

Similarly sized objects–ATM camera. Variably sized objects–Pedestriandetection.




Similarly sized objects: single, optimized subsampling level

One subsampling level sufficient for license plate detetection application.

Similarly sized objects.

Unsubsampled 2x2 subsampled 4x4 subsampled0

20

40

60

80

100

Relative energy consumption

Detection accuracy

Characterization accuracy

Aggregate accuracy




Variably sized objects: adaptive subsampling

Subsampling level will depend on object size.




Variably sized objects: face detection

Calculate ideal subsampling levels for each category and given constraints.

0 Unsubsampled 2x2 subsampled 4x4 subsampled 8x8 subsampled

0

20

40

60

80

100

Category 1 (s = 0-0.09)

Category 2 (s = 0.09-0.17)

Category 3 (s = 0.17-0.35)




Variably sized objects: face detection

Adaptive subsampling improves accuracy with similar energy.

Coarse sampling for nearby (large) objects, fine for distant (small) objects.

Accuracy preserved if large objects misclassified as small.

Unsubsampled 2x2 subsampled 4x4 subsampled Adaptively subsampled

0

20

40

60

80

100

Detection

accuracy

Relative energy

consumption




Summary

Digital Foveation–an energy-aware machine vision framework inspired fromthe multi-round, multi-resolution human vision pipeline.

The framework reduces energy consumption by 75–85% at minimal to nodecrease in accuracy.




Video based applications

Reconfiguration latency, depending on the camera, can be a bottleneck forDigital Foveation

Android cameras’ latency can range from 300-400 ms

The presented analysis implicitly accounted for reconfiguration latency

Saccades–resolution reconfiguration in humans

latency order: 50–60 ms




Hardware support

Most modern sensors allow random access.

Some even use energy-saving routines for standard subsampling modes–ONSemiconductors NOII4SM6600A-D.

What needs to be updated? Firmware.




Integration with vision-accelerators

The framework is generic and reasons for efficiency are orthogonal toparallelized vision accelerators.

More absolute savings are expected. We hope to uptake this work in future.




Future work

Generative networks for finding ideal sensing patterns at minimal datarequirements.

Variable bit rates and SAR-ADCs’ integration.

Application-oriented accelerators and algorithms, for aesthetics do notmatter.


Digital Foveation: an Energy-Aware Machine Vision Framework

Documents

Transcript of Digital Foveation: an Energy-Aware Machine Vision Framework