"Computer Vision in Cars: Status, Challenges, and Trends," a Presentation from videantis

Copyright © 2016 videantis GmbH 1

Marco Jacobs

May 3, 2016

Computer Vision in Cars:

Status, Challenges, and Trends


About videantis

#1 vision

processor

10+ years in

business

100% vision

company

automotive

since 2008


Market status


Driverless vehicles

Life saver, time saver, cost saver


WHO: 1.2M deaths per year world-wide



USA: ~5 years of your life in a car



USA insurance: ~$0.10/mile



90% autonomous cars: 2x road capacity



Future of transportation?

www.next-future-mobility.com

Hyperloop

“All you can eat” subscriptions?


Only autonomous people mover today…


Bosch CEO at CES 2016:

• “Next decade maybe”

• What works:

• Low speed, parking: now

• Highway, exit to exit: ~2020

Fully autonomous cars possible?


Driver

monitors

at all

times

Vehicle

drives

itself – not

100%

safely

Change is gradual (OICA’s 6 levels)

Driver

operates

vehicle

Driver

holds

wheel

or

controls

pedals

Driver

ready to

regain

control

Driver not

required

at all times

Vehicle

steers or

controls

speed

Vehicle

drives

itself – but

may give

up control

Vehicle

drives

itself –

during

specific

use case

(e.g.

highway)

Vehicle

drives

itself

door to

door

L0 L4 L1 L2 L3 L5 L

0

L

1

L

2

L

3

L

4

L

5


Driver

monitors

at all

times

Vehicle

drives

itself – not

100%

safely

Gradually increasing automation

Driver

operates

vehicle

Driver

holds

wheel

or

controls

pedals

Driver

ready to

regain

control

Driver not

required

at all times

Vehicle

steers or

controls

speed

Vehicle

drives

itself – but

may give

up control

Vehicle

drives

itself –

during

specific

use case

(e.g.

highway)

Vehicle

drives

itself

door to

door

L0 L4 L1 L2 L3 L5 L

0

L

1

L

2

L

3

L

4

L

5


>$1T business excluding

infrastructure, fuel,

insurance

Market in numbers

1.2B vehicles on the road

100M cars sold each year

20 OEMs produce >1M

each year

100 Tier 1s >$1B revenue

$800B combined

~25% of cost is electronics

Today: ~0.4 cameras/new car Opportunity: 10 cameras/car


Drivers: NHTSA & Euro NCAP ratings

L

2

L

0

Rear camera mandatory Front camera rating


Where do the cameras go?


Cameras to extend visibility

Rear Mirror Surround

L

0

L

0

L

0


Cameras with embedded vision (all of them)

Rear Mirror Surround Front Driver

L

2

L

2

L

2

L

3

L

0


Typical functions included:

• Wide angle lens dewarp

• Graphics for guidelines

• H264 compress for transmission over

automotive Ethernet

Vision technology:

• Real-time camera calibration

• Dirty lens detection

• Parking assist

• Cross traffic alert (1)

• Backover protection (2)

• Trailer steering assist (3)

Rear camera – enhances visibility

(2)

(1)

(1)

(3)

L

2


Surround view – enhances visibility


• Image stitching and reprojection

Vision technology:

• Structure from Motion

• Automated parking assist:

• marker detection

• free parking space detection

• obstacle detection

(includes everything rear camera has)

Courtesy: Magna

L

2


Mirror replacement - reduces drag, expands view

Courtesy: Ficosa


• image stitching and warping

• blind spot detect (1)

• rear collision warning (2)

Vision technology:

• object detection

• optical flow

• structure from motion

(includes rear camera features)

(2)

(1) L

0


Front camera – controls speed & steering wheel


• Emergency braking, auto cruise control

• Pedestrian and vehicle detection

• Lane detection and keeping

• Traffic sign recognition

• Headlight control

• Bicycle recognition (2018)

• Intersections (2020)

(Often combined with radar)

L

3


Driver monitoring

Typical features:

• Driver drowsiness detection

• Driver distraction detection

• Airbag deployment

• Seatbelt adjustment

• Driver authentication

Vision technology:

• Face detect

• Face analysis

• Posture detection

L

0


Technologies inside


Image processing pipeline

Computer

vision

Image

processing

• ISP (HDR)

• Comp photo

• Lens correct

• Reprojection

• 360º video

• Depth / 3D extraction:

• Structure from Motion

• Stereo disparity

• Object detection

• CNN, Haar, HOG, etc

• Face analysis

• Optical flow: track objects

Transmit

• ETH or LVDS • H.264, JPEG • HDR, 10/12b • Low delay

User interface

• Graphics / Video

Vision path

Display path


Structure from Motion

Structure from

motion algorithm

camera origin

and direction

+ calibrated camera

• Example applications:

Automated parking

Cross-traffic alert

3D pointcloud


Automotive challenges (& opportunities)


• Cold & hot: low power

• Dark & light: HDR, noise

• Dirty lens: detect algos

• 0-120mph: different algos

• Car loaded/dinged: calibrate

• … and many more

Challenge: work under all conditions…


Challenge: under severe power constraints…

Power source

100kW

Small form

factors limits

heat dissipation

Complete smart

camera <1W


Central processing

• Pros

• Single processing platform

eases software dev

• Cons

• Entry-level car also needs

high-end head unit

• Not scalable, not modular

• Adding cameras causes

system overload

Challenge: central or distributed processing?

Central processing uses expensive

head unit, even in low-end car


Distributed processing

• Pros

• Low-end head unit

• Options become plug-and-

play

• Every camera adds

processing capabilities

• Cons

• More complex system

This is reality today:

• Some cars have 250 ECUs

Scalable architecture provides

most flexibility, lowest entry cost

Challenge: central or distributed processing?


• Business opportunity is huge: >$1T

• Self-driving car tech causes paradigm shift

• New players can grab market share

• Automotive is not like consumer electronics

• Next 10 years no self-driving cars:

• change will be gradual, lots of driver

assist functions with vision technologies

• Efficient computer vision systems are the

key enabler for making our cars safer

Conclusion


Thank you

Marco Jacobs

May 3, 2016

"Computer Vision in Cars: Status, Challenges, and Trends," a Presentation from videantis

Technology

Transcript of "Computer Vision in Cars: Status, Challenges, and Trends," a Presentation from videantis