Fast Reactive Illumination through Rain and Snow

Sponsors: ONR, NSF, Intel, Samsung

Carnegie Mellon University

Raoul de Charette Robert Tamburo

Peter Barnum

Anthony Rowe Takeo Kanade

Srinivasa Narasimhan

Driving in Snow at Night

Berlin, Germany 55

Brussels, Belgium 65

Cape Town, SA 52

Chicago, USA 48

Kuala Lumpur, Malaysia 79

Paris, France 56

Pittsburgh, USA 59

Seattle, USA 62

Tokyo, Japan 50

Zurich, Switzerland 64

[ World Meteorological Organization, 30 year average ]

How many Rainy/Snowy Nights in a Year?

Post-processing: De-raining and De-snowing

=

Rain frequencies

Scene frequencies

[Barnum et al, 07]

Headlight that sees through Rain and Snow

Goal: High Light Throughput and Accuracy

[Similar in spirit to “Lighting up dust”, Ken Perlin]

… a detour

[ Barnum, Narasimhan, Kanade, 10 ]

Rain Streaks versus Rain Drops

Long Exposure Time (12 ms) Short Exposure Time (1 ms)

Rain streaks appear dense but drops are sparse

Light Throughput and Camera Exposure

0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 80

85

90

95

100

Exposure time (ms)

Lig

ht

thro

ug

hp

ut

(%)

0.5mm/h

2mm/h

17mm/h

Operating Range: Visibility of Rain Streaks

Distance to the light source (m)

Lig

ht

inte

nsi

ty (

ftc)

0

500

1000

1500

2000

0 1 2 3 4 5 6

Headlight

DLP Projector

Halogen Light

Not visible beyond 5-6 meters from the source

Reactive Illumination

operating range

camera

projector

beam splitter

capture

system latency

reactive control

Capture

Frame #

Frame 1

Frame 2

Frame 3

Frame 4

TX TX Projection Process

Time (ms) 5 9 14 18 27 36

5 4 5 4 9

Projection

System Pipeline

Stage 3 Stage 2 Stage 1

Capture TX TX Process

Projection Capture TX TX Process

Capture TX TX Process

Latency

Prototype System in Lab

Detecting, Tracking and Predicting Drops

View from the system camera

Performance with Tracking Latency

0 5 10 15 20 25 0

20

40

60

80

100

Tracking latency (frames)

Acc

ura

cy (

%)

1000 Hz

500 Hz

200 Hz

100 Hz

50 Hz

Performance under Detection Errors

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 90

92

94

96

98

100

Detection error (pixel)

Lig

ht

thro

ug

hp

ut

(%)

Rain (5mm/h)

Snow (1mm/h)

Slow Capture and Projection

Standard headlight Smart headlight Not so

Projecting black streaks instead of white streaks

High Speed Bit-Plane Projection

[Raskar et al, Narasimhan et al, Debevec et al]

What we see….or don’t see (30 Hz)

What a high-speed camera sees (2000 Hz)

[ Temporal integration ]

Making Rain Disappear (90 mm/h)

System Latency

Standard headlight Smart headlight

Making Rain Disappear (90 mm/h)

Standard headlight Smart headlight

Performance of our system

System Latency: 13 ms Imaging exposure: 5ms Tracking latency: 1 frames

Resolution: 1024 x 1024 Car motion: 30 kph Projection rate: 120 Hz

0 5 10 15 20 25 40

50

60

70

80

90

100

Fallrate (mm/h)

Lig

ht

thro

ug

hp

ut

(%)

Snow (accuracy = 63.4%)

Rain (accuracy = 68.9%)

Performance of an Ideal system

System Latency: 0 ms Imaging exposure: 1ms Tracking latency: 0 frames

Resolution: 1024 x 1024 Car motion: 30 kph Projection rate: 10000 Hz

0 5 10 15 20 25 40

50

60

70

80

90

100

Fallrate (mm/h)

Lig

ht

thro

ug

hp

ut

(%)

Rain (accuracy = 100%)

Snow (accuracy = 100%)

+

-

-

-

-

High-speed computations

Projector and camera in one device?

Embedded Design for Imaging and Illumination

Rain Rain Go Away…

• Improves driver visibility directly • Simulations suggest that the idea is feasible

• Initial lab prototype is encouraging

• Still a long way to go:

• Wind, turbulence, vibrations

• Making the device compact, fast moving car