Computational Illumination - MIT CSAILgroups.csail.mit.edu/graphics/classes/CompPhoto06/... · 7...

1

Ramesh Raskar, Computational Illumination

Computational IlluminationComputational Illumination

Ramesh RaskarMitsubishi Electric Research Labs

Course WebPage : http://www.merl.com/people/raskar/photo/


Computational Illumination

Traditional Traditional ‘‘filmfilm‐‐likelike’’ PhotographyPhotography

Lens

Detector

Pixels

Image

Computational PhotographyComputational Photography: : Optics, Sensors and ComputationsOptics, Sensors and Computations

GeneralizedSensor

GeneralizedOptics

Computations

Picture

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Computational PhotographyComputational Photography

Novel CamerasGeneralizedSensor

GeneralizedOptics

Processing

Computational PhotographyComputational Photography Novel Illumination

Novel CamerasGeneralizedSensor

GeneralizedOptics

Processing

Light Sources

2


Novel Cameras

Scene: 8D Ray Modulator

GeneralizedSensor

GeneralizedOptics

Processing

Light Sources


Novel Cameras


Display

GeneralizedSensor

GeneralizedOptics

Processing

Recreate 4D Lightfield

Light Sources


Novel Cameras


Display

GeneralizedSensor

GeneralizedOptics

Processing


Ray Sampler

Ray Reconstruction

GeneralizedOptics


Light Sources

Modulators

4D Incident Lighting

4D Light Field


Novel Cameras


Display

GeneralizedSensor

GeneralizedOptics

Processing


Ray Sampler

Ray Reconstruction

GeneralizedOptics


Light Sources

Modulators

4D Light Field

Programmable 4D Illumination field + time + wavelength


‘Smarter’ Lighting Equipment

What Parameters Can We Change ?What Parameters Can We Change ?

Edgerton 1930Edgerton 1930’’ss

3

Edgerton 1930Edgerton 1930’’ss

Multi‐flash sequential photography

Stroboscope(Electronic Flash)

CameraExposure

Flash Time


Computational Illumination:Computational Illumination:Programmable 4D Illumination Field + Time + WavelengthProgrammable 4D Illumination Field + Time + Wavelength

•• Presence or Absence, Duration, BrightnessPresence or Absence, Duration, Brightness–– Flash/NoFlash/No--flashflash

•• Light positionLight position–– MultiMulti--flash for depth edgesflash for depth edges–– Programmable dome (image reProgrammable dome (image re--lighting and matting)lighting and matting)

•• Light color/wavelengthLight color/wavelength

•• Spatial ModulationSpatial Modulation–– Synthetic Aperture IlluminationSynthetic Aperture Illumination

•• Temporal ModulationTemporal Modulation–– TV remote, Motion Tracking, Sony IDTV remote, Motion Tracking, Sony ID--cam, RFIGcam, RFIG

•• Exploiting (uncontrolled) natural lighting conditionExploiting (uncontrolled) natural lighting condition–– Day/Night FusionDay/Night Fusion


Computational IlluminationComputational Illumination•• Presence or Absence, Duration, BrightnessPresence or Absence, Duration, Brightness

–– Flash/NoFlash/No--flashflash•• Light positionLight position

–– MultiMulti--flash for depth edgesflash for depth edges–– Programmable dome (image reProgrammable dome (image re--lighting and matting)lighting and matting)




•• General lighting conditionGeneral lighting condition–– Day/NightDay/Night

Denoising Challenging Images

Available light:+ nice lighting

- noise/blurriness- color

No-flash

Flash:+ details+ color

- flat/artificial

Flash

Denoise no-flash image using flash image

Flash

No-flash

Result

Elmar Eisemann and Frédo Durand , Flash Photography Enhancement via Intrinsic Relighting

Georg Petschnigg, Maneesh Agrawala, Hugues Hoppe, Richard Szeliski, Michael Cohen, Kentaro Toyama. Digital Photography with Flash and No-Flash Image Pairs

4

+ original lighting+ details/sharpness+ color

Result

No-flash

Transfer detail from flash image to no-flash imageCross-Bilateral Filter based Approach

Cross Bilateral Filter

• When no-flash image is too noisy

• Borrow similarity from flash image

edge stopping from flash image

Bilateral Cross Bilateral

Detail Layer

Intensity Large-scale

/ =Detail

Recombination: Large scale * Detail = Intensity

Need flash component!

FlashAmbient

Build Exposure HDR image

• Multiple images with different exposure

– Debevec & Malik, Siggraph 97– Nayar & Mitsunaga, CVPR 00

Increasing Exposure

5

Flash Intensity

Build Flash HDR image Flash‐Exposure Sampling

Build Flash‐ExposureHDR image

Exposure

Flash Intensity

Agrawal, Raskar, Nayar, LiSiggraph05


Varying Exposure time Varying Flash brightness Varying both

Capturing HDR Image

Flash Result Reflection LayerAmbient

Flash and Ambient Images[ Agrawal, Raskar, Nayar, Li Siggraph05 ]

Intensity Gradient Vector Projection

Intensity Gradient Vectors in Flash and Ambient Images

Same gradient vector direction Flash Gradient Vector

Ambient Gradient Vector

Ambient Flash

No reflections

6

Reflection Ambient Gradient Vector

Different gradient vector direction

With reflections

Ambient Flash

Flash Gradient Vector Residual Gradient Vector

Intensity Gradient Vector Projection

Result Gradient Vector

Ambient Flash Result Residual

Reflection Ambient Gradient Vector

Flash Gradient Vector

FlashProjection = Result

Residual =Reflection Layer

Co-located Artifacts

Ambient


Computational IlluminationComputational Illumination•• Presence or Absence, Duration, BrightnessPresence or Absence, Duration, Brightness


–– Programmable dome (image reProgrammable dome (image re--lighting and matting)lighting and matting)–– MultiMulti--flash for depth edgesflash for depth edges






Synthetic LightingSynthetic LightingPaul Paul HaeberliHaeberli, Jan 1992, Jan 1992


DebevecDebevec et al. 2002: et al. 2002: ‘‘Light Stage 3Light Stage 3’’

7


ImageImage--Based Actual ReBased Actual Re--lightinglighting

Film the background in Milan,Film the background in Milan,Measure incoming light,Measure incoming light,

Light the actress in Los AngelesLight the actress in Los Angeles

Matte the backgroundMatte the background

Matched LA and Milan lighting.Matched LA and Milan lighting.

Debevec et al., SIGG2001


courtesy of P. Debevec

courtesy of A Agrawala

Photomontage

Ramesh Raskar, Computational Illumination Ramesh Raskar, Computational Illumination

TableTable--top Computed Lighting for top Computed Lighting for Practical Digital Photography Practical Digital Photography

Ankit Mohan, Jack TumblinAnkit Mohan, Jack TumblinNorthwestern UniversityNorthwestern University

Bobby Bobby BodenheimerBodenheimerVanderbilt UniversityVanderbilt University

Cindy Grimm, Cindy Grimm, ReynoldReynold BaileyBaileyWashington University in St. LouisWashington University in St. Louis


Move shadow back

Soften this shadow

Make this brighter

Remove this specular highlight

Make this darker


Target

Result

Sketch Your Desires, OptimizeSketch Your Desires, Optimize

8


Acquisition for RelightingAcquisition for Relighting

object

•• Uniquely lit Uniquely lit basis imagesbasis images

•• KnownKnown lightlight--positionspositions


‘‘Aimed SpotAimed Spot’’: low: low--risk movementrisk movement


From Jack Tumblin


Overlapped Spots avoid aliasingOverlapped Spots avoid aliasing


““Light WavingLight Waving””Tech Sketch (Tech Sketch (WinnemollerWinnemoller, Mohan, Tumblin, Gooch), Mohan, Tumblin, Gooch)


Light Waving: Estimating Light Light Waving: Estimating Light Positions From Photographs Positions From Photographs

AloneAlone

HolgerHolger WinnemWinnemööllerller, Ankit Mohan, , Ankit Mohan, Jack Tumblin, Bruce GoochJack Tumblin, Bruce Gooch

Northwestern University Northwestern University

9


Computational IlluminationComputational IlluminationQuest for 4D IlluminationQuest for 4D Illumination

Novel Cameras


Display

GeneralizedSensor

GeneralizedOptics

Processing


Ray Sampler

Ray Reconstruction

GeneralizedOptics

Light Sources

Modulators

4D Light Field



Ramesh Raskar, Computational Illumination[Debevec et al. 2002]

[Debevec et al. 2000] [Masselus et al. 2002]

[Masselus et al. 2003] [Malzbender et al. 2002]

[Matusik et al. 2002]

A 4A 4--D Light SourceD Light Source

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera

Ramesh Raskar, Karhan Tan, Rogerio Feris, Jingyi Yu, Matthew Turk

Mitsubishi Electric Research Labs (MERL), Cambridge, MAU of California at Santa BarbaraU of North Carolina at Chapel Hill

NonNon--photorealistic Camera: photorealistic Camera: Depth Edge Detection Depth Edge Detection andand Stylized Rendering Stylized Rendering usingusing

MultiMulti--Flash ImagingFlash Imaging


Depth Edge Camera

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera

10




11




Depth Discontinuities

Internal and externalShape boundaries, Occluding contour, Silhouettes


Depth Edges

12


Sigma = 9 Sigma = 5

Sigma = 1

Canny Intensity Edge Detection

Our method captures shape edges


Our MethodCanny


Our Method

PhotoMitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera

Canny Intensity Edge Detection

Our Method

Photo Result


Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

13


Shadows

Clutter

Many Colors

Highlight Edges

Mark moving parts

Basic colors

A New ProblemA New Problem



Imaging Geometry

Shadow lies along epipolar ray


Shadow lies along epipolar ray,

Epipole and Shadow are on opposite sides of the edge

Imaging Geometry

m


Shadow lies along epipolar ray,

Shadow and epipole are on opposite sides of the edge

Imaging Geometry

m

14


Depth Edge Camera

Light epipolar rays are horizontal or vertical


Normalized

Left / Max

Right / Max

Left Flash

Right Flash

Input U{depth edges}


Normalized

Left / Max

Right / Max

Left Flash

Right Flash

Input U{depth edges}Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera

Normalized

Left / Max

Right / Max

Left Flash

Right Flash

Input U{depth edges}

Negative transition along epipolar ray is depth edge

Plot


Normalized

Left / Max

Right / Max

Left Flash

Right Flash

Input

Negative transition along epipolar ray is depth edge

Plot U{depth edges}Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera

% Max compositemaximg = max( left, right, top, bottom);

% Normalize by computing ratio imagesr1 = left./ maximg; r2 = top ./ maximg;r3 = right ./ maximg; r4 = bottom ./ maximg;

% Compute confidence mapv = fspecial( 'sobel' ); h = v';d1 = imfilter( r1, v ); d3 = imfilter( r3, v ); % vertical sobeld2 = imfilter( r2, h ); d4 = imfilter( r4, h ); % horizontal sobel

%Keep only negative transitionssilhouette1 = d1 .* (d1>0); silhouette2 = abs( d2 .* (d2<0) );silhouette3 = abs( d3 .* (d3<0) );silhouette4 = d4 .* (d4>0);

%Pick max confidence in eachconfidence = max(silhouette1, silhouette2, silhouette3, silhouette4);imwrite( confidence, 'confidence.bmp');

No magicparameters !

15


Related WorkRelated Work

• Stylized image processing– [Hertzmann 98] [DeCarlo and Santella 02][Waking Life]

• Relies on segmentation of image, • Stylization rather than comprehension

• Shape from shadow– Shadow grams [Savarese et al 01]

• Look at smooth surfaces– Building heights [Lin et al 1998]

• Assumes flat ground and uniform albedo


ComparisonComparisonLess data but more robustLess data but more robust

• Traditional stereo (camera pair)– Correspondence matching fails at depth edges– Requires texture

• Photometric stereo (moving light source)– For smooth surfaces, fails at depth edges– Large lighting variation required– Not a self-contained device

• 3D range scanners– Expensive, low resolution, low frame rate


LimitationsLimitations

• Difficult conditions– Outdoor, bright scenes– Transparent, low albedo, mirror-like surfaces– Thin narrow objects

• Issues– Baseline between camera and flash– Specularities– Flash non-uniformity, area light source

• Comprehensibility– Sharp edges not captured


Change DetectionChange Detection

Before After


Change DetectionChange DetectionMitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera

Change DetectionChange Detection

Reconstructed from gradient field of new depth edges

16


Computational IlluminationComputational Illumination•• Presence or AbsencePresence or Absence




•• Spatial Modulation (IntraSpatial Modulation (Intra--flash 2D Modulation)flash 2D Modulation)–– Synthetic Aperture IlluminationSynthetic Aperture Illumination



6-D Methods and beyond...Relighting with 4D Incident Light Fields Vincent Masselus, Pieter Peers,

Philip Dutre and Yves D. Willems SIGG2003

© 2004 Marc Levoy

Synthetic Aperture Illumination: Comparison with Long-range synthetic aperture photography

• width of aperture 6’• number of cameras 45• spacing between cameras 5”• camera’s field of view 4.5°

© 2004 Marc Levoy

The scene

• distance to occluder 110’• distance to targets 125’• field of view at target 10’

© 2004 Marc Levoy

Synthetic aperture photographyusing an array of mirrors

• 11-megapixel camera (4064 x 2047 pixels)• 18 x 12 inch effective aperture, 9 feet to scene • 22 mirrors, tilted inwards → 22 views, each 750 x 500 pixels

© 2004 Marc Levoy

Synthetic aperture illumination

• technologies– array of projectors– array of microprojectors– single projector + array of mirrors

17

© 2004 Marc Levoy

What does synthetic aperture illumination look like?

What are good patterns?pattern one trial 16 trials

© 2004 Marc Levoy

Underwater confocal imagingwith and without SAP









Demodulating Cameras

• Simultaneously decode signals from blinking LEDs and get an image

– Sony ID Cam– Phoci

• Motion Capture Cameras– Visualeyez™ VZ4000 Tracking System– PhaseSpace motion digitizer

18

Demodulating Cameras

• Decode signals from blinking LEDs + image – Sony ID Cam– Phoci

• Motion Capture Cameras

Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang,

Dietz, Lee, Leigh, WillwacherR F I G Lamps

Ramesh Raskar, Paul Beardsley, Jeroen van Baar, Yao Wang, Paul Dietz, Johnny Lee, Darren Leigh, Thomas Willwacher

Mitsubishi Electric Research Labs (MERL), Cambridge, MA

R F I R F I GG LampsLamps : : Interacting with a SelfInteracting with a Self--describing World via describing World via Photosensing Wireless Tags and ProjectorsPhotosensing Wireless Tags and Projectors



Radio Frequency Identification Tags (RFID)Radio Frequency Identification Tags (RFID)

microchip

Antenna

No batteries,

Small size,

Cost few cents



WarehousingWarehousing RoutingRouting

Library Library

Baggage handling Baggage handling

CurrencyCurrency

Livestock trackingLivestock tracking



Micro Controller

Memory Computer

READER

Micro Controller

Memory

Conventional Passive RFID

19



Tagged Books in a LibraryTagged Books in a Library

IdEasy to get list of books in RF range

No Precise Location DataDifficult to find if the books in sorted

order ?Which book is upside down ?



Where are boxes with Where are boxes with Products close to Expiry Date ?Products close to Expiry Date ?



READER

Micro Controller

RF Data

Memory

Conventional RFID

Computer

READER

Micro Controller

RF Data

Light

Memory

Photosensor

Computer

Conventional RF tag

Photo-sensing RF tag



READER

Projector Micro

ControllerRF Data

Light

Memory

Photosensor

Computer

Photosensor ?

Compatible with RFID size and power needs

Projector ?

Directional transfer,AR with Image overlay



c. Tags respond via RF, with date and precise (x,y) pixel location. Projector beams ‘O’ or ‘X’ at that location for visual feedback

c. Tags respond via RF, with date and precise (x,y) pixel location. Projector beams ‘O’ or ‘X’ at that location for visual feedback

a. Photosensing RFID tagsare queried via RFa. Photosensing RFID tagsare queried via RF

d. Multiple users can simultaneously work from a distance without RF collision

d. Multiple users can simultaneously work from a distance without RF collision

b. Projector beams a time-varying pattern unique for each (x,y) pixel which is decoded by tags

b. Projector beams a time-varying pattern unique for each (x,y) pixel which is decoded by tags



20



RFID(Radio Frequency Identification)

RFIG

(Radio Frequency Id and Geometry)



Prototype TagPrototype Tag

RF tag + photosensor

RF tag + photosensor



PatternMSB

PatternMSB

PatternMSB-1PatternMSB-1

PatternLSB

PatternLSB

Projected Sequential Frames

•Handheld Projector beams binary coded stripes

•Tags decode temporal code



PatternMSB

PatternMSB


PatternLSB

PatternLSB






PatternMSB

PatternMSB


PatternLSB

PatternLSB






PatternMSB

PatternMSB


PatternLSB

PatternLSB




21



PatternMSB

PatternMSB


PatternLSB

PatternLSB






PatternMSB

PatternMSB


PatternLSB

PatternLSB

For each tag

a. From light sequence, decode x and y coordinate

b. Transmit back to RF reader (Id, x, y)

For each tag

a. From light sequence, decode x and y coordinate

b. Transmit back to RF reader (Id, x, y)

00 11 11 00 00 X=12X=12



Visual feedback of 2D positionVisual feedback of 2D position

a. Receive via RF {(x1,y1), (x2,y2), …} pixels

b. Illuminate those positions

a. Receive via RF {(x1,y1), (x2,y2), …} pixels

b. Illuminate those positions








•• Natural lighting conditionNatural lighting condition–– Day/Night FusionDay/Night Fusion

Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005

A Night Time Scene: Objects are Difficult to Understand due to Lack of Context

Dark Bldgs

Reflections on bldgs

Unknown shapes


Enhanced Context :All features from night scene are preserved, but background in clear

‘Well-lit’ Bldgs

Reflections in bldgs windows

Tree, Street shapes

22


Background is captured from day-time scene using the same fixed camera

Night Image

Day Image

Result: Enhanced Image


Mask is automatically computed from scene contrast


But, Simple Pixel Blending Creates Ugly Artifacts


Pixel Blending

Our Method:Integration of

blended Gradients


Nighttime imageNighttime image

Daytime imageDaytime image

Gradient fieldGradient field

Importance Importance image Wimage W

Fina

l res

ult

Fina

l res

ult


Mixed gradient fieldMixed gradient field

GG11 GG11

GG22 GG22

xx YY

xx YY

II11

I2

GG GGxx YY


Reconstruction from Gradient Field

• Problem: minimize error |∇ I’ – G|• Estimate I’ so that

G = ∇ I’

• Poisson equation

∇ 2 I’ = div G

• Full multigridsolver

II’’

GGXX

GGYY

23


Video Enhancement using Fusion

– Video from fixed cameras• Improve low quality InfraRed video using high-quality visible video• Fill in dark areas, enhance change in intensity• Output style: better context

– Current Demo• Fusion of Night video and Daytime image

Original Video FrameEasy-to-understand

Non-photorealistic (NPR)Image or Video


Details– Combine day and night time images

• Night videos have low contrast, areas with no detail• Same camera during day can capture static information• Dark areas of night video are replaced to provide context• Moving object (from night) + Static scene (from day)

Day time Photograph

Combine pixels depending on context, image and temporal gradient

Night time Video

(or Photo)

Enhanced Night Video (or Photo)

with context

Modified Surveillance CameraModified Surveillance Camera


Video Enhancement


Enhanced videoNote: exit ramp, lane dividers,

buildings not visible in original night video, but clearly seen here.

Overview of Process

Day time image: By averaging 5 seconds of day video

Original night time traffic camera 320x240 video

Mask frame (for frame above): Encodes pixel with intensity change

Input

Output


AlgorithmFrame Frame NN

Daytime imageDaytime image

TimeAveragedTimeAveragedimportance maskimportance mask


Final resultFinal result

Processed binary maskProcessed binary mask

Gra

dien

t fie

ldG

radi

ent f

ield

Frame Frame NN--11

Mixed gradient fieldMixed gradient field

‘Smarter’ Lighting Equipment

Programmable ParametersProgrammable Parameters

24


Novel Cameras


Display

GeneralizedSensor

GeneralizedOptics

Processing


Ray Sampler

Ray Reconstruction

GeneralizedOptics


Light Sources

Modulators

4D Light Field

Programmable 4D Illumination field + Time + Wavelength

Programmable 4D Illumination field + Time + Wavelength


Computational Illumination:Computational Illumination:Programmable 4D Illumination Field + Time + WavelengthProgrammable 4D Illumination Field + Time + Wavelength

•• Presence or Absence, Duration, BrightnessPresence or Absence, Duration, Brightness–– Flash/NoFlash/No--flashflash

•• Light positionLight position–– MultiMulti--flash for depth edgesflash for depth edges–– Programmable dome (image reProgrammable dome (image re--lighting and matting)lighting and matting)




•• Exploiting (uncontrolled) natural lighting conditionExploiting (uncontrolled) natural lighting condition–– Day/Night FusionDay/Night Fusion

Course WebPage : http:// www.merl.com/ people/ raskar/ photo/

Computational Illumination - MIT CSAILgroups.csail.mit.edu/graphics/classes/CompPhoto06/... · 7...

Documents

Transcript of Computational Illumination - MIT CSAILgroups.csail.mit.edu/graphics/classes/CompPhoto06/... · 7...