study Coded Aperture

39
Image and Deoth from a Conventional Camera with a Coded Apertrue Anat Levin, Rob Fergus, Frédo Durand, William Freeman MIT CSAIL study

description

study

Transcript of study Coded Aperture

Page 1: study Coded Aperture

Image and Deoth from a Conventional Camera

with a Coded Apertrue

Anat Levin, Rob Fergus, Frédo Durand, William

Freeman

MIT CSAIL

study

Page 2: study Coded Aperture

Single input image

real objects

output #1: Depth mapoutput #2: all-infocused image

Coded Aperture

Page 3: study Coded Aperture

Conventional aperture and depth of field

Small aperture

Big aperture

Focal plane

Object

Page 4: study Coded Aperture

LensCamera sensor

Point spread

function

http://groups.csail.mit.edu/graphics/CodedAperture/CodedAperture-LevinEtAl-SIGGRAPH07.pptFocal plane

Depth from defocus

Page 5: study Coded Aperture

LensObject

Camera sensor

Point spread

function

http://groups.csail.mit.edu/graphics/CodedAperture/CodedAperture-LevinEtAl-SIGGRAPH07.pptFocal plane

Depth from defocus

Page 6: study Coded Aperture

Focal plane

LensCamera sensor

Point spread

function

Object

http://groups.csail.mit.edu/graphics/CodedAperture/CodedAperture-LevinEtAl-SIGGRAPH07.ppt

Depth from defocus

Page 7: study Coded Aperture

Defocus as local convolution

Input defocused image

xfy k Local

observed sub-

window

Calibrated blur

kernels at depth K

Sharp sub-

window

Depth k=1

Depth k=2

Depth k=3

Page 8: study Coded Aperture

Introduction Estimation of depth – a branch of Computational Photography Most challenges of y = fk * x

Input Ringing with the traditional Richardson-Lucy algorithm

• Hard to de-convolve even when kernel is known

• Hard to identify correct scale:

?? Correct

scale

Smaller scale

? Larger scale

Page 9: study Coded Aperture

Related work – depth estimation Active methods – additional illumination sources• Structured light methods

Nayar et al. ICCV 95Zhang and Nayar, SIGGRAPH 06

Projection Defocus Analysis for Capture and Image Display, Zhang and Nayar, 06

Page 10: study Coded Aperture

Related work – depth estimation

Passive methods – changes of focus • Depth from defocus (DFD)

Pentland, IEEE 87Chaudhuri, Favaro et al. , 99

• Blind Deconvolution – image prior, maximum likelyhood

Kundur and Hatzinakos , IEEE 96Levin, NIPS 06

• Coded apertures for light gatheringFenimore and Cannon, Optics 78

Page 11: study Coded Aperture

Related work – depth estimation

• Plenoptic /light field cameraAdelson and Wang, IEEE 92Ng et al., 05

• Wavefront codingCathey & Dowski, Optics 94, 951.Rays don't converge anymore2.Image blur is the same for all

depth3.Blur spectrum does not have

too many zeros

Passive methods – changes of viewpoints

CompPhoto06/html/lecturenotes/25_LightField_6.pdf

Page 12: study Coded Aperture

OverviewTry deconvolving local input windows with different

scaled filters:

?? Correct

scale

Smaller scale

? Larger scale

Somehow: select best scale

yxfk

Page 13: study Coded Aperture

Challenges & contributions

Hard to de-convolve even when kernel is knownIDEA 1: Natural images prior

Hard to identify correct scaleIDEA 2: Coded Aperture

Page 14: study Coded Aperture

? =

=?

Solution 1:

Solution 2:

Deconvolution is ill posed

Page 15: study Coded Aperture

IDEA 1: Natural images prior

Image

gradient

put a penalty on gradients

Natural images have sparse gradients

Natural

Unnatural

What makes images special?

Page 16: study Coded Aperture

Deconvolution with prior

i ik xyxf )(min arg x

2 Convolution

errorDerivatives

prior

High

Low ?

?

2

2Equal convolution error

Page 17: study Coded Aperture

Comparing deconvolution algorithms

Input

Richardson-Lucy

Gaussian prior

“spread” gradients

Sparse prior

“localizes” gradients

8.0)( xx 2

)( xx

Page 18: study Coded Aperture

Statistical Model of Images

“Deconvolution using natural image priors”, Levin et. al., ETAI 07

Spatial domain

Frequency domain

Page 19: study Coded Aperture

Maximum a-posteriori P(x|y)

Image prior (gradient here)

Gradient operator

For Gaussian priors

For sparse priors

likelyhood

Page 20: study Coded Aperture

Minimize deconvolution error

Page 21: study Coded Aperture

Deconvolution using a Gaussian prior

Note: solved in the frequency domain in a few seconds for MB size file

Page 22: study Coded Aperture

Using an iterative reweighted least squares process (IRLS) [Meer 2004; Levin and Weiss to appear]

Cannot solve in frequency domain

Deconvolution using a sparse prior

Note: solved in the frequency domain around 1 hour on 2.4Ghz CPR for 2MB file

Page 23: study Coded Aperture

Iterative reweighted least squares process (IRLS)

Page 24: study Coded Aperture

Recall: OverviewTry deconvolving local input windows with different

scaled filters:

?? Correct

scale

Smaller scale

? Larger scale

Somehow: select best scale

yxfk

Challenge: smaller scale not so different than correct

Page 25: study Coded Aperture

IDEA 2: Coded Aperture

Mask (code) in aperture planeMake defocus patterns different from natural

images and easier to discriminate

Conventional aperture

Our coded aperture

Page 26: study Coded Aperture

Lens with coded aperture

Lens with coded

aperture

Camera sensor

Point spread

function

Image of a defocused point light

source

Aperture pattern

Object

Focal plane

Page 27: study Coded Aperture

Why coded ?

Convention

al Coded

Coded aperture- reduce uncertainty in scale identification

Correct

scale

Smaller scale

Larger scale

Page 28: study Coded Aperture

Why coded ?

Convention

al Coded

Coded aperture- reduce uncertainty in scale identification

Correct

scale

Smaller scale

Larger scale

Page 29: study Coded Aperture

Fourier transforms of 1D slide through the blur pattern

Page 30: study Coded Aperture

Coded aperture: Scale estimation and division by zero

?

?

=

=

0 Frequency

sp

ectr

um

0 Frequency

sp

ectr

um

0 Frequency

sp

ectr

um

0 Frequency

sp

ectr

um

0 Frequency

sp

ectr

um

Estimated image

spatial ringing

Division by zero

Estimated image

Filter, wrong scale

Filter, correct scale

Observed image

Page 31: study Coded Aperture

Division by zero with a conventional aperture ?

?

?

=

=

0 Frequency

sp

ectr

um

0 Frequency

sp

ectr

um

0 Frequency

sp

ectr

um

0 Frequency

sp

ectr

um

0 Frequency

sp

ectr

um

Estimated image

no spatial ringing

Tiny value at ω

Estimated image

Filter, wrong scale

Filter, correct scale

Observed image

No zero at ω !

No zero at ω !

ω is zero !

Page 32: study Coded Aperture

Filter Selection Criterion

The filter f has good depth discrimination - blurry image distributions Pk1(y) and Pk2(y) at depths k1 and k2 should not be similar

KL-divergence scores y

kkkkk dyyPyPyPyPyP 21121KL loglog,D

Page 33: study Coded Aperture

Filter Design Practical constrains

1. Binary filter to construct accurately2. Cut the filter from a single piece3. Avoid excessive radial distortion4. Avoid using the full aperture5. Diffraction impose a min size on the holes

in the file Spec.

13x13 patterns with 1 mm holesEach pattern, 8 different scales Varying between 5~15 pixels in width

Page 34: study Coded Aperture

Filter Design

Conventional

Conventional

Page 35: study Coded Aperture

Blur scale identification

Not robust at high-frequency noise

Un-normalized energy term

λk learn to minimize the scale misclassification error on a set of traning images

Ek is approximate by the reconstruction error by ML solution

x* is the deblurred image

Page 36: study Coded Aperture

Regularizing depth estimation

Page 37: study Coded Aperture

Results

Page 38: study Coded Aperture

Applications Digital refocusing from a single image

e.g. Synthesis an all-focus imagee.g. Post-exposure

Page 39: study Coded Aperture

Conclusion Pros.

All-infocus image and depth at a single shotNo loss of image resolution (compared with Plenoptic camera) Simple modificationCoded apertureConventional aperture

Cons.50 % light is blockedDepth is coarseMay need manual

correction