# study Coded Aperture

date post

17-May-2015Category

## Technology

view

1.455download

0

Embed Size (px)

description

### Transcript of study Coded Aperture

- 1. study

Image and Deoth from a Conventional Camera with a Coded Apertrue

Anat Levin, Rob Fergus,Frdo Durand, William Freeman

MIT CSAIL

2. Single input image

real objects

Coded Aperture

output #1: Depth map

output #2: all-infocusedimage

3. Conventional aperture and depth of field

Big aperture

Object

Focal plane

Small aperture

4. Depth from defocus

Camera sensor

Lens

Point spread function

Focal plane

http://groups.csail.mit.edu/graphics/CodedAperture/CodedAperture-LevinEtAl-SIGGRAPH07.ppt

5. Depth from defocus

Camera sensor

Lens

Object

Point spread function

Focal plane

http://groups.csail.mit.edu/graphics/CodedAperture/CodedAperture-LevinEtAl-SIGGRAPH07.ppt

6. Depth from defocus

Camera sensor

Lens

Object

Point spread function

Focal plane

http://groups.csail.mit.edu/graphics/CodedAperture/CodedAperture-LevinEtAl-SIGGRAPH07.ppt

7. Defocus as local convolution

Calibratedblur kernels at depth K

Local observed

sub-window

Sharp

sub-window

Input defocused image

Depth k=1

Depth k=2

Depth k=3

8. Introduction

Estimation of depth a branch of Computational Photography

Most challenges ofy = fk * x

- Hard to de-convolve evenwhen kernel is known

Input

Ringing with the traditional Richardson-Lucyalgorithm

- Hard to identify correct scale:

?

Larger scale

?

Correct scale

?

Smaller scale

9. Related work depth estimation

Active methods additional illumination sources

- Structured light methods

Nayar et al. ICCV 95

Zhang and Nayar, SIGGRAPH 06

Projection Defocus Analysis for Capture and Image Display, Zhang and Nayar, 06

10. Related work depth estimation

Passive methods changes of focus

- Depth from defocus (DFD)

Pentland, IEEE 87

Chaudhuri, Favaro et al. , 99

- Blind Deconvolution image prior, maximum likelyhood

Kundur and Hatzinakos , IEEE 96

Levin,NIPS 06

- Coded apertures for light gathering

Fenimore and Cannon, Optics 78

11. Related work depth estimation

- Passive methods changes of viewpoints

12. Plenoptic /light field cameraAdelson and Wang, IEEE 92

Ng et al., 05

- Wavefront coding

Cathey & Dowski, Optics 94, 95

1.Rays don't converge anymore

2.Image blur is the same for all depth

3.Blur spectrum does not have too many zeros

CompPhoto06/html/lecturenotes/25_LightField_6.pdf

13. Overview

Try deconvolving local input windows with different scaled filters:

?

Larger scale

?

Correct scale

?

Smaller scale

Somehow: select best scale

14. Challenges & contributions

Hard to de-convolve even when kernel is known

IDEA 1: Natural images prior

Hard to identify correct scale

IDEA 2: Coded Aperture

15. Deconvolution is ill posed

Solution 1:

=

?

Solution 2:

=

?

16. IDEA 1: Natural images prior

What makes images special?

Natural

Unnatural

Image

gradient

Natural images have sparse gradients

put a penalty on gradients

17. Deconvolution with prior

Convolution error

Derivatives prior

2

?

Low

Equal convolution error

2

?

High

18. Comparing deconvolution algorithms

Richardson-Lucy

Input

spread gradients

localizes gradients

Gaussian prior

Sparse prior

19. Statistical Model of Images

Deconvolution using natural image priors, Levin et. al., ETAI 07

Spatial domain

Frequency domain

20. Maximum a-posteriori P(x|y)

likelyhood

Image prior

(gradient here)

Gradient operator

For Gaussian priors

For sparse priors

21. Minimize deconvolution error

22. Deconvolution using a Gaussian prior

Note: solved in the frequency domain in a few seconds for MB size file

23. Deconvolution using a sparse prior

Using an iterative reweighted least squares process (IRLS) [Meer 2004; Levin and Weiss to appear]

Cannot solve in frequency domain

Note: solved in the frequency domainaround1 hour on 2.4Ghz CPR for 2MB file

24. Iterative reweighted least squares process (IRLS)

25. Recall: Overview

Try deconvolving local input windows with different scaled filters:

?

Larger scale

?

Correct scale

?

Smaller scale

Somehow: select best scale

Challenge: smaller scale not so different than correct

26. IDEA 2: Coded Aperture

Mask (code) in aperture plane

Make defocus patterns different from natural images and easier to discriminate

Conventional aperture

Our coded aperture

27. Lens with coded aperture

Image of a defocused point light source

Aperture pattern

Camera sensor

Lens with coded aperture

Object

Point spread function

Focal plane

28. Why coded ?

Coded aperture- reduce uncertainty in scale identification

Conventional

Coded

Larger scale

Correct scale

Smaller scale

29. Why coded ?

Coded aperture- reduce uncertainty in scale identification

Conventional

Coded

Larger scale

Correct scale

Smaller scale

30. Fourier transforms of 1D slide through the blur pattern

31. Coded aperture: Scale estimation and division by zero

spectrum

spectrum

spectrum

spectrum

spectrum

Frequency

Frequency

Frequency

Frequency

Frequency

Estimated image

?

Observed image

=

Filter, correct scale

Division by zero

Estimated image

?

spatial ringing

=

Filter, wrong scale

32. Division by zero with a conventional aperture ?

spectrum

spectrum

spectrum

spectrum

spectrum

Frequency

Frequency

Frequency

Frequency

Frequency

Estimated image

?

No zero at !

Observed image

=

Filter, correct scale

No zero at !

Tiny value at

no spatial ringing

Estimated image

?

=

Filter, wrong scale

is zero !

33. Filter Selection Criterion

The filter f has good depth discrimination - blurry image distributions Pk1(y) and Pk2(y) at depths k1 and k2 shouldnot be similar

KL-divergence scores

34. Filter Design

Practical constrains

Binary filter to construct accurately

Cut the filter from a single piece

Avoid excessive radial distortion

Avoid using the full aperture

Diffraction impose a min size on the holes in the file

Spec.

13x13 patterns with 1 mm holes

Each pattern, 8 differentscales

Varying between 5~15 pixels in width

35. Filter Design

Conventional

Conventional

36. Blur scale identification

Not robust at high-frequency noise

Un-normalized energy term

klearn to minimizethe scale misclassification error on a set of traning images

Ek is approximate by the reconstruction error by ML solution

x* is the deblurred image

37. Regularizing depth estimation

38. Results

39. Applications

Digital refocusing from a single image

e.g.Synthesis an all-focus image

e.g.Post-exposure

40. Conclusion

Pros.

All-infocus image and depth at a single shot

No loss of image resolution (compared with Plenoptic camera)

Simple modification

Coded aperture

Conventional aperture

Cons.

50 % light is blocked

Depth is coarse

May need manual correction