Image Destabilization: Programmable Defocus using Lens and Sensor Motion
description
Transcript of Image Destabilization: Programmable Defocus using Lens and Sensor Motion
MIT Media Lab Camera Culture
Image Destabilization:Programmable Defocus using
Lens and Sensor Motion
Ankit Mohan, Douglas Lanman,Shinsaku Hiura, Ramesh Raskar
MIT Media Lab
MIT Media Lab Camera Culture
MIT Media Lab Camera Culture
Lots of glass; Heavy; Bulky; Expensive
Defocus Blur
MIT Media Lab Camera Culture
Image Destabilization
Lens Sensor
Camera
Static Scene
MIT Media Lab Camera Culture
Image Destabilization
Static Scene
Lens Motion Sensor Motion
Camera
MIT Media Lab Camera Culture
Related Work
[Bae and Durand 2007]
[Hasinoff and Kutulakos 2007]
f8
f/4
f/2
f/1
extrapolated aperture
[Hiura et al. 2009][Vaish et al. 2004]
MIT Media Lab Camera Culture
LaminographyX-Ray Source
X-Ray Sensor
Plane of focus
Related technique: Time Delay and Integration (TDI)
Motion direction
Motion direction
MIT Media Lab Camera Culture
A
B A’
B’
Lens based Focusing
Lens Sensor
MIT Media Lab Camera Culture
A
B A’
B’
Lens based Focusing
Lens Sensor
MIT Media Lab Camera Culture
A
B
B’
A’
Lens
Smaller aperture Smaller defocus blur
Sensor
MIT Media Lab Camera Culture
A
B A’
B’
Pinhole
Pinhole: All In-Focus
Sensor
MIT Media Lab Camera Culture
A
B
A’
B’
Pinhole
Shifting Pinhole
Sensor
vp
MIT Media Lab Camera Culture
A
B
Pinhole
A’
B’
Shifting Pinhole
Sensor
vp
MIT Media Lab Camera Culture
A
B
Pinhole
A’
B’
Shifting Pinhole
Sensor
vp
MIT Media Lab Camera Culture
A
B
Pinhole
A’
B’
Shifting Pinhole
Sensor
vp
MIT Media Lab Camera Culture
B
Pinhole Sensor
A’
B’
Shifting Pinhole
A
da
db ds
tp
vp
MIT Media Lab Camera Culture
A
B
A’
B’
Pinhole
Shifting Pinhole and Sensor
vp
Sensor
vs
da
db dsFocus Here
MIT Media Lab Camera Culture
A
B
Pinhole
A’
B’vp
Sensor
vs
da
db dsFocus Here
Shifting Pinhole and Sensor
MIT Media Lab Camera Culture
A
B
Pinhole
A’
B’
vp
Sensor
vs
da
db dsFocus Here
Shifting Pinhole and Sensor
MIT Media Lab Camera Culture
A
B
Pinhole
A’
B’
vp
Sensor
vs
da
db dsFocus Here
Shifting Pinhole and Sensor
MIT Media Lab Camera Culture
A Lens in Time!
Analogous to shift and sum basedLight field re-focusing.
Lens Equation:
Virtual Focal Length:
Virtual F-Number:
MIT Media Lab Camera Culture
Our Prototype
MIT Media Lab Camera Culture
Adjusting the Focus Plane
all-in-focus pinhole image
MIT Media Lab Camera Culture
Adjusting the Focus Plane
focused in the front using destabilization (10 second exposure)
MIT Media Lab Camera Culture
Adjusting the Focus Plane
focused in the middle using destabilization (5 second exposure)
MIT Media Lab Camera Culture
Adjusting the Focus Plane
focused in the back using destabilization (10 second exposure)
MIT Media Lab Camera Culture
Adjusting the Virtual Aperture
focused in the middle using destabilization (5mm pinhole translation)
MIT Media Lab Camera Culture
Adjusting the Virtual Aperture
focused in the middle using destabilization (30mm pinhole translation)
MIT Media Lab Camera Culture
Shifting Lens and Sensor Defocus• Physical vs. synthetic focus• Similar physical and synthetic focus
aperture
sensor
real focus
virtual focus
Defocus Exaggeration
MIT Media Lab Camera Culture
Defocus Exaggeration
static lens with an f/2.8 aperture
MIT Media Lab Camera Culture
Defocus Exaggeration
destabilization simulates a reduced f-number
MIT Media Lab Camera Culture
Defocus Invariance
aperture
sensor
real focus
virtual focus
• Differing physical and synthetic focus
MIT Media Lab Camera Culture
* =real PSF virtual PSF overall PSF
Defocus Invariance• Related work
– [Nagahara et al. 2008]– [Cathey and Dowski 1995]– [Levin et al. 2008]
• PSF not depth invariant– only size is depth invariant
• Gaussian special case– depth invariant PSF – inversion is ill-conditioned* =
real PSF virtual PSF overall PSF
[Nagahara et al. 2008]
MIT Media Lab Camera Culture
Defocus Invariance
depth-invariant blur size (horizontal slit + destabilization)
MIT Media Lab Camera Culture
Defocus Invariance
Richardson-Lucy deconvolution result
MIT Media Lab Camera Culture
Tilted Sensor
C
DC′
D′
focus plane aperture plane sensor plane
Scheimpflug intersection
dC
dD
d′Cd′D
CC ddf 111
MIT Media Lab Camera Culture
Tilted Sensor
C
D
vs?
focus plane aperture plane sensor plane
vp
CC ddf 111
dC
dD
MIT Media Lab Camera Culture
Tilted Sensor
C
D
vp
C′ D′
vs
focus plane aperture plane sensor plane
dC
dD
d′Cd′D
s
p
DD
D
CC
C
vv
ddd
ddd
MIT Media Lab Camera Culture
Tuning the PSF
aperture
sensor
real focus
pinhole image (static f/22 aperture)
MIT Media Lab Camera Culture
Tuning the PSF
real focus
aperture
sensor
large aperture image (static f/2.8 aperture)
MIT Media Lab Camera Culture
Tuning the PSF
virtual focus
aperture
sensor
destabilized image using a pinhole (translated f/22 aperture)
MIT Media Lab Camera Culture
Tuning the PSF
aperture
sensor
real focusvirtual focus
destabilized image using a large aperture (translated f/2.8 aperture)
MIT Media Lab Camera Culture
Tuning the PSFreal focus
virtual focus
aperture
sensor
simulated aspheric lens using a vertical slit aperture and destabilization
MIT Media Lab Camera Culture
Extension to 2D Displacements
ellipticallinear circular
“figure 8” hypocycloidal trispiral
MIT Media Lab Camera Culture
Large apertures with tiny lenses?Benefits• No time or light inefficiency
wrt cheap cameras• Exploits unused area around
the lens• Compact design• With near-pinhole apertures
(mobile phones) many possibilities
Limitations• Coordinated mechanical
movement required• Diffraction (due to small aperture)
cannot be eliminated[Zhang and Levoy, tomorrow]
[Our group: augmented LF for wave analysis]
• Scene motion during exposure
MIT Media Lab Camera Culture
Acknowledgements
Grace WooMIT CSAIL
Quinn SmithwickMIT Media Lab
Gabriel TaubinBrown University
Jaewon KimMIT Media Lab
MIT Media Lab: Camera Culture
MIT Media Lab Camera Culture
SLRs with tiny lenses?
• Analysis of space of relative lens/sensor displacement• Destabilization as virtual focusing mechanism• Shallower depth of field than physical aperture• Depth-independent defocus blur size