Computational Light Transport Computational Photography...
Transcript of Computational Light Transport Computational Photography...
![Page 1: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/1.jpg)
Raskar, Camera Culture, MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Light Transport Computational Photography
Inverse problems
MIT Media Lab Ramesh Raskar http://raskar.info
![Page 2: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/2.jpg)
MIT Media Lab [email protected] http://cameraculture.info
• Co-design of Optics and Computation – Photons not just pixels
– Mid-level cues
• Computational Photography
– Coded Exposure
– Compressive Sensing for High Speed Events
• Limits of CS for general imaging
• Computational Light Transport
– Looking Around Corners, trillion fps
– Lightfields: • Compressive Campture
• 3D Displays and Holograms
Inverse Problems
![Page 3: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/3.jpg)
Fernald, Science [Sept 2006]
Shadow Refractive Reflective
Tools
for
Visual
Computing
![Page 4: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/4.jpg)
Traditional Photography
Lens
Detector
Pixels
Image
Mimics Human Eye for a Single Snapshot: Single View, Single Instant, Fixed Dynamic range and Depth of field for given Illumination in a Static world
Courtesy: Shree Nayar
![Page 5: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/5.jpg)
Computational Camera + Photography: Optics, Sensors and Computations
Generalized Sensor
Generalized Optics
Computations
Picture
4D Ray Bender
Upto 4D Ray Sampler
Ray Reconstruction
Merged Views, Programmable focus and dynamic range, Closed-loop Controlled Illumination, Coded exposure/apertures
![Page 6: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/6.jpg)
Computational Photography Novel Illumination
Computational Cameras
Scene: 8D Ray Modulator
Display
Generalized
Sensor
Generalized
Optics Processing
4D Ray Bender Upto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
![Page 7: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/7.jpg)
Camera Culture Creating new ways to capture and share visual information
MIT Media Lab Ramesh Raskar
http://cameraculture.info
Facebook.com/cameraculture
1.Light-Field Camera A new camera design
exploiting the fundamental
dictionary of light-fields for a
single-capture capture of
light-fields with full-
resolution refocusing
effects.
2. Color Primaries A new camera design with
switchable color filter arrays
for optimal color fidelity and
picture quality on scene
geometry, color and
illumination.
3. Flutter-Shutter A camera that codes the
exposure time with a binary
pseudo-sequence to de-
convolve and remove
motion blur in textured
backgrounds and partial
occluders.
4. Compressive
Capture We analyze the gamut of
visual signals from low-
dimensional images to light-
fields and propose non-
adaptive projections for
efficient sparsity exploiting
reconstruction.
Computational Photography
1. Looking around corners Using short laser pulses and fast detector, we
aim to build a device that can look around
corners with no imaging device in the line of
sight using time resolved transient imaging.
2. Reflectance Recovery We demonstrate a new technique that allows
a camera to rapidly acquire reflectance
properties of objects 'in the wild' from a single
viewpoint, over relatively long distances and
without encircling equipment.
3. Trillion Frames per Second Imaging A camera fast enough to capture light pulses
moving through objects. We can use such a
camera to understand reflectance, absorption
and scattering properties of materials.
Femtosecond Imaging 3D Displays
1. Tensor Display A family of compressive light
field displays comprising all
architectures employing a stack
of time-multiplexed, light-
attenuating layers illuminated by
uniform or directional
backlighting
2. Layered 3D Tomographic techniques for
image synthesis on displays
composed of compact volumes
of light-attenuating material.
Such volumetric attenuators
recreate a 4D light field or high-
contrast 2D image when
illuminated by a uniform
backlight.
3. Glasses-free 3D HDTV Light field displays with
increased brightness and
refresh rate by stacking a pair of
modified LCD panels, exploiting
rank and constraint of 3D
displays
4. BIDI Screen A thin, depth-sensing LCD for
3D interaction using light fields
which supports both 2D multi-
touch and unencumbered 3D
gestures.
5. Living Windows 6D
Display A completely passive display
that responds to changes in
viewpoint and changes in
incident light conditions.
May 2012
![Page 8: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/8.jpg)
1. Augmented Light Fields Expands light field representations
to describe phase and diffraction
effects by using the Wigner
Distribution Function
2. Hologram v Parallax Barrier Defines connections between
parallax barrier displays and
holographic displays by analyzing
their operations and limitations in
phase space
3. Ray–Based Diffraction Model Simplified capture of diffraction
model for computer graphics
applications.
Post-Doctorial Researchers: Doug Lanman, Gordon Wetzstein, Alex Olwal, Christopher Barsi
Research Assistants: Matthew Hirsch, Otkrist Gupta, Nikhil Naik, Jason Boggess, Everett Lawson, Aydın Arpa, Kshitij Marwah
Visiting Researchers & Students: Di Wu, Daryl Lim
1. Retinal Imaging With simplified optics and
cleaver illumination we visualize
images of the retina in a
standalone device easily
operated by the end user.
2. NETRA/CATRA Low-cost cell-phone
attachments that measures eye-
glass prescription and cataract
information from the eye.
3. Cellphone Microscopy A platform for computational
microscopy and remote
healthcare
4. High-speed Tomography A compact, fast CAT scan
machine using no mechanical
moving parts or synchronization.
5. Shield Fields 3D reconstruction of objects
from a single shot photo using
spatial heterodyning.
6. Second Skin Using 3D motion tracking with
real-time vibrotactile feedback
aids the correct of movement
and position errors to improve
motor learning.
Health & Wellness
1. Bokode Low-cost, passive optical
design so that bar codes can
be shrunk to fewer than 3mm
and read by ordinary
cameras several meters
away.
2. Specklesense Set of motion-sensing
configurations based on laser
speckle sensing . The
underlying principles allow
interactions to be fast,
precise, extremely compact,
and low cost.
3. Sound Around Soundaround is a multi-
viewer interactive audio
system, designed to be
integrated into multi-view
displays presenting localized
audio/video channels with no
need for glasses or
headphones.
Human Computer Interaction Visual Social Computing
1. Photocloud A near real-time system for
interactively exploring a
collectively captured
moment without explicit 3D
reconstruction.
2. Vision Blocks On-demand, in-browser,
customizable, computer-
vision application-building
platform for the masses.
Without any prior
programming experience,
users can create and share
computer vision
applications.
3. Lenschat LensChat allows users to
share mutual photos with
friends or borrow the
perspective and abilities of
many cameras.
Light Propagation Theory and Fourier Optics
Visit us online at
Cameraculture.info fb.com/cameraculture
![Page 9: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/9.jpg)
Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses [Ng et al 2005]
![Page 10: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/10.jpg)
Light Field Inside a Camera
![Page 11: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/11.jpg)
Lenslet-based Light Field camera
[Adelson and Wang, 1992, Ng et al. 2005 ]
Light Field Inside a Camera
![Page 12: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/12.jpg)
Mask based Light Field Camera
Mask Sensor
[Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ]
![Page 13: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/13.jpg)
Spatial Heterodyning with Masks
Main Lens Object Mask Sensor
•Mask == Light Field Modulator
•Intensity of ray gets multiplied by Mask
•Convolution in Frequency domain
•Lightfield without additional lenslets
Wavefront Analysis for ANY wavelength
Oh, Raskar, Barbastathis 2009: Augmented Light Field
![Page 14: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/14.jpg)
Captured 2D Photo
Encoding due to
Mask
![Page 15: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/15.jpg)
Mask-based 4D Light Field
2D Sensor Photo, 1800*1800 2D Fourier Transform, 1800*1800
2D
FFT
Rearrange 2D tiles into 4D planes
200*200*9*9 4D IFFT
4D Light Field Depth
9*9=81 spectral copies
200*200*9*9
![Page 16: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/16.jpg)
Radio Frequency Heterodyning
Baseband Audio Signal
Receiver: Demodulation High Freq Carrier
100 MHz
Reference
Carrier
Incoming
Signal
99 MHz
![Page 17: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/17.jpg)
Optical Heterodyning
Photographic
Signal
(Light Field)
Carrier Incident
Modulated
Signal
Reference
Carrier
Main Lens Object Mask Sensor
Recovered
Light
Field
Software Demodulation
Baseband Audio Signal
Receiver: Demodulation High Freq Carrier
100 MHz
Reference
Carrier
Incoming
Signal
99 MHz
![Page 18: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/18.jpg)
1/f0
Mask Tile
Cosine Mask Used
![Page 19: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/19.jpg)
Captured 2D Photo
Encoding due to
Mask
![Page 20: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/20.jpg)
2D FFT
Traditional Camera Photo
Heterodyne Camera Photo
Magnitude of 2D FFT
2D FFT
Magnitude of 2D FFT
![Page 21: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/21.jpg)
Computing 4D Light Field
2D Sensor Photo, 1800*1800 2D Fourier Transform, 1800*1800
2D
FFT
Rearrange 2D tiles into 4D planes
200*200*9*9 4D IFFT
4D Light Field
9*9=81 spectral copies
200*200*9*9
![Page 22: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/22.jpg)
fθ
fx
fθ0
fx0
Band-limited
Light Field
Sensor Slice – Fourier
Slice Theorem
Photo = Slice of Light Field in Fourier Domain
[Ren Ng, SIGGRAPH 2005]
![Page 23: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/23.jpg)
How to Capture 2D Light Field with 1D Sensor ?
fθ
fx
fθ0
fx0
Band-limited
Light Field
Sensor Slice
Fourier Light Field Space
![Page 24: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/24.jpg)
Extra sensor bandwidth cannot capture extra dimension of the light field
fθ
fx
fθ0
fx0
Sensor
Slice
Extra sensor
bandwidth
![Page 25: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/25.jpg)
fθ
fx
??? ???
??? ???
![Page 26: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/26.jpg)
Solution: Modulation Theorem
Make spectral copies of 2D light field
fθ
fx
fθ0
fx0
Modulation
Function
![Page 27: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/27.jpg)
fθ
Modulated Light Field
fx
fθ0
fx0
Modulation
Function
Sensor Slice captures entire Light Field
![Page 28: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/28.jpg)
1/f0
Mask Tile
Cosine Mask Used
![Page 29: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/29.jpg)
Demodulation to recover Light Field
fθ
fx
Reshape 1D Fourier Transform into 2D
1D Fourier Transform of Sensor Signal
![Page 30: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/30.jpg)
fθ
fx
fθ0
fx0
Modulation Function == Sum of Impulses
Physical Mask = Sum of Cosines
![Page 31: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/31.jpg)
Where to place the Mask?
Mask
Sensor
Mask
Sensor
Mask Modulation
Function Mask Modulation
Function
fx
fθ
![Page 32: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/32.jpg)
Mask Sensor
Where to place the Mask?
Mask
Modulation
Function fx
fθ
![Page 33: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/33.jpg)
Captured 2D Photo
Encoding due to
Cosine Mask
![Page 34: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/34.jpg)
Computing 4D Light Field
2D Sensor Photo, 1800*1800 2D Fourier Transform
2D
FFT
Rearrange 2D tiles into 4D planes
200*200*9*9 4D IFFT
4D Light Field
9*9=81 spectral copies
200*200*9*9
![Page 35: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/35.jpg)
Sensing Depth from Light Sensing LCD By creating Array of Virtual Cameras in LCD
BiDi Screen: Thin LCD for touch+hover
Hirsch, Lanman, Holtzman, Raskar Siggraph Asia 2009
![Page 36: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/36.jpg)
BiDi Screen: Multi-touch + Hover 3D interface LCD = Large Lensless Camera
![Page 37: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/37.jpg)
Bit Hacking
Ph
oto
n H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational Photography Illumination
Co-designing Optical and Digital Processing
![Page 38: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/38.jpg)
Motion Blur in Low Light
![Page 39: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/39.jpg)
Traditional
Deblurred Image
Blurred Photo
![Page 40: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/40.jpg)
Fluttered Shutter Camera Raskar, Agrawal, Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turned opaque or transparent in a rapid binary sequence
![Page 41: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/41.jpg)
Flutter Shutter: Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred Photo PSF == Broadband Function
Fourier Transform
![Page 42: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/42.jpg)
Traditional Coded
Exposure
Image of Static Object
Deblurred Image
Deblurred Image
![Page 43: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/43.jpg)
Motion Blur in Low Light
![Page 44: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/44.jpg)
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 40.4 Hz
500 fps hi-speed camera
Bottling line
![Page 45: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/45.jpg)
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz.
Fourier transform is non-zero only at multiples of fP=1/P ~ 63Hz.
4fP
3fP 0 fMax - fMax fP=1/P 2fP -fP -2fP -4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
![Page 46: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/46.jpg)
High speed camera
Periodic signal has regularly spaced, sparse Fourier coefficients.
Is it necessary to use a high-speed video camera? Why waste bandwidth?
0 fMax - fMax fP=1/P 2fP -fP -2fP -4fP -3fP 4fP 3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1/(2 fMax)
![Page 47: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/47.jpg)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies.
Low exposure to avoid blurring. Low light throughput.
Period known apriori.
![Page 48: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/48.jpg)
Random Projections Per Frame of Camera using
Coded Strobing Photography
In every exposure duration observe different linear combinations of
the periodic signal.
Advantage of the design
• Exposure coding independent of the frequency
• On an average, light throughput is 50%
t P
Coded Strobing Photography. Reddy, D., Veeraraghavan, A., Raskar, R. IEEE PAMI 2011
![Page 49: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/49.jpg)
Observation Model
x at 2000fps y at 25fps
![Page 50: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/50.jpg)
Signal Model
x at 2000fps y at 25fps
![Page 51: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/51.jpg)
Signal & Observation Model
A is M x N, M<<N
x at 2000fps y at 25fps
N / M = 2000 / 25 = 80
![Page 52: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/52.jpg)
Recovery: Sparsity
Very few
non-zero
elements
Structured Sparse
Coefficients
y = A s Mixing matrix
Basis Pursuit De-noising
Asytss ..min1
Observed values
![Page 53: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/53.jpg)
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
![Page 54: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/54.jpg)
Rotating mill tool
Normal Video: 25fps Reconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video: 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
![Page 55: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/55.jpg)
Compressive Sensing for Images .. A good idea?
NMNMxy where,
Single Pixel Camera
compressive image measurement matrix
image
![Page 56: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/56.jpg)
Is Randomized Projection-based Capture apt for Natural Images ?
[Pandharkar, Veeraraghavan, Raskar 2009]
Randomized Projections
Pro
gres
sive
Pro
ject
ion
s
Compression Ratio
Periodic Signals
![Page 57: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/57.jpg)
MIT Media Lab [email protected] http://cameraculture.info
• Co-design of Optics and Computation – Photons not just pixels
– Mid-level cues
• Computational Photography
– Coded Exposure
– Compressive Sensing for High Speed Events
• Limits of CS for general imaging
• Computational Light Transport
– Looking Around Corners, trillion fps
– Lightfields: • Compressive Campture
• 3D Displays and Holograms
Inverse Problems
![Page 58: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/58.jpg)
What is around the corner ?
![Page 59: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/59.jpg)
Can you look around the corner ?
![Page 60: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/60.jpg)
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
![Page 61: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/61.jpg)
Light in Slow Motion ..
10 Billion x Slow
![Page 62: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/62.jpg)
Femto-Camera
Wall
Door
Hidden Mannequin
Velten et al, Nature Communications 2012
![Page 63: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/63.jpg)
![Page 64: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/64.jpg)
![Page 65: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/65.jpg)
Scene with hidden elements
Raw Time profiles
t
Signal Proc.
Novel light transport models and inference
algorithms
Photo, geometry, reflectance,
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto Photography Time Resolved Multi-path Imaging
Tasks: Non-line of sight, Motion estimation, Material sensing, Volumetric Inversion: Sparsity, Rank, Bounded Approx, Scene Priors, Transforms Signal Processing: Compressive, SNR, Bandwidth, Noise models Illumination: Coding space/time/wavelength, Semicondctr lasers, nonlinear optics FormFactors: Endoscopy, OCT, Bench-top, Room size, Outdoors Applications: ComputerVision, MedicalImaging, Biochemistry, Rescue/Planning, IndustrialInspection, Robotics, Spectroscopy Art/Education Collaborations Welcome/ Apply for internships
http://raskar.info
![Page 66: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/66.jpg)
![Page 67: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/67.jpg)
View Dependent Appearance and Iridescent color Cross section through a single M. rhetenor scale
![Page 68: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/68.jpg)
Two Layer Displays
barrier
sensor/display
lenslet
sensor/display
PB = dim displays
Lenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter, High spatial resolution
![Page 69: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/69.jpg)
Parallax barrier
LCD display
Limitations of 3D Display
Lanman, Hirsch, Kim, Raskar Siggraph Asia 2010
Front
Back
![Page 70: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/70.jpg)
][][],[ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k] k
f[i] i
L[i,k]
L[i,k]
![Page 71: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/71.jpg)
f[i]
g[k]
L[i,k]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L
~
![Page 72: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/72.jpg)
Implementation
Components
• 22 inch ViewSonic FuHzion VX2265wm LCD [1680×1050 @ 120 fps]
![Page 73: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/73.jpg)
f[i]
g[k]
L[i,k]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
![Page 74: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/74.jpg)
0,for ,2
1 min arg
2
GF,
GFFGL
F
G
` L~ =
Content-Adaptive Parallax Barriers
![Page 75: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/75.jpg)
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0,for ,2
1 min arg
2
GF,
GFFGLW
F
G
` L~
=
Content-Adaptive Parallax Barriers
Lanman, Hirsch, Kim, Raskar Siggraph Asia 2010
![Page 76: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/76.jpg)
rear mask: f1[i,j] front mask: g1[k,l]
reconstruction (central view)
Optimization: Iteration 1
))](([
)]([
]))([(
])[(
FGWF
LWFGG
GFGW
GLWFF
t
t
t
t
Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.
Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
![Page 77: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/77.jpg)
Optimization: Iteration 10
rear mask: f1[i,j] front mask: g1[k,l]
reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.
Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
))](([
)]([
]))([(
])[(
FGWF
LWFGG
GFGW
GLWFF
t
t
t
t
![Page 78: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/78.jpg)
Optimization: Iteration 20
rear mask: f1[i,j] front mask: g1[k,l]
reconstruction (central view)
))](([
)]([
]))([(
])[(
FGWF
LWFGG
GFGW
GLWFF
t
t
t
t
Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.
Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
![Page 79: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/79.jpg)
Optimization: Iteration 30
rear mask: f1[i,j] front mask: g1[k,l]
reconstruction (central view)
))](([
)]([
]))([(
])[(
FGWF
LWFGG
GFGW
GLWFF
t
t
t
t
Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.
Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
![Page 80: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/80.jpg)
Optimization: Iteration 40
rear mask: f1[i,j] front mask: g1[k,l]
reconstruction (central view)
))](([
)]([
]))([(
])[(
FGWF
LWFGG
GFGW
GLWFF
t
t
t
t
Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.
Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
![Page 81: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/81.jpg)
Optimization: Iteration 50
rear mask: f1[i,j] front mask: g1[k,l]
reconstruction (central view)
))](([
)]([
]))([(
])[(
FGWF
LWFGG
GFGW
GLWFF
t
t
t
t
Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.
Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
![Page 82: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/82.jpg)
Optimization: Iteration 60
rear mask: f1[i,j] front mask: g1[k,l]
reconstruction (central view)
))](([
)]([
]))([(
])[(
FGWF
LWFGG
GFGW
GLWFF
t
t
t
t
Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.
Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
![Page 83: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/83.jpg)
Optimization: Iteration 70
rear mask: f1[i,j] front mask: g1[k,l]
reconstruction (central view)
))](([
)]([
]))([(
])[(
FGWF
LWFGG
GFGW
GLWFF
t
t
t
t
Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.
Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
![Page 84: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/84.jpg)
Optimization: Iteration 80
rear mask: f1[i,j] front mask: g1[k,l]
reconstruction (central view)
))](([
)]([
]))([(
])[(
FGWF
LWFGG
GFGW
GLWFF
t
t
t
t
Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.
Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
![Page 85: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/85.jpg)
Optimization: Iteration 90
rear mask: f1[i,j] front mask: g1[k,l]
reconstruction (central view)
))](([
)]([
]))([(
])[(
FGWF
LWFGG
GFGW
GLWFF
t
t
t
t
Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.
Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
![Page 86: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/86.jpg)
Content-Adaptive Front Mask (1 of 9)
![Page 87: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/87.jpg)
Content-Adaptive Rear Mask (1 of 9)
![Page 88: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/88.jpg)
Emitted 4D Light Field
![Page 89: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/89.jpg)
Conclusion
• Described a rank constraint for all dual-layer displays ‒ With a fixed pair of masks, emitted light field is rank-1
• Achieved higher-rank approximation using temporal multiplexing
‒ With T time-multiplexed masks, emitted light field is rank-T
‒ Constructed a prototype using off-the-shelf panels
• Demonstrated light field display is a matrix approximation problem
• Introduced content-adaptive parallax barriers ‒ Applied weighted NMF to optimize weighted Euclidean distance to target
Adaptation increases brightness and refresh rate of dual-stacked LCDs
0,for ,2
1 min arg
2
GF,
GFFGLW
F
G
` L~
=
Content-Adaptive Parallax Barriers
![Page 90: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/90.jpg)
Parallax Barrier: Np=103 pix. Hologram: NH=105 pix.
θp=10 pix
w
θH =1000 pix
ϕP∝w/d ϕH∝λ/tH
Fourier Patch
Horstmeyer, Oh, Cuypers, Barbastathis, Raskar, 2009
![Page 91: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/91.jpg)
Augmented Light Field
9
5
Wigner
Distribution
Function
Traditional
Light Field
WDF
Traditional
Light Field
Augmented LF
Interference & Diffraction
Interaction w/ optical elements
ray optics based
simple and powerful
wave optics based
rigorous but cumbersome
Oh, Raskar, Barbastathis 2009: Augmented Light Field
![Page 92: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/92.jpg)
Light Fields
Reference
plane
position
Goal: Representing propagation, interaction and image formation of
light using purely position and angle parameters
LF propagation
(diffractive)
optical
element
LF LF LF LF
LF propagation
light field transformer
![Page 93: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/93.jpg)
Augmented Lightfield for Wave Optics Effects
Wigner
Distribution
Function
Light
Field
LF < WDF
Lacks phase properties
Ignores diffraction, interferrence
Radiance = Positive
LF
Augmented
Light Field
WDF
ALF ~ WDF
Supports coherent/incoherent
Radiance = Positive/Negative
Virtual light sources
![Page 94: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/94.jpg)
Free-space propagation
Light field transformer
Virtual light projector
Possibly negative radiance
9
8
![Page 95: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/95.jpg)
Lightfield vs Hologram Displays
![Page 96: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/96.jpg)
Is hologram just another ray-based light field? Can a hologram create any intensity distribution in 3D? Why hologram creates a ‘wavefront’ but PB does not? Why hologram creates automatic accommodation cues? What is the effective resolution of HG vs PB?
![Page 97: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/97.jpg)
L(x,θ) W(x,u) Wm= sinc d = delta
q Wm
p
q
p d(θ)
p
q q d(θ) *
p Wm *
*
Rays: No Bending 1 Fresnel HG Patch
θ u
*
Zooming into the Light Field
![Page 98: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/98.jpg)
s1
m2
s1
m2 s1*
s1 s1
s1*
Rank-1 Rank-1 Rank-3
Algebraic Rank Constraint
![Page 99: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/99.jpg)
x
u
-Transform
<t(x+xʹ/2)t*(x-xʹ/2)>
Interference
xʹ
x
(a) Two Slits, Coherent
t(x+xʹ/2)t*(x-xʹ/2) W(x,u)
2x
1x
Rank-1
t(x1)t*(x2)
Transform
-1
u R45, D
![Page 100: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/100.jpg)
L2 L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(x,θ)
L2(x,θ)
L3(x,θ)
d
z1
hH
r
z2
L1(x,θ) L2(x,θ) L3(x,θ)
s1 m2
(a)
![Page 101: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/101.jpg)
Is hologram just another ray-based light field? Can a hologram create any intensity distribution in 3D? Why hologram creates a ‘wavefront’ but PB does not? Why hologram creates automatic accommodation cues? What is the effective resolution of HG vs PB?
![Page 102: Computational Light Transport Computational Photography ...groups.csail.mit.edu/netmit/sFFT/RaskarSparseFFTFeb2013.pdfWe demonstrate a new technique that allows a camera to rapidly](https://reader030.fdocuments.in/reader030/viewer/2022040301/5e722539b18c21283315b6e6/html5/thumbnails/102.jpg)
MIT Media Lab [email protected] http://cameraculture.info
• Co-design of Optics and Computation – Photons not just pixels
– Mid-level cues
• Computational Photography
– Coded Exposure
– Compressive Sensing for High Speed Events
• Limits of CS for general imaging
• Computational Light Transport
– Looking Around Corners, trillion fps
– Lightfields: • Compressive Campture
• 3D Displays and Holograms
Inverse Problems