Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer...
Click here to load reader
-
Upload
cameraculture-mit-media-lab -
Category
Technology
-
view
2.386 -
download
1
Transcript of Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer...
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Light Transport and
Computational Photography Inverse problems
MIT Media LabRamesh Raskar httpraskarinfo
raskarmitedu
Raskar Camera Culture MIT Media Lab
Ramesh Raskar MIT Media Lab
After X what is neXt
How to Invent
Ramesh Raskar MIT Media Lab
Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010
Ramesh Raskar MIT Media Lab
Xd
X++
X X+Y
X
X
neXt
Ramesh Raskar httpraskarinfo
Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010
Ramesh Raskar MIT Media Lab
Simple Exercise bull Image Compression
ndash Save Bandwidth and storage
What is neXt
Ramesh Raskar MIT Media Lab
Strategy 1 Xd
bull Extend it to next (or some other) dimension
Ramesh Raskar MIT Media Lab
X = bull Idea you just heardbull Conceptbull Patentbull New ProductBest projectinvention awardbull Product featurebull Designbull Artbull Algorithm
Ramesh Raskar MIT Media Lab
Xd
X++
X X+Y
X
X
neXt
Ramesh Raskar httpraskarinfo
Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010
Research bull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Is project worthwhile Heilmeiers Questions
bull Whatndash What are you trying to do Articulate your objectives using absolutely no jargon
bull Related workndash How is it done today and what are the limits of current practice
bull Contributionndash Whats new in your approach and why do you think it will be successful
bull Motivationndash Who caresndash If youre successful what difference will it make
bull Challengesndash What are the risks and the payoffsndash How much will it costndash How long will it take
bull Evaluationndash What are the midterm and final exams to check for success
bull Raskar additions ndash Why now (why not before whatrsquos new that makes possible)ndash Why us (wrong answers I am smart I can work harder than others)
httpenwikipediaorgwikiGeorge_H_HeilmeierHeilmeier27s_Catechism
Great Research Strive for Five
1 Before Five teamsBe first often let others do details
2 Beyond Five yearsWhat no one is thinking about
3 Within Five layers of lsquoHumanrsquo ImpactRelevance
4 Beyond Five minutes of descriptionDeep iterative participatory
5 Fusing Five+ ExpertiseMulti-disciplinary proactive
Ramesh Raskar httpraskarinfo
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Fernald Science [Sept 2006]
Shadow Refractive
Reflective
Tools for
Visual Computin
g
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Traditional Photography
Lens
Detector
Pixels
Image
Mimics Human Eye for a Single Snapshot
Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree
Nayar
Computational Camera + Photography
Optics Sensors and ComputationsGeneralized
Sensor
Generalized OpticsComputations
Picture
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Raskar Camera Culture MIT Media Lab
Ramesh Raskar MIT Media Lab
After X what is neXt
How to Invent
Ramesh Raskar MIT Media Lab
Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010
Ramesh Raskar MIT Media Lab
Xd
X++
X X+Y
X
X
neXt
Ramesh Raskar httpraskarinfo
Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010
Ramesh Raskar MIT Media Lab
Simple Exercise bull Image Compression
ndash Save Bandwidth and storage
What is neXt
Ramesh Raskar MIT Media Lab
Strategy 1 Xd
bull Extend it to next (or some other) dimension
Ramesh Raskar MIT Media Lab
X = bull Idea you just heardbull Conceptbull Patentbull New ProductBest projectinvention awardbull Product featurebull Designbull Artbull Algorithm
Ramesh Raskar MIT Media Lab
Xd
X++
X X+Y
X
X
neXt
Ramesh Raskar httpraskarinfo
Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010
Research bull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Is project worthwhile Heilmeiers Questions
bull Whatndash What are you trying to do Articulate your objectives using absolutely no jargon
bull Related workndash How is it done today and what are the limits of current practice
bull Contributionndash Whats new in your approach and why do you think it will be successful
bull Motivationndash Who caresndash If youre successful what difference will it make
bull Challengesndash What are the risks and the payoffsndash How much will it costndash How long will it take
bull Evaluationndash What are the midterm and final exams to check for success
bull Raskar additions ndash Why now (why not before whatrsquos new that makes possible)ndash Why us (wrong answers I am smart I can work harder than others)
httpenwikipediaorgwikiGeorge_H_HeilmeierHeilmeier27s_Catechism
Great Research Strive for Five
1 Before Five teamsBe first often let others do details
2 Beyond Five yearsWhat no one is thinking about
3 Within Five layers of lsquoHumanrsquo ImpactRelevance
4 Beyond Five minutes of descriptionDeep iterative participatory
5 Fusing Five+ ExpertiseMulti-disciplinary proactive
Ramesh Raskar httpraskarinfo
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Fernald Science [Sept 2006]
Shadow Refractive
Reflective
Tools for
Visual Computin
g
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Traditional Photography
Lens
Detector
Pixels
Image
Mimics Human Eye for a Single Snapshot
Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree
Nayar
Computational Camera + Photography
Optics Sensors and ComputationsGeneralized
Sensor
Generalized OpticsComputations
Picture
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Ramesh Raskar MIT Media Lab
After X what is neXt
How to Invent
Ramesh Raskar MIT Media Lab
Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010
Ramesh Raskar MIT Media Lab
Xd
X++
X X+Y
X
X
neXt
Ramesh Raskar httpraskarinfo
Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010
Ramesh Raskar MIT Media Lab
Simple Exercise bull Image Compression
ndash Save Bandwidth and storage
What is neXt
Ramesh Raskar MIT Media Lab
Strategy 1 Xd
bull Extend it to next (or some other) dimension
Ramesh Raskar MIT Media Lab
X = bull Idea you just heardbull Conceptbull Patentbull New ProductBest projectinvention awardbull Product featurebull Designbull Artbull Algorithm
Ramesh Raskar MIT Media Lab
Xd
X++
X X+Y
X
X
neXt
Ramesh Raskar httpraskarinfo
Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010
Research bull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Is project worthwhile Heilmeiers Questions
bull Whatndash What are you trying to do Articulate your objectives using absolutely no jargon
bull Related workndash How is it done today and what are the limits of current practice
bull Contributionndash Whats new in your approach and why do you think it will be successful
bull Motivationndash Who caresndash If youre successful what difference will it make
bull Challengesndash What are the risks and the payoffsndash How much will it costndash How long will it take
bull Evaluationndash What are the midterm and final exams to check for success
bull Raskar additions ndash Why now (why not before whatrsquos new that makes possible)ndash Why us (wrong answers I am smart I can work harder than others)
httpenwikipediaorgwikiGeorge_H_HeilmeierHeilmeier27s_Catechism
Great Research Strive for Five
1 Before Five teamsBe first often let others do details
2 Beyond Five yearsWhat no one is thinking about
3 Within Five layers of lsquoHumanrsquo ImpactRelevance
4 Beyond Five minutes of descriptionDeep iterative participatory
5 Fusing Five+ ExpertiseMulti-disciplinary proactive
Ramesh Raskar httpraskarinfo
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Fernald Science [Sept 2006]
Shadow Refractive
Reflective
Tools for
Visual Computin
g
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Traditional Photography
Lens
Detector
Pixels
Image
Mimics Human Eye for a Single Snapshot
Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree
Nayar
Computational Camera + Photography
Optics Sensors and ComputationsGeneralized
Sensor
Generalized OpticsComputations
Picture
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Ramesh Raskar MIT Media Lab
Xd
X++
X X+Y
X
X
neXt
Ramesh Raskar httpraskarinfo
Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010
Ramesh Raskar MIT Media Lab
Simple Exercise bull Image Compression
ndash Save Bandwidth and storage
What is neXt
Ramesh Raskar MIT Media Lab
Strategy 1 Xd
bull Extend it to next (or some other) dimension
Ramesh Raskar MIT Media Lab
X = bull Idea you just heardbull Conceptbull Patentbull New ProductBest projectinvention awardbull Product featurebull Designbull Artbull Algorithm
Ramesh Raskar MIT Media Lab
Xd
X++
X X+Y
X
X
neXt
Ramesh Raskar httpraskarinfo
Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010
Research bull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Is project worthwhile Heilmeiers Questions
bull Whatndash What are you trying to do Articulate your objectives using absolutely no jargon
bull Related workndash How is it done today and what are the limits of current practice
bull Contributionndash Whats new in your approach and why do you think it will be successful
bull Motivationndash Who caresndash If youre successful what difference will it make
bull Challengesndash What are the risks and the payoffsndash How much will it costndash How long will it take
bull Evaluationndash What are the midterm and final exams to check for success
bull Raskar additions ndash Why now (why not before whatrsquos new that makes possible)ndash Why us (wrong answers I am smart I can work harder than others)
httpenwikipediaorgwikiGeorge_H_HeilmeierHeilmeier27s_Catechism
Great Research Strive for Five
1 Before Five teamsBe first often let others do details
2 Beyond Five yearsWhat no one is thinking about
3 Within Five layers of lsquoHumanrsquo ImpactRelevance
4 Beyond Five minutes of descriptionDeep iterative participatory
5 Fusing Five+ ExpertiseMulti-disciplinary proactive
Ramesh Raskar httpraskarinfo
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Fernald Science [Sept 2006]
Shadow Refractive
Reflective
Tools for
Visual Computin
g
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Traditional Photography
Lens
Detector
Pixels
Image
Mimics Human Eye for a Single Snapshot
Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree
Nayar
Computational Camera + Photography
Optics Sensors and ComputationsGeneralized
Sensor
Generalized OpticsComputations
Picture
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Ramesh Raskar MIT Media Lab
Simple Exercise bull Image Compression
ndash Save Bandwidth and storage
What is neXt
Ramesh Raskar MIT Media Lab
Strategy 1 Xd
bull Extend it to next (or some other) dimension
Ramesh Raskar MIT Media Lab
X = bull Idea you just heardbull Conceptbull Patentbull New ProductBest projectinvention awardbull Product featurebull Designbull Artbull Algorithm
Ramesh Raskar MIT Media Lab
Xd
X++
X X+Y
X
X
neXt
Ramesh Raskar httpraskarinfo
Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010
Research bull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Is project worthwhile Heilmeiers Questions
bull Whatndash What are you trying to do Articulate your objectives using absolutely no jargon
bull Related workndash How is it done today and what are the limits of current practice
bull Contributionndash Whats new in your approach and why do you think it will be successful
bull Motivationndash Who caresndash If youre successful what difference will it make
bull Challengesndash What are the risks and the payoffsndash How much will it costndash How long will it take
bull Evaluationndash What are the midterm and final exams to check for success
bull Raskar additions ndash Why now (why not before whatrsquos new that makes possible)ndash Why us (wrong answers I am smart I can work harder than others)
httpenwikipediaorgwikiGeorge_H_HeilmeierHeilmeier27s_Catechism
Great Research Strive for Five
1 Before Five teamsBe first often let others do details
2 Beyond Five yearsWhat no one is thinking about
3 Within Five layers of lsquoHumanrsquo ImpactRelevance
4 Beyond Five minutes of descriptionDeep iterative participatory
5 Fusing Five+ ExpertiseMulti-disciplinary proactive
Ramesh Raskar httpraskarinfo
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Fernald Science [Sept 2006]
Shadow Refractive
Reflective
Tools for
Visual Computin
g
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Traditional Photography
Lens
Detector
Pixels
Image
Mimics Human Eye for a Single Snapshot
Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree
Nayar
Computational Camera + Photography
Optics Sensors and ComputationsGeneralized
Sensor
Generalized OpticsComputations
Picture
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Ramesh Raskar MIT Media Lab
Strategy 1 Xd
bull Extend it to next (or some other) dimension
Ramesh Raskar MIT Media Lab
X = bull Idea you just heardbull Conceptbull Patentbull New ProductBest projectinvention awardbull Product featurebull Designbull Artbull Algorithm
Ramesh Raskar MIT Media Lab
Xd
X++
X X+Y
X
X
neXt
Ramesh Raskar httpraskarinfo
Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010
Research bull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Is project worthwhile Heilmeiers Questions
bull Whatndash What are you trying to do Articulate your objectives using absolutely no jargon
bull Related workndash How is it done today and what are the limits of current practice
bull Contributionndash Whats new in your approach and why do you think it will be successful
bull Motivationndash Who caresndash If youre successful what difference will it make
bull Challengesndash What are the risks and the payoffsndash How much will it costndash How long will it take
bull Evaluationndash What are the midterm and final exams to check for success
bull Raskar additions ndash Why now (why not before whatrsquos new that makes possible)ndash Why us (wrong answers I am smart I can work harder than others)
httpenwikipediaorgwikiGeorge_H_HeilmeierHeilmeier27s_Catechism
Great Research Strive for Five
1 Before Five teamsBe first often let others do details
2 Beyond Five yearsWhat no one is thinking about
3 Within Five layers of lsquoHumanrsquo ImpactRelevance
4 Beyond Five minutes of descriptionDeep iterative participatory
5 Fusing Five+ ExpertiseMulti-disciplinary proactive
Ramesh Raskar httpraskarinfo
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Fernald Science [Sept 2006]
Shadow Refractive
Reflective
Tools for
Visual Computin
g
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Traditional Photography
Lens
Detector
Pixels
Image
Mimics Human Eye for a Single Snapshot
Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree
Nayar
Computational Camera + Photography
Optics Sensors and ComputationsGeneralized
Sensor
Generalized OpticsComputations
Picture
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Ramesh Raskar MIT Media Lab
X = bull Idea you just heardbull Conceptbull Patentbull New ProductBest projectinvention awardbull Product featurebull Designbull Artbull Algorithm
Ramesh Raskar MIT Media Lab
Xd
X++
X X+Y
X
X
neXt
Ramesh Raskar httpraskarinfo
Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010
Research bull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Is project worthwhile Heilmeiers Questions
bull Whatndash What are you trying to do Articulate your objectives using absolutely no jargon
bull Related workndash How is it done today and what are the limits of current practice
bull Contributionndash Whats new in your approach and why do you think it will be successful
bull Motivationndash Who caresndash If youre successful what difference will it make
bull Challengesndash What are the risks and the payoffsndash How much will it costndash How long will it take
bull Evaluationndash What are the midterm and final exams to check for success
bull Raskar additions ndash Why now (why not before whatrsquos new that makes possible)ndash Why us (wrong answers I am smart I can work harder than others)
httpenwikipediaorgwikiGeorge_H_HeilmeierHeilmeier27s_Catechism
Great Research Strive for Five
1 Before Five teamsBe first often let others do details
2 Beyond Five yearsWhat no one is thinking about
3 Within Five layers of lsquoHumanrsquo ImpactRelevance
4 Beyond Five minutes of descriptionDeep iterative participatory
5 Fusing Five+ ExpertiseMulti-disciplinary proactive
Ramesh Raskar httpraskarinfo
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Fernald Science [Sept 2006]
Shadow Refractive
Reflective
Tools for
Visual Computin
g
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Traditional Photography
Lens
Detector
Pixels
Image
Mimics Human Eye for a Single Snapshot
Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree
Nayar
Computational Camera + Photography
Optics Sensors and ComputationsGeneralized
Sensor
Generalized OpticsComputations
Picture
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Ramesh Raskar MIT Media Lab
Xd
X++
X X+Y
X
X
neXt
Ramesh Raskar httpraskarinfo
Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010
Research bull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Is project worthwhile Heilmeiers Questions
bull Whatndash What are you trying to do Articulate your objectives using absolutely no jargon
bull Related workndash How is it done today and what are the limits of current practice
bull Contributionndash Whats new in your approach and why do you think it will be successful
bull Motivationndash Who caresndash If youre successful what difference will it make
bull Challengesndash What are the risks and the payoffsndash How much will it costndash How long will it take
bull Evaluationndash What are the midterm and final exams to check for success
bull Raskar additions ndash Why now (why not before whatrsquos new that makes possible)ndash Why us (wrong answers I am smart I can work harder than others)
httpenwikipediaorgwikiGeorge_H_HeilmeierHeilmeier27s_Catechism
Great Research Strive for Five
1 Before Five teamsBe first often let others do details
2 Beyond Five yearsWhat no one is thinking about
3 Within Five layers of lsquoHumanrsquo ImpactRelevance
4 Beyond Five minutes of descriptionDeep iterative participatory
5 Fusing Five+ ExpertiseMulti-disciplinary proactive
Ramesh Raskar httpraskarinfo
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Fernald Science [Sept 2006]
Shadow Refractive
Reflective
Tools for
Visual Computin
g
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Traditional Photography
Lens
Detector
Pixels
Image
Mimics Human Eye for a Single Snapshot
Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree
Nayar
Computational Camera + Photography
Optics Sensors and ComputationsGeneralized
Sensor
Generalized OpticsComputations
Picture
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Research bull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Is project worthwhile Heilmeiers Questions
bull Whatndash What are you trying to do Articulate your objectives using absolutely no jargon
bull Related workndash How is it done today and what are the limits of current practice
bull Contributionndash Whats new in your approach and why do you think it will be successful
bull Motivationndash Who caresndash If youre successful what difference will it make
bull Challengesndash What are the risks and the payoffsndash How much will it costndash How long will it take
bull Evaluationndash What are the midterm and final exams to check for success
bull Raskar additions ndash Why now (why not before whatrsquos new that makes possible)ndash Why us (wrong answers I am smart I can work harder than others)
httpenwikipediaorgwikiGeorge_H_HeilmeierHeilmeier27s_Catechism
Great Research Strive for Five
1 Before Five teamsBe first often let others do details
2 Beyond Five yearsWhat no one is thinking about
3 Within Five layers of lsquoHumanrsquo ImpactRelevance
4 Beyond Five minutes of descriptionDeep iterative participatory
5 Fusing Five+ ExpertiseMulti-disciplinary proactive
Ramesh Raskar httpraskarinfo
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Fernald Science [Sept 2006]
Shadow Refractive
Reflective
Tools for
Visual Computin
g
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Traditional Photography
Lens
Detector
Pixels
Image
Mimics Human Eye for a Single Snapshot
Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree
Nayar
Computational Camera + Photography
Optics Sensors and ComputationsGeneralized
Sensor
Generalized OpticsComputations
Picture
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Is project worthwhile Heilmeiers Questions
bull Whatndash What are you trying to do Articulate your objectives using absolutely no jargon
bull Related workndash How is it done today and what are the limits of current practice
bull Contributionndash Whats new in your approach and why do you think it will be successful
bull Motivationndash Who caresndash If youre successful what difference will it make
bull Challengesndash What are the risks and the payoffsndash How much will it costndash How long will it take
bull Evaluationndash What are the midterm and final exams to check for success
bull Raskar additions ndash Why now (why not before whatrsquos new that makes possible)ndash Why us (wrong answers I am smart I can work harder than others)
httpenwikipediaorgwikiGeorge_H_HeilmeierHeilmeier27s_Catechism
Great Research Strive for Five
1 Before Five teamsBe first often let others do details
2 Beyond Five yearsWhat no one is thinking about
3 Within Five layers of lsquoHumanrsquo ImpactRelevance
4 Beyond Five minutes of descriptionDeep iterative participatory
5 Fusing Five+ ExpertiseMulti-disciplinary proactive
Ramesh Raskar httpraskarinfo
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Fernald Science [Sept 2006]
Shadow Refractive
Reflective
Tools for
Visual Computin
g
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Traditional Photography
Lens
Detector
Pixels
Image
Mimics Human Eye for a Single Snapshot
Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree
Nayar
Computational Camera + Photography
Optics Sensors and ComputationsGeneralized
Sensor
Generalized OpticsComputations
Picture
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Great Research Strive for Five
1 Before Five teamsBe first often let others do details
2 Beyond Five yearsWhat no one is thinking about
3 Within Five layers of lsquoHumanrsquo ImpactRelevance
4 Beyond Five minutes of descriptionDeep iterative participatory
5 Fusing Five+ ExpertiseMulti-disciplinary proactive
Ramesh Raskar httpraskarinfo
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Fernald Science [Sept 2006]
Shadow Refractive
Reflective
Tools for
Visual Computin
g
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Traditional Photography
Lens
Detector
Pixels
Image
Mimics Human Eye for a Single Snapshot
Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree
Nayar
Computational Camera + Photography
Optics Sensors and ComputationsGeneralized
Sensor
Generalized OpticsComputations
Picture
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Fernald Science [Sept 2006]
Shadow Refractive
Reflective
Tools for
Visual Computin
g
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Traditional Photography
Lens
Detector
Pixels
Image
Mimics Human Eye for a Single Snapshot
Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree
Nayar
Computational Camera + Photography
Optics Sensors and ComputationsGeneralized
Sensor
Generalized OpticsComputations
Picture
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Fernald Science [Sept 2006]
Shadow Refractive
Reflective
Tools for
Visual Computin
g
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Traditional Photography
Lens
Detector
Pixels
Image
Mimics Human Eye for a Single Snapshot
Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree
Nayar
Computational Camera + Photography
Optics Sensors and ComputationsGeneralized
Sensor
Generalized OpticsComputations
Picture
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Traditional Photography
Lens
Detector
Pixels
Image
Mimics Human Eye for a Single Snapshot
Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree
Nayar
Computational Camera + Photography
Optics Sensors and ComputationsGeneralized
Sensor
Generalized OpticsComputations
Picture
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Traditional Photography
Lens
Detector
Pixels
Image
Mimics Human Eye for a Single Snapshot
Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree
Nayar
Computational Camera + Photography
Optics Sensors and ComputationsGeneralized
Sensor
Generalized OpticsComputations
Picture
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Computational Camera + Photography
Optics Sensors and ComputationsGeneralized
Sensor
Generalized OpticsComputations
Picture
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
- Slide 1
- Slide 2
- Slide 3
- Slide 4
- Simple Exercise
- Strategy 1 Xd
- X =
- Slide 8
- Research
- Is project worthwhile Heilmeiers Questions
- Great Research Strive for Five
- Slide 12
- Slide 13
- Slide 14
- Traditional Photography
- Computational Camera + Photography Optics Sensors and Compu
- Computational Photography
- Computational Photography [Raskar and Tumblin]
- Slide 19
- Slide 20
- Take home points
- Slide 22
- Slide 23
- Digital Refocusing using Light Field Camera
- Slide 25
- Slide 26
- Fluttered Shutter Camera
- Slide 28
- Slide 29
- Slide 30
- Slide 31
- Slide 32
- Slide 33
- Slide 34
- Slide 35
- Slide 36
- Slide 37
- Slide 38
- Slide 39
- Slide 40
- Slide 41
- Slide 42
- Slide 43
- Is Randomized Projection-based Capture apt for Natural Images
- Slide 45
- Slide 46
- Slide 47
- Slide 48
- Convert single 2D photo into 3D
- Slide 50
- Slide 51
- Slide 52
- Slide 53
- Slide 54
- Slide 55
- Depth Edges with MultiFlash
- Slide 57
- Slide 58
- Slide 59
- Slide 60
- Slide 61
- Slide 63
- Questions
- Slide 66
- Slide 67
- Take home points (2)
- Slide 69
- Slide 70
- Slide 71
- Slide 72
- Slide 73
- Femto-Photography (Transient Imaging)
- Inverting Light Transport
- Slide 76
- Slide 77
- Slide 78
- Slide 79
- Femto-Photography Time Resolved Multi-path Imaging
- Slide 81
- Slide 82
- Slide 83
- Slide 84
- Slide 85
- Slide 86
- Slide 87
- Slide 88
- Slide 89
- Slide 90
- Slide 91
- Slide 92
- Slide 93
- Slide 94
- Slide 95
- Slide 96
- Slide 97
- Slide 98
- Slide 99
- Slide 100
- Slide 101
- Slide 102
- Slide 103
- Slide 104
- Slide 105
- Slide 106
- Slide 107
- Slide 108
- Slide 109
- Slide 110
- Slide 111
- Slide 112
- Slide 113
- Slide 114
- Slide 115
- Slide 116
- Slide 117
- Augmented Light Field
- Light Fields
- Augmented Lightfield for Wave Optics Effects
- Slide 121
- Slide 122
- Slide 123
- Slide 124
- Slide 125
- Slide 126
- Slide 127
- Slide 128
- Three Questions
- How to do Research in Imaging
- Take home points (3)
- Slide 132
-