Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3....
Transcript of Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3....
![Page 1: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/1.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Amit Agrawal
June 2010
Mitsubishi Electric Research Labs (MERL)
Cambridge, MA, USA
Coding and Modulation in Cameras
![Page 2: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/2.jpg)
Coded Computational Imaging Agrawal, Veeraraghavan, Narasimhan & Mohan
Schedule
Introduction Srinivasa, 10 mins
Assorted Pixels Srinivasa, 20 mins
Coding and Modulation in Cameras Amit, 45 mins
Break 10 min
Light Fields and Applications Ankit, 60 mins
Break 10 min
Computational Illumination Srinivasa, 45 mins
Future Trends Amit, 15 mins
Discussion
![Page 3: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/3.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Have Cameras Evolved?
Digital CamerasLens Based Camera Obscura, 1568
![Page 4: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/4.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Traditional Photography
Lens
Detector
Pixels
Image
![Page 5: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/5.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Computational Imaging: Optics, Sensors and Computations
GeneralizedSensor
GeneralizedOptics
Computations
Picture
4D Ray Bender
Upto 4D Ray Sampler
Ray Reconstruction
Slide Courtesy Ramesh Raskar
![Page 6: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/6.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
![Page 7: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/7.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Flash Hotspots
Glass Reflections
![Page 8: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/8.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Motion Blur
![Page 9: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/9.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Camera Shake
![Page 10: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/10.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
High Dynamic Range
![Page 11: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/11.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Out of Focus
![Page 12: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/12.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Fog, Haze, Bad
Weather
![Page 13: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/13.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Computational Imaging
• Photo Manipulations– Two photos are better than one!!
– Combine two+ photos
– Change camera parameters
• Coding and Modulation
– Capture relevant information
– Decode in software
![Page 14: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/14.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Two photos are better than one
Camera
Controls
Exposure
Time
![Page 15: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/15.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Changing Exposure Time
• High dynamic range imaging
– Mann and Picard, 95
– Devebec and Malik, 97
Photo 1 Photo 2 Photo 3
![Page 16: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/16.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Epsilon Photography
• Dynamic Range
– Multiple exposures
• Noise Reduction
– Flash/No-flash images, Mutiple Images
• Motion Blur
– Short + Long exposure
• Focus Blur
– Focal Stack (Images focused at different depths)
• Spectrum
– Visible + Near-IR
• Field of View
– Panaromas
• Resolution
– Multi-image super-resolution methods
![Page 17: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/17.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Computational Imaging
• Photo Manipulations– Two photos are better than one!!
– Combine two+ photos
– Change camera parameters
• Coding and Modulation
– Capture relevant information
– Decode in software
![Page 18: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/18.jpg)
detector
lens
image
Traditional Camera
detector
newoptics
computeimage
Computational Camera
Slide: Courtesy Shree Nayar
![Page 19: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/19.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Computational Cameras
• Adaptive Dynamic Range Imaging, Nayar & Branzoi, ICCV 2003
• Omnidirectional Cameras, Gluckman & Nayar, ICCV 98
• Folded Catadioptric Cameras, Nayar & Peri, CVPR 99
• Catadioptric Imaging, Nayar 88
• Flexible Field of View, Kuthirummal & Nayar, 07
• Cata-Fisheye Camera for Panoramic Imaging,“ Krishnan & Nayar, 08
• Generalized Mosaicing, Schechner & Nayar, ICCV 01
• Motion Deblurring using Hybrid Imaging, Ben-Ezra & Nayar
• Jitter Camera, Ben-Ezra et al CVPR 04
• Programmable Imaging, Nayar et al 2004
• Single Lens Depth Camera, Gao & Ahuja, 2006
• Omnidirectional Stereo Vision System, Yi and Ahuja, 06
• Omnifocus Nonfrontal Imaging Camera, Aggarwal et al
• Split Aperture Imaging, Aggarwal and Ahuja, 2001
• Plenoptic Cameras, Adelson & Wang, Ng et al., Stanford
• Multi-Aperture Photography, Green et al. SIGGRAPH 07
• Coded Apertures, Wavefront Coding (CDM Optics)
• Assorted Pixels, Narasimhan & Nayar
![Page 20: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/20.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
![Page 21: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/21.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Coded Exposure [Raskar, Agrawal, Tumblin SIGGRAPH 2006]
![Page 22: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/22.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Coded Exposure (Flutter Shutter) Camera
Raskar, Agrawal, Tumblin [Siggraph2006]
Coding in Time: Shutter is opened and closed
![Page 23: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/23.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Blurring == Convolution
Traditional Camera: Shutter is OPEN: Box Filter
PSF == Sinc Function
ω
Sharp Photo Blurred Photo
![Page 24: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/24.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Flutter Shutter: Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo Blurred Photo
PSF == Broadband Function
![Page 25: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/25.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
TraditionalCoded
Exposure
Image of Static Object
Deblurred Image
Deblurred Image
![Page 26: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/26.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
![Page 27: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/27.jpg)
![Page 28: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/28.jpg)
![Page 29: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/29.jpg)
Coded Exposure (Flutter Shutter) Camera
Raskar, Agrawal, Tumblin [Siggraph2006]
PointGrey Camera No additional Cost
Coding in Time: Shutter is opened and closed
External Shutter with SLR Camera
![Page 30: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/30.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit AgrawalHow to handle focus blur?
![Page 31: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/31.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Coded Exposure (Flutter Shutter)Raskar, Agrawal, Tumblin
SIGGRAPH 2006
Coded Aperturewith Veeraraghavan, Raskar, Tumblin, & Mohan,
SIGGRAPH 2007
Temporal 1-D broadband code: Motion Deblurring
Spatial 2-D broadband code: Focus Deblurring
![Page 32: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/32.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
In Focus Photo
LED
![Page 33: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/33.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Out of Focus Photo: Open Aperture
![Page 34: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/34.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Out of Focus Photo: Coded Aperture
![Page 35: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/35.jpg)
Blurred Photos
Coded Aperture, 7 * 7 MaskOpen Aperture
![Page 36: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/36.jpg)
Deblurred Photos
Coded Aperture, 7 * 7 MaskOpen Aperture
![Page 37: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/37.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit AgrawalCaptured Blurred
Photo
![Page 38: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/38.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Refocused on Person
![Page 39: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/39.jpg)
![Page 40: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/40.jpg)
![Page 41: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/41.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Blocking Light == More Information
Coded ExposureCoding in Time
Coded ApertureCoding in Space
![Page 42: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/42.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Key Concept 1: PSF Invertibility
• Modify the PSF to be invertible
– PSF == Impulse Response
• Traditional Camera
– Non-invertible PSF (loses information)
• Coding in Camera
– Invertible PSF
• Coding in Time == Coded Exposure
• Coding in Space == Coded Aperture
![Page 43: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/43.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Key Concept: PSF Null-Filling
• PSF invertibility using multiple photos
• No camera modification required
• Can do it on available SLR‟s
– Using Exposure Bracketing mode (AEB)
“Invertible Motion Blur in Video”, Agrawal, Xu and Raskar, SIGGRAPH 2009
Varying Exposure
Photo 1
Photo 2
Photo 3
![Page 44: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/44.jpg)
Traditional Exposure Video
Fourier
Transform
Motion PSF
(Box Filter)
Information is lost
Exposure Time
![Page 45: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/45.jpg)
Varying Exposure Video
Exposure Time Fourier
Transform
![Page 46: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/46.jpg)
Varying Exposure Video
Exposure Time
Exposure Time Fourier
Transform
No common nulls
![Page 47: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/47.jpg)
Varying Exposure Video
Exposure Time
Exposure Time
Exposure Time
Fourier
Transform
No common nulls
![Page 48: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/48.jpg)
Varying Exposure Video = PSF Null-Filling
Joint Frequency
Spectrum
Preserves All
Frequencies
Fourier
Transform
![Page 49: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/49.jpg)
Key Idea: PSF Null-Filling
• Individual non-invertible PSF’s
combined into jointly-invertible PSF
– Information lost in any single photo is
captured in some other photo
• For motion deblurring
– Achieve PSF null-filling by varying the
exposure time of successive photos
Varying Exposure
Photo 1
Photo 2
Photo 3
![Page 50: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/50.jpg)
Varying Exposure Video
![Page 51: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/51.jpg)
Deblurred Result
Blurred Photos
![Page 52: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/52.jpg)
Outdoor Car
Photo 1 Photo 2 Photo 3 Deblurring
![Page 53: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/53.jpg)
Blurred Photo 1 Blurred Photo 2
Blurred Photo 3 Deblurred
Face
![Page 54: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/54.jpg)
Auto Exposure Bracketing (AEB) for
Varying Exposure Deblurring
1/50s 1/80s 1/30s
![Page 55: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/55.jpg)
Deblurred Result
Blurred Photos
![Page 56: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/56.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Key Concept 2: PSF Invariance
• But…
– Need to estimate depth or velocity for deblurring
• Modify the PSF to be invariant
• Motion Blur
– Motion invariant Photography (MIP), Levin et al SIG08
• Focus Blur• Wavefront coding
• Focus Sweep Camera
• Spectral Sweep Camera
• Diffusion coding
![Page 57: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/57.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
PSF Invariance: Motion Blur
• Move the camera while taking photo
– Constant Camera Acceleration
• Leads to similar PSF for object velocity in a range
• But requires knowledge of motion direction
PSF
Coded
Exposure
Traditional
CameraMIP
![Page 58: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/58.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Comparison
• Coded Exposure
– Requires motion magnitude for deblurring
– But works for any motion direction
• PSF Invariance
– Requires motion direction to move the camera
– But invariant PSF for motion magnitude within a range
Optimal Single Image Capture for Motion Deblurring, Agrawal and Raskar,
CVPR 2009
![Page 59: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/59.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
PSF Invariance: Focus Blur
• Defocus PSF should be invariant of depth
Nagahara et al. ECCV 2008
![Page 60: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/60.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
![Page 61: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/61.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Wavefront Coding
• Traditional Lens:
– Defocus („circle of confusion) dependent on distance from plane of focus
http://www.cdm-optics.com
![Page 62: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/62.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Wavefront Coding
• Traditional Lens:
– Defocus dependent on distance from plane of focus
• Cubic Phase Plate
– Defocus nearly independent of distance
– All points „blurred‟
– Deconvolve to get sharper image
http://www.cdm-optics.com
![Page 63: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/63.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Spectral Focal Sweep Lens
Cossairt and Nayar, ICCP 2010
![Page 64: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/64.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Cossairt and Nayar, ICCP 2010
![Page 65: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/65.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
PSF Invariance: Focus Blur
• Vary focal length in captured photo
Focal Length Variation
Time
Hardware Implementation
Reference
Sensor Motion Nagahara et al. ECCV 2008
Phase/Angle Cubic Phase Plate Wavefront Optics
WavelengthLens with Chromatic
AberrationsCossairt and Nayar, ICCP 2010
ApertureDivide the aperture into
different lensBen-Eliezer, Applied Optics 2005,
Levin et al SIGGRAPH 2009
![Page 66: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/66.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
PSF Invariance: Diffusion Coding
• Use a radially symmetric diffuser in aperture
• Cossairt and Nayar, SIGGRAPH 2010
![Page 67: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/67.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
High Speed Imaging
• High speed cameras
– Expensive
– Require on-board memory
– Fundamental Light Loss
30 fps 500 fps 2000 fps 4000 fps
-24.5 dB -36.5 dB -42.5 dB
![Page 68: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/68.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Coded Strobing Camera: 100x Temporal Super-Resolution
• For Periodic Signals
• Coded Exposure Video
– Every frame is coded differently
![Page 69: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/69.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Battery powered Toothbrush
20fps normal camera 20fps coded strobing camera
Reconstructed frames 1000fps hi-speed camera
![Page 70: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/70.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Implementation
Captured at 10fps PGR Dragonfly2 External FLC Shutter
Can strobed at 1ms Can strobe at 250us
![Page 71: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/71.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Temporally at a pixel
observe different linear combinations of the periodic signal
Advantage of the design:
Exposure coding is independent of the frequency periodic signal.
50% light throughput, far greater than traditional strobing.
tP = 10ms
![Page 72: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/72.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Compressive sensing Reconstruction
Very few (K)
non-zero
elements
Sparse Basis Coeff
y = A sMixing matrixObserved low
rate frames
1 2min . .s s t y As
Basis Pursuit De-noising
![Page 73: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/73.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Battery powered Toothbrush
20fps normal camera 20fps coded strobing camera
Reconstructed frames 1000fps hi-speed camera
![Page 74: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/74.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Rotating Mill Tool captured by PointGrey Dragonfly2
Reconstructed Video at 2000fps
Coded Strobing Video: 25fpsNormal Video: 25fps
![Page 75: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/75.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
High Speed Imaging
• Coded Strobing Camera for Periodic Signals
• For General Scenes?
Agrawal et al. CVPR 2010, Wilburn et al. CVPR 2004, Shechtman et al. ECCV 2002
Camera Arrays
![Page 76: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/76.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Camera Arrays
• Point Sampling
– Wilburn et al CVPR 2004
– Each camera captures independent
sample of high speed video
High temporal resolution video
Point Sampling
Interleave Frames
C1
C2
C3
C4
in
fT
inT
inoutin
f
out
f TTNTT ,/
![Page 77: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/77.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Camera Arrays
• Box Sampling
Box Sampling
High temporal resolution video
C1
C2
C3
C4
Interleave Frames Solve Linear System
in
fT inT
NTT in
f
out
f / NTT inout /
![Page 78: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/78.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Space-Time Super-Resolution, Shechtman et al. ECCV 2002
![Page 79: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/79.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Camera Arrays
• Coded Sampling• Agrawal, Gupta, Veeraraghavan and Narasimhan CVPR 2010
Interleave Frames Solve Linear Systemout
f
outin
f
out
f TTNTT ,/
Coded Sampling
High temporal resolution video
C1
C2
C3
C4
in
fT in
fT
![Page 80: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/80.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Implementation
![Page 81: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/81.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Flexible Videography[Gupta, Agrawal, Veeraraghavan and Narasimhan, ECCV 2010]
• Resolution Tradeoff: Traditional Video Camera
– Fixed Space-Time Resolution
– Independent of the scene
– Same all over the image
• Flexible Videography
– Change space-time resolution in post capture
– Scene dependent Resolution
– Different for different parts of the image
![Page 82: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/82.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Flexible Voxels
• Per pixel coded exposure
– Different temporal modulation per pixel
Image Plane
ProjectorImage
Plane
Camera
Scene
Beam
Splitter
Time
Pixel 1
Pixel 2
Pixel K
Camera Integration TimeProjector Pattern
![Page 83: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/83.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Image Plane
ProjectorImage
Plane
Camera
Scene
Beam
Splitter
![Page 84: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/84.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
2 1 2
34
5 6
789 10
1112
13 14
1516
t
1
5
15
8
4
11
16
9 7 10
13 3 14
12 6
1 2
34
5 6
78
9 10
1112
13 14
1516t
1 234
5 678
9 101112
13 141516
t
12345678910111213141516
t
TR = 4, SR = 1/4 TR = 8, SR = 1/8 TR = 16, SR = 1/16
TR = 2, SR = 1/2TR = 1, SR = 1/1
1
5
15
8
4
11
216
9 7 10
13 3 14
12 6
x
y
Sampling Strategy for 1-16x
![Page 85: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/85.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Captured Video
![Page 86: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/86.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Naïve Reconstruction
8X Temporal Super-res, but 8 times lower spatial resolution
![Page 87: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/87.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Optical Flow Magnitudes
![Page 88: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/88.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Motion Aware Reconstruction
8X Temporal Super-res on moving fan, same spatial resolution on static parts
![Page 89: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/89.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Motion Aware Reconstruction
Optical Flow
Naïve Reconstruction
Captured Video
![Page 90: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/90.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Summary: Temporal Modulations
• Coded Exposure (Photo)
– Same for all pixels
– Motion deblurring
• Strobe Camera (Video)
– Same for all pixels in a frame
– Different across frames
– Temporal Super-Resolution (100x)
• Multi-Camera Arrays (Video)
– Same for all pixels in a frame
– Same across frames, different across cameras
– High Speed Imaging
• Flexible Voxels (Motion Aware Video)
– Different for pixels in a frame
– Same across frames
– Post Capture Space Time Resolution Tradeoff
![Page 91: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/91.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit Agrawal
Section Summary
• Coding and Modulation
– Beyond Photo Manipulations
• Key Concepts
– PSF Invertibility and PSF Invariance
• Motion Blur and Defocus blur
– Coded exposure, Coded aperture, Wavefront coding etc.
• High Speed Imaging
– Strobing Camera
– Coded Sampling for Camera Arrays
• Flexible Videography
– Post-capture Resolution Tradeoff
![Page 92: Amit Agrawal June 2010ILIM/projects/IM/aagrawal/...–Devebec and Malik, 97 Photo 1 Photo 2 Photo 3. Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Amit](https://reader033.fdocuments.in/reader033/viewer/2022041907/5e6404107885b42e3340dfba/html5/thumbnails/92.jpg)
Coded Computational Imaging Agrawal, Veeraraghavan, Narasimhan & Mohan
Schedule
Introduction Srinivasa, 10 mins
Assorted Pixels Srinivasa, 20 mins
Coding and Modulation in Cameras Amit, 45 mins
Break 10 min
Light Fields and Applications Ankit, 60 mins
Break 10 min
Computational Illumination Srinivasa, 45 mins
Future Trends Amit, 15 mins
Discussion