Compressive Imaging - Computer...
Transcript of Compressive Imaging - Computer...
![Page 1: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/1.jpg)
Compressive Imaging
Aswin Sankaranarayanan
(Computational Photography – Fall 2017)
![Page 2: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/2.jpg)
Traditional Models for Sensing
• Linear (for the most part)
• Take as many measurements as unknowns
sample
![Page 3: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/3.jpg)
Traditional Models for Sensing
• Linear (for the most part)
• Take as many measurements as unknowns
sampleTypically, M >= N
![Page 4: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/4.jpg)
signal
measurement matrix
measurements
Under-determined problems
Fewer measurements than unknowns!
An infinite number of solutions to such problems
![Page 5: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/5.jpg)
Credit: Rob Fergus and Antonio Torralba
![Page 6: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/6.jpg)
Credit: Rob Fergus and Antonio Torralba
![Page 7: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/7.jpg)
signal
measurement matrix
measurements
Under-determined problems
Fewer measurements than unknowns!
An infinite number of solutions to such problems
Is there any “hope” of solving these problems ?
![Page 8: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/8.jpg)
Complete the sentences
I cnt blv I m bl t rd ths sntnc.
Wntr s cmng. n wt, wntr hs cm.
Hy, I m slvng n ndr-dtrmnd lnr systm.
how: ?
![Page 9: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/9.jpg)
Complete the matrix
How: ?
![Page 10: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/10.jpg)
Complete the image
Model ?
![Page 11: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/11.jpg)
Image Dictionaries
![Page 12: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/12.jpg)
Real data has structure
Image gradients are sparse!
Image credit: David W Kennedy (Wikipedia)
![Page 13: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/13.jpg)
Real data has structure
Real world images: Only a few non-zero coefficients in a transformation
![Page 14: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/14.jpg)
signal
measurement matrix
measurements
Compressive Sensing
A toolset to solve under-determined systems by exploiting additional structure/models
on the signal we are trying to sense.
![Page 15: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/15.jpg)
Compressive Sensing
measurements
Sparse signal nonzero entries
- Suppose measurement matrix A satisfied certain conditions
- M ≥ c1K log(N/K)
- All K-sparse signals x can be recovered
- In the absence of noise, the recovery is exact!
A
[Candes and Tao, 2004]
+ noise
![Page 16: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/16.jpg)
Compressive Sensing: Big Picture
• If signal has structure, exploit it to solve under-determined problem
• Structure: Refers to a lower-dimensional parametrization of the signal class
– Sparsity in a basis (like Fourier or wavelets)
– Sparsity of gradients
– Low-rank, low-dim smooth manifold
– Any set with a projection operator
• Number of measurement is often proportional to the dim of the low-dim parameters
• Range of recovery techniques
• (Take 18-898G next semester for a deep dive)
![Page 17: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/17.jpg)
High-speed videography using CS
Key papersVeeraraghavan et al., Coded strobing, PAMI 2011Reddy et al., P2C2, CVPR 2011Hitomi et al., Coded exposure, ICCV 2011
![Page 18: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/18.jpg)
Image Formation Model
Low-speed captureworks well for static scenes
![Page 19: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/19.jpg)
High-speed scenes
open shut
33 ms
![Page 20: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/20.jpg)
High-speed scenes
Blurring in dynamic areas
High spatial resolution in static areas
![Page 21: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/21.jpg)
High speed scenes
Image credit: Boston.com
![Page 22: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/22.jpg)
Spatial Resolution = 1 MegapixelTemporal Resolution = 1 fpsBandwidth = 1 Megapixel/s
Spatio-Temporal Resolution Tradeoff
Single image
Slide credit: Mohit Gupta (Hitomi et al. 2011)
![Page 23: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/23.jpg)
Captured Thin-out Movie
(Row-wise sub-sampling)
Spatio-Temporal Resolution Tradeoff
Interpolated movie
Spatial Resolution = 1/4 MegapixelTemporal Resolution = 4 fpsBandwidth = 1 Megapixel/s
Slide credit: Mohit Gupta (Hitomi et al. 2011)
![Page 24: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/24.jpg)
Captured Thin-out Movie
(Row-wise sub-sampling)
Spatio-Temporal Resolution Tradeoff
Interpolated movie
Spatial Resolution = 1/36 MegapixelTemporal Resolution = 36 fpsBandwidth = 1 Megapixel/s
Slide credit: Mohit Gupta (Hitomi et al. 2011)
![Page 25: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/25.jpg)
Spatio-Temporal Resolution Tradeoff
High-speed, High-res Video
Challenges
1. Bandwidth of data
2. Light throughput
Slide credit: Mohit Gupta (Hitomi et al. 2011)
![Page 26: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/26.jpg)
From this photo …
![Page 27: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/27.jpg)
… to this one
Credit: Edgerton
![Page 28: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/28.jpg)
Idea 1: Multiplexing in Time
33 ms
![Page 29: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/29.jpg)
Idea 1: Multiplexing in Time
Optical coding
Benefits
1. Bandwidth of data remains the same
2. Light throughput is not significantly reduced
![Page 30: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/30.jpg)
Idea 1: Multiplexing in Time
Optical coding
Challenge:
More unknowns than measurements
How do we recover ?
Ay = x
![Page 31: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/31.jpg)
Idea 2: Signal Models
• Real-world signals are highly redundant
![Page 32: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/32.jpg)
Sparsity
N pixels
K < N
largewaveletcoefficients
(blue = 0)
Nwidebandsignalsamples
K < N
largeGabor (TF)coefficients
time
frequency
![Page 33: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/33.jpg)
Idea 2: Signal Models
• Real-world signals are highly redundant
• Models
– Sparse gradients
– Sparse in transform: Wavelets, Fourier
– Low rank: PCA, Union-of-subspaces
• Key idea: Constrain the solution space!
– Number of degrees of freedom significantly lesser than ambient dimensionality
![Page 34: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/34.jpg)
Periodic signals
Bottling line Toothbrush
Credit: Veeraraghavan et al, 2011
![Page 35: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/35.jpg)
High-speed Camera
0 fMax- fMax
Periodic signal has regularly spaced, sparse Fourier coefficients.
Is it necessary to use a high-speed video camera? Why waste bandwidth?
fP=1/P
2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t) – When each period of x has high frequency variations, Nyquist sampling rate is high.
P = 10ms Ts = 1/(2 fMax)
![Page 36: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/36.jpg)
Solving for the video
y
Camera observations at
a pixel
A x=
N unknowns
t
Coded Strobing
Frame 1
Frame M
Frame Integration Period TS
![Page 37: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/37.jpg)
Solving for the video
Non-zero elements
Fourier Basis
Basis Coeff
x = F s
b1 b2 bN
t
![Page 38: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/38.jpg)
Solving for the video
y F s= A
![Page 39: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/39.jpg)
Solving for the video
y F s= A
![Page 40: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/40.jpg)
Implementation
PGR Dragonfly2(25 fps)
FLC Shutter
Can flutter at 250us
![Page 41: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/41.jpg)
Toothbrush (simulation)
1000fps hi-speed camera
20fps normal camera 20fps coded strobing camera
Reconstructed frames
![Page 42: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/42.jpg)
Mill Tool
Normal Video: 25fps
Reconstructed Video at 2000fps
Mill tool rotating at 50Hz
Coded Strobing Video: 25fps
Mill tool rotating at 50Hz
Mill tool rotating at 50Hz
![Page 43: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/43.jpg)
Optical super-resolution
Key papersDuarte et al., Single pixel camera, SPM 2008Wang et al., LiSens, ICCP 2015Chen et al., FPA-CS, CVPR 2015
![Page 44: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/44.jpg)
Example
Video sensing in infrared
• Sensing in infra-red has applications in night-vision, astronomy, microscopy, etc.
• Materials that sense in certain infrared bands are extremely costly
– A 64 x 64 sensor costs upwards of USD 2000
– 1 Megapixel sensor costs
> USD 100k
Table courtesy of Gehm and Brady, Applied Optics, 2015
![Page 45: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/45.jpg)
Can we super-resolve a low-resolution sensor ?
• Spatial light modulation
– Introduce a high-resolution mask between scene and sensor
Digital micro-mirror
device
Photo-detector
![Page 46: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/46.jpg)
Single pixel camera
• Each pattern of micro-mirrors yield ONE compressive measurement
• A single photo-detector tuned to the wavelength of interest
• Resolution of the camera is that of the DMD, and not the sensor
with Kelly lab, Rice University
Digital micro-mirror
device
Photo-detector
![Page 47: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/47.jpg)
CS-MUVI on SPC
Single pixel camera setup
![Page 48: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/48.jpg)
CS-MUVI: IR spectrum
Joint work with Xu, Studer, Kelly, Baraniuk
InGaAs Photo-detector (Short-wave IR)
SPC sampling rate: 10,000 sample/s
Number of compressive measurements: M = 16,384
Recovered video: N = 128 x 128 x 61. Compression = 61x
Recovered Video
![Page 49: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/49.jpg)
Results
Final estimate (6 different videos)
• Real data acquired using a single pixel camera
• Sampling rate: 10,000 Hz
• Number of compressive measurements: 65536
• Total duration of data acquisition: 6 seconds
• Reconstructed video resolution: 128x128x256
![Page 50: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/50.jpg)
Motivation
SPC has very low measurement rate
DMD --- RDMD patterns/sec (typically, in 10s kHz)
ADC --- RADC samples/sec (typically, in 10s MHz)
Measurement rate of the SPC = min(RADC, RDMD)
objective lens
relay lens
~ ~ ~digital micromirror device(DMD)
photo-detector
ADC
![Page 51: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/51.jpg)
Parallel Compressive Imaging
• Use multiple pixels or a low-resolution sensor array
• How do we decide the specifications of the low-resolution sensor ?
– Number of pixels, geometry, etc …
objective lens
relay lens
~ ~ ~digital micromirror device(DMD)
Low-resolution
array ADC
![Page 52: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/52.jpg)
Measurement rate
# of pixels, F
Measure
ment
rate
RADC
RDMD SPC
Conventional sensor
F = 1 F = 106
![Page 53: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/53.jpg)
Measurement rate
# of pixels, F
Measure
ment
rate
RADC
RDMD SPC
F = 1 F = 106
Conventional sensor
![Page 54: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/54.jpg)
Optimal # of pixels
Foptimal is typically in 1000s of pixel for today’s DMDs and ADCs
Implications. Measurement rate of a conventional sensor but with a fraction of the number of pixels! (less than 0.1% pixels)
# of pixels, F
Measure
ment
rate RADC
Foptimal
![Page 55: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/55.jpg)
Two Prototypes
• Focal plane array-based CS (FPA-CS)– SWIR
– Map DMD onto a low-resolution 2D sensor
– Each pixel on the sensor observes a 2D patch of micromirrors on the DMD
• Line-sensor based Compressive Imager (LiSens)– Map DMD onto a line-array
sensor
– Each pixel on the sensor observes a line of micromirrors on the DMD
DMD
Low-resolution
sensor
DMD
Line-sensor
![Page 56: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/56.jpg)
FPA-CS
Objective Lens
DMD(1140x940)
64x64 SWIR Sensor
Relay optics
Relay optics
![Page 57: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/57.jpg)
FPA-CS Results
Scene (seen in a visible
camera)
Super-resolved image by FPA-CS
architecture
Image seen by 64x64 SWIR sensor
Chen et al. CVPR 2015
![Page 58: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/58.jpg)
objective lens
relay lens
digital micromirror device(DMD)
cylindrical lens
~ ~ ~
1. Use a linear array of pixels (a line-sensor)
2. Add a cylindrical lens
line sensor
Line-Sensor-based compressive camera (LiSens)
![Page 59: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/59.jpg)
Hardware prototype
DMD
SPC
objective lens
relay lens
cylindrical lensline-sensor
Measurement rate: 1 MHz
![Page 60: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/60.jpg)
![Page 61: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/61.jpg)
![Page 62: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/62.jpg)
Compressive Light Fields
Key Papers
• Marwah et al., Compressive coded apertures, SIGGRAPH 2013
• Tambe et al., Compressive LF videos, ICCV 2013
• Ito et al., Compressive epsilon photography, SIGGRAPH 2014
![Page 63: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/63.jpg)
Capture stack of photographs by varying camera parameters incrementally
Epsilon Photography
![Page 64: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/64.jpg)
Ex 1 - Epsilon photography applied to exposure
Slide credit: Verma and Mon-Ju. “High Dynamic Range Imaging”
Exposure Bracketting for HDR
![Page 65: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/65.jpg)
Ex 2 – Epsilon photography applied to focus
Slide credit: dpreview.com
Focus stack
![Page 66: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/66.jpg)
Ex 3 – Epsilon photography applied to aperture and focus
Hasinoff and Kutulakos, ECCV 2006
Confocal stereoPer-pixel depth estimation
![Page 67: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/67.jpg)
Confocal Stereo
Hasinoff and Kutulakos, ECCV 2006
Confocal stereoPer-pixel depth estimation
![Page 68: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/68.jpg)
Aperture Focus Images
focus
apert
ure
![Page 69: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/69.jpg)
Hasinoff and Kutulakos, ECCV 2006
![Page 70: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/70.jpg)
Pros and Cons
• Pros
– Per-pixel operations (for the most part)
• Cons
– Too many images
– Need texture (problem for everybody passive)
– Align ?
![Page 71: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/71.jpg)
Capture stack of photographs by varying camera parameters incrementally
Extremely slow!
Epsilon Photography
![Page 72: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/72.jpg)
1.5 1.75 2 2.5 3.5
0.44 0.5 0.6 0.8 1 1.5
Compressive Epsilon Photography
![Page 73: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/73.jpg)
1.5 1.75 2 2.5 3.5
0.44 0.5 0.6 0.8 1 1.5
Compressive Epsilon Photography
![Page 74: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/74.jpg)
Redundancies in Focus-Aperture Stacks
focus
apert
ure
![Page 75: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/75.jpg)
Redundancies in Focus-Aperture Stacks
focus
apert
ure
![Page 76: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/76.jpg)
Redundancies in Focus-Aperture Stacks
![Page 77: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/77.jpg)
Per-pixel models
• Key idea: Model intensity variations observed at an individual pixel
• Advantages
– No smoothing. Spatial resolution can be preserved
– Parallel recovery at each pixel
• Disadvantages
– Lack of spatial constraints
![Page 78: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/78.jpg)
Gaussian Mixture Models
focus
apert
ure
focus-aperture variations
Observation: Structure of EP intensity profiles tied to depth at a pixel
![Page 79: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/79.jpg)
Problem formulation
Given a few images captured with pre-selected parameters
+
per-pixel GMM of intensity variations
recover the entire epsilon photography intensity profile at each pixel.
Linear inverse problem
Lots of solvers
We use a maximum likelihood estimator
![Page 80: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/80.jpg)
Advantages of the GMM model
Figure 5: Greedy image selection. Comparison of PSNR as afunction of the number of acquired images for various methods ofsampling clearly shows the efficacy of our greedy sampling scheme.Note that the proposed scheme provides almost 10dB improvementover traditional sampling methods.
The MMSE estimator x(y) is the mean of the posterior f (x|y), i.e.,
x(y) =
K
k = 1
α( k )
(y)u( k )
x | y(y).
The corresponding MMSE is given by
MMSE(H ) = E ||x − x(y)||2
(2)
4.3 Greedy algorithm for image selection
Our goal is to reconstruct the focal and focus-aperture stacks byobserving a minimal number of images corresponding to certainchoices of camera focus and aperture settings. Thus, choice ofcamera parameters to observe is an important consideration1. Wepropose a greedy algorithm based on minimizing the MMSE (2).A tight analytic lower bound for the MMSE has been derived by[Flam et al. 2012][Flam et al. 2011] and we exploit this lower boundto derive a greedy sampling strategy. The lower bound of MMSE isgiven by:
MMSE(H ) =
K
k = 1
pk Tr C( k )
x | y, (3)
where C( k )
x | yis the posterior cluster Gaussian covariance (1).
Finding optimal camera parameters is a combinatorial problemsince we need to choose parameters from a pre-defined set of focus-aperture values. We instead rely on a greedy strategy that selectsone camera setting at a time that best minimizes the MMSE givenpreviously selected camera parameters. Suppose that there are atotal of N f focus settings and a total of Na aperture settings in agiven camera. If we are interested in capturing only m images cor-responding to m camera settings, then the brute-force version of the
algorithm will require evaluating the MMSE (3) for N f N a
mtimes,
which becomes practically impossible. Hence we device a greedyalgorithm. We first find the optimal pair of camera parameters, i.e.,
m = 2 by evaluating the MMSEN f N a
2times. Given this pair,
we then update the posterior covariance matrices Cx |y to take intoaccount the effect of the current selected camera parameters. Eachchoice of camera parameter correspond to a row hi in the H ma-
trix. After the i t h iteration, the posterior covariance is updated asfollows:
C( k )
x |y , i= C
( k )
x | y , i − 1− C
( k )
x |y , i − 1h
Ti (hi C
( k )
x | y , i − 1h
Ti + Cn )
− 1hi C
( k )
x |y , i − 1
1For focal stack, uniform sub-sampling of the focus axis is a good choice
as this corresponds to uniform sampling in scene depth. But for focus-
aperture stack, the choice of optimal camera parameters is not so obvious.
Figure 6: Geometric calibration for precise alignment. An inputnear-side focus image (a) is calibrated (b) so that the correspond-ing objects appear at the same locations in images captured witha different focus-aperture setting. Calibration is performed usinga farthest focus image (c) as reference. The figures in (d) clearlyshow that enlarged patches appearing at distinct depths are accu-rately aligned against the reference image after calibration. In (e),the trajectories of warping directions are shown which suggest thatthe images corresponding to near-side focus (a) are magnified incomparison with to far-side images (c) as can be seen clearly.
with the initial posterior covariances Ckx |y ,0 being the same as the
prior GMM covariances, C( k )x . After this covariance update step,
we find the next camera setting by evaluating the MMSE expres-sion with updated covariances. Note that from the second iteration,we need to evaluate the MMSE expression just N f Na times, whichprovides a significant reduction in computations. Figure 5 showshow the MMSE decreases as a function of the number of chosencamera parameters m . We can conclude that by capturing 8 im-ages with this prescribed camera settings, we should be a able toreconstruct the focus-aperture stack with N f = 45 and Na = 20.
Figure 5 shows a comparison of PSNR as a function of the num-ber of acquired images for (a) proposed sampling strategy, (b) fo-cus stack with large aperture (c) aperture stack with focus positionat mid point, (d) random sampling of focus-aperture pairs and (e)uniform sampling of focus-aperture pairs. Note that the proposedscheme provides almost 10 dB improvement over all traditionalsampling methods. This result indicates, that even when the goalis not complete post-capture control, but rather traditional focal oraperture stacking (say for depth estimation), our optimized sam-pling strategy is significantly better.
5 Geometric and photometric calibration
Since we employ a per-pixel based model for learning and recon-struction, precise geometric and photometric calibration with sub-pixel level accuracy is essential. For this purpose, we adopt theprocedure in [Hasinoff and Kutulakos 2009].
Geometric Calibration. It is a well known fact that changes in fo-cal settings result in a non-linear warp of the objects in the scene. In[Hasinoff and Kutulakos 2009], it was shown that this warp can beaccurately modeled by considering parameters for image magnifi-cation, lens distortion and translation. For estimating these param-eters, we collected images of a calibration chart containing blackdots on a grid at the largest aperture (F/1.4). Registration amongimages in the dataset was realized by unwarping the images ac-cording to the estimated parameters. Figure 6 shows an example ofgeometric calibration achieving precise alignment.
Photometric Calibration. Modifying the aperture causes a changenot only in depth of field but also results in vignetting. This vi-
Analytical bounds on performance.
Can greedily pre-select camera parameters that maximize average reconstruction performance
![Page 81: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/81.jpg)
Advantages of the GMM model
Small aperture leads to large DOF and provides textural cues
Large aperture leads to small DOF and provides depth cues
![Page 82: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/82.jpg)
Chess
![Page 83: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/83.jpg)
Fluffy
![Page 84: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/84.jpg)
Animals
![Page 85: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/85.jpg)
CS Summary
• Three questions
– Is sensing costly ? (how? )
– Is there a sparsifying/parsimonious representation ?
– Acquire some sort of randomized measurements ?
![Page 86: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/86.jpg)
A simple case study: MRI
Lustig et al., 2008
MRI obtains samples in Fourier space
Taking lesser samples
==higher speed of operation, less time etc.
![Page 87: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/87.jpg)
MRI
without a signal model
From 10 times lesser number of measurements
Lustig et al., 2008
![Page 88: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/88.jpg)
MRI + CS
with signal model
From 10 times lesser number of measurements
The recovery is exact, provided some conditions are satisfied
Lustig et al., 2008
![Page 89: Compressive Imaging - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture25.pdf · Slide credit: Mohit Gupta (Hitomi et al. 2011) Captured Thin-out Movie](https://reader034.fdocuments.in/reader034/viewer/2022051912/6002f28c005e896b3b6cacc7/html5/thumbnails/89.jpg)
Summary
• CS provides the ability to sense from far-fewer measurements than the signal’s dimensionality
• Implications
– Fewer pixels on the sensor
– Shorter acquisition time
– Slower rate of acquisition