ADMINISTRIVIA SENSORayan/courses/cse559a/PDFs/lec... · 2019. 8. 29. · Install Anaconda, LaTeX,...
Transcript of ADMINISTRIVIA SENSORayan/courses/cse559a/PDFs/lec... · 2019. 8. 29. · Install Anaconda, LaTeX,...
CSE 559A: Computer Vision
Fall 2020: T-R: 11:30-12:50pm @ Wrighton 300 / Zoom
Instructor: Ayan Chakrabarti ([email protected]).Course Staff: Adith Boloor, Patrick Williams
Sep 17, 2020
http://www.cse.wustl.edu/~ayan/courses/cse559a/
1
ADMINISTRIVIAADMINISTRIVIAAll information @ http://www.cse.wustl.edu/~ayan/courses/cse559a/
Read through course websiteSyllabusProblem Set and Resource SectionLate PoliciesCollaboration & Academic Honesty Policies
Join PiazzaReview Pre-req SlidesInstall Anaconda, LaTeX, GitSubmit your Public KeySubmit Problem Set 0Post on piazza if you have issues with your setup.
2
THE PINHOLE CAMERATHE PINHOLE CAMERA
3
THE PINHOLE CAMERATHE PINHOLE CAMERA
4
THE PINHOLE CAMERATHE PINHOLE CAMERA
5
THE PINHOLE CAMERATHE PINHOLE CAMERA
6
THE PINHOLE CAMERATHE PINHOLE CAMERA
7
THE PINHOLE CAMERATHE PINHOLE CAMERA
8
THE PINHOLE CAMERATHE PINHOLE CAMERA
9
THE PINHOLE CAMERATHE PINHOLE CAMERA
10
THE PINHOLE CAMERATHE PINHOLE CAMERA
11
THE PINHOLE CAMERATHE PINHOLE CAMERA
12
THE PINHOLE CAMERATHE PINHOLE CAMERA
13
THE PINHOLE CAMERATHE PINHOLE CAMERA
14
THE PINHOLE CAMERATHE PINHOLE CAMERA
15
THE PINHOLE CAMERATHE PINHOLE CAMERA
16
THE PINHOLE CAMERATHE PINHOLE CAMERA
17
THE PINHOLE CAMERATHE PINHOLE CAMERA
18
THE PINHOLE CAMERATHE PINHOLE CAMERA
19
THE PINHOLE CAMERATHE PINHOLE CAMERA
20
THE PINHOLE CAMERATHE PINHOLE CAMERA
21
THE PINHOLE CAMERATHE PINHOLE CAMERA
22
SENSORSENSOR
: Light energy, per unit area per unit time, arriving at point at time
Here, are real numbers (in meters) denoting actual position on the sensor plane.
: Intensity measured by the sensor element at grid location
Here, , are integers, indexing pixel location.
Note the convention for “indexing”, while for because it is a function.
: is a sensitivity function for a pixel, assuming (0,0) is the center of the pixel.
is ideally 1 inside pixel (for example, within [-W/2,-H/2] to [W/2,H/2)), and 0 outside. But may haveattenuation at boundaries.
is the location (in meters) of the center of the sensor element
is the sensitivity function for the pixel at , “centered” at .
E(x, y, t) (x, y) t
x, y
I[ , ]nx ny ,nx ny
nx ny
[] () E
p(x, y)
p(⋅, ⋅)
,xnx yny
p(x − , y − )xnx yny [ , ]nx ny ( , )xnx xny
23
SENSORSENSOR: Light energy, per unit area per unit time, arriving at point at time
Here, are real numbers (in meters) denoting actual position on the sensor plane.
: Intensity measured by the sensor element at grid location
Here, , are integers, indexing pixel location.
Note the convention for “indexing”, while for because it is a function.
: is a sensitivity function for a pixel, assuming (0,0) is the center of the pixel.
is ideally 1 inside pixel (for example, within [-W/2,-H/2] to [W/2,H/2)), and 0 outside. But may haveattenuation at boundaries.
is the location (in meters) of the center of the sensor element
is the sensitivity function for the pixel at , “centered” at .
Defining as the “quantum efficiency” of the sensor: Ratio of Light Energy to Charge/Voltage
Rate at which charge/voltage increases in sensor element at time .
An image capture involves “exposing” the image for an interval (seconds).
So the total intensity is going to involve integrating the charge/voltage rate over that interval.
E(x, y, t) (x, y) t
x, y
I[ , ]nx ny ,nx ny
nx ny
[] () E
p(x, y)
p(⋅, ⋅)
,xnx yny
p(x − , y − )xnx yny [ , ]nx ny ( , )xnx xny
q
∫ E(x, y, t) p(x − , y − ) q dx dyxnx yny,nx ny t
T
24
SENSORSENSOR
are integers indexing pixels in image array.
is spatial location
is recorded pixel intensity.
is light “power” per unit area incident at location on the sensor plane at time
is the “center” spatial location of the pixel / sensor element at .
is spatial sensitivity of the sensor (might be lower near boundaries, etc.)
is quantum efficiency of the sensor (photons/energy to charge/voltage)
is the duration of the exposure interval.
CCD/CMOS sensors measure total energy or “count photons” that arrived during exposure.
I[ , ] = [∫ E(x, y, t) p(x − , y − ) q dx dy]dtnx ny ∫T
t=0
xnx yny
,nx ny
(x, y)
I[ , ]nx ny
E(x, y, t) (x, y) t
( , )xnx yny [ , ]nx ny
p(x, y)
q
T
25
SENSORSENSOR
I[ , ] = [∫ E(x, y, t) p(x − , y − ) q dx dy]dtnx ny ∫T
t=0
xnx yny
is the ideal unquantized pixel intensityI[ , ]nx ny
26
SENSORSENSOR
[ , ] = [∫ E(x, y, t) p(x − , y − ) q dx dy]dtI 0 nx ny ∫T
t=0
xnx yny
I ← I 0
27
SENSORSENSOR
SHOT NOISESHOT NOISE
Caused by uncertainty in photon arrival
Actual number of photons is a discrete random variable with Poisson distribution
is the “expected” number of photons. In our case,
[ , ] = [∫ E(x, y, t) p(x − , y − ) q dx dy]dtI 0 nx ny ∫T
t=0
xnx yny
I ← I 0
K
P(K = k) = λke−λ
k!
λ ∝ I 0
28
SENSORSENSOR
SHOT NOISESHOT NOISE
Property of Poisson distribution: Mean and Variance both equalO�en, shot noise is modeled with additive Gaussian noise with “signal dependent” variance:
where (Gaussian random noise with mean 0, variance 1).
[ , ] = [∫ E(x, y, t) p(x − , y − ) q dx dy]dtI 0 nx ny ∫T
t=0
xnx yny
I ← I 0
I ← + I 0 I 0‾‾√ ϵ1
ϵ ∼ (0, 1)
I ∼ ( , )I 0 I 0
29
SENSORSENSOR
AMPLIFICATION & ADDITIVE NOISEAMPLIFICATION & ADDITIVE NOISE
Signal amplified by gain before digitization. Based on ISO (higher for higher ISO).
Some signal-independent Gaussian noise added before and a�er amplification.
where and are parameters (lower for high quality sensors),and are noise variables, all independent.
[ , ] = [∫ E(x, y, t) p(x − , y − ) q dx dy]dtI 0 nx ny ∫T
t=0
xnx yny
I ← + I 0 I 0‾‾√ ϵ1
g g
I ← g × ( + + ) +I 0 I 0‾‾√ ϵ1 σ2aϵ2a σ2bϵ2b
σ2a σ2b
, ,ϵ1 ϵ2a ϵ2b (0, 1)
I ← + +gI 0
⏟Amplified Signal
g I 0‾‾√ ϵ1
Amplified Shot Noise
( + )g2σ2
2a σ2
2b‾ ‾‾‾‾‾‾‾‾‾‾‾‾√ ϵ2
Amplified and un-amplified additive noise
30
SENSORSENSOR
DIGITIZATIONDIGITIZATION
Final step is rounding and clipping (by an analog to digital converter)
[ , ] = [∫ E(x, y, t) p(x − , y − ) q dx dy]dtI 0 nx ny ∫T
t=0
xnx yny
I ← g + g + I 0 I 0‾‾√ ϵ1 ( + )g2σ2
2a σ2
2b‾ ‾‾‾‾‾‾‾‾‾‾‾‾√ ϵ2
I ← Round(g + g + )I 0 I 0‾‾√ ϵ1 ( + )g2σ2
2a σ2
2b‾ ‾‾‾‾‾‾‾‾‾‾‾‾√ ϵ2
I = min( , Round(g + g + ))Imax I 0 I 0‾‾√ ϵ1 ( + )g2σ2
2a σ2
2b‾ ‾‾‾‾‾‾‾‾‾‾‾‾√ ϵ2
31
SENSORSENSOR
ignoring sensor saturation, dark current, …
WHY STUDY THIS ?WHY STUDY THIS ?
To understand the degradation process of noise (if we want to denoise / recover from ).
To prevent degradation during capture, because we control exposure time and ISO / gain .
To understand the different trade-offs for loss of information from noise, rounding, and clipping.
[ , ] = [∫ E(x, y, t) p(x − , y − ) q dx dy]dtI 0 nx ny ∫T
t=0
xnx yny
I = min( , Round(g + g + ))Imax I 0 I 0‾‾√ ϵ1 ( + )g2σ2
2a σ2
2b‾ ‾‾‾‾‾‾‾‾‾‾‾‾√ ϵ2
I 0 I
T g
32
SENSORSENSOR
ROUNDING VS CLIPPINGROUNDING VS CLIPPING
Ignoring noise, what is the optimal for a given ?
Keep low so that most values of are below .
But if is too low, a lot of the variation will get rounded to the same value.
[ , ] = [∫ E(x, y, t) p(x − , y − ) q dx dy]dtI 0 nx ny ∫T
t=0
xnx yny
I = min( , Round(g + g + ))Imax I 0 I 0‾‾√ ϵ1 ( + )g2σ2
2a σ2
2b‾ ‾‾‾‾‾‾‾‾‾‾‾‾√ ϵ2
g [ , ]I 0 nx ny
g g [ , ]I 0 nx ny Imax
g
33
SENSORSENSOR
34
SENSORSENSOR
35
SENSORSENSOR
36
SENSORSENSOR
37
SENSORSENSOR
Note that here, our ‘ideal’ intensity is , everything else is noise.
LIGHT VS AMPLIFICATIONLIGHT VS AMPLIFICATION
Say we have chosen the optimal target values for the product . Is it better:
To have a higher and lower magnitude
To have a lower and higher magnitude
Depends, based on
[ , ] = [∫ E(x, y, t) p(x − , y − ) q dx dy]dtI 0 nx ny ∫T
t=0
xnx yny
I = min( , Round(g + g + ))Imax I 0 I 0‾‾√ ϵ1 ( + )g2σ2
2a σ2
2b‾ ‾‾‾‾‾‾‾‾‾‾‾‾√ ϵ2
gI 0
gI 0
g I 0
g I 0
,σ2a σ2b
38
SENSORSENSOR
Note that here, our ‘ideal’ intensity is , everything else is noise.
LIGHT VS AMPLIFICATIONLIGHT VS AMPLIFICATION
Say we have chosen the optimal target values for the product . Is it better:
To have a higher and lower magnitude
To have a lower and higher magnitude
Depends, based on
Additional Reading (if interested):S. Hasinoff, F. Durand, W.T. Freeman, “Noise-Optimal Capture for High Dynamic Range Photography,” CVPR 2010.
[ , ] = [∫ E(x, y, t) p(x − , y − ) q dx dy]dtI 0 nx ny ∫T
t=0
xnx yny
I = min( , Round(g + g + ))Imax I 0 I 0‾‾√ ϵ1 ( + )g2σ2
2a σ2
2b‾ ‾‾‾‾‾‾‾‾‾‾‾‾√ ϵ2
gI 0
gI 0
g I 0
g I0
,σ2a σ2b
39
SENSORSENSOR
So how do we increase ?
Better sensors (higher )
Larger sensor elements: over a larger area.
But we’ve gone the other way: cameras stuff more ‘megapixels’ in smaller form factors.
[ , ] = [∫ E(x, y, t) p(x − , y − ) q dx dy]dtI 0 nx ny ∫T
t=0
xnx yny
I 0
q
p(⋅, ⋅) > 0
40
SENSORSENSOR
Increase exposure time ?
If scene is static and camera is stationary:
doesn’t change with
If scene is moving …
[ , ] = [∫ E(x, y, t) p(x − , y − ) q dx dy]dtI 0 nx ny ∫T
t=0
xnx yny
T
E(x, y, t) t ⇒ ∝ TI 0
41
SENSORSENSOR
Increase itself. How ?
Take pictures outdoors, or under brighter lights.Don’t use a pinhole camera !
[ , ] = [∫ E(x, y, t) p(x − , y − ) q dx dy]dtI 0 nx ny ∫T
t=0
xnx yny
E(x, y, t)
42
LENSESLENSES
43
LENSESLENSES
44
LENSESLENSES
45
LENSESLENSES
46
LENSESLENSES
47
LENSESLENSES
48
LENSESLENSES
49
LENSESLENSES
50
LENSESLENSES
51
LENSESLENSES
52
LENSESLENSES
53
LENSESLENSES
54
LENSESLENSES
55
TRADEOFFSTRADEOFFSPhotographers think about these tradeoffs every time they take a shot
Dynamic range and what part of the image should be well exposed (rounding and clipping)
Choosing between:ISO i.e. Gain & noiseExposure Time & motion blurF-stop i.e. aperture size & defocus blur
56
COLORCOLOR
We le� out at an important term in this equation: wavelength
[ , ] = [∫ E(x, y, t) p(x − , y − ) q dx dy]dtI 0 nx ny ∫T
t=0
xnx yny
57
COLORCOLOR
Light carries different amounts of power in different wavelengths
now refers to power per unit area per unit wavelength
In wavlength , incident at at time
Both spectral and spatial density function
: Quantum efficiency also a function of wavelength
CMOS/CCD sensors are sensitive (have high ) across most of the visible spectrum
Actually extend to longer than visible wavelengths (near infra red)Why cameras have NIR filter, to prevent NIR radiation from being ‘superimposed’ on the image
Q: But this measures ‘total’ power in all wavelengths. How do we measure color ?
Ans: By putting a color filter in front of each sensor element.
[ , ] = [∫ E(λ, x, y, t) p(x − , y − ) q(λ) dλ dx dy]dtI 0 nx ny ∫T
t=0
xnx yny
E(λ, x, y, t)
λ (x, y) t
q(λ)
q
58
COLORCOLOR
is the transmittance of a color filter for color channel
E.g, will transmit power in (be high for) wavelengths in the red part of the visible spectrum and attenuate power in (be low for) other wavelengths.Sometimes also called “color matching functions”
[ , , c] = [∫ E(λ, x, y, t) p(x − , y − ) (λ) q(λ) dλ dx dy]dtI 0 nx ny ∫T
t=0
xnx yny Πc
for c ∈ {R,G,B}
Πc c
ΠR
(actually XYZ not RGB)
59
COLORCOLOR
But we can only put one filter in front of each sensor element / pixel location.So color cameras “multiplex” color measurements: they measure a different color channel at each location.Usually in an alternating pattern called the Bayer pattern:
Note: a disadvantage is that color filters block light, so measured values are lower.
That’s why black and white / grayscale cameras are “faster” than color cameras.
[ , , c] = [∫ E(λ, x, y, t) p(x − , y − ) (λ) q(λ) dλ dx dy]dtI 0 nx ny ∫T
t=0
xnx yny Πc
for c ∈ {R,G,B}
I 0
60
FINAL STEPSFINAL STEPSFinal steps in camera processing pipelines (except for some DSLR cameras shooting in RAW):
Filter Colors to Standard RGB:
Cameras o�en use their own color filters .
Apply a linear transformation to map those measurements to standard RGB.White-balance: scale color channels to remove color cast from a non-neutral illuminant.Tone-mapping:
The simplest form is “gamma correction” (approximately raising each intensity to the power )
Done based on standard developed for what old display devices expectedFits the full set of measurable colors into the gamut that can be displayed / printedModern cameras o�en do more advanced processing (to make colors look vibrant)
Compression
And that’s how you get your PNG / JPEG images !
Optional Additional Reading: Szeliski Sec 2.3
Πc
(1/2.2)
61