EE 7700
description
Transcript of EE 7700
EE 7700
Demosaicking Problem in Digital Cameras
Bahadir K. Gunturk 2
Multi-Chip Digital Camera
Lens
Scene
Spectral
filters
Beam-splitter
s
Sensors
To produce a color image, at least three spectral components are needed at each pixel.
One approach is to use beam-splitters and multiple chips.
Bahadir K. Gunturk 3
Single-Chip Digital Camera Multi-chip approach is expensive. Precise chip alignment is
required. The alternative is to use a color filter array.
Lens
Scene
Color filter array
Sensors
Bahadir K. Gunturk 4
Single-Chip Digital Camera The missing color samples must be estimated to produce
the full color image. Since a mosaic of samples are available, this estimation
(interpolation) process is called demosaicking.
Bahadir K. Gunturk 5
Single-Chip Digital Camera Images suffer from color artifacts when the samples are not
estimated correctly.
Original image Bilinearly interpolated from CFA-filtered
samples
Bahadir K. Gunturk 6
Demosaicking Approaches Non-Adaptive Single-Channel Interpolation: Interpolate
each color channel separately using a standard technique, such as nearest-neighbor interpolation, bilinear interpolation, etc.
Edge-Directed Interpolation: Estimate potential edges, avoid interpolating across the edges.
1 2
3
4
x
Edge-directed interpolation 1.Calculate horizontal gradient ΔH = |G1 – G2| 2.Calculate vertical gradient ΔV = |G3 – G4|3.If ΔH > ΔV,Gx = (G3 + G4)/2 Else if ΔH < ΔV, Gx = (G1 + G2)/2 Else Gx = (G1 + G2 + G3 + G4)/4
Bahadir K. Gunturk 7
Demosaicking Approaches Edge-Directed Interpolation: Based on the assumption that
color channels have similar texture, various edge detectors can be used.
Edge-directed interpolation 1. Calculate horizontal gradient ΔH = | (R3 + R7)/2 – R5 | 2. Calculate vertical gradient ΔV = | (R1 + R9)/2 – R5 | 3. If ΔH > ΔV,
G5 = (G2 + G8)/2 Else if ΔH < ΔV, G5 = (G4 + G6)/2 Else G5 = (G2 + G8 + G4 + G6)/4
1
2
4 5 6 7
8
9
3
Bahadir K. Gunturk 8
Demosaicking Approaches Constant-Hue-Based Interpolation: Hue does not change
abruptly within a small neighborhood. Interpolate green channel first. Interpolate hue (defined as either color differences or color
ratios). Estimate the missing (red/blue) from the interpolated hue.
Red Interpolated Red
InterpolateGreen
Interpolate
Bahadir K. Gunturk 9
Demosaicking Approaches Edge-Directed Interpolation of Hue: It is a combination of
edge-directed interpolation and constant-hue-based interpolation. Hue is interpolated as in constant-hue-based interpolation approach, but this time, hue is interpolated based on the edge directions (as in the edge-directed interpolation algorithm).
Bahadir K. Gunturk 10
Demosaicking Approaches Using Laplacian For Enhancement: Use the second-order
gradients of red/blue channels to enhance green channel.
1
2
4 5 6 7
8
9
3
1. Calculate horizontal gradient ΔH = |G4 – G6| + |R5 – R3 + R5 – R7| 2. Calculate vertical gradient ΔV = |G2 – G8| + |R5 – R1 + R5 – R9| 3. If ΔH > ΔV,
G5 = (G2 + G8)/2 + (R5 – R1 + R5 – R9)/4Else if ΔH < ΔV,
G5 = (G4 + G6)/2 + (R5 – R3 + R5 – R7)/4 Else
G5 = (G2 + G8 + G4 + G6)/4 + (R5 – R1 + R5 – R9 + R5 – R3 + R5 – R7)/8
Bahadir K. Gunturk 11
Aliasing
mf 1f
2f
2f
1f
2f
1f
Green channel Red/Blue channel
Frequency spectrum of an image:
After CFA sampling:
Bahadir K. Gunturk 12
Demosaicking Approach Alias Cancelling: Based on the assumption that red, green,
and blue channels have similar frequency components, the high-frequency components of red and blue channels are replaced by the high-frequency components of green channel.
2f
1f
Red/Blue channel
Bahadir K. Gunturk 13
Experiment
Full Red/Green/Blue
channels
Subband decomposition
CFA Sampling
Subband decomposition
Interpolate
LL
HL
HH
LH
LLLL
HLHL
LHLH
LL
HL
HH
LH
LLLL
HLHL
LHLH
Bahadir K. Gunturk 14
Constraint Sets
Detail Constraint Set: Detail subbands of the red and blue channels must be similar to the detail subbands of the green channel.
HL
HH
LH
HL
LH
HH
HLRHLG
1 2 1 2 1 2 1 2( , ) : ( , ) ( , ) ( , ),
, ,k k
d
R n n R n n G n n T n nC
for k HL LH HH
Bahadir K. Gunturk 15
Constraint Sets
Observation Constraint Set: Interpolated channels must be consistent with the observed data.
CFASensors
1 2( , )O n n
R
1 2 1 2 1 2 1 2( , ) : ( , ) ( , ), ( , )o RC R n n R n n O n n n n
Bahadir K. Gunturk 16
HL
LH
HH
Projection Operations
Projection onto the Detail Constraint Set:
Decompose the color channels.
Update the detail subbands of red and blue channels.
1 2( , )HLG n n
1 2 1 2( , ) ( , )HLG n n T n n
Apply synthesis filters to reconstruct back the channels.
1 2( , )HLR n n
Bahadir K. Gunturk 17
Projection Operations
Projection onto the Observation Constraint Set:
Insert the observed data to their corresponding positions.
CFASensors
1 2( , )O n n
Bahadir K. Gunturk 18
Alternating Projections Algorithm
Samples of color channels
Initial interpolation
0h
1h
0g
1gUpdate
Insert the observed data
Projection onto the detail constraint set
Projection onto the observation constraint set
Iteration
Bahadir K. Gunturk 19
Results
Original Hibbard 1995 Laroche and Prescott 1994
Hamilton and Adams 1997 Kimmel 1999 Gunturk 2002
Bahadir K. Gunturk 20
Results
OriginalHibbard
1995
Laroche and
Prescott 1994
Hamilton and
Adams 1997
Kimmel 1999
Gunturk 2002
Bahadir K. Gunturk 21
Previous Methods
Gunturk et al, “Demosaicking: Color Filter Array Interpolation in Single-Chip Digital Cameras,” to appear in IEEE Signal Processing Magazine.
[Gunturk02]
Bahadir K. Gunturk 22
References
[Gunturk02] Gunturk et al, “Color Plane Interpolation Using Alternating Projections,” IEEE Trans. Image Processing, 2002.
[Hibbard 1995] R. H. Hibbard, “Apparatus and method for adaptively interpolating a full color image utilizing luminance gradients,” U.S. Patent 5,382,976, January, 1995.
[Laroche and Prescott 1994] C. A. Laroche and M. A. Prescott, “Apparatus and method for adaptively interpolating a full color image utilizing chrominance gradients,” U.S. Patent 5,373,322, December, 1994.
[Hamilton and Adams 1997] J. F. Hamilton Jr. and J. E. Adams, “Adaptive color plane interpolation in single sensor color electronic camera,” U.S. Patent 5,629,734, May, 1997.
[Kimmel 1999] R. Kimmel, “Demosaicing: Image reconstruction from CCD samples,” IEEE Trans. Image Processing, vol. 8, pp. 1221-1228, 1999.