Paper Camera Demosaicing

8/18/2019 Paper Camera Demosaicing

http://slidepdf.com/reader/full/paper-camera-demosaicing 1/6

(a) (b) (c) Fig. 1. (a) Bayer CFA pattern, (b) Example of a mosaic image, (c) Full

color image reconstructed from (b).

Compact and Efficient Algorithm for Color

Demosaicing

Tien Ho-Phuoc

The University of Danang - University of Science andTechnology, Department of Electronics and

Telecommunications Engineering

Dung-Nghi Truong Cong

Ho Chi Minh City University of Technology, Faculty ofElectrical and Electronics Engineering

Abstract — The present paper proposes a new method for

color image demosaicing. This method separates luminance from

chrominance and allows luminance to be extracted directly from

known pixel values. Since interpolation for missing pixels is

applied only to chrominance channels, it can reduce artifacts.

Moreover, in chrominance channels the proposed method tries to

interpolate pixels along edges. As consequence, edges are better

preserved in the demosaiced image and false colors, which are a

popular problem in demosaicing, are correctly eliminated.

Comparisons with other algorithms show satisfying

performances of our method in both qualitative and quantitative

criteria. Another advantage of the proposed method relates to its

low computational complexity, which can facilitate hardware

implementation.

I. I NTRODUCTION

Digital cameras use a Color Filter Array (CFA) pattern

(Fig. 1a) to sense only one color at each pixel location. The raw

image generated from the CFA pattern is called mosaic image

(Fig. 1b). Given the latter, the full color image (Fig. 1c), i.e.

three colors per pixel, is reconstructed by interpolating two

missing colors at each pixel location. The process to recover a

full color image from a mosaic one is known as demosaicing;this step has crucial effect on image quality of a digital camera.

Among various CFA patterns, the one proposed by Bayer [1] is

the most popular and has been widely used in the digital

camera industry. In this paper we will only consider the BayerCFA pattern.

Since about four decades a broad spectrum of demosaicing

algorithms have been proposed in the literature: they span from

low to high complexity [2-4]. Most algorithms in the first

category treat the mosaic image in the spatial domain. At the

beginning, bilinear interpolation was used thanks to its simplehardware implementation. The main drawback of this method

is blurring and false colors in the reconstructed image. Toimprove the quality of interpolation, some authors proposed to

interpolate the color plans based on direction or edge in an

image. The idea is to use pixels along rather than across edges

for interpolation. Such algorithms have been subject to several

patents [5, 6].

Another demosaicing approach by Alleysson is to analyze

the mosaic image in the frequency domain [7]. This method

simulates a characteristic of the human visual system that is the

separation between luminance and chrominance and helps to

reduce color artifacts. It is important to note that although

Alleysson’s method is analyzed in the frequency domain, its

implementation is carried out in the spatial domain and is also

computationally efficient. Nevertheless, the main problem of

this method, as well as in many other algorithms, is that in

highly detailed regions or salient edges it tends to fail in

reconstructing true colors.

The second category of demosaicing algorithms often tries

to reconstruct a full color image through solving an

optimization problem using techniques such as Alternating

Projections [8, 9], Nonlocal Adaptive Thresholding [10],Minimized-Laplacian Residual Interpolation [11], Inter-Color

Correlation [12], or Compressive Sensing [13]. Noisy imagedemosaicing has just been examined in [14]. These methods

may produce better performance but with highly computational

complexity.

Yet, in order to be implementable on image sensors a

demosaicing algorithm usually must show both high

performance and low complexity due to limited hardwareresources. Aiming at such algorithm, in this paper, we will

focus mainly on low complexity but effective performance

demosaicing methods. We propose a compact method that is

able to overcome often encountered artifacts in demosaicing

such as blurring and false colors. In particular, we will exploit

the separation between luminance and chrominance as in [7] to

extract luminance entirely from known pixel values. Moreover,

in chrominance channels we use edge-preserving interpolation

to obtain true colors along edges or in textured regions.

The remaining of this paper is described as follows. In

section II, several effective demosaicing methods are revisited.

In section III, we introduce our proposed method to improvedemosaicing performance. Experiment and results will be

presented in section IV. Finally, section V summaries some



(a) (b)

Fig. 2. CFA patterns used for computation of the G plan.

conclusions and perspective.

II. REVIEW OF DEMOSAICING ALGORITHMS

We will review some popular algorithms used for the

demosaicing problem. Some of them are quite simple and

serve as baseline methods, others provide much interest in

industry. As mentioned earlier, low complexity algorithms

attract our attention in this section.

A. Bilinear Interpolation

Bilinear interpolation deals with each layer (R, G, or B)separately by calculating missing pixels using linearinterpolation on the available ones. Consequently, each layercan be reconstructed using linear convolution with thecorresponding mask in Eq. 1.

(1)

It is worth noting that the same mask is used for the R andB plans thanks to their identical mosaic structure. Although this

method is very computationally efficient, it tends to generate blurring and color artifacts, due to the used low-pass filter andthe fact that correlation between the three plans is not takeninto account.

B. Edge-Based Interpolation

The main idea of edge-based demosaicing methods is totake into account the edge direction at each pixel to reduceinterpolation error. Thus, it is required to interpolate pixelsalong rather than across the edge. Besides, since the CFA

pattern has more G pixels than R or B pixels, the G plan is preferred to be first reconstructed. Once the complete G planis obtained, the R and B plans are interpolated thanks to theircolor correlation with the G plan.

Various edge-based interpolation algorithms have been proposed. Their difference mostly concerns the Ginterpolation step, i.e. variation estimation and G valuecomputation at the missing G pixels. It is worth noting that theG interpolation step is the most important and determines the

performance of an edge-based demosaicing algorithm. In thefollowing we will review some methods estimating the G plan.

The simplest method of this category uses only G values torecover the complete G plan in two steps. First, the horizontaland vertical variations are estimated as | | and | | when we want to reconstruct the value of G5in Fig. 2a. Second, G5 is interpolated along the direction withless variation as in Eq. 2.

{

In [5], the horizontal and vertical variations are estimated by the second order derivative in a larger region and with R orB values rather than G values themselves. Concretely, in order

to compute the missing G5 value in Fig. 2a, these variationsare estimated as | | and | | Then, G5 is determined in the same wayas above (Eq. 2).

Hamilton and Adams [6] proposed to use the Laplacian(second order derivative) for the R and B pixels to correct thesimple average interpolation for the G values. The advantage

of this correction is to reduce aliasing in the final G plan. Thehorizontal and vertical variations are hence computed as | | | | and | | | | for the center pixel in Fig. 2a.The value of G5 is determined as:

{

The horizontal and vertical variations and can also be computed in a more sophisticated way as in [15]. In thismethod, variation at a missing G pixel is estimated using notonly the current horizontal or vertical lines but alsoneighboring lines. For example, to compute in Fig. 2b,the horizontal variation is computed as in Eq. 4. The verticalvariation can be estimated in the same manner.

∑ ∑

∑ ∑ ∑

∑

After the G plan is completed, the reconstruction of thetwo other plans will be carried out based on the assumptionthat color difference (or color ratio) in an object is constant[2]. This technique is widely used in the literature. In



particular, to reconstruct the R plan, we compute thedifference R-G at existing R positions. Remember that Gvalues are now available everywhere. Then the difference R-Gis interpolated, for example, using bilinear interpolation.Finally, the complete R plan is obtained by adding G to theabove difference. The B plan is recovered in the same way.

In section IV, the methods in [5], [6], and [15] – calledEdge1, Edge2, and Edge3 respectively – combined with theconstant-color-difference-based interpolation will be retainedfor testing.

C. Luminance-Chrominance Separation

In [7] Alleysson proposed a very interesting idea fordemosaicing. Inspired by the human visual system in whichvisual stimulus is divided into luminance and twochrominance pathways, Alleysson also separates luminanceand chrominance in the demosaicing process. First, luminanceis estimated using the following filter:

Second, luminance is subtracted from each color plan (R,G, or B) at corresponding existing positions. For each plan, theresulting difference – in fact, opponent chrominance – is theninterpolated using bilinear interpolation with the masksdescribed in Eq. 1. Finally, the complete plan for R, G or B isobtained by adding the above luminance.

Advantage of Alleysson’s algorithm is that it shows clearlythe separation of luminance and chrominance in the frequencydomain and, hence, explains easily phenomena oftenencountered in demosaicing such as false color, blurring, andaliasing. Readers who are interested in this method can find

more detailed mathematical explanations in [7].

III. PROPOSED ALGORITHM

Alleysson’s method allows us to extract luminancedirectly, i.e. without interpolation, from the mosaic image.Interpolation is applied only to opponent chrominance.Interestingly, human vision is less sensitive to highfrequencies of chrominance. Hence, simple interpolation – e.g.

bilinear interpolation – of chrominance channels generally isenough to provide satisfying performance. Nevertheless, foran image in which high frequency components are prominent,if chrominance is not well interpolated false colors may appearin the demosaiced image. While Alleysson’s method uses

bilinear interpolation, which is not good at edges, it maygenerate false colors along edges (see more in Fig. 4).

In the present paper we extend Alleysson’s work [7] to better treat chrominance channels. On the one hand, we extractluminance directly from all known pixels of a mosaic image;i.e. we do not need interpolation for the G plan and, therefore,reduce interpolation error for luminance (or G) estimation asin edge-based methods. On the other hand, we propose anotherway to interpolate chrominance channels to obtain true colorsalong edges or in textured regions. Our proposed method isdescribed as follows.

From a 2D mosaic image (Fig. 1b) its luminance is extracted with the mask in Eq. 5. The multiplexedchrominance is then obtained by subtracting theluminance from :

As in [7], is demultiplexed into three chrominancechannels:

with } and represents the position of a pixel.For simplicity, in the following indices may be omittedfrom a corresponding matrix if there is no confusion. and are the sampling matrices for the three colors

– R, G, and B – in the Bayer CFA pattern (Fig. 1a). Forexample,

Hence, is also sub-sampled.

Now our objective is to reconstruct a full color image from

three chrominance channels . In particular, we willinterpolate and, then, add luminance to thesechannels to generate the full RGB plans.

Suppose that we have a full channel, for instance, aftersome certain interpolation and the two other channels, and, are still sub-sampled. This full channel is considered torepresent the difference between plan G and luminance L, or (it explains why adding luminance will generate afull G plan). Besides, in the sub-sampled channel, at theexisting R pixels we have . Thus, (9)

That means at the existing R pixels the difference betweenchannels and is also the color difference R-G. Similarlyfor the difference between channels and : . This observation suggests that we can use the constant-color-difference assumption to interpolate channels and once the full channel is known.

The remaining issue is how to compute the full channel.Since we want to recover correct colors along edges, we

propose to use the edge-preserving method described in [6]due to its proved performance and low complexity. Thus,chrominance channel is interpolated in the same way as inEq. 3. The proposed algorithm can be summarized as follows:

Algorithm 1:

-

Input : a mosaic image (only one color per pixel)- Estimate the luminance,- Compute three chrominance channels ,- Interpolate using edge-preserving method,- Interpolate and using the constant-color-difference

assumption,- Add the luminance to ,- Output : a full color image.



Fig. 3. Demosaicing results of different algorithms: (a) original image Kodim07, (b) zoom of the green square in the original image, (c) bilinear

interpolation, (d) Edge2 [6], (e) Edge3 [15], (f) Alleysson’s method [7], (g) Kiku’s method [11], (h) Proposed method.

TABLE 1. PNSR for different methods with the Kodak base of images. The table shows PNSR results for each image and PNSR averaged over all 24images of the base.

Bilinear Edge1 [5] Edge2 [6] Edge3 [15] Alleysson [7] Kiku [11] Proposed

Kodim07 32.44 37.95 40.66 40.57 38.39 41.98 40.44

Kodim08 23.61 29.71 32.34 32.85 29.40 35.21 34.93

Kodim19 27.84 34.39 37.21 36.60 33.74 39.95 39.36

All images 29.51 34.31 36.96 37.08 35.45 39.23 38.50

It is important to note that while preserving edges inchrominance the proposed method can strengthen edges inluminance once luminance and chrominance are recombined.Furthermore, this method is very compact: it features lowcomplexity and only uses simple operations.

IV. EXPERIMENT

We will test our proposed demosaicing method andcompare it with existing ones. In particular, we simulate the

bilinear interpolation; the three edge-based methods describedin section II (Edge1, Edge2, and Edge3); and Alleysson’salgorithm [7]. Moreover, these methods are also compared

with a state-of-the-art algorithm [11] keeping in mind that thelatter features much higher computational complexity. TheKodak base of lossless color images (Fig. 6) – popular indemosaicing evaluation – will be used for testing. Weevaluated each method with all the images, however in orderto see difference between methods it is required to explore indetail some specific images.

The first one is Kodim07 of size 768×1024. From thisoriginal image, we simulate the Bayer CFA pattern to generatethe corresponding mosaic image (like Fig. 1b). Given thismosaic image full color images are reconstructed by different

demosaicing methods. It is important to note that in reality wedo not known the original image – this is exactly the one wewant to recover – but we have only the mosaic image comingfrom a camera’s sensors. Here the original image is used onlyfor the purpose of algorithms evaluation. Fig. 3 shows thedemosaiced images for different methods. Visually, methodEdge1 is often not much different from bilinear interpolationand, thus, is not showed in this figure. Because of the limit ofspace this paper only shows zoomed regions; the full-sizedemosaiced images, which are not easy to visually distinguish

between methods, are also omitted. In Fig. 3, the result from bilinear interpolation is not very good: it contains blurring and

false colors at leaves. Yet it is not easy to differentiate othermethods: they all give visually satisfying results. In otherwords, more difficult testing images are required.

We continue to examine images that have high frequencycomponents or salient edges. For image Kodim19, Fig. 4illustrates advantage of the proposed method (Fig. 4h): iteliminates much of false colors, which often occur clearly inother methods. Particularly, though the proposed method andAlleysson’s algorithm (Fig. 4f) share the separation betweenluminance and chrominance, a careful edge treatment ofchrominance channels in the proposed method does pay off as







it provides a much better result. Similarly, Fig. 5h showssatisfying performance of the proposed method for thin edges,while most of the other methods reveal artifacts. Anotherimage in the Kodak base, Kodim08, repeats the abovetendency, which we do not show in this paper due to spacelimitations. Quantitatively, table 1 confirms, through the

PSNR criterion [15], performance of the proposed method forthe above images. Interestingly, our proposed method givesresults equivalent – both qualitatively and quantitatively – tothose of Kiku’s algorithm, which is a state-of-the-art methodand has been proved to outperform many other state-of-the-artalgorithms [11]. However, our algorithm is of low complexity



Fig. 6. Kodak base of lossless color images: each image has size of768×1204. The numbers on the images are intentionally added for the

convenience of reading.

and only requires simple operations while Kiku’s methodfeatures much higher complexity.

To verify the methods with various types of image, weshowed PNSR averaged over all 24 images of the Kodak base(last row of table 1). The proposed method presents once againsatisfying result: its PSNR is close to Kiku’s algorithm.

Nevertheless, it is important to note that objective evaluation,e.g. PSNR, needs to be combined with subjective evaluationwhen it comes to demosaicing performance or image qualityin general. Artifacts such as false colors may produce littledeviation in quantitative criteria but show embarrassing resultswhen we look at them.

V. CONCLUSION

The proposed method inherited the idea of separation between luminance and chrominance, which actually happensin the human visual system, to deal with the demosaicing

problem. This allows extracting directly luminance fromknown pixels of the CFA. In other words, luminance replacesthe G plan in edge-based methods in which G values need to

be interpolated. Interpolation is now applied only tochrominance channels. Moreover, the proposed method triesto preserve edges in chrominance; this leads to reducing falsecolors and improving details in luminance. The experimentshowed promising results, qualitatively and quantitatively, forthe proposed method.

With the aim at hardware implementation in mind, we proposed a compact and effective algorithm, whichoutperforms popular low-complexity methods. The proposedmethod’s performance is even closed to Kiku’s state-of-the-artalgorithm although this latter is of much higher complexity.

The next step is to devise an efficient architecture forhardware implementation of the proposed method.

ACKNOWLEDGMENT

This work is funded by Vietnam National Foundation forScience and Technology Development (NAFOSTED) undergrant number 102.99-2013.36 and by Ministry Project No.B2014-01-17.

R EFERENCES

[1] B. E. Bayer, “Color imaging array,” U.S. Patent 3971065, 1976.

[2] B. K. Gunturk, J. Glotzbach, Y. Altunbasak, R. W. Schafer, and R. M.Mersereau, "Demosaicking: color filter array interpolation," IEEE SignalProcessing Magazine, 22(1):44 – 54, 2005.

[3] X. Li, B. Gunturk, and L. Zhang, “Image demosaicing: A systematicsurvey,” in proc. SPIE, vol. 6822, p. 68221J, 2008.

[4] D. Menon and G. Calvagno, "Color image demosaicking: An overview,"Journal Image Communication, 26(8-9): 518--533, 2011.

[5] C. A. Laroche and M.A. Prescott, “Apparatus and method for adaptivelyinterpolating a full color image utilizing chrominance gradients,” U.S.Patent 5 373 322, 1994.

[6] J. F. Hamilton Jr. and J. E. Adams, "Adaptive color plane interpolationin single color electronic camera," U.S. Patent 5 629 734, 1997.

[7] D. Alleysson, S. Susstrunk, and J. Herault, "Linear demosaicing inspired by the human visual system," IEEE Trans. Image Process., 14(4):439 – 449, 2005.

[8] B. K. Gunturk, Y. Altunbasak, and R. M. Mersereau, “Color planeinterpolation using alternating projections,” IEEE Trans. Image Process.,11(9): 997 – 1013, 2002.

[9] Y. M. Lu, M. Karzand, and M. Vetterli, “Demosaicking by AlternatingProjections: Theory and Fast One-Step Implementation,” IEEE Trans.Image Process., 19(8): 2085-2098, 2010.

[10] L. Zhang, X. Wu, A. Buades, and X. Li, “Color demosaicking by localdirectional interpolation and nonlocal adaptive thresholding,” in Journalof Electronic imaging, Vol. 20, No. 2, 2011.

[11] D. Kiku, Y. Monno, M. Tanaka, and M. Okutomi, "Minimized-Laplacian Residual Interpolation for Color Image Demosaicking," Proc.SPIE 9023, Digital Photography X, 90230L, 2014.

[12]

S. P. Jaiswal, O. C. Au, V. Jakhetiya, Y. Yuan, and H. Yang,"Exploitation of Inter-color correlation for Color Image Demosaicing,"Proc. of IEEE Int. Conf. on Image Processing (ICIP), 2014.

[13] A. A. Moghadam, M. Aghagolzadeh, M. Kumar, and H. Radha,"Compressive Framework for Demosaicing of Natural Images," IEEETrans. Image Process., 22(6): 2356 - 2371, 2013.

[14] G. Jeon and E. Dubois, "Demosaicking of Noisy Bayer-Sampled ColorImages With Least-Squares Luma-Chroma Demultiplexing and NoiseLevel Estimation," IEEE Trans. Image Process., 22(1): 146-156, 2013.

[15] K. H. Chung and Y. H. Chan, "Color demosaicing using variance ofcolor differences," IEEE Trans. Image Process., 15(10):2944 – 2955,2006.

Paper Camera Demosaicing

Documents

Transcript of Paper Camera Demosaicing