Stereoscopic Video Color Correction Based on Keyframe Extraction

Chao hui Lü, Jinghua Li, Xiaolong Chen, Jiaying Pan School of Information Engineering,Communication University of China,Beijing,100024,China

e-mail: llvch@hotmail.com

Abstract—Color inconsistency is an urgent problem to be solved in stereoscopic video. In this paper, a robust color correction method for stereoscopic video is proposed. We first take advantage of feature points matching based on Scale Invariant Feature (SIFT) to find the corresponding match points of the images. Next, parameter model between the images is calculated by the value of each match point pair and obtained the optimal from the least square method. Finally, keyframe extraction technology based on color feature is used to perform consistently the color correction of parameter model for each keyframe and its next every frame. Experimental results show that the proposed method can produce well correction quality for stereoscopic video.

Keywords-stereoscopic; color correction; SIFT; keyframe

I. INTRODUCTION With the considerable progress and wide application of

digital video technology, traditional Two-dimensional (2D) video can’t meet the needs of real and natural reproduction for people, so Three-dimensional (3D) stereo display technology comes around. As the impact of shooting parameter settings, the scenes illumination, camera noise, camera calibration and etc at the process of the three dimensional material acquisition, 3D images and 3D video production tend to a great difference for video colors of each view video and subsequent processing.

There are already many proposed color correction algorithm for stereo video at present. But most of these methods aim at single view image and have their limitations, including range of application or the complicated calculation. Otherwise, many algorithms can’t take into account about influence of calibration image and reference image for correction information in the overlapping area.

In order to eliminate color inconsistencies, many color correction methods had already been presented. Fecker et al. proposed a color correction algorithm based on histogram matching, who converted the YUV color space to RGB color space, took an intermediate viewpoint as a reference and adjusted RGB trichromatic histogram to make reference viewpoint match so good correction result has been obtained [1]. Hur et al. considering the effect of brightness variation for the search block matching came up with a new block matching criterion and transferred additional brightness difference to decoding side, which was a kind of local color correction (only for the brightness compensate for the Y value) [2]. Yamamoto et al. made correction for each color

value of viewpoint by the corresponding chrominance table in the code, which was a global correction method [3]. Ruderman et al. proposed a new l�� color space that the correlation of three-chromaticity had been minimal so that a simple algorithm was designed for three-chromaticity correction, which achieved good result when l�� was applied for color transfer between images [4].

In this paper, overall considering the accuracy of the result and the operation and extension of methods, we first construct the color mapping relation according to the corresponding feature points between images. Next, color corrections between images are made on the basis of the mapping relationship. Finally, we realize the correction of video color on the grounds of keyframe technology.

II. THE PROPOSED METHOD AND ITS PRINCIPLE This paper mainly researches on color correction

between two-way video. Namely, the viewpoint closer to natural scenery is took as reference viewpoint and makes color correction to keep two viewpoint images good consistent. Meanwhile, the method can achieve greater disparity estimation between left and right viewpoint about coding performance results.

A. SIFT Principle Scale invariant feature transform (SIFT) based on

computer vision is an algorithm. SIFT algorithm extracts distinctive local features, which are reasonable invariant to scaling translation, rotation, and partially invariant to illumination changes affine distortion, noise, and occlusion.

B. SIFT Color Correction Algorithm The template is used to format your paper and style the

text. All margins, column widths, line spaces, and text fonts are prescribed; please do not alter them. You may note peculiarities. For example, the head margin in this template measures proportionately more than is customary. This measurement and others are deliberate, using specifications that anticipate your paper as one part of the entire proceedings, and not as an independent document. Please do not revise any of the current designations.

1) Image feature point extraction We first preliminarily confirm the position and scale of

feature points across scale-space extreme detection. Next, the position and scale of feature points are made accuracy and low-contrast and the edge of unstable response points

2013 Sixth International Symposium on Computational Intelligence and Design

978-0-7695-5079-4/13 $26.00 © 2013 IEEE

DOI 10.1109/ISCID.2013.176

250

are eliminated. Finally, SIFT feature vector can get obtained.

2) Image Image feature points matching It’s essential to select a way as theirs similarity

measurement. All kinds of distance functions are usually taken as similarity measure of the feature matching such as Euclidean distance, mahalanobis distance [5]. Matching between images is weighted by computing and comparing similarity measurement. The Euclidean distance is usually used as their similarity measure for two images. After achieving SIFT feature vector, using priority k-d tree search algorithm equal to BBF approximation algorithm makes first search in another feature points of the image so that the two approximate nearest adjacent feature points with it can be found. We compute the minimum and the second smallest Euclidean and get its closest and second closest points in other images. If the ratio is smaller than a threshold, the two points between two images are considered as match. The number of matched points is relied on the threshold. Reduce this ratio threshold, the number of matching points will reduce, but more stable. We choose 0.7 as threshold.

3) Image correction parameter model The preceding text has made detailed introduction about

SIFT algorithm and feature points matching. Then parameter model is constructed through the color difference of matching point pair and relational mapping is obtained from the least square method between original images and reference images, so the original images can get color correction. The flow chart of SIFT image correction is shown in Figure1.

Figure 1. The flow chart of SIFT image correction

III. KEYFRAME TECHNOLOGY It assumes that video image sequence collections are F=

{ 1a , 2a , 3a , … , ka }. The needed keyframes are chosen

to put their in collection ka from F and every keyframe has enough differences between keyframes. So keyframe extraction algorithm is as follows:

1) Select the first frame a1 of video image sequence as keyframe and keep it in keyframe collections ka .

2) Calculate color feature distance d between adjacent images. The frequency of each color value in each frame is calculated in R, G, B channels respectively, such as R channel. That the correlation between the two frame images gets quantified is as follows.

12 / SRSRd R = (1)

Where Rd is quantization of the R channel correlation between two frame images. The frequency of each color value is calculated in R channel, the total of which is marked with 1SR . Compared the frequency of each color value of the current frame with the frequency of each color value of previous frame, the smaller value is chosen and add up the frequency of each color value marked with 2SR . In a similar way, G, B channels adopt the same ways to obtain

Gd , Bd . Gd and Bd are quantization of the G and B channels correlation between two frame images separately.

As human eyes are the most sensitive to luminance information, Y component information in YcbCr color space is selected to reflect the correlation of the current frame image and the previous frame image. The color feature distance between two images is as follows:

BGR dddd *114.0*587.0*299.0 ++= (2) “*” denotes multiplication operation Where d is the color distance between two images, Rd ,

Gd and Bd are quantization of the R, G, B channels correlation between two frame images separately.

When d is smaller than certain threshold, the frame is considered as keyframe. The current frame is kept down and its next every frame is continued to compute until the last frame.

IV. EXPERIMENTAL RESULT

A. Color Correction Result Comparison In order to confirm effectiveness of color correction

methods based on the SIFT feature matching and the least square method proposed by this paper, we take advantage of MATLAB 7.1 platform and perform related experiments for these algorithms. By comparing the proposed method with the histogram color correction algorithm, the former algorithm has more superior.

The histogram color correction algorithm is introduced in this paragraph. First, roughly corresponding areas are found between the two images and calculated R, G, B histogram separately. Next, the minimum value of its difference is obtained from the corresponding histogram of

Original image Reference image

Feature points extracting

Feature points extracting

Feature points matching

The least square method to obtain parameter model

Make correction for original image

251

the current image and reference image. So the corresponding value is the best matching value.

Experiments are performed on "Flamenco2" sequence with the size 640 × 480 and stereo video. The"Flamenco2" is provided by the KDDI Corporation and has 5views in total. Stereo video is provided by myself. Frame #90 for view 1and view 3 is selected to match in "Flamenco2" image sequence. Each color channel of matching point pairs between original image and reference image is taken to construct the sample values of the least square method parameter model. After getting the optimal parameter model, each color channel of images can be corrected. Then, the corrected images are that each corrected color channel is merged.

In this paper, we use Euler distance of overlap area between the corrected view point image and the reference view point image to evaluate the performance of color correction algorithm. If the value is smaller, the color deviation is smaller. The formula is as follows:

222 )()()( BBGGRRE −+−+−= (3) Where R , G , B are the average value of R, G, B

In order to compare the proposed color correction algorithm and the histogram color correction algorithm, four different experiments are designed in this paper. Here, we take two experiments as example.

The first experiment selects view 3 as reference view in "Flamenco2" and frame #90 of that view is taken to make correction for view 1, which makes contrast with the histogram algorithm. As each viewpoint of various video test sequences exist certain difference, we ourselves don’t need add color difference to make other experiments. Here, the frame #90 color correction results for view 1 of "Flamenco2" are shown in Fig.2.

Color Correction Matrix:

���

�

�

���

�

� −−

100071.0998.0012.0037.0007.0960.0

(4)

The Euler distance between the reference images and other images is following:

ans1=25.5490, ans1 is the distance between the original image and the reference image.

ans2=13.3058, ans2 is the distance between the original image and the reference image with the proposed method.

ans3=14.0944, ans3is the distance between the original image and the reference image with the histogram matching.

The second experiment researches on the left and right video images sequence, which is taken by myself. By scrambling for left and right image sequence, the first frame is sampled for color correction. Here, the frame #1 color correction results for left video are shown in Fig.3. The Euler distance between the reference images and other images is following:

ans1=12.0154, ans1 is the distance between the original image and the reference image.

ans2=5.2461, ans2 is the distance between the original image and the reference image with the proposed method.

ans3=7.1874, ans3is the distance between the original image and the reference image with the histogram matching.

Through many experiments, we can obtain that the proposed algorithm is much better than histogram algorithm and can be more efficiency for image color correction.

B. Stereo Video Color Correction Based on Keyframe Technology By contrast with the above, the conclusion can be drawn

that the proposed method is much better for image correction. But the correction result aims to every frame image. Although the video is divided into every frame image to make correction, the accuracy of correction will be much high. At the same time, the complexity of calculation will appear too much. However, the frames can consume the minimum calculation of costs with consistent mapping relation. The precise of correction can’t get guaranteed yet. In order to get the optimal balanced, the big change place selects many keyframes and the small change place selects few keyframes in image.

The experiment based on the color feature keyframe method is performed on “the main building of osmanthus fragrans” (239 frames in total), which is taken as the image to be corrected. The color feature distance threshold between two images is marked with d. We make four times experiments respectively when the thresholds are 0.65, 0.85, 0.9, and 0.95. As an example, The first four keyframes are extracted when we choose threshold with 0.95.

The results are shown in Figure.4. Color correction parameter model is calculated for all

keyframes and color corrections are made for the current keyframes and its next every frame with the same parameter model when all keyframes have been completely extracted. Then, each frame video image that has been corrected composes the video. So, stereo video has achieved a relatively high accuracy color correction under relatively low calculation of costs. The experiment performs color correction reasonably well for stereo video.

V. CONCLUSIONS This paper proposes the algorithm based on image

feature point matching and the least square method for color compensate between stereo images, which can solve the color differences between stereo images very well. Meanwhile, the extracted feature points are very robust with SIFT algorithm and assures the accuracy of color difference correction between viewpoint images. Parameter model with the least square is easily constructed and has good performance. On the basis of establishing image correction algorithm, the color correction method based on keyframe extraction for stereo video has been proposed and realize color correction for stereo video.

252

ACKNOWLEDGMENT This work is supported by the University Level

Engineering Research project of Communication University of China (No.3132013XNG1304), and the project of Science and Technology Support Program of China (No.2012BAH01F04-2-0).

REFERENCES [1] Fecker.U, Barkowsky.M, Kaup.A. Histogram-based profile-tering for

luminance and chrominance compensation of multiview video [J]. IEEE Transactions on Circuits and Systems for VideoTech-nology, 2008, 18(9): 1258-1267.J. Clerk Maxwell, A Treatise on Electricity and Magnetism, 3rd ed., vol. 2. Oxford: Clarendon, 1892, pp.68–73.

[2] Hur.JH, cho.S, Lee.Y.L. Adaptive local illumination Change Compensation method for H.264/AVC-based multiview video coding[J].IEEE, Transactions on Circuits and Systems For Video Technology, 2007, 17(11): 1496-1505.

[3] Yamamoto.K, Kitahara.M, Kimata.H,et al.Multiviewvideo coding using view interpolation and color correction [J].IEEE Transactions on Circuits and Systems For Video Technology, 2007, 17(11): 1436-1449.

[4] Ruderman.D.L, Cronin.T.W, Chiao.C.C. Statistics of coneresponses to natural images: Implications for visual coding [J].Journal of The Optical Society of America A, 1998, 15(8): 2036-2045.

[5] Rui Ting, Zhang Shengao. Registration of Multi-sensor Images Based on HarrisCorner with SIFT Description. Opto-Electroic Engineering. Audust 2019.39(8):26-31.

(a) (b) (c) (d) Figure 2. Color correction results of ‘Flamenco2’: (a) original image(view 1,frame #90); (b) reference image(view 3,frame #90);(c) corrected image with

histogram matching; (d)corrected image with the proposed method.

(a) (b) (c) (d)

Figure 3. Correction results of stereo video : (a)original image(left-view, frame #1); (b)reference image(right-view, frame #1); (c) corrected image with histogram matching; (d) corrected image with the proposed method.

(a) (b) (c) (d)

Figure 4. The corresponding image of the Keyframes is above when the threshold (d) is equal to 0.95 and keyFrameIndexes are equal to 1, 11, 24, 49.

253