Jinita Base Paper s1

7/28/2019 Jinita Base Paper s1

http://slidepdf.com/reader/full/jinita-base-paper-s1 1/6

I. Yoon et al.: Adaptive Defogging with Color Correction in the HSV Color Space for Consumer Surveillance System 111

Manuscript received 01/15/12

Current version published 03/21/12

Electronic version published 03/21/12. 0098 3063/12/$20.00 © 2012 IEEE

Adaptive Defogging with Color Correction in the HSV Color Space

for Consumer Surveillance System Inhye Yoon, Seonyung Kim, Donggyun Kim, Member , IEEE, Monson H. Hayes, Fellow, IEEE,

and Joonki Paik, Member , IEEE

Abstract — Consumer video surveillance systems often

suffer from bad weather conditions, observed objects lose

visibility and contrast due to the presence of atmospheric

haze, fog, and smoke. In this paper, we present an image

defogging algorithm with color correction in the HSV color

space for video processing. We first generate a modified

transmission map of the image segmentation using multiphase

level set formulation from the intensity (V) values. We also

estimate atmospheric light in the intensity (V) values. The

proposed method can significantly enhance the visibility of

foggy video frames using the estimated atmospheric light and

the modified transmission map. Another contribution of the

proposed work is the compensation of color distortion

between consecutive frames using the temporal difference

ratio of HSV color channels. Experimental results show that

the proposed method can be applied to consumer video

surveillance systems for removing atmospheric artifacts

without color distortion1.

Index Terms — Single image defogging, color correction,

enhancement of the video surveillance system, bad weather.

I. INTRODUCTION

Identification of the original color and shape of an object is

a challenging problem in consumer video surveillancesystems, and the problem becomes even worse with unclear

meteorological conditions including fog, smoke, storm, and

snow. For overcoming this problem, various image processing

approaches have been investigated.

For example, Narasimhan et al. [1] addressed the problem

of restoring the contrast of atmospherically degraded images

and videos. Narasimhan’s method estimates depth from two

images of the same scene that are captured under different

1 This research was supported by the Chung-Ang University excellent

freshman scholarship grants, by Business for Cooperative R&D between

Industry, Academy, and Research Institute funded by Korea Small and

Medium Business Administration in 2011 (00045420-1), and by Basic

Science Research Program through National Research Foundation (NRF) of Korea funded by the Ministry of Education, Science and Technology (2009-

0081059).

Inhye Yoon is with the Department of Image, Chung-Ang University,

Seoul, Korea (e-mail: [email protected]).

Seonyung Kim is with the Department of Image, Chung-Ang University,


Donggyun Kim is with the Department of Image, Chung-Ang University,


Monson H. Hayes is with the Department of Image, Chung-Ang

University, Seoul, Korea (e-mail: [email protected]).

Joonki Paik is with the Department of Image, Chung-Ang University,


weather conditions. In spite of the improved defogging

performance, this method cannot be used for dynamic scenes

because of the requirement to capture multiple images of the

same scene under different environmental conditions.

Shwartz [2] exploited two or more images of the same scene

having different degrees of polarization by rotating a

polarizing filter attached to the camera. This method is very

constrained in the image acquisition process, and cannot be

used on existing image databases. Hautiere [3] used multiple

images for the same scene under different weather

conditions to compute depth information for contrast

enhancement. Although existing methods addressed the possibility of enhancing foggy images, they are not suitable

for consumer cameras because of the need of multiple

images and special hardware devices.

From a theoretical point of view, defogging can be

considered as an under-constrained problem if only a single

foggy image is available. Fattal [4] estimated the albedo of

the scene and then inferred the medium of the transmission

under the assumption that the transmission and surface

shading are locally uncorrelated. However, this approach

cannot handle images that are heavily foggy, and may fail if

the uncorrelated assumption does not hold. Tan [5] observed

that the fog-free image must have higher contrast than theunprocessed version of the foggy image. Based on this

observation the foggy component is removed by maximizing

the local contrast of the restored image. Chen [6][7]

observed that most local regions of a fog-free image have a

set of pixels with very low intensity, and used the dark

channel prior to remove fog. Kratz [8] used a factorial

Markov random field to model the haze image, and takes

scene albedo and scene depth as two statistically

independent components, and removed the haze by

factorizing the image into scene albedo and depth. However,

the single image-based methods are prone to halo effect and

color distortion problem.

Fog removal method is similar to the contrast enhancementmethod. Kong [9] conducted histogram equalization over all

image pixels concurrently. On the other hand, local

equalization tackles image enhancement by dividing the image

into multiple sectors and equalizing them independently. Xu

[10] used the parameter-controlled virtual histogram

distribution method, and enhanced both overall contrast and

sharpness of an image. While it can increases the visibility of

specified portions or aspects of the image color, this approach

cannot correctly restore the original color in the foggy image.



112 IEEE Transactions on Consumer Electronics, Vol. 58, No. 1, February 2012

In order to solve the aforementioned problems, we present a

novel video defogging algorithm with color correction for the

consumer video surveillance system. The proposed algorithm

first generates a modified transmission map using the

multiphase level set formulation from the intensity (V) values

in the HSV color space, and also estimates the atmospheric

light from the intensity (V) values. Based on the estimated

atmospheric light and the modified transmission map, we can

simultaneously remove fog and correct color distortion. The

proposed color correction method compensates color

distortion between consecutive frames using the temporal

difference ratio of HSV color channels. Experimental results

demonstrate that the proposed algorithm not only increases the

visual quality of atmospherically degraded video inputs, but

also improves the performance of object detecting and

tracking in consumer video surveillance systems.

II. FOGGY IMAGE FORMATION MODEL

A scene is produced if one or more objects are illuminated

by the light source as shown in Fig. 1. The light reflected from

an object is scattered and absorbed by aerosols in the

atmosphere before it reaches the observer. More specifically,

the light arriving at the camera consists of two components; i)

directly attenuated light and ii) airlight. The former represents

the original scene components partially attenuated by

absorption in the atmosphere, and the latter the light reflected

from other directions by scattering in the atmosphere. Because

of the atmospheric absorption, the original color of an object

is changed into a “foggy” color. The quality of consecutive

frames in bad weather is usually degraded by the presence of

fog in the atmosphere, since attenuation in the incident light

decreases the contrast of the acquired image. Fig. 1 shows thefoggy image formation model.

Fig. 1. The optical model of foggy image acquisition.

In the computer vision research field, the image degradation

model due to fog is described in RGB color space as

( , ) ( , ) ( , ) (1 ( , ))c c

I O f x y f x y T x y A T x y ,

for { , , }c R G B , (1)

where ( , )c

I f x y represents the foggy image, ( , )c

O f x y

the

ideal or original fog-free image, A the global atmospheric

light, and ( , )T x y the transmission map that is independent of

the color.

In the right hand side of (1), ( , ) ( , )c

O f x y T x y

represents the

directly attenuated component, and the second term

(1 ( , )) A T x y represents the airlight. More specifically, thedirect attenuated component describes the decayed version of

the ideal fog-free image ( , )c

O x y , while airlight results from

scattered light in other directions and color shifts in the scene.

The defogging problem is to restore ( , )c

O f x y , given

( , )c

I f x y . In the restoration process, it is necessary to

estimate A and ( , )T x y .

III. THE PROPOSED ALGORITHM

The proposed video defogging algorithm consists of transmission map generation, atmospheric light estimation,

recovery of defogged frame, and color correction using

consecutive frames, as shown in Fig. 2, where time variable t

is added for extending the image formation model in (1) to the

video.

Fig. 2. The proposed video defogging algorithm for consumer video

surveillance systems.

Each step of the algorithm is described in the following

subsections. In this section, we use a set of test images; an

ideal fog-free image and its foggy version to demonstrate the

performance of the proposed algorithm as shown in Fig. 3.

(a) (b) (c)

Fig. 3. Test images; (a) original fog-free image, (b) the simulated foggy

image, and (c) the defogged image using the proposed method.

A. Modified transmission map generation using multiphase

level set

Existing methods for generating the transmission map

search for the lowest intensity in the patch centered at ( , ) x y ,




called ( , ) x y , from (1) as [6]

( , ) ( , )

( , )

{

( , )

, , }

{ , , }

( , )min min

( , )( , ) min min 1 ( , ) .

c

I

R G B

c

O

c

R G

p q x y

c p x y B q

f p q

A

f p qT x y T x y

A

(2)

Ideally, the minimum intensity in a patch of the ideal fog-

free image should be zero. Therefore, we compute the

transmission map as

{ , , } ( , ) ( , )

( , )( , ) 1 min min

c

I

Rc p y B qG x

f p qT x y

A

. (3)

We note that existing transmission map exhibit a halo effect

and color distortion since the intensity discontinuity across

edges is not considered in the reconstruction process.

Existing halo effect reducing method uses image matting, but it causes extremely high computational complexity, which

is unsuitable for on-line video processing. To solve this

problem, we generate a modified transmission map using the

multiphase level set formulation, which partitions the image

into piecewise uniform regions. Since the transmission map

determines how much the light reflected by the object reaches

the camera, we assume that the light traveling a longer

distance is more attenuated, which yield the modified

transmission map as

( , )( , ) d x y x y eT , (4)

where is the scattering coefficient of the atmosphere, and

( , )d x y represents the depth or distance of the location ( , ) x y .

For obtaining the depth map in (4), the multiphase level set

method is used to reduce the processing time.

Let multiple level set functions { , 1,......, } j

j n represent

the rgions { , 1,......, } j j N with 2n N as defined in [11].

The j -th level set function j

is typically defined as the

signed distance function of the corresponding region j . We

compute the depth map ( , )d x y using two level set functions

that generates four phases as shown in Fig. 4.

Fig. 4. Partitioning of the image into four phase using two curves.

The four phases provide an effective, implicit

representation for evolving curves and surface. Therefore, we

compute the four phases as

1 1 2 2 1 2

3 1 2 4 1 2

( ) ( ), ( )(1 ( ))

(1 ( )) ( ), (1 ( ))(1 ( ))

P H H P H H

P H H P H H

, (5)

where ( ) j H is the Heaviside function. The depth map

( , )d x y in each variable is then obtained by fixing other

variables as

4

4

( )

( , )

V j j

j

j j

j

f P c

d x y

P c

, (6)

wherev

f represents the intensity (V) values in the HSV color

space. The modified transmission map can be generated from

the depth map obtained in (6). The conventional and modifiedtransmission maps of Fig. 3(b) are shown in Fig. 5(a) and Fig.

5(b), respectively. Since the intensity values are discontinuous

at the boundary of each pixel, we used a multiphase level set

to mitigate the discontinuities. As a result, the proposed

modified transmission map has continuous intensity values in

the neighborhood of boundaries.

(a) (b)

Fig. 5. Two different transmission map generated by; (a) the existing

method [4] and (b) the proposed method.

B. Estimation of atmospheric light and image defogging

Existing atmospheric light estimation methods extract the

brightest pixel among all color channels in the foggy image,

which results in the color distortion problem if there are

originally white objects without atmospheric degradation. In

this paper, we use the highest intensity (V) value in the HSVcolor space for the atmospheric light [5]. Given the

atmospheric light and the modified transmission map

( , )T x y , the V component of the defogged image frame can

be recovered as

( , , ) ( )( , , ) ( )

( , , )

V V I

D

f x y t A t f x y t A t

x y t T

(7)

where t represents the frame number.




Since it is not easy to quantify the performance of

defogging, we provide the defogged image the proposed

method in Fig. 3(c). Extensive comparison with existing

methods will be presented in section 4.

C. Color Correction between Consecutive Frames

After removing the foggy components, each frame may

have different color tones because the atmospheric lightand the transmission map are generated without

consideration of the temporal correlation between

consecutive frames. For solving this problem, it is

necessary to perform color correction for the defogged

frames to reduce the changes in color tone between

frames.

In the HSV color space, given a certain intensity (V)

value saturation (S) is defined as the relative length of the

vector that points to the given color on the border, And

hue (H) is defined as the linear length along the loci of

constant Saturation (S) from the reference vector

indicating the red color, that is the vector r , as shown inFig. 6 [12].

(a) (b) (c)

Fig. 6. Saturation and hue of a color vector as defined in the HSV color

space; (a) intensity values, (b) saturation color channel, and (c) hue color

channel.

For the proposed color correction use compute frame

differences in all HSV color channels as

1

( , , ) ( , , 1) M N

d

x y

d d

D Dc f x y t f x y t M N

,

for { , , } H d S V , (8)

where M and N represents horizontal and vertical sizes of

the image, respectively. We can then obtain color corrected

HSV color channels as

( , , ) ( , , )d d d

E D f x y t f x y t c , for { , , } H d S V . (9)

By transforming the processed HSV color space, which are H

E f , S

E f , and V

E f in (9), into the RGB color space, we finally

obtain the enhanced video frames. Fig. 7(a) shows foggy input

video frames captured by a surveillance camera. The defogged

and color corrected frames are shown in Fig. 7(b) and Fig.

7(c), respectively.

(a)

(b)

(c)

Fig. 7. Experimental results of defogging and color correction method for

video surveillance system; (a) the input consecutive frames (40th, 60th,

80th, and 100th frames), (b) the defogged frame by using the proposed

method, and (c) the defogged and color corrected frames by using the

proposed method.

IV. EXPERIMENTAL R ESULTS

In this section, we demonstrate the performance of the

proposed algorithm for enhancing foggy video.

Fig. 8(a) shows another input foggy video frames. The

results of the modified transmission map using the proposed

method, the defogged frames using the proposed method, and

the color corrected frames using the proposed method are

shown in Fig. 8(b), Fig. 8(c), and Fig. 8(d), respectively.

Experimental results demonstrated that the proposed

algorithm outperforms the existing algorithm in the sense of

both preserving the original color and improving visibility.

(a)

(b)

(c)

(d)

Fig. 8. Experimental results of defogging with color correction method;

(a) the input consecutive frames (90th, 100th, 110th, and 120th

frames),(b) the modified transmission map by using the proposed method,

(c) the defogged frames by using the proposed method, and (d) the

defogged and color corrected frames by using the proposed method.




Fig. 9(a) shows a set of foggy video frames, and Fig. 9(b)

shows the defogged frames using the existing method

proposed in [4]. The results of modified transmission maps

using the proposed method are shown in Fig. 9(c) and Fig.

9(d) shows the defogged frames using the proposed algorithm.

Based on the experimental results, the proposed defogging

method significantly outperforms existing method in the sense

of both contrast recovery and color preservation.

(a)

(b)

(c)

(d)

Fig. 9. Experimental results of various defogging methods; (a) an input

foggy image, (b) the defogged image by using the existing method in [4],

(c) the modified transmission map by using the proposed method, and (d)

the defogged image by using the proposed method.

V. CONCLUSIONS

In this paper, we proposed a combined video defogging and

color correction methods in the HSV color space.

The proposed algorithm first generates a modified transmission

map using the multiphase level set formulation in the intensity

(V) component of the HSV color space, and estimates the

atmospheric light the intensity (V) values. Defogged framesare then produced using the atmospheric light and th

emodified transmission map. Finally, color distortion between

consecutive frames is corrected by using the tem

poraldifference ratio of HSV color channels. Experimental results

demonstrated that the proposed algorithm outperforms the

existing algorithm in the sense of both preserving the original

color and improving the visibility. The proposed method can

be applied to the extended areas of image enhancement by

removing fog-like haze, clouds, and dust from the consumer

video surveillance systems.

R EFERENCES

[1] S. Narasimhan and S. Nayar, “Contrast restoration of weather

degraded images,” IEEE Trans. Pat tern Analys is and Machine

Intell igence , vol. 25, no. 6, pp. 713-724, June 2003.

[2] S. Shwartz, E. namer, and Y. Schechner, “Blind haze separation,”

Proc. IEEE Int . Conf . Computer Vis ion , Pattern Recogni tion, pp.

1984-1991, October 2006.

[3] N. Hautiere, J . Tarel, and D. Aubert, “Towards fog-free in-vehicle

vision systems through contrast restoration,” Proc. IEEE Conf .

Computer Vision, Pattern Recognition, pp. 1-8, June 2007.[4] R. Fattal, “Single image dehazing,” ACM Transacti ons on

Graphics, vol. 27, no. 3, pp. 1-9, August 2008.

[5] R. Tan, “Visibility in bad weather from a single image,” Proc.

IEEE Conf . Computer Vis ion , Pat tern Recogni tion, pp. 1-8, June

2008.

[6] M. Chen, A. Men, P. Fan, and B. Yang, “Single image defogging,”

IEEE Conf erence on Network Infrastructu re and Digita l Content ,

pp. 675-679 , November 2009.

[7] I. Yoon, J. Jeon, J. Lee, and J. Paik, “Spatially adaptive image

defogging using edge analysis and gradient-based tone mapping,”

Proc. IEEE Int. Conf . Consumer E lec tronics, pp. 195-196, January

2011.

[8] L. Kratz and K. Nishino, “Factorizing scene albedo and depth

from a single foggy image,” IEEE Int . Conf . Computer Vis ion , pp.

1701-1708, September 2009.

[9] N. Kong and H. Ibrahim, “Color image enhancement using

brightness preserving dynamic his togram equalizat ion ,” IEEE

Trans. Consumer Electronics, vol. 54, no. 4, pp. 1962-1968,

November 2008.

[10] Z. Xu, H. Wu, X. Yu, and B. Qiu, “Colour image enhancement by

virtual histogram approach,” IEEE Trans. Consume r Ele ctronics,

vol. 56, no. 2, May 2010.

[11] K. Zhang, L. Zhang, and S. Zhang, “A variational multiphase level

set approach to simultaneous segmentation and bias correction,”

IEEE Int . Conf . Image Processing , pp. 4105-4108, December

2010.

[12] M. Ebner, “Color constancy,” John Wi ley & Sons, England, 2007.

BIOGRAPHIES

Inhye Yoon was born in Suwon, Korea in 1988. She

received the B.S. degree in electronic engineering from

Kangnam University, Korea, in 2010. Currently, she is

pursuing M.S. degree in image processing at Chung-Ang

University. Her research interests include image

restoration, digital auto-focusing, image and video

processing, real-time object tracking.

Seonyoug Kim was born in Pohang, Korea in 1988. She

received the B.S. degree in information

telecommunications engineering from Suwon University,

Korea, in 2011. Currently, she is pursuing M.S. degree in

image processing at Chung-Ang University. Her researchinterests include image restoration, digital auto-focusing,

image and video processing, real-time object tracking.

Donggyun Kim was born in Ulsan, Korea in 1983. He

received the B.S. and M. S. degree in electronic and

electrical engineering from Chung-Ang University, Korea,

in 2007 and 2009, respectively. Currently, he is pursuing

Ph.D. degree in image processing at Chung-Ang

University. His research interests include image

restoration, digital auto-focusing, image and video

processing, real-time object tracking.




Monson Hayes received his BS Degree from the

University of California at Berkeley in 1971, worked as a

Systems Engineer at Aerojet Electrosystems until 1974,

and then received his Sc.D. degree in electrical

engineering and computer science from the Massachusetts

Institute of Technology in 1981. He then joined the

faculty at the Georgia Institute of Technology where he is

a Professor of Electrical and Computer Engineering. Dr. Hayes was a recipient

of the Presidential Young Investigator Award and the recipient of the IEEE

Senior Award. He has served the Signal Processing Society of the IEEE in

numerous positions, including Chairman of the DSP Technical Committee(1995-1997), Associate Editor for the IEEE Transactions on Acoustics,

Speech, and Signal Processing (1984-1988) and Associate Editor for the IEEE

Transactions on Education (2000-2010), Secretary-Treasurer of the ASSP

Publications Board (1986-1988), and Chairman of the ASSP Publications

Board ((1992-1994), General Chairman of ICASSP 96, and General Chairman

of ICIP 2006. Currently, Dr. Hayes has served as an Associate Chair in the

School of ECE at Georgia Tech, and as Associate Director for Georgia Tech

Savannah. Since joining the faculty at Georgia Tech, Dr. Hayes has become

internationally recognized for his contributions to the field of digital signal

processing, image and video processing, and engineering education. He has

published over 150 papers, is the author of two textbooks, and has received

numerous awards and distinctions from professional societies and Georgia

Tech. His research interests are in the areas of face and gesture recognition,

image and video processing, adaptive signal processing, and engineering

education. In 1992 he was elected to the grade of Fellow of the IEEE. Dr.

Hayes is currently a Distinguished Foreign Professor at Chung-AngUniversity, Seoul, Korea, in the Graduate School of Advanced Imaging

Science, Multimedia, and Film.

Joonki Paik was born in Seoul, Korea in 1960. He

received the B.S. degree in control and instrumentation

engineering from Seoul National University in 1984. He

received the M.S. and the Ph.D. degrees in electrical

engineering and computer science from Northwestern

University in 1987 and 1990, respectively. From 1990 to

1993, he joined Samsung Electronics, where he designed

the image stabilization chip sets for consumer’s camcorders. Since 1993, he

has joined the faculty at Chung-Ang University, Seoul, Korea, where he is

currently a Professor in the Graduate school of Advanced Imaging Science,

Multimedia and Film. From 1999 to 2002, he was a visiting Professor at theDepartment of Electrical and Computer Engineering at the University of

Tennessee, Knoxville. Dr. Paik was a recipient of Chester-Sall Award from

IEEE Consumer Electronics Society, Academic Award from the Institute of

Electronic Engineers of Korea, and Best Research Professor Award from

Chung-Ang University. He has served the Consumer Electronics Society of

IEEE as a member of the Editorial Board. Since 2005, he has been the head of

National Research Laboratory in the field of image processing and intelligent

systems. In 2008, he has worked as a full-time technical consultant for the

System LSI Division in Samsung Electronics, where he developed various

computational photographic techniques including an extended depth of field

(EDoF) system. From 2005 to 2007 he served as Dean of the Graduate School

of Advanced Imaging Science, Multimedia, and Film. From 2005 to 2007 he

has been Director of Seoul Future Contents Convergence (SFCC) Cluster

established by Seoul Research and Business Development (R&BD) Program.

Dr. Paik is currently serving as a member of Presidential Advisory Board for

Scientific/Technical policy of Korean Government and a technical consultantof Korean Supreme Prosecutor’s Office for computational forensics.

Jinita Base Paper s1

Documents

Transcript of Jinita Base Paper s1