Ning Sun, Hassan Mansour, Rabab WardProceedings of 2010 IEEE 17th International Conference on Image Processing

September 26-29, 2010, Hong Kong

HDR Image Construction from Multi-exposed Stereo LDR Images

Andy {andrey.korea@gmail.com}

2Intelligent Systems

Lab.

Algorithm descriptionTwo LDR images

with different exposures

Initial disparity map

Camera response function

Radiance maps of LDR

images

Refined disparity

mapHDR image

Main concept:

1. Multi-exposed stereo images are captured using identical cameras placed adjacent to each other on a horizontal line.

2. Stereo matching is then used to find a disparity map that matches each pixel in one image to the corresponding pixel in another image.

3. A subset of the matched pixels is used to generate the camera response function which in turn is used to generate the scene radiance map for each view with an expanded dynamic range.

4. The disparity map is refined by performing a second stereo matching stage using the radiance maps

3Intelligent Systems

Lab.

Imaging models

RI l eRI r

Pp n

nrn

n

nlnn pIcepIccJ

Gamma-correction model Polynomial camera response

Imaging models are used to determine the scene radiance from the measured pixel data

nn cJc minarg

Left image Right image

Scene radiance

Correction factor

Exposure ration between images

Exposure ration between images

Left image Right image

Scene radiance

4Intelligent Systems

Lab.

Computing the disparity map

NfEfEf SdFf

,minarg*

Best disparity map

Dissimilarity term

Set of feasible disparities

Smoothing term

p

pp

ppd fpNCCfDfE ,1

p pNq

qps VqpNfE ,,,

Pixel dissimilarity Disparity smoothness

Used for initial disparity estimation

5Intelligent Systems

Lab.

Pixel dissimilarity

22 ~~

~~

,

prrll

pWqprlrl

p

fpIwpIw

fqIqIww

fpNCC

pW - Search window centered on p

pf - displacement tw - Bilateral weight

2

2

2

2

2

''

2exp

sd

pItItptw

Spatial smoothing Intensity smoothing

ReII logloglog' I’ - intensity in log space defined as:

0.146.2 rs

6Intelligent Systems

Lab.

Pixel dissimilarity

pWt

pWt

pWt

jpWt

ll tw

Rtw

Rtw

Itw

II

log

log~

pWt

pWt

pWt

pWtr tw

Rtw

Rtw

Retw

ReI

log

log

loglog

loglog~

7Intelligent Systems

Lab.

Disparity smoothness

max

2

, ,min, VffffV qpqpqp

p pNq

qps VqpNfE ,,,

2

2

2

2

2

2

2

2

2222exp,

r

bb

r

aa

r

LL

s

qIpIqIpIqIpIqpqp

NfEfEf SdFf

,minarg*

Initial disparity and camera response

1. Minimize using graph cut algorithm

2. Compute polynomial coefficients for camera response function

0.164.2 rs

8Intelligent Systems

Lab.

Error correction

NfEfEf SdFf

,minarg*

Minimize energy function one more time with different dissimilarity function

For valid pixels

R~

Convert images to radiance space (results should be same for both images)

p

ppd fDfE

otherviseK

ffiffD

initialpp

pp,

,0

For erroneous pixels

rlppprlpp RRpWfCfpRpRfD~,~,,

~~

Hamming distance between pixels p and p+fp after applying

Census transform

9Intelligent Systems

Lab.

Input LDR images

10Intelligent Systems

Lab.

Disparity maps

Reference disparity map Initial disparity estimation Final map

11Intelligent Systems

Lab.

HDR images

12Intelligent Systems

Lab.

Experimental results

Image name Exposure Ratio RMSE Error Error pixels (%)

Statue 416

0.99430.9976

8.238.82

Dolls 416

0.84540.8591

4.775.58

Clothes 416

1.54591.1556

7.438.15

Baby 416

1.4321.4642

9.4210.13

13Intelligent Systems

Lab.

ConclusionsDisparity map computation algorithm is proposed

Proposed method is able to compute disparity between differently exposed images

Can deal with saturated regions in the image

Can be used for capturing motion scenes with different exposures

Disadvantages

- High computational costs

- Generated images are slightly blurred

- No rotation is considered

14Intelligent Systems

Lab.

Ideal image formation system

eLfI

Image brightness

Sensor response

Camera exposureCamera response function

Response = Gray-level

Irra

dian

ce

L

I

BgBfL 1

Reverse camera response function

42

cos4

h

dRE

From optics

Image radiance

Scene radianceFocal length

Aperture

Angle from ray to optical axis

EtL

Radiometric response

Shutter speed

or

RkeL

Where

td

e4

24

2cos

1

hk

N

c

nn

n

Ic0

15Intelligent Systems

Lab.

Response function examples

Response functions of a few popular cameras provided by their manufacturers

I

L

16Intelligent Systems

Lab.

Graph-cut algorithm

1. Start with an arbitrary labeling f

2. Set success := 0

3. For each label 2 L

3.1. Find f* = arg min E(f’) among f’ within one α-expansion of f

3.2. If E(f*) < E(f), set f := f* and success := 1

4. If success = 1 goto 2

5. Return f

17Intelligent Systems

Lab.

Census transform

If (CurrentPixelIntensity<CentrePixelIntensity) boolean bit=0else boolean bit=1

Input image 3x3 transform 5x5 transform