www.elsevier.com/locate/jvci

J. Vis. Commun. Image R. 18 (2007) 439–451

Photometric image processing for high dynamic range displays

Matthew Trentacoste a,*, Wolfgang Heidrich a, Lorne Whitehead a,Helge Seetzen a,b, Greg Ward b

a The University of British Columbia, 2329 West Mall, Vancouver BC, V6T 1Z4 Canadab Dolby Canada, 1310 Kootenay Street, Vancouver BC, V5K 4R1 Canada

Received 16 November 2006; accepted 12 June 2007Available online 17 July 2007

Abstract

Many real-world scenes contain brightness levels exceeding the capabilities of conventional display technology by several orders ofmagnitude. Through the combination of several existing technologies, new high dynamic range displays have been constructed recently.These displays are capable of reproducing a range of intensities much closer to that of real environments. We present several methods ofreproducing photometrically accurate images on this new class of devices, and evaluate these methods in a perceptual framework.� 2007 Elsevier Inc. All rights reserved.

Keywords: High dynamic range; Displays; Image processing; Photometry

1. Introduction

The high dynamic range (HDR) imaging pipeline hasbeen the subject of considerable interest from the computergraphics and imaging communities in recent years. Theintensities and dynamic ranges found in many scenes andapplications vastly exceed those of conventional imagingtechniques, and the established practices and methods ofaddressing those images are insufficient.

Researchers have developed additions and modificationsto existing methods of acquiring, processing, and display-ing images to accommodate contrasts that exceed the lim-itations of conventional, low dynamic range (LDR)techniques and devices. Methods exist for acquiringHDR images and video from multiple LDR images[4,13]. New cameras are capable of capturing largerdynamic ranges in a single exposure [1]. File formats havebeen designed to accommodate the additional data storagerequirements [7,8,18]. Most relevant to this paper, high

1047-3203/$ - see front matter � 2007 Elsevier Inc. All rights reserved.

doi:10.1016/j.jvcir.2007.06.006

* Corresponding author.E-mail addresses: mmt@cs.ubc.ca (M. Trentacoste), heidrich@cs.ubc.

ca (W. Heidrich), whitehead@physics.ubc.ca (L. Whitehead), helge.seetzen@dolby.com (H. Seetzen), gward@lmi.net (G. Ward).

dynamic range display systems have been developed toaccurately reproduce a much wider range of luminance val-ues. The work done by Ward [17] and Seetzen et al. [14,15]has provided devices that vastly exceed the dynamic rangeof conventional displays. These devices are capable ofhigher intensity whites and lower intensity blacks, whilemaintaining adequately low quantization across the entireluminance range.

HDR displays are constructed by optically combining astandard LCD panel with a second, typically much lowerresolution, spatial light modulator, such as an array ofindividually controlled LEDs [14]. The latter replaces theconstant intensity backlight of normal LCD assemblies.Due to this design, pixel intensities in HDR displays cannotbe controlled independently of each other. Dependenciesare introduced since every LED overlaps hundreds ofLCD pixels, and thus contributes to the brightness of allof them. It is therefore necessary to employ image process-ing algorithms to factor an HDR image into LDR pixelvalues for the LCD panel, as well as LDR intensities forthe low resolution LED array.

In this paper, we discuss algorithms to perform this sep-aration and to accurately reproduce photometric images.Achieving this goal entails designing efficient algorithms

440 M. Trentacoste et al. / J. Vis. Commun. Image R. 18 (2007) 439–451

to produce the best images possible; characterizing themonitor; and calibrating it to reproduce the most faithfulapproximation of appearance, compared to the inputimage. We evaluate our methods by comparing the outputimage to the input using perceptual models of the humanvisual system.

The remainder of this paper is structured as follows:Section 2 covers the topics related to the work presented.Section 3 describes the task of rendering images and detailsthe difficulties faced in doing so. Section 4 details the mea-surements required to correct for the actual hardware andcalibrate the output, and how those measurements areincorporated into the image processing methods. Section5 presents the results of the work, and evaluates them usinga perceptually-based metric.

2. Related work

2.1. Veiling glare and local contrast perception

Any analysis of the display of images includes an inher-ent discussion about the viewer: the perceptual makeup ofthe human observer. While the human visual system is anamazing biological sensor, it does have shortcomings thatcan be exploited for the purpose of creating display devices.One such shortcoming is that, while humans can see a vastdynamic range across a scene, they are unable to see morethan a small portion of it within a small angle subtended bythe eye. This inherent limitation, called veiling glare can beexplained by the scattering properties of the cornea, lens,and vitreous fluid, and by inter-reflection from the retina,all of which reduce the visibility of low contrast featuresin the neighborhood of bright light sources.

Veiling glare depends on a large number of parametersincluding spatial frequency, wavelength, pupil size as afunction of adaptation luminance [10], and subject age.While different values are reported for the threshold pastwhich we cannot discern high contrast boundaries, mostagree that the maximum perceivable local contrast is inthe neighborhood of 150:1. Scene contrast boundariesabove this threshold appear blurry and indistinct, and theeye is unable to judge the relative magnitudes of the adja-cent regions. From Moon and Spencer’s original work onglare [11], we know that any high contrast boundary willscatter at least 4% of its energy on the retina to the darkerside of the boundary, obscuring the visibility of the edgeand details within a few degrees of it. When the edge con-trast reaches a value of 150:1, the visible contrast on thedark side is reduced by a factor of 12, rendering detailsindistinct or invisible. This limitation of the human visualsystem is central to the operating principle of HDR displaytechnology, as we will discuss in the following section.

2.2. HDR display technology

In a conventional LCD display, two polarizers and aliquid crystal are used to modulate the light coming from

a uniform backlight, typically a fluorescent tube assembly.The light is polarized by the first polarizer and transmittedthrough the liquid crystal where the polarization of thelight is rotated in accordance with the control voltagesapplied to each pixel of liquid crystal. Finally, the lightexits the LCD by transmission through the second pola-rizer. The luminance level of the light transmitted at eachpixel is controlled by the polarization state of the liquidcrystal.

It is important to note that, even at the darkest state of aLCD pixel, some remaining light is transmitted. Thedynamic range of an LCD is defined by the ratio betweenthe light transmitted at the brightest state and the lighttransmitted in the darkest state. For a typical color LCDdisplay, this ratio is usually around 300:1. Monochromaticspecialty LCDs have a contrast ratio of 700:1, with num-bers exceeding 2000:1 reported in some cases. The lumi-nance level of the display can be easily adjusted bycontrolling the brightness of the backlight, but the contrastratio will remain the limiting factor. In order to maintain areasonable ‘black’ level of about 1 cd/m2, the LCD is thuslimited to a maximum brightness of about 300 cd/m2.Approaches such as the dynamic contrast advertised inrecent LCD televisions can overcome this problem to adegree and increase the apparent contrast across multipleframes. However, such methods can only adjust the inten-sity of the entire backlight for each frame displayeddepending on its average luminance, and provide no benefitfor static images or scenes without fast-moving action.

The fundamental principle of HDR displays is to use anLCD panel as an optical filter of programmable transpar-ency to modulate a high intensity but low resolution imageformed by a second spatial light modulator. This setupeffectively multiplies the contrast of the LCD panel withthat of the second light modulator such that global con-trast ratios in excess of 100,000:1 can be achieved [14]. Inthe case of an HDR display, each element of the rear mod-ulator is individually controllable, and together these ele-ments represent a version of the 2D input image.Currently, this second modulator consists of an array ofLEDs placed behind the LCD panel, as depicted in theupper left panel of Fig. 1. The array of LEDs is placedon a hexagonal grid for optimal packing, and the upperright panel of Fig. 1 demonstrates the LEDs of differentintensities in the hexagonal arrangement that forms thebacklight.

In order to ensure uniform illumination upon the LCD,the LED grid is placed behind a diffuser to blur the discretepoints into a smoothly varying field. This lower-frequencyillumination reduces artifacts caused by misalignment ofthe LCD and LED grid, and parallax from viewing the dis-play from indirect angles, which would be very difficult tocompensate for, and would perceptually be much morenoticeable than low frequency errors. The width of thepoint spread function (PSF) is quite large compared tothe spacing of the LEDs, as seen in the lower left panelof Fig. 1 which shows the point spreads of two adjacent

Fig. 1. (a) Arrangement of LED grid positioned behind the LCD panel. (b) Representation of hexagonal LED arrangement with different intensities. (c)Plot of the shape of an LED point spread, with another copy of the point spread positioned at the location of an adjacent LED. (d) Photograph of theBrightSide/Dolby HDR display.

M. Trentacoste et al. / J. Vis. Commun. Image R. 18 (2007) 439–451 441

LEDs. This overlapping of PSFs implies that not only isthe peak intensity of the display greater than any individualLED, but that any attempt to derive LED values from aninput image requires some form of deconvolution.

Local contrast from one pixel to its neighbor is limitedto roughly the contrast of the LCD panel (300:1), sincethe illumination pattern produced by the low resolutionLED array varies only minimally at the scale of the LCDpixel size. Perceptually, this limitation does not impairimage quality, since local contrast perception around anedge is limited to about 150:1 (Section 2.1). From the psy-chophysical theory mentioned in Section 2.1 we can estab-lish the largest possible spacing of the backlight LEDs for agiven viewing distance. Veiling glare is therefore central tothe operating principle of HDR displays, in that it allowsfor the use of a LED array with significantly reduced reso-lution compared to the LCD panel.

It is important to note that, as long as the local contrastis below the maximum contrast of the LCD panel, relative(and even absolute) luminance can be maintained, andedges can be reproduced at full sharpness. Only once thiscontrast range of the LCD panel is exceeded is some fidelitylost near high contrast boundaries, but this effect is belowthe detectable threshold, as has been verified in user studies[15]. The lower right panel of Fig. 1 shows an HDR displaybased on these principles, the BrightSide/Dolby DR37-P.This is the display we use in our experiments for this paper.

3. HDR image processing algorithms

3.1. Problem statement and reference algorithm

This section details the primary contribution of thepaper: algorithms for processing images to drive HDR dis-plays. We first discuss the overall challenge and formulate ahigh-level approach. Working from that method, we iden-tify practical algorithms that can be used to drive HDR dis-plays in real-time.

Given an image within the luminance range of the HDRdisplay, the goal of our work is to determine LED drivingvalues and an LCD panel image so as to minimize the per-ceptual difference between the input image and the one

formed by the display. This process must take into accountthe physical limitations of the display hardware, includingthe limited contrast of the LCD panel, the feasible intensityrange of the LEDs, and the finite precision of both due toquantization.

Fig. 2 shows a sample of the desired output of the algo-rithm. The LED backlight image is a low-frequency, black-and-white version of the input image and contains themajor features of the input. The LCD panel contains thecolor information and the high frequency detail, adjustedfor the backlight. Similar to the tone-mapping operatorof Chiu et al. [2], the panel has reverse gradients aroundlight sources to compensate for the light leaking acrossthe edge in the backlight. While this effect is undesirablein a tone-mapped image, it is beneficial when processingimages for display. These artifacts compensate for the blurinherent in the backlight, such that when the two are opti-cally combined, the result is close to the original. Since theinput image and the final output are HDR images, theyhad to be tone-mapped to be printed, while the LCD imageand the backlight are both 8 bit images and are showndirectly.

The image processing task we face can be framed as aconstrained non-linear optimization problem, where theobjective function is based on a complex metric of percep-tual image differences, such as the visible difference predic-tor [3] or its HDR counterpart [9]. Although this approachis possible, and in fact results in a high-quality referencesolution [16], it is not attractive in a practical setting. Whilethe visible difference predictor is a very powerful method ofcomparing two images, it is very slow (on the order of min-utes per image). It is therefore not feasible to evaluate it inreal-time on image sequences or live video streams on hard-ware that can be incorporate into a display.

Precomputation is also not an attractive option, sincethe processing is heavily parameterized on the characteris-tics of the HDR display, such as its luminance range andLED layout. We therefore desire an efficient algorithm thatcan process full frame images at 60 Hz either using thegraphics processing unit (GPU) of a control computer, orusing a signal processor or field-programmable gate array(FPGA) within the display.

Fig. 2. Left: original HDR image, tone-mapped for print. Center left: tone-mapped HDR image produced by the HDR display after processing with thealgorithms presented in this paper. Center right: low-frequency luminance image of the LED backlight used to produce this image. Right: thecorresponding LCD image compensated for the backlight. The top row represents a real-world example, while the bottom row shows a synthetic test casealong with the intensity distribution on a single scanline.

442 M. Trentacoste et al. / J. Vis. Commun. Image R. 18 (2007) 439–451

3.2. Real-time algorithm

In order to develop such an algorithm, we have to aban-don the global optimization and perceptual error metric infavor of more efficient local optimization and a less expen-sive error metric such as least squares. Not only is the fullglobal optimization across all LEDs and LCD pixels toocomputationally intense to be performed in real-time byGPUs and FGPAs, but the hardware architecture is ineffi-cient at computing linear systems. Instead, we frame thealgorithm as in image processing problem that is moreamenable to implementation on real-time hardware. How-ever, it is still important to ensure that the perceptual erroris small, which is not necessarily the case even if the meansquare error is small. For this reason, we verify all ouralgorithms by evaluating them on a set of test images withthe HDR visible difference predicator of Mantiuk et al. [9](see Section 5).

A very effective way of improving the performance is todetermine the LED driving values and the LCD panelimage in two separate stages. Suppose there is a way ofdetermining the LED values first. One can then use thehigh resolution LCD panel to compensate for the blurcaused by the low-resolution nature of the LED array. Spe-cifically, given the driving values for the LEDs, one cancompute the spatial light distribution B of the backlight,taking into account both the geometric layout of the LEDs,

and their PSFs. One can then compute target pixel values P

for the LCD panel as a pixel-by-pixel division of the targetimage �I by the backlight B

P ¼ f �1�IB

� �; ð1Þ

where f represents the physical response of the LCD panel,and the 8-bit control signal quantization, non-linear re-sponse function, and the inability to reproduce quantitiesof light greater or less than its dynamic range. Since theHDR display optically multiplies f(P) and B, the resultingimage I = f(P) Æ B is the closest possible approximation ofthe input image �I for the selected LED values.

The full algorithm thus consists of the following steps,also illustrated in Fig. 3:

(1) Given the desired image �I , determine a desired back-light distribution �B.

(2) Determine the LED driving levels d that most closelyapproximate �B.

(3) Given d, simulate the resulting backlight B.(4) Determine the LCD panel P that corrects for the low

resolution of the backlight B.

The individual stages are explained in detail in the fol-lowing section. Fig. 4 shows the tone-mapped image of

Fig. 4. Tone-mapped original HDR image for reference.

M. Trentacoste et al. / J. Vis. Commun. Image R. 18 (2007) 439–451 443

an input image for which we will demonstrate the results ofthe individual stages.

3.2.1. Target backlight

The first stage is to process the desired image �I and togenerate a target light distribution �B on the backlight.The input �I should be in photometric units, and its colorrepresentation should use the same chromaticity, whitepoint, and primaries as the HDR display. The output �Bis a black-and-white image in photometric units.

Treating the task of deriving the set of LED drivingintensities as a system of equations, we note that it is signif-icantly over-constrained, as the number of pixels in theinput image is greater than the number of LEDs that arebeing solved for. Over-constrained problems require morecomplicated and computationally intensive solver methodsto guarantee the solution best satisfies all the constraints.Instead, we choose to reduce the number of constraintsby downsampling the image so the resolution of the outputcorresponds to the resolution of the LED grid rather thanthe resolution of the LCD panel. The resulting number ofpixels in �B directly corresponds to the degrees of freedomavailable by controlling the LED intensities, simplifyingthe solution process, and reducing the computationalrequirements of the subsequent stages.

Determining the target backlight itself requires threesub-stages:

(1) Since the LCD panel only absorbs light, the array ofwhite LEDs needs to produce at least as much light asrequired to produce each individual color channel ateach point on the image plane. The target backlightluminance for each pixel is therefore set to the maxi-mum of all color channels for that pixel.

(2) A nonlinear function is applied to these target lumi-nance values in order to divide the dynamic rangeevenly between the LED array and the LCD panel,and to spread quantization errors uniformly acrossboth components. We experimentally verified that asquare root function works best, meaning that the

Fig. 3. Flowchart of stages of the implementation. The HDR input image is usdetermine the LED driving values. The actual backlight is simulated from the

LED array and the LCD panel are both responsiblefor producing roughly the square root of the targetintensity, such that the optical multiplication of thetwo values results in a good approximation of the tar-get image.

(3) The final step is to downsample the image to the res-olution of the LED grid. There are several ways toimplement this step. On the display FPGA, the stepis implemented as the average of neighborhoods ofpixels around LED positions. On a GPU the samealgorithm is implemented as recursive block averagesforming an image pyramid in order to work withinthe finite number of texture accesses available.

Fig. 5 shows the output of this stage: a monochrome,low-resolution sampling of square root of the originalimage.

3.2.2. Deriving LED intensities

The output from the previous stage is a target light dis-tribution for the backlight, which is already at the resolu-tion of the LED array. However, the pixels in that imagecannot be used directly as LED driving values, since theLEDs are significantly blurred optically. In fact, the pointspread function of each LED is roughly Gaussian inshape, with significant overlap of between the PSFs of

ed to determine a desired backlight configuration, which in turn is used tose driving values, and this simulation determines the LCD image.

444 M. Trentacoste et al. / J. Vis. Commun. Image R. 18 (2007) 439–451

neighboring LEDs (see Fig. 1). As discussed, this blur is theresult of a deliberate design decision at the level of thedisplay optics, since it avoids hard edges in the backlightillumination.

In order to derive LED driving values, d, we must there-fore compensate for the optical blur, which we do by imple-menting an approximate deconvolution of the targetbacklight with the PSF of the LEDs. This operation canbe described as minimizing the error of the linear system

mindkWd� �Bk2 ð2Þ

subject to the physical constraints on d, where W is a ma-trix describing the optical blur. More specifically, W con-tains the intensity of the PSF of each LED at each pixellocation, such that multiplying by a vector of LED intensi-ties will result in the simulated backlight. In order to effi-ciently find an approximate solution to this system, wemake use of the fact that this matrix is sparse, and band-diagonal. We therefore expect that the pixels in the targetbacklight �B are already fairly close to the driving valuesd, and we can obtain good results with a small computa-tional investment.

We chose one of the simplest iterative solvers, theGauss–Seidel method, on which to base our implementa-tion. The basic Gauss–Seidel iteration

dðkÞj ¼�Bj �

Pi<jwjid

ðkÞi �

Pi>jwjid

ðk�1Þi

wjjð3Þ

is the result of the reordering of the system in Eq. (2) andsolving for the unknowns dj. Every step, a new estimate d(k)

of the solution is chosen by comparing the current value ofthe system to the desired value. The new solution estimateis used to update the value of the system.

We make several modifications to this formulation tosuit our purposes. Instead of considering all other LEDsfor each LED, we use a smaller neighborhood NðdjÞ,and only perform a single iteration. The resulting computa-tion is a weighted average of the neighborhood of LEDs.Given a desired backlight image �B, it tries to account forlight contributions from other LEDs weighted accordingto PSF. By choosing dð0Þ ¼ �B, it collapses to

Fig. 5. Output of target backlight pass.

dj ¼�Bj �

PNðdjÞi wji

�Bi

wjjð4Þ

for a given LED j, where wjj is the value of the point spreadfor that LED, or simply the (PSF). Then, for a given LEDj, the desired luminance value of the backlight at its posi-tion is compared to the luminance coming from the sur-rounding LEDs. The value of LED j is chosen tocompensate for any disparity between the desired backlightand the illumination present. The results are clamped to[0,1] and passed to the subsequent simulation stage andthe LED controller in the display.

Fig. 6 shows the output of this stage. While it has alower resolution than the original input image (Fig. 4), itshows more contrast than the target backlight (Fig. 5) tocompensate for the optical blur.

3.2.3. Backlight simulation

At this point, the LED values have been determined,and the remaining stages are aimed at computing theLCD pixel values. To this end, we first need to determinethe actual light distribution B on the backlight at the fullresolution of the LCD panel. B should be similar to the tar-get distribution �B, but in general there will be minor differ-ences due to quantization and clamping of the drivingvalues, as well as approximations in the computation ofthose values (Section 3.2.2).

Simulating the actual light distribution B involves con-volving the driving values d with the LED point spreadfunction. On an FPGA, we directly evaluate each pixelby reading the value of the PSF for the distance to the cur-rent pixel from a lookup table (LUT) and modulate it bythe current driving value. On GPUs, we use a splattingapproach, and simply draw screen aligned quadrilateralswith textures of the PSF into the framebuffer. Each textureis modulated by its driving value and we use alpha blendingto accumulate the results. Since the PSF is very smooth,both methods can be implemented at a lower resolution,and the results can be upsampled.

Fig. 7 shows the output of this stage. If the LED valueshave been chosen appropriately, it should closely resemblethe image in Fig. 5 before downsampling, although a per-fect match is not expected due to quantization and clamp-ing of LED intensities to the physically feasible intensityrange.

3.2.4. Blur compensationThe final step in the algorithm is to determine the LCD

pixel values by dividing the input image by the simulatedbacklight, according to Eq. (1). The division is performedper pixel and per color channel. The resulting LCD pixelvalues are processed with the inverse response function ofthe LCD panel such that the LCD transparency is con-trolled linearly.

Fig. 8 shows the output of this stage. It displays thesame characteristics as the LCD panel image in Fig. 2.Since the result was obtained dividing by a low-frequency

Fig. 8. Output of blur correction pass.Fig. 6. Output of pass to determine LED intensities.

M. Trentacoste et al. / J. Vis. Commun. Image R. 18 (2007) 439–451 445

version of the original image, the LCD panel contains thesame reverse gradients as the work of Chiu et al. [2]. TheLCD image still contains all of the high-frequency andcolor information as the original image, but the low fre-quencies are damped, since they are generated by the back-light. Fig. 9 shows a tone-mapped reproduction of the finalHDR image produced on the HDR display resulting fromall the operations we have described in this section.

3.2.5. Discussion

The algorithm described in this section is fast enough forreal-time processing of large (HDTV resolution) highdynamic range images. It has been implemented in soft-ware, on GPUs, and on an FPGA chip that is integratedin the commercial displays by BrightSide Technologies/Dolby. It is in daily use in these implementations.

While the algorithm produces images of good qualityunder most circumstances, it systematically under-esti-mates the brightness of small bright regions with dark sur-roundings. This artifact is caused by the use of adownsampled image for determining the LED values (Sec-tion 3.2.2), as well as the approximate solution of Eq. (2). Itis worth pointing out that in natural images these artifactsare not easily perceptible. However, for more demandingapplications such as medical imaging, a more faithful,

Fig. 7. Output of backlight simulation pass. In this image, featurescorresponding to the LED positions are visible because the image islinearly scaled. These features are not visible when the physical display isviewed because of human lightness sensitivity.

albeit more computationally expensive, representationcan be desirable.

3.3. Error diffusion

We can alleviate these artifacts for demanding applica-tions by optimizing the LED intensities at full image reso-lution. As explained above, a full global optimization iscomputationally not feasible in real-time on the hardwareunder consideration, and therefore we developed a greedylocal optimization scheme that is inspired by the error dif-fusion algorithm for dithering [5]. Starting from the initialLED estimate (Section 3.2.2), we improve on this solutionby processing LEDs in scanline order, adjusting the inten-sity of each LED to minimize per-pixel error over the areaof influence for that LED.

Specifically, we minimize

minDdj

k�I � W jDdj � aBðj�1Þk2 ð5Þ

over the pixels within a local neighborhood of the jth LED,where Ddj is the change of intensity that needs to be appliedto the LED, and B(j�1) is the full-resolution, simulatedbacklight after the first j � 1 LEDs have been updated.Note the B(j�1) can be updated incrementally at reasonablecost: the full update of all LEDs is as expensive as the ini-tial backlight simulation.

Fig. 9. Tone-mapped simulation of results.

446 M. Trentacoste et al. / J. Vis. Commun. Image R. 18 (2007) 439–451

The parameter a in the above equation corresponds tothe average LCD pixel transmissivity in the range [0, 1].Error diffusion chooses a backlight that results in the max-imum number of LCD pixels having an average value of a.A value of 1

2(corresponding to the LCD at half the maxi-

mum intensity) produces the best final image quality, sinceit provides the distribution of LCD pixels most able tocompensate for the low-frequency of the backlight �B. Mov-ing that value closer to 1

2the LCD transmission will provide

the maximum room to correct for differences in luminancebetween the backlight and desired image. If the averagepixel value in a region is already close to full white (orblack), then additional local changes towards brighter (dar-ker) tones are limited in that region.

However, it is worth noting that there may be reasons tochoose other, especially larger, values for a. Since large val-ues of a correspond to a more transparent LCD panel, thesame display brightness can be achieved with lower LEDpower. Therefore, it is possible to trade image quality forpower savings, which may be interesting for mobile devices.Fig. 10 compares the final image produced by the error dif-fusion algorithm to that produced by the algorithm fromSection 3.2.2 for a simple test scene. It shows a significantimprovement in the reproduction of small bright featureswith error diffusion.

4. Measurement and calibration

The image processing algorithms described in the previ-ous sections require accurate descriptions of the geometricand optical properties of the HDR display. The quality ofthis data is of paramount importance. In fact, a full solu-tion of the LEDs and LCD pixels using approximate cali-bration data almost always looks worse than theapproximate solution using accurate calibration data.

Many attributes of the display must be measured toensure that the simulation results are correct. These includethe LCD panel response, the LED array alignment, thepeak luminance of each LED, as well as the LED pointspread function. All attributes related to light intensities

Fig. 10. Comparison of error diffusion to the original method as a 2D image anis the result of the error diffusion algorithm, and at the bottom the result of t

are measured in absolute units, which provide the neces-sary means of comparing the original image to the simu-lated result.

4.1. LCD panel response

First, we need to determine the non-linear response ofthe LCD panel. Most LCD panel controller circuitryapproximates a power function with an exponent of 2.5.The production of correct images requires compensatingfor this nonlinearity. To obtain the inverse, we follow thesame procedure as for LDR display calibration: we mea-sure the luminance of each of the LCD panel driving val-ues, and represent the inverse as a fitted function or byusing a lookup table (LUT). This calibration procedure isstandard for displays, and can be performed with standardtools. Since the LCD panel acts as a modulator, we do notneed to capture any absolute measurement of its response,and instead use a normalized function. The response of theDR37-P LCD panel is shown in Fig. 11 compared with anideal function with exponent of 2.5 mapped to the samedynamic range.

4.2. LED array alignment

Since the LEDs in a display are automatically mountedon a circuit board, the relative positioning of the LEDswith respect to each other does not deviate in any signifi-cant way from the construction plans. However, in the finalassembly, the misalignment between LCD panel and LEDarray is on average around 3 pixels. This offset is calibratedby examining the difference between the location of severalLED PSFs and the corresponding LCD pixel positions.

4.3. LED response

Due to the variance in LED construction and the cir-cuitry that supplies power, the response of the LEDs is nei-ther linear nor is it the same for each LED. Withoutcalibration, they do not respond linearly to driving values

d intensity profile. The top circle represents the target image, in the centerhe algorithm from Section 3.2.2.

0 50 100 150 200 250101

100

101

102

103

LCD intensity level

Lum

inan

ce (

cd/m

2 )

Measured data2.5 power

Fig. 11. Semilog plot of LCD panel response (in solid blue) comparedwith the response x2.5 (in dashed green). (For interpretation of thereferences to colour in this figure legend, the reader is referred to the webversion of this article.)

M. Trentacoste et al. / J. Vis. Commun. Image R. 18 (2007) 439–451 447

and they have different peak intensities. We measure thesedifferences with a Lumetrix IQCam [6], but multi-exposureHDR imaging could also be used [4,13].

4.4. LED point spread function

The point spread function (PSF) of the LEDs, previ-ously discussed in Section 2.2 and shown in Fig. 1, is themost critical parameter in accurately rendering images.The measurement procedure is straightforward. We turnon a single LED and take a high dynamic range image ofthe display with a calibrated camera like the LumetrixIQCam. Because of the variation in peak intensity ofLEDs, we normalize the measured data, and later multiplyit by the peak value computed from calibrating the individ-ual LED intensities and responses.

Several sources of measurement error can affect thequality of the image PSF. Artifacts can appear due to theLCD pixel spacing and camera photosite spacing, andnoise present in the HDR image. For these and other rea-sons, we do not use the measured image data directly, butinstead fit a function to it. The PSF is similar to a Gauss-ian, but has a wider tail, so we model it as the sum of sev-eral Gaussians of varying scales and widths. We recoverthese values by solving a least-square minimization prob-lem for the relative scales and widths of the componentGaussians.

1 Display manufacturers often employ various methods of distorting thecalculation of dynamic range, such as altering room illumination betweenmeasurements. The ANSI 9 checkerboard provides a standard measure ofthe usable display dynamic range, which we use to determine this number.

5. Evaluation

To evaluate the quality of the discussed algorithms, weprocessed a large number of HDR images with the displayparameters of a commercial HDR display, the BrightSide/Dolby DR37-P. We then inspected the images visually onthe actual display, and performed a quantitative analysisusing the HDR visual difference predicator (VDP) by Man-tiuk et al. [9]. Ideally, the comparison could be based on auser study instead of an algorithmic comparison. However,it is technically extremely challenging to create photometri-

cally accurate, artifact-free reference images for a largevariety of test images, so that we resort to the VDPapproach for now.

The display contains 1380 2.5 W LEDs on a 18.8 mmhexagonal close-packing matrix, where each LED is indi-vidually controlled over its entire dynamic range with 256addressable steps. The LCD panel is a 37 in Chi Mei Opto-electronics V370H1-L01 LCD panel with a 250:1 simulta-neous contrast ratio1 and 1920 · 1080 resolution. For afull white box occupying the center third of the screen,the maximum luminance is measured as 4760 cd/m2. Fora black image, the minimum luminance is zero, since allLEDs are off. The minimum luminance is less than 6 cd/m2 on a ANSI 9 checkerboard (the VESA contraststandard).

The HDR VDP takes both the original and displayedimages as input, and computes probabilities for the detec-tion of image differences in local neighborhoods tiling theimage. It works with absolute photometric units, and takesinto account properties of the human visual system, such asveiling glare (Section 2.1), contrast sensitivity for differentspatial frequencies, and non-linear luminance perception,among others. The output of the VDP can be somewhathard to interpret. For computational efficiency, the VDPimplementation only filters the images with respect to asubset of spatial frequencies and orientations. This approx-imation results in banded areas of detection which, uponfirst inspection, appear unrelated to the feature (seeFig. 12). The true probability distribution should be muchsmoother, and if all frequency bands and orientations wereused, these features would be wider and more evenlydefined. For our purposes, this is not a serious limitationsince we only desire to infer the existence of a perceivabledifference and its spatial extent and magnitude, instead ofits exact shape.

We visualize the detection probabilities as color codedregions on top of a black-and-white version of the image.Probabilities over 95% are marked solid red, probabilitiesbetween 75% and 95% are shown as a gradient from greento red, and probabilities below 75% are not colored. It isimportant to note that the VDP computes detection prob-abilities based on side-by-side comparisons of individuallocal image regions with a reference. As such it is a veryconservative measure for the practically more relevant sit-uation where a large image is presented without areference.

5.1. Evaluation results

In the evaluation of our methods, we compare the origi-nal image to a simulation of the luminance values outputby the display device. The measurements taken during

Fig. 12. Example of HDR VDP output (see text for explanation).

448 M. Trentacoste et al. / J. Vis. Commun. Image R. 18 (2007) 439–451

the calibration process provide absolute luminance data,and we make use of it to accurately simulate the luminancevalues produced by the display hardware. We used version1.2 of the HDR VDP software, with a simulated viewingdistance of 3.5 m, and defaults for the other parameters.

We have run our algorithms on a large number of HDRimages, from which we choose five test patterns and twophotographs as a representative sample set for discussionhere. Each set is presented in the same way: the originalimage is on top, the display output is in the middle, andthe VDP probability overlay is at the bottom. Since boththe original and displayed images are HDR, for printing

Fig. 13. Test pattern and frequ

in this paper they are first tone-mapped to 8 bits usingReinhard et al.’s photographic tone-mapping operator [12].

5.1.1. Test pattern

The left column of Fig. 13 shows a combination of sev-eral features at different frequencies. In the center are ver-tical and horizontal frequency gratings of differentspacings, while the horizontal white bars above and beloware linear gradients. There are solid rectangles on the left,and the outlined boxes on the right can be used to checkalignment of the display. The black level is set to 1 cd/m2

and the peak intensity is set to 2200 cd/m2. 1.42% of thepixels had more than a 75% probability of detection, while0.71% had more than a 95% probability of detection indirect side-to-side comparison. This image is a very difficultimage to reproduce correctly with the display hardware,especially on the right side near the outlined boxes. Severalof the issues, especially on the right, stem from the fact thatnone of the outlined boxes are big enough to produce therequired light intensity. The bright area is too small to havethe veiling glare obscure the excess backlight in the sur-rounding dark areas. The bars outside the vertical box,and the patches on the dark areas indicate that there istoo much backlight. The larger patches are the result ofthe backlight being too bright for a large area. They donot appear adjacent to the outline rectangles because thereis veiling glare to obscure the differences in those areas.

ency ramp sample images.

M. Trentacoste et al. / J. Vis. Commun. Image R. 18 (2007) 439–451 449

5.1.2. Frequency ramp

The right column of Fig. 13 consists of alternating whiteand black boxes of various widths and heights, reminiscentof some of the DCT basis functions used by JPEG images.Once again, the black level is set to 1 cd/m2 and the peakintensity is set to 2200 cd/m2. 1.15% of the pixels had morethan a 75% probability of detection while 0.79% had morethan a 95% probability. Considering the edge contrastsand feature sizes, the algorithm performs well, but showsthe common problem of failing to maintain peak intensitytowards edges of features. The red bars inside the white rect-angles indicate where the LCD panel switched to full whiteand caused a perceivable discontinuity. The red bars in thecorners of dark areas indicate excessive light being spilledfrom the two adjacent bright areas. As expected, the differ-ences become more visible for the higher frequency regionsin the top right, where the feature size is smaller than anLED. The hexagonal packing of the LED grid is aligned hor-izontally, so while thin horizontal features can be accuratelydepicted, thin vertical features will cause a saw-tooth likevertical pattern that is detectable under certain circum-stances. This orientation difference is why the error isdetected in the upper right, but not in the lower left wherethe same features are present at a different orientation.

5.1.3. ApartmentThe left column of Fig. 14 is the first of several photo-

graphs of real scenes, and depicts an indoor environment.

Fig. 14. Apartment and

The values are roughly calibrated to absolute photometricunits; the minimum luminance is 10�2 cd/m2 and the max-imum value is 1620 cd/m2. 0.26% of the pixels had morethan a 75% probability of detection while 0.16% had morethan a 95% probability. Compared to the test patterns, ithas noticeably less error. Most natural images do not con-tain such drastic contrast boundaries as the test patterns,and the result is fewer areas where the display is not ableto accurately represent the image. Most of the error is inthe small bright reflections on the balcony, or in the reflec-tion of the lamp in the TV. While the these differences arepredicted to be detectable in direct comparison, the imagequality produced by HDR display is very good (centerrow), and free of disturbing artifacts.

5.1.4. Moraine

The right column of Fig. 14 is a sample of an outdoorscene. Again, the values are roughly calibrated to absolutephotometric units. For this image, the minimum luminanceis 0.5 cd/m2 and the maximum value is 2200 cd/m2. Thisimage is an example of an image that is perfectlyrepresented on the display with 0.0% of the pixels had morethan a 75% probability of detection in side-by-side compar-ison. No boundaries are so extreme that we cannot accu-rately reproduce luminance and detail on both sides.

Finally, Fig. 15 includes an additional three HDRimages processed with our algorithm. For the Belgiumimage, 0.29% of the pixels had more than a 75% probabil-

Moraine test images.

Fig. 15. Additional image samples. Left: Belgium courtesy Dani Lischinski. Center: Fog courtesy Jack Tumblin. Right: Atrium courtesy KarolMyszkowski.

450 M. Trentacoste et al. / J. Vis. Commun. Image R. 18 (2007) 439–451

ity of detection while 0.17% had more than a 95% probabil-ity. For the Fog image, 0.12% of the pixels had more than a75% probability of detection while 0.07% had more than a95% probability. For the Atrium image, 0.22% of the pixelshad more than a 75% probability of detection while 0.14%had more than a 95% probability. These percentages arerepresentative of the quality of our algorithm’s reproduc-tion of images. Most of the errors observed are in specularhighlights, where we are not able to reproduce the lumi-nances of the image. These artifacts are the result of thelow-frequency nature of the backlight, where we cannotincrease the intensity of small features without adverselyaffecting the quality of surrounding regions.

The tests we have performed show that, using the imageprocessing algorithms presented in this paper, the represen-tation of natural images on the HDR display is very faith-ful to the original HDR image. While it is certainly possibleto construct test patterns resulting in detectable image dif-ferences, natural images do not usually exhibit this behav-ior. Furthermore, the fact that the VDP indicatesdifferences detectable in direct comparison with a referencemakes it a very conservative measure. Moreover, the differ-

ences introduced by the processing are not usually per-ceived as degrading the image quality. In a visualinspection of real-world images without a reference, evenexpert viewers miss a large percentage of the areas high-lighted by the VDP algorithm, even more so for animatedscenes where visible artifacts are extremely rare.

6. Conclusions

In this paper, we have presented algorithms for the accu-rate depiction of photometrically calibrated images ondual-modulator HDR displays. The steady increase inHDR imaging research has created a strong desire to dis-play the additional luminance information that those tech-niques provide. Display hardware with the potential tofulfill these needs is now available, but due to material lim-itations the images produced are not pixel-perfect copies ofthe original image. Instead, the displays make use of funda-mental limits in local contrast perception, that are well doc-umented in the psychophysics literature. The two imageprocessing algorithms we discussed in this paper are inactive use in the commercially available HDR displays,

M. Trentacoste et al. / J. Vis. Commun. Image R. 18 (2007) 439–451 451

and have been shown to several thousand people on vari-ous occasions, including trade shows.

As discussed in Section 3, we have not addressed thetopics of remapping images with pixel values outside thedisplayable space of the monitor. Hence, there is an oppor-tunity to improve tone-mapping techniques from very highdynamic range images to HDR images that the monitorsupports, and color space transformations given the extraconsiderations required over larger contrast ranges. Thesetopics and others are all aimed at more accurate colorappearance models, which are needed for the accurate dis-play of images. Fundamentally, all the same constraintsfound with LDR display systems still apply to HDR dis-plays, but have been loosened. Limits on peak intensity,feasible chromaticities, and other characteristics still exist.Research needs to be conducted in how well current prac-tices work on HDR displays and how they could, orshould, be improved.

Our evaluation with the HDR visible difference predica-tor shows that reproduction of natural images is very good,but limitations of both the hardware and the algorithmscan be detected on test patterns and under direct compar-ison with ground-truth images. Although very difficult toimplement in practice, in the future we would like to con-duct a formal user study with photometrically calibratedreference scenes. It would also be interesting to design auser study centered around perceived image quality insteadof difference with respect to a reference image.

References

[1] P.M. Acosta-Serafini, I. Masaki, C.G. Sodini, Single-chip imagersystem with programmable dynamic range, U.S. Patent 6,977,685,2005.

[2] K. Chiu, M. Herf, P. Shirley, S. Swamy, C. Wang, K. Zimmerman,Spatially nonuniform scaling functions for high contrast images, in:Proceedings of Graphics Interface 1993 (1993) 245–253.

[3] S. Daly, The visible differences predictor: an algorithm for theassessment of image fidelity, Digital images and human vision (1993)179–206.

[4] P. Debevec, J. Malik. Recovering high dynamic range radiance mapsfrom photographs, in: SIGGRAPH 1997: Proceedings of the 24thAnnual Conference on Computer Graphics and Interactive Tech-niques, ACM Press/Addison-Wesley Publishing Co., New York, NY,USA (1997) 369–378.

[5] R. Floyd, L. Steinberg. An adaptive algorithm for spatial grey scale,in: Proceedings of the SID International Symposium, Digital Tech-nology Papers (1975) 36–37.

[6] Lumetrix. IQCam imaging photometer, 2005. http://www.lumetrix.com.

[7] R. Mantiuk, A. Efremov, K. Myszkowski, H. Seidel, Backwardcompatible high dynamic range MPEG video compression, ACMTransactions on Graphics (special issue SIGGRAPH 2006) 25 (3)(2006) 713–723.

[8] R. Mantiuk, G. Krawczyk, K. Myszkowski, H. Seidel, Perception-motivated high dynamic range video encoding, ACM Transactions onGraphics (special issue SIGGRAPH 2004) 23 (3) (2004) 733–741.

[9] R. Mantiuk, K. Myszkowski, H. Seidel. Visible difference predicatorfor high dynamic range images, in: Proceedings of IEEE InternationalConference on Systems, Man and Cybernetics (2004) 2763–2769.

[10] P. Moon, D. Spencer, Visual data applied to lighting design, Journalof the Optical Society of America 34 (605) (1944).

[11] P. Moon, D. Spencer, The visual effect of non-uniform surrounds,Journal of the Optical Society of America 35 (3) (1945) 233–248.

[12] E. Reinhard, M. Stark, P. Shirley, J. Ferwerda, Photographic tonereproduction for digital images, ACM Transactions on Graphics(special issue SIGGRAPH 2002) 21 (3) (2002) 267–276.

[13] M. Robertson, S. Borman, R. Stevenson, Dynamic range improve-ments through multiple exposures, in: Proceedings of InternationalConference on Image Processing (ICIP) 1999 (1999) 159–163.

[14] H. Seetzen, W. Heidrich, W. Stuerzlinger, G. Ward, L. Whitehead,M. Trentacoste, A. Ghosh, A. Vorozcovs, High dynamic rangedisplay systems, ACM Transactions on Graphics (special issueSIGGRAPH 2004) 23 (3) (2004) 760–768.

[15] H. Seetzen, L. Whitehead, G. Ward, A high dynamic range displayusing low and high resolution modulators, in: Society for InformationDisplay International Symposium Digest of Technical Papers (2003)1450–1453.

[16] M. Trentacoste, Photometric Image Processing for HDR Displays.Master’s thesis, The University of British Columbia (2006).

[17] G. Ward, A wide field, high dynamic range, stereographic viewer, in:Proceedings of PICS 2002, April 2002.

[18] G. Ward, M. Simmons, Subband encoding of high dynamic rangeimagery, in: APGV 2004: Proceedings of the First Symposium onApplied Perception in Graphics and Visualization, ACM Press, NewYork, NY, USA (2004) 83–90.