Image statistics for surface reﬂectance...

1TtaiLcinwliglb

fsmbofaabettsc

n

846 J. Opt. Soc. Am. A/Vol. 25, No. 4 /April 2008 Sharan et al.

Image statistics for surface reflectance perception

Lavanya Sharan,1,* Yuanzhen Li,2 Isamu Motoyoshi,3 Shin’ya Nishida,3 and Edward H. Adelson2

1Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology,77 Massachusetts Avenue, Cambridge, Massachusetts 02139, USA

2Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, 77 Massachusetts Avenue,Cambridge, Massachusetts 02139, USA

3Human and Information Science Laboratory, NTT Communication Science Laboratories, Nippon Telegraph andTelephone Corporation, 3-1, Morinosato–Wakamiya, Atsugi, Kanagawa 243-0198, Japan

*Corresponding author: [email protected]

Received August 3, 2007; revised November 27, 2007; accepted January 11, 2008;posted February 6, 2008 (Doc. ID 85917); published March 13, 2008

Human observers can distinguish the albedo of real-world surfaces even when the surfaces are viewed in iso-lation, contrary to the Gelb effect. We sought to measure this ability and to understand the cues that mightunderlie it. We took photographs of complex surfaces such as stucco and asked observers to judge their diffusereflectance by comparing them to a physical Munsell scale. Their judgments, while imperfect, were highly cor-related with the true reflectance. The judgments were also highly correlated with certain image statistics, suchas moment and percentile statistics of the luminance and subband histograms. When we digitally manipulatedthese statistics in an image, human judgments were correspondingly altered. Moreover, linear combinations ofsuch statistics allow a machine vision system (operating within the constrained world of single surfaces) toestimate albedo with an accuracy similar to that of human observers. Taken together, these results indicatethat some simple image statistics have a strong influence on the judgment of surface reflectance. © 2008 Op-tical Society of America

OCIS codes: 330.0330, 330.4060, 330.5000, 330.5510, 330.7310.

natrcit

ttfsgtpaggiccm

foftuntmt

. INTRODUCTIONhe albedo of a surface is a measure of its diffuse reflec-ivity. Perceived albedo is known as “lightness,” and thebility to judge albedo with some accuracy, despite chang-ng viewing conditions, is known as “lightness constancy.”ightness constancy is not perfect, especially in extremeonditions such as those arranged by Gelb [1]. When andeal matte, planar surface is viewed in isolation, one can-ot determine its albedo. A black surface may be seen ashite, an illusion known as the Gelb effect. Here surface

uminance is the only relevant stimulus parameter, sincellumination and albedo are confounded (they multiply to-ether to produce the observed luminance). Therefore,ightness constancy is poor due to the lack of any disam-iguating information from the context.With nonideal surfaces, the Gelb demonstration fails. It

ails badly for complex surfaces such as stucco [2], ashown in Fig. 1. The two stucco images have the sameean luminance and are surrounded by the same dark

ackground, yet one looks darker and glossier than thether. The interreflections and specularities in these sur-aces seem to provide extra information, and observersre evidently able to utilize some of this information tochieve lightness constancy. Gilchrist and Jacobsen builtoxes containing miniature rooms and painted them withither black or white matte paint [3,4]. Observers viewedhe rooms, one at a time, through a small aperture, sohey were immersed in a field of uniform reflectance. Ob-ervers could tell which room was which, presumably be-ause of differences in interreflections.

Most research in surface perception has been domi-ated by the case of smooth, Lambertian patches and pla-

1084-7529/08/040846-20/$15.00 © 2

ar 3-D configurations, such as those shown in Figs. 2(a)nd 2(b) [1,5–14]. Existing theories of lightness percep-ion have no way of predicting the effects that occur witheal-world surfaces such as stucco. However, several re-ent studies, including ours, have examined stimuli thatncorporate some of the complexity of real-world condi-ions [Figs. 2(c) and 2(d)] [2–4,15–23].

In the fields of computer graphics and computer vision,here has been an interest in characterizing the bidirec-ional reflectance distribution function (BRDF), which is aull description of the reflectance properties of an opaqueurface [24]. Estimating BRDF of surfaces from photo-raphs is a challenging machine vision problem. Work inhese fields has mainly followed the inverse optics ap-roach, aiming to recover such a full model of 3-D layoutnd illumination of the scene as being consistent with aiven 2-D image. This is an impossibly difficult problemiven the many-to-one mapping from 3-D scenes to a 2-Dmage. Therefore, existing algorithms require additionalonstraints or assumptions that go far beyond those in-luded in a single picture such as Fig. 1 [25–35]. Theechanism of human perception must be different.We have taken up the lightness perception problem

rom several points of view. First, we ask how well humanbservers can judge albedo when viewing isolated sur-aces using materials such as stucco. Second, we proposehat simple statistics of the 2-D image of a surface can besed in a cue-based approach to lightness perception. Fi-ally, we show that manipulating these candidate statis-ics in an image alters human judgments in a predictableanner. Our approach is to be contrasted with inverse op-

ics approaches, which depend on the estimation of the

008 Optical Society of America

pammeaThigfnn[lut

onf

tcwgswgnp

ptstolopbrmtstihotaB

2LIafpepca

FGb

FWflis

Sharan et al. Vol. 25, No. 4 /April 2008 /J. Opt. Soc. Am. A 847

arameters of an internal model that can explain the im-ge data in detail. There is evidence that inverse opticsodels can be useful in understanding some human judg-ents. However, in real-world scenes, the surface geom-

try, illumination distribution, and BRDF are too complexnd too uncertain for inverse optics to have much success.herefore, it is reasonable that the visual system will useeuristics based on statistical cues when these cues are

nformative. The importance of image statistics was sug-ested in a study by Nishida and Shinya [15], whichound that reflectance perception of non-Lambertian andonsmooth surfaces was critically dependent on the lumi-ance histogram of the 2-D image of a surface. Dror et al.16,17] studied the appearance of spheres in real-world il-umination and found that simple image statistics wereseful in characterizing the reflectance properties of syn-hetic and natural spheres.

We consider lightness perception for photographs ofpaque surfaces viewed in isolation, with the mean lumi-ance scaled to a constant value for all surfaces. Our sur-aces have significant mesostructure so that shading, in-

ig. 1. (a) In the Gelb demonstration a smooth Lambertian blacelb effect fails for complex surfaces. The stucco samples have thlack.

ig. 2. Stimuli used for studying reflectance perception: (a)allach’s disc annulus displays, (b) Mondrian-like displays with

at Lambertian surfaces, (c) Fleming et al.’s simulated spheresn complex real-world illumination [18], (d) simulated locallymooth bumpy surfaces used by Nishida and Shinya [15].

erreflection, and specular highlights become significantomponents of the appearance. In our previous work [2]e reported that the skewness of the luminance histo-ram is correlated with the albedo and gloss of real-worldurfaces. Human judgments of lightness and glossinessere also correlated with luminance skewness. We sug-ested that this statistic can be easily computed by earlyeural mechanisms and found an aftereffect that sup-orts this hypothesis.In the present work, we took a more computational ap-

roach to the problem, focusing in detail on the statisticshat are associated with lightness. We evaluated the ab-olute effectiveness of a variety of image statistics andheir combinations in estimating the physical albedo notnly from correlations, but also from how well machineearning algorithms can tell light and dark surfaces basedn those statistics. The results suggest that moment andercentile statistics of the luminance histogram and sub-and histograms are informative. Although learning algo-ithms cannot predict lightness perfectly, their perfor-ance is similar to that of human observers. In addition,

he pattern of errors made by the algorithms was veryimilar to that of human errors. On changing these sta-istics of images, human judgments were affected accord-ngly. These findings suggest that human observers useistogram statistics for lightness estimation. Finally, inrder to manipulate the subband histograms in additiono the luminance histogram [2] without introducing imagertifacts, we developed a modification to the Heeger–ergen texture synthesis algorithm.

. IMAGE STATISTICS AS CUES TOIGHTNESS

n this section, from the viewpoint of ecological optics, wenalyze how simple statistics of the 2-D image of a sur-ace tell us about surface reflectance properties. Our ap-roach is similar to that of Dror et al. [16,17], who consid-red images of smooth, shiny spheres rendered orhotographed under complex, real-world illuminationonditions. They measured moment (2nd, 3rd, and 4th)nd percentile (10th, 50th, and 90th) statistics of pixel in-

ce can look the same as a smooth Lambertian white surface. (b)e mean luminance, yet it is easy to tell the white stucco from the

k surfae sam

tsTfc

sedgmastt

AWssffptspltACicmAd

(tcrcF

nbsa

Fw2lscffsoi


ensities and wavelet coefficients on the surface of thepheres via a cylindrical projection of the 2-D image.hese statistical measurements were found to be useful

or classifying the spheres into shiny, matte, white, gray,hrome, etc.

Like Dror et al., we are interested in identifying imagetatistics that are diagnostic of surface reflectance prop-rties. However, we operate under a set of assumptionsifferent from theirs. We do not assume a known surfaceeometry; rather we allow our surfaces to possess 3-Dedium-scale structure. We consider surfaces in simple,

rtificial illumination conditions, and while some of oururfaces are glossy (non-Lambertian), we focus on statis-ics that are predictive of albedo, i.e., the diffuse reflec-ance component.

. Image Datae gathered high-dynamic-range color photographs of

everal real-world surfaces, such as paper, candies, cloth,tucco, etc. (Fig. 3). Opaque surfaces with spatially uni-orm reflectance properties were selected so that each sur-ace is associated with a unique albedo value. We usedlanar surface samples with medium-scale surface struc-ure or surface mesostructure [36]. While we allowed theurfaces to be specular, we studied only the diffuse com-onent. Surfaces were photographed under three indoorighting conditions [see Figs. 4(a)–4(c)]. The specifics ofhe camera and lighting setup are provided in Fig. 4(d).ll images were acquired in RAW 12-bit format by aanon EOS 10D camera. The RAW images were linear-

zed using “dcraw” software [37]. The linearization pro-ess converts the pixel intensities in a RAW image to theeasured luminance up to a multiplicative scaling factor.ppendix C contains details of the linearization proce-ure.Our surfaces were orange, yellow, red, white, or black

Fig. 3). We used 30 surfaces of various shapes and reflec-ance properties. As we are interested in lightness, allolor photographs were converted to gray scale by sepa-ating the color channels. For colored surfaces, individualolor channels were treated as distinct gray-scale images.igures 4(e) and 4(f) show an example of the color compo-

Fig. 3. Examples of surfaces in our data set. All surfaces sh

ents of an orange surface. The blue channel looks like alack surface, while the red channel looks like a whiteurface. This happens because, for orange colored materi-ls, the different colors of light are reflected in different

ere were photographed under an overhead fluorescent light.

ig. 4. Image data acquisition. Three indoor lighting conditionsere used: (a) light 1, overhead fluorescent light source; (b) light, focused halogen spotlight; (c) light 3, diffuse tungsten halogenamp. (d) Schematic layout of the setup. Two views, one from theide and one from front, are shown. (e), (f) Red and blue coloromponents of an orange surface look like white and black sur-aces, respectively. (g) Ground truth is acquired using a uni-ormly illuminated flat material sample and a standard whiteurface. A user clicks on two regions, one on the sample and onen the standard. The ratio of mean luminance of the two regionss used to calculate the albedo for each color channel.

own h

wsv

tiimgs[mstf�

goa4bagfr

srs

BWoaatfdtMTahfidamtht

g

F(t


ays. Thus, we can acquire photographs of surfaces thathare identical geometry and illumination conditions butary in their reflective properties.

Many of our materials exhibit a strong specular reflec-ion component. One example is the crumpled black papern Fig. 3. In order to capture such materials with a lim-ted dynamic range camera, we used the technique of

ultiple exposure imaging. Multiple exposure photo-raphs of the same scene were combined using HDRShopoftware to produce a single high-dynamic-range image38]. As a final step, all images were multiplicatively nor-alized to have the same mean image luminance. This

tep is essential, because we want to know which statis-ics of an image, other than mean luminance, are usefulor reflectance perception. In total, we had 30 materials3 lighting conditions �3 color channels�270 images.For all the surfaces in our data set, we acquired the

round truth for diffuse reflectance. A smooth, flat samplef each surface, devoid of any mesostructure, was selectednd placed next to a standard white surface [see Fig.(g)]. For handmade surfaces, we prepared a flat sampley hand. For other surfaces, we used the flattest samplesvailable. Both the standard and the sample were photo-raphed under uniform illumination conditions. The dif-use reflectance of the sample was calculated by using theatio of the linearized intensity in a region containing the

ig. 5. Luminance histograms of light and dark materials exhibb), (c) the respective luminance histograms. (c), (f) Standard devhe ground truth for albedo for all the surfaces in our data set. A

ample to that of a region containing the standard. Theegions in the photograph were selected carefully to avoidhadows and highlights.

. Statistics of the Luminance Histograme studied the luminance histograms for the images in

ur data set and found that histograms of light (high-lbedo) and dark (low-albedo) materials (most materialsre non-Lambertian and nonsmooth) display characteris-ic differences. The luminance histograms for dark sur-aces tend to have higher Michelson contrast (standardeviation divided by the mean) and have longer, positiveails. For lighter surfaces, the histograms have lowerichelson contrast and are usually symmetric (Fig. 5).hese differences can be attributed to ways in which lightnd dark surfaces interact with light. Light surfaces haveigher diffuse reflectance; therefore light bounces aroundlling up the shadows, leading to a lower contrast thanark surfaces. If a light and a dark surface have the samemount of specular reflection, the specular highlights areore visible in the darker surface owing to higher con-

rast. Therefore, contributions from interreflections andighlights lead to different shapes for the luminance his-ograms of light and dark surfaces.

These systematic differences in the luminance histo-rams can be captured by a host of statistical measures—

ematic differences. (a) Light modeling clay; (d) dark stucco; andand skewness of the log-luminance histogram is plotted againsts pertain to the overhead fluorescent lighting condition.

it systiationll plot

ms[swwettWntft

lssu

ntspfir1aweoslRw[nys(

fatioflafF6tpltotant

ttlsusf

sasiIij

CAtovusspags

tctlbtoastaptac

srtHoa

DGwramRsbo


oment (standard deviation or skewness) or percentiletatistics (10th, 50th, or 90th). In our companion paper2], we had focused on one statistic, skewness or the thirdtandardized moment of the luminance histogram, ande showed that skewness is correlated with albedo asell as gloss. In the present study, from the viewpoint ofcological optics, we consider a wide range of statisticshat are correlated with albedo, such as standard devia-ion and percentiles of the original and filtered images.e compute these statistics either directly on the lumi-

ance values or on the log of luminance values. We foundhat statistics of the log-luminance histogram are distinctor light and dark surfaces, just as for the luminance his-ogram.

In the rest of this paper, we will discuss the results ofog-luminance analysis. In our experience, the conclu-ions remain the same for luminance and log-luminancetatistics. This is trivially true for order statistics that arenchanged by a log transformation.In Figs. 5(c) and 5(f), the standard deviation and skew-

ess of the log-luminance histogram is plotted against therue albedo of a surface. The dependence of the momenttatistics on the physical property can be seen in theselots. Receiver operating characteristic (ROC) analysis of-ers another way of visualizing this correlation. The ROCs a plot of the true positive rate versus the false alarmate of a binary classifier. A perfect classifier achieves00% classification accuracy with a 0% false alarm rate,nd the area under the ROC curve (AUC) is 1.0. For theorst classifier (unbiased coin flip), the true positive ratequals the false alarm rate (AUC�0.5). In our case, if anyf the statistics—90th percentile, standard deviation, orkewness of log-luminance—is used to classify surfaces asight (physical albedo �0.5) or dark (albedo �0.5), theOC curves lie somewhere in between the ideal and theorst classifier curves (AUC ranges from 0.73 to 0.77)

Fig. 6(a)]. The performance of moment statistics is sig-ificantly above chance, implying that both statisticsield useful information about albedo. Other percentiletatistics (10th and 50th) also have similar ROC curvesAUC�0.69 and 0.77, respectively).

While ROC analysis considers the utility of a statisticor binary reflectance classification, we can use regressionnalyses to see how well the statistics estimate reflec-ance. Figure 6(b) shows that a linear regression fit is annadequate model for the relationship between skewnessf log-luminance and albedo. Similar plots were obtainedor other moment and percentile statistics of log-uminance. We also conducted nonlinear regressionnalyses in order to model the data in Fig. 6(b) better. Weound that applying a log transformation to the axes ofig. 6(b) leads to a somewhat improved linear fit [Fig.(c)]. As skewness and albedo are dimensionless quanti-ies, applying a log transformation does not change thehysical significance of our results. Figure 6(d) shows theog–log plot for the 90th percentile of log-luminance his-ogram and albedo. Visualizing the relationship betweenur statistics and albedo is easier after applying the logransformations. However, as the r2 statistic in Figs. 6(c)nd 6(d) indicates, these nonlinear transformations doot capture the dependence of statistics on albedo en-irely. We did not use more complex models to nail down

he behavior of our data because of the danger of overfit-ing. We have only 30 materials in our data set. While theinear fits in Figs. 6(b)–6(d) are not perfect, they are stilltatistically significant. Therefore, our statistics containseful information, although the relationship betweentatistics and diffuse reflectance is not entirely straight-orward.

It is important to emphasize that the statistics de-cribed thus far are predictive of albedo as long as the im-ges on which they are computed look like surfaces. Theame statistics are of no use when measured on arbitrarymages that are not associated with a value of albedo [2].ndeed if we pixel scramble our images, thereby destroy-ng the perception of a surface, the luminance statisticsust described remain unchanged.

. Statistics of Subband Histogramss luminance (or log-luminance) statistics are insensitive

o spatial structure, we examined the statistics of filterutputs next. This is a reasonable thing to do because theisual system is more likely to have access to filtered val-es than to raw luminance values. We used center–urround and oriented edge detection filters in a multi-cale decomposition [39,40]. In Fig. 7, one observes thatixel histograms of filtered images look different for lightnd dark surfaces. For dark surfaces, filter output histo-rams have heavier tails and, in the case of center–urround filtering, the outputs are also skewed.

The filters amplify the local contrast differences be-ween white and black surfaces. The skewness of theenter–surround filter outputs is presumably related tohe asymmetry in the distribution of shadows and high-ights in natural images. Unlike shadows, which tend toe spread over a larger image region, specular highlightsend to be small and concentrated. It is likely that filterutput statistics are affected by these characteristicsymmetries. Figures 8(a) and 8(b) plot ROC curves forubband statistics—standard deviation and 90th percen-ile. Individual statistics fare much better than chancend hence are predictive of albedo. Figures 8(c) and 8(d)lot the log of the 90th percentile of filter outputs againsthe log of albedo. The linear regression fits in Figs. 8(c)nd 8(d) demonstrate that the filter statistics are highlyorrelated with albedo.

In our experiments, we also found that the skewnesstatistic is sensitive to the choice of filters and filter pa-ameters. Center–surround filters are somewhat betterhan oriented filters at the task of skewness detection.owever, in terms of albedo prediction, all statistics, both

f log-luminance and the two kinds of filters, performbout equally well.

. Combining Statisticsiven that individual statistics can predict albedo fairlyell, it is interesting to ask how the statistics perform

elative to one another. In Figs. 6 and 8 we observe thatll statistics—moments and percentiles derived from lu-inance or filter outputs—perform about equally well inOC tests and regression analyses. Not only do all thetatistics predict albedo with the same degree of success,ut we found that they are also correlated with one an-ther. Consider Fig. 9(a): At first glance, it is not clear

wpstpstt

ptdTob

ass9

fttsgrttlo

Fapiac


hy the standard deviation of log-luminance and the 90thercentile statistic of the center–surround filter outputshould covary. We performed chi-square independenceests and mutual information values to confirm these em-irical correlations. In our previous work [2], we had ob-erved that the skewness of filter outputs is highly,hough not completely, correlated with the skewness ofhe luminance histogram for images like those in Fig. 3.

In a sense, it is not surprising that the statistics are de-endent on one another. We noted earlier that shapes ofhe histograms of log-luminance and filter outputs haveistinctly different shapes for light and dark surfaces.hese characteristic differences can be captured in vari-us ways by various statistics. The surfaces with lower al-edo have higher local contrast as well as more structure

ig. 6. (a) ROC curves for the 90th percentile, standard deviatiobove chance at the task of classifying surfaces as light or darkhysical diffuse reflectance. A linear regression model is a poor fitmproves the fit of the linear regression model (p�0.05, r2=0.34gainst the log of the diffuse reflectance. The linear fit is still notonditions for all plots in this figure.

t higher frequencies. Therefore, the statistics that mea-ure contrast (e.g., standard deviation) covary with thetatistics that measure energy in higher frequency (e.g.,0th percentiles) for each surface.In spite of the high degree of correlation, we found that

or purposes of albedo estimation, combining a few statis-ics is better than using just one. We used a support vec-or regression technique with a linear kernel to combinetatistics [41]. The image data set was divided into threeroups, one of which was chosen as the training set. Theegression technique learned a linear relationship be-ween a chosen set of statistics (features) measured onhe training set and the ground truth values for albedo. Ainear kernel was chosen for simplicity. The � parameterf the regression was set to 0.1, and the penalty param-

skewness of log-luminance values. These statistics perform wellkewness of the log-luminance histogram is plotted against the05, r2=0.27). (c) Applying a log transformation to both axes of (b)og of 90th percentile of the log-luminance histogram is plottedood (p�0.05, r2=0.29). Statistics were pooled across all lighting

n, and. (b) S(p�0.). (c) Lvery g

ettatt(bfrtp

euftt

ttlcotwfmco

gnitao

FL(t


ter C was chosen by fivefold cross validation on theraining set. Once the regression parameters are learned,he technique can provide an estimate of the albedo forny new image of a surface. It is important to clarify thathe regression technique fits a linear model to our statis-ics to predict the albedo. Even though we noted earlierFigs. 6 and 8) that a linear model can predict “log(al-edo)” from “log(statistic)” somewhat better than albedorom statistic, we did not use log transformations in ouregression model. This is because the increase in predic-ion performance is not enough to justify the added com-lexity of the regression model.Figures 9(b)–9(d) show the outputs of three linear mod-

ls that differ in the number and type of statistics theyse to predict albedo. In these figures we see that the per-ormance of a linear model improves by using two statis-ics instead of one. However, on adding any more statis-ics, the gains to be made are not significant. As we tried

ig. 7. Pixel histograms of filtered images look different for whiaplacian of Gaussian filter (�=0.5 of size 5�5 pixels); (c) outpu

f) LoG and Sobel filter outputs; (g) pixel histograms of images ino the overhead fluorescent lighting condition.

o incorporate more than two or three statistics as fea-ures in our models, the correlation among the statisticsed to saturation in performance. We found that the pre-ise choice of features (moments or percentiles, luminancer filter statistics) or the exact number of features (two,hree, or four) is not too critical. So, for the rest of thisork, we will use a fixed linear model (henceforth re-

erred to as the “model” or “regression technique”). Theodel uses three statistics—standard deviation, 10th per-

entile, and 90th percentile of the center–surround filterutput.

In Fig. 10, the output of the model is compared to theround truth for albedo. We see that the regression tech-ique is not perfect at estimating the physical albedo, but

t does a fairly good job. Interestingly, we found that theechnique makes larger errors on surfaces that are flatnd nearly Lambertian than those with more complex ge-metry and reflectance properties. This performance is

black surfaces. (a) Light modeling clay from Fig. 5; (b) output ofrizontal Sobel filter (3�3 pixels); (d) dark stucco from Fig. 5. (e),

(e); (h) pixel histograms of images in (c) and (f). All plots pertain

te andt of ho(b) and

stdp

3WtGoiL

ATsim5

imt

BTwflap17lw�lsban

FpLl


imilar to human observers (Fig. 1). To study the correla-ion between statistics and perception better, we con-ucted psychophysical experiments to measure humanerformance on our images.

. EXPERIMENT Ie asked human observers to rate the lightness for all

he photographs in our data set in a context similar to theelb conditions of Fig. 1. From our informal “anti-Gelb”bservations in Fig. 2, we know that such a task is mean-ngful. It is easy to judge the lightness of rough non-ambertian surfaces in isolation.

. Stimulihe image data of the previous section were used astimuli for this experiment. The mean luminance equal-zed images were displayed on a gamma-corrected LCD

onitor. The images were displayed at a resolution of12�512 pixels against a middle gray background. The

ig. 8. ROC curves for statistics of filter outputs: (a) 90th percercentile and standard deviation for Sobel filter output are signoG and Sobel filter outputs, respectively, against the log of phy

ines �p�0.05�; r2 values are 0.63 and 0.62 for (c) and (d), respec

ntensity of the background was set to the mean image lu-inance; thus both variables do not change throughout

he experiment.

. Apparatushe three indoor light sources [see Figs. 4(a)–4(d)] thatere used to photograph our surfaces were an overheaduorescent lamp (Kino Flo Diva Lite 200) placed 60 cmbove the sample surface, a halogen spotlight (LTM Pep-er 300 W Quartz-Fresnel) placed 175 cm away and20 cm above, and finally a light box (Lowel Rifa 66,50 W tungsten halogen lamp) that produced diffuse, softighting 120 cm away from the surface. The LCD monitoras a Dell 20.1 in. flat panel �1 in.=2.54 cm� at 12801024 resolution, 75 Hz frame rate, and 70 cd/m2 mean

uminance. To obtain the ground truth [see Fig. 4(g)] thetandard white surface was chosen from the Gretag Mac-eth Color Checker chart. Light meters Sekonic L-608nd Minolta CS-100 were used to ensure uniform illumi-ation.

skewness, and standard deviation of LoG filter output; (b) 90thly above chance; (c), (d) plot the log of the 90th percentile of theiffuse reflectance. The linear regression fits are shown as black

entile,ificantsical dtively.

bw(

btslsts[tMbgi

COpMsalmwetftol

DTsn

EWattOtnsctsoatscrafli

Ffobt

Fp�aMosotccraonf

Fgmp


Observers viewed the LCD monitor in a dark room. Aox enclosing the two light sources and a Munsell chartith standard surface patches served as the reference

see Fig. 11). The box was constructed from white foam

ig. 9. (a) Standard deviation of log-luminance and the 90thercentile of LoG filter output are correlated (r=0.6237, p0.05). (b), (c), (d) Outputs of three linear models are plotted

gainst the true diffuse reflectance for a subset of our surfaces.odel A uses one statistic, the standard deviation of LoG filter

utputs to predict the albedo of a surface. Model B uses twotatistics—standard deviation and 10th percentile of LoG filterutputs. Model C uses three statistics—standard deviation andhe 10th and 90th percentiles of LoG filter outputs. For all threeases, the model ratings were averaged over all three lightingonditions. The error bars indicate the minimum and maximumatings. If the models were perfect at predicting physical albedo,ll points would lie along the black line with slope�1. The slopesf the best fit lines are indicated in each plot. The asterisk de-otes statistical significance �p�0.05�. The r2 statistic is similaror all plots—0.42 for (b), 0.38 for (c), and 0.40 for (d).

ig. 10. Output of the regression technique is plotted versusround truth for diffuse reflectance (albedo). Bars indicate maxi-um and minimum ratings. If the technique were perfect, all

oints would lie along the diagonal.

oard panels and covered with dark gray craft paper onhe outside. One side of the box was left open to allow ob-ervers to view the Munsell chart. Compact fluorescentight bulbs of color temperature 5500 K (SunWave fullpectrum CFL bulbs) were used to uniformly illuminatehe chart. The Munsell chart comprised eight grayquares, numbered 1 to 8, on a random noise backgroundsee Fig. 11(b)]. The gray squares were matched by eye tohe Munsell standard reflectance, N2 through N9 (Gretagacbeth 31-step neutral value scale) under the SunWave

ulbs. The squares as well as the random noise back-round were printed on Epson enhanced matte paper us-ng an Epson Stylus Photo R800 printer.

. Procedurebservers viewed the photographs, one after another, androvided ratings between 1 and 8 to indicate the standardunsell patch that was closest in reflectance to the

timuli. Fractional ratings such as 4.5 were permitted tollow observers to express their answers at a finer reso-ution than the Munsell scale provided to them. However,

ost observers did not use fractional ratings. For the fewho did, the fractional ratings were converted to thequivalent albedo value. Observers were divided intohree groups. Each group viewed a different set of sur-aces. The experiment was self-paced. For each surface,hree repetitions were run for each lighting condition. Therder of the images was randomized. The experimentasted 30 min.

. Observerswelve observers with normal or corrected-to-normal vi-ion participated in the experiment. All observers wereaive to the purpose of the experiment.

. Resultse found that observers can, to some extent, estimate the

lbedo or diffuse surface reflectance under our experimen-al conditions. Figure 12 plots the perceived diffuse reflec-ance versus the ground truth for observers in one group.bservers are not perfect at estimating ground truth, but

hey perform reasonably well. In Fig. 12, we reject theull hypothesis �p�0.05� that there is no linear relation-hip between observer data and ground truth. Therefore,ontrary to the predictions made by classical lightnessheories, human observers can judge lightness in the ab-ence of mean luminance information and context. Ourbservers tend to agree with one another (see Fig. 13). Wenalyzed the deviation of observer ratings from groundruth, i.e., the errors observers made. We found that theize of the error does not seem to be related to the physi-al reflectance of the surface. In other words, black mate-ials, are not harder to judge than white materials, for ex-mple. Instead, it seems that the closer a surface is to theat, smooth, purely matte ideal of Fig. 1(a), the harder it

s to judge its diffuse reflectance (see Fig. 14).In our data, the effect of lighting was not significant. In

ig. 15 we see that changing the illumination does not af-ect the perceived reflectance of a surface too much. Thisbservation is consistent with the work of Fleming et al.ecause our illumination conditions did not vary as dras-ically as theirs [18]. Fleming et al. showed that reflec-

tli

civdmmetlst

cnBoistg

wopcaalwh

FglTlee

Fd al layo


ance perception is significantly poorer under atypical il-umination (e.g., Gaussian noise) than under real-worldllumination. Our lighting conditions fall in the real-world

ig. 12. Results of experiment I. Perceived albedo versusround truth for four observers. Responses were pooled across allighting conditions. Error bars indicate 95% confidence intervals.he responses of a veridical observer would lie along the black

ine with slope�1. The gray line is the linear regression fit toach observer’s data. The slope of the best fit line is indicated inach plot. The asterisk denotes significance (p-value �0.05).

ig. 11. Observers viewed stimuli on the LCD panel and matcheard surfaces on the Munsell chart. (a) Dimensions of experiment

ategory and are all indoor laboratory setups. Therefore,t is not surprising that observer performance did notary with illumination. It is important to note that in ourata set, we used a fixed viewpoint and probed three illu-ination directions. Unlike databases acquired for BRDFeasurements, such as the CURET database, we did not

xplore the space of lighting directions and viewpoints ex-ensively [42]. One may ask, What effect do variations inighting direction and camera viewpoint have on our re-ults? In Appendix B we have addressed this issue in de-ail.

Finally, as a consequence of using colored surfaces, weould study how observers rated the different color chan-els. Figure 16 displays observer ratings for the R, G, andchannels of a colored surface. The color channels differ

nly in the diffuse and specular reflectance properties; thellumination conditions and surface geometry remain theame. Observers rate each color channel differently,hereby establishing that interreflections and surfaceloss influence lightness perception greatly.

Figure 17 allows a comparison of observer performanceith that of diagnostic image statistics. We note that bothbservers and statistics cannot predict the ground trutherfectly, but both of them make similar mistakes. Theorrelation coefficient r2 ranges from 0.6 to 0.78. Thisgreement is surprising since the statistics have no clues to how observers estimate reflectance properties. Theearning technique that employs statistics as featuresas trained to predict the physical diffuse reflectance, notuman performance. These findings suggest that image

ctance properties of the surface on the screen to one of the stan-ut, (b) photograph of the reference box, (c) dimensions of the box.

d refle

sp

4Gt

ftthndp

Fnafl

FId


tatistics of the kind we have considered must captureerceptually relevant information.

. EXPERIMENT IIiven the strong correlation between perceived reflec-

ance and the informative image statistics, we posed the

ig. 14. Surfaces in (a) and (b) have similar diffuse reflectanceents and surface shapes. (c), (d) Errors in human judgments fornd mesostructure. Error bars indicate maximum and minimumuorescent lighting condition.

ig. 13. Observers tend to agree with one another. Perceived reflf all observers behaved in the same way, all data points would lieeal of agreement among observers.

ollowing question: How do changes in the histogram sta-istics of an image affect reflectance perception? We knowhat as the physical reflectance of a surface changes, theistogram statistics change accordingly. If instead we ma-ipulate the statistics of a given image, what happens? Asiscussed earlier, most of our statistics, moments, andercentiles of luminance and filtered outputs are corre-

s (0.085 and 0.092), respectively but dissimilar specular compo-o surfaces seem to vary with the change in specular reflectanceThe data and photographs in this figure pertain to the overhead

e ratings for every pair of observers from Fig. 12 are plotted here.black lines with slope�1; r2 values indicate that there is a great

valuethe tw

errors.

ectancon the

lnaohhssitmct

ttrLatatasfrv

Fsoitf

Fhalogen source. (a)–(d) show the averaged observer ratings for four sur


ated with one another in our data set. It is difficult to ma-ipulate the statistics independently of one another bypplying monotonic transformations on the images. Inur previous study, we manipulated only the luminanceistogram [2]. In the present work, we manipulate theistogram statistics all at once. As the luminance andubband statistics are correlated, an iterative techniqueuch as the Heeger–Bergen texture synthesis algorithm,s required to simultaneously constrain both kinds of sta-istics [43]. The Heeger–Bergen algorithm iterativelyatches the pixel histogram and the histogram of wavelet

oefficients of a source image texture to a target imageexture.

The algorithm converges in a few iterations to an imagehat has nearly the same pixel and wavelet coefficient his-ograms as the target image. The Heeger–Bergen algo-ithm may be applied in our case in the following manner:et an image of a black surface be the source, and an im-ge of a white surface be the target. The result of runninghe Heeger–Bergen algorithm on the black surface will ben image of a surface that has the same histogram statis-ics as the white surface. If histogram statistics capturenything of perceptual relevance, the resulting imagehould look lighter than the original black surface. Weound that applying the Heeger–Bergen technique di-ectly to our images in the way just described resulted inisible image artifacts.

To remove the artifacts, we introduce the following

t light, light 2 is the halogen spotlight, and light 3 is the diffusefaces. Error bars indicate 95% confidence intervals.

ig. 16. Observers do not rate individual color channels in theame way. (a) Red channel, (b) green channel, and (c) B channelf an orange surface and the respective averaged observer rat-ngs. (d) Even with the same mean image luminance, illumina-ion conditions, and surface geometry, observers can extract use-ul information from the image to discern diffuse reflectance.

ig. 15. Effect of lighting conditions. Light 1 is the overhead fluorescen

mmmabB1tHtbw

meoaTpiswtmp

Ft

F


odification to the Heeger–Bergen algorithm: Instead ofatching histograms of the filter subbands directly, weatch histograms of activity maps instead. Activity maps

re obtained by blurring the absolute value of filter sub-ands. A more detailed discussion of the modified Heeger–ergen technique may be found in Appendix A. Figure8(b) shows the effect of swapping the histogram statis-ics of a white and a black surface with the activity mapeeger–Bergen technique. We see that the perception of

he surfaces is remarkably altered. The manipulatedlack surface looks much lighter, and the manipulatedhite surface looks much darker. To test this observation

ig. 17. Agreement between observers and the regression technhan the agreement between observers but are not very differen

ig. 18. (a) Light surface and (b) dark surface. (c) Result of ma

ore conclusively, we ran an additional experiment withxactly the same conditions as before (experiment I). Thenly change was that we used histogram-manipulated im-ges along with the original photographs of the surfaces.he same 12 observers who took part in experiment I alsoarticipated in experiment II. Observers were dividednto three groups as before. Each group viewed a differentet of surfaces. The partitioning of surfaces across groupsas precisely the same as in experiment I. We ensured

hat for a given observer, stimuli presented during experi-ent I were not repeated in experiment II. Figure 19

lots observer data for this experiment. Observers consis-

he model) is fairly high. The r2 values here are somewhat lower13).

histogram statistics of (a) to those of (b) and (d) and vice versa.

ique (tt (Fig.

tching

ttaTpss

5IfccocttatrmflutatsMip

tsn

nswcnararnlineRoaTiIwca

Wttricmodgviwtfnn

Brttncsttiwojwps

FtntintcAtsv


ently rate the manipulated images as more similar to thearget image than the source image. In other words, im-ges with similar histogram statistics are rated similarly.hese results establish a two-way relationship betweenerceived reflectance and histogram statistics. Changingtatistics affects reflectance perception, and knowing thetatistics lets us predict the perception.

. DISCUSSIONn this work, we considered a range of real-world sur-aces. The interaction of light with our surfaces is moreomplex than with planar, Lambertian surfaces that haveommonly been considered thus far. We found that humanbservers were not perfectly lightness constant, but theironstancy improved as surfaces became more complex. Inhe absence of context and mean luminance information,he visual system relies on other cues present in the im-ge. For the surfaces we considered, diffuse interreflec-ions and specular highlights offer rich information abouteflectance properties. We proposed quantitative imageeasurements that are correlated with the diffuse sur-

ace reflectance. Moment and percentile statistics of theuminance histogram and of filter output histograms wereseful. A linear combination of these statistics was usedo estimate the diffuse reflectance of our surfaces with anccuracy similar to that of human observers. Not only didhe statistics achieve the same degree of success, theyeemed to make the same mistakes as humans [10].oreover, manipulating these candidate statistics in an

mage of a surface altered the perception of lightness (ex-eriment II). Therefore, it is conceivable that our diagnos-

ig. 19. Observer data for histogram-manipulated images (a)hrough (d) show four different surfaces in one group. The chan-els R and B refer to the red and blue color channels, respec-ively, of the color photographs of each surface. The R2B channels the result of matching the histogram statistics of the R chan-el image to that of the B channel. B2R is the result of matchinghe histogram statistics of the B channel image to that of the Rhannel. The data are pooled across all observers in the group.ll plots here are from the overhead fluorescent lighting condi-

ion. Error bars are 95% confidence intervals. We note that ob-ervers consistently rate R2B similar to B rather than to R andice versa.

ic statistics are used as cues to lightness by the visualystem, especially when other cues such as mean lumi-ance or luminance ratios offer no information.How do our results compare with other studies on light-

ess perception that have used complex real-worldtimuli? Gilchrist and Jacobsen’s black-and-white roomsere viewed in isolation, without a context with which to

ompare [4]. As the rooms were matte and mean lumi-ance was accounted for, the only useful informationvailable to the observers was the pattern of diffuse inter-eflections. It is likely that observers employed cues suchs luminance histogram statistics to distinguish theooms. Our findings confirm Nishida and Shinya’s origi-al observation that the luminance histogram influences

ightness perception [15]. Rutherford and Brainard’s find-ng that observers do not explicitly perceive the illumi-ant appears to favor our approach of using image cues tostimate lightness over inverse optics approaches [44].obilotto and Zaidi asked observers to judge the lightnessf crumpled gray papers, some with patterns on them, in

3-D setup under natural viewing conditions [19,20].hey found that observer performance could be explained

n terms of low-level cues such as brightness and contrast.n their experiments, unlike ours, a surrounding contextas always included. Therefore, cues from the surround

ompete with any information available from highlightsnd interreflections within the paper stimulus.Several issues remain unanswered in the present work.e considered the restricted case of surfaces with spa-

ially homogenous reflectance properties under simple ar-ificial illumination. We know from our daily visual expe-ience that we estimate lightness under more challengingllumination and surface conditions. Therefore, it is con-eivable that there exist other cues or informative imageeasurements that apply to a less restricted setting than

urs. Another issue that requires resolution is the inter-ependence of histogram statistics. We found a high de-ree of correlation in our set of statistics. Does this obser-ation extend beyond our limited set of surfaces andllumination conditions? In fact, in the companion study,e manipulated the standard deviation and skewness of

he luminance histogram independently of each other andound that skewness was a stronger cue [2]. Such a ma-ipulation was not possible in the current work, given theumber of dependent statistics we considered.Our experiments with the application of the Heeger–

ergen texture synthesis technique to images of surfacesevealed a potential application—material transforma-ion. One can imagine a Photoshop plugin that changeshe appearance of a surface in an image region by ma-ipulating the local image statistics. Our current modifi-ation to the Heeger–Bergen technique has had modestuccess in achieving this goal. Finally, a major questionhat remains unanswered is, What image statistics dis-inguish an image of a natural surface from an arbitrarymage? Our statistics can predict perception only whene are given an image of a real-world surface. For anyther image, the statistics and indeed even lightnessudgments are not very meaningful. In our companionork, we addressed this question to some extent [2]. Forixel-scrambled images (which look like noise), when ob-ervers are asked to make lightness judgments, they

jtpmobsbcswq

wWhhucrgt

6Hswctamtf

lpgpasasamtwp

AAIgi[BgvdtnWhwmti


udge the overall brightness instead. Since luminance sta-istics are unaffected by pixel scrambling, they cannot ex-lain perception. Clearly, in such cases, other perceptualechanisms must be at work [45]. On phase scrambling

ur surface images, luminance and subband statistics cane retained, but spatial structure is lost. For such phase-crambled images, lightness perception can be explainedy our set of diagnostic image statistics, but not the per-eption of surface gloss. Gloss perception seems very sen-itive to the interpretation of an image as a plausible real-orld situation [18,23]. Further progress on theseuestions is required.In the current work, as well as in the companion paper,

e suggest a new perspective on natural image statistics.hile variance and kurtosis, both even-order statistics,

ave been extensively studied, skewness, an odd statistic,as been largely neglected. We also demonstrated thetility of percentile statistics. These findings have impli-ations for machine vision systems that perform materialecognition as well as for psychophysicists and physiolo-ists studying midlevel vision mechanisms and represen-ations [46–48].

. CONCLUSIONumans use a variety of cues in judging the albedo of a

urface. For an ideal planar surface, context is essential;hen the surface is viewed in isolation, there is insuffi-

ient information to estimate albedo. However, surfaces inhe real world are more complex. They have both diffusend specular reflection, and they often have significantesostructure. The resulting image has a visual texture

hat offers some cues about the surface’s albedo. We haveound that a variety of simple image statistics are corre-

Fig. 20. Block diagram of the activ

ated with albedo. Useful statistics, which may be com-uted on the luminance histogram or on subband histo-rams, include standard deviation, skewness, andercentiles. Linear combinations of these statistics allow

machine vision system, operating within the con-trained world of single surfaces, to estimate the physicallbedo reasonably well. The machine tends to make theame mistakes that humans do, suggesting that humansre using similar statistics in making their judgments. Byanipulating these statistics, we can increase or decrease

he apparent albedo of a given surface in a predictableay, giving further evidence that these image statisticslay a major role in the surface judgments.

PPENDIX A: MODIFIED HEEGER–BERGENLGORITHM

n the Heeger–Bergen algorithm the luminance histo-ram as well the wavelet subband histograms of a targetmage are iteratively matched to those of a source image43]. The histogram-matching procedure in the Heeger–ergen algorithm involves applying a nonlinear pointwiseain; i.e., each pixel value in an image is mapped to a newalue, independent of other pixels. A pointwise gain is notesirable in the overcomplete filter subbands because ifhe value of a pixel is manipulated independently of itseighbors, local distortions can occur in the final output.e propose the following solution: Instead of matching

istograms of the source and target subbands directly, weill modify the target subband histograms via activityaps. An activity map is defined as the result obtained by

aking the absolute value of a subband and then blurringt with a Gaussian kernel (Fig. 20).

p-based Heeger–Bergen technique.
ity-ma

fwgttstbb

wohtatbmpB

AIOIt(fscntlsp

ard[cdnHi

Cbereipthfcca

aottn1sippooTitBup

icwet23sqbh

isgon

FH


The combination of absolute value and blurring trans-orms the subband image into a local energy map. Whene match the histograms of the activity maps of the tar-et and source images, then a pointwise gain is applied tohe source activity map. As the activity map may behought of as a local energy map, a pointwise gain on theource activity map is effectively a locally smooth gain onhe original subband. Let the original source activity mape Aorig and the histogram matched source activity mape Amodified. Then the gain map G is calculated as

G =Amodified

Aorig,

here G is multiplied to the original source subband tobtain the modified subband. Therefore, matching theistograms of the activity maps allows us to apply a spa-ially local gain, which results in fewer image artifactsnd smoother-looking pictures. The local gain modifieshe value at a pixel depending on the values of its neigh-ors; therefore the distortions introduced by histogramatching are reduced locally. Figure 21 shows the im-

rovements obtained by using the modified Heeger–ergen technique.

PPENDIX B: EFFECTS OF VARYINGLLUMINATION AND VIEWPOINTN IMAGE STATISTICS AND PERCEPTION

n our image data, we used a fixed viewpoint and sampledhree illumination directions with different light sourcesFigs. 3 and 4). In our psychophysical experiments, weound that the perceived reflectance of a surface was notignificantly affected by these changes in illuminationonditions (Fig. 15). It may be argued that since we didot explore the space of viewing and illumination direc-ions extensively, the results of Fig. 15 are somewhat pre-iminary. In order to address this concern, we performedome additional computational and psychophysical ex-eriments.Image data that are acquired for BRDF measurement

re sampled densely along viewpoint and illumination di-ections. For example, the Columbia–Utrecht (CURET)atabase contains photographs of 61 real-world surfaces42]. Each surface has been captured from 205 distinctombinations of viewing and illumination angles. Such aatabase would be useful for studying the effect of illumi-ation and viewpoint variation on reflectance perception.owever, there are two reasons why the CURET database

s not ideal for our purposes. First, the resolution of all

ig. 21. Comparison of histogram-matching techniques: (a) soeeger–Bergen output, (e) activity-map-based Heeger–Bergen ou

URET images is 640�480 pixels, which includes pixelselonging to the dark background and the mountingquipment. So the effective pixel resolution of the mate-ial samples is much lower. It can be as low as 50�50 pix-ls at oblique viewing angles. Second, while the materialsn the CURET database have wide-ranging reflectanceroperties, some of them have spatially varying reflec-ance functions (e.g., straw, peacock feather, and cornusk) or possess shallow mesostructure (e.g., leather,rosted glass, and corduroy). In the present study, we fo-used on materials with significant mesostructure thatan be associated with a unique value of albedo. Thus, were limited to only a quarter of the 61 CURET materials.In spite of the concerns just outlined, the CURET im-

ges are very useful because they span a broad spectrumf illumination and viewing directions. We chose nine ma-erials from the database (materials 10–12 and 16–21)hat matched our criteria of spatially uniform albedo andontrivial mesostructure. For each material, we selected59 of the 205 photographs and cropped out the materialample to obtain 100�100 pixel patches. In the CURETmages, there is an inherent trade-off between desiredixel resolution of a material and the number of view-oints that can be used. In order to accommodate as manyblique views as possible without sacrificing too much res-lution, we decided on 100�100 pixels [see Fig. 22(a)].he CURET image data come calibrated to ensure linear-

ty between pixels and radiance. For CURET materials,he ground truth for reflectance is known in terms ofRDF tables and Oren–Nayar model parameters [49]. Wesed the Oren–Nayar model parameterization, since thearameter � corresponds to albedo.In addition to the CURET images, we acquired more

mage data for our own materials. The time and resourceosts of reproducing something like the CURET databaseith our materials are immense. Therefore, we consid-red only two viewpoints and three illumination direc-ions. The images were 512�512 pixel resolution [see Fig.2(b)]. We chose nine materials from our collection (Fig.). Our selection included materials such as handmadetucco, modeling clay, and Tic Tacs. The images were ac-uired and linearized in the same manner as describedefore. The ground truth for albedo for these materialsas already been measured earlier.We analyzed the image statistics for both sets of

mages—CURET images and ours. Figure 23 plots thekewness of center–surround filtered images against theround truth for albedo. Similar results were obtained forther moment and percentile statistics, both for lumi-ance and filter outputs. We observed that the statistics

mage, (b) target image, (c) luminance histogram matching, (d)
urce itput.

vtaomTOmtfpsasdbswt

tip

onsC[FtgsiFtfar

Fdtt

Fatsr


aried enormously with viewpoint and illumination varia-ion in the CURET data set but much less so for our im-ges. These differences may be attributed to the fact thatur images do not explore the space of viewpoint and illu-ination variation as aggressively as the CURET images.here is another, more subtle, reason for these results.ur images differ from the CURET images in a funda-ental way. Our surfaces have deeper mesostructure

han the CURET surfaces (Fig. 22). For shallower sur-aces, the appearance changes more dramatically as view-oint and illumination directions are varied. Consider thehadows in an image of a surface. For a shallow surface,s the illumination (or viewpoint) becomes grazing, darkhadows appear in the image. For a 3-D surface witheeper mesostructure, dark shadows are always presentecause no matter what the illumination (or viewpoint),elf-shadowing occurs due to higher surface relief. Indeed,hen one considers the CURET images of Fig. 22(a), par-

icularly those in the first row, it is hard to believe that

ig. 22. (a) Some of the CURET images we used. The three rowifferent views are shown for each material. The images were muhe materials from our data set. The two rows correspond to twoions. All images were normalized to have the same mean.

ig. 23. Effect of varying illumination and viewpoint on imagegainst the ground truth for diffuse reflectance. A Laplacian of aistics is plotted for all 159 images of the 9 CURET materials amear at each value of rho. (b) All six images of the nine materiaeflectance for the materials.

he same surface can be made to look so different by mov-ng the lights and camera around. We conducted psycho-hysical experiments to study these effects further.We asked three observers (two naïve subjects and one

f the authors) to rate the lightness of mean luminanceormalized images (both CURET and ours) in exactly theame experimental setup as before. For each of the nineURET materials, six representative images were chosen

Fig. 22(a) show three such images for three materials].or our materials, all six views were used for the nine ma-

erials. Images were viewed against a midgray back-round, one at a time, and observers indicated the Mun-ell chip that was closest in reflectance to the samplemage on screen. Two repetitions were run for each image.igure 24 plots the perceived diffuse reflectance against

rue reflectance for both sets of images. Observers wereairly accurate in their lightness judgments for our im-ges but performed poorly on the CURET images. Theseesults confirm the informal observations we made earlier

spond to materials 11 (plaster), 18 (rug), and 21 (sponge). Threeatively normalized to have the same mean luminance. (b) One ofoints and the columns to the three different illumination direc-

ics. The skewness of center–surround filtered images is plottedsian filter was used (�=0.5, size 5�5 pixels). (a) Skewness sta-the Oren–Nayar model parameter “rho” [49]. Note the vertical

ur data set were used. The x axis is the ground truth for diffuse

s correltiplicviewp

statistGaus

gainstls in o

pCstataa

ttao

sdor

ttamadaq

Fob

FabS


ertaining to Fig. 22(a). The poorer resolution of theURET images was an issue. Observers reported thatome CURET images appeared to have wallpaperlike tex-ures rather than 3-D surfaces of single albedo. Observerslso complained about the grazing-angle CURET pictureshat are formed of alternating stripes of dark shadowsnd bright highlights. Even when these pictures are seens 3-D surfaces, it is hard to gauge the true albedo.How well do image statistics account for these percep-

ual results? Figure 25 plots the perceived reflectance forhe observer of Fig. 24 against the skewness of the imagefter center–surround filtering. Again, observer data forur images can be explained reasonably well by our cho-

ig. 24. Perceived reflectance for an observer is plotted against tll viewing and illumination directions. Error bars are 95% conetween perceived and true reflectance �p�0.05�. (b) Our imagesimilar trends were obtained for other observers.

en statistics. For the CURET images also, we see evi-ence for a correlation between perceived reflectance andur statistics. This is encouraging, given the dismal cor-elations in Figs. 23(a) and 24(a).

Taken together, these results lead us to conclude thathe relationship among image statistics, physical reflec-ance, and reflectance perception holds for the images wecquired but breaks down for the CURET data set. Asentioned earlier, our images differ from the CURET im-

ges in many ways. It is reasonable to assume that theseifferences in computational and psychophysical resultsre correlated with the nature of the data sets. A moreuantitative investigation of how factors such as surface

nd truth. For each material, observer ratings were pooled acrosse intervals. (a) CURET images: There is no linear relationshiprver data can be explained by a linear model (p�0.05, r2=0.85).

ig. 25. Perceived reflectance of an image of a surface is plotted against the skewness of the center–surround filter output. A Laplacianf a Gaussian filter was used (�=0.5, size 5�5 pixels). (a) CURET images and (b) our images. A linear relationship can be observed inoth plots. For (c), p�0.05, r2=0.42 and for (d), p�0.05, r2=0.64.

he groufidenc: Obse

srp

AVTmugitilttap

(dmpdorwWmaaestesatcd

tt=fd

AWaRptd

R

1

F[cna


tructure, viewing and illumination conditions, and mate-ial properties influence image statistics (and reflectanceerception) remains a direction for future research.

PPENDIX C: LINEARIZATION OF PIXELALUES

o ensure a linear relationship between pixel values andeasured luminance, we calibrated our raw image datasing dcraw software [37]; dcraw is an open-source pro-ram that decodes the raw images acquired by a cameran its native, manufacturer-specific format. The advan-age of taking photographs in the raw format is that rawmages have the most information and are processed theeast by the camera. Almost all digital cameras can be seto produce a JPEG output instead of a raw output. Al-hough the JPEG format is convenient, it is a lossy formatnd valuable data are lost as a result of the JPEG com-ression.We acquired our image data in the Canon RAW format

CRW). The CRW images files were then converted by thecraw program to 16-bit PSD (Adobe Photoshop) file for-at. In order to verify the correctness of the calibration

rocedure, we used the following methodology. Since weid not take photometric measurements at each point ofur scenes, the precise luminance measurements that cor-espond to the pixels in an image are unknown. However,e did capture multiple exposure shots of the same scene.e used a fixed-focal-length �50 mm� lens. We appliedanual settings for focus, aperture, and white balance

nd varied the shutter speeds in intervals of 1 / 2 f-stopsnd acquired between 9 and 12 consecutive exposures forach scene. We made use of the following principle: If thehutter is open for twice as long, the amount of light cap-ured is twice as much. If the dcraw outputs are truly lin-ar, the pixels values for two images of the same scenehould be related to each other by exactly the same ratios the lengths of their exposure times. Figure 26 illus-rates this reasoning for a pair of example images. If theombination of internal camera processing followed bycraw was perfectly linear, the data in Fig. 26 would fit

ig. 26. Verifying linearity correction. Images A and B are the li37]. The original exposures were recorded in the Canon RAW foamera parameters were the same for A and B. The pixel valuesation of internal camera processing followed by dcraw was perctual fit is quite good: slope�2.1055±0.0084 (p�0.05, r2=0.94).

he linear model perfectly with a slope equal to 2. The ac-ual fit is quite good: slope=2.1055±0.0084 (p�0.05, r2

0.94). We repeated this analysis for other image pairsor each of our scenes and verified that the accuracy ofcraw linearization was similar to Fig. 26.

CKNOWLEDGMENTSe thank our reviewers for their insightful comments

nd suggestions. We also thank R. W. Fleming and R.osenholtz for useful discussions. This research was sup-roted by a grant from the NTT-MIT Research Collabora-ion. It was also supported by the National Science Foun-ation under grant no. 03545805 to E. Adelson.

EFERENCES1. A. Gelb, “Die ‘Farbenkonstanz’ dez Sehdinge [Color

constancy of visual objects],” in Handbuchder normalenund pathologischen Psychologie, W. A. von Bethe, ed.(Springer, 1929), pp. 594–678.

2. I. Motoyoshi, S. Nishida, L. Sharan, and E. H. Adelson,“Image statistics and the perception of surface qualities,”Nature 447, 206–209 (2007).

3. A. L. Gilchrist, “The perception of surface blacks andwhites,” Sci. Am. 240, 112–124 (1979).

4. A. L. Gilchrist and A. Jacobsen, “Perception of lightnessand illumination in a world of one reflectance,” Perception13, 5–19 (1984).

5. E. H. Land and J. J. McCann, “Lightness and retinextheory,” J. Opt. Soc. Am. 61, 1–11 (1971).

6. A. L. Gilchrist, “Perceived lightness depends on perceivedspatial arrangement,” Science 195, 185–187 (1977).

7. E. H. Adelson, “Perceptual organization and judgment ofbrightness,” Science 262, 2042–2044 (1993).

8. M. D’Zmura and G. Iverson, “Color constancy. I. Basictheory of two-stage linear recovery of spectral descriptionsfor lights and surfaces,” J. Opt. Soc. Am. A 10, 2148–2165(1993).

9. D. H. Brainard, “Color constancy in the nearly naturalimage. 2. Achromatic loci,” J. Opt. Soc. Am. A 15, 307–325(1998).

0. A. L. Gilchrist, C. Kossyfidis, F. Bonato, T. Agostini, J.Cataliotti, X. Li, B. Spehar, V. Annan, and E. Economou,“An anchoring theory of lightness perception,” Psychol. Rev.106, 795–834 (1999).

utputs of the dcraw program for two exposures of the same scene(CRW). The exposure time for B was twice that of A. All other

linearized images are plotted against each other. If the combi-linear, all data points would lie along a line with slope�2. The

near ormatof the

fectly

1

1

1

1

1

1

1

1

1

2

2

2

2

2

2

2

2

2

2

3

3

3

3

3

3

3

3

3

3

4

4

4

4

4

4

4

4

4

4


1. M. G. Bloj, D. Kersten, and A. C. Hurlbert, “Perception ofthree-dimensional shape influences color perceptionthrough mutual illumination,” Nature 402, 877–879 (1999).

2. H. Boyaci, L. T. Maloney, and S. Hersh, “The effect ofperceived surface orientation on perceived surface albedo inbinocularly viewed scenes,” J. Vision 3, 541–553 (2003).

3. D. H. Brainard, J. M. Kraft, and P. Longere, “Colorconstancy: developing empirical tests of computationalmodels,” in Color Perception: From Light to Object, R.Mausfeld and D. Heyer, eds. (Oxford U. Press, 2003), pp.307–334.

4. L. T. Maloney and J. N. Yang, “The illumination estimationhypothesis and surface color perception,” in Color Vision:Connecting the Mind to the Physical World, R. Mausfeldand D. Heyer, eds. (Oxford U. Press, 2003), pp. 335–358.

5. S. Nishida and M. Shinya, “Use of image-based informationin judgments of surface reflectance properties,” J. Opt. Soc.Am. A 15, 2951–2965 (1998).

6. R. O. Dror, E. H. Adelson, and A. S. Willsky, “Recognitionof surface reflectance properties from a single image underunknown real-world illumination,” in Proceedings of theIEEE Workshop on Identifying Objects across Variation inLighting: Psychophysics and Computation (IEEE, 2001),available at http://web.mit.edu/persci/people/adelson/pub_pdfs/dror_cvpr01_goem.pdf.

7. R. O. Dror, “Surface reflectance recognition and real-worldillumination statistics,” Ph.D. dissertation (MassachusettsInstitute of Technology, Department of ElectricalEngineering and Computer Science, 2002).

8. R. W. Fleming, R. O. Dror, and E. H. Adelson, “Real-worldillumination and the perception of surface reflectanceproperties,” J. Vision 3, 347–368 (2003).

9. R. Robilotto and Q. Zaidi, “Limits of lightness identificationof real objects under natural viewing conditions,” J. Vision4, 779–797 (2004).

0. R. Robilotto and Q. Zaidi, “Lightness identification ofpatterned three-dimensional, real objects,” J. Vision 6,18–36 (2006).

1. J. Beck and S. Prazdny, “Highlights and the perception ofglossiness,” Percept. Psychophys. 30, 401–410 (1981).

2. F. Pellacini, J. A. Ferwerda, and D. P. Greenberg, “Towardsa psychophysically-based light reflection model for imagesynthesis,” in Proceedings of the 27th Annual Conference onComputer Graphics and Interactive Techniques, J. R.Brown and K. Akeley, eds. (ACM Press, 2000), pp. 55–64.

3. J. T. Todd, J. F. Norman, and E. Mingolla, “Lightnessconstancy in the presence of specular highlights,” Psychol.Sci. 15, 33–39 (2004).

4. B.-T. Phong, “Illumination for computer generatedpictures,” Commun. ACM 18, 311–317 (1975).

5. Y. Sato, M. D. Wheeler, and K. Ikeuchi, “Object shape andreflectance modeling from observation,” in Proceedings ofthe 24th Annual Conference on Computer Graphics andInteractive Techniques, G. O. Owen, T. Whitted, and B.Mones-Hattal, eds. (ACM Press, 1997), pp. 379–387.

6. Y. Yu and J. Malik, “Recovering photometric properties ofarchitectural scenes from photographs,” in Proceedings ofthe 25th Annual Conference on Computer Graphics andInteractive Techniques, S. Cunningham, W. Bransford, andM. F. Cohen, eds. (ACM Press, 1998), pp. 207–217.

7. S. R. Marschner, S. H. Westin, E. P. F. Lafortune, K. E.Torrance, and D. P. Green, “Image-based BRDFmeasurement including human skin,” in Proceedings of the10th Eurographics Workshop on Rendering, D. Lischinskiand G. W. Larson, eds. (Springer, 1999), pp. 139–152.

8. Y. Yu, P. Debevec, J. Malik, and T. Hawkins, “Inverse globalillumination: recovering reflectance models of real scenesfrom photographs,” in Proceedings of the 26th AnnualConference on Computer Graphics and InteractiveTechniques, W. Waggenspack, ed. (ACM Press, 1999), pp.215–224.

9. S. Boivin and A. Gagalowicz, “Image based rendering ofdiffuse, specular and glossy surfaces from a single image,”

in Proceedings of the 28th Annual Conference on ComputerGraphics and Interactive Techniques, L. Pocock, ed. (ACMPress, 2001), pp. 107–116.

0. S. Tominaga and N. Tanaka, “Estimating reflectionparameters from a single color image,” IEEE Comput.Graphics Appl. 20, 58–66 (2000).

1. K. Nishino, Z. Zhang, and K. Ikeuchi, “Determiningreflectance parameters and illumination distributions froma sparse set of images for view-dependent imagesynthesis,” in Proceedings of IEEE InternationalConference on Computer Vision (IEEE, 2001), pp. 599–601.

2. R. Ramamoorthi and P. Hanrahan, “A signal processingframework for inverse rendering,” in Proceedings of the28th Annual Conference on Computer Graphics andInteractive Techniques, L. Pocock, ed. (ACM Press, 2001),pp. 117–128.

3. P. Debevec, T. Hawkins, C. Tchou, H.-P. Duiker, W. Sarokin,and M. Sagar, “Acquiring the reflectance field of a humanface,” in Proceedings of the 27th Annual Conference onComputer Graphics and Interactive Techniques, J. R.Brown and K. Akeley, eds. (ACM Press, 2000), pp. 145–156.

4. P. Debevec, C. Tchou, A. Gardner, T. Hawkins, C. Poullis, J.Stumpfel, A. Jones, N. Yun, P. Einarsson, T. Lundgren, M.Fajardo, and P. Martinez, “Estimating surface reflectanceproperties of a complex scene under captured naturalillumination,” Tech. Rep. ICT-TR-06 (University ofSouthern California Institute for Creative TechnologiesGraphics Laboratory, 2004).

5. J. J. Koenderink, A. J. Van Doorn, K. J. Dana, and S.Nayar, “Bidirectional reflectance distribution function ofthoroughly pitted surfaces,” Int. J. Comput. Vis. 31,129–144 (1999).

6. J. J. Koenderink and A. J. van Doorn, “Illuminance texturedue to surface mesostructure,” J. Opt. Soc. Am. A 13,452–463 (1996).

7. D. Coffin, “Raw digital photo decoding in Linux,” http://www.cybercom.net/~dcoffin/dcraw.

8. P. Debevec, C. Tchou, and T. Hawkins, “HDRShop: highdynamic range image processing and manipulation,” http://www.hdrshop.com.

9. P. Burt and E. H. Adelson, “Laplacian pyramid as acompact image code,” IEEE Trans. Commun. 31, 532–540(1983).

0. E. P. Simoncelli and W. T. Freeman, “The steerablepyramid: a flexible architecture for multi-scale derivativecomputation,” in Proceedings of IEEE Conference on ImageProcessing (IEEE, 1995), pp. 444–447.

1. C.-C. Chang and C.-J. Lin, “LIBSVM: a library for supportvector machines,” http://www.csie.ntu.edu.tw/~cjlin/libsvm.

2. Columbia–Utrecht Reflectance and Texture (CURET)database, http://www.cs.columbia.edu/CAVE/curet.

3. D. J. Heeger and J. R. Bergen, “Pyramid-based textureanalysis/synthesis,” in Proceedings of the 22nd AnnualConference on Computer Graphics and InteractiveTechniques, R. Cook, ed. (ACM Press, 1995), pp. 229–238.

4. M. D. Rutherford and D. H. Brainard, “Lightnessconstancy: a direct test of the illumination estimationhypothesis,” Psychol. Sci. 13, 142–149 (2002).

5. C. Chubb, M. S. Landy, and J. Econopouly, “A visualmechanism tuned to black,” Vision Res. 44, 3223–3232(2004).

6. B. A. Olshausen and D. J. Field, “Emergence of simple-cellreceptive field properties by learning a sparse code fornatural images,” Nature 381, 607–609 (1996).

7. E. P. Simoncelli and B. A. Olshausen, “Natural imagestatistics and neural representation,” Annu. Rev. Neurosci.24, 1193–1216 (2001).

8. M. Varma and A. Zisserman, “A statistical approach totexture classification from single images,” Int. J. Comput.Vis. 62, 61–81 (2005).

9. M. Oren and S. K. Nayar, “Generalization of theLambertian model and implications for machine vision,”Int. J. Comput. Vis. 14, 227–251 (1995).

Image statistics for surface reﬂectance...

Documents

Transcript of Image statistics for surface reﬂectance...