Self-calibrating Photometric Stereo

Boxin Shi1∗ Yasuyuki Matsushita2 Yichen Wei2 Chao Xu1 Ping Tan3

1Key Lab of Machine Perception (MOE), Peking University2Microsoft Research Asia

3Department of Electrical & Computer Engineering, National University of Singapore

Abstract

We present a self-calibrating photometric stereo method.From a set of images taken from a fixed viewpoint underdifferent and unknown lighting conditions, our method au-tomatically determines a radiometric response function andresolves the generalized bas-relief ambiguity for estimatingaccurate surface normals and albedos. We show that colorand intensity profiles, which are obtained from registeredpixels across images, serve as effective cues for address-ing these two calibration problems. As a result, we developa complete auto-calibration method for photometric stereo.The proposed method is useful in many practical scenarioswhere calibrations are difficult. Experimental results val-idate the accuracy of the proposed method using variousreal-world scenes.

1. IntroductionPhotometric stereo [23] recovers surface orientations and

albedos from a set of images taken from a fixed viewpointunder different lighting directions. There are two necessarycalibrations for a standard photometric stereo algorithm towork correctly. First, the camera needs to be radiometri-cally calibrated to obtain the scene irradiance from mea-sured pixel values. Second, lighting directions and inten-sities need to be known to uniquely determine the surface.With these two calibrations, surface orientations and albe-dos can be estimated uniquely from three images for Lam-bertian scenes.

Typically, these calibrations are performed separatelybefore applying photometric stereo algorithms. The ra-diometric camera calibration is often performed as a com-pletely independent preprocessing step by, for example, us-ing multiple images [15] captured from a fixed viewpointunder a static lighting configuration with different exposuretimes. In practice, however, this involves an additional datameasurement that is not always possible. The lighting direc-

∗This work was done while the first author was visiting Microsoft Re-search Asia.

tions and intensities are often recorded by putting a mirrorsphere and a Lambertian sphere in the scene. These ad-ditional calibration objects make the process complicatedand could even introduce interreflections that eventually in-crease error. On the other hand, if the lighting parametersare unknown, surface normals of a Lambertian scene canonly be recovered up to a 3 × 3 linear ambiguity [10]. Ifthe surface is known to be integrable, this ambiguity can bereduced to the generalized bas-relief (GBR) ambiguity [3].Without additional prior knowledge about the scene, or thelighting, this shape/lighting ambiguity cannot be resolved.Hence, these two calibrations limit the practical applica-tions of photometric stereo.

In this paper, we address these two calibration problemswithout using any additional input data, or calibration ob-jects. From a set of images taken from a fixed viewpoint un-der unknown and varying lightings with an unknown cam-era response function, our method automatically recoversboth the camera’s radiometric response function and the un-known lighting directions and intensities. Our method usescolor and intensity profiles to achieve this goal. A colorprofile is defined as the set of measured RGB colors at apixel across images. A color profile draws a 3D curve inthe RGB color space. An intensity profile is defined asan ordered sequence of scene irradiance at a pixel acrossimages. An intensity profile corresponds to a 2D curve inthe irradiance-time space, similar to the appearance profilesshown by Koppal and Narasimhan [12]. Our method usescolor profiles to determine the radiometric response func-tion, and intensity profiles to resolve the GBR ambiguity.Putting them together, we build a self-calibrating photomet-ric stereo method.

The rest of the paper is organized as follows. After dis-cussing prior work, we describe the problem and overviewof our approach in Section 2. Section 3 describes the pro-posed radiometric calibration method based on the colorprofiles, and Section 4 explains the method for GBR dis-ambiguation using intensity profiles. Section 5 shows theexperimental results followed by discussions.

Radiometric

calibration

Surface normal grouping

by intensity profiles

Linearization

of color profiles

Albedo

grouping

Scene

Sequence Surface normal

Depth

I

t

R G

B

GBR disambiguation

Figure 1. Overview of the proposed method. The proposed method automatically performs radiometric self-calibration and GBR disam-biguation from the input images and gives estimated surface normals and depth maps as output.

1.1. Related works

Our work is related to both radiometric calibration andGBR disambiguation. Several methods have been pro-posed to estimate camera response functions. One ofthe most widely used methods is Mitsunaga and Nayar’smethod [15], which uses multiple images of a static scenewith multiple different known exposure times. More re-cent methods work under more general imaging conditions.Lin et al. [13] proposed a method that only requires a sin-gle color image. Later they extended it to work with a sin-gle gray-scale image [14]. These methods are based oncolor/intensity mixtures on edge boundaries. There are afew works that allow scene/camera movements during datacapturing. Kim and Pollefeys [11] computed pixel corre-spondence across image frames for radiometric calibration.Grossberg and Nayar [8] used intensity histograms of twoimages of different exposure rather than exact pixel corre-spondence. Wilburn et al. [22] showed the response func-tion can be recovered from a motion blur in a single im-age. Unlike all of these methods, our radiometric calibra-tion method uses photometric stereo images captured by astatic camera under varying lighting conditions.

The GBR ambiguity is an intrinsic shape/lighting am-biguity for uncalibrated photometric stereo with unknownlighting parameters. This ambiguity arises because differentcombinations of shapes and lightings can produce the sameappearance. To resolve the GBR ambiguity, additional cuesare necessary. Prior knowledge about surface albedo can beused to resolve the GBR [10, 1]. Chandraker et al. [4] usedinterreflections in a Lambertian scene to resolve the ambi-guity. It has been shown that the GBR can also be resolvedif the surface reflectance is known to be Lambertian com-pounded with a Torrance-Sparrow specular reflection [7], orwith specular reflections containing specular spikes [6, 5].More recently, Tan et al. [20, 21] showed that the GBR canbe reduced and resolved in a stratified manner for isotropicsurfaces. Besides resolving the GBR ambiguity, Basri et al.[2] proposed a general solution under environment lighting.

Furthermore, to handle various BRDFs, Sato et al. [19] sug-gested using the similarity in radiance changes and Okabe etal. [17] utilized the attached shadow. Our method takes adifferent approach from these. As we will see later in thispaper, our method can achieve a high-accuracy estimationusing the intensity profile as a cue.

The radiometric calibration and GBR disambiguationhave been discussed separately in prior works. To achievea complete auto-calibration in photometric stereo, we solvethe mixture of these problems. We take a fundamentallydifferent approach to achieve this goal using the color andintensity profiles.

2. Problem statement

We perform photometric stereo with images taken froma fixed viewpoint under varying and unknown directionallighting (both intensities and directions are unknown) us-ing a camera where the response function is also unknown.We address the two calibration problems in this setting, i.e.,radiometric camera calibration and GBR disambiguation.

In this paper, we use the following imaging model:

M = f(I) = f((ρn) · (El)

), (1)

where M is the recorded pixel value, and ρ and n are thesurface reflectance (or albedo) and the surface normal vec-tor at a pixel. E and l are the intensity and direction of thedirectional lighting. I is the irradiance, and f is the cam-era’s radiometric response function. Since the albedo ρ andsurface normal vector n are location dependent and light-ing intensity E and direction l are image dependent, we usesubscripts i and j to index pixels and images respectively.For multiple images under varying directional lighting, theimaging model can be written as

Mij = f(Iij) = f((ρini) · (Ejlj)

). (2)

Our goal is to estimate ρi,ni, Ej , lj and f from the obser-vations Mij .

Radiometric response function The goal of radiometriccalibration is to estimate the radiometric response functionf that maps the irradiance I to a pixel intensity M . As theresponse function f is a monotonic function, there is alwaysa unique inverse mapping g(= f−1). Radiometric calibra-tion amounts to estimating the inverse response function g.

The generalized bas-relief ambiguity As in previouswork [10], the photometric stereo problem is formulated ina matrix form:

I = SL, (3)

where I is a P × F matrix containing irradiance valuesfrom all images, and P and F are the number of pixelsand images respectively. Each row of I corresponds to apixel position in an image, and each column correspondsto a different image. The surface matrix S ∈ RP×3 rep-resents surface properties, i.e., the i-th row of S encodesthe surface normal at the i-th pixel scaled by its albedo asSi∗ = ρini. The lighting matrix L ∈ R3×F represents thelighting directions and intensities, i.e., the j-th column ofthe matrix L corresponds to the lighting direction in the j-th image scaled by its intensity as L∗j = Ejlj . When L isunknown, S and L can be estimated only up to a GBR trans-formation G by enforcing integrability over the surface asI = SL = (S′G)(G−1L′). The GBR transformation G isa three-parameter matrix:

G =

1 0 00 1 0µ ν λ

, G−1 =1λ

λ 0 00 λ 0−µ −ν 1

. (4)

The overview of the proposed method is illustrated inFig. 1. Our method uses color profiles for radiometric cali-bration (Section 3) and intensity profiles for resolving GBRambiguity (Section 4).

3. Radiometric calibration using color profilesThis section describes our method for determining the

radiometric response function using color profiles. A colorprofile is a set of RGB color values at the same coordinatein a set of photometric stereo images. Our method capital-izes on the fact that color profiles form straight lines in theRGB color space when the radiometric response function fis linear. On the other hand, if f is nonlinear, color profilesform nonlinear curves in the RGB space.

To understand this relationship between color profilesand the radiometric response function f , let us consider theratio between red, green and blue colors at a pixel. In acolor image, Eq. (2) is evaluated at each color channel, i.e.,Mr = f(Ir), Mg = f(Ig) and M b = f(Ib), where su-perscripts r, g, and b correspond to the RGB color chan-nels. If the radiometric response function f is linear, i.e.,

0.4

0.5

0 2

0.3B

0.1

0.2

0 0.2 0.4 0.6 0.80.20.40.60.80

RG

Figure 2. Color profiles of a real-world scene taken by a cam-era with a non-linear response function. The profiles show clearcurves in the RGB space.

f(x) = kx, the color ratio at a Lambertian point is

(Mr : Mg : M b) = (kIr : kIg : kIb) = (ρr : ρg : ρb), (5)

which is constant over the photometric stereo image se-quences. Hence, in this case, a color profile is a straight linewith direction (ρr, ρg, ρb) passing through the origin in theRGB space. On the other hand, if f is nonlinear, this ratiowill change over the image sequence. As a result, the colorprofile bends to a nonlinear curve in the RGB space. Thisresult is consistent with the empirical observation in [18].Examples of color profiles obtained from a real-world sceneare shown in Fig. 2.

The correct inverse response function g (g(M) = I)should linearize the curved color profiles. Using this prop-erty as a constraint, we estimate the inverse response func-tion g that maps curved color profiles to straight lines. Wefirst define a nonlinearity measurement for color profiles toassess how the profiles are bended. We then formulate theestimation problem in an energy minimization frameworkto derive the inverse response function g.

Degree of nonlinearity We assess the degree of nonlin-earity of a transformed color profile by its line fitting error.Let us denote the RGB triplet of the measured color valueas M and use g(M) to represent that the inverse responsefunction g is applied to each element of M independently.For a color profile at the i-th pixel, it is computed as

Di(g) =∑

j

‖(X0i −X1

i )× (g(M ij)−X1i )‖

‖X0i −X1

i ‖, (6)

where X0i and X1

i are two points in the RGB space thatdefine the line best fitting to the transformed color profileg(M i∗). Intuitively, D measures the summed Euclideandistance between g(M ij) to the fitted line. It approaches tozero when g is the correct inverse function.

Energy function We compute the inverse response func-tion g from a set of color profiles Ω by minimizing the non-linearity. We use a polynomial representation for the in-verse response function to reduce the computational com-plexity as in [15]. Thus, the inverse response function g isdescribed by a small number of unknown coefficients ck inthe form of K-order polynomial as

I = g(M) =K∑

k=0

ckMk. (7)

We estimate the coefficients c = c0, ..., cK and fittedlines X = X0

i ,X1i |i ∈ Ω by minimizing the summed

nonlinearity for each color profile i ∈ Ω:

c, X = argminc,X

1|Ω|

∑i∈Ω

Di(c), (8)

where |Ω| is the number of color profiles used for the com-putation. The minimization is performed with normalizedcolor values in the range [0, 1].

For minimization, we put two constraints on the inverseradiometric response function g, namely monotonicity andboundary conditions, which are general for any inverse re-sponse functions. The monotonicity constraint is repre-sented as ∂g/∂M > 0, and the boundary conditions aredescribed as g(0) = 0 and g(1) = 1. We use these bound-ary conditions as a hard constraint by setting c0 = 0 andcK = 1 −

∑K−1k=1 ck because g(1) =

∑Kk=1 ck = 1. The

final formulation of the energy minimization becomes

c, X = argminc,X

1|Ω|

∑i∈Ω

Di(c) + λ∑

t

H

(−∂g(t)

∂M

). (9)

Here, H(·) is the Heaviside step function (H(x) = 1 whenx ≥ 0, and H(x) = 0, otherwise). The derivative ∂g/∂Mare assessed at various t in the range of [0, 1].

Optimization of Eq. (9) is performed with the Nelder-Mead simplex method [16] due to its simplicity, but othersimilar methods could be used as well. We initialize g as alinear function and X as least square fitted lines from colorprofiles.

4. GBR disambiguation using intensity profilesThis section describes the proposed method for resolving

the GBR ambiguity using intensity profiles. By Hayakawa’ssingular value decomposition (SVD) method [10], the irra-diance I in Eq. (3) can be factorized into a surface compo-nent S and a lighting component L, up to a linear ambigu-ity. If albedos are known at 6 different normals or lightingintensity is known at 6 images, this linear ambiguity canbe reduced to a rotational ambiguity. On the other hand,

0 4

0.5

Group 1Group 2

0.3

0.4 p

dian

ce

0.1

0.2Irra

d

1 2 3 4 5 6 7 8 9 10 11 120

Frame number

Figure 3. Intensity profiles from a real-world scene. Group 1 andGroup 2 represent two different surface normal groups. Withineach group, the intensity profiles bear the similar shapes.

Yuille et al. [25] showed the original linear ambiguity canbe reduced to a GBR ambiguity by enforcing the integrabil-ity constraint.

We observe that the intersection of the rotational groupand the GBR transformation group contains only the iden-tity matrix. Hence, if both constraints from [10] and [25]can be used, the original linear ambiguity is fully resolved.To do so, we first apply the integrability constraint to derivea result up to the GBR ambiguity. We then resolve the GBRambiguity by identifying pixels that have different normalsbut with the same albedo.

Suppose we are given pixels that have the same albedoa but different normals. The GBR transformation shouldsatisfy ||siG|| = a, i.e.,

siCsTi = a2, (10)

where

C = GGT =

1 0 µ0 1 νµ ν µ2 + ν2 + λ2

, (11)

and si is the i-th row vector of S. Let λ = µ2 + ν2 +λ2 be an auxiliary variable. The constraint in Eq. (10) islinear in all 4 unknowns µ, ν, λ, a2. Therefore, given atleast 4 pixels with different normals but with the commonalbedo, the GBR ambiguity can be resolved. Our goal isto automatically identify these pixels to resolve the GBRambiguity.

Surface normal grouping Following Koppal andNarasimhan’s work [12], we use intensity profiles to clustersurface normals. As shown in Fig. 3, pixels with the samesurface normal show strong correlation in their intensityprofiles, while pixels with different surface normals becomeless correlated. Since we work in the irradiance domain,instead of using the extrema of the profiles as in [12], wesimply use the correlation of the entire profiles. This allowsus to use fewer images for surface normal clustering. ThePearson’s correlation rii′ between two intensity profiles Ii

Input Normal grouping Albedo grouping

Input Normal grouping Albedo grouping

PDF created with pdfFactory Pro trial version www.pdffactory.com

Figure 4. Surface normal grouping (middle) and albedo grouping(right) on the CAT scene. One of the input images is shown on theleft. Different colors indicate different cluster labels.

and Ii′ is assessed by

rii′ =

∑Fj=1 (Iij − Ii)(Ii′j − Ii′)

(F − 1)σiσi′, (12)

where Ii is the mean value of the i-th row vector of matrixI, and σi is its standard deviation. Using the correlation asa distance, we perform K-means clustering to partition im-ages into a set of groups that have similar surface normals.Hence, by selecting from different groups, we can obtainpoints with different normal directions. For this purpose,we use a large K. An example of the surface normal group-ing result is shown in Fig. 4 (middle).

Albedo grouping Our albedo grouping aims at identify-ing pixels with the same albedo. As we have discussed inSection 3, the RGB ratio of a Lambertian scene point shouldbe constant (ρr : ρg : ρb) across images in the irradiancedomain. Hence, we can approximately measure the simi-larity of pixel albedos by their difference of chromaticity.To reduce the influence of shadows and noise, we computethe chromaticity of each pixel from its averaged irradiancevalue I . The chromaticity is defined as a vector of two el-ements [Ir/(Ir + Ig + Ib), Ig/(Ir + Ig + Ib)]. We thencluster pixels in the chromaticity domain to obtain pixelswith similar albedo. Similar to the surface normal group-ing, we apply K-means clustering using the Euclidean dis-tance of chromaticity vectors. As a result, we obtain albedogroups as shown in Fig. 4 (right).

Pixel selection We combine the two grouping results toautomatically select pixels with the same albedo but differ-ent surface normals.

First, our method selects reliable albedo groups. All thealbedo groups are sorted by their variances, and half of thegroups with larger variances are eliminated. After that, ineach albedo group, we further remove pixels that have alarge distance from the group center in the chromaticity do-main. Specifically, pixels that have a distance larger thanthe average are discarded. The selected groups are split intosubgroups based on the surface normal grouping result. Theresulting subgroups become the groups of pixels with the

CAT (12) OWL (12) PILLOW (15) SHEEP (12)

CAT (12) OWL (12) PILLOW (15) SHEEP (12)

Figure 5. Images of the real-world scene for experiments with thenumber of images used.

same albedo and the same surface normal. Since we aimat finding pixels with the same albedo but different surfacenormals, only one pixel from each subgroup is selected.

To implement this pixel selection algorithm, we havetested two different approaches. One is random selectionfrom subgroups, and the other is an optimization approach.The second approach selects pixels that minimize the sum-mation of the surface normal similarity among selected pix-els. This optimization problem is discrete and can be ef-ficiently solved using a belief propagation algorithm [24].Although the optimization-based method theoretically guar-antees to select pixels with diverse surface normals, wefind that the random sampling produces as good result asthe optimization-based method when the number of surfacenormal groups is high. For this reason, we use the randomsampling method in this work.

Finally, a set of pixels are selected from R albedo groupsof unknown albedo values a1, a2, . . . , aR. From thesepixels, we obtain multiple linear equations Eq. (10) withunknowns µ, ν, λ, a2

1, · · · , a2R to resolve the GBR ambi-

guity by a least square solution.

5. Experiments

We use four real-world scenes, CAT, OWL (courtesyof Dan Goldman and Steven Seitz1), PILLOW and SHEEPscenes, to verify our method. The scenes are shown inFig. 5. CAT and OWL are radiometrically calibrated, whilePILLOW and SHEEP are recorded by a Nikon D70 andCanon 20D, respectively. Both of these cameras have non-linear radiometric response functions. In this experiment,we first quantitatively evaluate the radiometric calibrationpart. Second, we use the CAT and OWL scenes to assessthe proposed GBR disambiguation method. Finally, weevaluate the entire pipeline with both radiometric calibra-tion and GBR disambiguation. We implement our methodusing Matlab. The optimization of Eq. (9) can be imple-mented simply by a Matlab built-in function “fminsearch”.Throughout our experiments, we set λ = 5000, |Ω| = 100(100 color profiles), K = 200 for the surface normal group-ing, and K = 20 for the albedo grouping in two K-meansstages. With an Intel E6550 (2.33GHz) CPU, the entire pro-cess took 10 minutes without optimization.

1http://www.cs.washington.edu/education/courses/csep576/05wi//projects/project3/project3.htm

1 EstimatedGround truth

Inverse response function

0.6

0.7

0.8

0.9 Ground truth

radi

ance

Nikon D70

RMSE:0.0252

0.3

0.4

0.5

Nor

mal

ized

irr

Disparity:0.0387

Std. dev.:

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

Normali ed meas rement

0.0131

0.9

1 EstimatedGround truth

Normalized measurement

Inverse response function

0 5

0.6

0.7

0.8

d irr

adia

nce

Canon 20D

RMSE:0.0254

0.2

0.3

0.4

0.5

Nor

mal

ized Disparity:

0.0459

Std. dev:0.0286

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

Normalized measurement

Figure 6. Result of radiometric calibration. The mean curve ofestimated inverse response functions with standard deviation areshown in comparison with the ground truth.

Radiometric calibration result Using a Nikon D70 andCanon 20D, we take 50 datasets for testing the radiometriccalibration method. From each dataset, we randomly se-lect pixels to produce color profiles. The result is shownin Fig. 6 in comparison with the ground truth obtained byMitsunaga and Nayar’s method [15]. The averages of rootmean squared error (RMSE) and disparity (the maximumdifference) are computed from all the 50 datasets. Thestatistics show that our method consistently works well withvarious scenes and different cameras.

Surface normal and albedo grouping result We showan example of the grouping result of the SHEEP scenein Fig. 7. It depicts the accuracy of the grouping result forthe following GBR disambiguation.

GBR disambiguation result We evaluate the proposedGBR disambiguation method using the CAT and OWL

Input Normal grouping Albedo grouping

Input Normal grouping Albedo grouping

PDF created with pdfFactory Pro trial version www.pdffactory.com

Figure 7. Surface normal grouping (middle) and albedo grouping(right) on the SHEEP scene. One of the input images is shown onthe left. Different colors indicate different cluster labels.

Angular error CAT OWL

Our method Mean 6.15 10.47Std. dev. 2.83 4.75

Minimum Mean 15.82 27.10entropy [1] Std. dev. 6.15 6.08

Table 1. GBR disambiguation results. Mean and standard devi-ation of the angular error of the estimated normal maps. Ourmethod gives lower errors compared to the minimum entropymethod in these two scenes.

scenes. Fig. 8 and Fig. 9 show the results.2 On the toprow, the color coded normal maps (x, y, z components arelinearly mapped in the RGB channel) are shown. On thebottom row is a depth map integrated from the normalmap by Poisson solver. For a validation, we show theground truth obtained by the calibrated photometric stereoon the left. We compare our method with the minimum en-tropy method [1]. Because our method uses surface normalgroups in addition to the albedo information, it performswell in GBR disambiguation. The average angular errorsare summarized in Table 1. Comparing the results betweenthe CAT and OWL scenes, because of the smoothly varyingalbedo of OWL scene, the performances of both our methodand the minimum entropy method degrade.

Self-calibrating photometric stereo result To test theentire pipeline, we use the PILLOW and SHEEP scenes takenby a Nikon D70 and Canon 20D, respectively. The resultsare shown in Fig. 10 and Fig. 11. In the figures, from leftto right, we show the calibrated result (known lighting andresponse function), the result of our method with and with-out the radiometric calibration step. These results show thatour method produces results that are close to the calibrateddata, although we only use the input images taken under un-known lighting parameters with an unknown response func-tion. The importance of radiometric calibration can also beseen clearly in the side-by-side comparison. Table 2 showsthe quantitative comparison. Our self-calibration methodsignificantly improves the accuracy. Fig. 12 shows the ren-dering of the reconstructed surface.

2We recommend viewing the electronic version of this paper for bettervisualization.

Calibrated Our method Min-entropy

Dep

th

N

orm

al

PDF created with pdfFactory Pro trial version www.pdffactory.com

Figure 8. GBR disambiguation result for CAT scene. From left toright, calibrated, our method and minimum entropy method [1] areshown. Top row shows the computed surface normal, and the bot-tom row shows the integrated depth maps. In the depth maps, thered color represents nearer, and blue color corresponds to furtherdepth values. The reference sphere shows the normal directions.

Calibrated Our method Min-entropy

Dep

th

N

orm

al

Angular error Cat Owl

Min-entropy mean 15.82 27.10

std. dev. 6.15 6.08

Our method mean 6.15 10.47

std. dev. 2.83 4.75

PDF created with pdfFactory Pro trial version www.pdffactory.com

Figure 9. GBR disambiguation result for OWL scene. From leftto right, calibrated, our method and minimum entropy method [1]are shown. Top row shows the computed surface normal, and thebottom row shows the integrated depth maps.

Angular error PILLOW SHEEP

Our method Mean 5.63 7.30Std. dev. 3.71 3.02

Without Mean 16.08 20.18radiometric calib. Std. dev. 7.35 6.86

Table 2. Mean angular error and standard deviation of self-calibrating photometric stereo result, and the result without radio-metric calibration.

6. Conclusion and discussionAuto-calibration for photometric stereo is important for

photometric stereo to be applicable in various settings.

Dep

th

Nor

mal

Calibrated Our method W/o radiometric calib.

PDF created with pdfFactory Pro trial version www.pdffactory.com

Figure 10. Result of the proposed method for PILLOW scene. Fromleft to right, calibrated, our method with and without radiometriccalibration.

Dep

th

Nor

mal

Calibrated Our method W/o radiometric calib.

Angular error Pillow Sheep

Withoutradiometric calib.

mean 16.08 20.18

variance 7.35 6.86

Our complete solution

mean 5.63 7.30

variance 3.71 3.02

PDF created with pdfFactory Pro trial version www.pdffactory.com

Figure 11. Result of the proposed method for SHEEP scene. Fromleft to right, calibrated, our method with and without radiometriccalibration.

PDF created with pdfFactory Pro trial version www.pdffactory.com

Figure 12. The surfaces of CAT and PILLOW scenes.

In this paper, we propose a self-calibrating photometricstereo method that handles both radiometric calibration andGBR disambiguation. The mixture of these problems issolved by analyzing color/intensity profiles in the RGB andirradiance-time domains. We show that radiometric calibra-tion can be performed from images from a fixed viewpointunder varying lightings. We also develop a method to au-tomatically resolve the GBR ambiguity from at least fourpixels with different normals but the same albedo.

Limitations Our radiometric calibration method has acouple of limitations. First, it cannot work with gray-scaleimages and scenes. Second, it cannot handle a certain classof response functions f where f(x)/f(ax) = const. holds.

This is a special case when the response function is non-linear while the color profiles remain straight. This classof function includes f(x) = xn, n ∈ R, when boundaryconditions f(0) = 0 and f(1) = 1 is imposed.

To assess the practical limitation of our method, wetested our method using the database of response functions(DoRF) [9] with a simulation dataset. We have confirmedthat all the response functions in the database, includinggamma correcting functions, bend the color profiles in theRGB color space. Among the degenerate case where the re-sponse function is in the form of f(x) = xn, only the linearresponse function f(x) = x is likely dominant in practice.3

Therefore, by setting the initial guess of the response func-tion as a linear function, our method works well for a widevariety of real-world camera response functions.

AcknowledgementThe authors would like to thank Neil Alldrin for provid-

ing the code to compare the results. Boxin Shi and Chao Xuwas partially supported by the Chinese and Beijing Natu-ral Science Foundation (Grant No.60975014 and 4102024).Ping Tan was supported by Singapore MOE Grant R-263-000-555-112.

References[1] N. Alldrin, S. P. Mallick, and D. J. Kriegman. Resolving the

generalized bas-relief ambiguity by entropy minimization. InProc. of Computer Vision and Pattern Recognition, pages 1–7, 2007.

[2] R. Basri, D. Jacobs, and I. Kemelmacher. Photometric stereowith general, unknown lighting. Int’l Journal of ComputerVision, 72(3):239–257, 2007.

[3] P. Belhumeur, D. J. Kriegman, and A. L. Yuille. The bas-relief ambiguity. Int’l Journal of Computer Vision, 35(1):33–44, 1999.

[4] M. Chandraker, F. Kahl, and D. J. Kriegman. Reflections onthe generalized bas-relief ambiguity. In Proc. of ComputerVision and Pattern Recognition, pages 788–795, 2005.

[5] O. Drbohlav and M. Chantler. Can two specular pixels cal-ibrate photometric stereo? In Proc. of Int’l Conf. on Com-puter Vision, pages 1850–1857, 2005.

[6] O. Drbohlav and R. Sara. Specularities reduce ambiguity ofuncalibrated photometric stereo. In Proc. of European Conf.on Computer Vision, pages 46–62, 2002.

[7] A. S. Georghiades. Incorporating the torrance and sparrowmodel of reflectance in uncalibrated photometric stereo. InProc. of Int’l Conf. on Computer Vision, pages 816–823,2003.

[8] M. D. Grossberg and S. K. Nayar. Determining the cameraresponse from images: What is knowable? IEEE Trans.on Pattern Analysis and Machine Intelligence, 25(11):1455–1467, 2003.

3The linear response as well as other response functions in the form off(x) = xn does not exist in the DoRF.

[9] M. D. Grossberg and S. K. Nayar. Modeling the space ofcamera response functions. IEEE Trans. on Pattern Analysisand Machine Intelligence, 26(10):1272–1282, 2004.

[10] H. Hayakawa. Photometric stereo under a light source witharbitrary motion. Journal of the Optical Society of America,11(11):3079–3089, 1994.

[11] S. Kim and M. Pollefeys. Radiometric alignment of imagesequences. In Proc. of Computer Vision and Pattern Recog-nition, pages 645–651, 2004.

[12] S. J. Koppal and S. G. Narasimhan. Clustering appearancefor scene analysis. In Proc. of Computer Vision and PatternRecognition, pages 1323–1330, 2006.

[13] S. Lin, J. Gu, S. Yamazaki, and H. Shum. Radiometric cal-ibration from a single image. In Proc. of Computer Visionand Pattern Recognition, pages 938–945, 2004.

[14] S. Lin and L. Zhang. Determining the radiometric responsefunction from a single grayscale image. In Proc. of ComputerVision and Pattern Recognition, pages 66–73, 2005.

[15] T. Mitsunaga and S. K. Nayar. Radiometric self-calibration.In Proc. of Computer Vision and Pattern Recognition, pages374–380, 1999.

[16] J. A. Nelder and R. Mead. A simplex method for functionminimization. Computer Journal, 7(4):308–313, 1965.

[17] T. Okabe, I. Sato, and Y. Sato. Attached shadow coding: es-timating surface normals from shadows under unknown re-flectance and lighting conditions. In Proc. of Int’l Conf. onComputer Vision, pages 1693–1700, 2009.

[18] I. Omer and M. Werman. Color lines: Image specific colorrepresentation. In Proc. of Computer Vision and PatternRecognition, pages 946–953, 2004.

[19] I. Sato, T. Okabe, Q. Yu, and Y. Sato. Shape reconstructionbased on similarity in radiance changes under varying illu-mination. In Proc. of Int’l Conf. on Computer Vision, pages1–8, 2007.

[20] P. Tan, S. P. Mallick, L. Quan, D. J. Kriegman, and T. Zickler.Isotropy, reciprocity and the generalized bas-relief ambigu-ity. In Proc. of Computer Vision and Pattern Recognition,pages 1–8, 2007.

[21] P. Tan and T. Zickler. A projective framework for radiometricimage analysis. In Proc. of Computer Vision and PatternRecognition, pages 2977–2984, 2009.

[22] B. Wilburn, H. Xu, and Y. Matsushita. Radiometric calibra-tion using temporal irradiance mixtures. In Proc. of Com-puter Vision and Pattern Recognition, pages 1–7, 2008.

[23] R. Woodham. Photometric method for determining sur-face orientation from multiple images. Optical Engineering,19(1):139–144, 1980.

[24] J. S. Yedidia, W. T. Freeman, and Y. Weiss. Understandingbelief propagation and its generalizations. In Proc. Int’l JointConf. on Artificial Intelligence, pages 239–269, 2001.

[25] A. L. Yuille, D. Snow, R. Epstein, and P. N. Belhumeur.Determining generative models of objects under varying il-lumination: Shape and albedo from multiple images usingsvd and integrability. Int’l Journal of Computer Vision,35(3):203–222, 1999.