JOURNAL OF LA SymPS: BRDF Symmetry Guided ...ysato/papers/Lu-TPAMI17.pdfA BRDF measures the ratio of...

0162-8828 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TPAMI.2017.2655525, IEEETransactions on Pattern Analysis and Machine Intelligence

JOURNAL OF LATEX CLASS FILES, VOL. , NO. , 2016 1

SymPS: BRDF Symmetry Guided PhotometricStereo for Shape and Light Source Estimation

Feng Lu, Xiaowu Chen, Imari Sato, and Yoichi Sato

Abstract—We propose uncalibrated photometric stereo methods that address the problem due to unknown isotropic reflectance. Atthe core of our methods is the notion of “constrained half-vector symmetry” for general isotropic BRDFs. We show that such symmetrycan be observed in various real-world materials, and it leads to new techniques for shape and light source estimation. Based on the 1Dand 2D representations of the symmetry, we propose two methods for surface normal estimation; one focuses on accurate elevationangle recovery for surface normals when the light sources only cover the visible hemisphere, and the other for comprehensive surfacenormal optimization in the case that the light sources are also non-uniformly distributed. The proposed robust light source estimationmethod also plays an essential role to let our methods work in an uncalibrated manner with good accuracy. Quantitative evaluationsare conducted with both synthetic and real-world scenes, which produce the state-of-the-art accuracy for all of the non-Lambertianmaterials in MERL database and the real-world datasets.

Index Terms—Photometric stereo, BRDF symmetry, uncalibrated light sources, isotropic reflectance

F

1 INTRODUCTION

PHOTOMETRIC stereo is an important research topicfor 3D scene recovery in computer vision. It aims

at recovering surface normals of a scene from a set ofimages captured under varying light sources directions.Compared with other techniques, e.g., multi-view stereo,photometric stereo takes into consideration surface nor-mal, illumination, and surface reflectance simultaneous-ly, and thus, it has broader prospects than other methodsand shows a good potential to be of great benefit to otherfields like computer graphics and virtual reality.

Photometric stereo problems can be defined andsolved under different assumptions. For instance, Wood-ham et al. [1] proposed the first well known methodthat assumes known light source directions and pureLambertian reflectance. However, without knowing thelight sources directions, the problem becomes muchmore difficult. As a result, most of the previous uncali-brated (i.e., when light sources are unknown) methodshave to assume simple surface reflectances, e.g., pureLambertian reflectance, or that the diffuse componentof the surface reflectance follows the Lambertian law. Indoing so, the surface normals can be routinely solved upto the Generalized Bas-Relief (GBR) ambiguity [2]. TheGBR ambiguity can be further resolved by using vari-ous properties observed in real-world BRDFs, includingsymmetries [3], [4], diffuse maxima [5] and pixel color

• F. Lu and X. Chen are with the State Key Laboratory of Virtual RealityTechnology and Systems, School of Computer Science and Engineering,Beihang University. F. Lu is also with the International Research Institutefor Multidisciplinary Science, Beihang University, Beijing, 100191, China.E-mail: lufeng,[email protected]

• I. Sato is with National Institute of Informatics, Tokyo 101-0003, Japan.E-mail: [email protected]

• Y. Sato is with the Institute of Industrial Science, the University of Tokyo,Tokyo 153-8505, Japan. E-mail: [email protected]

profiles [6]. Methods along this line cannot deal withsurfaces with non-Lambertian reflectances.

However, real-world scenes show a variety of complexsurface reflectances (represented by the bidirectional re-flectance distribution functions, BRDFs), ignoring whichwill greatly limit the applicability of photometric stereo.Therefore, it is desired to develop a technique that canhandle various surface reflectances/BRDFs when lightsources are unknown and no additional input is giv-en. Several recent methods are capable of solving thisproblem to some extent [7], [8], [9]. However, they stillface challenges especially when recovering the elevationangles of surface normals. For instance, [7], [8] mustassume that the light sources cover the whole sphereuniformly or else their accuracies drop significantly;while in practice, lighting a real scene from back isinconvenient and problematic – severe cast shadowscould be produced to badly affect the accuracy.

In this paper, we propose novel photometric stereomethods that work with general and unknown surfacereflectances and also unknown light sources distributedonly in the visible hemisphere, i.e., in the camera side.At the core of our methods is the idea to exploit surfacereflectance symmetries, namely, the 1D/2D constrainedhalf-vector symmetries, which are the key findings inthis paper (Section 2). We observe these symmetries invarious real-world isotropic BRDFs and use them todevelop two photometric stereo techniques. In particular,we investigate the 1D constrained half-vector symmetryand introduce an effective elevation angle optimizationmethod (Section 3). We also propose a second methodbased on the 2D constrained half-vector symmetry towork with non-uniform lightings accurately (Section 4).In addition, we present a technique that recovers un-known light sources with good accuracy (Section 5).

Our methods make the following assumptions: (1)




directional point light sources are positioned in thevisible hemisphere centered by the target scene with thesame intensity; (2) the target surface has homogeneousisotropic materials up to only color differences. Otherassumptions like an orthographic camera are consistentwith traditional photometric stereo settings. With theseassumptions, our methods are capable of estimating bothshape and light sources of the scene.

Our methods show the following merits over existingapproaches. First, we handle both unknown light sourcesand general and unknown surface BRDFs, which is chal-lenging for most previous photometric stereo methods.Second, we propose the constrained half-vector symme-tries, which can be observed in a variety of real-worldisotropic BRDFs. They are essential to allow our methodsto effectively solve the above challenging problem with-out requiring additional inputs. Third, our light sourceestimation is also able to handle various BRDFs in thescene and output robust results. Overall, our methodsachieve an average surface normal error of around 6 for100 real-world BRDFs in our setting, which outperformsexisting approaches.

1.1 Previous workWhereas conventional photometric stereo methods as-sume known light directions and Lambertian reflectance[1], recent studies have been proposed to relax these as-sumptions. In the case of unknown light sources, surfacenormals can be recovered up to the GBR ambiguity [2]by assuming Lambertian reflectance and integrability ofthe surface. The GBR ambiguity may be then resolved byusing additional scene properties, e.g., specularity posi-tion [10], [11], interreflections [12], low-dimensional fea-ture space [13], minimum entropy [14], color profiles [6],diffuse maxima [5] and reflectance symmetry [3], [4], orusing special light source configurations [15]. Besides, ifreplacing the orthogonal projection assumption by theperspective projection assumption, the GBR ambiguitycan also be solved during camera calibration [16].

Overall, these methods can work with unknown lightsources under the condition that the diffuse reflectancecomponent obeys the Lambertian law. However, for real-world surfaces having strong non-Lambertian reflectancecomponents, their applicability cannot be confirmed.

In order to handle non-Lambertian reflectances, mostexisting methods either regard the non-Lambertian com-ponents as outliers, or introduce various reflectancemodels for fitting and analysis. Robust estimation frame-works have been proposed to detect and remove non-Lambertian components from the original observations.To this end, techniques such as color analysis [17], shad-ow cues [18], median filter [19], rank-minimization [20],sparse regression [21], and Markov random field [22]are commonly used. Some other methods explore thereflectance properties, e.g., bilateral symmetry [23],halfway vector symmetry [24]1, isotropy and monotonic-

1 Detailed comparison is provided in the Supplemental Material.

ity [25], [26], or use explicit BRDF models, e.g., the Wardmodel [27], [18], the Lambertian+specular models [28],and the bi-variate BRDF model [29] to solve the problem.For these methods, light source directions are usuallyessential to know to ensure good accuracy.

There are fewer methods that work with both uncal-ibrated light sources and general isotropic reflectances.They may need to put additional objects with knownshapes and reflectances in the scene as references [30],[31], and use complex reflectance models in formula-tion [32]. Given the BRDF dataset, the extracted principalcomponents also help handle unknown reflectances [33]and solve the photometric stereo problem [34]. Othermethods include those using light sources placed in aring [9] or exploiting the similarity of pixel intensityprofiles especially when the light sources cover thewhole sphere [7], [35], [8]. They work well in recoveringazimuth angles of surface normals, but face difficulty [7],[35], [8] or require additional inputs [9] to estimate theelevation angles of the surface normals accurately. Inparticular, if the light sources only cover the visible hemi-sphere in the camera side, or their distribution is highlynon-uniform, the methods described in [7], [35], [8] willproduce large errors especially in the elevation anglesof the surface normals. Compared with these methods,our method accurately solves for surface normals undermore general light source settings, and it achieves state-of-the-art results without requiring additional inputs.

2 CONSTRAINED HALF-VECTOR SYMMETRY

In this section, we review the definition and parameteri-zations of BRDFs and some of the important symmetries.Then, we propose the constrained half-vector symmetry.Specifically, the 1D version of the symmetry can beobserved in a 1D BRDF slice, and the 2D version ofthe symmetry is related to a set of BRDF iso-contours.They are suitable to help optimize the solution of surfacenormals in photometric stereo problems.

2.1 BRDF and Its ParameterizationA BRDF measures the ratio of the reflected radiance froma surface patch. It is a function f(ωin,ωout) of incomingand outgoing light directions in a local coordinate sys-tem. Many recent studies have used halfway/differenceparameterization [36] to represent BRDFs. By introduc-ing the half vector, which is the bisector of ωin andωout, a BRDF can be parameterized as f(θh, φh, θd, φd),as illustrated in Fig. 1 (a). Thus, a pixel value capturedat a surface patch can be expressed as

I = f(θh, φh, θd, φd)(nTs), (1)

where n is the surface normal of the surface patch ands is the point light source.

The above 4D BRDF parameterization domain canbe reduced by using some of the symmetries observedin many real-world materials. For instance, the widelyobserved ‘isotropy’ reduces the BRDF to a 3D function




n

hωin

ωout

ϕh

θd

θhϕd

v (ωout) h

S(ωin)

Elevation angles

n

h

Elevation angle of n

BRD

F va

lue

1D symmetry v (ωout) h

S(ωin)

n

θh θh

BRD

F va

lue

2D symmetryBRDF iso-contours

BRDF 1D slice h

ϕd

(a) Local C.S. (b) 1D Symmetry in view-centered C.S. (c) 2D Symmetry in view-centered C.S.

Fig. 1. (a) Standard BRDF parameterization; (b) in a view-centered coordinate system, we examine the surfacenormals having the same azimuth angle as the light source. We observe the 1D constrained half-vector symmetry inthe resulting 1D BRDF slice; (c) in a view-centered coordinate system, we examine surface normals centered by thehalf vector. We observe BRDF iso-contours and the 2D constrained half-vector symmetry.

f(θh, θd, φd) which is invariant with respect to φh [37].This is the most widely adopted assumption in the relat-ed research, and nearly all materials in the MERL BRDFdatabase conform to it [38]. Moreover, the ‘reciprocity’and ‘bilateral’ symmetries further fold the domain of φdfrom [0, π] to [0, π/2]. Finally, by considering the ‘half-vector’ symmetry, which assumes an invariant BRDFagainst a rotation φd of both ωin and ωout around thehalf vector, the BRDF function can be further reduced toa bivariate one, denoted as f(θh, θd).

Note that these symmetries are studied in the localcoordinate system at a single surface normal, as shownin Fig. 1 (a). On the contrary, in this paper, we proposeconstrained half-vector symmetries in the view-centeredcoordinate system, where multiple surface normals areinvolved and constrained. As a result, the proposed sym-metries show potentials to help directly solve/optimizesurface normals in photometric stereo problems.

2.2 1D Constrained Half-Vector SymmetryLet us start with a special case of isotropic reflectancewherein a set of surface normals n with differentelevation angles share the same azimuth angle with alight source s in the view-centered coordinate system, asshown in Fig. 1 (b).

This results in a fixed θd and also φd = 0 or π1

in the conventional BRDF parameterization with localcoordinates in Fig. 1 (a). Consequently, this simplifiesthe isotropic BRDF function to either fθd,φd=0(θh) orfθd,φd=π(θh). If one further assumes reciprocity, meaningthat the roles of ωin and ωout can be interchanged, thetwo representations can be unified to be fθd,φd=0(θh).Note that in Fig. 1 (b), θh is simply the differencebetween the elevation angles of h and n. Since h isfixed, fθd,φd=0(θh) is fully determined by the elevationangle of n, which corresponds to a 1D slice from theoriginal BRDF, as shown in Fig. 1 (b). Since fθd,φd=0(θh)is symmetric about the half-vector (i.e., θh = 0), it leadsto the following observation shown in Fig. 1 (b).

Observation 1: By assuming isotropy and reciprocity,BRDF values captured at surface normals having thesame azimuth angle as the light source direction con-stitute a 1D slice (Fig. 1 (b)). BRDF data in the 1D slice

1ωin and ωout can be rotated by π to interchange their positions,while still ensuring that n, ωin, and ωout are coplanar.

should be distributed symmetrically about the half vec-tor regarding the elevation angle of the surface normal.

While this property is a direct result of assumingisotropy and reciprocity, it can also be derived from thehalf-vector symmetry by fixing θd in a bivariate BRDFfunction2. For convenience, we will use the term “1Dconstrained half-vector symmetry” in the rest of thispaper since the concept of the half vector is important.

2.3 2D Constrained Half-Vector SymmetryKnowing the light source s, we consider a more generalcase where surface normals n are no longer con-strained by the same azimuth angle. Instead, we takeinto account all the surface normals near the half vectorh, as shown in Fig. 1 (c).

In such a case, θd is fixed and the isotropic BRDFfunction can be written as fθd(θh, φd). According to [36],fθd(θh, φd) for a common isotropic material is invariantagainst the changes of φd, meaning that the BRDF valueis fully determined by θh; therefore, if we plot surfacenormals n in the view-centered coordinate system inFig. 1 (c), those surface normals with the same θh shouldcorrespond to the same BRDF value: there are BRDF iso-contours centered by h, and the BRDF value at each iso-contour is fully deteremined by θh. Since all iso-contoursconstitute a 2D surface, the resulting symmetry is called“2D constrained half-vector symmetry”.

Observation 2: By assuming half-vector symmetry,BRDF values captured under the same light sourceconstitute concentric iso-contours (Fig. 1 (c)), and eachcontour corresponds to surface normals sharing the sameangular difference from the half vector.

The “1D constrained half-vector symmetry” is clearlya special case of the “2D constrained half-vector symme-try” that corresponds to the φd = 0 and φd = π cases.

3 SURFACE NORMAL’S ELEVATION ANGLERE-MAPPING USING 1D SYMMETRY

In this section, we propose recovering accurate elevationangles of surface normals by using the 1D constrainedhalf-vector symmetry. This method works efficiently e-specially when scene images are captured by a fixed

2In most cases, assuming half-vector symmetry is stronger thanassuming isotropy and reciprocity.




camera with roughly uniform light sources in the visiblehemisphere. Note that we will use known light sourcepositions to describe the method. However, this assump-tion will be discarded in Section 5.

3.1 Symmetric Data Acquisition from ImagesWe employ the previously proposed method [7] forobtaining the initial estimates of the surface normalsfrom images. Pixel profiles are obtained to measure thesurface normals’ similarities. Then the surface normalsare recovered from these similarity measurements. Be-cause the light sources only cover the visible hemisphere,the measurements are asymmetric along the viewingdirection. Preliminary experiments indicate that the re-sulting average error in the elevation angle is more thantwice as that in the azimuth angle.

In order to optimize the elevation angles using theproposed 1D symmetry, we extract symmetric pairs from1D BRDF slices based on the initialized surface normals.For simplicity, let us describe this procedure for a givenlight source s. We will do the same for each of the otherlight sources. In the view-centered coordinate system,we find a set of captured pixels whose correspondingsurface normals nj share the same azimuth angle withs. In practice, a threshold of the azimuth angle differencecan be used, e.g., ∆a = 1.0, to ensure that the numberof selected nj is larger than 10. Then, each pixel valueis converted into a BRDF value by dividing it by nj · s,using Eq. (1). Finally, we obtain a 1D slice ρ containingall BRDF values observed at nj.

The next step is to extract symmetric pairs. Given theviewing direction [0, 0,−1]T and the light source s, wefirst compute the half vector as their bisector and recordits elevation angle as h. We also find the surface normalin nj which corresponds to the largest BRDF value inρ and record its elevation angle as δ. It is reasonable toassume that the largest BRDF is observed at h, and thus,δ should be close to h. Then, we extract the elevationangles of nj in a pairwise manner which satisfy the1D constrained half-vector symmetry. We traverse allelements of ρ, and for each of them, indexed by theelevation angle αj , we find in ρ another element βj suchthat 1) αj and βj correspond to the same BRDF value;2) αj and βj are located on two sides of δ. Note that αj ,βj and δ are the initial elevation angles or their currentestimates during the iteration (see Section 3.3).

In practice, βj may not have exactly the same BRDFvalue as αj . Therefore, instead of pursuing an ideal βj ,we find two elevation angles βj and γj , whose BRDFvalues are closest to that of αj . By assuming the BRDFdata are not extremely sparse and their values changesmoothly, we compute the weights µj and νj satisfyingthe linear interpolation ρ(αj) = µjρ(βj) + νjρ(γj). Thenthe symmetry partner of αj can be well approximated byµjβj + νjγj . As a result, (αj , µjβj + νjγj) is consideredto be a symmetric pair, as illustrated in Fig. 2 (a). In ourexperiments, the number of collected BRDF symmetricdata ranges from 1800 to 2200 for different materials.

h

BR

DF

valu

e

Elevation angle of n

Interpolation

Symmetric pair

αjβj

γj

λ(k)

k

c

45

Smoothness strength

(a) (b)

Fig. 2. (a) For each αj , we interpolate its symmetricpartner in the 1D slice; (b) smoothness level in Eq. (4).

Ideally, the data collected under the light source sshould satisfy the 1D constrained half-vector symmetryin the following ways: 1) δ = h and 2) αj+(µjβj+νjγj)

2 =h, which means h should be the bisector of the elevationangles of any symmetric pair, as shown in Fig. 2 (a).However, if these data are computed from inaccurate es-timates of the elevation angles, the symmetry cannot besatisfied. Therefore, we can use symmetry as a criterionto correct the elevation angles of the surface normals.

3.2 Formulation with 1D BRDF Slices

By using the 1D symmetric data described in the pre-vious section, we propose refining the elevation anglesof the surface normals so that they will satisfy the1D constrained half-vector symmetry for all 1D BRDFslices. We model the refinement as an elevation anglere-mapping process2. We establish a one-to-one globalmapping ε 7→ ε : ε = m(ε) with boundary conditionsm(0) = 0 and m(π/2) = π/2 to refine any elevationangle ε. The mapping can be simple or highly non-linear,depending on the accuracy of the initial elevation angles.

We formulate the problem by using constraints de-rived from the 1D constrained half-vector symme-try: for the ith light source si, the collected eleva-tion angle pairs should satisfy 1) m(δi) = hi and2) m(αj

i )+[µjim(βj

i )+νjim(γj

i )]

2 = hi after the re-mapping.This directly leads to the formulation,

Edata(m(·)) =∑i

[κ [m(δi)− hi]2︸︷︷︸align extreme to hi

+

∑j

[m(αji ) + [µjim(βji ) + νjim(γji )]− 2hi

]2︸︷︷︸enforce symmetry of every elevation angle pair about hi

], (2)

where κ balances the influence of different terms. Inpractice, we use κ = 10, due to the importance ofinformation from the half vector itself. Note that Eq. (2)uses an outer summation to take into account equationsfrom all light sources si.

In addition, a smoothness term can be added to Eq. (2)to suppress the influence of outliers. To this end, we force

2Azimuth angles of surface normals can be assumed correctlyestimated beforehand, supported by results from [7], [8]




the mapping function to change gradually by minimiz-ing the sum of the squares of the second derivative ofm(·) as

Esmooth(m(·)) =90−1∑k=0+1

λ(k) [m′′(k)]2, (3)

where the mapping function m(·) is sampled at k =0, · · · , 90, and λ(k) determines the level of elevation-angle-dependent smoothness. In this discrete formula-tion, we use m′′(k) = m(k − 1) − 2m(k) + m(k + 1) tocompute the second derivative. Note that although wesample m(·) at elevation angles from 0 to 90 by a stepof 1, it is straightforward to use a formulation with ahigher or lower sampling density.

The balance weight λ(·) is specially designed to be

λ(k) =

c× (2× cos(4k) + 3) k < 45

c otherwise, (4)

where c is a constant value (= 100 in our experiments).The curve of the smoothness level is in Fig. 2 (b). Byusing this spatially varying smoothness, small elevationangles (which tend to suffer from noise because theirsurface points are always dark without lighting from theback) are smoothed out to avoid effects due to the lowsignal-noise-ratio.

The final formulation of the problem takes into con-sideration both terms as follows:

m(·) = argminm(·)

(Edata + Esmooth),

s.t. m(0) = 0,m(π/2) = π/2,

m([0, π/2]) ⊆ [0, π/2]. (5)

3.3 Solution and AnalysisWe solve the unknown mapping m(·) via matrix calcula-tions. Under the condition that the elevation angles aresampled from 0 to 90 in steps of 1, we can re-definem(·) as a 1D vector m ∈ R91×1, while its kth elementequals m(k) which is the mapping result for elevationangle k. Thus, m numerically defines a mapping ofelevation angles, which allows us to re-write Eq. (5) as

m = argminm

‖Am− b‖2 s.t. m ≥ 0, (6)

where the matrices A and B are constructed as follows:column: [βj

i ] [αji ] [δi] [γj

i ]

A =

...

... ...... ...

... ...... ...

... ... ... 2·κ ... ...

... µji ... 1 ... ... ... νj

i ...

...... ...

... λ(k) −2·λ(k) λ(k)... ...

......

.... . .

...Inf 0

0 Inf

,b =

...2·hi

2·hi

...0...0

90·Inf

,

(7)where [βji ] takes the rounded value of βji and indicatesthe column index. Note that the same row from thematrix A and vector b exactly constitute a sub-formula

in either Eq. (2) or Eq. (3), except for the last tworows representing the boundary conditions for 0 andπ/2, where ‘Inf’ indicates a constant large value (= 105

in our experiments). Therefore, the solution of Eq. (6)numerically approximates the solution of Eq. (5). Thesolution can be obtained by using least squares solverswith non-negative constraints.Iteration In Section 3.1, we compute the BRDF values byusing the initial elevation angles, and thus, those valuesare inaccurate. After the elevation angles are updatedusing our solution, we re-compute the BRDF values andsolve for the elevation angles again. Note that we do notupdate the choice of nj since the azimuth angles areassumed correct. In practice, 3 iterations suffice to get afinal solution.Solution Uniqueness It is important to know whetherthe solution to Eq. (6) is unique or not in order todetermine the surface normals.

Proposition 1: Without considering the smoothnessterm, the problem in Eq. (5) (Eq. (6)) has a uniquesolution under the following conditions: 1) the surfacereflectance is not Lambertian; 2) there are at least twolight sources with different elevation angles.

The first condition is obvious, as Lambertian re-flectance has the same BRDF value everywhere, whichcannot provide information to determine a unique sym-metric structure. Many diffuse materials like latex andrubber cause this problem, which can be detected by thenon-convergence of the solution; in such cases, we cansimply keep the original results produced by methodslike [7], since Lambertian-like materials are easy for theconventional methods to handle.

The second condition can be understood intuitively.Our 1D symmetry cannot determine the elevation anglesof the surface normals only with one half vector, sincewhatever distribution the elevation angles on one sideof the half vector have, the 1D symmetry can still besatisfied by changing the other side symmetrically. Onlyanother 1D slice with a different half vector can removethis additional degree of freedom. A stricter prove isgiven in the Supplemental Material.

4 SURFACE NORMAL OPTIMIZATION USING2D SYMMETRY

In this section, we propose optimizing surface normalsby using the 2D constrained half-vector symmetry. Com-pared to the method introduced in Section 3, this newmethod has an increased complexity; in return, it canhandle light sources with heavily non-uniform distribu-tions. Note that we will use known light source positionsto describe the method here, and this assumption will bediscarded in Section 5.

4.1 Symmetric Data Acquisition and Preparation

Similar to Section 3.1, we employ [7] to obtain the initialestimates of the surface normals from images. Then, the




θh

BRD

F va

lue h

ϕd BRD

F va

lue

θh

Captured data

Fitted data(ideal)

(a) (b)

Fig. 3. (a) BRDF values show 2D symmetry regardingθh; (b) relation between BRDF values and θh values. Notethat inaccurate surface normal will bias the captured dataand cause error in θh.

method introduced in Section 3 can effectively optimizethe elevation angles of surface normals by a globalelevation angle remapping. However, this requires thatthe light source distribution is uniform; while in the caseof a heavily non-uniform light source distribution, wepropose fully optimizing every surface normal by usingthe 2D version of the proposed symmetry.

Data preparation and acquisition can be done as be-low. Knowing the viewing direction [0, 0,−1]T and the i-th light source si, the half vector hi, surface normals njand their corresponding BRDF values ρi(nj) should becomputed before collecting the 2D symmetric data, asshown in Fig. 1 (c). The half vector hi can be computedas the bisector of [0, 0,−1]T and si; the surface normalsnj are directly set to their initial estimates from [7]; theBRDF values ρi(nj) are then computed by dividing thepixel value at each surface normal by nj · si.

Consequently, to collect 2D symmetric data, the moststraightforward way is to densely sample different sur-face normals nj, compute their corresponding θjh andφjd, and group them into multiple BRDF iso-contours,as shown in Fig. 3 (a). However, in practice the cap-tured data cannot be dense enough to constitute idealiso-contours for possible BRDF values. To handle thisdifficulty, we fold the 2D BRDF space by removing thedifference in φd, which transform every iso-contour to bea point. The resulting data pattern is shown in Fig. 3 (b).Then, relation between the BRDF value ρ and θh can becomputed as ρ = g(θh) via nonlinear fitting using thecaptured data. In practice, if a capture data point showsa different θh from that in the fitting result, it should becorrected to satisfy the symmetry. Following this idea,we compute the symmetric data as below.

Let nj denote the current estimate of a surface nor-mal, and thus the corresponding θh value is θih(nj) =arccos(hT

i ·nj). On the other hand, according to the fittingresult, the ideal θh value should be g−1(ρi(n

j)). Thentheir difference is ∆θh = arccos(hT

i · nj) − g−1(ρi(nj)).

As a result, nj should be corrected to an ideal surface

normal eji to satisfy the symmetry:

eji = argmineji

[((nj)Teji − cos(∆θh)

)2+

(hTi eji − cos(g−1(ρi(n

j))))2

+(‖eji‖2 − 1

)2]. (8)

The above obtained surface normal pairs (nj , eji ) arecollected and stored as the symmetric data. As a specialcase, let nmax

i denote the surface normal correspondingto the largest BRDF value under light source si, and itsexpected ideal surface normal will be hi according toEq. (8). Besides, note that in practice, we only recorddata for a subset Ωi of all the surface normals, whichhave limited angular differences from h (in our setting,a threshold is set to 60, which produces around 15000symmetric data in average for each surface.)

4.2 Formulation with 2D symmetric data

By using the 2D symmetric data obtained in the previoussection, we propose optimizing all the surface normalsby using the 2D constrained half-vector symmetry. Theoptimization function contains three terms, namely, the2D symmetry term, the half vector term, and the locallinear structure term. They are specified as below.2D Symmetry Term Given the 2D symmetric data(nj , eji ) for all the light sources, and by assuming thatnj should be corrected to eji , as explained before, we canwrite down the following energy function to refine nj :

Esym(nj) =∑i

∑j∈Ωi

(nj − eji )2, (9)

where Esym penalizes the differences between nj and ejiand sums them up for all the light sources.Half Vector Term For the special pair (nmax

i ,hi), wewrite down its individual energy function as:

Eh(nmaxi ) =

∑i

(nmaxi − hi)

2. (10)

This term ensures that the maximum of the BRDF lobecoincides with the half vector’s direction. It is a specialcase of the 2D symmetry term but with higher impor-tance, due to the fact that hi determines the axis ofsymmetry as shown in Fig. 3 (a).Local Linear Structure Term Besides, we introduce thethird term Estr to preserve local structures3 of similarsurface normals during iteration, so that local surfacenormals can be updated as a whole. This is because Esymand Eh can only involve and optimize a set of surfacenormals, and there are rest surface normals unhandled.For this seek, we first solve for a set of weights wj,k,which are commonly considered representing the linearstructure of a manifold, based on the surface normals

3Note that here ‘local structure’ is defined in the surface normaldomain, not in the image domain.




v (ωout)

n

v (ωout) h1

h2

v (ωout)

n2 h2max n1 h1

max

Preserved structure

BRDF iso-contours Anchors

(a) (b) (c)

Fig. 4. Illustration of the proposed optimization terms:(a) 2D BRDF symmetry; (b) half vector’s direction as aspecial case; (c) local linear structure preservation whichinvolves additionally added anchor normals.

from the initialization or the previous iteration, via thefollowing minimization:

wj,k = argminwj,k

∑j

njpre −∑k∈Nj

wj,k · nkpre

2

, (11)

where Nj indicates that surface normals nkpre and njpreare neighbors with a limited angular difference.

To constrain the solution especially in the boundaryarea (surface normals perpendicular to the viewing di-rection), we add an additional set of virtual surfacenormals ηt (t = 1, · · · , 24 in our experiments) as anchorsto the linear reconstruction and their orientations are per-pendicular to the viewing direction as shown in Fig. 4 (c).These anchor normals are unchangeable during iterationand thus they help establish the boundary conditions.Consequently, the term wj,k · nkpre in Eq. (11) is replacedby wj,k · χk, where χk comprises of both nkpre or ηt ifthey are neighbors of nj .

Then, the local linear structure term Estr can be writ-ten as

Estr(nj) =

∑j

nj −∑k∈Nj

wj,k · χk2

, (12)

where the weights wj,k are computed as describedabove. This term is designed to preserve local structurein the surface normal domain. The change of a singlesurface normal during optimization will affect all itsneighbors so that their local relationship remains.

By now, we have introduced three terms for optimiza-tion. There are visually illustrated in Fig. 4. Finally, wecan optimize for all the surface normals by solving thefollowing problem containing all those terms:

nj = argminnj

(λsym · Esym︸︷︷︸

2D symmetry

+ λh · Eh︸︷︷︸half vector

+ λstr · Estr︸︷︷︸local linear structure

),

(13)where weights are set to be λsym = λstr = 0.01λh.

4.3 Solution and AnalysisWe solve the problem in Eq. (13) by first converting itinto the matrix form. Let X ∈ RNd×3 be a matrix witheach row being an unknown surface normal χd:

X = χd = [n1, . . . ,nNj

,η1, . . . ,ηNt

]T, (14)

and Nd = N j +N t. Then, Eq. (13) can be re-written as

X = argminX

∑i

(X −Ei)TΛi(X −Ei)

+XT(I −W )T(I −W )X.

(15)

Definition of each notation is given here. I is the identitymatrix. W is the weight matrix whose elements arewj,k from Eq. (11). Each Ei is a matrix with the samesize of X , and each Λi is a diagonal matrix. In particular,Ei and Λi are formed by

(Ei)d =

edi d ∈ Ωi

hi nd = nmaxi

ηd−Nj

d > N j

0 otherwise

, (Λi)dd =

1 d ∈ Ωi

100 nd = nmaxi

∞ d > N j

0 otherwise

.

(16)Eq. (15) is a quadratic function w.r.t.X , whose mini-

mum can be achieved when the following equation issatisfied:[∑

i

Λi + (I −W )T(I −W )]X =

∑i

ΛiEi, (17)

which is a linear system and can be solved routinely.Finally, the first N j rows ofX are taken as the estimationresults for surface normals nj.Iteration In Section 4.1, the 2D symmetric data are com-puted from the initial surface normals and their corre-sponding BRDF values. Since the initial surface normalsare unreliable, their errors may affact the final surfacenormal accuracy. In order to obtain robust solutions, were-compute the 2D symmetric data by using the surfacenormals from Eq. (17), and use them to solve Eq. (17)again. Besides, in the first one or two iterations, we set(Λi)dd = 0 when d ∈ Ωi in Eq. (16). This helps reduce theinfluence of inaccurate symmetric data due to unreliablesurface normal initialization. In practice, no more thanfive iterations suffice to output a final solution.Solution for Diffuse Materials Although the 2D sym-metry can be well satisfied by common isotropic re-flectance, the resulting method in Section 4.2 may faceproblem with diffuse materials. The reason is that adiffuse material have the same BRDF value everywhere,and therefore different θd correspond to the same BRDFvalues in Fig. 3 (b), which becomes incapable to con-strain the solution. Such cases can be easily identifiedby examining the fitting results as that in Fig. 3 (b).In such cases, as discussed in Section 3.3, we simplykeep the original results produced by methods like [7],since diffuse materials can be effectively handled byconventional methods and need no more optimization.

5 ROBUST LIGHT SOURCE ESTIMATION

The previous sections use known light source position-s to describe the proposed symmetry and methods.If these sources are unknown, we propose estimatingthem from the input images. By assuming that the lightsources cover the visible hemisphere, Winnemoller etal. [39] show that 3D light source positions can be




Inpu

t im

ages

Dat

a lin

earit

yLi

ght s

ourc

e re

cove

ry

0 0.5 10

20

40

60

80

0 0.5 1 1.50

20

40

60

80

0 0.5 1 1.5 20

20

40

60

80

Ambiguity

Image diff.

Ligh

t diff

. [d

eg.]

…

(a) (b) (c)

Fig. 5. Examples of 3D light source recovery from im-ages. We compare different methods in terms of theirresulting data linearity and the recovered light sources. (a)[39]: corr.=0.35, err.> 20; (b) [41]: corr.=0.86, err.=8.50;(c) our method: corr.=0.96, err.=4.16.

recovered by using image differences caused by varyinglight source directions. In particular, let us denote apixel profile consisting of pixel intensities Ipi fromthe image captured under the i-th light source as ξi =[I1i , · · · , I

pi , · · · ]T. As demonstrated in [39], by varying

the light source position moderately, the geodesic differ-ences4 between ξi show a strong linear relation withthe differences between the light sources directions. Fol-lowing this observation, one can apply dimensionalityreduction (e.g., ISOMAP [40]) to embed all the ξi intothe 3D space and map them onto the visible hemisphereto approximate the unknown light sources.

Inspired by the same idea, we can also use images toestimate their corresponding light sources. However, ourproposed techniques are distinctive in that:

(1) While [39] only handles nearly diffuse reflectancewithout strong specularity, our method can workrobustly for general isotropic reflectances.

(2) Instead of applying dimensionality reduction, ourmethod directly solves for 3D light sources.

(3) We resolve their remaining rotation/flip ambiguityby aligning the light sources to the surface normals.

General Reflectances An example of light sources recov-ered by [39] is shown in Fig. 5 (a), where the capturedsurface material is ‘aventurnine’. The resulting averagelight source error is very large, showing that [39] has dif-ficulty in handling specular materials. To understand thereason, we also plot the relation between ξi’s geodesicdifferences and light source differences in that case. Thecorrelation coefficient is only 0.25, meaning that thebasic assumption of linear relation is no longer valid. Toovercome this problem, [41] converts pixel profiles ξiinto pixel order profiles ξi = [o(I1

i ), · · · , o(Ipi ), · · · ]T,

4The geodesic distance between two nodes adds up local Euclideandistances along the shortest path between the two nodes [40].

where any pixel value Ipi is replace by its ascendingorder o(Ipi ) among all the pixels in the image. This avoidsextreme pixel values due to specularities and results inmuch better estimates as shown in Fig. 5 (b). However,determining pixel orders within the shadow areas isproblematic, since those values are extremely small andnoisy. Therefore, ξi contain inventible errors.

We propose a robust method with improved perfor-mance. First, pixel order profiles ξi are computed inthe same manner with [41]. Then, the difference betweentwo profiles is computed by

dist(i, j) =(∑

p

ςp

)−1∑p

ςp · (Ipi − Ipj )2, (18)

where ςp = 1 only when Ipi and Ipj are all from non-shadow regions, otherwise ςp = 0. In this way, thenegative effect cause by shadow regions is eliminated.For synthetic data, we simply find shadow regions withpixel intensities equal to 0, while for real data wemanually chose a threshold according to the measuredpixel intensities. Finally, we compute the geodesic dis-tances distg(i, j) from dist(i, j), which show a verygood linear relation (correlation coefficient: 0.96) withthe corresponding light source differences, as shown inFig. 5 (c). The estimated light source directions are almostidentical to the ground truth.Formulation and Solution Consequently, we recoverlight source directions. Observing that distg(i, j) is lin-early correlated to the angular difference between lightsources si and sj , we can explicitly model this by

arccos(sTi sj) ∝ distg(i, j) = λs · distg(i, j), (19)

where λs is a scale factor. By assuming that light sourcescover the visible hemisphere, it is reasonable to haveλs = max(distg(i, j))/π . Now, let S = [s1, · · · , si, · · · ]Tand Dg = distg(i, j), and then Eq. (19) can be re-written in the matrix form:

STS = cos(λsDg) s.t. rank(S) = 3, (20)

which can be solved iteratively by using SVD decom-position, while during each iteration only the first threesingular values are kept to initialize the next iteration.Rotation/Flip Ambiguity The solution is invariant a-gainst rotations/flips of all light sources around/aboutthe viewing direction, as shown in Fig. 5 (a). Win-nemoller et al. [39] suggest to manually solve this am-biguity, whereas we propose an automatic method thataligns the light sources to the recovered surface normals.

We first model the rotation and flip ambiguity as

Q =

σ · cosφv −σ · sinφv 0

sin θv cos θv 00 0 1

, (21)

where θv controls the rotation around the viewing direc-tion, and σ = ±1 flips the axis. Then, any light source siwill be converted to Qsi given the Q.




(a) Light sources (b) 3D surfaces for synthesis

Fig. 6. Light sources and 3D surfaces (Hemisphere,Bunny, Dragon, Rabbit, Buddha, Armdillo, Beethoven andLion) used in the experiments.

A simple observation enables us to solve Q. That is,any light source si always shares the same azimuth anglewith the surface normal ni, which corresponds to thebrightest pixel Imax

i that si can produce. Thus,

Q = argminQ

∑i

[Imaxi · arccos

c2,1(Qsi)T · c2,1(nmax

i )

|c2,1(Qsi)| · |c2,1(nmaxi )|

],

(22)where c2,1(·) keeps the first two rows and one columnof the input vector, and the pixel value Imax

i balancesthe influence of the current term since bright pixels arereliable. Eq. (22) can easily be solved since the onlyunknown is the angle θv and σ = ±1. In this way, Qyields correct light sources Qsi.

6 EXPERIMENTAL EVALUATION

6.1 Quantitative Evaluation with Synthetic DataWe produce synthetic scenes with different surface re-flectances (BRDFs), light sources, and 3D surfaces forquantitative evaluation. The methods proposed in Sec-tion 3 and Section 4 are mentioned as “1D-sym” and“2D-sym”, respectively.

We use 100 different BRDFs in the MERL BRD-F database [38], which cover a large variety of realworld surface reflectances. We generate two sets of lightsources, as shown in Fig. 6 (a), which contain 82 and 83sources, respectively. Unlike in [7], [8], these light sourcesonly cover the visible hemisphere, not the whole sphere.Details about the light sources are discussed later. Foreach BRDF and light source, we produce images using3D surfaces as shown in Fig. 6 (b). Note that inter-reflections are not considered during rendering. Theseimages serve as inputs to our method. To initialize thesurface normals, we use the method in [7].100 BRDFs + roughly uniform light source distri-bution. First, we test our method using hemisphericalsurfaces. A hemispherical surface contains all visiblesurface normals and thus it is suitable to use for quan-titative evaluation. We choose 82 light sources as shownin the top of Fig. 6 (a). These sources cover the visiblehemisphere with a roughly uniform distribution.

Table 1 shows the average results for all the 100BRDFs. The average surface normal error is 6.01 for

TABLE 1Average surface normal/light source errors for 100

BRDFs under roughly uniformly distributed light sources.

Method Init. err. Final err. Superior cases Light err.1D-sym 15.63 6.01 17× 3.00 6.12

2D-sym 15.63 5.65 61× 1.42 6.12

Skipped (Good enough initialization)Processed

Fig. 7. Examples of automatic material type classification.

the 1D-sym method and is 5.65 for the 2D-sym method.These results are pretty accurate considering the fact thatboth the BRDFs and the light sources are unknown. Incontrast, the initial surface normals obtained by using[7] shows a much larger average error of 15.63 due tothe fact that it lacks the ability to handle light sourcesonly in the visible hemisphere. Also note that among allthe 100 BRDFs, the 2D-sym method outperforms the 1D-sym method in 61 times. However, their average errorsare still quite similar to each other.

Moreover, our methods detect and skip the diffuseBRDFs automatically as described in Section 3.3 andSection 4.3. Their initial surface normals are used to bethe final outputs. We show examples of processed andskipped materials in Fig. 7. Our methods clearly showthe ability to determine which types of BRDFs shouldbe chosen for further refinement.

Our methods also recover light source directions. Asshown in Table 1, the average light source error is6.12, while the method [7] used for initialization cannotpredict light source directions.

Finally, for the 1D-sym method, we plot its error foreach BRDF in Fig. 8 and show some representativeexamples of elevation angle mappings and the recovered1D symmetries. In particular, material A, B and C areprocessed correctly and their 1D BRDF symmetries canbe seen; while material D has the worst accuracy due toits additional lobe besides the specular lobe that breaksthe 1D symmetry. Fig. 8 also shows highly identicalshapes between the computed elevation angle mappingcurves (blue) and the ground truths (red).100 BRDFs + non-uniform light source distribution.We then evaluate the proposed methods when the lightsources are non-uniformly distributed. We choose 83light sources as shown in the bottom of Fig. 6 (a). Theirdistribution is obviously non-uniform: light sources aredense in one area and sparse in the other area.

Table 2 shows the average results for all 100 BRDFs.The average surface normal error is 9.68 for the 1D-symmethod and is 6.58 for the 2D-sym method. Althoughboth methods improve the initial solution greatly, the2D-sym method has clearly a better average performancethan the 1D-sym method; it is better for 86 BRDFswhich are almost non-diffuse ones. Remember that thetwo methods perform similarly when the light sources




05

101520253035404550

yello

w-p

last

icre

d-fa

bric2

spec

ular

-whi

te-…

blac

k-fa

bric

poly

ethy

lene

pink

-pla

stic

red-

spec

ular

-…pi

nk-fa

bric

poly

uret

hane

-foam

viol

et-ru

bber

light

-bro

wn-

fabr

icw

hite

-pai

ntm

aroo

n-pl

astic

gree

n-ac

rylic

yello

w-p

heno

licbe

ige-

fabr

icsp

ecul

ar-v

iole

t-…gr

een-

fabr

icav

entu

rnin

esp

ecul

ar-g

reen

-…pi

nk-ja

sper

whi

te-fa

bric

whi

te-a

cryl

icgr

een-

plas

ticw

hite

-mar

ble

colo

nial

-map

le-2

23re

d-ph

enol

ical

umin

a-ox

ide

purp

le-p

aint

pear

l-pai

ntpi

nk-fe

ltgr

ay-p

last

icpu

re-ru

bber

pick

led-

oak-

260

blue

-acr

ylic

blac

k-so

ft-pl

astic

whi

te-d

iffus

e-bb

all

spec

ular

-ora

nge-

…ne

opre

ne-ru

bber

blue

-rubb

ersil

ver-p

aint

spec

ular

-blu

e-…

yello

w-m

atte

-…da

rk-s

pecu

lar-…

spec

ial-w

alnu

t-224

blue

-fabr

icda

rk-b

lue-

pain

tco

lor-c

hang

ing-

…gr

een-

met

allic

-…te

flon

fruitw

ood-

241

spec

ular

-…ni

ckel

cher

ry-2

35sp

ecul

ar-re

d-…

pvc

light

-red-

pain

tre

d-m

etal

lic-p

aint

spec

ular

-yel

low

-…pi

nk-fa

bric2

silve

r-met

allic

-…bl

ack-

obsid

ian

yello

w-p

aint

blac

k-ox

idize

d-…

gold

-pai

ntco

lor-c

hang

ing-

…go

ld-m

etal

lic-…

delri

nbl

ue-m

etal

lic-p

aint

whi

te-fa

bric2

blue

-met

allic

-…al

umin

ium

tung

sten

-car

bide

colo

r-cha

ngin

g-…

viol

et-a

cryl

icda

rk-re

d-pa

int

gold

-met

allic

-…he

mat

itena

tura

l-209

oran

ge-p

aint

two-

laye

r-gol

dgo

ld-m

etal

lic-p

aint

silve

r-met

allic

-…ch

rom

egr

een-

met

allic

-…al

um-b

ronz

eip

swich

-pin

e-22

1sp

ecul

ar-b

lack

-…sil

icon-

nitr

ade

two-

laye

r-silv

erbl

ack-

phen

olic

ss44

0re

d-pl

astic

red-

fabr

icst

eel

gree

n-la

tex

grea

se-c

over

ed-…

bras

sch

rom

e-st

eel

nylo

n

erro

rs [d

eg.]

A B CD

Elevation angle mapping Recovered 1D symmetry

A B C D

Fig. 8. Results for 100 BRDFs by using the 1D-sym method under roughly uniformly distributed light sources. Elevationangle re-mapping curves (blue: estimated; red: ground truth) and recovered symmetry data are shown.

Fig. 9. Average surface normal error maps of 100 BRDFSunder non-uniform lights by using: initialization (left), 1D-sym method (middle) and 2D-sym method (right).

TABLE 2Average surface normal/light source errors for 100

BRDFs under non-uniformly distributed light sources.

Method Init. err. Final err. Superior cases Light err.1D-sym 15.50 9.68 14× 2.63 5.96

2D-sym 15.50 6.58 86× 5.11 5.96

are roughly uniformly distributed. Therefore, results inTable 2 reflect the fact that the performance of the1D-sym method heavily depends on the light sourcedistribution, while the 2D-sym method has the merit inthat it can be applied in more general cases. For a moreintuitive understanding, we plot the average error mapsof surface normals produced by initialization, 1D-symand 2D-sym methods in Fig. 9 for a direct comparison.

The proposed method for light source estimation al-so works well for the non-uniformly distributed lightsources. As shown in Table 2, the average light sourceerror is 5.96, which is similar to and even smaller thanthat in the case of uniform lights. It indicates that ourlight source estimation method is robust to the changeof light source distribution and can produce consistentlyaccurate results.

For the 2D-sym method, we plot its error for eachBRDF in Fig. 10 and also show some representativeexamples of 2D BRDF symmetry before/after using the2D-sym method. The BRDF data (blue dots) are mappedto the ideal symmetric values (red dots) via optimization.Different from the results in Fig. 8, the 2D-sym methodcan better handle specular BRDFs, while it producesrelatively larger errors for 1) some BRDFs in the MERL

database that contain visually noticeable artifacts, and 2)some diffuse BRDFs that are ‘skipped’ without optimiza-tion since diffuse BRDFs do not show clear 2D symmetry(case D in Fig. 10).Comparison with Other Methods. We further compareour results with those of previous methods. The samedataset as above, i.e., hemispherical surfaces renderedwith 100 BRDFs and both 82 roughly uniform and 83non-uniform light directions, is used to conduct experi-ments for all methods. Because handling both unknownlight sources and unknown general BRDFs is very chal-lenging, it is not easy to find previous methods that cansolve this problem without additional assumptions toours. Eventually, we choose the following methods thatcan output reasonable results using the same input:

1) LowRank [20]+light: a low-rank minimization-basedmethod that robustly handles non-LambertianBRDFs. We provide it with our estimated light sourcesto produce the final output.

2) UPS-IP [42]: a state-of-the-art method for uncali-brated photometric stereo with general unknownBRDFs. Note that [42] proposes two methods andhere we refer to the one that does not requireadditional database as input.

3) LDR [43]: a recent method that solves uncalibratedphotometric stereo by analyzing diffuse maxima.

4) Entropy [14]: a well-known method that solvesuncalibrated photometric stereo via entropy min-imization.

Note that although [20] is designed to handle variousBRDFs, it needs known light sources to output accuratesolutions. For this purpose, we provide it with ourestimated light sources and denote the resulting methodas “LowRank+light”. This is unfair to other methods butit helps determine the upper bound performance of [20].

Average estimation accuracy of all methods are shownand compared in Table 3. The proposed 1D-sym and2D-sym methods achieve significantly better estimationaccuracies than other ones in both cases of 82 and 83 lightsources, while the 2D-sym method appears to be theonly one that can solve the problem accurately with non-




05

101520253035404550

pear

l-pai

ntco

loni

al-m

aple

-…al

umin

a-ox

ide

whi

te-a

cryl

icsp

ecul

ar-w

hite

-…sil

ver-p

aint

spec

ial-w

alnu

t-…fru

itwoo

d-24

1go

ld-p

aint

pick

led-

oak-

260

blac

k-ox

idize

d-…

red-

spec

ular

-…da

rk-b

lue-

pain

tgr

een-

met

allic

-…sil

ver-m

etal

lic-…

spec

ular

-…na

tura

l-209

whi

te-m

arbl

eye

llow

-phe

nolic

blac

k-so

ft-pl

astic

blue

-rubb

ergo

ld-m

etal

lic-…

blue

-met

allic

-…re

d-pl

astic

cher

ry-2

35ne

opre

ne-ru

bber

spec

ular

-red-

…sp

ecul

ar-y

ello

w-…

teflo

ngr

een-

plas

ticw

hite

-diff

use-

…ye

llow

-pai

ntda

rk-re

d-pa

int

gold

-met

allic

-…sil

ver-m

etal

lic-…

two-

laye

r-silv

erpi

nk-ja

sper

light

-red-

pain

tpu

re-ru

bber

mar

oon-

plas

ticav

entu

rnin

eor

ange

-pai

ntye

llow

-pla

stic

nylo

nde

lrin

spec

ular

-…ip

swich

-pin

e-22

1ye

llow

-mat

te-…

gray

-pla

stic

two-

laye

r-gol

dni

ckel

poly

ethy

lene

pink

-felt

pink

-pla

stic

alum

-bro

nze

colo

r-cha

ngin

g-…

red-

fabr

icvi

olet

-acr

ylic

blue

-met

allic

-…sp

ecul

ar-v

iole

t-…gr

een-

met

allic

-…pu

rple

-pai

ntvi

olet

-rubb

erbl

ack-

obsid

ian

hem

atite

colo

r-cha

ngin

g-…

red-

met

allic

-pai

ntal

umin

ium

pink

-fabr

ic2go

ld-m

etal

lic-…

colo

r-cha

ngin

g-…

ss44

0w

hite

-fabr

icw

hite

-pai

ntgr

een-

late

xsp

ecul

ar-b

lack

-…sp

ecul

ar-g

reen

-…bl

ue-fa

bric

pvc

red-

fabr

ic2ch

rom

ebe

ige-

fabr

icre

d-ph

enol

icgr

een-

fabr

icpi

nk-fa

bric

poly

uret

hane

-…tu

ngst

en-c

arbi

dest

eel

gree

n-ac

rylic

bras

sbl

ack-

phen

olic

blue

-acr

ylic

spec

ular

-blu

e-…

chro

me-

stee

lw

hite

-fabr

ic2da

rk-s

pecu

lar-…

silico

n-ni

trad

egr

ease

-cov

ered

-…bl

ack-

fabr

iclig

ht-b

row

n-fa

bric

erro

rs [d

eg.]

A B CD

A B C2D BRDF symmetry

NotApplicable

DInitialized Optimized

Fig. 10. Results for 100 BRDFs by using the 2D-sym method under non-uniformly distributed light sources. Examplesof the 2D BRDF symmetry (folded for easy illustration as in Fig. 3) before/after optimization are shown.

TABLE 3Comparison of average estimation accuracy for different methods with 100 BRDFs and 82/83 light sources.

Light sources 1D-sym 2D-sym LowRank [20]+light UPS-IP [42] LDR [43] Entropy [14]

82 (roughly uniform)Normal err. 6.0± 2.4 5.7± 1.9 13.7± 10.0 15.0± 8.4 13.1± 4.2 16.7± 5.6

Light err. 6.1± 2.0 6.1± 2.0 − − 15.1± 2.8 12.9± 4.7

83 (non-uniform)Normal err. 9.7± 4.6 6.6± 2.0 14.6± 11.4 16.8± 8.5 14.7± 4.5 22.7± 7.6

Light err. 6.0± 2.2 6.0± 2.2 − − 15.4± 3.2 21.9± 9.8

Surfa

ce n

orm

al e

rror [

deg.

]

Fig. 11. Comparison of average surface normal errorsproduced by different methods.

uniform illumination. Average results are also shownand compared in Fig. 11 for both 82 and 83 lights.Results for Complex 3D Models. We conduct experi-ments using seven synthetic 3D surfaces shown in Fig. 6.They are rendered with seven typical BRDFs under 83non-uniformly distributed light sources. Their normalmaps are recovered by using the method in [7] andthe proposed 1D-sym and 2D-sym methods. Table 4show the numerical results, including surface normalestimation errors and light source estimation errors. Bycomparing the surface normal estimation errors, we canagain confirm the best performance of the proposed 2D-sym method. Representative results produced by the 2D-sym method are further given in Fig. 12 for visualization,where the 3D surfaces are reconstructed by using thepoisson reconstruction algorithm [44].

6.2 Results for Real Scenes

We evaluate our method by using images captured inreal scenes. We use the recent datasets provided by

TABLE 4Results for 3D models with 10 BRDFs under 83 lights.

3D Model Init. [7] 1D-sym 2D-sym Light err.Bunny 21.2 11.9 6.6 10.2

Lion 7.9 7.9 7.9 10.5

Armadillo 18.8 20.1 9.1 6.8

Beethoven 22.0 6.9 5.2 7.3

Dragon 7.4 7.5 5.3 7.2

Buddha 10.7 7.3 5.5 7.1

Rabbit 14.4 12.0 10.0 7.4

Average 14.6 10.5 7.1 8.1

Ikehata et al. [21] and Shi et al. [45]. These datasetsare difficult for common photometric stereo methodssince the captured objects have non-lambertian or evenhigh specular BRDFs. Besides, handling some objectswith complex surface geometry and large concavity maysuffer from severe cast shadows. In our setting, we con-duct all the experiments without light source calibration,which makes the task far more difficult.

Experimental results are provided in Fig. 13, wherewe show the estimated surface normal maps, the depthmaps and the reconstructed 3D surfaces. These resultsdemonstrate the ability of our method to handle complex3D surfaces with general BRDFs. Some noise due to spec-ular highlights exists in the recovered surface normalmap, e.g., for the 4-th and 7-th objects, whose positionsare consistent with those of the specular highlights un-der various light source directions. Note that the lightsources used to produce the input images do not coverthe whole visible hemisphere and this does not meet ourprevious assumption in Section 5. To solve this problem,we just let the method know the maximum light source




One

of i

nput

im

ages

Estim

ated

sur

face

no

rmal

map

Angu

lar e

rror 40

0

3D s

urfa

ce

reco

nstru

ctio

n

Fig. 12. Results for synthetic surfaces under 83 non-uniform lights. They are produced by using the 2D-sym method.More details and comparisons between different methods are provided in Table 4.

One

of i

nput

im

ages

Estim

ated

norm

al m

apEs

timat

ed

dept

h m

ap3D

sur

face

re

cons

truct

ion

Fig. 13. Results for real objects. They are produced by using the 2D-sym method. Quantitative comparisons withother methods are provided in Table 5.




TABLE 5Estimation accuracy on real world data.

Scene Ours AM07 [14] SM10 [6] WT13 [46] LM13 [8] PF14 [43]

Cow 15.0 54.7 22.7 22.5 19.8 19.5

Goblet 18.3 46.5 48.8 29.2 20.6 29.9

Reading 22.3 53.7 27.0 48.2 59.0 24.2

Harvest 28.0 61.7 73.9 34.5 55.5 29.2

Average-C 20.9 54.2 43.1 33.6 38.7 25.7

Ball 9.3 7.3 8.9 4.4 22.4 4.8

Cat 12.6 31.4 19.8 36.6 25.0 9.5

Pot1 12.4 18.4 16.7 9.4 32.8 9.5

Bear 10.9 16.8 12.0 6.4 15.4 9.1

Pot2 15.7 49.2 50.7 14.5 20.6 15.9

Buddha 19.0 32.8 15.5 13.2 25.8 14.9

Average-all 16.3 37.2 29.6 23.9 27.6 16.7

Surfa

ce n

orm

al e

rror [

deg.

]

Fig. 14. Comparison of surface normal errors and thestandard deviations on the most challenging scenes.

variation range to scale the solution. Such a simple hintallows the method to work properly.

We also conduct quantitative comparisons with recentphotometric stereo methods. In particular, we use all ofthe ten real scenes from the latest DiLiGenT dataset [45].Ground truth data and the performances of other meth-ods are provided by [45], and we compare these methodswith ours and summarize the results in Table 5.

Meanwhile, we emphysize on four particular scenes,i.e., Cow, Goblet, Reading and Harvest, which are con-sidered as ‘challenging’ cases. These scenes are the mostchallenging ones due to their highly non-lambertianBRDFs and complexity in geometry, and thus exist-ing method all perform poorly on them (refer to the‘Average-C’ row). For these scenes, our method reportsan average error of of 20.9, which is significantly small-er than any other methods. A more intuitive comparisoncan be found in Fig. 14, and it confirms the reasonablygood visual quality in Fig. 13.

7 DISCUSSION AND CONCLUSION

In this paper, we propose constrained half-vector sym-metry and its 1D and 2D representations, which revealthe fact that the observed BRDF values in a 1D sliceor the 2D normal space have a symmetric distribution

about the half vector. By using the collected symmetricdata, we develop two efficient and accurate modelingand solution methods for surface normal estimation.Our method also recovers light sources robustly in thecase of general isotropic BRDFs, and this helps ourmethods work in an uncalibrated manner. Experimentalresults confirm that our methods can handle the mostchallenging problem of uncalibrated photometric stereowith general isotropic reflectances and produce state-of-the-art results. Future works include eliminating theeffects caused by cast shadows and inter-reflections toimprove the performance for real scenes.

REFERENCES[1] R. Woodham, “photometric method for determining surface ori-

entation from multiple images,” Optical Engineering, vol. 1, no. 7,pp. 139–144, 1980.

[2] P. N. Belhumeur, D. J. Kriegman, and A. L. Yuille, “The bas-reliefambiguity,” Int’l Journal of Computer Vision, vol. 35, no. 1, pp. 33–44, 1999.

[3] P. Tan, L. Quan, and T. Zickler, “The geometry of reflectance sym-metries,” IEEE Trans. on Pattern Analysis and Machine Intelligence,vol. 33, no. 12, pp. 2506–2520, 2011.

[4] Z. Wu and P. Tan, “Calibrating photometric stereo by holisticreflectance symmetry analysis,” in Proc. of IEEE Conf. on ComputerVision and Pattern Recognition (CVPR), 2013.

[5] P. Favaro and T. Papadhimitri, “A closed-form solution to uncal-ibrated photometric stereo via diffuse maxima,” in Proc. of IEEEConf. on Computer Vision and Pattern Recognition (CVPR), 2012, pp.821–828.

[6] B. Shi, Y. Matsushita, Y. Wei, C. Xu, and P. Tan, “Self-calibratingphotometric stereo,” in Proc. of IEEE Conf. on Computer Vision andPattern Recognition (CVPR), 2010, pp. 1118–1125.

[7] I. Sato, T. Okabe, Q. Yu, and Y. Sato, “Shape reconstruction basedon similarity in radiance changes under varying illumination,” inProc. of Int’l Conf. on Computer Vision (ICCV), 2007, pp. 1–8.

[8] F. Lu, Y. Matsushita, I. Sato, T. Okabe, and Y. Sato, “Uncalibratedphotometric stereo for unknown isotropic reflectances,” in Proc.of IEEE Conf. on Computer Vision and Pattern Recognition (CVPR),2013.

[9] M. Chandraker, J. Bai, and R. Ramamoorthi, “A theory of differ-ential photometric stereo for unknown isotropic brdfs,” in Proc.of IEEE Conf. on Computer Vision and Pattern Recognition (CVPR),2011, pp. 2505–2512.

[10] O. Drbohlav and R. Sara, “Specularities reduce ambiguity ofuncalibrated photometric stereo,” in Proc. of European Conf. onComputer Vision (ECCV), 2002, pp. 644–645.

[11] O. Drbohlav and M. Chaniler, “Can two specular pixels calibratephotometric stereo?” in Proc. of Int’l Conf. on Computer Vision(ICCV), 2005, pp. 1850–1857.

[12] M. Chandraker, F. Kahl, and D. Kriegman, “Reflections on thegeneralized bas-relief ambiguity,” in Proc. of IEEE Conf. on Com-puter Vision and Pattern Recognition (CVPR), 2005, pp. 788–795.

[13] R. Basri, D. Jacobs, and I. Kemelmacher, “Photometric stereowith general, unknown lighting,” Int’l Journal of Computer Vision,vol. 72, no. 3, pp. 239–257, 2007.

[14] N. Alldrin, S. Mallick, and D. Kriegman, “Resolving the general-ized bas-relief ambiguity by entropy minimization,” in Proc. ofIEEE Conf. on Computer Vision and Pattern Recognition (CVPR),2007.

[15] Z. Zhou and P. Tan, “Ring-light photometric stereo,” in Proc. ofEuropean Conf. on Computer Vision (ECCV), 2010, pp. 265–279.

[16] T. Papadhimitri and P. Favaro, “A new perspective on uncalibrat-ed photometric stereo,” in Proc. of IEEE Conf. on Computer Visionand Pattern Recognition (CVPR), 2013.

[17] S. Barsky and M. Petrou, “The 4-source photometric stereo tech-nique for three-dimensional surfaces in the presence of highlightsand shadows,” IEEE Trans. on Pattern Analysis and Machine Intel-ligence, vol. 25, no. 10, pp. 1239–1252, 2003.

[18] H.-S. Chung and J. Jia, “Efficient photometric stereo on glossysurfaces with wide specular lobes,” in Proc. of IEEE Conf. onComputer Vision and Pattern Recognition (CVPR), 2008, pp. 1–8.




[19] D. Miyazaki, K. Hara, and K. Ikeuchi, “Median photometric stereoas applied to the segonko tumulus and museum objects,” Int’lJournal of Computer Vision, vol. 86, no. 2, pp. 229–242, 2010.

[20] L. Wu, A. Ganesh, B. Shi, Y. Matsushita, Y. Wang, and Y. Ma,“Robust photometric stereo via low-rank matrix completion andrecovery,” in Proc. of Asian Conf. on Computer Vision (ACCV), 2010,pp. 703–717.

[21] S. Ikehata, D. Wipf, Y. Matsushita, and K. Aizawa, “Robustphotometric stereo using sparse regression,” in Proc. of IEEE Conf.on Computer Vision and Pattern Recognition (CVPR), 2012, pp. 318–325.

[22] T. Wu, K. Tang, C. Tang, and T. Wong, “Dense photometric stereo:A markov random field approach,” IEEE Trans. on Pattern Analysisand Machine Intelligence, vol. 28, no. 11, pp. 1830–1846, 2006.

[23] N. Alldrin and D. Kriegman, “Toward reconstructing surfaceswith arbitrary isotropic reflectance : A stratified photometricstereo approach,” in Proc. of Int’l Conf. on Computer Vision (ICCV),2007, pp. 1–8.

[24] M. Holroyd, J. Lawrence, G. Humphreys, and T. Zickler, “A pho-tometric approach for estimating normals and tangents,” ACMTransactions on Graphics, vol. 27, p. 133, 2008.

[25] T. Higo, Y. Matsushita, and K. Ikeuchi, “Consensus photometricstereo,” in Proc. of IEEE Conf. on Computer Vision and PatternRecognition (CVPR), 2010, pp. 1157–1164.

[26] B. Shi, P. Tan, Y. Matsushita, and K. Ikeuchi, “Elevation angle fromreflectance monotonicity: Photometric stereo for general isotropicreflectances,” in Proc. of European Conf. on Computer Vision (ECCV),2012.

[27] D. Goldman, B. Curless, A. Hertzmann, and S. Seitz, “Shape andspatially-varying brdfs from photometric stereo,” in Proc. of Int’lConf. on Computer Vision (ICCV), 2005, pp. 341–348.

[28] S. Nayar, K. Ikeuchi, and T. Kanade, “Determining shape andreflectance of hybrid surfaces by photometric sampling,” IEEETransactions on Robotics and Automation, vol. 6, no. 4, pp. 418–431,1990.

[29] N. Alldrin, T. Zickler, and D. Kriegman, “Photometric stereo withnon-parametric and spatially-varying reflectance,” in Proc. of IEEEConf. on Computer Vision and Pattern Recognition (CVPR), 2008, pp.1–8.

[30] A. Hertzmann and S. Seitz, “Example-based photometric stereo:shape reconstruction with general, varying BRDFs,” IEEE Trans.on Pattern Analysis and Machine Intelligence, vol. 27, no. 8, pp. 1254–1264, 2005.

[31] P. Ren, J. Wang, J. Snyder, X. Tong, and B. Guo, “Pocket reflec-tometry,” in Proc. of ACM SIGGRAPH, 2011, pp. 45:1–45:10.

[32] A. Georghiades, “Incorporating the torrance and sparrow modelof reflectance in uncalibrated photometric stereo,” in Proc. of Int’lConf. on Computer Vision (ICCV), 2003, pp. 816–823.

[33] S. Lombardi and K. Nishino, “Single image multimaterial esti-mation,” in Proc. of IEEE Conf. on Computer Vision and PatternRecognition (CVPR), 2011, pp. 238–245.

[34] K. Midorikawa, T. Yamasaki, and K. Aizawa, “Uncalibrated pho-tometric stereo by stepwise optimization using principal compo-nents of isotropic brdfs,” in Proc. of IEEE Conf. on Computer Visionand Pattern Recognition (CVPR), 2016.

[35] T. Okabe, I. Sato, and Y. Sato, “Attached shadow coding: Estimat-ing surface normals from shadows under unknown reflectanceand lighting conditions,” in Proc. of Int’l Conf. on Computer Vision(ICCV), 2009, pp. 1693–1700.

[36] S. M. Rusinkiewicz, “A new change of variables for efficient brdfrepresentation,” in Rendering techniques 98. Springer, 1998, pp.11–22.

[37] S. R. Marschner, “Inverse rendering for computer graphics,” Ph.D.dissertation, Cornell University, 1998.

[38] W. Matusik, H. Pfister, M. Brand, and L. McMillan, “A data-drivenreflectance model,” in Proc. of ACM SIGGRAPH, 2003, pp. 27–31.

[39] H. Winnemoller, A. Mohan, J. Tumblin, and B. Gooch, “Lightwaving: Estimating light positions from photographs alone,”Computer Graphics Forum, vol. 24, pp. 433–438, 2005.

[40] J. Tenenbaum, V. De Silva, and J. Langford, “A global geometricframework for nonlinear dimensionality reduction,” Science, vol.290, no. 5500, pp. 2319–2323, 2000.

[41] F. Lu, I. Sato, and Y. Sato, “Uncalibrated photometric stereo basedon elevation angle recovery from brdf symmetry of isotropicmaterials,” in Proc. of IEEE Conf. on Computer Vision and PatternRecognition (CVPR), 2015.

[42] F. Lu, Y. Matsushita, I. Sato, T. Okabe, and Y. Sato, “From intensityprofile to surface normal: Photometric stereo for unknown lightsources and isotropic reflectances,” IEEE Trans. on Pattern Analysisand Machine Intelligence, vol. 37, no. 10, pp. 1999–2012, 2015.

[43] T. Papadhimitri and P. Favaro, “A closed-form, consistent androbust solution to uncalibrated photometric stereo via local dif-fuse reflectance maxima,” Int’l Journal of Computer Vision, vol. 107,no. 2, pp. 139–154, 2014.

[44] A. Agrawal, R. Raskar, and R. Chellappa, “What is the range ofsurface reconstructions from a gradient field?” in Proc. of EuropeanConf. on Computer Vision (ECCV), 2006, pp. 578–591.

[45] B. Shi, Z. Wu, Z. Mo, D. Duan, S.-K. Yeung, , and P. Tan, “A bench-mark dataset and evaluation for non-lambertian and uncalibratedphotometric stereo,” in Proc. of IEEE Conf. on Computer Vision andPattern Recognition (CVPR), 2016.

[46] Z. Wu and P. Tan, “Calibrating photometric stereo by holisticreflectance symmetry analysis,” in Proc. of IEEE Conf. on ComputerVision and Pattern Recognition (CVPR), 2013, pp. 1498–1505.

Feng Lu received the B.S. and M.S. degreesin automation from Tsinghua University, in 2007and 2010, respectively, and the Ph.D. degreein information science and technology from TheUniversity of Tokyo, in 2013. He is currentlya Professor with the State Key Laboratory ofVirtual Reality Technology and Systems, Schoolof Computer Science and Engineering, BeihangUniversity. His research interests including 3Dshape recovery, reflectance analysis, and hu-man gaze analysis, .

Xiaowu Chen received the Ph.D. degree in com-puter science from Beihang University, Beijing,China, in 2001. He is currently a Professor withthe State Key Laboratory of Virtual Reality Tech-nology and Systems, School of Computer Sci-ence and Engineering, Beihang University. Hiscurrent research interests include virtual reality,computer graphics, and computer vision.

Imari Sato received her B.S. degree in policymanagement from Keio University in 1994 andM.S. and Ph.D. degrees in interdisciplinary in-formation studies from the University of Tokyo in2002 and 2005. She is a professor at the Nation-al Institute of Informatics. Her research interestsinclude physics-based vision and image-basedmodeling.

Yoichi Sato received the BS degree from theUniversity of Tokyo in 1990, and the MS and PhDdegrees in robotics from the School of ComputerScience, Carnegie Mellon University, in 1993and 1997, respectively. He is a professor at theInstitute of Industrial Science, the University ofTokyo. His research interests include physics-based vision, reflectance analysis, and gaze andgesture analysis. He served/is serving in sever-al conference organization and journal editorialroles including IEEE Transactions on PAMI and

International Journal of Computer Vision.

JOURNAL OF LA SymPS: BRDF Symmetry Guided ...ysato/papers/Lu-TPAMI17.pdfA BRDF measures the ratio of...

Documents

Transcript of JOURNAL OF LA SymPS: BRDF Symmetry Guided ...ysato/papers/Lu-TPAMI17.pdfA BRDF measures the ratio of...