Optics and Lasers in Engineering 42 (2004) 461–468

Surface shape estimation fromphotometric images

Jes !us Villaa, Juan B. Hurtado-Ramosb,*a Laboratorio de Procesamiento Digital de Senales, Unidad de Ingenier!ıa El!ectrica,

Universidad Aut !onoma de Zacatecas, C.P. 98000. Zacatecas, Mexicob Centro de Ingenier!ıa y desarrollo Industrial, Av. Playa Pie de la Cuesta #702,

Desarrollo Habitacional San Pablo, C.P. 76130, Quer!etaro, Qro., Mexico

Received 16 June 2003; accepted 30 December 2003

Abstract

One of the most studied techniques for recovering surface shapes using a computer vision

system is the photometric. This method is based on the analysis of one or several images of an

object illuminated from a known direction . This kind of images can be considered reflectance

maps that give us information of the surface gradient. In computer vision, the problem of

recovering 3D information from shaded images is considered an inverse problem. To integrate

surface gradient information it is proposed a regularization technique that gives a stable

solution of the inverse problem and allows the possibility of reducing errors caused by noise.

Results applied on synthetic and real experimental images are presented.

r 2004 Elsevier Ltd. All rights reserved.

Keywords: Shape from shading; Regularization

1. Introduction

Extraction of 3D information from 2D images is a fundamental problem incomputer vision. There are different methodologies to realize this task. Among themost known methods is the so-called shape from shading or photometric [1–3]. Thistechnique is based on three main factors: the shape of the object, the reflectanceproperties of the surface and the illumination, i.e. the light source position. Becausephysical parameters of reflectance and illumination direction are known, the shape

ARTICLE IN PRESS

*Corresponding author.

E-mail address: jhurtado@cidesi.mx (J.B. Hurtado-Ramos).

0143-8166/$ - see front matter r 2004 Elsevier Ltd. All rights reserved.

doi:10.1016/j.optlaseng.2003.12.004

of the object can be obtained considering that the brightness at every point in theimage depends on the surface reflectance and orientation at that point.

A number of shape from shading techniques have been developed in the last threedecades. Some of them use a single image for recovering 3D information [4,5],however, good results with these methods are restricted to simple objects and idealexperimental conditions, mainly because of the lack of a robust solutionmethodology. Other researchers have developed techniques using multiple imagesfor more reliable results. Works have also been made using the photometric stereomethod for deformation detection [6].

In general, the application of any method mentioned above does not give theheight distribution itself but the gradient of the surface. For obtaining the heightdistribution an integration method is required. However, the gradient integrationhas a key problem: the orientations are not always consistent, which makes itdifficult to define an appropriate relation for slope-to-height conversion. Thisproblem, called the integrability problem [2], can be analyzed from a differentperspective. Remember that it is an inverse problem, where it is required to recover3D information from 2D arrays (i.e., images). Essentially, this inverse problem isconsidered ill-posed, because observations (i.e., the slopes) does not determine thevalue of the height distribution in a unique an stable way [7] due to the loss ofinformation during the imaging process that projects a three-dimensional world intotwo-dimensional images. Lee and Kuo [3] reported work that alleviates in part theintegrability problem by using a quadratic cost functional with nodal basis-functionsfor representation of the surface, however, the method is not robust enough in thepresence of noise or in the case in which data is missing or is unreliable because itdoes not takes into account prior information of the solution.

Existence of an ill-posed problem suggests a regularization method to solve it. Thekey idea to solve an ill-posed problem is to restrict the possible solutions byintroducing prior knowledge. In this work, it is proposed a regularization approachto solve the integration problem in the shape from shading technique with threeimages. This method provides suitable results and allows the possibility of reduceerrors caused by noise.

2. Shape from shading with three images

Assuming an object whose surface is Lambertian and is illuminated from threedifferent directions (light source is supposed to be sufficiently away from theobject in such a way that rays of light illuminating it can be considered parallel).Three different images are then recorded each of them having a differentillumination direction. The intensities of the three recorded images are functionsproportional to the reflectance maps that provide information of the relationshipbetween the shape gradient and the irradiance. The three images can be representedas [1]

Ek ¼ rð#sk � #nÞ; k ¼ 1; 2; 3; ð1Þ

ARTICLE IN PRESSJes !us Villa, J.B. Hurtado-Ramos / Optics and Lasers in Engineering 42 (2004) 461–468462

where r is the albedo of the surface and

#sk ¼ð�pk;�qk; 1Þ

T

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi1 þ p2

k þ q2k

q ; k ¼ 1; 2; 3; ð2Þ

#n ¼ð�p;�q; 1ÞTffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi

1 þ p2 þ q2p ; ð3Þ

where #n denotes the unit vector of the gradient of the height z (i.e.,p ¼ @z=@x; q ¼ @z=@y) for each object’s surface point and #sk the unit vector definingthe slope of an imaginary plane whose normal is the illumination source’s directionand is used to define the light source position. Combining Eqs. (1–3) we rewriteE ¼ rS#n; where the matrix S contains the unit vectors #sk; and components of vectorE are the three intensities Ek: Thus, assuming that S is not singular, the gradient ofthe surface is obtained with

r#n ¼ S�1E; ð4Þ

where

S ¼

#s1

#s2

#s3

264

375: ð5Þ

3. Integration of the gradient

As explained before, gradient integration is an ill-posed problem that must beregularized to obtain proper results. Regularization of linear problems has beenstrongly studied. Tikhonov [8] analyzed the solution of the problem of finding z fromany given data set g (e.g. gradient data) which are related in the following way:

g ¼ Az; ð6Þ

where A is a noninjective linear operator. In that work it is proposed that theregularized solution can be obtained by finding the value of z that minimizes thefunctional

UðzÞ ¼ jjAz � gjj2 þ mjjPzjj2; ð7Þ

where m is the so-called regularization parameter, and P is the regularizationoperator that is formed by a linear combination of derivatives.

Following the same reasoning on (7), a regularization procedure for obtaining theheight distribution in the shape from shading problem can be applied considering theoperator A as a first order difference, so that

jjAz � gjj2 ¼Xði;jÞAL

½zi;j � zi�1;j � pi;j2 þXði;jÞAL

½zi;j � zi;j�1 � qi;j2; ð8Þ

ARTICLE IN PRESSJes !us Villa, J.B. Hurtado-Ramos / Optics and Lasers in Engineering 42 (2004) 461–468 463

where (i,j) represents the coordinates in the image, zi,j the height distribution to beestimated and L a finite regular lattice where the observations are available.

As can be seen, for the case m ¼ 0 this problem corresponds to the least-squaresprocedure of integration [9,10] and height distribution could be obtained minimizingthis cost functional with respect to zi,j, however, the least-squares procedure does notalways provide suitable results because it assumes a noiseless information and theobservations ðp; qÞ does not determine the value of the field zi,j in an unique andstable way, mainly because the algorithm does not include any information about thesolution, e.g., smoothness of the estimated signal.

In the proposed regularization procedure it is assumed ma0 which restricts thesolution to have specific characteristics, i.e., to be smooth. If it is assumed that theoperator P is a finite approximation to the second-order partial derivatives, thesecond term of Eq. (7) becomes

mjjPzjj2 ¼ mXði;jÞAL

½ziþ1;j � 2zi;j þ zi�1;j 2 þ mXði;jÞAL

½zi;jþ1 � 2zi;j þ zi;j�12: ð9Þ

In this way the solution is restricted to have smooth derivatives. Parameter mcontrols the smoothness of the field to be estimated and its value may depend on theamount of noise. There is not a precise procedure to determine the value of thisparameter in a given problem so that it is selected intuitively [11].

Finally, the estimated field zi,j is the minimizer of the cost function (7), so thattaking the gradient of U(z) with respect to zi,j and equating it to zero:

@UðzÞ@zi;j

¼ 2 DULS þ DURð Þ ¼ 0; ð10Þ

where

DULS ¼ ½zi;j � zi�1;j � pi;j � ½ziþ1;j � zi;j � piþ1;j

þ ½zi;j � zi;j�1 � qi;j � ½zi;jþ1 � zi;j � pi;jþ1 ð11Þ

and

DUR ¼ � m½6zi;j � 4ðziþ1;j þ zi�1;jÞ þ ziþ2;j þ zi�2;j

� m½6zi;j þ 4ðzi;jþ1 þ zi;j�1Þ þ zi;jþ2 þ zi;j�2: ð12Þ

One can obtain the minimizer zi,j solving the linear system (10) by applying anyiterative algorithm from the literature [12].

The minimizer of Eq. (12) smoothes out the observations, hence, the minimizationoperation can also be considered a low-pass filtering operation. To analyze thefrequency response of the minimization it is taken the discrete Fourier transform ofEq. (12) only in the i direction (i.e., considering only the slope p for simplicity)obtaining the transfer function that is expressed as

HðoÞ ¼ZðoÞPðoÞ

¼1 � expðioÞ

2 � 2 cosðoÞ þ m½8 cosðoÞ � 2 cosð2oÞ � 6; ð13Þ

which represents a low-pass filter whose cut-frequency is controlled by parameter m:A graphic of the normalized response for different values of m is shown in Fig. 1. As

ARTICLE IN PRESSJes !us Villa, J.B. Hurtado-Ramos / Optics and Lasers in Engineering 42 (2004) 461–468464

can be seen the analytic response has an asymptote for o ¼ 0 which is an expectedresult for p=0. On the other hand, in the particular case of least squares (i.e, m ¼ 0)the transfer function is represented by

HðoÞ ¼1 � expðioÞ2 � 2 cosðoÞ

: ð14Þ

4. Experimental results

Performance of the proposed technique was tested in two experiments. The first onewas a numerical estimation with synthetic images of reflectance maps using thesimulated sphere shown in the contour map of Fig. 2(a). The images were corruptedwith –5 dB of uniformly distributed noise then, the gradient (p,q) was calculated usingEq. (4). The height distribution obtained applying least-squares (m ¼ 0) is shown inFig. 2(b). Fig. 2(c) shows the results applying the proposed technique with m ¼ 3:

In a second experiment an estimation of the height distribution of a pottery objectwas made. In the experiment the values p1=0.36, p2=0, p3=�0.36 and q1=0.05,q2=0.36, q3=0.05 for the positions of the light source were used. The resultingimages of 250 250 pixels are shown in Fig. 3(a)–(c). The contour map of the heightdistribution applying the proposed technique is shown in Fig. 3(d).

5. Conclusions

Recovering of height distribution from gradients using the shape from shadingtechnique is an ill-posed problem which requires the introduction of prior

ARTICLE IN PRESS

Fig. 1. Frequency response of the minimization operation for different values of m:

Jes !us Villa, J.B. Hurtado-Ramos / Optics and Lasers in Engineering 42 (2004) 461–468 465

ARTICLE IN PRESS

Fig. 2. (a) Contour map of the height distribution of a simulated sphere. Estimated shape using (b) the

simple least-squares (m ¼ 0) and (c) the proposed method for m ¼ 3:

Jes !us Villa, J.B. Hurtado-Ramos / Optics and Lasers in Engineering 42 (2004) 461–468466

information about the behavior of the solution to solve it, i.e., regularizing thesolution. In this work it was presented a procedure for gradient integration in shapefrom shading technique with three images. As shown, this inverse problem can beeasily solved by using a regularization technique that improves the estimationsobtained in the particular case of the least-squares method. It was explained and shownthat the technique allows the possibility of reduce errors caused by noise. Results of theproposed algorithm were presented with simulated and real experiments.

Acknowledgements

Authors would like to acknowledge the financial support of the Centro deIngenier!ıa y Desarrollo Industrial (Quer!etaro, M!exico) to complete the present work.

ARTICLE IN PRESS

Fig. 3. Shaded images of a pottery object using the source positions described by the slopes (a) p1=0.36,

q1=0.05; (b) p2=0, q2=0.36; (c) p3=�0.36, q3=0.05. (d) contour map of the estimated height distribution

(all dimensions are in pixels).

Jes !us Villa, J.B. Hurtado-Ramos / Optics and Lasers in Engineering 42 (2004) 461–468 467

References

[1] Horn BKP. Robot vision. Cambridge, MA: MIT Press; 1986 [Chapters 10,11].

[2] Horn BKP. Height and gradient from shading. Int J Comput Vision 1990;5:584–95.

[3] Lee KM, Kuo J. Surface reconstruction from photometric stereo images. J Opt Soc Am 1993;10:

855–68.

[4] Zheng Q, Chellapa R. Estimation of illumination direction, albedo, and shape from shading. IEEE

Trans Pattern Anal Mach Intell 1991;13:680–702.

[5] Szeliski R. Fast surface interpolation using hierarchical basis functions. IEEE Trans Pattern Anal

Mach Intell 1990;12:513–28.

[6] Carver AE, Schalkoff RJ. Deformation detection on composite reflectance surfaces using two image

photometric stereo. Opt Eng 2001;40:295–302.

[7] Hadamard J. Sur les problems aux derives partielles et leur signification physique. Princeton

University Bulletin 13. Princeton, NJ: Princeton University; 1902.

[8] Tikhonov AN. Solution of incorrectly formulated problems and the regularization method. Sov Math

Dock 1963;4:1035–8.

[9] Fried DL. Least-squares fitting a wave front distortion estimate to an array of phase difference

measurements. J Opt Soc Am 1977;67:370–5.

[10] Hudgin RH. Wave-front reconstruction for compensated imaging. J Opt Soc Am 1977;67:375–8.

[11] Bertero M, Boccacci P. Introduction to inverse problems in imaging. London: Institute of Physics;

1998 [Chapter 5].

[12] Gollub GH, Van Loan CF. Matrix computations. Baltimore, MD: Johns Hopkins U. Press; 1990.

ARTICLE IN PRESSJes !us Villa, J.B. Hurtado-Ramos / Optics and Lasers in Engineering 42 (2004) 461–468468